i2vgen xl

Providerali-vilab
Categorytext-to-video
Licensemit
Downloads35.4K
Stars0

Overview

i2vgen-xl is a high-fidelity image-to-video diffusion model designed to animate static images with strong temporal consistency. Unlike basic text-to-video generators, it focuses on precise motion control based on a reference frame, making it ideal for developers building cinematic tools, automated social media content, or dynamic UI elements. It handles complex motion trajectories better than many open-source alternatives, reducing the 'morphing' effect common in latent video diffusion. Integration is straightforward for those familiar with the Diffusers library, allowing for scalable deployment in creative pipelines where visual stability and high resolution are critical.

Highlights

  • High-fidelity image-to-video animation with temporal stability
  • MIT license allows for flexible commercial integration
  • Reduced visual artifacts compared to standard diffusion models
  • Optimized for cinematic motion and realistic frame transitions

Usage

Install
# Install Hugging Face transformers
pip install transformers torch
SDK Usage
# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("ali-vilab/i2vgen-xl")
tokenizer = AutoTokenizer.from_pretrained("ali-vilab/i2vgen-xl")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance 
pip install -U huggingface_hub

CLI Download

Download the full repository

Download the full repository 
huggingface-cli download ali-vilab/i2vgen-xl

Download a single file to a local folder (e.g. config.json into ./dir)

Download a single file to a local folder (e.g. config.json into ./dir) 
huggingface-cli download ali-vilab/i2vgen-xl config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

SDK Download 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('ali-vilab/i2vgen-xl')

Git Download

Make sure git-lfs is installed first

Git Download 
git lfs install
git clone https://huggingface.co/ali-vilab/i2vgen-xl

To skip LFS large-file downloads, use:

Skip LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/ali-vilab/i2vgen-xl

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

Install Transformers 
pip install -U transformers torch

Load the model and run inference

Load the model and run inference 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('ali-vilab/i2vgen-xl')
tokenizer = AutoTokenizer.from_pretrained('ali-vilab/i2vgen-xl')

Full Documentation

来源: HuggingFace

---
license: mit
tags:

  • image-to-video

pipeline_tag: text-to-video
---

VGen

!figure1

VGen is an open-source video synthesis codebase developed by the Tongyi Lab of Alibaba Group, featuring state-of-the-art video generative models. This repository includes implementations of the following methods:

  • [A Recipe for Scaling up Text-to-Video Generation with Text-free Videos]()
  • [InstructVideo: Instructing Video Diffusion Models with Human Feedback]()

VGen can produce high-quality videos from the input text, images, desired motion, desired subjects, and even the feedback signals provided. It also offers a variety of commonly used video generation tools such as visualization, sampling, training, inference, join training using images and videos, acceleration, and more.

<a href='https://i2vgen-xl.github.io/'><img src='https://img.shields.io/badge/Project-Page-Green'></a> <a href='https://arxiv.org/abs/2311.04145'><img src='https://img.shields.io/badge/Paper-Arxiv-red'></a> ![YouTube](https://youtu.be/XUi0y7dxqEQ) <a href='https://cloud.video.taobao.com/play/u/null/p/1/e/6/t/1/441039979087.mp4'><img src='source/logo.png'></a>

🔥News!!!

  • __[2023.12]__ We release the high-efficiency video generation method VideoLCM
  • __[2023.12]__ We release the code and model of I2VGen-XL and the ModelScope T2V
  • __[2023.12]__ We release the T2V method HiGen and customizing T2V method DreamVideo.
  • __[2023.11]__ We release a high-quality I2VGen-XL model, please refer to the Webpage

TODO

  • [x] Release the technical papers and webpage of I2VGen-XL
  • [x] Release the code and pretrained models that can generate 1280x720 videos
  • [ ] Release models optimized specifically for the human body and faces
  • [ ] Updated version can fully maintain the ID and capture large and accurate motions simultaneously
  • [ ] Release other methods and the corresponding models

Preparation

The main features of VGen are as follows:

  • Expandability, allowing for easy management of your own experiments.

  • Completeness, encompassing all common components for video generation.

  • Excellent performance, featuring powerful pre-trained models in multiple tasks.

Installation

code
conda create -n vgen python=3.8
conda activate vgen
pip install torch==1.12.0+cu113 torchvision==0.13.0+cu113 torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cu113
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple

Datasets

We have provided a demo dataset that includes images and videos, along with their lists in `data.

*Please note that the demo images used here are for testing purposes and were not included in the training.*

Clone codeb

code
git clone https://github.com/damo-vilab/i2vgen-xl.git
cd i2vgen-xl

Getting Started with VGen

(1) Train your text-to-video model

Executing the following command to enable distributed training is as easy as that.

code
python train_net.py --cfg configs/t2v_train.yaml

In the t2v_train.yaml configuration file, you can specify the data, adjust the video-to-image ratio using frame_lens, and validate your ideas with different Diffusion settings, and so on.

  • Before the training, you can download any of our open-source models for initialization. Our codebase supports custom initialization and grad_scale settings, all of which are included in the Pretrain item in yaml file.
  • During the training, you can view the saved models and intermediate inference results in the workspace/experiments/t2v_traindirectory.

After the training is completed, you can perform inference on the model using the following command.

code
python inference.py --cfg configs/t2v_infer.yaml

Then you can find the videos you generated in the workspace/experiments/test_img_01 directory. For specific configurations such as data, models, seed, etc., please refer to the t2v_infer.yaml file.

<!-- <table>
<center>
<tr>
<td ><center>
<video muted="true" autoplay="true" loop="true" height="260" src="https://cloud.video.taobao.com/play/u/null/p/1/e/6/t/1/441754174077.mp4"></video>
</center></td>
<td ><center>
<video muted="true" autoplay="true" loop="true" height="260" src="https://cloud.video.taobao.com/play/u/null/p/1/e/6/t/1/441138824052.mp4"></video>
</center></td>
</tr>
</center>
</table>
</center> -->

<table>
<center>
<tr>
<td ><center>
<image height="260" src="https://img.alicdn.com/imgextra/i4/O1CN01Ya2I5I25utrJwJ9Jf_!!6000000007587-2-tps-1280-720.png"></image>
</center></td>
<td ><center>
<image height="260" src="https://img.alicdn.com/imgextra/i3/O1CN01CrmYaz1zXBetmg3dd_!!6000000006723-2-tps-1280-720.png"></image>
</center></td>
</tr>
<tr>
<td ><center>
<p>Clike <a href="https://cloud.video.taobao.com/play/u/null/p/1/e/6/t/1/441754174077.mp4">HRER</a> to view the generated video.</p>
</center></td>
<td ><center>
<p>Clike <a href="https://cloud.video.taobao.com/play/u/null/p/1/e/6/t/1/441138824052.mp4">HRER</a> to view the generated video.</p>
</center></td>
</tr>
</center>
</table>
</center>

(2) Run the I2VGen-XL model

(i) Download model and test data:

code
!pip install modelscope
from modelscope.hub.snapshot_download import snapshot_download
model_dir = snapshot_download('damo/I2VGen-XL', cache_dir='models/', revision='v1.0.0')

(ii) Run the following command:

code
python inference.py --cfg configs/i2vgen_xl_infer.yaml

In a few minutes, you can retrieve the high-definition video you wish to create from the workspace/experiments/test_img_01` directory. At present, we find that the current model performs inadequately on anime images and images with a black background due to the lack of relevant training data. We are consistently working to optimize it.

<span style="color:red">Due to the compression of our video quality in GIF format, please click 'HRER' below to view the original video.</span>

<center>
<table>
<center>
<tr>
<td ><center>
<image height="260" src="https://img.alicdn.com/imgextra/i1/O1CN01CCEq7K1ZeLpNQqrWu_!!6000000003219-0-tps-1280-720.jpg"></image>
</center></td>
<td ><center>
<!-- <video muted="true" autoplay="true" loop="true" height="260" src="https://cloud.video.taobao.com/play/u/null/p/1/e/6/t/1/442125067544.mp4"></video> -->
<image height="260" src="https://img.alicdn.com/imgextra/i4/O1CN01hIQcvG1spmQMLqBo0_!!6000000005816-1-tps-1280-704.gif"></image>
</center></td>
</tr>
<tr>
<td ><center>
<p>Input Image</p>
</center></td>
<td ><center>
<p>Clike <a href="https://cloud.video.taobao.com/play/u/null/p/1/e/6/t/1/442125067544.mp4">HRER</a> to view the generated video.</p>
</center></td>
</tr>
<tr>
<td ><center>
<image height="260" src="https://img.alicdn.com/imgextra/i4/O1CN01ZXY7UN23K8q4oQ3uG_!!6000000007236-2-tps-1280-720.png"></image>
</center></td>
<td ><center>
<!-- <video muted="true" autoplay="true" loop="true" heigh

View official source
Join our Telegram