i2vgen xl

---
license: mit
tags:

• image-to-video

VGen

!figure1

VGen is an open-source video synthesis codebase developed by the Tongyi Lab of Alibaba Group, featuring state-of-the-art video generative models. This repository includes implementations of the following methods:

• I2VGen-xl: High-quality image-to-video synthesis via cascaded diffusion models

• VideoComposer: Compositional Video Synthesis with Motion Controllability

• Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation

• [A Recipe for Scaling up Text-to-Video Generation with Text-free Videos]()

• [InstructVideo: Instructing Video Diffusion Models with Human Feedback]()

• DreamVideo: Composing Your Dream Videos with Customized Subject and Motion

• VideoLCM: Video Latent Consistency Model

• Modelscope text-to-video technical report

VGen can produce high-quality videos from the input text, images, desired motion, desired subjects, and even the feedback signals provided. It also offers a variety of commonly used video generation tools such as visualization, sampling, training, inference, join training using images and videos, acceleration, and more.

<a href='https://i2vgen-xl.github.io/'><img src='https://img.shields.io/badge/Project-Page-Green'></a> <a href='https://arxiv.org/abs/2311.04145'><img src='https://img.shields.io/badge/Paper-Arxiv-red'></a>  <a href='https://cloud.video.taobao.com/play/u/null/p/1/e/6/t/1/441039979087.mp4'><img src='source/logo.png'></a>

🔥News!!!

• __[2023.12]__ We release the high-efficiency video generation method VideoLCM

• __[2023.12]__ We release the code and model of I2VGen-XL and the ModelScope T2V

• __[2023.12]__ We release the T2V method HiGen and customizing T2V method DreamVideo.

• __[2023.12]__ We write an introduction docment for VGen and compare I2VGen-XL with SVD.

• __[2023.11]__ We release a high-quality I2VGen-XL model, please refer to the Webpage

TODO

• [x] Release the technical papers and webpage of I2VGen-XL

• [x] Release the code and pretrained models that can generate 1280x720 videos

• [ ] Release models optimized specifically for the human body and faces

• [ ] Updated version can fully maintain the ID and capture large and accurate motions simultaneously

• [ ] Release other methods and the corresponding models

Preparation

The main features of VGen are as follows:

• Expandability, allowing for easy management of your own experiments.

• Completeness, encompassing all common components for video generation.

• Excellent performance, featuring powerful pre-trained models in multiple tasks.

Installation

conda create -n vgen python=3.8
conda activate vgen
pip install torch==1.12.0+cu113 torchvision==0.13.0+cu113 torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cu113
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple

Datasets

We have provided a demo dataset that includes images and videos, along with their lists in `data.

git clone https://github.com/damo-vilab/i2vgen-xl.git
cd i2vgen-xl

python train_net.py --cfg configs/t2v_train.yaml

In the t2v_train.yaml configuration file, you can specify the data, adjust the video-to-image ratio using frame_lens, and validate your ideas with different Diffusion settings, and so on.

• Before the training, you can download any of our open-source models for initialization. Our codebase supports custom initialization and grad_scale settings, all of which are included in the Pretrain item in yaml file.

• During the training, you can view the saved models and intermediate inference results in the workspace/experiments/t2v_traindirectory.

python inference.py --cfg configs/t2v_infer.yaml

!pip install modelscope
from modelscope.hub.snapshot_download import snapshot_download
model_dir = snapshot_download('damo/I2VGen-XL', cache_dir='models/', revision='v1.0.0')

python inference.py --cfg configs/i2vgen_xl_infer.yaml

<span style="color:red">Due to the compression of our video quality in GIF format, please click 'HRER' below to view the original video.</span>

<center>
<table>
<center>
<tr>
<td ><center>
<image height="260" src="https://img.alicdn.com/imgextra/i1/O1CN01CCEq7K1ZeLpNQqrWu_!!6000000003219-0-tps-1280-720.jpg"></image>
</center></td>
<td ><center>
<!-- <video muted="true" autoplay="true" loop="true" height="260" src="https://cloud.video.taobao.com/play/u/null/p/1/e/6/t/1/442125067544.mp4"></video> -->
<image height="260" src="https://img.alicdn.com/imgextra/i4/O1CN01hIQcvG1spmQMLqBo0_!!6000000005816-1-tps-1280-704.gif"></image>
</center></td>
</tr>
<tr>
<td ><center>
<p>Input Image</p>
</center></td>
<td ><center>
<p>Clike <a href="https://cloud.video.taobao.com/play/u/null/p/1/e/6/t/1/442125067544.mp4">HRER</a> to view the generated video.</p>
</center></td>
</tr>
<tr>
<td ><center>
<image height="260" src="https://img.alicdn.com/imgextra/i4/O1CN01ZXY7UN23K8q4oQ3uG_!!6000000007236-2-tps-1280-720.png"></image>
</center></td>
<td ><center>
<!-- <video muted="true" autoplay="true" loop="true" heigh