Cinemo: Consistent and Controllable Image Animation with Motion Diffusion Models
Official PyTorch Implementation
Cinemo: Consistent and Controllable Image Animation with Motion Diffusion Models
Xin Ma, Yaohui Wang*†, Gengyun Jia, Xinyuan Chen, Yuan-Fang Li, Cunjian Chen*, Yu Qiao
(*Corresponding authors, †Project Lead)
This repo contains pre-trained weights, and sampling code of Cinemo. Please visit our project page for more results.
-
(🔥 New) Jul. 29, 2024. 💥 HuggingFace space is added, you can also launch gradio interface locally.
-
(🔥 New) Jul. 23, 2024. 💥 Our paper is released on arxiv.
-
(🔥 New) Jun. 2, 2024. 💥 The inference code is released. The checkpoint can be found here.
Download and set up the repo:
git clone https://github.com/maxin-cn/Cinemo
cd Cinemo
conda env create -f environment.yml
conda activate cinemo
You can sample from our pre-trained Cinemo models with animation.py
. Weights for our pre-trained Cinemo model can be found here. The script has various arguments for adjusting sampling steps, changing the classifier-free guidance scale, etc:
bash pipelines/animation.sh
Related model weights will be downloaded automatically and following results can be obtained,
Input image | Output video | Input image | Output video |
![]() |
![]() |
![]() |
![]() |
"People Walking" | "Sea Swell" | ||
![]() |
![]() |
![]() |
![]() |
"Girl Dancing under the Stars" | "Dragon Glowing Eyes" | ||
![]() |
![]() |
![]() |
![]() |
"Bubbles Floating upwards" | "Snowman Waving his Hand" |
We also provide a local gradio interface, just run:
python app.py
You can specify the --share
and --server_name
arguments to meet your requirement!
You can also utilize Cinemo for other applications, such as motion transfer and video editing:
bash pipelines/video_editing.sh
Related checkpoints will be downloaded automatically and following results will be obtained,
Input video | First frame | Edited first frame | Output video |
![]() |
![]() |
![]() |
![]() |
or motion transfer,
Input video | First frame | Edited first frame | Output video |
![]() |
![]() |
![]() |
![]() |
Xin Ma: xin.ma1@monash.edu, Yaohui Wang: wangyaohui@pjlab.org.cn
If you find this work useful for your research, please consider citing it.
@article{ma2024cinemo,
title={Cinemo: Latent Diffusion Transformer for Video Generation},
author={Ma, Xin and Wang, Yaohui and Jia, Gengyun and Chen, Xinyuan and Li, Yuan-Fang and Chen, Cunjian and Qiao, Yu},
journal={arXiv preprint arXiv:2407.15642},
year={2024}
}
Cinemo has been greatly inspired by the following amazing works and teams: LaVie and SEINE, we thank all the contributors for open-sourcing.
The code and model weights are licensed under LICENSE.