Skip to content
/ LAMP Public

[CVPR 2024] | LAMP: Learn a Motion Pattern for Few-Shot Based Video Generation

License

Notifications You must be signed in to change notification settings

RQ-Wu/LAMP

Repository files navigation

[CVPR 2024] | LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation

Python 3.8 pytorch 1.12.0

This repository is the official implementation of LAMP

LAMP: Learn A Motion Pattern for Few-Shot Video Generation
Ruiqi Wu, Linagyu Chen, Tong Yang, Chunle Guo, Chongyi Li, Xiangyu Zhang
( * indicates corresponding author)

[Arxiv Paper]  [Website Page]  [Google Drive]  [Baidu Disk (pwd: ffsp)]  [Colab Notebookmethod 

🚀 LAMP is a few-shot-based method for text-to-video generation. You only need 8~16 videos 1 GPU (> 15 GB VRAM) for training!! Then you can generate videos with learned motion pattern.

News

  • [2024/02/27] Our paper is accepted by CVPR2024!
  • [2023/11/15] The code for applying LAMP on video editing is released!
  • [2023/11/02] The Colab demo is released! Thanks for the PR of @ShashwatNigam99.
  • [2023/10/21] We add Google Drive link about our checkpoints and training data.
  • [2023/10/17] We release our checkpoints and Arxiv paper.
  • [2023/10/16] Our code is publicly available.

Preparation

Dependencies and Installation

  • Ubuntu > 18.04
  • CUDA=11.3
  • Others:
# clone the repo
git clone https://github.com/RQ-Wu/LAMP.git
cd LAMP

# create virtual environment
conda create -n LAMP python=3.8
conda activate LAMP

# install packages
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113
pip install -r requirements.txt
pip install xformers==0.0.13

Weights and Data

  1. You can download pre-trained T2I diffusion models on Hugging Face. In our work, we use Stable Diffusion v1.4 as our backbone network. Clone the pretrained weights by git-lfs and put them in ./checkpoints

  2. Our checkpoint and training data are listed as follows. You can also collect video data by your own (Suggest websites: pexels, frozen-in-time) and put .mp4 files in ./training_videos/[motion_name]/

  3. [Update] You can find the training video for video editing demo in assets/run.mp4

Motion Name Checkpoint Link Training data
Birds fly Baidu Disk (pwd: jj0o) Baidu Disk (pwd: w96b)
Firework Baidu Disk (pwd: wj1p) Baidu Disk (pwd: oamp)
Helicopter Baidu Disk (pwd: egpe) Baidu Disk (pwd: t4ba)
Horse run Baidu Disk (pwd: 19ld) Baidu Disk (pwd: mte7)
Play the guitar Baidu Disk (pwd: l4dw) Baidu Disk (pwd: js26)
Rain Baidu Disk (pwd: jomu) Baidu Disk (pwd: 31ug)
Turn to smile Baidu Disk (pwd: 2bkl) Baidu Disk (pwd: l984)
Waterfall Baidu Disk (pwd: vpkk) Baidu Disk (pwd: 2edp)
All Baidu Disk (pwd: ifsm) Baidu Disk (pwd: 2i2k)

Get Started

1. Training

# Training code to learn a motion pattern
CUDA_VISIBLE_DEVICES=X accelerate launch train_lamp.py config="configs/horse-run.yaml"

# Training code for video editing (The training video can be found in assets/run.mp4)
CUDA_VISIBLE_DEVICES=X accelerate launch train_lamp.py config="configs/run.yaml"

2. Inference

Here is an example command for inference

# Motion Pattern
python inference_script.py --weight ./my_weight/turn_to_smile/unet --pretrain_weight ./checkpoints/stable-diffusion-v1-4 --first_frame_path ./benchmark/turn_to_smile/head_photo_of_a_cute_girl,_comic_style.png --prompt "head photo of a cute girl, comic style, turns to smile"

# Video Editing
python inference_script.py --weight ./outputs/run/unet --pretrain_weight ./checkpoints/stable-diffusion-v1-4 --first_frame_path ./bemchmark/editing/a_girl_runs_beside_a_river,_Van_Gogh_style.png --length 24 --editing

#########################################################################################################
# --weight:           the path of our model
# --pretrain_weight:  the path of the pre-trained model (e.g. SDv1.4)
# --first_frame_path: the path of the first frame generated by T2I model (e.g. SD-XL)
# --prompt:           the input prompt, the default value is aligned with the filename of the first frame
# --output:           output path, default: ./results 
# --height:           video height, default: 320
# --width:            video width, default: 512
# --length            video length, default: 16
# --cfg:              classifier-free guidance, default: 12.5
#########################################################################################################

Visual Examples

Few-Shot-Based Text-to-Video Generation

Horse run
A horse runs in the universe. A horse runs on the Mars. A horse runs on the road.
Firework
Fireworks in desert night. Fireworks over the mountains. Fireworks in the night city.
Play the guitar
GTA5 poster, a man plays the guitar. A woman plays the guitar. An astronaut plays the guitar, photorealistic.
Birds fly
Birds fly in the pink sky. Birds fly in the sky, over the sea. Many Birds fly over a plaza.

Video Editing

Origin Videos Editing Result-1 Editing Result-2
A girl in black runs on the road. A man runs on the road.
A man is dancing. A girl in white is dancing.

Citation

If you find our repo useful for your research, please cite us:

@inproceedings{wu2024lamp,
      title={LAMP: Learn A Motion Pattern for Few-Shot Video Generation},
      author={Wu, Ruiqi and Chen, Liangyu and Yang, Tong and Guo, Chunle and Li, Chongyi and Zhang, Xiangyu},
      booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
      year={2024}

License

Licensed under a Creative Commons Attribution-NonCommercial 4.0 International for Non-commercial use only. Any commercial use should get formal permission first.

Acknowledgement

This repository is maintained by Ruiqi Wu. The code is built based on Tune-A-Video. Thanks for the excellent open-source code!!