Paper | Project Page | Dataset | Video
Text2Performer synthesizes human videos by taking the text descriptions as the only input.
📖 For more visual results, go checkout our project page
Clone this repo:
git clone https://github.com/yumingj/Text2Performer.git
cd Text2Performer
Dependencies:
conda env create -f env.yaml
conda activate text2performer
In this work, we contribute a human video dataset with rich label and text annotations named Fashion-Text2Video Dataset.
You can download our processed dataset from this Google Drive. After downloading the dataset, unzip the file and put them under the dataset folder with the following structure:
./datasets
├── FashionDataset_frames_crop
├── xxxxxx
├── 000.png
├── 001.png
├── ...
├── xxxxxx
└── xxxxxx
├── train_frame_num.txt
├── val_frame_num.txt
├── test_frame_num.txt
├── moving_frames.npy
├── captions_app.json
├── caption_motion_template.json
├── action_label
├── xxxxxx.txt
├── xxxxxx.txt
├── ...
└── xxxxxx.txt
└── shhq_dataset % optional
Pretrained models can be downloaded from the Google Drive. Unzip the file and put them under the pretrained_models folder with the following structure:
pretrained_models
├── sampler_high_res.pth
├── video_trans_high_res.pth
└── vqgan_decomposed_high_res.pth
After downloading pretrained models, you can use generate_long_video.ipynb
to generate videos.
Train the decomposed VQGAN. If you want to skip the training of this network, you can download our pretrained model from here.
For better performance, we also use the data from SHHQ dataset to train this stage.
python -m torch.distributed.launch --nproc_per_node=4 --master_port=29596 train_vqvae_iter_dist.py -opt ./configs/vqgan/vqgan_decompose_high_res.yml --launcher pytorch
Train the video transformer. If you want to skip the training of this network, you can download our pretrained model from here.
python -m torch.distributed.launch --nproc_per_node=4 --master_port=29596 train_dist.py -opt ./configs/video_transformer/video_trans_high_res.yml --launcher pytorch
Train the appearance transformer. If you want to skip the training of this network, you can download our pretrained model from here.
python train_sampler.py -opt ./configs/sampler/sampler_high_res.yml
If you find this work useful for your research, please consider citing our paper:
@inproceedings{jiang2023text2performer,
title={Text2Performer: Text-Driven Human Video Generation},
author={Jiang, Yuming and Yang, Shuai and Koh, Tong Liang and Wu, Wayne and Loy, Chen Change and Liu, Ziwei},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
year={2023}
}
Distributed under the S-Lab License. See LICENSE
for more information.