- Implement multicontrolnet for high quality and consistency of generated video (that can preserve facial and hand features more effectively)
- Setting weight average for each controlnet model if necessary
- Colabs for more consistent generation of densepose and openpose
- The code is currently under process (usable, but contains a significant amount of hardcoding).
- Here is the repo for more details of multicontrolnet (branch multicontrolnet): https://github.com/takuma104/diffusers
- Resize your input video to 512x512 and 25 fps at: https://www.video2edit.com/convert-to-video
- Vid2DensePose colab link: https://colab.research.google.com/drive/1Vs9WnyIQmv0f7iI9R0HR25Onf7aX-az3?usp=sharing
- Vid2OpenPose colab link: https://colab.research.google.com/drive/1NxH02XZBdDur_gkQtJgjbtZ7qrgvEt1q?usp=sharing
- Vid2FaceOnly colab link: https://colab.research.google.com/drive/1wrXGs9awXxk7G80lONzK0qTA6OYfzusZ?usp=sharing
- Download controlnet models to folder [pretrained_models/MagicAnimate]
- Set up configs in [configs/prompts/animation.yaml]
- Then run installation and inference like the original repository.
Zhongcong Xu
·
Jianfeng Zhang
·
Jun Hao Liew
·
Hanshu Yan
·
Jia-Wei Liu
·
Chenxu Zhang
·
Jiashi Feng
·
Mike Zheng Shou
National University of Singapore | ByteDance
- [2023.12.4] Release inference code and gradio demo. We are working to improve MagicAnimate, stay tuned!
- [2023.11.23] Release MagicAnimate paper and project page.
Please download the pretrained base models for StableDiffusion V1.5 and MSE-finetuned VAE.
Download our MagicAnimate checkpoints.
Place them as follows:
magic-animate
|----pretrained_models
|----MagicAnimate
|----appearance_encoder
|----diffusion_pytorch_model.safetensors
|----config.json
|----densepose_controlnet
|----diffusion_pytorch_model.safetensors
|----config.json
|----temporal_attention
|----temporal_attention.ckpt
|----sd-vae-ft-mse
|----...
|----stable-diffusion-v1-5
|----...
|----...
prerequisites: python>=3.8
, CUDA>=11.3
, and ffmpeg
.
Install with conda
:
conda env create -f environment.yaml
conda activate manimate
or pip
:
pip3 install -r requirements.txt
Run inference on single GPU:
bash scripts/animate.sh
Run inference with multiple GPUs:
bash scripts/animate_dist.sh
Try our online gradio demo quickly.
Launch local gradio demo on single GPU:
python3 -m demo.gradio_animate
Launch local gradio demo if you have multiple GPUs:
python3 -m demo.gradio_animate_dist
Then open gradio demo in local browser.
We would like to thank AK(@_akhaliq) and huggingface team for the help of setting up oneline gradio demo.
If you find this codebase useful for your research, please use the following entry.
@inproceedings{xu2023magicanimate,
author = {Xu, Zhongcong and Zhang, Jianfeng and Liew, Jun Hao and Yan, Hanshu and Liu, Jia-Wei and Zhang, Chenxu and Feng, Jiashi and Shou, Mike Zheng},
title = {MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model},
booktitle = {arXiv},
year = {2023}
}