En3D - Official PyTorch Implementation

Project page | Paper | Video | Online Demo

En3D: An Enhanced Generative Model for Sculpting 3D Humans from 2D Synthetic Data
Yifang Men, Biwen Lei, Yuan Yao, Miaomiao Cui, Zhouhui Lian, Xuansong Xie
In: CVPR 2024

En3D is a large 3D human generative model trained on millions of synthetic 2D data, independently of any pre-existing 3D or 2D assets. This repo contains an implementation of En3D and provides a series of applications built upon it. In addition, this repo aims to be a useful creative tool to produce realistic 3D avatars from seeds, text prompts or images, and support automatic character animation FBX production. All outputs are compatible with the modern graphics workflows.

Generative 3D humans

gallery.mp4

Text guided synthesis

Image guided synthesis

More results can be found in project page.

Updates

(2024-10-29) The code of avatar generation & animation are available.

(2024-10-29) The pretrained weights are available from Modelscope.

(2024-01-15) ModelScope and HuggingFace Online demo is available! Try out .

(2024-01-15) A Rigged & Animated 3D Human library (3DHuman-Syn) is released, containing ~1000 avatars produced by En3D for quick experience. Infinite avatars and actions support will be coming soon!

(2024-01-03) The paper and video are released.

Web Demo

• Integrated an online demo into ModelScope. Try out and have fun!

• Integrated an online demo into Huggingface Spaces 🤗. Try out and have fun!

Requirements

• We recommend Linux for performance and compatibility reasons.
• 1–8 high-end NVIDIA GPUs. We have done all testing and development using V100, RTX3090, and A100 GPUs.
• 64-bit Python 3.8 and PyTorch 1.11.0 (or later). See https://pytorch.org for PyTorch install instructions.
• CUDA toolkit 11.3 or later. We used the custom CUDA extensions from the StyleGAN3 repo. Please see Troubleshooting
• Python libraries: see requirements.txt for exact library dependencies. You can use the following commands with Anaconda to create and activate your Python environment:
  • cd En3d
  • conda create -n en3d python=3.8
  • conda activate eg3d
  • pip install -r requirements.txt

Quick Start

WebUI usage

[Recommended] A deployed Online Demo

With own machine, you can also deploy our demo as below, which provides flexible user interface. Both CPU/GPU are supported for avatar animation, only GPU (>24G memory) is supported for avatar generation and render.

python app.py

Synthetic avatar library

We released a Rigged & Animated 3D Human library (3DHuman-Syn), containing ~1000 characters produced by En3D, and 1000+ actions are provided for animation.

• Avatar download and rendering

python render.py

• Avatar animation

python animation.py

• AR application

Convert the generated animation file (.glb) to .usdz format using Sketchfab (upload glb and download usdz) or other tools, and insert the animated avatar to web with model-viewer.

<!-- Import the component -->
<script type="module" src="https://ajax.googleapis.com/ajax/libs/model-viewer/3.3.0/model-viewer.min.js"></script>

<!-- Use it like any other HTML element -->
<model-viewer src="assets/human.glb" camera-controls ar shadow-intensity="1" ios-src="assets/human.usdz"></model-viewer>

AR function is only supported with iPhone, try the AR example in the phone browser and click the right-down icon for AR experience.

Avatar Generation

Download

• Download the whole models folder from the link and put it under the root dir.

python download_models.py

Generate 360 renderings

bash run_seed_pretrained.sh

Text guided synthesis

bash run_text_synthesis.sh

res_text2.mp4

Image guided synthesis

Repose the human image to canonical A-pose first, and then using the following command to generate the 3D avatar.

bash run_img_synthesis.sh

res_img2.mp4

Avatar Animation

Auto-rig and animation for the generated 3D avatar

bash run_animate.sh

res_anim2.mp4

Acknowledgments

Here are some great resources we benefit from:

• Multiview-Avatar for 2D avatar synthesis
• EG3D for the general 3D GAN structure and 3D representation
• Fantasia3D for DMTET optimization
• ICON for normal estimation

Citation

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{men2024en3d,
  title={En3D: An Enhanced Generative Model for Sculpting 3D Humans from 2D Synthetic Data},
  author={Men, Yifang and Lei, Biwen and Yao, Yuan and Cui, Miaomiao and Lian, Zhouhui and Xie, Xuansong},
  journal={arXiv preprint arXiv:2401.01173},
  website={https://menyifang.github.io/projects/En3D/index.html},
  year={2024}}