This is a minimal PyTorch implementation of DreamFusion and its variant Magic3D, where we utilize Instant-NGP as the neural renderer and Stable Diffusion/DeepFloyd IF as the guidance.
It takes ~30min to train with Stable Diffusion and ~40min with DeepFloyd IF on a single 3090.
git clone https://github.com/chinhsuanwu/dreamfusionacc.git
cd dreamfusionacc
pip install -r requirements.txt
Dependencies (click to expand)
- torch
- tinycudann
- nerfacc
- numpy
- imageio
- einops
- diffusers
- trainsformers
You may install the pre-built wheels for NerfAcc. Please check out here. This repo is built upon torch 1.13.0 + cu117.
To train
python train.py --config config/peacock.yaml
You can find all controllable settings in the yaml file. After the training is done, run
python test.py --config config/peacock.yaml
to render 360˚ visualizations.
@article{poole2022dreamfusion,
author = {Poole, Ben and Jain, Ajay and Barron, Jonathan T. and Mildenhall, Ben},
title = {DreamFusion: Text-to-3D using 2D Diffusion},
journal = {arXiv},
year = {2022},
}
@inproceedings{lin2023magic3d,
title={Magic3D: High-Resolution Text-to-3D Content Creation},
author={Lin, Chen-Hsuan and Gao, Jun and Tang, Luming and Takikawa, Towaki and Zeng, Xiaohui and Huang, Xun and Kreis, Karsten and Fidler, Sanja and Liu, Ming-Yu and Lin, Tsung-Yi},
booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
year={2023}
}
This implementation is heavily based on NerfAcc and stable-dreamfusion. Kudos to the authors for their amazing work!
![tyro logo](https://user-images.githubusercontent.com/3310961/199084143-0d63eb40-3f35-48d2-a9d5-78d1d60b7d66.png)