Official implementation of MicroDreamer: Zero-shot 3D Generation in ~20 Seconds by Score-based Iterative Reconstruction.
display.mp4
[10/2024] Add a new mesh export method from LGM
The codebase is built on DreamGaussian. For installation,
conda create -n MicroDreamer python=3.11
conda activate MicroDreamer
pip install -r requirements.txt
# a modified gaussian splatting (+ depth, alpha rendering)
git clone --recursive https://github.com/ashawkey/diff-gaussian-rasterization
pip install ./diff-gaussian-rasterization
# The commit hash we used
# d986da0d4cf2dfeb43b9a379b6e9fa0a7f3f7eea
# simple-knn
pip install ./simple-knn
# nvdiffrast
pip install git+https://github.com/NVlabs/nvdiffrast/
# The version we used
# pip install git+https://github.com/NVlabs/nvdiffrast/@0.3.1
# kiuikit
pip install git+https://github.com/ashawkey/kiuikit/
# The version we used
# pip install git+https://github.com/ashawkey/kiuikit/@0.2.3
# To use ImageDream, also install:
pip install git+https://github.com/bytedance/ImageDream/#subdirectory=extern/ImageDream
# The commit hash we used
# 26c3972e586f0c8d2f6c6b297aa9d792d06abebb
Image-to-3D:
### preprocess
# background removal and recentering, save rgba at 256x256
python process.py test_data/name.jpg
# save at a larger resolution
python process.py test_data/name.jpg --size 512
# process all jpg images under a dir
python process.py test_data
### training gaussian stage
# train 20 iters and export ckpt & coarse_mesh to logs
python main.py --config configs/image_sai.yaml input=test_data/name_rgba.png save_path=name_rgba
### training mesh stage
# auto load coarse_mesh and refine 3 iters, export fine_mesh to logs
python main2.py --config configs/image_sai.yaml input=test_data/name_rgba.png save_path=name_rgba
Image+Text-to-3D (ImageDream):
### training gaussian stage
python main.py --config configs/imagedream.yaml input=test_data/ghost_rgba.png prompt="a ghost eating hamburger" save_path=ghost_rgba
Calculate for CLIP similarity:
PYTHONPATH='.' python scripts/cal_sim.py
total_1.mp4
total_2.mp4
This work is built on many amazing open source projects, thanks to all the authors!
@misc{chen2024microdreamerzeroshot3dgeneration,
title={MicroDreamer: Zero-shot 3D Generation in $\sim$20 Seconds by Score-based Iterative Reconstruction},
author={Luxi Chen and Zhengyi Wang and Zihan Zhou and Tingting Gao and Hang Su and Jun Zhu and Chongxuan Li},
year={2024},
eprint={2404.19525},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2404.19525},
}