Official Repository for CVPR 2024 paper SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting.
The embedding points of 3DGS on the triangle mesh are updated by the walking on triangle scheme.
See the phongsurface module implemented in c++ and pybind11.
- Create conda env with pytorch.
conda create -n splatting python=3.9
conda activate splatting
# pytorch 1.13.1+cu117 is tested
pip install torch==1.13.1 torchvision torchaudio functorch --extra-index-url https://download.pytorch.org/whl/cu117
# pytorch3d
git clone https://github.com/facebookresearch/pytorch3d.git
cd pytorch3d
pip install -e .
# install other dependencies
pip install tqdm omegaconf opencv-python libigl
pip install trimesh plyfile imageio chumpy lpips
pip install packaging pybind11
pip install numpy==1.23.1
- Clone this repo recursively. Install Gaussian Splatting's submodules.
git clone --recursive https://github.com/initialneil/SplattingAvatar.git
cd SplattingAvatar
cd submodules/diff-gaussian-rasterization
pip install .
cd ../submodules/simple-knn
pip install .
cd ..
- Install
simple_phongsurffor walking on triangles.
cd model/simple_phongsurf
pip install -e .
-
Download FLAME model, choose FLAME 2020 and unzip it, copy 'generic_model.pkl' into
./model/imavatar/FLAME2020, -
Download SMPL model (1.0.0 for Python 2.7 (10 shape PCs)) and move them to the corresponding places:
mv /path/to/smpl/models/basicModel_f_lbs_10_207_0_v1.0.0.pkl model/smplx_utils/smplx_models/smpl/SMPL_FEMALE.pkl
mv /path/to/smpl/models/basicmodel_m_lbs_10_207_0_v1.0.0.pkl model/smplx_utils/smplx_models/smpl/SMPL_MALE.pkl
We provide the preprocessed data of the 10 subjects used in the paper.
- Our preprocessing followed IMavatar and replaced the Segmentation with RobustVideoMatting.
Pre-trained checkpoints are provided together with the data.- Please find the data from https://github.com/Zielon/INSTA. We'll update the checkpoint link soon.
python train_splatting_avatar.py --config configs/splatting_avatar.yaml --dat_dir <path/to/subject>
# for example:
python train_splatting_avatar.py --config configs/splatting_avatar.yaml --dat_dir /path-to/bala
# you may specify gpu id by adding CUDA_VISIBLE_DEVICES=x before calling python:
CUDA_VISIBLE_DEVICES=0 python train_splatting_avatar.py ...
# to disable network_gui, set ip to 'none'
CUDA_VISIBLE_DEVICES=0 python train_splatting_avatar.py ... --ip none
# use SIBR_remoteGaussian_app.exe from 3DGS to watch the training
SIBR_remoteGaussian_app.exe --path <path/to/output_of_any_standard_3dgs>
# <output_of_any_standard_3dgs> is generated by running the original 3dgs on any original dataset
# SIBR_remoteGaussian_app.exe somehow requires a standard 3dgs output to start
# it is recommended to change "FPS" to "Trackball" in the viewer
# you don't need to change the "path" everytime
python eval_splatting_avatar.py --config configs/splatting_avatar.yaml --dat_dir <path/to/model_path>
# for example:
python eval_splatting_avatar.py --config configs/splatting_avatar.yaml --dat_dir /path-to/bala/output-splatting/last_checkpoint
We conducted experiments on PeopleSnapshot.
- Please download the parameter files (the same with InstantAvatar) from: Baidu Disk or Google Drive.
- Download 4 sequences from PeopleSnapshot (male/female-3/4-casual) and unzip
imagesandmasksto corresponding folders from above. - Use
scripts/preprocess_PeopleSnapshot.pyto preprocess the data. - Training:
# override with instant_avatar.yaml for PeopleSnapshot in InstantAvatar's format
python train_splatting_avatar.py --config "configs/splatting_avatar.yaml;configs/instant_avatar.yaml" --dat_dir <path/to/subject>
# for example:
python train_splatting_avatar.py --config "configs/splatting_avatar.yaml;configs/instant_avatar.yaml" --dat_dir /path-to/female-3-casual
# pretrained checkpoints provided in `output-splatting/last_checkpoint` can be evaluated by `eval_splatting_avatar.py`
# for example:
python eval_splatting_avatar.py --config "configs/splatting_avatar.yaml;configs/instant_avatar.yaml" --dat_dir /path-to/female-3-casual --pc_dir /path-to/female-3-casual/output-splatting/last_checkpoint/point_cloud/iteration_30000
# to animate to noval pose `aist_demo.npz`
python eval_animate.py --config "configs/splatting_avatar.yaml;configs/instant_avatar.yaml" --dat_dir /path-to/female-3-casual --pc_dir /path-to/female-3-casual/output-splatting/last_checkpoint/point_cloud/iteration_30000 --anim_fn /path-to/aist_demo.npz
We conducted our experiments on a single NVIDIA RTX 3090 with 24GB. Training with less GPU memory can be achieved by setting a maximum number of Gaussians.
# in configs/splatting_avatar.yaml
model:
max_n_gauss: 300000 # or less as needed
or set by command line
python train_splatting_avatar.py --config configs/splatting_avatar.yaml --dat_dir <path/to/subject> model.max_n_gauss=300000
If you find our code or paper useful, please cite as:
@inproceedings{shao2024splattingavatar,
title = {{SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting}},
author = {Shao, Zhijing and Wang, Zhaolong and Li, Zhuang and Wang, Duotun and Lin, Xiangru and Zhang, Yu and Fan, Mingming and Wang, Zeyu},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2024}
}
We thank the following authors for their excellent works!
SplattingAvatar
The code is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License for Noncommercial use only. Any commercial use should get formal permission first.
Gaussian Splatting
Inria and the Max Planck Institut for Informatik (MPII) hold all the ownership rights on the Software named gaussian-splatting. The Software is in the process of being registered with the Agence pour la Protection des Programmes (APP).