Monocular Dynamic Gaussian Splatting is Fast and Brittle but Smooth Motion Helps

This repository is the official PyTorch(Lightning) implementation of the paper:

Monocular Dynamic Gaussian Splatting is Fast and Brittle but Smooth Motion Helps
Yiqing Liang, Mikhail Okunev, Mikaela Angelina Uy†‡, Runfeng Li, Leonidas Guibas‡, James Tompkin, Adam W Harley‡

   †    ‡

  Paper  |  Data

We aim to benchmark Monocular View Dynamic Gaussian Splatting from motion perspective.

Methods Included

Method	Abbrev Name in this Repo
Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting (ICLR 2024)	FourDim
A Compact Dynamic 3D Gaussian Representation for Real-Time Dynamic View Synthesis (ECCV 2024)	Curve
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (CVPR 2024)	HexPlane
Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (CVPR 2024)	MLP
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (CVPR 2024)	TRBF

Existing Datasets Included

Dataset	Abbrev Name in this Repo
D-NeRF: Neural Radiance Fields for Dynamic Scenes (CVPR 2021)	dnerf
Nerfies: Deformable Neural Radiance Fields (ICCV 2021)	nerfies
A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields (SIGGRAPH Asia 2021)	hypernerf
Monocular Dynamic View Synthesis: A Reality Check (NeurIPS 2022)	iphone
NeRF-DS: Neural Radiance Fields for Dynamic Specular Objects (CVPR 2023)	nerfds

Our Created Datasets

Dataset	Abbrev Name in this Repo
Camera-Pose-Rectified HyperNeRF	fixed
Instructive Dataset	dnerf/custom

All data could be prepared by downloading this folder and extracted as follow diagram:

this_repo
│   README.md  
└───data
│    │ 
│    └───dnerf
│    │    │ 
│    │    └───data
│    │         │ 
│    │         └───bouncingballs
│    │         └───...
│    └───fixed
│    │    │ 
│    │    └───chickchicken
│    │    └───...
│    └───hypernerf 
│    │    │ 
│    │    └───aleks-teapot
│    │    └───...
│    └───iphone
│    │    │ 
│    │    └───apple
│    │    └───...
│    └───nerfds
│    │    │ 
│    │    └───as
│    │    └───...
│    └───nerfies
│    │     │ 
│    │     └───broom
│    │     └───...
│    │
|    └───custom
|          |
│          └───dynamic_cube_dynamic_camera_textured_motion_range_0.0
│          └───...
└...       
...

Installation

This code has been developed with Anaconda (Python 3.7), CUDA 11.8.0 on Red Hat Enterprise Linux 9.2, one NVIDIA GeForce RTX 3090 GPU.

conda create -p [YourEnv] python=3.9
conda activate [YourEnv]

conda install -c anaconda libstdcxx-ng
conda install -c menpo opencv 

conda install -c conda-forge plyfile==0.8.1
pip install tqdm imageio

pip install torch==2xx # find the torch version that works for your cuda device

pip install torchmetrics

pip install requests 

pip install tensorboard 

pip install scipy

pip install kornia



pip install lightning=2.2.1 # recommend to use this version for stability!
pip install "jsonargparse[signatures]"

pip install wandb
pip install lpips

pip install pytorch-msssim

pip install ninja
pip install timm==0.4.5

# install from local folders 
pip install submodules/diff-gaussian-rasterization
pip install submodules/depth-diff-gaussian-rasterization
pip install submodules/gaussian-rasterization_ch3
pip install submodules/gaussian-rasterization_ch9
pip install submodules/simple-knn

After activating conda environment

wandb init

paste API key and create first project following instruction

Usage

On Our Instructive Dataset

We provide a python utility to run training and testing for the instructive dataset. The utility trains the model as well as runs evaluations for masked and non-masked metrics. The utility by default is going to run the code locally. However, this amount of experiments will likely require a cluster to finish in a reasonable time. For this case, the utility can instead run scripts on the slurm cluster. Check slurms/custom.sh as an example and specify --slurm_script parameter. For training and testing method ${method} on the instructive dataset scene ${scene},

exp_group_name="vanilla"
exp_name="${scene}_${method}"

python runner.py \
    --config_file configs/custom/${method}/vanilla1.yaml \
    --group ${exp_group_name}_${scene} \
    --name ${exp_name} \
    --dataset data/custom/${scene} \
    --slurm_script slurms/custom.sh \
    --output_dir output/custom/${exp_group_name}/${scene}/${method}

On All Other Datasets

For training and testing method ${method} on dataset ${dataset}'s scene ${scene},

base="${dataset}/${scene}/${method}"
name="vanilla1"
variant="${base}/${name%?}1"
output_path="./output/${base}"

python main.py fit \
    --config configs/${variant}.yaml \
    --output ${output_path} \
    --name "${base##*/}_$name" 

python main.py test \
    --config configs/${variant}.yaml \
    --ckpt_path  last \
    --output ${output_path} \
    --name "${base##*/}_$name" 

Citation

If you find our repository useful, please consider giving it a star ⭐ and citing our paper:

@misc{liang2024monocular,
  title        ={Monocular Dynamic Gaussian Splatting is Fast and Brittle but Smooth Motion Helps},
  author       ={Yiqing Liang and Mikhail Okunev and Mikaela Angelina Uy and Runfeng Li and Leonidas Guibas and James Tompkin and Adam W. Harley},
  year         ={2024},
  eprint       ={2412.04457},
  archivePrefix={arXiv},
  primaryClass ={cs.CV}
}