Deform3DGS: Flexible Deformation for Fast Surgical Scene Reconstruction with Gaussian Splatting

Official code implementation for Deform3DGS, a Gaussian Splatting based framework for surgical scene reconstruction.

Deform3DGS: Flexible Deformation for Fast Surgical Scene Reconstruction with Gaussian Splatting
Shuojue Yang, Qian Li, Daiyun Shen, Bingchen Gong, Qi Dou, Yueming Jin
MICCAI2024, Early Accept

Demo

Reconstruction within 1 minute

demo1.mp4

Compared to previous SOTA method in fast reconstruction, our method reduces the training time to 1 minute for each clip in EndoNeRF dataset, demonstrating remarkable superiority in efficiency.

Reconstruction of various scenes

visual_demo.mp4

Pipeline

Deform3DGS is composed of (a) Point cloud initialization, (b) Flexible Deformation Modeling, and (c) 3D Gaussian Splatting. Experiments on DaVinci robotic surgery videos indicate the efficacy of our approach, showcasing superior reconstruction fidelity PSNR: (37.90) and rendering speed (338.8 FPS) while substantially reducing training time to only 1 minute/scene.

Environment setup

Tested with NVIDIA RTX A5000 GPU.

git clone https://github.com/jinlab-imvr/Deform3DGS.git
cd Deform3DGS
conda create -n Deform3DGS python=3.7 
conda activate Deform3DGS

pip install -r requirements.txt
pip install -e submodules/depth-diff-gaussian-rasterization
pip install -e submodules/simple-knn

Datasets

We use 6 clips from EndoNeRF and 3 clips manually extracted from StereoMIS to verify our method.

To use the two available examples in EndoNeRF dataset. Please download the data via this link and organize the data according to the guideline.

To use the StereoMIS dataset, please follow this github repo to preprocess the dataset. After that, run the provided script stereomis2endonerf.py to extract clips from the StereoMIS dataset and organize the depth, masks, images, intrinsic and extrinsic parameters in the same format as EndoNeRF. In our implementation, we used RAFT to estimate the stereo depth for StereoMIS clips. Following EndoNeRF dataset, this script only supports fixed-view settings.

The data structure is as follows:

data
| - endonerf_full_datasets
|   | - cutting_tissues_twice
|   |   | -  depth/
|   |   | -  images/
|   |   | -  masks/
|   |   | -  pose_bounds.npy 
|   | - pushing_soft_tissues
| - StereoMIS
|   | - stereo_seq_1
|   | - stereo_seq_2

Training

To train Deform3DGS with customized hyper-parameters, please make changes in arguments/endonerf/default.py.

To train Deform3DGS, run the following example command:

python train.py -s data/endonerf_full_datasets/pulling_soft_tissues --expname endonerf/pulling_fdm --configs arguments/endonerf/default.py 

Testing

For testing, we perform rendering and evaluation separately.

Rendering

To run the following example command to render the images:

python render.py --model_path output/endonerf/pulling_fdm  --skip_train --reconstruct_test --configs arguments/endonerf/default.py

Please follow EndoGaussian to skip rendering. Of note, you can also set --reconstruct_train, --reconstruct_test, and --reconstruct_video to reconstruct and save the .ply 3D point cloud of the rendered outputs for train, test andvideo sets, respectively.

Evaluation

To evaluate the reconstruction quality, run following command:

python metrics.py --model_path output/endonerf/pulling_fdm -p test

Note that you can set -p video, -p test, -p train to select the set for evaluation.

Acknowledgements

This repo borrows some source code from EndoGaussian, 4DGS, depth-diff-gaussian-rasterizer, 3DGS, and EndoNeRF. We would like to acknowledge these great prior literatures for inspiring our work.

Thanks to EndoGaussian for their great and timely effort in releasing the framework adapting Gaussian Splatting into surgical scene.

Citation

If you find this code useful for your research, please use the following BibTeX entries:

@misc{yang2024deform3dgs,
      title={Deform3DGS: Flexible Deformation for Fast Surgical Scene Reconstruction with Gaussian Splatting}, 
      author={Shuojue Yang and Qian Li and Daiyun Shen and Bingchen Gong and Qi Dou and Yueming Jin},
      year={2024},
      eprint={2405.17835},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Questions

For further question about the code or paper, welcome to create an issue, or contact 's.yang@u.nus.edu' or 'ymjin@nus.edu.sg'