Free-SurGS: SfM-Free 3D Gaussian Splatting for Surgical Scene Reconstruction

Jiaxin Guo¹    Jiangliu Wang¹    Di Kang²    Wenzhen Dong¹    Wenting Wang¹    Yun-hui Liu^{1, 3   
1}CUHK    ²Tencent AI Lab    ³HKCLR   

The repository contains the official implementation for the MICCAI 2024 paper Free-SurGS: SfM-Free 3D Gaussian Splatting for Surgical Scene Reconstruction.

3DGS with SfM fails to recover accurate camera poses and geometry in surgical scenes due to the challenges of minimal textures and photometric inconsistencies. In our paper, we propose Free-SurGS as the first SfM-free 3DGS-based method for surgical scene reconstruction from monocular video by jointly optimizing the camera poses and scene representation.

demo.mp4

Pipeline

Update

• Release the training and evaluation code.
• Release the web-based gaussian visualizer for pose free 3DGS.

Installation

Our code is tested on Ubuntu22.04 + CUDA 12.1 + Pytorch 2.2.1

conda create -n freesurgs python=3.10
conda activate freesurgs
pip install -r requirements.txt

Dataset

We evaluate our method on the SCARED dataset. To obtain the dataset and code, please sign the challenge rules and email them to max.allan@intusurg.com.

To reproduce our result quickly, we provide a sequence of preprocessed demo example, please download here.

Train

To train the Free-SurGS, please follow:

python train.py -s ./data/scared_demo/ \
        --model_path ./outputs/scared_demo/ \
        --visualize True \
        --port 8039 \
        --log True

After training, the checkpoints and rendered test views can be found in ./outputs/.

Evaluation

To validate our method, please follow:

python train.py -s ./data/scared_demo/ \
        --model_path ./outputs/scared_demo/ \
        --test True \
        --start_checkpoint <path_to_the_checkpoints>

Interactive Web-based Visualizer

We use the visualizer adapted from shape-of-motion based on Viser and nerfview. You can easily visualize reconstructed 3D gaussian via connecting to http://localhost:<your_port>.

Acknowledgement

Our code is based on 3D Gaussian Splatting and has incorporated some parts from SplaTAM. Thanks for their excellent work!

Citing

If you find our work helpful, please cite:

@article{guo2024free,
  title={Free-SurGS: SfM-Free 3D Gaussian Splatting for Surgical Scene Reconstruction},
  author={Guo, Jiaxin and Wang, Jiangliu and Kang, Di and Dong, Wenzhen and Wang, Wenting and Liu, Yun-hui},
  booktitle={MICCAI},
  year={2024},
  organization={Springer}
}