SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views [ECCV2022]

We present a novel neural surface reconstruction method, called SparseNeuS, which can generalize to new scenes and work well with sparse images (as few as 2 or 3).

Project Page | Paper

Setup

Conda Environment

conda create -n spneus python=3.10.6
conda activate spneus
pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu117 
# install torchsparse 1.4.0
conda install -c conda-forge sparsehash
pip install --upgrade git+https://github.com/mit-han-lab/torchsparse.git@v1.4.0
# install inplace-abn
pip install inplace-abn
# install tensorboard
pip install tensorboard

Dataset

• DTU Training dataset. Please download the preprocessed DTU dataset provided by MVSNet. As stated in the paper, we preprocess the images to obtain the masks about the "black empty background" to remove image noises. The preprocessed masks can be downloaded here. Training without the masks will not be a problem, just ignore the "masks" in the dataloader.
• DTU testing dataset. Since our target neural reconstruction with sparse views, we select two set of three images from the 15 testing scenes (same as IDR) for evaluation. Download our prepared testing dataset.

Easy to try

Just run the provided bash file to get the teaser result.

bash ./sample_bashs/dtu_scan118.sh

Training

Our training has two stages. First train the coarse level and then the fine level.

python exp_runner_generic.py --mode train --conf ./confs/general_lod0.conf
python exp_runner_generic.py --mode train --conf ./confs/general_lod1.conf --is_continue --restore_lod0

Finetuning

The reconstructed results generated by generic model can be further improved using our consistency-aware fine-tuning scheme.

We use current powerful segmentation models to improve the performance (a bit difference from original paper). First, use segmentation model to remove backgrounds, and save the foreground images into mask directory. Then run such code:

#!/usr/bin/env bash
python exp_runner_finetune.py \
--mode train --conf ./confs/finetune.conf --is_finetune \
--checkpoint_path ./weights/ckpt.pth \
--case_name scan118  --train_imgs_idx 0 1 2 --test_imgs_idx 0 1 2 --near 700 --far 1100 

Results

You can download the DTU results and BMVS results of the paper reports here.

Citation

Cite as below if you find this repository is helpful to your project:

@inproceedings{long2022sparseneus,
  title={Sparseneus: Fast generalizable neural surface reconstruction from sparse views},
  author={Long, Xiaoxiao and Lin, Cheng and Wang, Peng and Komura, Taku and Wang, Wenping},
  booktitle={European Conference on Computer Vision},
  pages={210--227},
  year={2022},
  organization={Springer}
}

Acknowledgement

Some code snippets are borrowed from IDR, NeuS and IBRNet. Thanks for these great projects.