3DFIRES: Few Image 3D REconstruction for Scenes with Hidden Surfaces

Linyi Jin¹, Nilesh Kulkarni¹, David F. Fouhey^2
1University of Michigan ²New York University

3DFIRES reconstructs scene-level 3D from posed images, which works with as few as one view, reconstructs the complete geometry of unseen scenes, including hidden surfaces.

Install

conda env create -f environment.yml
conda activate 3dfires

# Download Pretrained Model
cd ckpts
wget https://www.dropbox.com/scl/fi/n5en8p4rqyudrj4kg8f3g/model_0299999.pth?rlkey=mva5crd8smkegkzgl41zh89tv
cd ..

In dropbox, we provide preprocessed file for testing, download and save to dataset/

wget -O dataset.tar "https://www.dropbox.com/scl/fi/hv4zn8s09vjsy0en0au0v/dataset.tar?rlkey=4910bmzoflnhpymfp1m1inhtj&dl=1"
tar -xvf dataset.tar

To get the full dataset for evaluation and training, you need to download the omnidata Taskonomy Medium dataset as tar files (no need to decompress since our code directly read from tar). We also need to create tarindex for faster IO.

conda install -c conda-forge aria2
pip install 'omnidata-tools'

omnitools.download rgb depth_zbuffer \
   --components taskonomy \
   --subset medium \
   --dest ./dataset/omnidata \
   --name YOUR_NAME --email YOUR_EMAIL --agree_all \
   --only_download True \
   --keep_compressed True 

Create tarindexer for fast tar IO.

git clone git@github.com:devsnd/tarindexer.git
cd tarindexer
ROOT=../dataset/omnidata
tar=$ROOT/compressed
tarindex=$ROOT/tarindex
mkdir -p $tar
mkdir -p $tarindex
ls $tar | xargs -I{} python tarindexer.py -i $tar/{} $tarindex/{}

Demo

bash demo.sh

Evaluation

bash eval.sh

Training

bash train.sh

BibTeX

@article{jin20243dfires,
  author    = {Jin, Linyi and Kulkarni, Nilesh and Fouhey, David},
  title     = {3DFIRES: Few Image 3D REconstruction for Scenes with Hidden Surfaces},
  journal   = {CVPR},
  year      = {2024},
}