Neural Scene Graphs for Dynamic Scene (CVPR 2021) PyTorch

Project Page | Paper

Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, Felix Heide

Implementation of Neural Scene Graphs, that optimizes multiple radiance fields to represent different objects and a static scene background. Learned representations can be rendered with novel object compositions and views.

Original repository forked from the Implementation of "NeRF: Neural Radiance Fields" by Mildenhall et al.: Original NeRF Implementation, original readme; and Origional TensorFlow Implementation.

---

Getting started

The whole script is currently optimized for the usage with Virtual KITTI 2 Dataset and KITTI

Quick Start

Train a Virtual KITTI 2 Scene

conda create -n neural_scene_graphs python=3.11
conda activate neural_scene_graphs
cd neural-scene-graphs-pytorch
pip install -r requirements.txt
bash download_virtual_kitti.sh
python main.py --config example_configs/config_vkitti2_Scene06.txt
tensorboard --logdir=example_weights/summaries --port=6006

Data Preperation

KITTI

1. Get the KITTI MOT dataset, from which you need:
   1. Left color images
   2. Right color images
   3. GPS/IMU data
   4. Camera Calibration Files
   5. Training labels
2. Extract everything to ./data/kitti and keep the data structure
3. Neural Scene Graphs is well tested and published on real front-facing scenarios with only small movements along the camera viewing direction. We therefore prepared selected config files for KITTI Scenes (0001, 0002, 0006)

Virtual KITTI 2

bash ./download_virtual_kitti.sh

---

Training

To optimize models on a subsequence of Virtual KITTI 2 or KITTI, create the environment, download the data set (1.2) and optimize the (pre-trained) background and object models together:

conda create -n neural_scene_graphs --file requirements.txt -c conda-forge -c menpo
conda activate neural_scene_graphs

vkitti2 example:

python main.py --config example_configs/config_vkitti2_Scene06.txt
tensorboard --logdir=example_weights/summaries --port=6006

KITTI example:

python main.py --config example_configs/config_kitti_0006_example_train.txt
tensorboard --logdir=example_weights/summaries --port=6006

Rendering a Sequence

Render a pretrained KITTI sequence

bash download_weights_kitti.sh
python main.py --config example_configs/config_kitti_0006_example_render.txt

To render a pre-trained download the weights or use your own model.

bash download_weights_kitti.sh

To make a full render pass over all selected images (between the first and last frame) run the provided config with 'render_only=True'.

• To render only the outputs of the static background node use 'bckg_only=True'
• for all dynamic parts set 'obj_only=True' & 'white_bkgd=True'

python main.py --config example_configs/config_kitti_0006_example_render.txt

---

Citation

@InProceedings{Ost_2021_CVPR,
    author    = {Ost, Julian and Mannan, Fahim and Thuerey, Nils and Knodt, Julian and Heide, Felix},
    title     = {Neural Scene Graphs for Dynamic Scenes},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2021},
    pages     = {2856-2865}
}