E-NeRF

E-NeRF computes a Neural Radiance Field from event-camera data. It is a joint work by

Simon Klenk¹   Lukas Koestler¹   Davide Scaramuzza²   Daniel Cremers¹  

_{¹Computer Vision Group, Technical University of Munich, Germany
  ² Robotics and Perception Group, University of Zurich, Switzerland.}

IEEE Robotics and Automation Letters (RA-L), 2023 &
International Conference on Intelligent Robots and Systems (IROS), 2023

If you use this code or our paper results, please cite our work. link to paper

@article{klenk2022nerf,
  title={E-NeRF: Neural Radiance Fields from a Moving Event Camera},
  author={Klenk, Simon and Koestler, Lukas and Scaramuzza, Davide and Cremers, Daniel},
  journal={IEEE Robotics and Automation Letters},
  year={2023}
} 

Data

Datasets were simulated using esim. It can be found under https://vision.in.tum.de/research/enerf.

Code

Our python implementation is based on torch-ngp - a pytorch implementation of instant-ngp, as described in Instant Neural Graphics Primitives with a Multiresolution Hash Encoding.

Install

git clone --recursive https://github.com/knelk/enerf
cd enerf

Install with conda

conda env create -f environment.yml
conda activate enerf

Install with pip

python3 -m venv env
source env/bin/activate
pip install -r requirements.txt

Tested environments

• Ubuntu 20.04 with torch 1.10 & CUDA 11.12.2 on an NVIDIA RTX A4000.

Usage

We support 3 data formats: esim, tumvie, and eds. The data format is quite flexible and easy-to-adapt, e.g. to feed COLMAP poses or similar (have a look at nerf/provider.py and the scripts folder for data preprocessing, and associated issues in torch-ngp).

Training

./configs should be a good starting point to understand the most important flags of our code. Important event-related flags include C_thres (-1 for using the normalized loss function), events (boolean), event_only (boolean) and accumulate_evs (boolean). The scene is assumed to be in [-bound, bound], and centered at (0, 0, 0).

Please refer to torch-ngp and its issues for more details regarding the installation, usage and scene assumptions.

Data Preprocessing:

• Check the scripts folder.
• For tumvie, download data to TUMVIEDATA. Left grayscale camera = camId0, right grayscale camera = camId1.
• Run python ./scripts/undistort_images_tumvie.py --indir TUMVIEDATA --camId 0 to undistort the left images, events (by creating rectify_map_left/right.h5), and intrinsics. Before running the script, copy the respective calibration file (e.g. camera-calibrationA.json for mocap-desk2) as calibration.json into TUMVIEDATA, and copy the respective imu-calibration file (e.g. camera-calibrationA.json for mocap-desk2) as mocap-imu-calib.json into TUMVIEDATA. See tumvie homepage for details.
• Similarly, run ./scripts/undistort_images_eds.py to undistort the images, events (by creating rectify_map_left/right.h5), and intrinsics of the eds dataset (download tgz version).

Rendering

We provide a script for rendering. For example, to render the trained E-NeRF model at the keyframe (kfs) validation (val) poses, simply run python scripts/render.py --model_dir EXPDIR --infile EXPDIR/val_final_quatlist_kfs_ns.txt. If you want random poses (around the training poses), do not provide and infile and set rand_poses=1.

Troubleshooting

• Take care that the poses you provide are in the correct format (right_up_back, check e.g. this issue or this issue which also explains the idea behind poses_bounds.py format).
• Take care to undistort the images/events/e2vid-reconstructions that you are feeding to the NeRF model, as described above in Data Preprocessing.
• If the program sometimes does not start due to CUDA issues (or it gets stuck after loading the model), I often found it useful to clear the cache rm -rf ~/.cupy/ ~/.ccache/ ~/.cache/, and sometimes (rm -rf __pycache__/ raymarching/__pycache__/ gridencoder/__pycache__/ utils/__pycache__/ nerf/__pycache__/ shencoder/__pycache__/).

Acknowledgement

• torch-ngp:

@misc{torch-ngp,
    Author = {Jiaxiang Tang},
    Year = {2022},
    Note = {https://github.com/ashawkey/torch-ngp},
    Title = {Torch-ngp: a PyTorch implementation of instant-ngp}
}

• tiny-cuda-nn and instant-ngp:

  @misc{tiny-cuda-nn,
      Author = {Thomas M\"uller},
      Year = {2021},
      Note = {https://github.com/nvlabs/tiny-cuda-nn},
      Title = {Tiny {CUDA} Neural Network Framework}
  }
  
  @article{mueller2022instant,
      title = {Instant Neural Graphics Primitives with a Multiresolution Hash Encoding},
      author = {Thomas M\"uller and Alex Evans and Christoph Schied and Alexander Keller},
      journal = {arXiv:2201.05989},
      year = {2022},
      month = jan
  }
  

• The tumvie dataset:

@inproceedings{klenk2021tum,
  title={Tum-vie: The tum stereo visual-inertial event dataset},
  author={Klenk, Simon and Chui, Jason and Demmel, Nikolaus and Cremers, Daniel},
  booktitle={2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  pages={8601--8608},
  year={2021},
  organization={IEEE}
}

• The eds dataset:

 @inproceedings{hidalgo2022event,
  title={Event-aided Direct Sparse Odometry},
  author={Hidalgo-Carri{\'o}, Javier and Gallego, Guillermo and Scaramuzza, Davide},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={5781--5790},
  year={2022}
}

• E2VID reconstruction were created with this repository: e2calib

@Article{Rebecq19pami,
  author        = {Henri Rebecq and Ren{\'{e}} Ranftl and Vladlen Koltun and Davide Scaramuzza},
  title         = {High Speed and High Dynamic Range Video with an Event Camera},
  journal       = {{IEEE} Trans. Pattern Anal. Mach. Intell. (T-PAMI)},
  url           = {http://rpg.ifi.uzh.ch/docs/TPAMI19_Rebecq.pdf},
  year          = 2019
}