In this project, we present immersive NGP, the first open-source VR NERF Unity package that brings high resolution, low-latency, 6-DOF NERF rendering to VR. This work is based on Nvidia's ground breaking instant-ngp technique. Current version uses this commit of instant-ngp.
-
Stereoscopic, 6-DOF, real-time, low-latency NERF VR rendering in Unity.
-
DLSS Support for rendering at higher framerate.
-
6-DOF continuous locomotion in VR.
-
Offline volume image slices rendering via Unity Volume Rendering Toolkit.
-
Integration with MRTK 2.8 for building mixed reality applications with NERF.
- Unity 2019.4.29 ( Use the legacy XR manager for compatibility with OpenVR)
- instant-ngp
- Unity OpenVR desktop plugin && SteamVR
- Microsoft Mixed Reality Toolkit MRTK 2.8
- OpenGL Graphics API
- Current version of the repository was tested on Windows 10, and Oculus Quest 2.
-
Clone this repository:
git clone --recursive https://github.com/uhhhci/immersive-ngp
-
Make sure you have all the dependencies for instant-ngp installed before proceed.
-
Update dependencies for submodules
git submodule sync --recursive git submodule update --init --recursive
-
Build the instant-ngp project, similar to the build process for the original instant-ngp project.
cmake . -B build cmake --build build --config RelWithDebInfo -j
-
After succesful build, copy the following plugin files from
\instant-ngp\build\
folder to the\stereo-nerf-unity\Assets\Plugins\x86_64
folder. -
Now instant-ngp can be loaded as native plugins via Unity.
-
Open the stereo-nerf-unity Unity project with Unity 2019.4.29.
-
For a quick VR test of your own NERF scene, go to the
Assets\NERF_NativeRendering\Scenes\XRTest
scene. -
Copy the path to your nerf model, images folder, and transform.json file to the
Stereo Nerf Renderer
in theNerf path
parameters, as ilustrated below. -
Adjust DLSS settings, and image resolution as you like.
-
Now you can run the scene in Editor :)
-
Disclaimer: There is currently a small issue with running the scene in Unity Editor with native plugin clean up, you might need to restart the editor when running a new scene.
- Fix Editor restart issue
- Time-warp algorithm for latency compensation
- Dynamics Resolution
- Foveated NERF
- Support for OpenXR
- Support for higher Unity Version
- Real-time SLAM capture for dynamic grow dataset
Many thanks to the authors of these open-source repositories:
*Ke Li1, 2, * Tim Rolff1,3, Susanne Schmidt 1, Reinhard Bacher 2 , Simone Frintrop 3 , Wim Leemans 3 , Frank Steinicke 1
*These authors contributed equally to the work.
1 Human-Computer Interaction Group, Department of Informatics, Universität Hamburg
2 Deutsches Elektronen-Synchrotron DESY, Germany
3 Computer Vision Group, Department of Informatics, Universität Hamburg
Contact: ke.li1@desy.de, tim.rolff@uni-hamburg.de
@misc{Immersive-NGP,
author = {Tim Rolff and Ke Li and Susanne Schmidt and Reinhard Bacher and Simone Frintrop and Wim Leemans and Frank Steinicke},
title = {Immersive Neural Graphics Primitives},
year = {2022},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/uhhhci/Immersive-Neural-Graphics-Primitives}},
}
This work was supported by DASHH (Data Science in Hamburg - HELMHOLTZ Graduate School for the Structure of Matter) with the Grant-No. HIDSS-0002, and the German Federal Ministry of Education and Research (BMBF).
Please check here to view a copy of Nvidia's license for instant-ngp and for this repository.