PLINK

Probabilistic LiDAR NeRF Kodebase

Novel Viewpoint Generation

PLINK can be used to generate high quality synthetic LiDAR scans from previously unvisited locations. In the demo below, given a noisy trajectory of raw data (left), PLINK is able to accurately produce new LiDAR scans along an arbitrary sensor trajectory within that space (right). PLINK is particularly well suited for handling unstructured and noisy environments with porous and partially reflective surfaces.

Raw LiDAR data used to train PLiNK

Synthetic LiDAR Scans Generated by PLiNK

Learning a probabilistic world representation from LiDAR data

Consecutive pulses from a static LiDAR sensor should in theory return the same range measurement every time. In practice, the interplay of semi-transparent surfaces, beam divergence, fluttering of loose foliage, discretization effects, and high sensitivity at sharp corners inevitably introduces a level of stochasticity into recorded data. PLiNK rexamines the process through which a NeRF can be trained from LiDAR data in order to reconcile this difference from deterministic RGB data.

Because 3D structure is learned indirectly when training off image data, "floater" artifacts are produced in situations where degenerate geometry under-constrains the multi-view reconstruction problem. Training directly on depth information from LiDAR data alleviates this ambiguity, however, conflicting LiDAR data over-constrains the problem, introducing another kind of "phantom surface" as shown below. Attempting to directly optimize conflicting depth measurements using the traditional NeRF L2 loss regime will cause the network to learn phantom surfaces at compromise locations.

By formulating loss as an integral of probability (rather than as an integral of optical density) PLiNK allows a network to learn multiple peaks for a given ray, allowing the sampling of first, $\text{n}^{\text{th}}$, or strongest returns from a single output channel.

In the figure below, the inverse cumulative distribution can be set to a low value to render nearest returns (left), a high value to render more distant returns (right). The two renderings are different for directions viewing semi-transparent surfaces (windows) but identical for solid surfaces (walls).

The gif below demonstrates the resulting point clouds for a constant sensor pose as C is adjusted between (0,1]. For values close to 0, the rendering process produces points placed mostly on the first surface in each look direction, such as on windows and on the outermost leaves of the tree. As the value of C is increased, surfaces further along the CDF for each look direction returned-- revealing the rooms beyond the courtyard windows and the inner leaves of the tree. For values of C close to 1, the NeRF reproduces view-dependent reflections artifacts that are occasionally present in the raw sensor data.

Allowing the implicit representation to learn multiple peaks along a given ray allows PLiNK to outperform other LiDAR-based NeRF implementations in accurate scene completion.

Additional Resources

To train your own map using PLiNK, check out this jupyter notebook

For additional information about how our coarse-to-fine sampling process works, check out this jupyter notebook

For additional information on how training data is prepared, check out this jupyter notebook

If you use this work, please cite it using the following BibTeX entry:

@article{mcdermott2024plink,
  title={A Probabilistic Formulation of LiDAR Mapping With Neural Radiance Fields},
  author={Matthew McDermott and Jason Rife},
  year={2024},
  journal={arXiV preprint arXiv:2411.01725}
}