This repository is a collection of deep learning based localization and mapping approaches. A survey on Deep Learning for Visual Localization and Mapping is offered in the following paper:
Deep Learning for Visual Localization and Mapping: A Survey
Changhao Chen, Bing Wang, Chris Xiaoxuan Lu, Niki Trigoni and Andrew Markham
IEEE Transactions on Neural Networks and Learning Systems [PDF]
A survey on Deep Learning for Inertial Positioning is offered in the following paper:
Deep Learning for Inertial Positioning: A Survey
Changhao Chen, Xianfei Pan
IEEE Transactions on Intelligent Transportation Systems [PDF]
Previous Version.
A Survey on Deep Learning for Localization and Mapping: Towards the Age of Spatial Machine Intelligence
Changhao Chen, Bing Wang, Chris Xiaoxuan Lu, Niki Trigoni and Andrew Markham
arXiv:2006.12567 [PDF]
- We released our survey paper.
- Our Survey "Deep Learning for Visual Localization and Mapping: A Survey" was accepted to IEEE TNNLS.
- Our Survey "Deep Learning for Inertial Positioning: A Survey" was accepted to IEEE TITS.
@misc{chen2020survey,
title={A Survey on Deep Learning for Localization and Mapping: Towards the Age of Spatial Machine Intelligence},
author={Changhao Chen and Bing Wang and Chris Xiaoxuan Lu and Niki Trigoni and Andrew Markham},
year={2020},
eprint={2006.12567},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
*The Date in the table denotes the publication date (e.g. date of conference).
| Models | Date | Publication | Paper | Code |
|---|---|---|---|---|
| VINet | 2017 | AAAI | VINet: Visual-Inertial Odometry as a Sequence-to-Sequence Learning Problem | |
| VIOLearner | 2019 | TPAMI | Unsupervised deep visual-inertial odometry with online error correction for rgb-d imagery | |
| SelectFusion | 2019 | CVPR | Selective Sensor Fusion for Neural Visual-Inertial Odometry | |
| DeepVIO | 2019 | IROS | DeepVIO: Self-supervised deep learning of monocular visual inertial odometry using 3d geometric constraints |
| Models | Date | Publication | Paper | Code |
|---|---|---|---|---|
| IONet | 2018 | AAAI | IONet: Learning to Cure the Curse of Drift in Inertial Odometry | |
| RIDI | 2018 | ECCV | RIDI: Robust IMU Double Integration | Py |
| Wagstaff et al. | 2018 | IPIN | LSTM-Based Zero-Velocity Detection for Robust Inertial Navigation | PT |
| Cortes et al. | 2019 | MLSP | Deep Learning Based Speed Estimation for Constraining Strapdown Inertial Navigation on Smartphones | |
| MotionTransformer | 2019 | AAAI | MotionTransformer: Transferring Neural Inertial Tracking between Domains | |
| AbolDeepIO | 2019 | TITS | AbolDeepIO: A Novel Deep Inertial Odometry Network for Autonomous Vehicles | |
| Brossard et al. | 2019 | ICRA | Learning wheel odometry and imu errors for localization | |
| OriNet | 2019 | RA-L | OriNet: Robust 3-D Orientation Estimation With a Single Particular IMU | PT |
| L-IONet | 2020 | IoT-J | Deep Learning based Pedestrian Inertial Navigation: Methods, Dataset and On-Device Inference |
| Models | Date | Publication | Paper | Code |
|---|---|---|---|---|
| Velas et al. | 2018 | ICARSC | CNN for IMU Assisted Odometry Estimation using Velodyne LiDAR | |
| LO-Net | 2019 | CVPR | LO-Net: Deep Real-time Lidar Odometry | |
| DeepPCO | 2019 | IROS | DeepPCO: End-to-End Point Cloud Odometry through Deep Parallel Neural Network | |
| Valente et al. | 2019 | IROS | Deep sensor fusion for real-time odometry estimation |
- Joint learning of depth and ego-motion has been discussed in Visual Odometry. We do not include these works here, although they can produce depth representation.
| Models | Date | Publication | Paper | Code |
|---|---|---|---|---|
| Eigen et al. | 2014 | NeurIPS | Depth Map Prediction from a Single Image using a Multi-Scale Deep Network | |
| Liu et al. | 2015 | TPAMI | Learning depth from single monocular images using deep convolutional neural fields | |
| Garg et al. | 2016 | ECCV | Unsupervised cnn for single view depth estimation: Geometry to the rescue | |
| Demon | 2017 | CVPR | Demon: Depth and motion network for learning monocular stereo | |
| Godard et al. | 2017 | CVPR | Unsupervised monocular depth estimation with left-right consistency | |
| Wang et al. | 2018 | CVPR | Learning depth from monocular videos using direct methods |
| Models | Date | Publication | Paper | Code |
|---|---|---|---|---|
| SurfaceNet | 2017 | CVPR | SurfaceNet: An End-to-end 3D Neural Network for Multiview Stereopsis | |
| Dai et al. | 2017 | CVPR | Shape completion using 3d-encoder-predictor cnns and shape synthesis | |
| Hane et al. | 2017 | 3DV | Hierarchical surface prediction for 3d object reconstruction | |
| OctNetFusion | 2017 | 3DV | Octnetfusion: Learning depth fusion from data | |
| OGN | 2017 | ICCV | Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs | |
| Kar et al. | 2017 | NeurIPS | Learning a multi-view stereo machine | |
| RayNet | 2018 | CVPR | RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials |
| Models | Date | Publication | Paper | Code |
|---|---|---|---|---|
| Fan et al. | 2017 | CVPR | A point set generation network for 3d object reconstruction from a single image |
| Models | Date | Publication | Paper | Code |
|---|---|---|---|---|
| Ladicky et al. | 2017 | ICCV | From point clouds to mesh using regression | |
| Mukasa et al. | 2017 | ICCVW | 3d scene mesh from cnn depth predictions and sparse monocular slam | |
| Wang et al. | 2018 | ECCV | Pixel2mesh: Generating 3d mesh models from single rgb images | |
| Groueix et al. | 2018 | CVPR | AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation | |
| Scan2Mesh | 2019 | CVPR | Scan2mesh: From unstructured range scans to 3d meshes | |
| Bloesch et al. | 2019 | ICCV | Learning meshes for dense visual SLAM |
| Models | Date | Publication | Paper | Code |
|---|---|---|---|---|
| SemanticFusion | 2017 | ICRA | Semanticfusion: Dense 3d semantic mapping with convolutional neural networks | |
| DA-RNN | 2017 | RSS | DA-RNN: Semantic mapping with data associated recurrent neural networks | |
| Ma et al. | 2017 | IROS | Multi-view deep learning for consistent semantic mapping with rgb-d cameras | |
| Sunderhauf et al. | 2017 | IROS | Meaningful maps with object-oriented semantic mapping | |
| Fusion++ | 2018 | 3DV | Fusion++: Volumetric object-level SLAM | |
| Grinvald et al. | 2019 | RA-L | Volumetric instance-aware semantic mapping and 3d object discovery | |
| PanopticFusion | 2019 | IROS | Panopticfusion: Online volumetric semantic mapping at the level of stuff and things |
- neural scene representation, task-driven representation
| Models | Date | Publication | Paper | Code |
|---|---|---|---|---|
| Mirowski et al. | 2017 | ICLR | Learning to navigate in complex environments | |
| Zhu et al. | 2017 | ICRA | Target-driven visual navigation in indoor scenes using deep reinforcement learning | |
| Eslami et al. | 2018 | Science | Neural scene representation and rendering | |
| CodeSLAM | 2018 | CVPR | CodeSLAM — Learning a Compact, Optimisable Representation for Dense Visual SLAM | |
| Mirowski et al. | 2018 | NeurIPS | Learning to navigate in cities without a map | |
| SRN | 2019 | NeurIPS | Scene representation networks: Continuous 3d-structure-aware neural scene representations | |
| Tobin et al. | 2019 | NeurIPS | Geometry-aware neural rendering | |
| Lim et al. | 2019 | NeurIPS | Neural multisensory scene inference |
| Models | Date | Publication | Paper | Code |
|---|---|---|---|---|
| Laskar et al. | 2017 | ICCV Workshops | Camera Relocalization by Computing Pairwise Relative Poses Using Convolutional Neural Network | |
| DELS-3D | 2018 | CVPR | Dels-3d: Deep localization and segmentation with a 3d semantic map | |
| AnchorNet | 2018 | BMVC | Improved visual relocalization by discovering anchor points | |
| RelocNet | 2018 | ECCV | RelocNet: Continuous Metric Learning Relocalisation using Neural Nets | |
| CamNet | 2019 | ICCV | Camnet: Coarse-to-fine retrieval for camera re-localization |
| Models | Date | Publication | Paper | Code |
|---|---|---|---|---|
| DSAC | 2017/07 | CVPR | DSAC - Differentiable RANSAC for Camera Localization | |
| DSAC++ | 2018/06 | CVPR | Learning less is more-6d camera localization via 3d surface regression | |
| Dense SCR | 2018/07 | RSS | Full-Frame Scene Coordinate Regression for Image-Based Localization | |
| DSAC++ angle | 2018/09 | ECCV | Scene coordinate regression with angle-based reprojection loss for camera relocalization | |
| Confidence SCR | 2018/09 | BMVC | Scene Coordinate and Correspondence Learning for Image-Based Localization | |
| ESAC | 2019/10 | ICCV | Expert Sample Consensus Applied to Camera Re-Localization | |
| NG-RANSAC | 2019/06 | CVPR | Neural-Guided RANSAC: Learning Where to Sample Model Hypotheses | |
| SANet | 2019/10 | ICCV | SANet: scene agnostic network for camera localization | |
| HSC-Net | 2020 | CVPR | Hierarchical scene coordinate classification and regression for visual localization | |
| KF-Net | 2020 | CVPR | Kfnet: Learning temporal camera relocalization using kalman filtering |
| Models | Date | Publication | Paper | Code |
|---|---|---|---|---|
| LocNet | 2018 | IV | Locnet: Global localization in 3d point clouds for mobile vehicles | |
| PointNetVLAD | 2018 | CVPR | Pointnetvlad: Deep point cloud based retrieval for large-scale place recognition | |
| Barsan et al. | 2018 | CoRL | Learning to localize using a lidar intensity map | |
| L3-Net | 2019 | CVPR | L3-net: Towards learning based lidar localization for autonomous driving | |
| PCAN | 2019 | CVPR | PCAN: 3D Attention Map Learning Using Contextual Information for Point Cloud Based Retrieval | |
| DeepICP | 2019 | CVPR | Deepicp: An end-to-end deep neural network for 3d point cloud registration | |
| DCP | 2019 | CVPR | Deep closest point: Learning representations for point cloud registration | |
| D3Feat | 2020 | CVPR | D3feat: Joint learning of dense detection and description of 3d local features |
| Models | Date | Publication | Paper | Code |
|---|---|---|---|---|
| LS-Net | 2018 | ECCV | Learning to solve nonlinear least squares for monocular stereo | |
| BA-Net | 2019 | ICLR | BA-Net: Dense bundle adjustment network |
| Models | Date | Publication | Paper | Code |
|---|---|---|---|---|
| CNN-SLAM | 2017 | CVPR | CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction | |
| Li et al. | 2019 | ICRA | Pose graph optimization for unsupervised monocular visual odometry | |
| DeepTAM | 2020 | IJCV | DeepTAM: Deep Tracking and Mapping with Convolutional Neural Networks | |
| DeepFactors | 2020 | RA-L | DeepFactors: Real-Time Probabilistic Dense Monocular SLAM |
| Models | Date | Publication | Paper | Code |
|---|---|---|---|---|
| Sunderhauf et al. | 2015 | RSS | Place recognition with convnet landmarks: Viewpoint-robust, condition-robust, training-free | |
| Gao et al. | 2017 | AR | Unsupervised learning to detect loops using deep neural networks for visual slam system | |
| Huang et al. | 2018 | RSS | Lightweight unsupervised deep loop closure | |
| Sheng et al. | 2019 | ICCV | Unsupervised Collaborative Learning of Keyframe Detection and Visual Odometry Towards Monocular Deep SLAM | |
| Memon et al. | 2020 | RAS | Loop closure detection using supervised and unsupervised deep neural networks for monocular slam systems |
| Models | Date | Publication | Paper | Code |
|---|---|---|---|---|
| Kendall et al. | 2016 | ICRA | Modelling uncertainty in deep learning for camera relocalization | |
| Kendall et al. | 2017 | NeurIPS | What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? | |
| VidLoc | 2017 | CVPR | VidLoc: A Deep Spatio-Temporal Model for 6-DoF Video-Clip Relocalization | |
| Wang et al. | 2018 | IJRR | End-to-end, sequenceto-sequence probabilistic visual odometry through deep neural networks | |
| Chen et al. | 2019 | TMC | Deep neural network based inertial odometry using low-cost inertial measurement units |
This list is maintained by Changhao Chen and Bing Wang, Department of Computer Science, University of Oxford.
Please contact them (email: changhao.chen@cs.ox.ac.uk; bing.wang@cs.ox.ac.uk), if you have any question or would like to add your work on this list.