This paper has been accepted as both a journal paper in Robotics and Automation Letters (RA-L), as well as being accepted as a conference paper at IROS 2022!!!
This is the offical implementation of paper "DeepFusionMOT: A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion with Deep Association "
- (2022/6/21)This paper has been accepted as a journal paper in Robotics and Automation Letters (RA-L).
- (2022/7/31)This paper has been accepted as a conference paper in IROS 2022.
- (2022/10/25)Online presentation on IROS2022.
- (2022/11/6)Another article"StrongFusionMOT" based on DeepFusionMOT improvement was accepted by IEEE Sensors Journal.
- (2023/4/28)The code has been refactored and different cost functions have been introduced, resulting in further improvement in performance.
In the recent literature, on the one hand, many 3D multi-object tracking (MOT) works have focused on tracking accuracy and neglected computation speed, commonly by designing rather complex cost functions and feature extractors. On the other hand, some methods have focused too much on computation speed at the expense of tracking accuracy. In view of these issues, this paper proposes a robust and fast camera-LiDAR fusion-based MOT method that achieves a good trade-off between accuracy and speed. Relying on the characteristics of camera and LiDAR sensors, an effective deep association mechanism is designed and embedded in the proposed MOT method. This association mechanism realizes tracking of an object in a 2D domain when the object is far away and only detected by the camera, and updating of the 2D trajectory with 3D information obtained when the object appears in the LiDAR field of view to achieve a smooth fusion of 2D and 3D trajectories. Extensive experiments based on the KITTI dataset indicate that our proposed method presents obvious advantages over the state-of-the-art MOT methods in terms of both tracking accuracy and processing speed.
- Ubuntu 20.04
git clone https://github.com/wangxiyang2022/DeepFusionMOT
Please download the official KITTI object tracking dataset.
The final dataset organization should be like this:
DeepFusionMOT
├── data
│ ├── kitti
│ │ │── tracking
│ │ │ │──testing
| │ │ │ ├──calib
| │ │ │ | ├──0000.txt
| │ │ │ | ├──....txt
| │ │ │ | └──0028.txt
| │ │ │ ├──image_02
| │ │ │ | ├──0000
| │ │ │ | ├──....
| │ │ │ | └──0028
| │ │ │ ├──oxts
| │ │ │ | ├──0000.txt
| │ │ │ | ├──....
| │ │ │ | └──0028.txt
| │ │ │ ├──label_02
| │ │ │ | ├──0000.txt
| │ │ │ | ├──....txt
| │ │ │ | └──0028.txt
| │ │ │ ├──velodyne
| │ │ │ | ├──0000
| │ │ │ | ├──....
| │ │ │ | └──0028
│ │ │ │──training # the structure is same as testing set
| │ │ │ ├──calib
| │ │ │ ├──image_02
| │ │ │ ├──pose
| │ │ │ ├──label_02
| │ │ │ └──velodyne
cd your_path/DeepFusionMOT
pip install -r requirements.txt
Thanks to these researchers for making their code public, in this repository, for convenience, we provide the 3D detections of PointRCNN and 2D detections of RRC on the KITTI MOT dataset for car. Of course you can also use the results of other detectors.
- For 3D detections
| Frame | Type | 2D BBOX (x1, y1, x2, y2) | Score | 3D BBOX (h, w, l, x, y, z, rot_y) | Alpha |
|---|---|---|---|---|---|
| 0 | 2 | 298.3125,165.1800,458.2292,293.4391 | 8.2981 | 1.9605,1.8137,4.7549,-4.5720,1.8435,13.5308,-2.1125 | -1.7867 |
- For 2D detections
| Frame | 2D BBOX (x1, y1, x2, y2) | Score |
|---|---|---|
| 0 | 296.021000,160.173000,452.297000,288.372000 | 0.529230 |
The format definition can be found in the object development toolkit here: https://github.com/JonathonLuiten/TrackEval/blob/master/docs/KITTI-format.txt
python main.py
If you want to evaluate the tracking results using the evaluation tool on the KITTI website, you will need to go https://github.com/JonathonLuiten/TrackEval to download the evaluation code and follow the appropriate steps to set.
Using 3D detections of PointRCNN and 2D detections of RRC, the following results will be obtained.
| HOAT( ↑) | DetA( ↑) | AssA(↑) | IDSW（↓） | MOTP(↑) | MOTA(↑) | FPS（↑） | |
|---|---|---|---|---|---|---|---|
| Training dataset | 77.45% | 75.35% | 79.85% | 83 | 86.60% | 87.28% | 104 |
| Testing dataset | 75.46% | 71.54% | 80.05% | 84 | 85.02% | 84.63% | 110 |
A portion code is borrowed from AB3DMOT and Deepsort, and the visualization code from 3D-Detection-Tracking-Viewer. Many thanks to their wonderful work!
If you find this work useful, please consider to cite our paper:
@ARTICLE{9810346,
author={Wang, Xiyang and Fu, Chunyun and Li, Zhankun and Lai, Ying and He, Jiawei},
journal={IEEE Robotics and Automation Letters},
title={DeepFusionMOT: A 3D Multi-Object Tracking Framework Based on Camera-LiDAR Fusion With Deep Association},
year={2022}, volume={7}, number={3}, pages={8260-8267}, doi={10.1109/LRA.2022.3187264}}