CoAlign (ICRA2023)

Robust Collaborative 3D Object Detection in Presence of Pose Errors

Paper | Video ｜ Readme in Feishu

Update🌟 2024.1.24

HEAL is accepted to ICLR 2024. We implement a unified and integrated multi-agent collaborative perception framework for LiDAR-based, camera-based and heterogeneous setting! See HEAL GitHub.

Update🌟 2023.7.11

Camera-based collaborative perception support!

We release the multi-agent camera-based detection code, based on Lift-Splat-Shoot. Support OPV2V, V2XSet and DAIR-V2X-C dataset.

LiDAR's feature map fusion method can seamlessly adapt to camera BEV feature. Support CoAlign's multiscale fusion, V2XViT, V2VNet, Self-Att, FCooper, DiscoNet(w.o. KD). Please feel free to browse our repo. Example yamls are listed in this folder: CoAlign/opencood/hypes_yaml/opv2v/camera_no_noise

New features (Compared with OpenCOOD):

• Modality Support

  • LiDAR
  • Camera 🌟

• Dataset Support

  • OPV2V
  • V2X-Sim 2.0 🌟
  • DAIR-V2X 🌟
  • V2XSet

• SOTA collaborative perception method support

  • Attentive Fusion [ICRA2022]
  • Cooper [ICDCS]
  • F-Cooper [SEC2019]
  • V2VNet [ECCV2022]
  • FPV-RCNN [RAL2022]
  • DiscoNet [NeurIPS2021]
  • V2X-ViT [ECCV2022]
  • MASH [IROS 2021]
  • V2VNet(robust) [CoRL2020]
  • CoAlign [ICRA2023]

• Visualization support

  • BEV visualization
  • 3D visualization 🌟

• 1-round/2-round communication support

  • transform point cloud first (2-round communication)
  • warp feature map (1-round communication, by default in this repo. 🌟)

• Pose error simulation support

Installation

Please visit the feishu docs CoAlign Installation Guide for details!

Or you can refer to OpenCOOD data introduction and OpenCOOD installation guide to prepare data and install CoAlign. The installation is totally the same as OpenCOOD, except some dependent packages required by CoAlign.

Data Preparation

mkdir a dataset folder under CoAlign. Put your OPV2V, V2X-Sim, V2XSet, DAIR-V2X data in this folder. You just need to put in the dataset you want to use.

CoAlign/dataset

. 
├── my_dair_v2x 
│   ├── v2x_c
│   ├── v2x_i
│   └── v2x_v
├── OPV2V
│   ├── additional
│   ├── test
│   ├── train
│   └── validate
├── V2XSET
│   ├── test
│   ├── train
│   └── validate
├── v2xsim2-complete
│   ├── lidarseg
│   ├── maps
│   ├── sweeps
│   └── v1.0-mini
└── v2xsim2_info
    ├── v2xsim_infos_test.pkl
    ├── v2xsim_infos_train.pkl
    └── v2xsim_infos_val.pkl

Note that

1. *.pkl file in v2xsim2_info can be found in Google Drive
2. use our complemented annotation for DAIR-V2X in my_dair_v2x

Complemented Annotations for DAIR-V2X-C 🌟

Originally DAIR-V2X only annotates 3D boxes within the range of camera's view in vehicle-side. We supplement the missing 3D box annotations to enable the 360 degree detection. With fully complemented vehicle-side labels, we regenerate the cooperative labels for users, which follow the original cooperative label format.

Original Annotations	Complemented Annotations

Download: Google Drive

Website: Website

Checkpoints

Single detection with uncertainty

Download coalign_precalc and save it to opencood/logs

CoAlign Checkpoints

• CoAlign OPV2V
• CoAlign V2X-Sim 2.0
• CoAlign DAIR-V2X

Download them and save them to opencood/logs

Citation

@inproceedings{lu2023robust,
  title={Robust collaborative 3d object detection in presence of pose errors},
  author={Lu, Yifan and Li, Quanhao and Liu, Baoan and Dianati, Mehrdad and Feng, Chen and Chen, Siheng and Wang, Yanfeng},
  booktitle={2023 IEEE International Conference on Robotics and Automation (ICRA)},
  pages={4812--4818},
  year={2023},
  organization={IEEE}
}

Acknowlege

This project is impossible without the code of OpenCOOD, g2opy and d3d!

Thanks again to @DerrickXuNu for the great code framework.

Q&A

1. Different AP results between arxiv v2 and arxiv v3? and different from OPV2V[ICRA 22']?

   See Issue 4.

2. How to get V2X-Sim-2.0 pickle file?

   See Issue 2.