This repository contains the source code for training and evaluating models described in the NeurIPS '21 paper Trust, but Verify: Cross-Modality Fusion for HD Map Change Detection. [Paper] [Supplementary Material]
The Trust but Verify (TbV) dataset is publicly available for download, as part of the Argoverse 2.0 family of datasets. Download instructions can be found here. You can find a short invited talk at the CVPR 2021 VOCVALC workshop summarizing our work here on Youtube.
The Trust but Verify (TbV) dataset is the first public dataset for the task of high-definition (HD) map change detection, i.e. determining when sensor data and map data are no longer in agreement with one another due to real-world changes. We collected TbV by mining thousands of hours of data from over 9 months of autonomous vehicle fleet operations.
The dataset, consisting of maps and logs collected in six North American cities, is one of the largest AV datasets to date with more than 7.9 million images and will be made available to the public, along with code and models under the the CC BY-NC-SA 4.0 license. Above, we show before-and-after images that showcase a few examples of map changes featured in TbV logs.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
First, clone the repo:
git clone https://github.com/johnwlambert/tbv.gitNext, install Miniconda or Anaconda, and create the conda environment:
conda env create -f environment_linux.yml
conda env create -f environment_mac.ymlNote: rendering data is only supported on Linux with a CUDA-supported GPU.
cd tbv
pip install -e .When you clone the repo, the structure should be as follows:
- tbv/
|--- setup.py
|--- tbv-raytracing/
|---setup.py
|---pybind11/
Next, install mseg-api anywhere on your machine using:
git clone https://github.com/mseg-dataset/mseg-api.git
cd mseg-api
pip install -e .
cd ..Next, install mseg-semantic anywhere on your machine using:
git clone https://github.com/mseg-dataset/mseg-semantic.git
pip install -e .Next, install av2 (av2-api is the official repo for Argoverse 2.0):
pip install av2==0.1.0Next, install Eigen. On Linux, sudo apt install libeigen3-dev. Next, cd tbv-raytracing and download pybind11 via git clone https://github.com/pybind/pybind11.git, where it should be downloaded into the second-level tbv-raytracing dir.
Compile the GPU library using setup.py as follows:
python setup.py bdist_wheel
pip install dist/tbv_raytracing-0.0.1-cp{PY_VERSION}-cp{PY_VERSION}-linux_x86_64.whle.g. this file could be named one of the following:
pip install dist/tbv_raytracing-0.0.1-cp38-cp38-linux_x86_64.whl
pip install dist/tbv_raytracing-0.0.1-cp310-cp310-linux_x86_64.whlDownload the dataset per the instructions found here.
Create a folder, and then logs/
To render data in a bird's eye view, run
python scripts/run_dataset_rendering_job.py --config_name bev_config.yamlTo render data in the ego-view, run
python scripts/run_dataset_rendering_job.py --config_name egoview_config.yamlProgram output will be saved in a logging_output directory.
We use the following abbreviations for city names featured in TbV:
| City Name | Abbreviation |
|---|---|
| Washington, DC | WDC |
| Miami, FL | MIA |
| Pittsburgh, PA | PIT |
| Palo Alto, CA | PAO |
| Austin, TX | ATX |
| Detroit, MI | DTW |
After rendering a dataset, you're ready to train models. Start training by running:
python scripts/train.py --config_name {CONFIG_NAME}python scripts/test.py --config_name {CONFIG_NAME}Pretrained Models: will be released shortly.
@inproceedings{Lambert21neurips_TrustButVerifyHDMapChangeDetection,
author = {Lambert, John W. and Hays, James},
booktitle = {Advances in Neural Information Processing Systems Track on Datasets and Benchmarks},
title = {{Trust, but Verify}: Cross-Modality Fusion for HD Map Change Detection},
url = {https://openreview.net/pdf?id=cXCZnLjDm4s},
year = {2021}
}All code within this repository and all data included in the TbV Dataset are governed by the CC BY-NC-SA 4.0 license. By downloading the software, you are agreeing to the terms of this license agreement. If you do not agree with these terms, you may not use or download this software. It may not be used for any commercial purpose. See LICENSE for more details.
This code and dataset are owned by Argo AI, LLC (Licensor), but are distributed by John Lambert with Argo's permission under CC BY-NC-SA 4.0.
Exclusive Remedy and Limitation of Liability: To the maximum extent permitted under applicable law, Licensor shall not be liable for direct, indirect, special, incidental, or consequential damages or lost profits related to Licensee's (you or your organization) use of and/or inability to use the Software, even if Licensor is advised of the possibility of such damage.
Disclaimer of warranties: The software is provided "as-is" without warranty of any kind including any warranties of performance or merchantability or fitness for a particular use or purpose or of non-infringement. Licensee bears all risk relating to quality and performance of the software and related materials.
Copyright: The Software is owned by Licensor and is protected by United States copyright laws and applicable international treaties and/or conventions.
Q: How you make your decision on change: In your paper, you mention that each change task is given a buffer of sensor data from time 0 to t, but in your model architectures in figure 3, I can't figure out how you incorporate the buffer. Is it at each time stamp, a change decision is made, and then you average the decision from all of the time stamps? A: Using the buffer is not strictly necessary, but in some cases, it can be useful to have. Having a buffer of past info is also fairly realistic w.r.t. onboard settings. For the bird's eye view models we trained, we used a ring buffer to keep around the past 3d points w/ their RGB values, to make a richer input texture map (see code here. For the ego-view models, we didn't use a buffer of sensor data, but there would be ways to feed into a buffer of data as input. We discuss this a bit Appendix F, page 5 of the supplement.