Revive driving scene simulation by simulator-conditioned generative models
Yunsong Zhou, Michael Simon, Zhenghao Peng, Sicheng Mo, Hongzi Zhu, Minyi Guo, and Bolei Zhou
- Presented by MetaDriverse, GenForce, and Shanghai Jiao Tong University
- 📬 Primary contact: Yunsong Zhou ( zhouyunsong2017@gmail.com )
- arXiv paper | Blog TODO | Slides
🔥 The first simulator-conditioned generative model for controllable driving scene generation with appearance
and layout
diversity.
🌟 SimGen addresses simulation to reality (Sim2Real)
gaps via cascade diffusion paradigm, and follows layout guidance from simulators and cues of the rich text prompts to realistic driving scenarios.
📊 DIVA dataset comprises 147.5 hours of web videos
and synthesized data
for diverse scene generation and advancing Sim2Real research.
[2024/06]
SimGem paper released.[2024/06]
DIVA dataset subset released.
- Release DIVA dataset
- Release SimGen code
- Toolkits for novel scene generation
You could install simgen package to enable simulator-conditioned generation.
# You don't have to create new environment, the only requirement is python>=3.10.
conda create -n simgen python=3.10
conda activate simgen
# Install this package
cd ~/SimGen
pip install -e .
# Test torch (expect True)
python -c "import torch; print(torch.cuda.is_available())"
# Install MetaDrive
cd ~/
git clone https://github.com/metadriverse/metadrive.git
cd ~/metadrive
pip install -e .
After installation, you could run the following script to test simgen pipeline. Model checkpoint will automatically be downloaded from huggingface:SichengMo-UCLA/SimGen.
python test/test.py
You cloud also use simgen with metadrive.
python metadrive_simgen.py
DIVA-Real. It collects driving videos from YouTube, covering a worldwide range of geography, weather, scenes, and traffic elements and preserving the appearance diversity of a wide range of traffic participants. Here we provide a sample of 🔗 YouTube video list we used. For privacy considerations, we are temporarily keeping the complete data labels private.
DIVA-Sim. The Sim2Real data is induced from the same real-world scenarios, in which we can obtain real-world map topology, layout, and raw sensor data. It also includes hazardous driving behaviors through interactions introduced by adversarial traffic generation. The digital twins (on nuScenes dataset) and safety-critical scenarios (on Waymo Open dataset) can be obtained through this 🔗data link.
All assets and code in this repository are under the Apache 2.0 license unless specified otherwise. The annotation data is under CC BY-NC-SA 4.0. Other datasets (including nuScenes, Waymo, and MetaDrive) inherit their own distribution licenses. Please consider citing our paper and project if they help your research.
@article{zhou2024simgen,
title={SimGen: Simulator-conditioned Driving Scene Generation},
author={Zhou, Yunsong and Simon, Michael and Peng, Zhenghao and Mo, Sicheng and Zhu, Hongzi and Guo, Minyi and Zhou, Bolei},
journal={arXiv preprint arXiv:2406.09386},
year={2024}
}
We acknowledge all the open-source contributors for the following projects to make this work possible:
You are welcome to follow other related work from , MetaDriverse, and GenForce.
- ELM | OpenScene | DriveAGI
- ScenarioNet | CAT | FreeControl