Zheng Zhu*, Xiaofeng Wang*, Wangbo Zhao*, Chen Min*, Nianchen Deng*, Min Dou*, Yuqi Wang*, Botian Shi#, Kai Wang#, Chi Zhang#, Yang You#, Zhaoxiang Zhang#, Dawei Zhao#, Liang Xiao#, Jian Zhao#, Jiwen Lu#, Guan Huang#
(* denotes equal contributions, # denotes corresponding authors)
(Source:Sora, DriveDreamer, DriveDreamer-2, Drive-WM, UniSim, UniPi, RoboDreamer)
This is the official repository for the technical report:
Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond.
In our report, we present a holistic examination of recent advancements in world model research, encompassing profound philosophical perspectives and detailed discussions. Our analysis delves deeply into the literature surrounding world models for video generation, autonomous driving, and autonomous agents, uncovering their applications in media production, artistic expression, end-to-end driving, games, and robots. We assess the existing challenges and limitations of world models and delve into prospective avenues for future research, with the intention of steering and igniting further progress in world models.
| Methods | Task | Github |
|---|---|---|
| Open-Sora-Plan | T2V Generation |  |
| Open-Sora | T2V Generation |  |
| Sora | T2V Generation & Editing | - |
| IRC-GAN | T2V Generation | - |
| TGANs-C | T2V Generation | - |
| TFGANs | T2V Generation | - |
| StoryGAN | T2V Generation |  |
| TiVGAN | T2V Generation | - |
| GODIVA | T2V Generation |  |
| VideoGPT | C2V Generation |  |
| StoryDALL-E | C2V Generation | - |
| CogVideo | T2V Generation |  |
| Imagen Video | T2V Generation | - |
| MAGViT | C2V Generation |  |
| MAGViT-V2 | C2V Generation |  |
| VideoPoet | T2V Generation | - |
| SVD | T2V Generation |  |
| WorldDreamer | T2V Generation |  |
| Latte | T2V Generation |  |
| StreamingT2V | T2V Generation |  |
| Methods | Task | Github |
|---|---|---|
| Iso-Dream | End-to-end Driving | - |
| MILE | End-to-end Driving |  |
| SEM2 | End-to-end Driving | - |
| TrafficBots | End-to-end Driving | - |
| Think2Drive | End-to-end Driving | - |
| GAIA-1 | Neural Driving Simulator (2D) | - |
| Tesla | Neural Driving Simulator | - |
| DriveDreamer | Neural Driving Simulator (2D) |  |
| ADriver-I | Neural Driving Simulator (2D) | - |
| DrivingDiffusion | Neural Driving Simulator (2D) | - |
| Panacea | Neural Driving Simulator (2D) |  |
| Drive-WM | Neural Driving Simulator (2D) & End-to-end Driving |  |
| WoVoGen | Neural Driving Simulator (2D) | - |
| DriveDreamer-2 | Neural Driving Simulator (2D) |  |
| GenAD | Neural Driving Simulator (2D) |  |
| SubjectDrive | Neural Driving Simulator (2D) | - |
| Copilot4D | Neural Driving Simulator (3D) | - |
| OccWorld | Neural Driving Simulator (3D) |  |
| MUVO | Neural Driving Simulator (3D) | - |
| LidarDM | Neural Driving Simulator (3D) | - |
| UniWorld | Neural Driving Simulator (3D) & 4D Pre-training | - |
| ViDAR | Neural Driving Simulator (3D) & 4D Pre-training |  |
| DriveWorld | Neural Driving Simulator (3D) & 4D Pre-training | - |
| Methods | Task | Github |
|---|---|---|
| PlaNet | Robotics |  |
| World Models | Game Agent |  |
| RobotDreamPolicy | Robotics | - |
| Plan2Explore | Robotics |  |
| DreamerV1 | Robotics |  |
| SimPLe | Game Agent |  |
| Dreaming | Robotics | - |
| DreamerV2 | Game Agent |  |
| LEXA | Robotics |  |
| PathDreamer | Indoor Navigation |  |
| DreamerPro | Robotics |  |
| DreamingV2 | Robotics | - |
| TransDreamer | Game Agent & Robotics |  |
| IRIS | Game Agent |  |
| JEPA | Framework | - |
| Dr.G | Robotics |  |
| SWIM | Robotics | - |
| DreamerV3 | Game Agent & Robotics |  |
| HarmonyDream | Game Agent & Robotics | - |
| DayDreamer | Robotics |  |
| TWM | Game Agent |  |
| STORM | Game Agent |  |
| MC-JEPA | Optics Flow Prediction | - |
| A-JEPA | Audio Classification | - |
| I_JEPA | Image Semantics |  |
| SafeDreamer | Robotics |  |
| Genie | Generative Interactive Environment | - |
| V-JEPA | Video Prediction |  |
| RoboDreamer | Robotics | - |
| UniSim | Generative Interactive Environment | - |
If you find our survey is useful in your research or applications, please consider giving us a star 🌟 and citing it by the following BibTeX entry.
@article{generalworldmodelsurvey,
title={Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond},
author={Zheng Zhu and Xiaofeng Wang and Wangbo Zhao and Chen Min and Nianchen Deng and Min Dou and Yuqi Wang and Botian Shi and Kai Wang and Chi Zhang and Yang You and Zhaoxiang Zhang and Dawei Zhao and Liang Xiao and Jian Zhao and Jiwen Lu and Guan Huang},
journal={arXiv preprint arXiv:2405.03520},
year={2024}
}