SigmaRL: A Sample-Efficient and Generalizable Multi-Agent Reinforcement Learning Framework for Motion Planning

• SigmaRL: A Sample-Efficient and Generalizable Multi-Agent Reinforcement Learning Framework for Motion Planning
  • Welcome to SigmaRL!
  • Install
  • How to Use
    • Training
    • Testing
  • Customize Your Own Maps
  • News
  • Publications
    • 1. SigmaRL
    • 2. XP-MARL
  • TODOs
  • Acknowledgements

Note

Check out our recent work XP-MARL! It is an open-source framework that augments MARL with auxiliary prioritization to address non-stationarity in cooperative MARL.

Welcome to SigmaRL!

This repository provides the full code of SigmaRL, a Sample efficiency and generalization multi-agent Reinforcement Learning (MARL) for motion planning of Connected and Automated Vehicles (CAVs).

SigmaRL is a decentralized MARL framework designed for motion planning of CAVs. We use VMAS, a vectorized differentiable simulator designed for efficient MARL benchmarking, as our simulator und customize our own RL environment. The first scenario in Fig. 1 mirrors the real-world coditions of our Cyber-Physical Mobility Lab (CPM Lab). Besides, we also support maps handcrafted in JOSM, an open-source editor for OpenStreetMap. Below you will find detailed guidance to create your OWN maps.

(a) CPM scenario.	(b) Intersection scenario.
(c) On-ramp scenario.	(d) "Roundabout" scenario.

Install

We use Python 3.9. Other versions may also work well. After git clone this repository, install the necessary packages using

pip install -r requirements.txt

We have tested this repository and confirmed that it works well on Windows and macOS. You may need to take additional steps to make it work on Linux.

How to Use

Training

Run /main_training.py. During training, all the intermediate models that have higher performance than the saved one will be automatically saved. You are also allowed to retrain or refine a trained model by setting the parameter is_continue_train in the config.json from the root directory file to true. The saved model will be loaded for a new training process.

/scenarios/road_traffic.py defines the RL environment, such as observation function and reward function. Besides, it provides an interactive interface, which also visualizes the environment. To open the interface, simply run this file. You can use arrow keys to control agents and use the tab key to switch between agents. Adjust the parameter scenario_type to choose a scenario. All available scenarios are listed in the variable SCENARIOS in utilities/constants.py. It is recommended to use the virtual visualization to check if the environment is as expected before training.

Testing

After training, run /main_testing.py to test your model. You may need to adjust the parameter path therein to tell which folder the target model was saved. Note: If the path to a saved model changes, you need to update the value of where_to_save in the corresponding JSON file as well.

Customize Your Own Maps

We support maps customized in JOSM, an open-source editor for ​OpenStreetMap. Follow these steps:

• Install and open JOSM, click the green download button
• Zoom in and find an empty area (as empty as possible)
• Select the area by drawing a rectangle
• Click "Download"
• Now you will see a new window. Make sure there is no element. Otherwise, redo the above steps.
• Customize lanes. Note that all lanes you draw are considered center lines. You do not need to draw left and right boundaries, since they will be determined automatically later by our script with a given width.
• Save the osm file and store it at assets/maps. Give it a name.
• Go to utilities/constants.py and create a new dictionary for it. You should at least give the value for the key map_path, lane_width, and scale.
• Go to utilities/parse_osm.py. Adjust the parameters scenario_type and run it.

News

• [2024-09-15] Check out our recent work XP-MARL, an open-source framework that augments MARL with auxiliary prioritization to address non-stationarity in cooperative MARL!
• [2024-08-14] We support customized maps in OpenStreetMap now (see here)!
• [2024-07-10] Our CPM Scenario is now available as an MARL benchmark scenario in VMAS (see here)!
• [2024-07-10] Our work SigmaRL was accepted by the 27th IEEE International Conference on Intelligent Transportation Systems (IEEE ITSC 2024)!

Publications

We would be grateful if you would refer to the papers below if you find this repository helpful.

1. SigmaRL

Jianye Xu, Pan Hu, and Bassam Alrifaee, "SigmaRL: A Sample-Efficient and Generalizable Multi-Agent Reinforcement Learning Framework for Motion Planning," arXiv preprint arXiv:2408.07644, 2024.

• BibTeX

  @inproceedings{xu2024sigmarl,
    title={{{SigmaRL}}: A Sample-Efficient and Generalizable Multi-Agent Reinforcement Learning Framework for Motion Planning},
    author={Xu, Jianye and Hu, Pan and Alrifaee, Bassam},
    booktitle={2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC), in press},
    year={2024},
    organization={IEEE}
  }

• Reproduce Experimental Results in the Paper:

  • Git checkout to the corresponding tag using git checkout 1.2.0
  • Go to this page and download the zip file checkpoints.zip. Unzip it, copy and paste the whole folder checkpoints to the root of this repository.
  • Run utilities/evaluation_itsc24.py.

  You can also run /testing_mappo_cavs.py to intuitively evaluate the trained models. Adjust the parameter path therein to specify which folder the target model was saved. Note: The evaluation results you get may slightly deviate from the paper since we carefully improved computation of the performance metrics.

2. XP-MARL

Jianye Xu, Omar Sobhy, and Bassam Alrifaee, "XP-MARL: Auxiliary Prioritization in Multi-Agent Reinforcement Learning to Address Non-Stationarity," arXiv preprint arXiv:2409.11852, 2024.

• BibTeX

  @article{xu2024xp,
    title={{{XP-MARL}}: Auxiliary Prioritization in Multi-Agent Reinforcement Learning to Address Non-Stationarity},
    author={Xu, Jianye and Sobhy, Omar and Alrifaee, Bassam},
    journal={arXiv preprint arXiv:2409.11852},
    year={2024},
  }

• Reproduce Experimental Results in the Paper:

  • Git checkout to the corresponding tag using git checkout 1.2.0
  • Go to this page and download the zip file checkpoints.zip. Unzip it, copy and paste the whole folder checkpoints to the root of this repository.
  • Run utilities/evaluation_icra25.py.

  You can also run /testing_mappo_cavs.py to intuitively evaluate the trained models. Adjust the parameter path therein to specify which folder the target model was saved.

TODOs

• Effective observation design
  • Image-based representation of observations
  • Historic observations
  • Attention mechanism
• Improve safety
  • Integrating Control Barrier Functions (CBFs)
  • Integrating Model Predictive Control (MPC)
• Address non-stationarity
  • Integrating prioritization (see the XP-MARL paper here)
• Misc
  • OpenStreetMap support (see guidance here)
  • Contribute our CPM scenario as an MARL benchmark scenario in VMAS (see news here)

Acknowledgements

This research was supported by the Bundesministerium für Digitales und Verkehr (German Federal Ministry for Digital and Transport) within the project "Harmonizing Mobility" (grant number 19FS2035A).