We demonstrate the effectiveness of using Hybrid Deep Reinforcement Learning architecture for rough terrain navigation of an ackermann-steered platform called AgileX HunterSE. The algorithm combines DRL and LQR controller and is deployed on NVIDIA's Isaac Sim simulator. The paper has been accepted at 2024 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM) held in Boston, MA, USA. Following video shows a brief overview of the paper
DhruvMehtaAIM024_FinalVideoV2.mp4
Train_Evaluate.mp4
I hope you find this research useful and would appreciate it if you cite it - Preprint is available here with more details Link: https://www.researchgate.net/publication/380245264_Rough_Terrain_Path_Tracking_of_an_Ackermann_Steered_Platform_using_Hybrid_Deep_Reinforcement_Learning
@INPROCEEDINGS{10636992,
author={Mehta, Dhruv and Salvi, Ameya and Krovi, Venkat},
booktitle={2024 IEEE International Conference on Advanced Intelligent Mechatronics (AIM)},
title={Rough Terrain Path Tracking of an Ackermann Steered Platform using Hybrid Deep Reinforcement Learning},
year={2024},
volume={},
number={},
pages={685-690},
keywords={Training;Uncertainty;Regulators;Mechatronics;Computational modeling;Deep reinforcement learning;Robustness},
doi={10.1109/AIM55361.2024.10636992}}
Following are the instructions to replicate the scenario.
PLEASE NOTE: Version 4.0.0 will be the last release of OmniIsaacGymEnvs. Moving forward, OmniIsaacGymEnvs will be merging with IsaacLab (https://github.com/isaac-sim/IsaacLab). All future updates will be available as part of the IsaacLab repository.
For tutorials on migrating to IsaacLab, please visit: https://isaac-sim.github.io/IsaacLab/source/migration/migrating_from_omniisaacgymenvs.html.
This repository contains Reinforcement Learning examples that can be run with the latest release of Isaac Sim. RL examples are trained using PPO from rl_games library and examples are built on top of Isaac Sim's omni.isaac.core and omni.isaac.gym frameworks.
Please see release notes for the latest updates.
It is recommended to have at least 32GB RAM and a GPU with at least 12GB VRAM. For detailed system requirements, please visit the Isaac Sim System Requirements page. Please refer to the Troubleshooting page for a detailed breakdown of memory consumption.
Follow the Isaac Sim documentation to install the latest Isaac Sim release.
Examples in this repository rely on features from the most recent Isaac Sim release. Please make sure to update any existing Isaac Sim build to the latest release version, 4.0.0, to ensure examples work as expected.
Once installed, this repository can be used as a python module, omniisaacgymenvs, with the python executable provided in Isaac Sim.
To install omniisaacgymenvs, first clone this repository:
git clone https://github.com/NVIDIA-Omniverse/OmniIsaacGymEnvs.gitOnce cloned, locate the python executable in Isaac Sim. By default, this should be python.sh. We will refer to this path as PYTHON_PATH.
To set a PYTHON_PATH variable in the terminal that links to the python executable, we can run a command that resembles the following. Make sure to update the paths to your local path.
For Linux: alias PYTHON_PATH=~/.local/share/ov/pkg/isaac_sim-*/python.sh
For Windows: doskey PYTHON_PATH=C:\Users\user\AppData\Local\ov\pkg\isaac_sim-*\python.bat $*
For IsaacSim Docker: alias PYTHON_PATH=/isaac-sim/python.sh
Install omniisaacgymenvs as a python module for PYTHON_PATH:
PYTHON_PATH -m pip install -e .The following error may appear during the initial installation. This error is harmless and can be ignored.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
Additional setup and example details can be found at https://github.com/isaac-sim/OmniIsaacGymEnvs
- Folder structure (Pointing out some necessary files in order to successfully train/evaluate the scenario)
├── omniisaacgymenvs
│ ├── cfg
│ │ ├── config.yaml
│ │ ├── task
│ │ │ ├── HunterTaskE2E.yaml
│ │ │ ├── HunterTask.yaml
│ │ └── train
│ │ ├── HunterTaskE2EPPO.yaml
│ │ ├── HunterTaskE2ESAC.yaml
│ │ ├── HunterTaskPPO.yaml
│ ├── robots
│ │ └── articulations
│ │ ├── hunter.py
│ ├── scripts
│ │ ├── random_policy.py
│ │ ├── rlgames_demo.py
│ │ └── rlgames_train.py
│ ├── tasks
│ │ ├── HunterTask_PyTorchE2E.py
│ │ ├── HunterTask_PyTorch.py
│ ├── USD_Files
│ │ └── hunter_aim4.usd
│ ├── utils
│ │ ├── task_util.py
│ └── Waypoints
│ ├── Austin_centerline2.csv
│ ├── BrandsHatch_centerline.csv
│ └── Silverstone_centerline.csv
├── README.md
├── setup.py
├── terrainfile_link.txt
- Config files: Simulation and DRL algorithm related parameters
- hunter.py: Imports the USD file with a specified confiuration
- Task files: HDRL (HunterTask_PyTorch.py) and End-to-End DRL (HunterTask_PyTorchE2E.py), These are scripts where states, actions, rewards, observations, etc. are defined.
- hunter_aim4.usd: USD file of the robot AgileX HunterSE
- task_util.py: Define task keys for running training and evaluation commands
- terrainfile_link.txt: Rough terrain USD file
Important: Make sure to replace the directory in task file for each of the waypoints and terrain as "home/your_username/..." and add the terrain USD file in USD_Files folder
PYTHON_PATH scripts/rlgames_train.py task=HunterTask headless=True
PYTHON_PATH scripts/rlgames_train.py task=HunterTaskE2E headless=True
- Also, running SAC gives an error, kindly change the sac_agent.py located in the following directory
cd ~/.local/share/ov/pkg/isaac-sim-4.0.0/kit/python/lib/python3.10/site-packages/rl_games/algos_torch
and modify lines 494 - 497 to the following. The issue is reported here Denys88/rl_games#263
if isinstance(next_obs, dict):
next_obs_processed = next_obs['obs']
self.obs = next_obs_processed.clone()
else:
self.obs = next_obs.clone()
PYTHON_PATH scripts/rlgames_train.py task=HunterTaskE2E train=HunterTaskE2ESAC headless=True
The trained models are saved in the run folder and to use tensorbaord
PYTHON_PATH -m tensorboard.main --logdir runs/EXPERIMENT_NAME/summaries
In order to evaluate the trained agents run the following code as per the task
PYTHON_PATH scripts/rlgames_train.py task=Ant checkpoint=runs/Experiment_Name/nn/Experiment_name.pth test=True num_envs=64
We'd like to acknowledge the following references:
@misc{makoviychuk2021isaac,
title={Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning},
author={Viktor Makoviychuk and Lukasz Wawrzyniak and Yunrong Guo and Michelle Lu and Kier Storey and Miles Macklin and David Hoeller and Nikita Rudin and Arthur Allshire and Ankur Handa and Gavriel State},
year={2021},
journal={arXiv preprint arXiv:2108.10470}
}
@misc{rudin2021learning,
title={Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning},
author={Nikita Rudin and David Hoeller and Philipp Reist and Marco Hutter},
year={2021},
journal = {arXiv preprint arXiv:2109.11978}
}
@unknown{unknown,
author = {Sakai, Atsushi and Ingram, Daniel and Dinius, Joseph and Chawla, Karan and Raffin, Antonin and Paques, Alexis},
year = {2018},
month = {08},
pages = {},
title = {PythonRobotics: a Python code collection of robotics algorithms}
}
https://github.com/AtsushiSakai/PythonRobotics
@INPROCEEDINGS{9981913,
author={Baek, Donghoon and Purushottam, Amartya and Ramos, Joao},
booktitle={2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
title={Hybrid LMC: Hybrid Learning and Model-based Control for Wheeled Humanoid Robot via Ensemble Deep Reinforcement Learning},
year={2022},
volume={},
number={},
pages={9347-9354},
keywords={Deep learning;Training;Uncertainty;Torque;Regulators;Humanoid robots;Reinforcement learning},
doi={10.1109/IROS47612.2022.9981913}}