Would AirSim ever become a viable RL simulator?

[HarrySoteriou]

I have been working on an autonomous landing project using computer vision and Deep RL for almost 2 years. I had chosen AirSim due to the Python API, superior documentation, photo-realistic graphics, the diverse sensor suit and the OpenAI gym integration. But I ran into some major problems that have led me to abandon the simulator for good;

1. Rendering and physics are coupled leading to annoyingly slow training. Even when using reward shaping and a pretrained feature extractor (hence only the RL policy needs to be trained), training a PPO algorithm using single 416x416 RGB images as inputs to convergence (120k timesteps) takes more than 5d on an NVIDIA 1080Ti (Pascal architecture) with a staggering 1 FPS.
2. only PPO would converge, DQN and A2C would fail miserably , even when my rewards were normalized and I made sure the agent was always attempting to complete the task and would not terminate immediately or hover to avoid the failure penalty.
3. Unable to run vectorized environments: a crucial step to make gradient policies perform (A2C is heavily dependent on this feature)
4. Unable to train multiple agents in parallel (This was definitely not possible in Python, I wonder if it is a C++ API feature)
5. Observations and actions are not synchronized, therefore an RL policy can never be learned (even the RL example on AirSim's website is sketchy). I have tried adding an observation to a queue and then perform a moveByVelocityAsynch(Vx,Vy,Vz, duration=0.5s)
   with and without the .join() command. Supposedly .join() ensures that no movement is performed until the action is completed and hopefully a mapping is learned from state s ->action a-> state s'. In my evaluation runs only when I remove the .join() argument could a trained policy achieve 100% landing success which is irrational as a new action is taken as soon as a new observation is recorded and a consistent mapping is not being utilized.
6. After training my agent with random initializations (random position and orientation towards the landing target), when attempting to validate performance on a different background the performance drastically degrades from 100% to 2%, which is unacceptable as input features are; bounding box coordinates, confidence score and class id. In addition the detector has been validated separately on real and simulated images with an accuracy of 98%.

Are there any examples using the AirSim project where RL policies have converged? Or does my 5th point stand and the AirSim simulator should not be used for RL research purposes ?

[ramzibou0735]

Hi Harry, I'm a masters student doing some research about multi-agent reinforcement learning on drones (indoor target search to be specific) and finding a simulator the could at the same time do reinforcement learning and is compatible with ROS and PX4 has been a wild ride so far, I'm curious if 1 year later you've been successful in completing your project elsewhere?
Also thanks for your post, you've raised some important things for me to look out for.

[HarrySoteriou]

I managed to train a single RL agent in AirSim to perform autonomous landing but changed jobs before I could implement it on a real drone using ROS. You should check Flightmare, PyBullet-Drones, RotoRs (Gazebo) and Unity-ML agents as viable multi-agent options and decide for yourself.

What I disliked about RotoRs (Gazebo) was wildly outdated packages and lack of documentation. I had a colleague that was running multi-agents algorithms in Gazebo (Not sure if it was RotoRs) and some other PhD's incorporate autonomous landing using Unity ML agents + ROS.

Best of luck in your endeavors.