This repo serves as a companion to An Overview of Reinforcement Learning. It contains Pytorch implementations of the DQN, DDPG and PPO (for continuous action spaces) algorithms.
The DQN and DDPG algorithms are implemented in a parallel fashion in the style of APE-X with multiple CPU actor processes asynchronously communicating with a single CPU replay buffer process and a single GPU learner process.
To run any algorithm, run the command python agent.py --env <ENV_ID>.
Run python agent.py --help for further information about command line arguments.
While it is straightforward to extend the implementation of PPO to be parallel as well, it was found that using decoupled actor processes did not perform very well due to differences in the running mean and standard deviation of the observations and rewards. Solving that by having all actor processes interact with a single process that kept track of the running means and standard deviations worked but caused a communication bottleneck that obviated the advantage of parallelism in the first place. OpenAI's baseline implementation solves this by using lower-level primitives that I chose to avoid for the sake of simplicity.