Skip to content

jfpettit/flare

Repository files navigation

flare

PPO agent on LunarLanderContinuous-v2

PPO agent trained to play LunarLanderContinuous-v2. Reward per episode at this point was ~230.

Table of Contents

flare is a small reinforcement learning library. Currently, the use case for this library is small-scale RL experimentation/research. Much of the code is refactored from and built off of SpinningUp, so massive thanks to them for writing quality, understandable, and performant code.

(old) Blog post about this repository here.

Installation

MPI parallelization will soon be removed. Work is being done to rebase the code using PyTorch Lightning which uses PyTorch's multiprocessing under the hood.

Flare supports parallelization via MPI! So, you'll need to install OpenMPI to run this code. SpinningUp provides the following installation instructions:

On Ubuntu:

sudo apt-get update && sudo apt-get install libopenmpi-dev

On Mac OS X

brew install openmpi

If using homebrew doesn't work for you, consider these instructions.

On Windows

If you're on Windows, here is a link to some instructions.

Installing flare

It is recommended to use a virtual env before installing this, to avoid conflicting with other installed packages. Anaconda and Python offer virtual environment systems.

Clone the repository and cd into it:

git clone https://github.com/jfpettit/flare.git
cd flare

The next step depends on your package manager.

If you are using pip, pip install the requirements file:

pip install -r requirements.txt

Alternatively, if you're using Anaconda, create a new Anaconda env from the environments.yml file, and activate your new conda environment:

conda env create -f environment.yml
conda activate flare

A third option, if you don't want to clone a custom environment or run through the requirements.txt file, is to simply pip install the repository via:

pip install -e git+https://github.com/jfpettit/flare.git@98d6d3e74dfadc458b1197d995f6d60ef516f1ee#egg=flare

Usage

Running from command line

Each algorithm implemented can be run from the command line. A good way to test your installation is to do the following:

python -m flare.run

This will run PPO on LunarLander-v2 with default arguments. If you want to change the algorithm to A2C, run on a different env, or otherwise change some defaults with this command line interface, then do python -m flare.run -h to see the available optional arguments.

Running in a Python file

Import required packages:

import gym
from flare.polgrad import a2c 

env = gym.make('CartPole-v0') # or other gym env
epochs = 100
a2c.learn(env, epochs)

The above snippet will train an agent on the CartPole environment for 100 epochs.

You may alter the architecture of your actor-critic network by passing in a tuple of hidden layer sizes to your agent initialization. i.e.:

from flare.polgrad import ppo 
hidden_sizes = (64, 32)
ppo.learn(env, epochs=100, hidden_sizes=hidden_sizes)

Details

This repository is intended to be a lightweight and simple to use RL framework, while still getting good performance.

Algorithms will be listed here as they are implemented:

The policy gradient algorithms (REINFORCE, A2C, PPO), support running on multiple CPUs/GPUs via PyTorch Lightning. The Q Policy Gradient algorithms (SAC, DDPG, TD3) do not yet use Lightning, they will soon be brought up to parity with the policy gradient algorithms.

If you wish to build your own actor-critic from scratch, then it is recommended to use the FireActorCritic as a template.

Contributing

We'd love for you to contribute! Any help is welcome. See CONTRIBUTING.md for contributor guidelines and info.

References

More to come!

  • Update Q-policy gradient algorithms to use Pytorch Lightning
  • Comment code to make it clearer
  • Test algorithm performance