main.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1><center>Hackathon RL@XHEC 2020</center></h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What's your objective"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of this hackathon is successfully (and consistantly) land a SpaceX rocket on its ship.\n",
    "\n",
    "To do this you will train a RL agent using the PPO (Proximal Policy Optimization) algorithm and OpenAIGym and  tensorforce libraries. \n",
    "\n",
    "You will have to understand and select the agent's hyperparameters and define a reward to allow the rocket to land.\n",
    "\n",
    "Your score will be computed based on the consistency and precision of your landings (how often you land out of 100 episodes and how close to the center you land)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Environment description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The agent will have access to the following episodic environment :\n",
    "\n",
    "Actions:\n",
    "\n",
    "-Thrust : float between 0 and 1\n",
    "\n",
    "    The force the rocket's reactor outputs \n",
    "    \n",
    "-Left(/Right) booster : float between 0 and 1\n",
    "\n",
    "    The force of the left(/Right) lateral booster of the rocket, used to correct its angle.\n",
    "    \n",
    "-Gimbal : float between -1 and 1\n",
    "\n",
    "    The angle at which the rocket outputs its thrust (see explanation below or search for gimbal rocket)\n",
    "    \n",
    "    \n",
    "States:\n",
    "\n",
    "-Angle : float between -1 and 1\n",
    "\n",
    "    The angle between the vertical axis and the rocket (negative to the left, positive to the right)\n",
    "    \n",
    "-x : float between -1 and 1\n",
    "\n",
    "    Horizontal distance from the center of the ship to the rocket. (negative to the left, positive to the right)\n",
    "    \n",
    "-y : float between 0 and 1\n",
    "\n",
    "    Vertical distance from the ship to the rocket (0 being the ship).\n",
    "    \n",
    "-distance : float between 0 and 1\n",
    "\n",
    "    Euclidean distance between the ship's center and the rocket.\n",
    "    \n",
    "-first/second leg contact : boolean\n",
    "\n",
    "    Wether or not the first/second leg has touched the ship\n",
    "    \n",
    "-velocity x/y :  float between -1 and 1\n",
    "\n",
    "    The horizontal/vertical velocity (negative when going left/up and positive when going right/down)\n",
    "    \n",
    "-velocity : float between -1 and 1\n",
    "\n",
    "    The absolute velocity (negative when going away from the ship, positive when going towards the ship)\n",
    "    \n",
    "-angular_velocity : float between -1 and 1\n",
    "\n",
    "    The rate at which the rocket turns around its center (negative when turning left positive when turning right)\n",
    "    \n",
    "-landed : boolean\n",
    "\n",
    "    Wether or not both legs are touching the ground\n",
    "    \n",
    "-landed_full : boolean\n",
    "\n",
    "    Wether or not both legs have been touching the ground for 60frames (1s), which we will aknowledge as a succesful landing!\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![\"explications\"](images/explanation.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How-to"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup\n",
    "First time:\n",
    "\n",
    "-Run the Setup cell\n",
    "\n",
    "If you have save files:\n",
    "\n",
    "-See \"Loading\" section.\n",
    "\n",
    "### Landing SpaceX\n",
    "1-Modify main.py:\n",
    "\n",
    "    -change the GROUP_NAME and COMMENT to personnalize your submissions\n",
    "    \n",
    "    -Modify create_agent\n",
    "    \n",
    "    -Modify reward_function\n",
    "    \n",
    "    -Modify the number of episodes in the main if you wish (but don't touch the number of episodes per batch)\n",
    "    \n",
    "    \n",
    "2-Run the \"Run the agent!\" cell\n",
    "\n",
    "3-While it's running you can observe the graphs available in the \"figs\" folder once the have been created/updated after each batch to monitor your training. You can also do the same with the \"vids\" folder. \n",
    "\n",
    "4-Once the \"Run the agent!\" cell has finished running, you can run the \"Show videos\" cell to see all of your videos and compare batches together (the displayed vid for each batch is the last episode of the batch, it may not be representative of your batch).\n",
    "\n",
    "### Saving\n",
    "Run the \"Save your files\" cells from time to time to save your progress\n",
    "### Loading\n",
    "1-Upload your *hackathon_save_file.zip* file in the environment (be sure it fits this exact name, rename it otherwise)\n",
    "\n",
    "2-Run the \"Load your files\" cell\n",
    "\n",
    "3-Go back to \"Landing SpaceX\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Disclaimer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\">Local files are not saved between sessions, don't forget to download your environment from time to time to be able to upload it again</span>.\n",
    "\n",
    "The environment is based on https://github.com/EmbersArc/gym_rocketLander"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!apt-get update\n",
    "!git clone https://github.com/YannBerthelot/hackathon_RL_XHEC_2020.git\n",
    "!pip install -q --pre poetry\n",
    "%cd hackathon_RL_XHEC_2020\n",
    "!apt-get install -y xvfb python-opengl ffmpeg > /dev/null 2>&1\n",
    "!poetry install\n",
    "!pip install pyvirtualdisplay"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run the agent !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!poetry run python main.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Helpful docs:\n",
    "\n",
    "https://tensorforce.readthedocs.io/en/latest/agents/ppo.html\n",
    "\n",
    "https://tensorforce.readthedocs.io/en/latest/modules/parameters.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Show videos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gym.wrappers import Monitor\n",
    "import glob\n",
    "import io\n",
    "import base64\n",
    "from IPython.display import HTML\n",
    "from pyvirtualdisplay import Display\n",
    "from IPython import display as ipythondisplay\n",
    "import os\n",
    "from show_videos import show_video\n",
    "\n",
    "number_vids = len(os.listdir(\"vids\"))\n",
    "for i in range(number_vids):\n",
    "        print(f\"Batch {i+1}\")\n",
    "        show_video(i + 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Save your files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!zip -r hackathon_save_files.zip hackathon_RL_XHEC_2020/\n",
    "from google.colab import files\n",
    "files.download(\"/content/hackathon_save_files.zip\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm hackathon_save_files.zip\n",
    "%cd hackathon_RL_XHEC_2020/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load your files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!unzip hackathon_save_files.zip\n",
    "%rm -rf hackathon_save_files.zip\n",
    "%cd hackathon_RL_XHEC_2020/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!apt-get update\n",
    "!pip install -q --pre poetry\n",
    "%cd hackathon_RL_XHEC_2020\n",
    "!apt-get install -y xvfb python-opengl ffmpeg > /dev/null 2>&1\n",
    "!poetry install\n",
    "!pip install pyvirtualdisplay"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}