CS677_FinalProject.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "71d62341",
   "metadata": {
    "id": "71d62341"
   },
   "source": [
    "# CS673 Deep Learning | Final Project\n",
    "\n",
    "## Proposal -  Improved Bot Learning process on Atari games by using Transfer Learning\n",
    "\n",
    "## Team: \n",
    "- Ching-Hao Sun\n",
    "- Chia-Lin Hsieh\n",
    "- Rahul Gautham Putcha\n",
    "\n",
    "\n",
    "## Presentation Video\n",
    "[Presentation Video](https://drive.google.com/drive/folders/1_x79wppxM3DEz-7aGSr9ND59dF2WytP-?usp=sharing)\n",
    "\n",
    "## Index\n",
    "- [Abstract](#Abstract)\n",
    "- [Setting up Google Drive for Colab (Recommended for Training)](#Setting-up-Google-Drive-for-Colab-(Recommended-for-Training))\n",
    "- [Baseline Models](#Baseline-Models:)\n",
    "- [Candidate ML Models / Methods](#Candidate-ML-Models-/-Methods)\n",
    "- [Project Environment Setup](#Project-Environment-Setup)\n",
    "  - [Part 1: Installation of GYM](#Part-1)\n",
    "  - [Part 2: Reinforcement Learning Dependencies](#Part-2)\n",
    "- [Working on the Project: PART I - Learning the first game](#Working-on-the-Project:-PART-I---Learning-the-first-game)\n",
    "  - [Hyperparameters](#Hyperparameters)\n",
    "  - [About Reinforcement Learning](#About-Reinforcement-Learning) :\n",
    "  - [Replay Memory](#Replay-Memory)\n",
    "  - [Agent](#Agent)\n",
    "  - [Starting the Game Environment](#Starting-the-Game-Environment)\n",
    "  - [A short demo: Of how the model predicts](#A-short-demo:-Of-how-the-model-predicts)\n",
    "  - [Q-Learning](#Q-Learning)\n",
    "  - [Model Checkpointing](#Model-Checkpointing)\n",
    "  - [Game Play](#)\n",
    "  - [Training](#)\n",
    "  - [Testing](#)\n",
    "- [Transfer Learning: PART II - Learning the second game](#Transfer-Learning:-PART-II)\n",
    "  - [Training](#)\n",
    "  - [Testing](#)\n",
    "\n",
    "\n",
    "## Abstract\n",
    "Reinforcement learning algorithms require tens of thousands or millions of time steps -\n",
    "which is equivalent to several weeks of training in real time to learn how to play a\n",
    "single game. Having a bot trained from scratch is costly in terms of time and processing\n",
    "power.\n",
    "\n",
    "Suppose we have a pre-trained model of a bot that has already learnt to play one game. \n",
    "We intend to make use of the same trained-model for a bot in learning another game of \n",
    "a similar traits/environment, thereby improving the efficiency of learning the second \n",
    "game and expanding the bot’s knowledge in tackling multiple games in less time.\n",
    "\n",
    "## Baseline Models:\n",
    "CNN (Convolutional Neural Network) with DQN (Deep-Q-Network; a Q-Learning variant)\n",
    "\n",
    "\n",
    "## Candidate ML Models / Methods\n",
    "- Deep Convolutional Neural Network\n",
    "- Deep Q-Network\n",
    "- Transfer Learning\n",
    "\n",
    "### Summary\n",
    "The main goal of this project is to train the agent/model to play Breakout game from scratch and get a time estimation on training the model. Then use this same model and their weights to train on the game of Space Invader, which is having few extra high complex actions (2 ACTIONS ADDED over Breakout)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33ba0e11",
   "metadata": {
    "id": "33ba0e11"
   },
   "source": [
    "## Setting up Google Drive for Colab (Recommended for Training)\n",
    "Before carring out this section please visit the Git Repo for this repository and find IPYNB **[FIRST-PRIORITY-RUN]: Project Setup File** before this file.\n",
    "\n",
    "- [GitHub project repo](https://github.com/RPG-coder/atari-transfer-learning)\n",
    "\n",
    "Feel free to skip this process if you are doing locally on a GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "wLFFBRYomP-8",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wLFFBRYomP-8",
    "outputId": "55af5579-8257-483f-b70b-2c5618577b6d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Drive already mounted at /content/drive/; to attempt to forcibly remount, call drive.mount(\"/content/drive/\", force_remount=True).\n"
     ]
    }
   ],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "QILv3hsumUbJ",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "QILv3hsumUbJ",
    "outputId": "e4173330-4f7e-46cf-cee5-d1414d2bf776"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/content/drive/MyDrive/CS677DeepLearning/atari-transfer-learning\n"
     ]
    }
   ],
   "source": [
    "cd \"drive/MyDrive/CS677DeepLearning/atari-transfer-learning\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68cc4ae2",
   "metadata": {
    "id": "68cc4ae2"
   },
   "source": [
    "## Project Environment Setup\n",
    "\n",
    "### Part 1\n",
    "#### Requirements:\n",
    "  - Development Environment Window (Installation procedure is similar for Linux and Mac too...)\n",
    "  - Miniconda or Anaconda with conda cmd installed \n",
    "\n",
    "\n",
    "#### Install Microsoft Visual Studio 2022 (For Windows only)\n",
    "  - Select Build Tools Desktop Development with C++\n",
    "\n",
    "#### Installation process\n",
    "- Open a Terminal (For example: Command prompt) ... Require Conda cmd installed by using Miniconda installation setup\n",
    "- Setup a new environment **(Recommended)**\\\n",
    "    <code>$ conda create -n env3</code>\n",
    "    \n",
    "    <code>$ conda activate env3</code>\n",
    "\n",
    "- Install Necessary Package in our new environments\n",
    "\n",
    "  - Install Python3.7 \\\n",
    "    <code>$ conda install python=3.7</code>\n",
    "    \n",
    "  - Install OpenAI Gym for Atari games \\\n",
    "    <code>$ pip install gym[atari]</code>\n",
    "\n",
    "- With this project comes a git repository where you can download the project folder structure and the necessary file after environment setup shown above.\n",
    "   - The link to [GitHub project repo](https://github.com/RPG-coder/atari-transfer-learning)\n",
    "\n",
    "\n",
    "- After Setting up the repo locally into your computer, put all of your atari game into the './roms' folder.\n",
    "- Choose a Atari game from any of the following sources or your choice:\n",
    "  - [Breakout from oldgames.sk](https://www.oldgames.sk/en/game/breakout/download/8314/)\n",
    "  - [SpaceInvaders from consoleroms.com](https://www.consoleroms.com/roms/atari-2600/space-invaders)\n",
    "  - [SpaceInvaders from atarimania.com](http://www.atarimania.com/game-atari-2600-vcs-space-invaders_s6947.html)\n",
    "\n",
    "- Also, you can see by default the roms folder contains Breakout and SpaceInvaders '.bin' files in it.\n",
    "- After putting all of your games that you want to run in this project, go back to the terminal where you are running conda environment.\n",
    "- Run following cmd to load the game into the arcade learning environment (A way for us to use the atari games using open-ai gym) \\\n",
    "    <code>$ ale-import-roms /roms</code>\n",
    "\n",
    "**You are now all set to Run this project...**\n",
    "\n",
    "FacedError: If not all's set, no need to worry. Execute below project steps sequentially to get all dependencies setup in no time."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "184a5065",
   "metadata": {
    "id": "184a5065"
   },
   "source": [
    "### Making Gym[Atari] work on our localhost\n",
    "At first we load the games by importing the Arcade Learning Environment package, where we upload the games using ale-import-roms into this program and then use it inside gym emulator. \n",
    "\n",
    "This is a setup tutorial, if you have already have gym setup feel free to skip this section and proceed to [Part 2](#Part-2) or [Working on the Project](#Working-on-the-Project:-PART-I---Learning-the-first-game)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5bbe99c",
   "metadata": {
    "id": "b5bbe99c"
   },
   "source": [
    "This project requires Python 3.7 and gym[atari]==0.21.0\n",
    "\n",
    "**Execute below line (START to END) if you are on Google Colab**\\"
   ]
  },
  {
   "cell_type": "raw",
   "id": "c8783ea8",
   "metadata": {
    "id": "c8783ea8"
   },
   "source": [
    "!apt-get install python3.7 # GOOGLE COLAB only: Install the python versionr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74b28603",
   "metadata": {
    "id": "74b28603"
   },
   "source": [
    "\n",
    "\n",
    "**Else execute below line in conda CLI environment such as env3 as mentioned above. GOOGLE COLAB may not require below steps as it may already have them installed as pre-configured packages.**\n",
    "\n",
    "Execute on conda CLI, For Example,\\\n",
    "Conda Terminal or CMD prompt(Windows) or Terminal(Linux or Mac OS)\\\n",
    "<code>(env3) path> conda uninstall python</code>\\\n",
    "<code>(env3) path> conda install python=3.7</code>\n",
    "\n",
    "**START**"
   ]
  },
  {
   "cell_type": "raw",
   "id": "ffc34964",
   "metadata": {
    "id": "ffc34964",
    "scrolled": true
   },
   "source": [
    "# Only on Local machine\n",
    "!conda uninstall python\n",
    "!conda install python=3.7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4G6uqrPEmqdV",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4G6uqrPEmqdV",
    "outputId": "6e26a5f0-8e02-439e-8e90-030259552d6e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found existing installation: gym 0.21.0\n",
      "Uninstalling gym-0.21.0:\n",
      "  Would remove:\n",
      "    /usr/local/lib/python3.7/dist-packages/gym-0.21.0.dist-info/*\n",
      "    /usr/local/lib/python3.7/dist-packages/gym/*\n",
      "  Would not remove (might be manually added):\n",
      "    /usr/local/lib/python3.7/dist-packages/gym/envs/atari/__init__.py\n",
      "    /usr/local/lib/python3.7/dist-packages/gym/envs/atari/environment.py\n",
      "Proceed (y/n)? n\n",
      "Requirement already satisfied: gym[atari]==0.21.0 in /usr/local/lib/python3.7/dist-packages (0.21.0)\n",
      "Requirement already satisfied: numpy>=1.18.0 in /usr/local/lib/python3.7/dist-packages (from gym[atari]==0.21.0) (1.19.5)\n",
      "Requirement already satisfied: importlib-metadata>=4.8.1 in /usr/local/lib/python3.7/dist-packages (from gym[atari]==0.21.0) (4.8.2)\n",
      "Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from gym[atari]==0.21.0) (1.3.0)\n",
      "Requirement already satisfied: ale-py~=0.7.1 in /usr/local/lib/python3.7/dist-packages (from gym[atari]==0.21.0) (0.7.3)\n",
      "Requirement already satisfied: importlib-resources in /usr/local/lib/python3.7/dist-packages (from ale-py~=0.7.1->gym[atari]==0.21.0) (5.4.0)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.8.1->gym[atari]==0.21.0) (3.6.0)\n",
      "Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.8.1->gym[atari]==0.21.0) (3.10.0.2)\n"
     ]
    }
   ],
   "source": [
    "!pip uninstall gym\n",
    "!pip install gym[atari]==0.21.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "H76uLQhsmvNP",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "H76uLQhsmvNP",
    "outputId": "2b60a37b-4649-4344-9a53-8ee71762dfe2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: ale_py in /usr/local/lib/python3.7/dist-packages (0.7.3)\n",
      "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from ale_py) (1.19.5)\n",
      "Requirement already satisfied: importlib-resources in /usr/local/lib/python3.7/dist-packages (from ale_py) (5.4.0)\n",
      "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from ale_py) (4.8.2)\n",
      "Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->ale_py) (3.10.0.2)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->ale_py) (3.6.0)\n",
      "Requirement already satisfied: pyglet in /usr/local/lib/python3.7/dist-packages (1.5.0)\n",
      "Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from pyglet) (0.16.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install ale_py\n",
    "!pip install pyglet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df016e9c",
   "metadata": {
    "id": "df016e9c"
   },
   "source": [
    "**END**\n",
    "\n",
    "**(Mandatary execution)** Executing below step will import games that are necessary for our project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "51900677",
   "metadata": {
    "id": "51900677"
   },
   "outputs": [],
   "source": [
    "import ale_py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "66567d64",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "66567d64",
    "outputId": "44d3c520-ff3c-4234-8490-b5540f19d50c",
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m[SUPPORTED]    \u001b[0m             breakout              roms\\Breakout.bin\n",
      "\u001b[92m[SUPPORTED]    \u001b[0m       space_invaders         roms\\SpaceInvaders.bin\n",
      "\n",
      "\n",
      "\n",
      "Imported 2 / 2 ROMs\n"
     ]
    }
   ],
   "source": [
    "!ale-import-roms roms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0ea1c7af",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0ea1c7af",
    "outputId": "f2ca8496-921e-4319-c968-2093f09a3e7e"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\rahul\\miniconda3\\envs\\cs677\\lib\\site-packages\\ale_py\\roms\\utils.py:90: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.\n",
      "  for external in metadata.entry_points().get(self.group, []):\n",
      "C:\\Users\\rahul\\miniconda3\\envs\\cs677\\lib\\site-packages\\ale_py\\roms\\__init__.py:94: DeprecationWarning: Automatic importing of atari-py roms won't be supported in future releases of ale-py. Please migrate over to using `ale-import-roms` OR an ALE-supported ROM package. To make this warning disappear you can run `ale-import-roms --import-from-pkg atari_py.atari_roms`.For more information see: https://github.com/mgbellemare/Arcade-Learning-Environment#rom-management\n",
      "  _RESOLVED_ROMS = _resolve_roms()\n"
     ]
    }
   ],
   "source": [
    "import warnings\n",
    "from ale_py import ALEInterface\n",
    "from ale_py.roms import Breakout\n",
    "from ale_py.roms import SpaceInvaders\n",
    "\n",
    "ale = ALEInterface()        # Ignore any Deprecation warnings cause by this line\n",
    "ale.loadROM(Breakout)       # This line will load your Breakout game into this project\n",
    "ale.loadROM(SpaceInvaders)  # This line will load your SpaceInvaders game into this project"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16e0dbe2",
   "metadata": {
    "id": "16e0dbe2"
   },
   "source": [
    "Let Try to see the Breakout atari game inside of gym,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "41694f30",
   "metadata": {
    "id": "41694f30"
   },
   "outputs": [],
   "source": [
    "import gym\n",
    "env = gym.make('BreakoutDeterministic-v4')#('SpaceInvaders-v4')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "676e1f5e",
   "metadata": {
    "id": "676e1f5e"
   },
   "source": [
    "Actions moves the bot can make in this game:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9d936c5d",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9d936c5d",
    "outputId": "76b1c4ab-edb1-4da5-fb1d-519c8eaa1f5d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This game supports 4 action moves\n",
      "The moves are ['NOOP', 'FIRE', 'RIGHT', 'LEFT']\n"
     ]
    }
   ],
   "source": [
    "print(f\"This game supports {env.action_space.n} action moves\")\n",
    "print(f\"The moves are {env.unwrapped.get_action_meanings()}\") # Note that the NOOP means no operation or no move"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dedd4cfd",
   "metadata": {
    "id": "dedd4cfd"
   },
   "source": [
    "A basic game play can be executed as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "58f96f96",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "58f96f96",
    "outputId": "76189d7e-b57a-4d19-fac0-05f8f8a52541"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode is finished after the 133 timesteps\n",
      "Episode info: {'ale.lives': 0}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "env.reset() # Start the game from beginning\n",
    "t=0 # timestamp (epoch)\n",
    "while True: # Run the game till the game is over, for every timestep\n",
    "    #env.render() # COMMENT THIS LINE ON COLAB // Print the game to the screen ...\n",
    "    action = env.action_space.sample() # Random action\n",
    "    observation, reward, done, info = env.step(action) # At each step try random action\n",
    "    if done: # if the game is over (End of the game: can be win, lose or draw in any game) => Stop the game\n",
    "        print(\"Episode is finished after the {} timesteps\".format(t+1))\n",
    "        print(\"Episode info: {}\".format(info)) # What the reason? for the game to stop\n",
    "        break\n",
    "    t=t+1\n",
    "\n",
    "env.close() # Close the window\n",
    "print() # Just for format: This one just prints nothing so we can avoid the print of previous line in jupyter"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0058e6b9",
   "metadata": {
    "id": "0058e6b9"
   },
   "source": [
    "You will see the game window pop up and close automatically. \n",
    "\n",
    "If you did Hurray!! We are now able to work with any game using Gym in our project."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ba82d35",
   "metadata": {
    "id": "4ba82d35"
   },
   "source": [
    "### Part 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38b8601b",
   "metadata": {
    "id": "38b8601b"
   },
   "source": [
    "Install Reinforcement Learning process dependencies\n",
    "\n",
    "**Execute below line (START to END) in conda CLI environment such as env3 as mentioned above. GOOGLE COLAB may already have following packages pre-installed**\n",
    "\n",
    "**START**"
   ]
  },
  {
   "cell_type": "raw",
   "id": "3c5ecbfc",
   "metadata": {
    "id": "3c5ecbfc"
   },
   "source": [
    "!pip uninstall tensorflow\n",
    "!pip install --upgrade tensorflow==2.7.0\n",
    "!pip install --upgrade tensorflow-gpu==2.7.0"
   ]
  },
  {
   "cell_type": "raw",
   "id": "ef2cac8d",
   "metadata": {
    "id": "ef2cac8d"
   },
   "source": [
    "!pip uninstall keras\n",
    "!pip install keras==2.7.0"
   ]
  },
  {
   "cell_type": "raw",
   "id": "c1167ece",
   "metadata": {
    "id": "c1167ece",
    "scrolled": true
   },
   "source": [
    "!pip install numpy\n",
    "!pip install opencv-python\n",
    "!pip install pyglet\n",
    "!pip install scikit-image\n",
    "# For PIL\n",
    "!pip install pillow\n",
    "!pip install matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e97dbb62",
   "metadata": {
    "id": "e97dbb62"
   },
   "source": [
    "**END**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d647d03d",
   "metadata": {
    "id": "d647d03d"
   },
   "source": [
    "## Working on the Project: PART I - Learning the first game\n",
    "If you run above commands we will see below import modules to be successfully executed in our program"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "cb48b1d7",
   "metadata": {
    "id": "cb48b1d7"
   },
   "outputs": [],
   "source": [
    "# Basic Python Libraries\n",
    "import os\n",
    "import sys\n",
    "import itertools\n",
    "import random\n",
    "from collections import deque\n",
    "\n",
    "# Gym for loading Atari Environment compatible for Reinforcement Learning\n",
    "import gym\n",
    "from gym.wrappers import Monitor\n",
    "\n",
    "# Basic Data Science Libraries (Useful for Reinforcement Learning)\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from PIL import Image, ImageDraw\n",
    "import keras\n",
    "\n",
    "# Image Processing Libraries\n",
    "import cv2\n",
    "import imageio\n",
    "from skimage.color import rgb2gray\n",
    "from skimage.transform import resize\n",
    "from skimage import img_as_ubyte"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "21c0d9da",
   "metadata": {
    "id": "21c0d9da"
   },
   "outputs": [],
   "source": [
    "# For Deep Learning: Building Neural Network\n",
    "import tensorflow as tf\n",
    "import keras.backend as K\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Input, Flatten, Dense, Multiply, Concatenate, LeakyReLU, Lambda, Conv2D\n",
    "from tensorflow.keras.optimizers import Adam, RMSprop, SGD"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c18951f3",
   "metadata": {
    "id": "c18951f3"
   },
   "source": [
    "### Hyperparameters\n",
    "Below are the hyperparameters that we are using to tune the learning process of the agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8b31edc6",
   "metadata": {
    "id": "8b31edc6"
   },
   "outputs": [],
   "source": [
    "# Hyperparameters space\n",
    "HYPERPARAMS = {\n",
    "    # Google COLAB Setting\n",
    "    \"ON_COLAB\": False,\n",
    "    \"COLAB_COMPATIBLE_MODEL\": False,  # We have seen that Colab's GYM env considers all 18 actions of Atari game machine\n",
    "                                      # This is alleveated by installing newer gym=0.21.0 and pyglet, !pip install pyglet\n",
    "                                      # This parameter say our model to consider 18 actions by default.\n",
    "                                      # In our project we mistakenly trained old project model on 18 action which made the training time high due to larger actions\n",
    "                                      # In case we are required to make use of backward compatible model we set this to TRUE\n",
    "    \n",
    "    # Reinforcement Learning Parameters\n",
    "    \"ENV_NAME\"        : 'BreakoutDeterministic-v4', # Name of environment to be used\n",
    "    # ('SpaceInvaders-v0') # ('Assault-ram-v0')\n",
    "    \"MEM_SIZE\"        : 18000, # Size of replay memory\n",
    "    \"GAMMA\"           : 0.99,   # Gamma (Discount rate) of Markov decision process\n",
    "    \n",
    "    # Exploration vs Exploitation (for Epsilon Decay Policy)\n",
    "    \"EPSILON\"         : 0.5,     # Start agent at exploration stage=1, or exploitation=0\n",
    "    \"EPSILON_MIN\"     : 0.01,\n",
    "    \"EPSILON_DECAY\"   : 0.5,\n",
    "    \"TOTAL_FRAMES\"    : 5000000,\n",
    "    \"EPSILON_MAX\"     : 1,\n",
    "    \n",
    "    # Model Training Hyper Parameters\n",
    "    \"LEARNING_RATE\"   : 0.00000001,\n",
    "    \"MOMENTUM\"        : 0.001,\n",
    "    \"STACK_SIZE\"      : 4,\n",
    "    \"HIDDEN_NEURONS\"  : 1024,  # Number of Neurons in the Deep Neural Network\n",
    "    \"MINIBATCH_SIZE\"  : 96,\n",
    "    \"NUM_EPISODES\"    : 20000, # Number of episodes/gameplay for the agent training\n",
    "    \"RETRAIN\"         : 32,  # Number of times the agent background model trains on its Replay memory before proceeding\n",
    "    \n",
    "    # Model used in Replay Memory for Exploitation (Learning to win) on What has be Explored (or What has been found).\n",
    "    \"TGT_UPDATE_FREQ\" : 1500,\n",
    "    \"NUM_EXPLORE\"     : 1000,\n",
    "    \n",
    "    # For demo purpose\n",
    "    \"VIS_DIR\"               : \"GIFs\", # For GOOGLE COLAB Environment only\n",
    "    \"AUTOSAVE_CHECKPOINT\"   : 500,   # Auto save model after a number of episode = 100\n",
    "    \"SAVED_MODEL_NAME\"      : \"model_dqn_breakout.h5\", # Name of final model, second game \"model_dqn_spaceshoot.h5\"\n",
    "    \"TRANSFER_MODEL_NAME\"   : \"model_dqn_breakout.h5\",\n",
    "    \n",
    "    # Extra Tuning\n",
    "    \"NOOPMAX\"         : 8, # Maximum number of No operation actions taken at the beginning of the game (For using every exploration)\n",
    "    \n",
    "    # Testing\n",
    "    \"NUM_EVAL\"        : 20, # Number of Evals (Test runs)\n",
    "    \n",
    "    \"LIVES\"           : \"ale.lives\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d3dbb925",
   "metadata": {
    "id": "d3dbb925"
   },
   "outputs": [],
   "source": [
    "#Lists to store loss and reward value per game!\n",
    "LOSS_HISTORY = []\n",
    "REWARD_HISTORY = []\n",
    "\n",
    "# Number of Frame viewed OR number of times env.step(action) called.\n",
    "FRAME_COUNT = 0\n",
    "\n",
    "model_swap =1\n",
    "if not os.path.exists(\"models\"): \n",
    "    os.mkdir(\"models\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8973e18a",
   "metadata": {
    "id": "8973e18a"
   },
   "source": [
    "### About Reinforcement Learning\n",
    "Reinforcement Learning is a process of teaching the machine to learn by itself without any external/human intervention. The model is said to learn from its environment it is put in and also learn from its past moves (REPLAY MEMORY).\n",
    "\n",
    "**In simple words reinforcement learning is the process of finding the optimal plays that can be made from the Markov Decision Tree process.**\n",
    "\n",
    "In Reinforcement Learning a model is also called as the agent. The agent learns from an environment (GYM-ATARI, in this case) its put in. The only dependency for reinforcement learning is for the agent to differentiate a good moves or bad moves or decent moves made by its predict(), and along with this it must also know whether the game is done(or over) after it's interaction with the environment. One may say, we can use OpenCV for taking screenshots & image processing or PyGame for emulator, but it may become a burden for initial setting. \n",
    "\n",
    "GYM come to rescue for such a behavior as it is built for satisfying such reinforcement learning dependencies.\n",
    "\n",
    "**State-Sequence: Reinforcement Learning**\n",
    "![State Sequence - Reinforcement Learning](./Designs/StateSequence.png)\n",
    "\n",
    "\n",
    "**Reinforcement Learning**\n",
    "![Reinforcement Learning](./Designs/ReinforcementLearning.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b56db728",
   "metadata": {
    "id": "b56db728"
   },
   "source": [
    "### Replay Memory\n",
    "Replay Memory for improving the agent model by making it play (or fitting) over it past experience, stored in it's Memory."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f7ed55e",
   "metadata": {},
   "source": [
    "![ReplayMemory](./Designs/ReplayMemory.png)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea0910d1",
   "metadata": {},
   "source": [
    "- Replay Memory is basically like human memory, that stores \n",
    "  - state: STACK_SIZE=4 number of images, for representing current state being played\n",
    "  - action: Current Action being played\n",
    "  - next_state: Next STACK_SIZE=4 images seen after an actions is played on current state \n",
    "  - done: Is the game over after the action is played\n",
    "  - reward: What is the reward (100:good,-10:bad or 0:decent)\n",
    "- **The unbalanced dataset problem and solution:**\n",
    "  - In any real world problem the dataset which we get is unbalanced.\n",
    "  - In our case, the unbalanced data set is due to the number of wrong moves made early into the game.\n",
    "  - This may over shadow the good moves and made in the game as the come to be at minor population.\n",
    "  - To avoid this we decided to split the Replay Memory into 3 different queues:\n",
    "    - Good Moves\n",
    "    - Bad Moves\n",
    "    - Decent Moves\n",
    "  - The grouping of these moves is performed using variable `done` and variable `reward`\n",
    "  - Below diagram gives and idea of how the grouping is done. The 32 records comes from the **HYPERPARAMS[MIN_BATCH_SIZE]**\n",
    "  - Before Training we randomly pick 32 samples features from each memory and Merge them into one minibatch for training model as a batch.\n",
    "  \n",
    "![Balanced Replay Memory](./Designs/BalancedReplayMemory.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a7cec6e5",
   "metadata": {
    "id": "a7cec6e5"
   },
   "outputs": [],
   "source": [
    "# For Boosting Experience\n",
    "class Replay_Memory:\n",
    "    '''\n",
    "        This replay memory clas would act as a buffer in which previous experiences would be stored. \n",
    "        Agent Experience = [ state=current_state, action=current_action, reward, next_state, done]\n",
    "    '''\n",
    "    def __init__(self, MEM_SIZE = 2000): \n",
    "        self.wrong_move_memory = deque(maxlen = MEM_SIZE//3)  # Bad Moves\n",
    "        self.decent_move_memory = deque(maxlen = MEM_SIZE//3) # No Change\n",
    "        self.right_move_memory = deque(maxlen = MEM_SIZE//3)  # Good Moves\n",
    "        self.max_size = MEM_SIZE\n",
    "        \n",
    "    def add(self,  state, action, reward, next_state, done):\n",
    "        if reward < 0 or done:\n",
    "            self.wrong_move_memory.append(( state, action, reward, next_state, done))\n",
    "        elif reward > 0:\n",
    "            self.right_move_memory.append(( state, action, reward, next_state,done))\n",
    "        else:\n",
    "            self.decent_move_memory.append(( state, action, reward, next_state,done))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "885004f1",
   "metadata": {
    "id": "885004f1"
   },
   "source": [
    "### Agent\n",
    "The agent class: contains the following attribute,\n",
    "- Performs Learning using Reinforcement Learning\n",
    "- Saves/Loads the model\n",
    "- Contains Actual training model and Target model for estimating future rewards\n",
    "- The Actual model plays the game (Exploration and Trains)\n",
    "- Target model is a copy of Actual model that gives the future Qvalues for each action made on current state.\n",
    "- For every sample feature from replay memory, the target value gives the estimated Q-value based on the state and action\n",
    "- Output of Targert model is determined by **Q(s', a')**\n",
    "$$ Q_{current\\_state}(s,a) = r + \\gamma \\cdot \\max _{a'} Q_{next\\_state}(s',a') $$\n",
    "- The result is used as the target for that particular sample feature containing the state and next state in replay memory\n",
    "- This value is sum up with the reward received with conditional done/terminal condition that say if there is any future left for current state and upon which action is played.\n",
    "\n",
    "\n",
    "- **Special:** This agent can also perform Transfer Learning. Use a previous lower complex model to learn a similar higher complex task.\n",
    "- Transfer Learning is a ability of making an agent that has learned to play one game to play another game of similar but higher complexity, in a short duration.\n",
    "\n",
    "![Model Design](./Designs/ModelDesign.PNG)\n",
    "\n",
    "**Note that we are taking images as grayscale not RGB as depicted in the picture above**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ee6790fc",
   "metadata": {
    "id": "ee6790fc"
   },
   "outputs": [],
   "source": [
    "# Model/Agent\n",
    "class Agent:\n",
    "    '''This class contains all methods for an agent to function.'''\n",
    "    \n",
    "    def __init__(self, env, action_size=None, model_pathname=\"\"):\n",
    "        print(\"Setting up the agent ...\")\n",
    "        self.reset_parameters(env, action_size)\n",
    "        self.model = self.build_model(model_pathname) if model_pathname else self.build_model()\n",
    "        self.target_model = self.get_tgt_model()\n",
    "        \n",
    "        print(\"Agent has been Sucessfully setup ...\")\n",
    "        \n",
    "    def reset_parameters(self, env, action_size=None):\n",
    "        self.state_size = env.observation_space.shape[0]\n",
    "        self.action_size = 18 if HYPERPARAMS[\"COLAB_COMPATIBLE_MODEL\"] else env.action_space.n\n",
    "        if action_size: self.action_size = action_size\n",
    "        self.memory = Replay_Memory(HYPERPARAMS[\"MEM_SIZE\"])\n",
    "        self.gamma = HYPERPARAMS[\"GAMMA\"]\n",
    "        self.epsilon = HYPERPARAMS[\"EPSILON\"]\n",
    "        self.epsilon_max = HYPERPARAMS[\"EPSILON_MAX\"]\n",
    "        self.epsilon_min = HYPERPARAMS[\"EPSILON_MIN\"]\n",
    "        self.total_frame = HYPERPARAMS[\"TOTAL_FRAMES\"]\n",
    "        self.slope = (self.epsilon_max - self.epsilon_min)/self.total_frame\n",
    "        self.epsilon_decay = HYPERPARAMS[\"EPSILON_DECAY\"]\n",
    "        self.lr = HYPERPARAMS[\"LEARNING_RATE\"]\n",
    "        self.momentum = HYPERPARAMS[\"MOMENTUM\"]\n",
    "        self.dummy_input = np.zeros((1,self.action_size))\n",
    "        self.dummy_batch = np.zeros((HYPERPARAMS[\"MINIBATCH_SIZE\"],self.action_size))\n",
    "        \n",
    "    # Function for Agent Model Setup\n",
    "    def lambda_out_shape(self, input_shape):\n",
    "        shape = list(input_shape)\n",
    "        shape[-1] = 1\n",
    "        return tuple(shape)\n",
    "        \n",
    "    \n",
    "    def build_model(self, model_pathname=\"\"):\n",
    "        '''Model to train the agent'''\n",
    "        return self._build_compatible_model(self.action_size, model_pathname=model_pathname)\n",
    "\n",
    "    def transfer_learning(self, new_env, prev_model_pathname, old_actions_size, new_actions_size):\n",
    "        '''\n",
    "            Perform a transfer of knowledege about a game of similar less complex enviroment to this game.\n",
    "            Please supply:\n",
    "            - model_pathname: the location of model file that has already learnt to training on a game.\n",
    "            - old_action_size: the estimation of size for the already learnt game.\n",
    "            - new_action_size: the estimation of size for the current game being played.\n",
    "            Returns model\n",
    "        '''\n",
    "        self.model = self._build_compatible_model(old_actions_size, True, new_actions_size,prev_model_pathname)\n",
    "        self.target_model = self._build_compatible_model(old_actions_size, True, new_actions_size,prev_model_pathname)\n",
    "        self.reset_parameters(new_env)\n",
    "        return\n",
    "\n",
    "    def _build_compatible_model(self, actions_size, is_transfer_learning=False, new_actions_size=0, model_pathname=\"\"):\n",
    "        '''A single method to build model normally or build model after transfer learning from another game'''\n",
    "        prev_actions_size = actions_size\n",
    "        agent_bot = self\n",
    "        \n",
    "        input_layer = Input(shape=(84, 84, HYPERPARAMS[\"STACK_SIZE\"]), name=\"image\")     # Sending the stack of 4 resized image of 84*84\n",
    "\n",
    "        # Convolution-Max Pooling parts\n",
    "        conv_layer1 = Conv2D(32, (8, 8), strides=(4, 4), activation='relu', use_bias=False, name=\"conv2D_1\", kernel_initializer=tf.keras.initializers.VarianceScaling(scale=2))(input_layer)\n",
    "        # May Introduce Max Pooling here ...\n",
    "        conv_layer2 = Conv2D(64, (4, 4), strides=(2, 2), activation='relu', use_bias=False, name=\"conv2D_2\", kernel_initializer=tf.keras.initializers.VarianceScaling(scale=2))(conv_layer1)\n",
    "        # May Introduce Max Pooling here ...\n",
    "        conv_layer3 = Conv2D(64, (3, 3), strides=(1, 1), activation='relu', use_bias=False, name=\"conv2D_3\", kernel_initializer=tf.keras.initializers.VarianceScaling(scale=2))(conv_layer2)\n",
    "        # May Introduce Max Pooling here ...\n",
    "        conv_layer4 = Conv2D(HYPERPARAMS[\"HIDDEN_NEURONS\"], (7, 7), strides=(1, 1), activation='relu', use_bias=False, name=\"conv2D_4\", kernel_initializer=tf.keras.initializers.VarianceScaling(scale=2))(conv_layer3)\n",
    "\n",
    "        # Densely connected Neural Network\n",
    "        flat_feature = Flatten(name=\"flat_1\")(conv_layer4)\n",
    "        q_value_prediction = Dense(prev_actions_size, name=\"q_values\", activation='relu')(flat_feature) # Setting up the Output Layer\n",
    "        model = Model(inputs=[input_layer], outputs=[q_value_prediction])\n",
    "        \n",
    "        # LOAD AUTO_SAVE CHECKPOINT: If filename is provided and not for Transfer learning\n",
    "        if len(model_pathname)>0 and is_transfer_learning==False:\n",
    "            model.load_weights(model_pathname)\n",
    "\n",
    "        # PREFORM TRANSFER LEARNING FROM OLD MODEL: If filename is provided and not for Transfer learning\n",
    "        if is_transfer_learning and len(model_pathname)>0:\n",
    "            # Load model for previous game\n",
    "            model.load_weights(model_pathname)\n",
    "            \n",
    "            # Sibling layers for learning actions\n",
    "            prev_Qlayer = model.get_layer(name=\"q_values\")\n",
    "            prev_Qlayer._name = \"old_action_q_values\"\n",
    "            new_Qlayer = Dense(new_actions_size, name=\"new_action_q_values\", activation='relu')\n",
    "\n",
    "            # Merge the Sibling layers to form one layer to estimate Q-values\n",
    "            q_value_prediction = Concatenate(name=\"q_values\")([prev_Qlayer(flat_feature), new_Qlayer(flat_feature)])\n",
    "            model = Model(inputs=[input_layer], outputs=[q_value_prediction])\n",
    "        \n",
    "        model.compile(loss=['mse','mse'], loss_weights=[0.0,1.0],optimizer=Adam(agent_bot.lr))\n",
    "        return model\n",
    "        \n",
    "    def get_tgt_model(self):\n",
    "        '''This method would clone the architecture as well as the initial weights of the base model into target model'''\n",
    "        self.target_model = self.build_model()\n",
    "        self.update_target_model()\n",
    "        return self.target_model\n",
    "    \n",
    "    def update_target_model(self): \n",
    "        '''This method would update weights of target model'''\n",
    "        self.target_model.set_weights(self.model.get_weights())\n",
    "    \n",
    "    def load_model(self, pathname): \n",
    "        '''This method would load weights of model'''\n",
    "        # load weights into new model\n",
    "        self.model.load_weights(pathname)\n",
    "        self.target_model = self.get_tgt_model()\n",
    "        print(\"Loaded model from disk\")\n",
    "        \n",
    "    def save_model(self, pathname):\n",
    "        '''Save method would save weights of model model'''\n",
    "        # serialize weights to HDF5\n",
    "        self.model.save_weights(pathname)\n",
    "        self.target_model = self.get_tgt_model()\n",
    "        print(\"Saved model to disk\")\n",
    "        \n",
    "\n",
    "    # Agent Play Prediction function\n",
    "    def next_action(self, state):\n",
    "        '''Get the next action using epsilon greedy policy for deciding whether to exploit or explore'''\n",
    "        \n",
    "        if self.epsilon > self.epsilon_min: # Epsilon \n",
    "            self.epsilon = self.epsilon_max - self.slope*(FRAME_COUNT)\n",
    "            \n",
    "            \n",
    "        if (np.random.rand() <= self.epsilon):\n",
    "            return env.action_space.sample()\n",
    "        #print(self.dummy_input.shape)\n",
    "        q_values = self.model.predict([np.expand_dims(state,axis=0)])[0]\n",
    "        return np.argmax(q_values[0])\n",
    "\n",
    "    # Replay Functions\n",
    "    def store_experience(self, state, action, reward, next_state, done):\n",
    "        '''Store the experience in our replay memory'''\n",
    "        self.memory.add(state, action, reward, next_state, done)\n",
    "\n",
    "    def replay(self, batch_size, model_swap):\n",
    "        '''\n",
    "            Does the back propogation to adjust weights during exploitation action.\n",
    "            - batch_size: total number of random samples that the agent can recollect from memory\n",
    "            - The higher the batch_size more is the time for training process.\n",
    "        '''\n",
    "        # REINFORCEMENT LEARNING\n",
    "        print(\"Game Play Paused! Model is training on it's past Memory\")\n",
    "        # First we set all input to NOOP or no move for every observations(stack of frames or images)\n",
    "        # Dummy_Inputs_batch.shape = [(MINBATCH_SIZE = 32 images), (action_size = 4 moves for breakout)]\n",
    "        #dummy_batch = np.zeros((batch_size,18 if HYPERPARAMS[\"COLAB_COMPATIBLE_MODEL\"] else self.action_size)) \n",
    "            \n",
    "        #for i in np.arange(HYPERPARAMS[\"RETRAIN\"]): # Experimental 'RETRAIN'\n",
    "        # Experience batch set\n",
    "        state_batch      = []\n",
    "        action_batch     = []\n",
    "        reward_batch     = []\n",
    "        next_state_batch = []\n",
    "        terminal_batch   = []  # recording Is_done?\n",
    "        \n",
    "        # Actual Move that should have played (This is also an Assumption)\n",
    "        y_batch = []\n",
    "\n",
    "        # Sample random minibatch of transition from replay memory\n",
    "        minibatch = random.sample(list(self.memory.wrong_move_memory), batch_size//3) + random.sample(list(self.memory.right_move_memory), batch_size//3) + random.sample(list(self.memory.decent_move_memory), batch_size//3)\n",
    "        #print(minibatch[0])\n",
    "        # For every experience thats in our Replay Memory\n",
    "        for data in minibatch: # We organize the data\n",
    "            state_batch.append(data[0])\n",
    "            action_batch.append(data[1])\n",
    "            reward_batch.append(data[2])\n",
    "            next_state_batch.append(data[3])\n",
    "            terminal_batch.append(data[4]) # is_done!!\n",
    "        \n",
    "        # Convert the is_done to a numpy array\n",
    "        terminal_batch = np.array(terminal_batch, dtype=\"int8\")\n",
    "\n",
    "         \n",
    "        \n",
    "        # Get what agent is assuming with the trained model till now. Supplying NOOP/NoAction input for every move... \n",
    "        # Model is predicting the Q-Value or we can all it as Future reward from current move (or action) made\n",
    "        target_q_values_batch = self.target_model.predict([np.float32(np.array(next_state_batch))])[0]\n",
    "        # What model should assume (The Assumption is to predict its own output without any gameover) Outrageous!!\n",
    "        y_batch = reward_batch + (1 - terminal_batch) * self.gamma * np.max(target_q_values_batch, axis=-1)\n",
    "        # (1 - terminal_batch) above is to indicate the game is_done(0, return only reward for bad move) or in_progress(1, return reward with discounted sum)\n",
    "        # y_batch is also called Future Reward or the reward model is expecting to get in the future.\n",
    "\n",
    "        # START TRAINING PROCESS\n",
    "        # Get the loss between current state Q value & the future state that Q-values\n",
    "        loss = self.model.train_on_batch([np.float32(np.array(state_batch))], [y_batch])\n",
    "\n",
    "        LOSS_HISTORY.append(loss)\n",
    "    \n",
    "        # END TRAINING PROCESS: 'RETRAIN\"\n",
    "        \n",
    "            #At target network's update frequency, update the target network\n",
    "        if(model_swap % HYPERPARAMS[\"TGT_UPDATE_FREQ\"] == 0):\n",
    "            self.update_target_model()  # Swap Model\n",
    "            print(\"Target model swapped successfully with Trained model!\")   \n",
    "            \n",
    "        return "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7e8ee6a1",
   "metadata": {
    "id": "7e8ee6a1"
   },
   "outputs": [],
   "source": [
    "# Helper functions\n",
    "\n",
    "def preprocess(image):\n",
    "    '''This method downsamples and resizes the images to 84*84 and converts it to grayscale for CNN compatibility and processing efficiency'''\n",
    "    # Downsample(image) & resize image into square for CNN compatibility\n",
    "    #resized_image = preprocess_rgb(image)\n",
    "    #grayscale_image = rgb2gray(resized_image)\n",
    "    \n",
    "    # crop image (top and bottom, top from 34, bottom remove last 16)\n",
    "    img = image[34:-16, :, :]\n",
    "    # resize image\n",
    "    img = cv2.resize(img, (84, 84))\n",
    "    img = img.mean(-1,keepdims=True)\n",
    "    img = img.astype('float32') / 255.\n",
    "    return img #grayscale_image\n",
    "\n",
    "def preprocess_rgb(image):\n",
    "    '''This method downsamples and resizes the images to 84*84 and converts it to grayscale for CNN compatibility and processing efficiency'''\n",
    "    # Downsample(image) & resize image into square for CNN compatibility\n",
    "    resized_image = cv2.resize(image[::2, ::2], (84, 84), interpolation = cv2.INTER_AREA)\n",
    "    return resized_image\n",
    "\n",
    "def generate_gif(frame_no, frames, reward, path, e):\n",
    "    '''Utility method to generate gif from frames'''\n",
    "    for idx, frame_idx in enumerate(frames): \n",
    "        frames[idx] = resize(frame_idx, (420, 320, 3), preserve_range=True, order=0).astype(np.uint8)\n",
    "        \n",
    "    imageio.mimsave(f'{path}{\"episode_{0}_frame_{1}_reward_{2}.gif\".format(e, frame_no, reward)}', frames, duration=1/100)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20f9af08",
   "metadata": {
    "id": "20f9af08"
   },
   "source": [
    "### Starting the Game Environment\n",
    "As seen previously we have installed the environment for running any game within gym emulator setup. Now is the time to get things working in action, for the main aim of this project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1e955804",
   "metadata": {
    "id": "1e955804"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\rahul\\miniconda3\\envs\\cs677\\lib\\site-packages\\ale_py\\roms\\utils.py:90: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.\n",
      "  for external in metadata.entry_points().get(self.group, []):\n",
      "C:\\Users\\rahul\\miniconda3\\envs\\cs677\\lib\\site-packages\\ale_py\\roms\\__init__.py:94: DeprecationWarning: Automatic importing of atari-py roms won't be supported in future releases of ale-py. Please migrate over to using `ale-import-roms` OR an ALE-supported ROM package. To make this warning disappear you can run `ale-import-roms --import-from-pkg atari_py.atari_roms`.For more information see: https://github.com/mgbellemare/Arcade-Learning-Environment#rom-management\n",
      "  _RESOLVED_ROMS = _resolve_roms()\n"
     ]
    }
   ],
   "source": [
    "from ale_py import ALEInterface\n",
    "from ale_py.roms import Breakout, SpaceInvaders,Tetris\n",
    "ale = ALEInterface()\n",
    "ale.loadROM(Breakout) #ale.loadROM(Breakout)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b7c716e3",
   "metadata": {
    "id": "b7c716e3"
   },
   "outputs": [],
   "source": [
    "import gym\n",
    "env = gym.make(HYPERPARAMS[\"ENV_NAME\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b0dff4f1",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "b0dff4f1",
    "outputId": "bf21bd88-206a-4027-f1b4-baa894884542"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ENVIRONMENT BreakoutDeterministic-v4:\n",
      "This environment requires 4 actions.\n",
      "The actions are ['NOOP', 'FIRE', 'RIGHT', 'LEFT'].\n"
     ]
    }
   ],
   "source": [
    "print(f'ENVIRONMENT {HYPERPARAMS[\"ENV_NAME\"]}:')\n",
    "print(f'This environment requires {env.action_space.n} actions.')\n",
    "print(f'The actions are {env.unwrapped.get_action_meanings()}.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "9c62ec12",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9c62ec12",
    "outputId": "8e260c6f-66ca-4cce-a121-2d115ac769e2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setting up the agent ...\n",
      "Agent has been Sucessfully setup ...\n"
     ]
    }
   ],
   "source": [
    "agent = Agent(env)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a74f5574",
   "metadata": {
    "id": "a74f5574"
   },
   "source": [
    "### A short demo: Of how the model predicts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1259284a",
   "metadata": {
    "id": "1259284a"
   },
   "source": [
    "For the first step we are delivering the **STACK_SIZE=4** number of images at a time in our agent model. Along with this is the current action being played. The action signifies the last move that was played, i.e. the move played by in the last image of the 4 image stack/sequence.\n",
    "\n",
    "You may wonder why we are considering the **STACK_SIZE**. Firstly, DeepMind choose to use the past 4 frames. Why?\n",
    "1. frame doesnot describe the movement of player or the enemies or any items. (Relative motion of any object)\n",
    "2. frames bare minimum requirement to learn about the speed of objects. (We capture the relative position on object between 2 frames)\n",
    "3. frames is necessary to infer acceleration. Why? \n",
    "   - Every frame we are received with, provides the derivative of position w.r.t time.\n",
    "4. and so on..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "13d85ae6",
   "metadata": {
    "id": "a36d9d52"
   },
   "outputs": [],
   "source": [
    "# Also, Please Run this in the Local environment, where gym takes in playable action rather than complete 18 Action as in COLAB.\n",
    "# Then Rerun, agent=Agent(env)\n",
    "env.reset()\n",
    "image1, image2, image3, image4 = preprocess(env.step(1)[0]), preprocess(env.step(2)[0]), preprocess(env.step(0)[0]), preprocess(env.step(0)[0])\n",
    "cv2.imshow(\"image1\", image1)\n",
    "cv2.imshow(\"image2\", image2)\n",
    "cv2.imshow(\"image3\", image3)\n",
    "cv2.imshow(\"image4\", image4)\n",
    "cv2.waitKey(0)\n",
    "cv2.destroyAllWindows()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25db0382",
   "metadata": {
    "id": "25db0382"
   },
   "source": [
    "4 windows pop up showcasing how the images look like. **Warning: Hit Space or any button to Resume. Else, your IPython Kernel may die/crash.**\n",
    "\n",
    "You may see an image of below kind.\n",
    "![Image Observation by the Model](./Designs/ImageObservation.png)\n",
    "\n",
    "By executing below you are letting make it's first prediction. As said before the model takes STACK_SIZE=4 images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "68e36975",
   "metadata": {
    "id": "ca538aa4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.27388984, 0.        , 0.14055967, 0.07903521]], dtype=float32)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Also, Please Run this in the Local environment, where gym takes in playable action rather than complete 18 Action as in COLAB.\n",
    "# Make Sure that HYPERPARAMS[\"COLAB_COMPATIBLE_MODEL\"]=False for this\n",
    "# Then Rerun, agent=Agent(env)\n",
    "env.reset()\n",
    "image1, image2, image3, image4 = preprocess(env.step(1)[0]), preprocess(env.step(2)[0]), preprocess(env.step(0)[0]), preprocess(env.step(0)[0])\n",
    "agent.model.predict([\n",
    "    np.expand_dims(\n",
    "        np.stack([\n",
    "            image1, image2, image3, image4\n",
    "        ], axis=2), \n",
    "        axis=0\n",
    "    )\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "052fef5b",
   "metadata": {},
   "source": [
    "The output of predict function is the Q-values for each action that the emulater/game currently being played is supporting.\n",
    "Also the index with maximum Qvalue of all the action is considered as the current move being made.\\\n",
    "**action = (argmax(Q-values))**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb7eca92",
   "metadata": {
    "id": "eb7eca92"
   },
   "source": [
    "### Q-Learning\n",
    "The first four floats, shown above, are the Q-values for each action move that can be played in the game. Also, the Q-values are for the action for the current state passed in the model. These values give the estimation of what the chances are to get rewards in the future by using current_state and reward receive on current_state by playing an action.\n",
    "$$ Q_{current\\_state}(s,a) = r + \\gamma \\cdot \\max _{a'} Q_{next\\_state}(s',a') $$\n",
    "\n",
    "with and added exception of Q(s',a') must only be consider if current state s and action a are progressing to valid next state without game getting over."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "779a5b44",
   "metadata": {},
   "source": [
    "### EPSILON-DECAY GREEDY POLICY\n",
    "During Game play the agent was to focused on winning the game that it became greedy in finding the best move from what it has seen before. This was a counter-productive approach as the agent was not able to find new ways to achieve same score with better run.\n",
    "\n",
    "Due to this there we are required to have some randomness in movement of the agent. This procedure is known as Epsilon Greedy Policy usage. The Policy says to use the model to perform movement only when a random value is less that a value epsilon (eps), else we use the model to predict the value.\n",
    "\n",
    "- if epsilon >= RANDOM_VALUE:\n",
    "  - use random action\n",
    "- else\n",
    "  - then, use action supplied by the model\n",
    " \n",
    "This introduces another problem of epsilon being fixed. We want the randomness in program to decrease as we progress the game. For this we can use either FRAME_COUNT or NUMBER_OF_EPISODES_SEEN. We have used FRAME_COUNT that shows us that, as number of frames that agent observes increases over time, the randomness in action decreses (EXPLOITS). Therefore, as FRAME_COUNT increseses, epsilon value decreases (decays) over time\n",
    "\n",
    "![Epsilon-Decay Greedy Policy](./Designs/EpsilonGreedy.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89cb94bd",
   "metadata": {
    "id": "89cb94bd"
   },
   "source": [
    "### Model Checkpointing\n",
    "During the training process we found that it is really inefficient to produce the complete training model of 30hrs in a single run. It's better to work checkpoint of 5-10 hrs and save the progress in middle. For this we have devised the load_model and save_model functionality within our agent class.\n",
    "\n",
    "Use below function to check the trained model performance after training."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ab6b667",
   "metadata": {
    "id": "6ab6b667"
   },
   "source": [
    "**Convert below line to Code block by pressing 'Y', if you already have a saved model. Make sure that the model file exist.**"
   ]
  },
  {
   "cell_type": "raw",
   "id": "c072f494",
   "metadata": {
    "id": "c072f494"
   },
   "source": [
    "agent.load_model('./models/model_dqn_breakout.h5') # \"models/\" + HYPERPARAMS[\"SAVED_MODEL_NAME\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3c37470",
   "metadata": {
    "id": "a3c37470"
   },
   "source": [
    "**OR** run Training blocks to run initial exploration stage, for our agent to get an understanding of the game, and then the Training. Exploration is not exactly understanding, but is a way for us to fill the replay memory with images of the game being played with random actions.\n",
    "\n",
    "### Game Play"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e7c21e1a",
   "metadata": {
    "id": "e7c21e1a"
   },
   "outputs": [],
   "source": [
    "EXPLORE = 1\n",
    "TRAIN   = 2\n",
    "TEST    = 3\n",
    "\n",
    "def slow_start(env,image_stack=[], rgb_stack=[],NOOPMAX=10):\n",
    "    idle_times = random.randint(4, NOOPMAX)\n",
    "    \n",
    "    #print(f'Agent is Staying Idle for {idle_times} times. Agent is thinking about what move to make...')\n",
    "    for idle_time in range(idle_times):\n",
    "        if HYPERPARAMS[\"ON_COLAB\"]==False: env.render() # NOTE: Comment this in Google Colab\n",
    "        state, reward, done, info = env.step(0) # Zero means: NOOP or No Operation\n",
    "        processed_frame = preprocess(state)\n",
    "        image_stack.append(processed_frame)\n",
    "        rgb_stack.append(preprocess_rgb(state))\n",
    "        \n",
    "    return image_stack, rgb_stack\n",
    "\n",
    "#agent.epsilon = 0.5  # Agent is Exploring the game by default\n",
    "def gameplay(PLAY_TYPE=TEST,MAX_EPISODE_PLAYTIME=1000000):\n",
    "    global FRAME_COUNT\n",
    "    #print(f'Agent is starting a new game: {e} games played.')\n",
    "    # Reset Game    \n",
    "    state = env.reset()\n",
    "    times_rewarded = times_penalized = 0\n",
    "    last_lives = 5\n",
    "    terminal_life_lost = False # False if last_lives==0 else True\n",
    "    \n",
    "    #print('Agent has made the start move.')\n",
    "    # Start the game by 'FIRE' action, incase if it doesnot start the game without it\n",
    "    state, _, _, _ = env.step(1) \n",
    "    \n",
    "    # Fill agent's memory with random times of no operation played\n",
    "    image_stack,rgb_stack = slow_start(env=env, NOOPMAX=HYPERPARAMS[\"NOOPMAX\"])\n",
    "    \n",
    "    \n",
    "    #print(f'Agent is now playing the game...')\n",
    "    i, state = 0, np.stack(image_stack[-4:], axis = 2)\n",
    "    while i < MAX_EPISODE_PLAYTIME:\n",
    "        \n",
    "        # If agent has lost a life then start the game with 'FIRE' again.\n",
    "        if(terminal_life_lost == True):\n",
    "            state, _, _, _ = env.step(1) # 'FIRE' to start the game\n",
    "            slow_start(env, image_stack, rgb_stack, HYPERPARAMS[\"NOOPMAX\"])\n",
    "            state = np.stack(image_stack[-4:], axis = 2)\n",
    "\n",
    "        FRAME_COUNT = FRAME_COUNT + 1\n",
    "        action = env.action_space.sample() if PLAY_TYPE==EXPLORE else (agent.next_action(state) if PLAY_TYPE==TRAIN else np.argmax(agent.model.predict([np.expand_dims(state,axis=0)])[0]))\n",
    "\n",
    "        if HYPERPARAMS[\"ON_COLAB\"]==False: env.render() # NOTE: Comment this in Google Colab\n",
    "        # Agent Makes random moves here...\n",
    "        action = action if action < env.action_space.n else env.action_space.sample()\n",
    "        next_state, reward, done, info = env.step(action)\n",
    "        last_lives = info[HYPERPARAMS[\"LIVES\"]]\n",
    "        #rgb_stack.append(preprocess_rgb(next_state))\n",
    "      \n",
    "\n",
    "        # Agent updates it's game status here...\n",
    "        terminal_life_lost = True if info[HYPERPARAMS[\"LIVES\"]] < last_lives else False\n",
    "        #print(reward, terminal_life_lost, done, info)  \n",
    "            \n",
    "        if reward > 0:      \n",
    "            times_rewarded = times_rewarded + 1\n",
    "            #reward = 100\n",
    "        elif reward < 0 or done or terminal_life_lost: \n",
    "            times_penalized = times_penalized + 1\n",
    "            #reward = -10\n",
    "        #elif terminal_life_lost: times_penalized = times_penalized + 1\n",
    "        #reward = 10 if reward > 0 else (-30 if reward < 0 else reward)\n",
    "        #reward = -30 if terminal_life_lost else reward\n",
    "        \n",
    "        # Store the stack of images for new a experience\n",
    "        processed_frame = preprocess(next_state)\n",
    "        image_stack = image_stack[-3:]\n",
    "        image_stack.append(processed_frame)\n",
    "        \n",
    "        next_state = np.stack(image_stack, axis = 2)\n",
    "        if(len(image_stack) != 4): print(\"Something's not right!! The stack size is less than expected.\")\n",
    "            \n",
    "        #Store experience in replay mem\n",
    "        if(PLAY_TYPE==EXPLORE or PLAY_TYPE==TRAIN): \n",
    "            agent.store_experience(state, action, reward, next_state, terminal_life_lost)\n",
    "        elif PLAY_TYPE==TEST:\n",
    "            next_state = np.stack([image_stack[-2], image_stack[-2], image_stack[-1], image_stack[-1]], axis=2)\n",
    "            \n",
    "        state = next_state\n",
    "        \n",
    "        if done: break\n",
    "        i+=1\n",
    "        \n",
    "    REWARD_HISTORY.append(times_rewarded)\n",
    "    return image_stack, times_rewarded, times_penalized, rgb_stack"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22f1bef6",
   "metadata": {
    "id": "22f1bef6"
   },
   "source": [
    "### TRAINING STAGE\n",
    "This step is recommended to be executed on Google Colab or over machine with GPU.\n",
    "\n",
    "**CONVERT THE BLOCKS to Code block. BELOW for TRAINING.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65e84041",
   "metadata": {
    "id": "65e84041"
   },
   "source": [
    "#### Initial Exploration stage\n",
    "This process takes around 3 min 28 seconds for Breakout on COLAB"
   ]
  },
  {
   "cell_type": "raw",
   "id": "948b1c95",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8c71d5e3",
    "outputId": "9bcf7de9-6c7d-441e-d76e-f2ae2b20770b"
   },
   "source": [
    "\n",
    "'''\n",
    "    THIS IS DONE TO POPULATE REPLAY MEMORY WITH COMPLETE EXPLORATION TO INITIALIZE THE MEMORY \n",
    "'''\n",
    "total_times_rewarded=total_times_penalized=0\n",
    "for e in range(HYPERPARAMS[\"NUM_EXPLORE\"]):\n",
    "  image_stack, times_rewarded, times_penalized, rgb_stack = gameplay(PLAY_TYPE=EXPLORE, MAX_EPISODE_PLAYTIME=1000)\n",
    "  total_times_rewarded  = total_times_rewarded + times_rewarded\n",
    "  total_times_penalized = total_times_penalized+ times_penalized\n",
    "  if(e % 100 == 0): \n",
    "      print(\"Finished exploring for {} episodes\".format(e))\n",
    "      print(\"Total Times Rewarded: {}, Total Times Penalized: {}\".format(total_times_rewarded, total_times_penalized))\n",
    "\n",
    "print(\"EXPLORATION STEP COMPLETED\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c4b1e7c",
   "metadata": {
    "id": "9c4b1e7c"
   },
   "source": [
    "#### Train using DQN\n",
    "Every 100 episodes takes around 10-20 min for Breakout, depending on the Hyperparameters."
   ]
  },
  {
   "cell_type": "raw",
   "id": "973abfef",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "kphrsAijn-fx",
    "outputId": "e3398351-ea2d-4deb-b8d3-90af2cbefcf8"
   },
   "source": [
    "# Model Checkpointing: Use it when you are continuing the training process mid-way\n",
    "FRAME_COUNT=35000\n",
    "agent.epsilon=0.5\n",
    "agent.load_model('./models/tmp_model_breakout_4000.h5')"
   ]
  },
  {
   "cell_type": "raw",
   "id": "be488686",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "1db8d6ad",
    "outputId": "178a2f14-5faa-42a8-d9b4-ba1dfbaa2a35"
   },
   "source": [
    "from IPython.display import clear_output\n",
    "%matplotlib inline\n",
    "fig = plt.figure(figsize=(20,5))\n",
    "fig,ax = plt.subplots(1,1)\n",
    "ax.set_xlabel('Episodes')\n",
    "ax.set_ylabel('LOSS')\n",
    "\n",
    "'''\n",
    "TRAIN DQN ON THE FIRST GAME\n",
    "'''\n",
    "\n",
    "from datetime import datetime\n",
    "start_time = datetime.now()\n",
    "\n",
    "rew_list = []\n",
    "e=0\n",
    "for e in range(1, HYPERPARAMS[\"NUM_EPISODES\"]+1):\n",
    "    print(\"Agent is ready for gameplay...\")\n",
    "    image_stack, times_rewarded, times_penalized, rgb_stack = gameplay(PLAY_TYPE=TRAIN, MAX_EPISODE_PLAYTIME=1000)\n",
    "            \n",
    "    rew_list.append(times_rewarded)\n",
    "            \n",
    "    if(e % HYPERPARAMS[\"AUTOSAVE_CHECKPOINT\"] == 0):\n",
    "        clear_output(wait=True)\n",
    "        print(\"Finished episode \", e , \"/\", HYPERPARAMS[\"NUM_EPISODES\"], \" Total reward = \", sum(rew_list)/len(rew_list), \" eps = \", agent.epsilon)\n",
    "        if HYPERPARAMS[\"ON_COLAB\"]:\n",
    "            if not os.path.exists(HYPERPARAMS[\"VIS_DIR\"]):\n",
    "                os.mkdir(HYPERPARAMS[\"VIS_DIR\"])\n",
    "            generate_gif(len(rgb_stack), rgb_stack, sum(rew_list)/(len(rew_list)), HYPERPARAMS[\"VIS_DIR\"] + \"/\", e)\n",
    "        \n",
    "        rew_list = []\n",
    "        print(f\"Saving Model Checkpoint at episode {e}...\")\n",
    "        agent.save_model(\"models/tmp_model_breakout_\" + str(e) + \".h5\")        \n",
    "        ax.set_xlim(0,e)\n",
    "        ax.set_ylim(0,max(LOSS_HISTORY))\n",
    "        plt.plot(np.arange(len(LOSS_HISTORY)),LOSS_HISTORY)\n",
    "        plt.show()\n",
    "\n",
    "    # BACKPROP INITIATED AT THE END OF EVERY EPISODE AND NOT AT TGT_FREQ\n",
    "    agent.replay(HYPERPARAMS[\"MINIBATCH_SIZE\"], model_swap)\n",
    "    model_swap = model_swap + 1\n",
    "\n",
    "print(\"Training Completes\")\n",
    "end_time = datetime.now()\n",
    "print('Duration: {}'.format(end_time - start_time))\n",
    "\n",
    "print(f\"Agent is now prepared now, after training for {e} episodes.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "031e1414",
   "metadata": {
    "id": "031e1414"
   },
   "source": [
    "#### Saving the final model"
   ]
  },
  {
   "cell_type": "raw",
   "id": "56969669",
   "metadata": {
    "id": "HgFqqT4vs73g"
   },
   "source": [
    "agent.save_model(\"models/\" + HYPERPARAMS[\"SAVED_MODEL_NAME\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4512222e",
   "metadata": {
    "id": "4512222e"
   },
   "source": [
    "## Testing the model\n",
    "If you open checkpoint \"model_dqn_breakout_1.h5\" we can see that model is able to move towards the ball direction, but still has to keep up to get the score. This is some progress from random movement from random sample of env.action_space.n. We have achieved this model about 18hrs of training on GOOGLE COLAB in the background."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5db55da3",
   "metadata": {},
   "outputs": [],
   "source": [
    "env = gym.make(HYPERPARAMS[\"ENV_NAME\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ff2c1e67",
   "metadata": {
    "id": "ff2c1e67",
    "outputId": "9afd6bae-9881-4090-a7ff-c8617045bcf3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setting up the agent ...\n",
      "Agent has been Sucessfully setup ...\n"
     ]
    }
   ],
   "source": [
    "agent = Agent(env, model_pathname=\"models/model_dqn_breakout_1.h5\" ) # model_pathname=(\"models/\" + HYPERPARAMS[\"SAVED_MODEL_NAME\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "3c627446",
   "metadata": {
    "id": "3c627446",
    "outputId": "b6f43cab-b5b4-45c1-b198-0f666382f3d7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_2\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " image (InputLayer)          [(None, 84, 84, 4)]       0         \n",
      "                                                                 \n",
      " conv2D_1 (Conv2D)           (None, 20, 20, 32)        8192      \n",
      "                                                                 \n",
      " conv2D_2 (Conv2D)           (None, 9, 9, 64)          32768     \n",
      "                                                                 \n",
      " conv2D_3 (Conv2D)           (None, 7, 7, 64)          36864     \n",
      "                                                                 \n",
      " conv2D_4 (Conv2D)           (None, 1, 1, 1024)        3211264   \n",
      "                                                                 \n",
      " flat_1 (Flatten)            (None, 1024)              0         \n",
      "                                                                 \n",
      " q_values (Dense)            (None, 4)                 4100      \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 3,293,188\n",
      "Trainable params: 3,293,188\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "agent.model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "5d2a848f",
   "metadata": {
    "id": "aa84b109",
    "outputId": "c5b4b580-39e5-400a-c20d-2877fb1204f9"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\rahul\\miniconda3\\envs\\cs677\\lib\\site-packages\\pyglet\\image\\codecs\\wic.py:289: UserWarning: [WinError -2147417850] Cannot change thread mode after it is set\n",
      "  warnings.warn(str(err))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished episode  0 / 20  Total reward =  30\n",
      "Finished episode  1 / 20  Total reward =  30\n",
      "Finished episode  2 / 20  Total reward =  30\n",
      "Finished episode  3 / 20  Total reward =  0\n",
      "Finished episode  4 / 20  Total reward =  30\n",
      "Finished episode  5 / 20  Total reward =  30\n",
      "Finished episode  6 / 20  Total reward =  30\n",
      "Finished episode  7 / 20  Total reward =  30\n",
      "Finished episode  8 / 20  Total reward =  30\n",
      "Finished episode  9 / 20  Total reward =  30\n",
      "Finished episode  10 / 20  Total reward =  30\n",
      "Finished episode  11 / 20  Total reward =  30\n",
      "Finished episode  12 / 20  Total reward =  30\n",
      "Finished episode  13 / 20  Total reward =  30\n",
      "Finished episode  14 / 20  Total reward =  30\n",
      "Finished episode  15 / 20  Total reward =  0\n",
      "Finished episode  16 / 20  Total reward =  30\n",
      "Finished episode  17 / 20  Total reward =  30\n",
      "Finished episode  18 / 20  Total reward =  30\n",
      "Finished episode  19 / 20  Total reward =  0\n",
      "Testing Completes\n",
      "Duration: 0:03:26.076870\n"
     ]
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "start_time = datetime.now()\n",
    "\n",
    "rew_list = []\n",
    "total_times_rewarded=total_times_penalized=0\n",
    "for e in range(HYPERPARAMS[\"NUM_EVAL\"]):\n",
    "    image_stack, times_rewarded, times_penalized, rgb_stack = gameplay(PLAY_TYPE=TEST, MAX_EPISODE_PLAYTIME=1000)\n",
    "    \n",
    "    total_times_rewarded  = total_times_rewarded + times_rewarded\n",
    "    total_times_penalized = total_times_penalized+ times_penalized\n",
    "    rew_list.append(times_rewarded)\n",
    "    \n",
    "    print(\"Finished episode \", e , \"/\", HYPERPARAMS[\"NUM_EVAL\"], \" Total reward = \", times_rewarded*(10))\n",
    "    if HYPERPARAMS[\"ON_COLAB\"]:\n",
    "        if not os.path.exists(\"test\"):\n",
    "            os.mkdir(\"test\")\n",
    "        generate_gif(len(rgb_stack), rgb_stack, total_times_rewarded, \"test/\", e)\n",
    "\n",
    "print(\"Testing Completes\")\n",
    "end_time = datetime.now()\n",
    "print('Duration: {}'.format(end_time - start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "e0ab81bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()\n",
    "exit(1) # Close Gym Windows"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ecc8b9e",
   "metadata": {
    "id": "1ecc8b9e"
   },
   "source": [
    "## Transfer Learning: PART II"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "327165af",
   "metadata": {},
   "source": [
    "**MODEL FOR**\n",
    "![BREAKOUT](./Designs/Breakout.jpeg)\n",
    "**IS TRANSFERED TO**\n",
    "![Space Invader](./Designs/Spaceshooter.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2853d2a2",
   "metadata": {
    "id": "2853d2a2",
    "outputId": "a3a90795-feb1-445e-d0a5-b10b0b38ac89"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setting up the agent ...\n",
      "Agent has been Sucessfully setup ...\n"
     ]
    }
   ],
   "source": [
    "prev_env = gym.make('BreakoutDeterministic-v4')\n",
    "env = gym.make('SpaceInvaders-v0')\n",
    "\n",
    "agent = Agent(prev_env) # Using Previous model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "e74e6426",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['NOOP', 'FIRE', 'RIGHT', 'LEFT', 'RIGHTFIRE', 'LEFTFIRE']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.unwrapped.get_action_meanings()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9b1a44bd",
   "metadata": {
    "id": "9b1a44bd"
   },
   "outputs": [],
   "source": [
    "# Transfer Learning condition: Requires (prev_env.action_space.n <= env.action_space.n)\n",
    "# We are assuming the next game to be played is higher complex than old game we have trained.\n",
    "agent.transfer_learning(\n",
    "    env,\n",
    "    'models/model_dqn_breakout_1.h5', # HYPERPARAMS[\"TRANSFER_MODEL_NAME\"],\n",
    "    agent.action_size, # Number of Old actions=4\n",
    "    (env.action_space.n-agent.action_size) # Number of New actions, Excluding old one's = 2\n",
    ") "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b2459ab",
   "metadata": {
    "id": "2b2459ab"
   },
   "source": [
    "### Training (On new Game using model of old game)\n",
    "\n",
    "#### Transfer Learning - Load Breakout model\n",
    "![Transfer Learning - Load Breakout model](./Designs/TransferLearningPart1.png)\n",
    "\n",
    "#### Transfer Learning - Extend Breakout model to Play SpaceInvader\n",
    "![Transfer Learning - Extend Breakout model to Play SpaceInvader](./Designs/TransferLearningPart2.png)"
   ]
  },
  {
   "cell_type": "raw",
   "id": "2bd9a948",
   "metadata": {
    "id": "1e7a79a6",
    "outputId": "7ad2f44b-4c14-4d7e-d628-c2e45c833f26"
   },
   "source": [
    "from IPython.display import clear_output\n",
    "%matplotlib inline\n",
    "fig = plt.figure(figsize=(20,5))\n",
    "fig,ax = plt.subplots(1,1)\n",
    "ax.set_xlabel('Episodes')\n",
    "ax.set_ylabel('LOSS')\n",
    "\n",
    "\n",
    "from datetime import datetime\n",
    "start_time = datetime.now()\n",
    "'''\n",
    "TRAIN DQN ON THE SECOND GAME\n",
    "'''\n",
    "rew_list = []\n",
    "e=0\n",
    "for e in range(1, HYPERPARAMS[\"NUM_EPISODES\"]+1):\n",
    "    print(\"Agent is ready for gameplay...\")\n",
    "    image_stack, times_rewarded, times_penalized, rgb_stack = gameplay(PLAY_TYPE=TRAIN, MAX_EPISODE_PLAYTIME=1000)\n",
    "            \n",
    "    rew_list.append(times_rewarded)\n",
    "            \n",
    "    if(e % HYPERPARAMS[\"AUTOSAVE_CHECKPOINT\"] == 0):\n",
    "        print(\"Finished episode \", e , \"/\", HYPERPARAMS[\"NUM_EPISODES\"], \" Total reward = \", sum(rew_list)/len(rew_list), \" eps = \", agent.epsilon)\n",
    "        if HYPERPARAMS[\"ON_COLAB\"]:\n",
    "            if not os.path.exists(HYPERPARAMS[\"VIS_DIR\"]):\n",
    "                os.mkdir(HYPERPARAMS[\"VIS_DIR\"])\n",
    "            generate_gif(len(rgb_stack), rgb_stack, sum(rew_list)/(len(rew_list)), HYPERPARAMS[\"VIS_DIR\"] + \"/\", e)\n",
    "        \n",
    "        rew_list = []\n",
    "        print(f\"Saving Model Checkpoint at episode {e}...\")\n",
    "        agent.save_model(\"models/tmp_model_spaceinvader_\" + str(e) + \".h5\")\n",
    "        ax.set_xlim(0,e)\n",
    "        ax.set_ylim(0,max(LOSS_HISTORY))\n",
    "        plt.plot(LOSS_HISTORY)\n",
    "        plt.show()\n",
    "\n",
    "    # BACKPROP INITIATED AT THE END OF EVERY EPISODE AND NOT AT TGT_FREQ\n",
    "    agent.replay(HYPERPARAMS[\"MINIBATCH_SIZE\"], model_swap)\n",
    "    model_swap = model_swap + 1\n",
    "    \n",
    "print(f\"Agent is now prepared now, after training for {e} episodes.\")\n",
    "\n",
    "print(\"Transfer Learning's Training Completes\")\n",
    "end_time = datetime.now()\n",
    "print('Duration: {}'.format(end_time - start_time))"
   ]
  },
  {
   "cell_type": "raw",
   "id": "06050117",
   "metadata": {
    "id": "6bef9558"
   },
   "source": [
    "agent.save_model('./models/model_dqn_spaceinvader.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8cba1e3",
   "metadata": {
    "id": "a8cba1e3"
   },
   "source": [
    "If you have conducted any training the Loss History can be showcased w.r.t episodes trained using below code."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cee3a180",
   "metadata": {
    "id": "cee3a180"
   },
   "source": [
    "### Testing Transfer learning of Breakout model on Spaceshooter game\n",
    "\n",
    "Initially if you run this with the 'model_dqn_breakout_1.h5' you can see that the agent forgot how to FIRE in SpaceInvader. Because, the fire action is only used once throught Breakout, at Start. Also we can see that the agent is moving close to the bullet at time, as it is assuming bullets of SpaceInvader is the ball it was trained to save from dropping in Breakout. Now all we have to do is Train the model of SpaceInvader from here by using above code block for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "10eb97ae",
   "metadata": {
    "id": "e3513dd1",
    "outputId": "d10ae16f-dedc-4b67-83f2-68a2bc693320"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished episode  0 / 20  Total reward =  0\n",
      "Finished episode  1 / 20  Total reward =  10\n",
      "Finished episode  2 / 20  Total reward =  0\n",
      "Finished episode  3 / 20  Total reward =  30\n",
      "Finished episode  4 / 20  Total reward =  0\n",
      "Finished episode  5 / 20  Total reward =  0\n",
      "Finished episode  6 / 20  Total reward =  0\n",
      "Finished episode  7 / 20  Total reward =  0\n",
      "Finished episode  8 / 20  Total reward =  0\n",
      "Finished episode  9 / 20  Total reward =  0\n",
      "Finished episode  10 / 20  Total reward =  0\n",
      "Finished episode  11 / 20  Total reward =  10\n",
      "Finished episode  12 / 20  Total reward =  10\n",
      "Finished episode  13 / 20  Total reward =  0\n",
      "Finished episode  14 / 20  Total reward =  0\n",
      "Finished episode  15 / 20  Total reward =  0\n",
      "Finished episode  16 / 20  Total reward =  0\n",
      "Finished episode  17 / 20  Total reward =  0\n",
      "Finished episode  18 / 20  Total reward =  0\n",
      "Finished episode  19 / 20  Total reward =  0\n",
      "Transfer Learning's Testing Completes\n",
      "Duration: 0:27:24.041639\n"
     ]
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "start_time = datetime.now()\n",
    "\n",
    "rew_list = []\n",
    "total_times_rewarded=total_times_penalized=0\n",
    "for e in range(HYPERPARAMS[\"NUM_EVAL\"]):\n",
    "    image_stack, times_rewarded, times_penalized, rgb_stack = gameplay(PLAY_TYPE=TEST, MAX_EPISODE_PLAYTIME=1000)\n",
    "    \n",
    "    total_times_rewarded  = total_times_rewarded + times_rewarded\n",
    "    total_times_penalized = total_times_penalized+ times_penalized\n",
    "    rew_list.append(times_rewarded)\n",
    "    \n",
    "    print(\"Finished episode \", e , \"/\", HYPERPARAMS[\"NUM_EVAL\"], \" Total reward = \", times_rewarded*(10))\n",
    "    if HYPERPARAMS[\"ON_COLAB\"]:\n",
    "        if not os.path.exists(\"test\"):\n",
    "            os.mkdir(\"test\")\n",
    "        generate_gif(len(rgb_stack), rgb_stack, total_times_rewarded, \"test/\", e)\n",
    "\n",
    "print(\"Transfer Learning's Testing Completes\")\n",
    "end_time = datetime.now()\n",
    "print('Duration: {}'.format(end_time - start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "53abaf4e",
   "metadata": {
    "id": "53abaf4e"
   },
   "outputs": [],
   "source": [
    "env.close()\n",
    "exit(1) # Close Gym Windows"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d195599",
   "metadata": {
    "id": "6eb40f02"
   },
   "source": [
    "### Training phase - Reward History and Loss History (Breakout)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d849e92",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.figure(figsize=(20,5))\n",
    "plt.plot(REWARD_HISTORY)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c51fc758",
   "metadata": {},
   "source": [
    "For reward History we found that the reward was low at the starting and increased till a point of time, as show below.\n",
    "![REWARD_HISTORY](./Designs/REWARD_HISTORY.png)\n",
    "\n",
    "**X-axis: episodes**\\\n",
    "**Y-axis: rewards**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10971ed2",
   "metadata": {},
   "source": [
    "We have also found that the loss steadly grow up more as the agent played the game and decreased after episodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c62094c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20,5))\n",
    "plt.plot(LOSS_HISTORY)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "343ae27a",
   "metadata": {},
   "source": [
    "![LOSS_HISTORY](./Designs/LOSS_HISTORY.png)\n",
    "**X-axis: episodes**\\\n",
    "**Y-axis: loss**"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "background_execution": "on",
   "collapsed_sections": [
    "8973e18a",
    "eb7eca92",
    "2b2459ab",
    "cee3a180"
   ],
   "machine_shape": "hm",
   "name": "CS673-FinalProject.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}