QLearning_SARSA_FrozenLake.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "auburn-pittsburgh",
      "metadata": {
        "id": "auburn-pittsburgh"
      },
      "source": [
        "# Lab 1 - FrozenLake MDP: Part 2 \n",
        "# Assignment\n",
        "\n",
        "- In this assignment you will learn how to tackle problems with limited state spaces.\n",
        "- In particular we consider the FrozenLake MDP problem.\n",
        "\n",
        "# Outline\n",
        "\n",
        "- Part 0 introduces us to [gym](https://gymnasium.farama.org/), an environment that allows us to test our reinforcement learning algorithms in various problems\n",
        "- In Part 1, you implemented a policy iteration algorithm (in previous homework HW1)\n",
        "- In Part 2, you will implement Q-Learning and SARSA (This homework HW2) \n",
        "\n",
        "# Deliverable\n",
        "\n",
        "Regarding the Lab:\n",
        "\n",
        "- Make sure your code runs from top to bottom without any errors.\n",
        "- Your submitted Notebook must contain saved outputs."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "medium-dancing",
      "metadata": {
        "id": "medium-dancing"
      },
      "source": [
        "# Recall: FrozenLake-v1 environment in Gym\n",
        "- We look [FrozenLake-v1 environment](https://gymnasium.farama.org/environments/toy_text/frozen_lake/) in gym. \n",
        "- You don't need to write any code for this part\n",
        "- you should still understand the code to help you solve Part 2\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "necessary-character",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "necessary-character",
        "outputId": "987da8b7-c2b4-4f57-f593-67e9f2caa389"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting gymnasium\n",
            "  Downloading gymnasium-0.27.1-py3-none-any.whl (883 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m883.7/883.7 KB\u001b[0m \u001b[31m11.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.8/dist-packages (from gymnasium) (1.22.4)\n",
            "Requirement already satisfied: importlib-metadata>=4.8.0 in /usr/local/lib/python3.8/dist-packages (from gymnasium) (6.0.0)\n",
            "Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from gymnasium) (2.2.1)\n",
            "Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.8/dist-packages (from gymnasium) (4.5.0)\n",
            "Collecting gymnasium-notices>=0.0.1\n",
            "  Downloading gymnasium_notices-0.0.1-py3-none-any.whl (2.8 kB)\n",
            "Collecting jax-jumpy>=0.2.0\n",
            "  Downloading jax_jumpy-0.2.0-py3-none-any.whl (11 kB)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata>=4.8.0->gymnasium) (3.14.0)\n",
            "Installing collected packages: gymnasium-notices, jax-jumpy, gymnasium\n",
            "Successfully installed gymnasium-0.27.1 gymnasium-notices-0.0.1 jax-jumpy-0.2.0\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting pygame\n",
            "  Downloading pygame-2.1.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.7/13.7 MB\u001b[0m \u001b[31m17.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: pygame\n",
            "Successfully installed pygame-2.1.3\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "# You will need numpy and gym. You can try running the following lines to install them\n",
        "# The assignment is tested on Python3.8 so in case you are having installation issues you might \n",
        "# want to try installing that version. \n",
        "\n",
        "# !{os.sys.executable} -m pip install numpy\n",
        "!{os.sys.executable} -m pip install gymnasium\n",
        "# !{os.sys.executable} -m pip install Pillow\n",
        "# !{os.sys.executable} -m pip install ipython\n",
        "!{os.sys.executable} -m pip install pygame\n",
        "\n",
        "import gymnasium as gym\n",
        "import numpy as np\n",
        "from PIL import Image\n",
        "from IPython.display import display"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "considerable-swing",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "considerable-swing",
        "outputId": "6a01727e-98e5-4a96-e8bf-1bfece5eaede"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of States 16, Number of Actions 4\n",
            "Reward range (0, 1)\n"
          ]
        }
      ],
      "source": [
        "# Import the environment we will use in this assignment\n",
        "env=gym.make('FrozenLake-v1',is_slippery=True, render_mode=\"rgb_array\")\n",
        "\n",
        "# Note: If you have installed the newest gym version, FrozenLake-v0 might not be available. In this case, instead of env=gym.make('FrozenLake-v0') you can use:\n",
        "# env=gym.make('FrozenLake-v1',is_slippery=True)\n",
        "\n",
        "# Show the model\n",
        "print(f\"Number of States {env.observation_space.n}, Number of Actions {env.action_space.n}\")\n",
        "print(f\"Reward range {env.reward_range}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "71L4uVanjIaZ",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 273
        },
        "id": "71L4uVanjIaZ",
        "outputId": "aa6269b1-3b29-4d6c-d357-7e0f13803fec"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<PIL.Image.Image image mode=RGB size=256x256 at 0x7F4C498484C0>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQAAAAEACAIAAADTED8xAAAeXklEQVR4nO2df4xd1XHHv0bPD7Nm8drYGFeVwdRKKMVuiymoBtEFFP9BhJAMUdUWlSpt3UaRatIIGlUCoSJFTfIHOFVp66BKliBV1YBEMFaBVFmB6xpau6kNrhOtYoEq7wYbsDFdvOut6R/z7ntz79yZO/e9t7985vOHtXvuOefOXvnMzJkz55xFu/cfRxCkSgPAlWvX2JWuWjbV/nnlQLPvQpycmCqUvHPa9Zbxd8cQ8vdMyvI3AFy1bMpukH86Bd+fwcWy6/On1Ip/sso/JuQvJeQve3uRiyrlCIILmEbdBtl4Kh+dNP6oRI5aaao08m3VVuPO7hghv4d05A8LECRNywLU8vnadaiVHJcHxop9EjQu+SgnZA+a/6fpgJA/5O9O/rAAQdI0wIaInEET2pjm5byVDHvZM3rb25N+YWEch/whf9fyhwUIkiYXBSod3ygb03Z9Xi7HIq+jYY9pz3tD/pDfI39YgCBpStYBtLFlz7XtcjnWtToaHs/PLg/5Q35ZEhYgSJoG2GCqO4Zs7r1xuP3zc/8xAt3nI/weXqH+DMmvSRXyF+ovaPnDAgRJ01oHoNEgV+C0KKxcjSOo/I9u2wJg+dc65WQN/u61VwqtqH5ZnLh6NLfL+y4/L9cyVUL+C0P+sABB0rSiQNk6Gf1WHoXlaKNK6n6CSugpzQfy/XTeQqO5bgZ5v+TvNhIS8i9U+cMCBEnTsgB+30uLv5KX/7O7hgAsx6nSl9FTbT6gITPIJb3LXxfNi5WE/PNZ/rAAQdIs2r3/eHtT8yZzc7McuzS2vn73MDLt7mf1nlPI5gNaz5UUNmV3Jz9FLbrTQ3bPlYT8mGv5wwIESZPLBeJxXG3Grc3r//T7EwB+/YqPK1/5b+9dCuBZ9mbZM8ngHMdtupM/ixsAVTpMI+QnFqL8YQGCpCnJBvWPSPL+f2e6G++NWlFEiGcKETSau9NDvWgU0mGExzflsYiQn1hY8ocFCJJm0e79x2/ecDn9oo0VORZ5pifp8ju/Pg3g2Mh7AH7l7odL+/nRi98EsG74CgD/8ucNAM82On6blinEKXh7bxx+H0AX8vOe62o4bX8Tl1C+hT8N+eeP/GEBgqTJzQHssSXH06NfugfAL3/mcmTafeTo9QBGju4B8OBDd7VrPvmtPQCA6wEAbwHY+lcPA/iFn7wP4C/+5gXkV+y4V2fvA+pFft7Ko4e0mvnykL+c+Sl/WIAgaXLZoBJt1JK/TtmdxKNf+n0AD959OYB7v/L3AB5k9V8/Pg7guSe+COC/fnJzuw7vjecScjxrhHXll/hPs9FaZRGPkL/D/Jc/LECQNBWnQ9ujhzQ3QdaAZgWk6TmZ7u94/LxtL3kgNp4osobm0WpnVs4EIb+nvBf5wwIESdOyAHwFzr96J0feD194A8APu2pbd/cQp1/ye3zWuv5uyO+XZPblDwsQJE3J/QAHxpqoM45pVZh2/b6FcQCP75xClvVJUJboI9uaAJazVmX7gzvIfaXaaO5Ffk4v3mTITyws+cMCBEmTiwLxyCt5ddo4ljsyn179RQC3vvo8gEe2nQLAd2HuQRPA0UNDAPZ+biuAe9FZB5A9a9ijuRf5PXhahfzEQpE/LECQNLmT4STaOJajkLJ97sMSAL+5fwrAwINL2k8nnjwL4B+XLAHw5I/2tMu7281ZuFuqL/J78EQtQn7O/Jc/LECQNIt27z/++ZvXePKz7Vw/Wgnmu8N4rr8s5/EfQtMihjyUj94X+T3w7EJ/DDvkb9evL3KOmZA/LECQNLlsUHsGrY2kkVdfArD5Tx4C8Oy3v9Uup5IVl0wDePYbT7TLf+uRhwB874WXANx3z+fb5dpeHk++eC/yc+R9t/7Mk5B/IcofFiBImtYcgBfR6Km7U/PQ3hfaP3/wSQOZ7h9sliecct3vodTLpJPJ+iK/pnv6lSkZ8tvMlfxhAYKkMW6JrJ7Xczbeeg90j016h3VPCeb9tN8yrtesKz/XOvKv6CVLkT8N+TXmSv6wAEHSqDvC5IjxzPR5W4mtCegsMXnDlNazLUnIXyZhyF+UPyxAkDSte4Lt2KrMq5ZPeYnWj8cv9N/u1C4P+UP+ruUPCxAkTW4OoGVZeObXEtlb6Uwc1aO5WkNob/S0CvlTlj8sQJA06i2Rds6dZ33On7NhZ2tIr04S8ttyauUhf1iAIGlK5gCEZ/z5W/ULPprt9cKQfya48OQPCxAkjboS7Fmxs/F4e3Je788itGuG/CG/DdVctHv/8R4FDYKFS7hAQdI0wK6617ATjHqnu8MtkG3ICPl7JGX5GwCuWjZVlWfnza/QxPKvBfrX+dp0Lb8dvZ7/8tvMf/nnw/cPFyhImoobYiR2fgUfwXbE1ybfVm1VuiOJsGMC2Xn2TWR55Bm8pNM2X6f4Fknv8mvE95dvkfjlDwsQJE3J/QB1d3BKvULj294fLPUWR/P/NB0m5fdlhvSKdmdt7/J73hvfv/fvHxYgSJrW6dD0i5xBE57RzFvJsJc9o7e9VenXFsaxIb9H32zdMVJZR/L89uHS8rI3Vnje8f0r60j69f3DAgRJk4sClY5vVM217b2eUpfwOhqenEHtvbK+RGqdgRs2G/Ub09MAJiYAYHr0zdIeCKmZ7AxKTnx/jZn7/mEBgqQxToYrYscK7HKpM2z94ZHKri99wbpap1VniPppIotLTw3cVKizeHoKwOmpZuEtmjayT72M75+rM8PfPyxAkDStc4Hol+5279t4fEetprPngvz81nKCawWP1uFMnCpKO7ByQNQaADAEINNP5K3a2kiTXyO+f6uHvn7/sABB0uRuiZQriFoU2b/n3z4pgErs2G2lr1yQn+PXPRs3nwcwNnbOqJMxCeDEsYvLHzYaAIbWNpFpo48OdaIWXA9p8sf3r6Kf3z8sQJA0uTvCMueqPIrMqRvB0J5mJXYmiRUz4f3wnEFb96xaN1koGWPaa82axQC2rFwC4JWTZ1GmmWQP1OrQvovQ9lwbDQCXbSzXQ9yVje8/V98/LECQNOrJcJ59OrKmH8+ef0JmwEu49+nXPaQzNDTdI1tRHfp31bpOHdJG041GW5KJg/varbR8GyK+/+x8/7AAQdI00Bq+3Actoukerhvq6iGtvraTqG4eOdc9MsLwwIbBQn1N32g1yUPddfhM4Sn3X7H5LDI9JGn/RfH9MXffPyxAkDS5XCAeh9YiBpoeogjApooDMsqRPdc95YuQ2SaZx9nxGqVGIbjuyWkRBal7eM/8KXmlatyaEd+fmM3vHxYgSJqSbNBeNAqPBnhu+tZOMejutEeCvE8ebVhxZgmALRuKGoXrCR5bkLrHE5HQ/NEsOjEJ4J2D1fLH95/N7x8WIEia1slw9Isc69Ir5f6i52YOeaKL51x5+Rb+tF1/vNPc0lI3riuWSN2jeZxUk+rwmlRO2oXrIf4z11j/+2HR971q2dR49gOVxPeXzPT3DwsQJE1uDmDvWLXPhPF4ilrNfHm5trNl4MgsEcLvcWo15dNcD0ruCvVw5AQAPPrYMAB8WC55fH+7pnya66Gr7x8WIEiaXDaoxH+apP80G61VFvHQMha7p67HKWtKveKJZ1Odp14DgGYDAO6cBIBLRXwmvr9dc+a+f1iAIGkqTof2RJE1NI9WO7OyX0jNwfF7nHa2idaDLJ+aHgTw4i0AcPUyADg5AZTtn5LE95/p7x8WIEialgXg2si/+mifH+ZvZbf166eVdwwDAM5C1ys8rkxIb5Lq7HJkp2i6h6DYM3mf7090/iU/e9MavPRuq2Z8f85sfv+wAEHSlNwPcGCsNT66oBdvUtNndc+rGX17CYClK84h0y7cE/VHoAkt2iC9W8n6684CGD1SfIv8tvH9idn//mEBgqTJRYF45Ji8Uk0P+XeU1m1le7GaNiINunNDE8C2w5360uPkcC0i1w61mho891BCUu3c0Pkr5HeI70/M5vcPCxAkTe5kOImmh7rzNT1RC//6JT+SUvYj9YEWmdZ0hlZftpIxDd5K80Rt+Yn4/jP9/cMCBElTkg1qZ6X3gjzXUtuFxNFOKNDkuWQxUHXKpLa3iMclPLpKi1HYaPLH95/97x8WIEiaXDaoHQGom12o6QnPuZZ1TyjgZ1PuuBYAth0eROb/kebYpegk7kF6tI62KumPaWjyx/ef/e8fFiBImpJbImlPUN2zADTd0694hXGC8cmJKVmfPFGa+y9dXq17JPZJNXaeo6btOPILx/fnzM73DwsQJI1xS2RxxNtwrWOfO6DhOYG+EKkYzzekeDnFTMgT3X4UyLICly4v5ploUQV7B1OF7hFrn6QFaQ2SKMhf+pfG9ydm+vuHBQiSRt0RJke8J1LB20psTUYxBHkKjdazdkuhXw9JfVNX98j4htQ9Z9gZBHzfrf0l4/vPzvcPCxAkTeueYDs2rN1XJXVSVYy52q/1306lZbZwPUTsAJDpofMfDwL45Bzg22/KT6MntDVOqXsoErLzNhTkKWRlxvdvM/vfPyxAkDS5OYCWL+6JD0i0e69kzkmVNqq3I4nffUtv2bSmiUwPEdwrJcg31bSLJ64s8w3JA85yJys7iO8/B98/LECQNOotkf67yDX8OSeeU44zqrNlZCyc3kJ6iH7ee3sTWaSC9grlTg/+VHS9yHhtCx5pJrJ4SPHvKuTTI77/3H3/sABB0pTMAYi6c3+7Vb/g2kjLp9eQupM0xMtm5NuG/+1aHr9Ey9OM7z/73z8sQJA06kqw/1xiu4e6GSb+LMi6+ZJcKn/NuZI/vv/syL9o9/7jzgZBcOERLlCQNA0AV66tOIbPTpDqne4O5wAw/u4YQv6eSVn+1i2RVXmC3vwQTSz/WqZ/nbJNyF/K/JffXv2YHfnDBQqSpuKGGImdH6KdOcOfesi3VVuV7qiyCfk99C6/HZPJIvRNFG8y5iWdtvK2Y/4WiV/+sABB0pTcD1B3B6rUK9pJZrbnx9H8P02HhfzzTX5fZlGvaHce++UPCxAkTet0aPrFfzqkLOetZNjLntHb3qr0awvjOOSft/J79P3WHSOVdSTPbx8uLS97ozXzCQsQJE3JyXDST7Ln2vZeValLeB0NT86jZ49syD/78sv6Eqn1B27YbNRvTE8DmJgAgOnRN0t7IKRlsDNYwwIESWOcDFfEjhXY5VJn2PrDI5U/Y157Y8jvea/21K4vffG6Wr9VZ4j6aSJbF5gauKlQZ/H0FIDTU83CWzRrwM9ZCgsQJE3rXCD6pbvTB2w8vqNW09lzyO95C2ZLfrkPi2tlj9bnTJwqSjuwckDUGgAwBCCzDzRbsK1B7AkOgvwtkXIFUYsia7t1ZLmWqcJL7Nhtpa8c8s8r+Tl+3b9x83lU3SyWMQngxLGLyx82GgCG1jaRWYOPDnWiRtwOxBwgCPJ3hGXOVXkUmVM3gqE9zUrsTJJybSf7CfnnVn6es2nr/lXrJgslY8x68HOh6WxQaRlkD9Tq0L6L0J45NBoALttYbgfoa4cFCJJGPRlO8x3rxo81NC9WIjPgPb2F/Db9lZ97/37db98Rpul+2Yrq0L+r1nXqkDWYbjTakkwc3NduFVGgIEADreHLfdAimu7huqGuHtLqazuJjLzCkN+QR6OP8pfCdb+M8Mgb3jV9r9WkGYJ2w0DrPoHNZ5HZAUncEhkE+VwgHofWIgaaHqIIwKaKAzLKkT3XPeWLCPnnVn6Z7ZN5/B2vXWp0wr4VWCJ1P++ZP6VZgbZuEBYgSJqSbNBeNAqPBnhuKpe7jXh5d3oo5CfmSn7y/nm0Z8WZJQC2bChqdHknJCF1vycipM0HsujQJIB3DhalDQsQJE3rZDj6RTthnesS7i96bhaRJ7p4zsWXb+FP2/XH8x2G/HMi/8mJKX6Gj+TGdcUS+z5gWZPq8JpUTtqd2wH+M7cYuRtoAPq74lygIHVycwDPDR8Sv6eo1cyXl2s7Wwbej0bIP9PyEzJLh/B7/FpN+TTXg5I7RD0cOQEAjz42DADs7viwAEHS5LJBJZ4RT/hPs9FaZREPLWPRIuSfW/lt6nr8sqbU6571BKrz1GsA0GwAwJ2TAHBp7AkOAqLidGhPFFlD82i1MytngpDfU95f+aXm5vg9fjvbR+tBlk9NDwJ48RYAuHoZANCt8RRhCwsQJE3LAvAVRP/qo7YntW4ru61HP4X8tgweeXqRv1XzjmEAwFnoep3H9QnpzVOdXY7sIE33ExT7J+///YnOvzTP2bQGL70bFiBIm5L7AQ6MtcZHF/TiTWr6zHNeTchv999dW7/8nNG3lwBYuuIcMu3OZwL+FQBCi/bI2YVk/XVnAYweKb6Ff9uwAEHS5KJAPHJMXqmmh/w7Suu2sr1YWxuF/DYzJz9ZsJ0bmgC2He7Ulx4/h2txuXar1dTguZ8Skmrnhs5fEesAQZA/GU6i6aHufE1P1MK/fsmPpAz5Pcym/FIfaysDms7W6stWMqbEW2kzgTgVIgjKskHtrPRe4NmF9i4kjnZCgawZ8s+t/MQli4GqUz61vV08LuSxFVqMyIbLHxYgSJpcNqgdAaibXajpCc1/ldqI92m/PeTn7+L1Z1N+2n2241oA2HZ4EJn/TZp7l2ITuAfv0fraqrA/psQJCxAkTcktkbQnqO5ZAJru6Ve8ouoEY94q5J9V+U9OTMn6NBOg2MvS5dW6X2KfFGTnmWrWhhMnwwWBdUtkccTbcK1jnzugYeuq0nMKxq1+Qv4OMyp/uyGtV1DMimYC248CWVbm0uXFPB8tqmPvIKvQ/WLtmawQrQETXP6wAEHSqDvC5Ij3RCp4W4mtySiGIE+h0Xq2JQn5yyScEfn5kjDq2AGp7+vqfhlfkrr/DDsDgu97jpXgIMjuCbZjw9p9VVInVcWYq/1a/+1U7fKQf27lb8PXmHn+0g4AmR04//EggE/OAb79vvw2AEJbY5a6nyJRO29DQZ7IBg2CFrk5gJYv7okPSGRvpZEEVGujag2nvdHTKuTvl/z8vTwqtWlNE5kdIPisgKC5gabdPXF9me9JM5Asd7W8VViAIGnUWyI1beTfi+TPOfGccpxRHgMJ+W05tfJ+yc9ryrUIegvZAfp57+1NZJEi2quVO735U9H1IuO1LXikn8jiUcW/i+9nCAsQJE3JHICoO/e3W/ULro0859xLQv5esOX3nG7NJSQN/bK58mDD/3ZtH4Uk9gMEQQt1Jdh/LrHdQ90ME38WpF0z5J9b+W2p/DVnWv5Fu/cfdzYIgguPcIGCpGkAuHJtxTF8doJU73R3OAeA8XfHEPL3TMryt26JrMoTLM8PsaPX/uiEFpl2/jFdy28T8pe9vYQFLX+4QEHSqFEge06dRVibKN5Ey0s6beVttfwtkvyIV+MGpTuqbOz8Fu3MHP7UQ8ivMd/kDwsQJE3J/QDamQX8ab/Q7qzV/D9NB9TyWdt15LsI7SQ2e+bDCfkXivxhAYKkaZ0OTb90EQEAsHXHSBcvfn77cGl52RsrPD9Dfr3PYjlvJcN2dkTC9lalX1vQQyH/HMofFiBImtKT4YCqGbfU+gM3bLZeMz0NYGICAKZH3yztgZCWwc5A5JTqJ6OVVl/bidtdZFpr69njG/LPtPxhAYKkUdcBpN9WV+u36gxRP01k6wJTAzcV6iyengJweqpZeItmDexTL+09snaJVi51lVZHw7/TN+SfTfnDAgRJ0zoXiH6hMSH30XCt7NH6nIlTxZE6sHJA1BoAMAQgsw80W7CtQVvsgvwa3aVheXxfraaz52Tl/97WXy2U3Pf8f1ZKxUtG7r+5/fPwM28Y75LEHCAI8rdE9qL7N24+j6qboTImAZw4drEiUQPA0NomMmvw0aFO1IjbgfYoL8jviSJru41kuZapwkvstYtKXzkp+e+56w8BfGXFIQCf+4PlhT7JJjzxwcbSNxLUluA9vHr/ze22L+z5jlP+sABB0uTuCOM5m7buX7VuslAyxqwHP9eXznaUlkH2QK0O7bsI7ZlDowHgso3ldoBPJbJ1PvqtPIrMqRvB0J5mJcW187oZ8CnIb+t+olX+9CEAz9z7l4Wn9z/3NWdbete3/2FXpfxhAYKkaVkA7v37db99x5Om+2UrqkP/rlrXqUPWYLrRaEsycXBfu5WW70Fovm/d+LGG5oVLZAa8p7cLT37JsZH3AIyOdv4/kBanf9ePfLXwdP364v+lV5/+UDwt/u+y5Q8LECSNuhLMdb+M8MgbujV9r9WkGYJ2QnzrPPjNZ5HZAUnbTz0wBvI7SxYYAOi6k+uGunpUq6/tpDPyahORn0O6f93wFQCAjh0gjU66nEpycZ6Wvu/Ulzbh5JkjAIBiHEmTPyxAkDQ5CyCzfTKPv+NXSY1O2Le6SqTu5z3zpzQrUNcNGDwOrUU8ND1KEbBNFQd8lCN7rntKGXFhyy8hO0BwXc79fl6H/h/ypxwq/+ff+z4AfOe7TpnDAgRJUzIHIO+fR3tWnFkCYMuGokaXd/oRUvd7IkLafCCLDk0CeOeg9ccQvWhEHg3zaDLtFAO+N6quHr2w5afoPkX0f/reqKO/9e2fZP2fvgcA11yxHmXrBh75wwIESdPOBrVGyY3riiX2fa6yJtXhNamctDu3A/xnbjFyN4gAAK5aNjWe/UAl2gnxXBdyf9dzM4o80chzrr98C3/arp+U/Byurf9v7wEAD/zsnwp1slkB9xrWF54+cua3Aey7dVPxj2TY8ocFCJKmZA4gs3QIv8ev1ZRPcz0ouUPUw5ETAPDoY8MA8CFK8dxQorXyeLpazXx5uba2ZeD9aCx0+TWeufsHAIDlyK8NZ6sEHUZHO6sE9PQZUNsfAHjgwz8r1PfIHxYgSBp1JZhT1+OXNaVe96wnUJ2nXgOAZgMA7pwEgEtFfMO/D1XDf5qN1iqLeGgZlxZpyr9r+TcKJaT7d63+AoDPtbQ7q7/6CwAeGKXZAl9L7vSm2QGNsABB0pRYAKm5OX6P38720XqQ5VPTgwBevAUArl4GAHTrt3YHIMcTBdfQPFrtzMqZ4MKTn2I+jw9+F8AxVs51v02rzhkAeHykuOL7GL4K4Muw4kKcsABB0uRPhrtjGABwFrpe53F9QnrzVGeXIztI0/0Exf7J+39/ovMv+amb1uCld1s1uTXwr55qe2rrtrLbevRravJT/J4ga0B7eVes7tTJZwFZ/VAPvE/gx/DteQgLECRNyRxg9O0lAJauOIdMu/OZgH8FgNCiPXJ2IVl/3VkAo0eKb5E6hvu1B8aapXU89OINa/rYc95OCvL/68guANdfvwXAspWdx9xf/+C/fwzg9MkxAK8MvVPo4dipkULbfA+dtm+99YpT/rAAQdLkbojZuaEJYNvhzmPp8XO4Fpdrt1pNDZ77KSGpdm4AdN+OR77JK9X0qH9HbN1Wthdua6N05F/xi5813rICnwWwZfB4ofz1M8NGK+L062PQ/wopf1iAIGlyJ8NxpD7WVgY0na3Vl61kTIm30mYC/EhKbcRreqg7X9kTdfGvv6Ys/0xj/xVc/rAAQdKURIEuWQxUnfKp7e3icSGPrdBiRDb8lgCOlu2oZaXXRZ7Lqe2iktKG/Lw+Re41Wrt+xX8N2jMgT4Lg3I+fK7zLlj8sQJA0JWeD7rgWALYdHkTmf5Pm3qXYBO7Be7S+tirsjylJtBMTOHWzIzU95zmXs+4JCynIv+ff/8d4OnPY8ocFCJKmtSdYagWaCVDsZenyat0vsU8KsvNMNWvDkWOan3Vc9ywDTXf2K95SdQIzb3Xhy5/P2xGsBoDHUZwn8DxQnRFk1uZ3f+MaVl4uf1iAIGlyN8VTvJbm7DQT2H4UyLIyly4vDj0tqmPvIKvQ/WLtmawQrQETbW03XvYnaWfe23CtaZ+boOE5Qb8QaQn5Zw1N/rAAQdKU3BLptwNS39fV/TK+JHX/GXYGBN+3ams1OeI9kQreVmJrMoqhyVNotJ5Tk/9tAMDDf/23AL755T822hIU0c8zUtmK+relImIlOAjyK8F8jZDnb+wAkNmB8x8PAvjkHODb78tvAyC0NWap+ykStfM2FOQpZDWWRrEIroe0u1U8+e5+v9Z/u1Zq8t/1az+PLD5j6+ne+aXPXAPx/6RUtrAAQdKU5ALxu2PJe9u0ponMDhB8VkDQ3EDT7p64vsz3pBlIlrtX2YE64j3xDU9v2omZVdq0WkNrb/S0Wljy52PzFlpv/gxWj/xhAYKkyVkALZZMY47sAP289/YmskgR7dXKnd78qXjPompReKSfyOJRxVFbyEdHmbaw9Ucvp51pGZR+r1oS8ttyauW9yx8WIEianAXwnE7Mxx9p6JfNyLEN1+JaHrlEy3OsEw+p16pf8L8x5Pe36hdS/rAAQdK4Tofm+M8r9nh7Mi7hzxuxa/rltHsI+TUuDPkX7d5/vEdBg2Dh8v/LTEi46XMPqgAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "env.reset()\n",
        "display(Image.fromarray(env.render()))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "obvious-implementation",
      "metadata": {
        "id": "obvious-implementation"
      },
      "source": [
        "# Part 2 - Model-Free Control\n",
        "\n",
        "Recall that in Part 1 you implemented Policy Iteration with Policy Evaluation and Policy Improvement. However this required knowledge of the model (i.e. `env.P`). In this section we study methods that do not require knowing the model."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "expired-promotion",
      "metadata": {
        "id": "expired-promotion"
      },
      "source": [
        "# Q-Learning\n",
        "* In this part you will implement Q-Learning with epsilon-greedy exploration to find the optimal policy.  \n",
        "  * You can experiment with fixed and adaptive learning rate and epsilon\n",
        "  * Report your best result and mention the differences you observed in answer to final questions\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 115,
      "id": "focal-capability",
      "metadata": {
        "id": "focal-capability"
      },
      "outputs": [],
      "source": [
        "from matplotlib.pylab import e\n",
        "# Initialize Running Parameters. \n",
        "# You should try to maintain the *number of episodes* and *max length of episode* at these numbers for your final results\n",
        "# However feel free to modify them for debugging purposes\n",
        "num_episodes       = 5000\n",
        "episode_max_length = 100\n",
        "\n",
        "# initialize discount factor and learning rate\n",
        "gamma     = 0.95\n",
        "learnRate = 0.6\n",
        "\n",
        "#create Q table\n",
        "Q=np.zeros([env.observation_space.n,env.action_space.n]) #matrix Q[s,a]\n",
        "\n",
        "reward_of_episodes = [] # list containing total reward of each episode (these rewards don't need to be discounted)\n",
        "\n",
        "# exponential decay for epsilon\n",
        "min_epsilon = 0\n",
        "max_epsilon = 1.0\n",
        "decay_rate = 0.003\n",
        "\n",
        "#epsilon = 0.175\n",
        "\n",
        "#execute in episodes\n",
        "for episode in range(num_episodes):\n",
        "    \n",
        "    # reset the environment at the beginning of an episode\n",
        "\n",
        "    # state = env...\n",
        "    # ....\n",
        "    state = env.reset()\n",
        "    state = state[0]\n",
        "    total_reward = 0.0\n",
        "\n",
        "    for t in range(episode_max_length):\n",
        "        \n",
        "        ########### SELECT ACTION a for your current state using Q-values ##################\n",
        "        # for example, choosing greedily:\n",
        "        #action = np.argmax(Q[state,:])\n",
        "        epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode)\n",
        "\n",
        "        if np.random.rand() > epsilon:\n",
        "            action = np.argmax(Q[state])\n",
        "        else:\n",
        "            action = env.action_space.sample()\n",
        "        \n",
        "        # new state, reward, done\n",
        "        newstate, reward, finished, _, _ = env.step(action)\n",
        "        ##### update Q(state,action) ############\n",
        "        Q[state][action] = Q[state][action] + learnRate * (reward + gamma * np.max(Q[newstate]) - Q[state][action])\n",
        "        total_reward += reward\n",
        "        \n",
        "        # stop if we have reached terminal state \n",
        "        # ...\n",
        "        # update state\n",
        "        if finished:\n",
        "            break\n",
        "        state = newstate\n",
        "    reward_of_episodes.append(total_reward)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "unnecessary-relations",
      "metadata": {
        "id": "unnecessary-relations"
      },
      "source": [
        "- Construct `reward_of_episodes`, a list with the total rewards of each episode (these rewards do not need to be discounted)\n",
        "- Plot the moving average of the list\n",
        "- Ideally you want to achieve a reward greater than 0.7 (the problem is technically considered solved for 0.78)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "answering-boards",
      "metadata": {
        "id": "answering-boards"
      },
      "outputs": [],
      "source": [
        "# You don't need to change the code here.\n",
        "# you can use this function to plot the moving average of a list\n",
        "from numpy import convolve, ones\n",
        "def movingaverage(interval, window_size):\n",
        "    window= np.ones(int(window_size))/float(window_size)\n",
        "    return np.convolve(interval, window, 'valid')\n",
        "from pylab import plot"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 116,
      "id": "uKZpBQq9ywSe",
      "metadata": {
        "id": "uKZpBQq9ywSe",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "outputId": "550ee48c-f7cb-438a-8775-9a729a3e2eb5"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f4b79c9beb0>]"
            ]
          },
          "metadata": {},
          "execution_count": 116
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAA9sklEQVR4nO2deXwU9fnHP8/u5ibhDPeRcIRLOSPFingBgihaWxV7/LT16EXV2gurUmvVovan1f6wlZ+1rf5qPWuLEkFAKYqKBDkTCIQQIAFCAuRO9vz+/piZ3ZnZ2d3ZzW72yPN+vfJi5rvfmfl+l9lnnnm+z0FCCDAMwzCphSXeA2AYhmGiDwt3hmGYFISFO8MwTArCwp1hGCYFYeHOMAyTgtjideEBAwaIgoKCeF2eYRgmKdmxY0eDECI/VL+4CfeCggKUlpbG6/IMwzBJCREdNdOPzTIMwzApCAt3hmGYFISFO8MwTArCwp1hGCYFYeHOMAyTgrBwZxiGSUFYuDMMw6QgLNwZhkk4DpxqxsufVoNTkkdO3IKYGIZhAnH3P3ahoq4Fl44fiBH9suM9nKSENXeGYboFh8sTVBN3uj3e7Yq6FgBAS6cratd3qc4fCo9HwO1J7rcGU8KdiBYSUQURVRLRcoPPRxLRh0S0k4j2ENFV0R8qwzDJSqfTjaIH3kPhfSX4z8F6v89f/uwoxj/wHs62OTTtS/7n44ivOfHBdbj9b9sBAFsrGzBpxXqUn2g2dezoX5ZgzC9LIr52IhBSuBORFcAqAIsATAJwMxFN0nV7AMDrQojpAJYCeC7aA2UYJnlpbHd6t9/ZfcLv879sPQKPAE41dQIAhvbOBAC4uqA9dzjd2Lj/NABg7d6TcLg9+OLYubDO4XCZ1/YTDTOa+ywAlUKIKiGEA8CrAK7V9REA8uTt3gD8//cYJkq8taMGNz3/abyHETZv7qjBvtqmeA8jLtS32L3bHgOBXVXfBgDYduQMWjqdOCELeQCoPN0S9vVe/rTau33T85/i8OlWAIDNQmGdp+iB91CwfC1++sZuTXun043rn9uKguVr8cS6A2GPrzswI9yHATiu2q+R29Q8BOCbRFQDoATAj4xORER3ElEpEZXW1/u/mjGMGX7yxm5sO3I26TwpfvrGblz9h8jNDMlMY4fP3OIx+H8bnZ8DADhY14KfvbFH89kv394X9vVWvucTuNuOnMW2I2cBAFnp1pDH2l1uv7Y3d9Ro9v/88RF8cawRAPDc5sMJeS9Ga0H1ZgB/FUIMB3AVgJeJyO/cQojVQohiIURxfn7IdMQME5RKWRtLdFxuD5o6nAE/93gEqhvacKgufA21OxBC4OiZti6do1W1MHq23YmGVrvm87zMNADAR4casPO41nRSVS/9Px+sa8Hxs+2mxtvm8BfQANBqD71Aaw9gilEL8IpT2v+r9gDXiydmhHstgBGq/eFym5rbALwOAEKITwFkAhgQjQEyTCDmP70FL3xUFe9hhOSe13Zh6q/f9+4frtc+lJ7fUoVLf7cZ85/egq2VDd09vJDctPozXPLkZvzzi5rQnQNQoXpwbTlYj+JHNmqEpSJQa851oK5ZK/gbWh3YV9uEBU9vwcVPfBjSDr7/ZOCH5P0m3gICnX+zvBC8vfos1ujWDS5+4sOQ5+1uzAj37QDGEVEhEaVDWjBdo+tzDMAVAEBEEyEJd7a7MDFH/7qciLy756Rmv05lTwaATw77BPrnsvkgkVDGZLQQahYr+du6nW6VcHf6a74Xju7v3a4559PY20Jo3/XyW8EfvzEDEwbnhj1WveaeYZPE5Olm6f/N6I1R7+WTCIQU7kIIF4BlANYD2A/JK6aMiB4moiVyt58AuIOIdgP4B4BbRSIaoZiUIxnvsuc2H0bB8rXYWF4HACCV4IvUt/rfu2px1TMfoWD5Wry7Jzr+DA/8ay9e2XbMu3/8XIdhv3/trEXB8rX40T92Gn4uhMB/bzjo196hEuidBsK9ULbDA8CT6yu82/+9ocKvrxrF3bEwPwd/v/1LQfsaoX/QvHBLMQDg2U2VAID7/rk37HPGA1M2dyFEiRCiSAgxRgjxqNy2QgixRt4uF0JcJISYKoSYJoR4P/gZGSY63FA8PN5DCJuPZdPL7S9JZSbVOu3Ygb0iOuf9b+9D+UlJqC17xVjIhoPT7cH/fXYMv3zbJ8iumzbUsO89r+0CEFizP91iN2xXC1G9tnz99GH4+ZXjvfuH6302///77BiCoZhVRvbLRt/s9KB9DcelG0ufrHRYCKht9H+4ff/SMQCAqSP6hH2dWMMRqkzSUNvYgaZ27cJkmjX5b+Fq1WJlc2fghddAlJ1oMrVQaJYvjp3DBvmtQs3qLVUoP9EMt0f4LSgGQ+0dU71ysXe7/GSzd6H2jM6s8dRN09AnOx2zCvsZnvOxkv0Br9fpciPNSshOt8FiIcweLZ3DKrtBOkNEqp5s8hfi98wrAuAf5fqLhROQbrWg3e7C0TNtONfmMDw+HnBuGSZpuGjlB35tRm5ricT+k8EjIp1uD46e8dmTT+rs8aHocLix+Nnoulde/9wnhu3NnS5c9exHGNYnC7WNHVh71xzk98rQ9DnS0IbCATmaNrvTWJje+pftyE634qOfXxZwLGdajbX+1VuqMHNUX1w5ebDfZ51ONzJtPpfHKyYMwmdVZ70mr5K9J3HtNL03t4/v/FV6o7pwdH98WnUGA/MykJUmna9TpdXn50pzd7g9OHS6FYue+Qhuj4Dd5dE8xOJF8qs9TI/koWukIOlAgiNR0Jskrpw8yLt9/YxhfouD6WG+iURTYzeLYp6oPdfh1biH9ckCADS2+y8sdsoP4N/fNA0A8NJ3Znk/a3e4cU4+5umbpuKdZXNQ/vCVpsZRFiAgrNPpQUaa73u8/eJCfPHgfO9+MLdUNfcuKMLOB+djUF4mMmX/+HaH9H0vmToU798zV9O/3eEO6EYZD1i4M0nJ9JF9YbWQV3AkGruON+LqP3yEW178XNM+UpXh8J9f1OKrf9Rqyc9sOoSC5Wtx/XNbTV3HaCESCC9JVqScbrF7/c6vmTpUHo//dd/eKXlO52ZKhoLeWWmaz+c9tQWA5Ot+/vDeyE73GRQU7diIZz+oxDkDLxW7y40MleZOROiX47O9r/h3WcDvTU2mzYq+8nGZssfMrEc3AQDGD871fhYOZ9scmPP4B/jXTr03efRh4c4kJYX5OciwWRJWc3/k3XLsq/WZZOaMHYDXv3shKuq0bnTqhUI1SvRjKDoCCKlIXfPUqQEybBbk52bggcUTvRGkaqwWgkN+iAyUhbCRmez5/0ixCMqpJw/N8+sDAMUF/vb1FVdP1ux/ZbrWnGK0NmB3epCZ5i/a3v3RHO/2obrQAXBq7T8jTRvZqrhHAv4Pq2B0Ot2oOddh6uHSVVi4M0lJboYNaVYLdtc0xnsohjTqXv2funEqZhX2w6QhxoLNiLdM+PCfaTUW4uF6VNpdbjy14SC2V0s+7T+7cjwqHlmE7ffPw+0Xj8Z7d1/sd8yppk6vpj5UNst0Oj0oP9FsGEmqvE3YrBbkZfov9xkJyUlD8/D8t2YCAOZNHIinb5qG1+6cHXAeDpcHa/eehNH0zxvW27udZgudY0YtwEf0zdJ8lqkS9nfOHe13rFH2SSEESvZKMQ/pttiLXhbuTFKgD5sgIrTaXThmIhw9HnTowtHzZMF16XjjtBtThvf2a/vJG7tDenYcafBp/komRSD8hean3j+IZzcdwk2rPwOgFWyA8VpAS6fL++bQNzvNe92rnv1IE7GpaNHjVQFFzbo87TNG9gk4NkXTv36G5PY6Oj+wu6giPKsCvBEpeEy88KnNLsqagoI6HuGwQVDTdav8zWrry07hkbX7/Y6PFSzcmaRAHc1YNEj6cd8wc3jCpmRVC4OPf3GZV9ObPbo/Kh5ZiKrHtCUP1iybg4vH+WfsCGR2UXCohPiWn1+GP9w8HUDg/CiBOH5O+5DUu5iqA62qVy5GXqYNTrcHnfJDTBGERmayOWPzMW5gr6BC+fXvXhjws+F9s1G9cjGuOn8IgOB2eH3OmkA4gjw0B+dl4ivTh3nz3QDAwLxMTZ9xqngE5btWFvkDnV+dGTPYdxEtWLgzCcH6slMoWL7W+1ev8zJR/1iKBkkaoMPlwbl2JzZXnNb0bepwYvZjm7C+7FTsBx4A9Wt3lp+91gqLhTAoTyukthmkHvji6DlsrWzAlx7b6M11ruahd8q92zarxXutb/15W1jjPdGoPbclRGrc7HQbXv7sKLYdOQMA3mChd3TRsVsO1mPj/rqQC9+2COMVylWupi63x6sZB3sAAIHzx5xtc+BUcydKjwZPA5Gpyi6pXKtA5wLa6XSjYPlavL1TMq+99YVvEdXILBVtWLgzCcGzmw5p9vXRgE7Vj3HF1ZKGpPywn96oPba6oQ2nmjvx3IeVsRiqKbJVP/7+vYwFzSt3zMbwvln41w8vAmAscF7bfhx/2XoEdc127AmyvvDirVKIvLIIqE++FYqJurWAwTpNFQCe/9ZMvCKH80+STSWKtUwRcHpzzuot0mLq8bPa/883v3chLimKPDPs/3xdekNRp/BVu4WuvWuO3zFqApmtlOAs/XgB4ImvTfFuD1GZwH6xcAKe+OoUXFKUj1u/XOBtVxSU362XUi/sOt7o/WwMa+5MT0GfI8atM4qqNXfFfq14cOw+3qjJ0vehrMnvrunewhjHz7bj8t9txvGz7XjfwItDz5j8Xvj4F5djmi50Pd1m8bapBZbeSqteh7h8guQ/r3YBfOTdcrz82dGgucZPNHbghj99gnd1qQOG6xYQAeDKyYPx5bGS6WjeROl6mw6c9tr6c9Kt3spHyvWVVAt6m3pxQT/8TeXvHi5XTxmKXhk2rxnI4xG486Ud3s8H5vo/nNQE0txbgkQI31jsS46rNtlkpVtx4wUjQET43iVjvO1KZK7R9x/qzSgasHBnEgKL7k7U227VP0ZFO7z9Yp+Xwl2qpFW/12ny3cV9/9yLqoa2iNO/Pii/kTx63Xm4Q57brIJ+ULLP6GVEMLsxALzw8RE8+K992HIocBrhX79Thu3V59CiC4YKlZNF7WqoVE3S51B/4eMj3u2cDGMzxNyifNw7vyjotYKNQdHAqxra8Hl16IyaSvRsoDWJOlnbvn66cQTr/VdNxMQheZo3MzVqf/rXtks1jvSiXR3IFks4/QCTEBC0moz+x6cIsmeWTvMu7s0Y2TfkeT0e0WUtqcPhhtVCAd3XFM1M7bkSCbfNKcRtcwoB+NLLptssUIav97DQe+QAxsFL7bLgFkJACK3WeDCAv3ffnOC+2+EmOCODlL+ANlo1XDJsVu99ovbrX/X1GQGPefHWC3DZ7zYH1NwV89+Ka/RloiXumDsadxi4Piqo75ETsmlRqQObm2nD4vOHYOVXpxgeG21Yc2cSgr26UHJ9kMej8kKZJYCQAOBNoatGXSQjUmY9uhE3BKnZWnhfCQrvKzHMGhgpiqb72/cOoE0OeV+zWxvVqGRjVJNn4Cv+ruwe+I0XtmH0L0tQqtJwA32bodIghJuw7fxh5v37zZJhs3iF+42q/5/sjMCl9BThG8jmriz8ZqaFLscXCGX9Yn2ZdD/Wy5G8LZ0u2KyxN8cosHBnEhK95v7BAcmW69LZ4tfd4wuuUWztiqskAD9zQ7i43B602F3YrVoMM8vdV4zD1uWXR3RdtRlDcQO16WxXmyukejg3z/LZgtWBOgpKEq1PDkueLSV7fV5ErgD+1oE0bQWjtxgl/a2euy4fix9eNjbo+SIh3Wbxy72+9IIRmDM2cBE4xaQXSHPvdLhB5L8wHA5vfV9y61SbaJSEcBcYROHGClMzIKKFRFRBRJVEtNzg86eJaJf8d5CIGqM+UqZH8crnxjm7nS6tMJowOM/rU155uhX/3lUb0NQQCUa22eNn2zH7sU0oWL4WB4PUPb1u+jC/4JdIUCohrd170msCUmdL1Ee96j1f9FrqvtomFCxfi13HGyOOEzDS7AN5gNy7YLwmX0y0qGpow/vldZirWuO4d35R0LcKn+buP2+3R+DZDyqRZrGEfLgFIzvdhrEDe2ne5JQ3i2jcD2YJKdyJyApgFYBFACYBuJmINAYpIcSP5SId0wD8AcA/YzBWpgcwc5RkR9eXm1O8Ny4u8tfKVn1DsrGOH5yLu1/d5fd5V6IB1UJAMRX952A9Tsk28R8bmEYU1O5ykXDFhIF+bUqWyZ++sdvbdvUUbRGNny8cr9lXUgR4v1vZLHPdqq0hF2UDodbcn5RdBI3yudxxcWFE5zeD8mBSRykHcjtVyAgi3JXzDMwLfg4zBIos7oq5J1zMaO6zAFQKIaqEEA4ArwK4Nkj/myGV2mOYsAnkHtc7Kw3nDcvDkN7+ms+0EX3QLyddkwxKrSG1dDoD5gUPhVqzVQppqAVDmUEOEYWu/pDvXzzRr83tEdhX26TJ+67PTnjZ+IGaJFk159rR7nD55VkHtNWQbptTiOqVi03lIrfJi7K9s9Jwg+wiqHbD9M3BeGEyFnx37mhvQY5AKG8cmytOa+qyAr4F6gcMvvdwudzgwQx0zdwTLmauNAzAcdV+jdzmBxGNAlAIwL+qAsMEQO0HnKm6+dXpXMtONGuyLOo52+bAp1VnvPvqhcVpD2/AzEc2RjQ2tUljxb/KAAROsxttjOzap5o7cfUfPsaBEJWQ1Em4DpxqwV3/2In1+/wjdtWRnIGyNRqhZEm8YqJPiOUEcA+MFXrLScm+k8YdNcdIB22vPoc5j2tdVpVUD9HQro2KaEfr3GaJ9mNkKYA3hRCGdz8R3UlEpURUWl9fH+VLM8mKsmB425xC2KwWfPuiAgD+pdfCoXeWDf11Gm2oJFxGqDX3dXI6g2B5W4xME5Gitmsr2/psi9t+eYXhsXrb7sb9p70CWRnjgkmDUDggB70ybNh471y/dLrB6JVhw5afXYaV1/vc+iYbLObGkj99c6Zm3yiqNByUtxh9uohIqAlQTDzRNPdaACNU+8PlNiOWIohJRgixWghRLIQozs+PPPSYSS3+uPkwAF8xh4vGSHb1J9YdACDV9AwXIfw9RyLRuI0EeSA3ui8V9jMM248GSjRum9137ex0KwYFuJ6Rb7+SVEuxwb9fXoez7U5MGJyLsQNzw15EHNk/W/N2oRdc0RCSwegTRh71QKjjAhSbezS060Bz1+eFjyVmhPt2AOOIqJCI0iEJ8DX6TkQ0AUBfAIEdghnGgKc3Srk31slmA0XIKyH8Sk1Po7S4CjfMHK7Zb3e48bsbpmraQmVYNEIt3G+eNVJqM8h8eM3UofjVNZPx51svQO+sNG8O8q6gNpkoZhj1gyWUp8tPFxQFdQsEgAMnmzVFKbqC+k3je5eMwer/6vp3EIyZo/pqcrkoGTHDoV61FuOWzYP9IqiwpKd/L+kcswr64VuzR+HqKUMwdmCvbkkYphDySkIIFxEtA7AegBXAi0KIMiJ6GECpEEIR9EsBvCqCJbJgmCAod06fAKHvwQTVkmlD8YaquMXI/tl+mQGNIjoDsa+2CUWDcjXCNM1K+M275fjrJ9V+/dWCZfevFpi+TjDUmnT/nHScaXPgDx/4kqEFC+gCgGWXj8NFY89587sYYXd5DBdCI0H9trB80YSonDMYNqsFDy2ZjPfLTuFEUyemB8kJr+Yn84vw3xskhUL9oD4mFyqPhnBXPKW+OnMYbrpgZJfPFwmmHtlCiBIhRJEQYowQ4lG5bYVKsEMI8ZAQws8HnmHM8jVZ+1YL5fdVaXvHDQoc8q5PFDV/on/+DrMRpLWNHbj6Dx/jV2vKNJr7S58exZ9V+VIUYmlHnVUoBb1cJ9vD1WH2ZtwYhxkkANMTblHuUCyW8653F1+ZIX03ofLhKKiLhqj/fxU7eTTMSZcUSQvNk4d27zqEGs4tw8SdiUPy0OFw4XbZJ1qtOVU1tGHq8N5o6nDiummBF/zGD87FJ8svR5rVgqx0K3rJEZ4/uHQMnpNt+i63uZdKJRfL9uqzASsnAcDehxbgTKsDQ/rExs4OAK/eMRu1jR1o6XT5PViGmvCjH5ibiV0r5mPawxsC9omWWQYA9j+8sFtKyKn5yfzx+OFlY00HSi2YPBi/vf583PfPvZo3MyLpO41GxsbFU4bgkvFXeu/DeMDpB5i4Unm6BftPNsNqIcMFPYfLg901TZg0NC/kgt/QPlnIz83Q/KCGqLxGXv7sKG58/lMULF8bdHFVuU7l6VZ892UpjayR/3RuZhoKBuREzaxhhMVCGNEv21AAnzAo3mFEIDOXQrh5YoKRlW4N6WsebSwWCjsCVgmKU69b2F0e9Dap/ZshnoIdYOHOxBkludJhXc1LRUAoi6BWfU5gk3xl+jBcdf5gAMCG8jpv5Ks+gEWNPn8N4F8gubsp7O8LQOpK5OtzcjTvMlWul0XnDY58YEmK8kC264R7d7oqxprUmQmTlOj9thWUYCbFTfK8MAJs1PTKsBnmCw/2FmDkhTLPwIbfnahNBd+6cFTE57nq/CGoXrkY35ztO4di1+9JZBhkh7Q73d1uUoolqTMTJil5VS5ooHZpA4AfRDGLoJEveLCAJqPPRulC97szAZQeZbF4gmphMBQ3zByuCbDqk+3zEe/OqMlEwZtAzJm6mjsvqDJhE2kBDCFEQI353gVa7frrs0biyfUVvmt2wcE2NzMNX50xHG994XOVDOYWqQ9cum/RBI0bZtVjV3VLmTQ91SsXw+0RsFrIVP4XNU/eMBVPqvz+1QI9mjb3ZEER4urC3XaXBwNiuH7S3fS8/1WmSzy14SAm/2o9mjoC15o04kyrHdMe3oB1AfJ/ZOp+VFm6PCW9ghRgMINNJ4y/IgdGGfH1/92mHVuaVeMeFw/BrtDdi5WpihIp+uPXfNk17S53VD2H4k3qzITpFlZ9WIkOp9tb2d0s1Wfa0dThxOotVZp2Jfe43tapNxVcP0MbgRouXUjPjcw0S8xD6Znuxcj84kgxs0zqzITpFpSK7uEn4ZKO++JYo6Y1N8OG2aNDL+gFKrBsFkW4FwUJhApEhs2KzHTpp8KKc2qgF+IHTjWj5lxHt2X87A5YuDNhoaQICLeCj8NlbDTvdLlDLuh1teiF+vrnRRAxOH5wLtKtFhT0z/bLV5PM3H3FOMwt6pkJ/PR+8Qt//xEAbQnCZIcXVJmICLeCz8km49D/hhZ7QOHdK8OGVrsL76gKT0TK+MGSxt4vJx3fv3QM/rj5sHdxUs+EwbkYlJfpVzhk888u6/I4EokfG7iI9hSsFsJdV4zDs5sOwdOV1foEhjV3JiL+/JF/jpVAuD0C976+2699Y3kdTjR1BizCofhfZ0ehCEThAEm4FwzI8Ray+M275YZ9D5xqYfNLD2DNLilz+bJ/fBHnkcQG1tyZiFhXZv71tc3hMmz/6JBUsCVQQq8/3DwdRxraolJcef6kQXjjexeieFRftNhdWPneAcPrKvm9zSTcYpKbajkLpNoUs/PB+fEaTtRhzZ0xjb502DmTlZLOtmr7dTrdeG5zJfbJ9UcDmWVyMmx+BTe6wgUF/UBEyMtMw8xRfbGvtsmvT4M81oL+/vVGmdRHX482mWHhzpjmf3VujI+V7Dd13Ll2rXBvaLXjiXUV2HFUqrDUHbm/9bTZXYbujbWN0avGwyQ2i6doUxNPDVIMJhlh4c6YplNXXi5YLVHNcXKI903FUrVGfXTo7NH9ozC68Jg0NM9wUVgJRx83MHyXSSa5UHtO5Wba8O9lXV+4TyRMCXciWkhEFURUSUSGBTmI6EYiKieiMiJ6JbrDZBIBve3bbbLolpKcKS9LOn53jdYc0lUf9kjIsFm9Dx01dlngp1ICKcYYddnGlk7jdaFkJuSvioisAFYBmA+gBsB2IlojhChX9RkH4D4AFwkhzhHRwFgNmIkfeq8Vsy5kihBVvFT00a3xyHudYbMYFrpW/PdZuKc+F6nyBaWid5SZO3gWgEohRJUQwgHgVQDX6vrcAWCVEOIcAAghTkd3mEwiUHaiCf1y0jFQLoN3SLfAGghFiCrCPdy8NLEgI81iaFZS2mJZgINJPIb0Tj3vKDPCfRiA46r9GrlNTRGAIiLaSkSfEdFCoxMR0Z1EVEpEpfX19ZGNmIkbvTLS0O5weTM0mq2FroR058nCvbnTJ9yXTB0a3UGaJMNmhcPl8ZuDwyvcWXPvCTz/rZkoGtQLL982K3TnJCNa78M2AOMAXApgOIAtRHS+EKJR3UkIsRrAagAoLi5OzbCwFKbT6cbkob1hsxAaWu2GNmsjahulcnB5Bpr7TxeMj/5ATeAr1uDReMYobxlslukZXDl5MK6cnJqVqMzcwbUARqj2h8ttamoArBFCOIUQRwAchCTsmRSiw+lGZpoF102XXtwG5WWYOu6UnHqgv+xD3KwS7n1z0gyPiTWKQNebZlhzZ1IFM3fwdgDjiKiQiNIBLAWwRtfnX5C0dhDRAEhmmiowKUWHw42sNCuWXjACRYN6oW8YxYT75aQjL1M2y8jC/eNfXIbczPgId6MyawAvqDKpQ8g7WAjhArAMwHoA+wG8LoQoI6KHiWiJ3G09gDNEVA7gQwA/E0KcidWgmfhQfrIZ6TYLiAi9Mmx+Lo2BsLs8yM20eQsh7JEjQ/vnmNP8Y4Gisf975wnD9vQeWJ2ISS1M2dyFECUASnRtK1TbAsC98h+TomSlWeFyS0sldpdHU4czGJ1ONzJtVq8HioUIbiH8qi11J0ph7kdL9uOOuaO97Q6XB1YLwcbCnUly+A5mTON0ezBWjtwcPyjXdGGD9WV1OH6u3WvqcHsE8nPjp7UDwIh+2d7tjeV1ON0sLfo63B7W2pmUgO9ixhQutwcuj/Bq34H8xI2OA4B22V6vEO+ydZeOl4pUzBk7ALe/VIqb//czAIDd6WZ7O5MS8F3MmEIR5Jmy3VwK3w+tuXeo+lgt5A35zoxzIeLRA6Ssj+MH5wIAqhraAMiaOwt3JgXgu5gxxU659mn5SSlNb2aa1ZTm3qF7AOyRF2GjkaO9KxAR0m0W/OegFEynxDLZnalVJJnpufBdzJiiZN9JAMDWSskJKjPNAofLEzK/TKdDegD86PKxmvZfL5kcg1GGR4bN4pej3s6aO5Mi8F3MmOJIvWS2yMuUNO40edExVH6ZM21SkrAJg/M07VNH9InyCMPHKBOgw8ULqkxqwHcxYwqlnumD10wCAJTLVZTueW1X0OOaZQFqs0pp9369ZDKuiVM+GTPYXR5kcKEOJgXgGqqMKZQApAvlwhqKK+H+k8bFrRVa5CRhI+X+t3y5ALd8uSBGo+w6LZ1OZLDmzqQAfBczpuh0uEHkC9ufOCTX1HEfHpAWLPslQW3Kc20OVDe0wekxlxCNYRIZFu6MKTrkKFMiybxiNlWvovEPyjMugh1PLinK1+yfbXcgJ8PmzTvPMMkMC/cehNPtwf1v70VtY0fYxyp5ZRQUIQ8AdXJ0pxGdTjeG9UnMQghFg7R1UjscbthdnpQs3MD0PFi49yA+P3IWf992DMvf2hP2sVlptoBBSxv31wU8TlqgTMzbTO/y2Ol0o9PpZj93JiXguzhFEUJgQ3kdNu2vQ9mJJrlN+iyYph2Io2faNAWF1bQGKC5cc64dZ1sdCVuyLt0qjetLsidQc6fTr3gHwyQrLNxTlNe2H8cdL5Xitr+VYvGzHwMADp1uAQAckzMihsOxs+1+EamzR0tC8bfvHTA8Zs7jH+LTqjNxTzUQCMWypNjYy2qb4XBxhCqTGvBdnKLsO+Gfa90iS7NpEQQQCQDnDdNq7v/z9Rmmjk10YVmYL+WZ6ZQLd7DmzqQCpn51RLSQiCqIqJKIlht8fisR1RPRLvnv9ugPlQkHAmn2C5av9drMP6s6G9a5hBBwuj0YoHNnzM0MHCZxWmX6Cfd63cUpeYxnWx2wEPCB7LaZ6A8jhjFDyCAmIrICWAVgPqRaqduJaI0QolzX9TUhxLIYjJGJACL/NqWgdbi5U+wuD4SAX+Sm2pbucns0BS4O1gVPS5AIvLb9OADgjR01AHwBV72CPLQYJlkw8yufBaBSCFElhHAAeBXAtbEdFtNV3AYJvZR6oUIET/alR9H4jXKwr7haSkfQrFtUDeZBk4iMG9gLTe2ScFfSATNMMmNGuA8DcFy1XyO36fkqEe0hojeJaITRiYjoTiIqJaLS+vr6CIbLmMWo8LSiuTvdwlD4B0IxXzjd/pGbVov0inCwrkXT/tdPqk2fP1788qqJAID7Fk1ARpoFLXbpAcU2dyYViJZx8R0ABUKIKQA2APibUSchxGohRLEQojg/P9+oCxMlFLvxgd8s9KbbbbP7tGuzJfIAKcc5AIzTBf0AvkXWYLndf3bleNPX6k5um1OI6pWL8d1LxiBTZWKKZ21XhokWZoR7LQC1Jj5cbvMihDgjhLDLuy8AmBmd4TGRoi6qoSx8vlbqewEzU2gDkGzp167aKp3LwF9dcXPscPgeFvtqtZ464ZqB4oFaW89m4c6kAGaE+3YA44iokIjSASwFsEbdgYiGqHaXANgfvSEykaD2ZJkz1v8tyazmrralG0WaKnZ49fneL9fa28OwAMUNtYfM4ATMg8Mw4RJSuAshXACWAVgPSWi/LoQoI6KHiWiJ3O0uIiojot0A7gJwa6wGzJjD7vJgtOy/XWiwQGhUqMKIs21277aRkFZMGM9vqVKd26npE459P14omnuGzaLJm8MwyYopny8hRAmAEl3bCtX2fQDui+7QmK5gd3q8ZhSjCNEzrXYAodP2HmnwRbOOyfe3uffNlnzf1XndlURhNxYPx5aDDbhi4sCwxh4P2hzSw86suYphEh126E1R7C6314xipIm22M1p7u0OXz+jnOxGniWKH/3PF07AE1/LMHWdeHNBQT9srmAPLiZ14FC8FMUeIEdK8ai+AIDvvrwDX/vjJyHPs+1I+NGlindNMkV6siWGSTWS59fHhIXd6TbMxvjY9ed7t0uPngt5njTZj33r8ssD9pk/aRAAybMG8AVLJaO/+FemG4VwMEzywcI9RQmkuffKCM8SV3aiGUN7ZwYtuHFBQV/vNZV/LQTYLMmnDg/MTQ4zEsOEgm3uKYpUJMOnOf94XhEEBPJVwqugf3bI82SmWUPa55U3BLvLg5wM5cFiZa8ThokjrLmnIE3tTpxq6kSmSnO/e9443DOvCGlWC6pXLkbhgBzDuqYej8Du44349PAZCCFgd7lx/jDjIh0KyhuCYo6pqm9Dsint3iyaSTZuhgkEa+4pyNSH3wcAOAxywSgcaWjDkYY2nGrqxODePiH/6vbj+OXbewEAl43PR6fTY5inRo3ileNwedDucCVd0jAAGDtQcvOcNCQvziNhmOjAmnsKY6SZ6znRpC2Wvb3a5x3zYUW9qZqiSrk6u8uDdof5nDWJxBUTBuKdZXNwzZSh8R4Kw0QFFu4pxtEzbd5tJcAoGA0tds2+vgTfodOt2BHCq0bJDLlp/2k4kjQIyGIhnD+8NyzJZk9imACwcE8x1IK4WZcGwAiLbtFzrEEU6mndA0BPU4cDAPD4ugNhZZtkGCZ2sHBPMdTh8x4TOV0aWu14e6dUieiDA3X49+7asBdDnW7fdY6eCb/4NsMw0YcXVFMMteY8ZXifgP1yM2xosbuw/J/S4mlTuxMPvaOvnGgOdeHsb/91e0TnYBgmurDmnmIo1Za2/OwyLJ4yJGC/9+65WLO/41ijZr9IVZij8tFFQa85bUQfXDxugKbtz7cUmxkuwzAxgoV7EvJJZQMKlq/Ff734OQBg9ZbD+MHfdwDwae7D+waOKAWA3Ayte+PWygbNvtoWry58bZZwi3AzDBNd2CyThDy+vgIAsOWglMXwsZID3s/sLg/SrZaQXh/6whtn2xze7f456ThwqkV/SFA+OqR9OFxQ0C+s4xmGiS6sXiUh7ap0APqi1X/6z+GgwUsKwXzXb541MvLBARjVPzspk4YxTCphSrgT0UIiqiCiSiJaHqTfV4lIEBEbXGPIpKG+KMqSvSe92w2twV0W1QTL+zJjVB/v9oTBoQt6AMA988Z5t9ljhmHiT0jhTkRWAKsALAIwCcDNRDTJoF8ugLsBbIv2IHsaoQpK5/fyJf860+ozp5xuNi/cA7Hx3ktw+YRB3v0bi0cE6e3jnnlFXb42wzDRw4zmPgtApRCiSgjhAPAqgGsN+v0GwOMAOqM4vh7Hix8fwQWPbgwaDPTCx0cM22/7W9fdEPXVltR5ZxiGSR7MCPdhAI6r9mvkNi9ENAPACCHE2mAnIqI7iaiUiErr67mkmREr1x1AQ6sD59odoTvDF/oPACebpOfqjy4fa+rY3Ez/9XRFuL//47m4dHw+Fp032NS5AODJr00x3ZdhmNjS5QVVIrIAeArAT0L1FUKsFkIUCyGK8/Pzu3rplKKl04mnNhz05mYp2XvK1HGKx4yayUODp+hVUOzpk2Ub/q+XTPZ+VjQoF3/99qywcrIvDONBwDBMbDEj3GsBqA2vw+U2hVwA5wHYTETVAGYDWMOLquGxvqwOz2465N3/zbvG0aItunwxmw6c9uuTZ6CRG/HjeUUYN7AXHrx6ErLSrFgweVDog4KQFoE/PMMwscGMFNgOYBwRFUIS6ksBfF35UAjRBMAbnkhEmwH8VAhRGt2hpjZKoYtQdASwxSvpBACgYECOqXN9eewAbLj3EgDA/t8sNHVMMNJZuDNMwhDy1yiEcAFYBmA9gP0AXhdClBHRw0S0JNYD7CmQQQkgo8RfdqexD7u6FF5WnHzMlcApLnjBMPHH1Pu7EKIEQImubUWAvpd2fVg9D4+B+6PD7UGmRSuoFQ3/G18aib9vO2Z4rqz0+AUQrb1rDob3CV2blWGY2MLpBxIEoyIXHQ63X6SnkhhsfJDgolCVk2KJ2cVchmFiCxtJEwS7kXA3sK8r/u+9s3yJv+YWaT2PwvFwYRgmNWHhngAIIbCuzN/18bOqM35tdXIUah9VCb2XvjPLu733oQUxGCHDMMkGC/cEYNP+09h9vNGv/d7Xd/ulIvjhK18A8Lk75ujs65ywi2EYgIV7QlDV0KrZf/qmqd5tI3MNAAzrm4XNP73U68qowL7mDMMALNzjzl+3HtHkYweAnHTfOveEB9fBbeAS2SvDhoIBORjaJ3hRDoZheibsLRNn1HVLF04ejOF9s3DJeO0C6anmTgzTCXG9L/tb378Qe2qaYjdQhmGSChbuceRkU4dm/0/fmmnYr8PhC1AaPSAHk4bm+XnEzBzVDzNHcfUjhmEk2CwTR77+v+ZS31ee9tnk7S4P1ydlGCYkLCXiyJGGNu92xSPa3C4HH1nk3Xa6fTZ3u8uDDBt7xDAMExw2y8SB35bsx/NbqjRteoGt1s7VhTscLndcI1AZhkkOWErEAb1gf+X2Lxn2U9rV7pAOt4eFO8MwIWEpkQB8eewAw/bJw6Q8LW+USoWwhBBsc2cYxhQsJboZp9s4KMmIzDTpv2d3TRNaOp1weQSE4LzpDMOEhm3u3Yw+4rR65eKAfdVCvNXu8ro/ZqSxcGcYJjimpAQRLSSiCiKqJKLlBp9/j4j2EtEuIvqYiCZFf6ipgT1AJSUj1L7sFadavGmBWXNnGCYUIaUEEVkBrAKwCMAkADcbCO9XhBDnCyGmAXgCUsFsxgBFc7922lC8+6M5po9rtbu8hToyODkYwzAhMKMCzgJQKYSoEkI4ALwK4Fp1ByFEs2o3B4B/MhQGgE+4XzZ+IM4bZr6wxbJXdqLiVAsA1twZhgmNGSkxDMBx1X6N3KaBiH5IRIchae53GZ2IiO4kolIiKq2vr49kvEmPV/s26fFy25xC7/bP3twDADjb5oj+wBiGSSmipgIKIVYJIcYA+AWABwL0WS2EKBZCFOfn5xt1SXma2p0AzC+KPni1zwJW3yIV6nAZZIlkGIZRY0bC1AIYodofLrcF4lUA13VhTCnNTas/A+AfkRoOI/pxml+GYYJjRrhvBzCOiAqJKB3AUgBr1B2IaJxqdzGAQ9EbYmoSTiCSPoJ10XlDoj0chmFSjJB+7kIIFxEtA7AegBXAi0KIMiJ6GECpEGINgGVENA+AE8A5ALfEctCpQDglrCcMydPsWy1cAJthmOCYCmISQpQAKNG1rVBt3x3lcaU8E3UCOxj6whwMwzCh4AjVbmZwXibmFg1ATob5r54ThTEMEy4sNboZu8sd9mKqhc0wDMOECQv3bqbT6fEmBAuHgbkZAID/u804PTDDMIwaFu7dSF1zJzqc7ohS9o7OzwEA9MlOi/awGIZJQdjm3o0o6QOG9ckO+9inb5qG17fXYPJQ8wuxDMP0XFi4dwMtnU7c+pftmFXYDwBwfhg5ZRSG9M7C3fPGhe7IMAwDFu7dwhPrKrDj6DnsOHoOAOdjZxgm9rCU6QZ2HW/U7LNrI8MwsYalTDcgdBmQu5JXhmEYxgws3LuB2YX9NfuRuEIyDMOEA0uZKHOm1Y6rnvkIG8vrvG1lJ5o1fVhzZxgm1rBwjzJPrq9A+clm3P5SqbfN5dEWxY7Ez51hGCYcWMpEGb2WDgAOt9bmzlkdGYaJNSzco8ze2ibv9tEzbQCAToc7XsNhGKaHwn7uMWTX8UaM6p+DTpcbl47Px/C+WbjqfC60wTBM7DGluRPRQiKqIKJKIlpu8Pm9RFRORHuIaBMRjYr+UBObTw+fwf1v79W07T7eBCEEjp5px5DemXjkuvPx5TED4jRChmF6EiGFOxFZAawCsAjAJAA3E9EkXbedAIqFEFMAvAngiWgPNNH59Ttl+Pu2Y5o2h9uN5k4XAMDp5qLWDMN0H2Y091kAKoUQVUIIB6QC2NeqOwghPhRCtMu7n0Eqot2jaJGFOAAsOm8w8nMzsKemCYfrWwEA00f2idPIGIbpiZixuQ8DcFy1XwMgWFLx2wC815VBJSPZ6T7f9QybBS2dTtS32HH9c58A4FJ5DMN0L1H1liGibwIoBvBkgM/vJKJSIiqtr6+P5qXjjjrPeobNioG5mZrPWbgzDNOdmBHutQBGqPaHy20aiGgegPsBLBFC2I1OJIRYLYQoFkIU5+fnRzLehOOlT6vx7b98ju3V57xtJ5o6MKq/Nmd7Jgt3hmG6ETPCfTuAcURUSETpAJYCWKPuQETTATwPSbCfjv4wE5cV/y7DhxXat5CPDjVg2WVjNW1qsw3DMEysCSnchRAuAMsArAewH8DrQogyInqYiJbI3Z4E0AvAG0S0i4jWBDhdj+FLo/trTDX9ctLjOBqGYXoapoKYhBAlAEp0bStU2/OiPK6k5oaZkrPQddOG4a+fVAMA+vfKiOOIGIbpaXCEahRZPGUI7p1fhIL+UjHrB6+ehB9cOgYOt4c1d4ZhuhUW7l3A5dZmexySl4kx+b28+1YLYWBepv4whmGYmMOJw7rAuXand3v6yD64Y+7oOI6GYRjGB2vuEdLU7sTyt/YAAP77hqn46sweF5TLMEwCw5p7hPzu/QpsOiB5fQ7pzaYXhmESCxbuEaLO2z6rsF8cR8IwDOMPC/cIIVUxJZuVv0aGYRILlkoMwzApCAv3CNknm2Uy0/grZBgm8WDJFCH9c6SI0x/PK4rzSBiGYfxh4R4hLo8H104biu9eMibeQ2EYhvGDhXsEHK5vRUOrA1YLhe7MMAwTB1i4R8Cppk4AwBUTBsV5JAzDMMawcA/A4+sOYNrD76PydIumXQiB37xbDgAY2S/b6FCGYZi4w8I9AH/cfBiN7U48u6lS036wrhUHTkkCnz1lGIZJVDi3jA6PR2CPKvr06Nl2AECb3YXTLXbsOu4rp8el8xiGSVRMCXciWgjgGQBWAC8IIVbqPp8L4PcApgBYKoR4M8rj7DY27K/Dd1/e4d0vkwX9Xf/Y6c0lo5CXlQaGYZhEJKRdgYisAFYBWARgEoCbiWiSrtsxALcCeCXaA+xuzrQ6NPs5GdLzTy/Y/3LrBejNwp1hmATFjNF4FoBKIUSVEMIB4FUA16o7CCGqhRB7AHiMTpBMdDrdAIBhfbIAAE0dThQsX+vXb2ZB324dF8MwTDiYEe7DABxX7dfIbWFDRHcSUSkRldbX10dyipjTIQv3Z5ZOQ066sU391i8XIDeDlysYhklcutXdQwixWghRLIQozs/P785Lm+ZEYwcAYOaovth2v3/d79xMGx5aMhlEHMDEMEziYka41wIYodofLrelJE0dUuk8IkK2gTdMS6eru4fEMAwTNmaE+3YA44iokIjSASwFsCa2w4ofQviCkyyq9ALPfWMGAGD2aC7MwTBM4hNSuAshXACWAVgPYD+A14UQZUT0MBEtAQAiuoCIagDcAOB5IiqL5aBjgdsjcPnvNmPt3pPok+3vBaO0DcrjknoMwyQ+plYFhRAlAEp0bStU29shmWuSlpZOJ6oa2gAA00b08bb/6Zsz0NjuxIWj++OBxRNxQ/GIAGdgGIZJHNjlQ+Yfn/scghaeN1i1PcS7ffvFo7t1TAzDMJHCyVEAdDjceHzdAe/+qP45cRwNwzBM1+lRmrvd5YaVCEQEp9sDq4WQZrWgpVPykHnomkn41oUFnKedYZikp8cI96c3HMQzmw4BAAblZaCu2Y4MmwXr7pnrFe7ZGTYW7AzDpAQ9xiyjCHYAqGu2o2hQL9hdHhw/245W2Xd9QK/0eA2PYRgmqvQI4f63T6r92maMlHLDvF9+Cl9/YRsAYGAuuzkyDJMapLxwF0Lg1+9o3e6H9cnCBQVSMNL/fXbM2z6CKysxDJMipIzNvc3u8qbn9XgEqs+0wSMkU4tHAMsXTcD3Lhnj7V8r55BRwyl8GYZJFVJCuH9+5CxufP5T/O07s3BJUT5e3HoEj6zdD8BnR7fpFkpzM1Ni6gzDMIakhFnmi2NS6butlQ0AJK08K82KCYNz0SAX35gxSpt/PS8zDWlWn8Df/asF3TRahmGY2JMSwv217VJ06eotVdhccRp/2VqN3EwbRvX32dD7Zvt7wkwckgcA+NHlY9kkwzBMSpESwl1J0wsAj6+rAABcMXEQ5ozz5YzPMkjfe+20YZgxsg/mFiVmbnmGYZhISTrDsxACu443avKqdzrduLF4OF4vrcHJJmmh9MGrJyLNasGD/9oHwFi43zanELfNKeyegTMMw3QjSSfcK+pa8JXnPvFrH9U/B70ybGhsd6JXhg1ZaVZNtaSsACXzGIZhUpGkE+4NLdIC6W+uOw+ThuQCkKomnTe0N66dNhR1zZ0YlJfpFeyfLL8cbo9Aui0lLFAMwzCmMCXciWghgGcAWAG8IIRYqfs8A8BLAGYCOAPgJiFEdXSHKrF27wkAwPQRfXDesN6az4b3zcbwvtpApKF9smIxDIZhmIQmpHAnIiuAVQDmA6gBsJ2I1gghylXdbgNwTggxloiWAngcwE2xGPBl4wcCIBQNyo3F6RmGYVICM5r7LACVQogqACCiVwFcC0At3K8F8JC8/SaA/yEiEkKIKI4VALBg8mAsmDw4dEeGYZgejBlD9DAAx1X7NXKbYR+55moTgP76ExHRnURUSkSl9fX1kY2YYRiGCUm3rjIKIVYLIYqFEMX5+exbzjAMEyvMCPdaAOqq0MPlNsM+RGQD0BvSwirDMAwTB8wI9+0AxhFRIRGlA1gKYI2uzxoAt8jbXwPwQSzs7QzDMIw5Qi6oCiFcRLQMwHpIrpAvCiHKiOhhAKVCiDUA/gzgZSKqBHAW0gOAYRiGiROm/NyFECUASnRtK1TbnQBuiO7QGIZhmEjhsE2GYZgUhIU7wzBMCkLxWvckonoARyM8fACAhigOJ1noqfMGeu7ced49CzPzHiWECOlLHjfh3hWIqFQIURzvcXQ3PXXeQM+dO8+7ZxHNebNZhmEYJgVh4c4wDJOCJKtwXx3vAcSJnjpvoOfOnefds4javJPS5s4wDMMEJ1k1d4ZhGCYILNwZhmFSkKQT7kS0kIgqiKiSiJbHezxdhYheJKLTRLRP1daPiDYQ0SH5375yOxHRs/Lc9xDRDNUxt8j9DxHRLUbXSiSIaAQRfUhE5URURkR3y+0pPXciyiSiz4lotzzvX8vthUS0TZ7fa3KSPhBRhrxfKX9eoDrXfXJ7BRFdGacphQURWYloJxG9K++n/LyJqJqI9hLRLiIqldtif58LIZLmD1LissMARgNIB7AbwKR4j6uLc5oLYAaAfaq2JwAsl7eXA3hc3r4KwHsACMBsANvk9n4AquR/+8rbfeM9txDzHgJghrydC+AggEmpPnd5/L3k7TQA2+T5vA5gqdz+JwDfl7d/AOBP8vZSAK/J25Pk+z8DQKH8u7DGe34m5n8vgFcAvCvvp/y8AVQDGKBri/l9HveJh/klXQhgvWr/PgD3xXtcUZhXgU64VwAYIm8PAVAhbz8P4GZ9PwA3A3he1a7plwx/AP4NqU5vj5k7gGwAXwD4EqSoRJvc7r3PIWVjvVDetsn9SH/vq/sl6h+kWhCbAFwO4F15Hj1h3kbCPeb3ebKZZcyU/EsFBgkhTsrbpwAMkrcDzT+pvxf5lXs6JC025ecumyZ2ATgNYAMk7bNRSCUqAe0cApWwTLp5A/g9gJ8D8Mj7/dEz5i0AvE9EO4joTrkt5ve5qZS/TPwQQggiSll/VSLqBeAtAPcIIZqJyPtZqs5dCOEGMI2I+gB4G8CE+I4o9hDR1QBOCyF2ENGlcR5OdzNHCFFLRAMBbCCiA+oPY3WfJ5vmbqbkXypQR0RDAED+97TcHmj+Sfm9EFEaJMH+dyHEP+XmHjF3ABBCNAL4EJI5og9JJSoB7RwClbBMtnlfBGAJEVUDeBWSaeYZpP68IYSolf89DelhPgvdcJ8nm3A3U/IvFVCXLbwFkj1aaf8veUV9NoAm+dVuPYAFRNRXXnVfILclLCSp6H8GsF8I8ZTqo5SeOxHlyxo7iCgL0jrDfkhC/mtyN/28jUpYrgGwVPYqKQQwDsDn3TKJCBBC3CeEGC6EKID0u/1ACPENpPi8iSiHiHKVbUj35z50x30e78WGCBYnroLkWXEYwP3xHk8U5vMPACcBOCHZ0W6DZFvcBOAQgI0A+sl9CcAqee57ARSrzvMdAJXy37fjPS8T854DyRa5B8Au+e+qVJ87gCkAdsrz3gdghdw+GpKQqgTwBoAMuT1T3q+UPx+tOtf98vdRAWBRvOcWxndwKXzeMik9b3l+u+W/MkVmdcd9zukHGIZhUpBkM8swDMMwJmDhzjAMk4KwcGcYhklBWLgzDMOkICzcGYZhUhAW7gzDMCkIC3eGYZgU5P8BPbNnuV7ldpIAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# Plot the moving average of total undiscounted reward across episodes here\n",
        "rm=movingaverage(reward_of_episodes, 100)\n",
        "plot(rm)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "sexual-aviation",
      "metadata": {
        "id": "sexual-aviation"
      },
      "source": [
        "#  SARSA\n",
        "\n",
        "- In the following you will implement SARSA with epsilon-greedy exploration.  \n",
        "    - You can experiment with fixed and adaptive learning rate and epsilon\n",
        "    - Report your best result and mention the differences you observed in answer to final questions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 119,
      "id": "careful-quick",
      "metadata": {
        "id": "careful-quick"
      },
      "outputs": [],
      "source": [
        "#initialize episodic structure - again try to maintain these two parameters as they are for your final submission\n",
        "num_episodes=5000;\n",
        "episode_max_length=100;\n",
        "\n",
        "#initialize discount factor, learning rate\n",
        "gamma=0.95\n",
        "learnRate=0.6\n",
        "\n",
        "#create Q table\n",
        "Q=np.zeros([env.observation_space.n,env.action_space.n]) #matrix Q[s,a]\n",
        "\n",
        "reward_of_episodes = [] # list containing total reward of each episode\n",
        "\n",
        "# exponential decay for epsilon\n",
        "min_epsilon = 0\n",
        "max_epsilon = 1.0\n",
        "decay_rate = 0.003\n",
        "\n",
        "#epsilon = 0.175\n",
        "\n",
        "#execute in episodes\n",
        "for episode in range(num_episodes):\n",
        "    \n",
        "    # reset the environment at the beginning of an episode \n",
        "    # and other possible initializations here\n",
        "\n",
        "    # state = env...\n",
        "    # ....\n",
        "    epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode)\n",
        "\n",
        "    state = env.reset()\n",
        "    state = state[0]\n",
        "    total_reward = 0.0\n",
        "\n",
        "    if np.random.random() > epsilon:\n",
        "        action = np.argmax(Q[state, :])\n",
        "    else:\n",
        "        action = env.action_space.sample()\n",
        "\n",
        "    for t in range(episode_max_length):\n",
        "        # take an action `action` and observe the next state ...\n",
        "        # state2, reward, done, info = ....\n",
        "        \n",
        "        # what is the next action?\n",
        "        # update Q[state,action]\n",
        "        \n",
        "        # other updates \n",
        "        # stop if we have reached terminal state \n",
        "        \n",
        "        newstate, reward, finished, _, _ = env.step(action)\n",
        "        if np.random.rand() > epsilon:\n",
        "            newaction = np.argmax(Q[newstate])\n",
        "        else:\n",
        "            newaction = env.action_space.sample()\n",
        "        \n",
        "        Q[state, action] += learnRate * (reward + gamma * Q[newstate, newaction] - Q[state, action])\n",
        "        total_reward += reward\n",
        "\n",
        "        if finished:\n",
        "            break\n",
        "        state, action = newstate, newaction\n",
        "    reward_of_episodes.append(total_reward)\n",
        "    # Other possible code here"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 120,
      "id": "complete-absence",
      "metadata": {
        "id": "complete-absence",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "outputId": "3190f00f-b0ab-439e-dbd3-b4adde842c9d"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f4b79bdcb20>]"
            ]
          },
          "metadata": {},
          "execution_count": 120
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAA7aklEQVR4nO2dd3gc1dX/v3e7umRZkm3Jttxwx73SCc0moYRmEwgtcV4SkhD45Y15Qwg1MRBIJRATWgg9EGKwqQbbobj3Ksu2ZMtNxerS9vv7Y8rOzM5skXZ3pNX5PI8ez9y5M3PvevfMmXNPYZxzEARBEOmFxewBEARBEImHhDtBEEQaQsKdIAgiDSHhThAEkYaQcCcIgkhDbGbduH///ry8vNys2xMEQfRKNm3aVM85L4rWzzThXl5ejo0bN5p1e4IgiF4JY6w6ln5kliEIgkhDSLgTBEGkISTcCYIg0hAS7gRBEGkICXeCIIg0hIQ7QRBEGkLCnSAIIg0h4U4QEdhU3Yjdx1rMHgZBxI1pQUwE0Ru46umvAABVSy41eSQEER+kuROEAVTIhkgEta1ueP3BlN+XhDtBGLD5cKPZQyB6OYEgx8xHVuJ//7Ut5fcm4U4QBhxtcps9BKKX0+bxAwDe23485fcmmztB6HD/sl148asqef9oUydK8zPMGxDR6zj/d6twsL4dgKDBpxrS3AlCB6VgB4DV++rMGQjRa5EEu1mQcCcIBU9+UoFl246Ftdc0dpgwGiKdeOHLQym9Hwl3glDwp5X78ZPXtsj7F44rAQC47FazhkSkCQ+8tzul9yPhThAi/oDaXe3KKaVYeuM0AIJGTxCxwjkHY+aOgYQ7QYi4dXyRmeIX6guk3leZ6J14/EHohUmkcmGVhDvRI/AFgqYLT7cvoNq/bPIg1X6HV32cIIyobfEAAH71zXGqdu13LJmQcCd6BNMe+gQzHvnU1DF8uPOEaj/XpfYU/n9vpT4Qheh9cM5x9uOfAwAcNrWI7SThTvQ1Wtx+NHX4TB2DVriPGZALALh5bjkA4JPdJ1M9JKIX4vaF3kBnlBdg868uVBzrYcKdMXYJY2wfY6ySMbZY5/gQxtjnjLEtjLHtjLH5iR8q0RcoX7wcHV6/KfeualD7JWc6BA+ZSYPzzBgO0UvZciSUtqJflgP9shz444LJAIDDDalzqY0q3BljVgBPAZgHYByAhYyxcZpu9wJ4k3M+BcACAH9N9ECJvsPm6iZT7nvV1DLVvrSYeu5pxQCAM0f2T/mYiN5HVb0gwM8a1R9F2U4AQGGW8G+rJ3WKSyya+0wAlZzzg5xzL4DXAVyu6cMB5IrbeQDCo0AIIkaqT5kT2RcIclgt4f5rBVkOlOZnoCTXZcKoCC1PfV6Jxz/aa0pIfyxUnGwFADx9wzRZQRiYL3x3UmmWiSW3TCmAI4r9GgCzNH3uB/AxY+zHALIAXKB3IcbYIgCLAGDIkCHxjpVIU7ReMmZl2u30BeCyWXDDnKFodas1LKfdArefvGXMprnTh8c/2gcAuHJKGUYWZ5s8ImOyHKHAN6e4sOpJYerfRC2oLgTwIue8DMB8AC8zxsKuzTlfyjmfzjmfXlRUlKBbE72dlk71QmqVSTk5Nh9uRLs3gHvmjcVvrpyoOuayWeHxkZ97Vzja1IlDCfo/7VS4oz764V6caO55mTvdvgCKc5yqGAmnTRD0nh62oHoUwGDFfpnYpuQ2AG8CAOf8awAuAGSgJGJi34lW1f6GqlOmjGPL4SbDY067BR7S3LvEgqVf47I/f5GQayldCT/ZfRI/fm1zQq6bSDq8AXkxXsJl75ma+wYAoxhjwxhjDggLpss0fQ4D+AYAMMbGQhDulEaPiAnJ3PHybTMxdUg+/CbYUrWpB7Q4bZaU/jC7gtlBYEYcOdWZsIVErSeVVjHQw+sPxl0JyRcIRv1OGNHY4Q3LRSRr7in8DkW1uXPO/YyxOwB8BMAK4HnO+S7G2IMANnLOlwG4G8CzjLGfQVhcvZlTjTIiRjq9whe+JNeFIAd2HWvBih3HMX/iwJSNob7NG/G402ZFU6e5fviRONrUiTOWfAagZ9V7TXRsQE1jp2rforMALvHd59djTUVIxxxRlIWVd58b9R7li5fL2yvvPgcjimK361fVt+O/++sxoTRX1W63MjDWA/3cOecrOOencc5HcM4fEdvuEwU7OOe7OedncM4ncc4nc84/TuagifRC0sYy7FYcqGsDALyx4UikUxJOsyi4b5ozVPe402ZJqb00XqobzM0dbkSix+UPCDrjjbOF/6dIBVSUgh0ADtTFP5Yv9tfH1V96+Hx7Srhbrctm7XFmGYJIKq+LgjzDYUV5YZYpY5A0qnNG6y/07zrWgr0xmAC6y8Pv78a7W7RLWpFZf+gUrn92nbxf3+ZJ9LC6jCTMIijYMfHPtdVYuHQtXllXDQBYdPZwXDpxYMI14YffV6fl/fWyXbjnne34qjI2IX/Dc8L/w8xh/cKOdfoCWLrmYMqC9Ei4E6Yjecv0y3TIC0+prhYvCQmXTT9v+9EmQSNL9rj+/sUh3PnG1rjO+XiXOm1CLHboVCEJ9yAXcq50lcc+3IuvDzbgqwMNAIT8+k67RRXqnwj+/kV4QY3X1h/B9X9fp9PbmAyHcf7/g114g+gKJNwJ0+n0BfDtKaWwWBgyHMIyUKp9yiUvDGeUohzJTPykXBCNZ3E0oBGaqX4wRkLpYdQdk0SLJu7AZbfAZbfKD91YGF2SA7cvgGv/9jXakxApuqk6lHYgI8L3yJuihW8S7oTpdHoDsqbz/bOGAQCGpdg8I2mA0puDlpJcIXy8M4lpf59ZdUDeXrrmYMznaR8EPcmrRxkbkMg4gUyHDT5xno3t4YvhLe7wxW+3P4ArnvoS6w+dwsV/WJOwsUhc9fRX8na/LEfY8Rwxy2iq4iVIuBOm09DulTWds0YVIcdp0/1xKml1+7DrWHNC7n+0qRMnW4RgGCON6555Y+X7JgtlQeXK2raYz9te04y8DDs+vetsAEBzZ2TPn1Si1FIT9TY2pF8mrBaGWcMLAQAnWsIDmZRaNABcN30waho75XUTrdcNoDYbaYPYnLbYReW9l47VLcv40q0zAaTurZSEO2Eqkias1D79QY5P99RGPG/i/R/j0j990W1hW1XfjjOWfIZfL9sFAMh26nsH52YI7Sv3Rh5Xd/i3YiF1dUVsYSKHGzqwvaYZzZ0+OKyCQPnF2zuSMr6uoNRSuxpNGtTEPUg5ZYpyhLepv60+oDre2O7FLS9skPdL8zPgCwRVuWgmlYVn+lx7MBQ8N2ZgjupYPHmF9PITAaH1nFR5XZFwJ0yl1SMI58H9MuW2K6YIFZBisR23ddN2ekxjs+0vZvHTcubIopjHlAiynLEV5K5tDQlMm9Xkop06KG3uXa1kpdV075k/BgBwlpilUxv01tgRenP5w3WT8eGdZ2GG6L0iuU5eMiE8huJ4c+i7MHVIAZb/5Ex5v7x/ZDOhcp4zysM9ZYCQyS/Ri8BGkHAnTEXS7HIz7HKbFDQSy+Kl5PecKIyCYhw2C6wWhhU7jif0fhKSf79ErI4lSs8Lrcb45oYjOP+JVXFFWjZ3+lC+eDnKFy8P05i7gvJhuPDZtV26htbXfGg/QdBaLAzjBuaq3CGrG9px/hOr5f0rppQix2WXF1ClBVg9F0rt5zd+UEi7X1NRh/H3fYhv/fkLPPDeLrn9nc01KF+8HC98WSW35brs0ENarE9VIBMJd8JUJI1HaaPMFD1mYlm87G6+l3jEV0GmPao3TVc5UNs14S55yvxxwWTZdDBuoBAd+av/7MTBuvawDJeR2H8y5EaZiIVZjz+IiaXdK3aitY8r8nHBpXGHNIqI3ViltsHr2b0ls81r358tt/31O1Pl7XZvADuONqsE+V1vCqUXl3ywV24zcoN0pTgzJAl3wlRkLxXFglWGQ9g+1qzv5qZc+Np2pHuLqkcVgkMSikbMGlaIHTVN3bqfEdq3lFh9wqUH4DmnCWajuSMKkemwor7NIwuRKQ99gp1Hm1G+eDl+9Opm3L9sl+FDUSl4EpEozesPqhapm+Mopcg5xx8/3Y9tms9caX5y2a2qz+7wKf1KR1matZS/rT4Y9kYgPRiGKUww8ycOVKXulfjn2mo0GASLaeumKscKQF7fSTYk3AlTkQSIUiOWXmvrW/V/PEoBdHc3i1Yr7bW7j7dE7Nvh9Rv+cLuLJKR/+23BSyPbFUuphdBDQdIWXXYr3P4AHlm+R9Xvm2JWxuXbj+PFr6qwbKt+PR3lG1QibMMefwBOhXvpZ/tizzXT6Qvg959W4D9bj8HCQlGfIxW5XjLsVpWZQ2kSkVIUAMAvLhkddn0pmlTiA7GGrtZj6oHLJ4Sde++7O1XausTE0rywwuoSykyRiTB5RYOEO2Eqepr70EJhcdVt8PqqXDADBLtnV6vyKLW+aCHyYwfmxnyffSda8WwcvupVYm3NS8YPwDdPHxhTZswOr18uXOGwCp+fy27BzqMtKs8bPXYc1X/j8SZYc/f4g/LYorGmog6d3gBW7jmJ+jaP6q3KYbPgzR/MQdWSS2FTXM9lt2LXsRZ8sOM4lnywVzXvh64ICeXiCN4u/kAQ/95SI+87NbEOV08rk9+MlLy3PfwB+dT1U1V53JUo25V1VpMFCXfCVOSwf4W2FK2wQcVJtX36rje34YUvw8PG47k/IITIR8Jps8IX4DEJ+Iv/sAaPrNiDytrYUgFIr/jZLhucMRYGWfSPTfK2JDhOtsSWV+YfX1frtitt0YnQ3L3+IJx2Cy4VM3xaDATf/pOt+O7z63HXm1tx20sbce+/d+LC34cCjYzGIvnR3/7KZjyz+kDEiNWb55brBqmtP3QKP3sj9Aao59N+wdjisDa9MRVk6S+matlek5gYjUiQcCdMJRT2H/oqSttGmrue0D/eRR9qZdWnAVF8mSWTTCR3SH8gqFoI1nP/CwR5mMeEP8hRmp8Bu9UCl92CVrcvqtulXlGTa6ersxH++4dzI15Di9urFO4BcM5x5FRHxAeaPxA09ADx+INw2qxYPE9wX6xr9ai8dxraPAgGuZyVc+uRJgDAlwfU9nC9RFwAcPnkQWFt10wr0017/OtvjcO2X18k79vEV7VahflvRnmBruZ945xy3fsr2f/IPOQYeMpIHPjNfGy77yIsnJn8MqMk3AlTkSIJlbZSSYs30tz1XCQj5fIw4kBdG97aFHodnz1cX4BIhOpgGpsrrn92Hcbe96G8r6ep/uT1LRjzqw9VbZ3egBxA5bRZ0eL247zfrYo4njEDcsLatFk1jfz2AX2XvP0Krx2PP4i3NtXgrMc+x4KlXxteZ+Gza8PmE7pGAA6rRV4TeHj5Hix+RwiyqjjZimkPf4rJD34sey1JY9J+ausP6Vfn0lY8AoyDiBhj8lshEFpvUSZq04ssldD7vCXGDsyFPQbzk9XCkJdpj3ifREHCnTAV6QcxSJGXO1oxYT0XyeJcYyFmhFJrf//HZ2LJVadH7O+MoVTaeo02rSdAl28XfOWVi2odvgBcoqCS7hMtKdY80dTxxS/Ok9tmDuuHn188GtdOL8NLt85UCZHrpg/G4nlj5Dw5esmzlAvGbl9A9uvfUGVsI450TDLLKMfxL/GBulUsa9ji9stvBtKDmzGGGeUFhteV0BOS0QSnUvZr30icBllBAeCNH8zRfVP45Gdn4/VFs3XOMBcS7oSpbDvSFJZkSVqAe/yjfbrCUfJqUHLff3bh9n9uQvni5TGnvF17sEHenlCaF1UoSD/8eKJUJbusxx/AlX/9Eh8qxj78/1bgwfd2w+sPYk1FnbyorLT5vq+zaCch2eUH5oUejIwx/Oi8kXjs6kk457Qilc/1xRNK8D/njMCdF5wmjil8HkozUnVDO1btM06DcOuLG1RVixrbvbjxuXV4W/E2JJhlLKoFc0CodvS/b2+X9yUXRunzau70RXxoSOi9sUV70Cvfbtq94dkmjcjLsOP0svyw9lElOcjLiM3WnkpIuBOmkuGwhkVQKm2eegm0JF3re2cOU7VLQj+ap4jEs/8VFmFvnlseU/9YzDJaJE20tsWDLYebVNGNAPD8l4fk4hrS24tSe7zj1S2G13b7A7BbmaEZAlDnyikrELyQpIen3kNK9TA1WPyU+EyTZ2dVRS3+u79e5Z7q8QfhsFlUHi56RFukfvt2/bWD4f3VJfBunluOa6YNjnitl26diRFFgoA/3qReq4mkuQPADbOHYNLg/Ih9egok3AnTqGnswKp9dXICKD2aNXVLg0GONRV1KM3PwL3fHKd7zubDjXh61YGIgUDKY+caVF/SIgn3eLxIjpzqAOccl/1F8DPXW/g9JJqHzhRzpZQWGJeOU+LxBQ2Li+hRkiMsGEtBQAfrhQdnc6cPT31eiWCQY++JVnmh8Vfv7gy7xqp9tYaVonyKVBAPvrcb5YuXIxDkUQUmANzzjn6ysysmD0LVkksxbai+iUZK6CZx/2XjI36fACGP0f+cMwJAuBktGCV4zGmz4n8vDveZ74mQcCdM462Nwut7pNqWv/1AHYyz54QQaCTZo39wzvCwc9YfOoVHP9wblvZVyTaFK5qR4NAiBVrFEj4+sljQKANBjsraNjRGiMy8Wwxh3yMGUc00SDylxa0JEIqGFBglpTp48D2hpNwDy3bh8Y/2YfX+Ojislog+9je/sAF3vrFV98GZ5QgJ2ucVrqndCfyK9lkzxnDhuBIUZjni8kCRHgBf7Febnd7bZmwGk+gtmntsYXAEkQQiaUmXTx6E/2w9Fqa5a3/s98wbix01zXL5NfX1w6/b1OFFU4cPTYpAqGjuaxKSOcPILCMVg771jGH45aVjMeL/VuCRFXt0++e4bHLOFykfuWQfH5DnwpPXTpLzlmjx+oN4c+MRVJxojUkrlpDMN5LwlgKn/ivWBw0GObYeacLYgbnyg0ZCuy6iV03IqDZorEFMesTy4H32u9Pjvu4EMd/NR7vUEbPSQzkS2U6brqtlT4M0d8J0rplWFtYmmUpOL81XtevZZi8cV6J7XT2T8cV/WINzf7dK9qeOh2jeMuc8vgqAsCintIP/7uOKsL56i7eSWQYIZSTUE4yvrKvGve/uxMbqxrg0d4m5I/qr9utEP29fgKO+zRMm2AGgTZN8TM9ery2FJ7HrmHC9qUPy4x5rslwGtZkbx4p5ha6ZHtle35sgzZ0wDWnxTs8Fcd6EgfjZG9swvjRX9xwl35o0CA+8tzusXSuQgFAEp+SCufW+C2Mer7ygGsXmHovP/Vkj++Mdje1aqiwEAKMH5KAox4nJg/PhCwRVPtQ7j4aEbyya+477L1Ltnzmqv26/Tl/o85o3YQA+2HkC350zFEHO8fr6I6q+Sq+aFT85C/P/9F+50LmWo03CG8IbP5iDQJCjod2LbIdQbevV9YfxtFhe8MoppWGL4ckS7g6bBSOKsmST4Eu3zgA4otrrexOkuROm4fYFUZBp1/X2MFq8vPG59WF9jaon3fLiBvx55X6UL16O+jZ1ZGRtixsWhrhc2OS0CDpmlp2KXC1GOeGVaLMU6tHU4cUnu09i1C8/ULW/vTnkahjDrZDjshuanpRvQsoQfEmTPee0IrhsVviDHP/4uko+Pus3K+VtaQH4y0p1VKnEaSVC8I8QfWtFaX4G8jLtGNwvUxUVrJcaOBmFrCWUpqYshw3FuS7DvDC9ERLuhGm4fQFDzYwxJubq1rdvKwN3XHYrbpw9FGeMLMRwTcWcJz4RTCKH6ttVVZv217Yhw26N68fsjJB+4AuFYCsThZ2e+95pJdl49XuzYopmVHqfGHn+FHdD0xxdkqNbyerH54/ED88dgaU3TsP5Y4pxSlyfUBbwlnjh5hnyA1J6SN8wW1jY/MetM3HV1DK5/qweSk+lm+aW441Fs/HpXefIbUZpnxOBcgE2FRGjqYaEO2Ea6w6dilh42GG1YPexkAlCuWAn+WxLPHTFBLzyvdkYXqS/IPbWxiOqXN9fHWiIu/CGZN/+WKcghBR1CoRC/qcNLcBzN6kX+/5x6yzMHdk/LGxez7aufKNpaBcErDYmIFJ6gUicMbIQvkAQkx74OOzYiKJs2KwWXDR+ABhjsn1az4vmLNHEMzDPJdvczx5VhKoll+Ls04rwxLWTDItXAEB+hkOeh1T0WrmomRPDG05Xkd4oAOOUBb0ZEu6Eabjskd3uPP6gSjB8sCM8MlWLnhcHALy5sQaPfqjOvx1vmmDJLKNX7UdKoWu3MkxRLBwqx79w5hBZ0/7unKEqT5BHr54Yds1/3jZL3paqEWnnt18nyCsWXDYrDtbru6BqC17cJAZ56dnqbXKqYatcrDwe18fcDBt+fP5IvPb9War2N8Rw/lvOGKZ3WkIYNzAXRTlOXKGTUiAdIOFOmEKH14+Kk22YrVhE1DKpLF/lubH5cPRw9LERkjt9Wal2l4yWKExLpLcMic2/ulAuEwhALn0HCIU4JHt8ca4LT98QKuF25ZRwj6E5Iwrlkm9SPp0DtYJAlnKcRAu6MSKSGUJrMpIqE72zWVjsLC/MDDvHabPID6B4hDtjDHdfNBqjStT/b7OGF6JqyaUxrU10FYuFYcMvL8AfFkxJ2j3MhIQ7YQpfi37pWv9pJa0ae/Ar6w5Hva5RmTU9Ys1BIxGLv7YykAfQz1oo4bRGNwtJmr/kyfItMdJ1lGi6+NF5I6NeQw+tT7py6eEbOrnLleRlhv+f1beF4gYGF4QLfyL1kHAnTEFayNPmH1dyemleXEm6AH27sBHK6vaxYLEwjB+Uq5ut8KxR/TF5cH6Yp4zkFqmXkCoWDVd6OGiLXE8eXICqJZfi4vEDYh6/kpnD1G9MUjh+UY5TZYvWI1/Xwyj0uQ/uR8K9J0DCnTAFt1z70/i1O8MRKn4sRX8CwKA846Ia8djRtdGvsRAIchxrCs8P4/EFdQW4pHnPnzAw7Jgk3KUqRXpIZolD9e1y+l0ActrerqId60DxMzVyPSzIDAn0/Mxw4a7U3ImeAQUxEaYgBcFECvjJcFhlW7NSeHyscJXTIgn3J66ZhPL+mdh/sg0vfV2NU+0enGr34qLxA1BemImnPj9g6B8fiUH5GThQF76I6fYHUKhjYnLarPj6nvNRmBUujK0WhrX3fCNiaTbpmnarBV8pqhNpbdTxovzc3759Lpo7hc93iIHW/eR1k3HLCxsACAWqX180G6X5ylTDAOfAi7fM6Na4iMRBmjthCnuPC/buSDZpKXhm0gMf42VFAE0koSxFGA4pzMS0of2wYOYQVNa24mSLB74AR3GOEzPExFwDIrwBGHGsqRPVDR2qoCVAWPA0WqQcmJdhaIIZkOeKGGXqtFnAmPCmk8jFRWmsE0pzMW1ogTwGo6CuXFfo3plOG2YPL1SZX6R13bIYM1oSyYeEO2EKkrkiktdGv2xBa23u9OHdrUK2vr9cH9mz4f7LxuOxq0/HdIWboTIYKMdlxzmnFWHJtyfKdT3jYa+4CLu6Qp1N0O03Fu7dgTEGl80Kty+A6UOFh9IfF0zu9nUlswwTC9pF8/I+vSxffou4WicXkEQkMxuRWki4E90mGOS4990dYdpsJDz+YNQAHKmggpJIdSwBQau/dvpgw8jTEUVZYIxhwcwh3RLGWtv+kVOdSYtydNkt6PQF5MXlMQNyo5wRyzWFsWrfKHJc+sLZbrVg068uRNWSS3W1e6mtK7VsieRAwp3oNseaO/HPtYdx33/CizsY4fEHovqNS9GLSrSRqbHw02+Mkre7K3xeuFmwKSsXFaVaqJ0GKW+7S4bdCrcvKOe0iVQKLlbGDcrFnOGFuON8wZVy8pB8fGNMMe66sGuFKJ64ZhLmTxxg4ElDmEFM3xLG2CWMsX2MsUrG2GKDPtcyxnYzxnYxxl5N7DCJnkyTWIhis1jwWOLT3SfhM4gY9fj1vUuU6HmEdEU7vmH20G6dr2TGMME0crQxlPNESgE8OgEatR4uuxUVJ1vl+8STw92I4hwXXls0G+eNFnzaMx02PHfzDIwb1LU5XDCuBH/9zrSYkqYRqSGqcGeMWQE8BWAegHEAFjLGxmn6jAJwD4AzOOfjAdyZ+KESPZWTLeGugWsq6vC9f2zEXz6r1D3H4wtGFVK5CdIClYu2XYvnVFxLfDgoCyvLbp0J0Kj1aHH70eb2wyPeJ5ZIWYKI5VsyE0Al5/wg59wL4HUAl2v6fB/AU5zzRgDgnNeC6DMoFywljovZ/KRyeFo8MZSIiyVzYiwotXVfnEFRWiwWhqIcp6psnls2lyTH3nzWqP44WN+OdtEttCsFOoi+RyzfklIAykz9NWKbktMAnMYY+5IxtpYxdkmiBkj0fPTym0v1OY2qFgmae2qElDLjXyKKMdgtDGv2hbxlJF/8ZAl3KcHX4x/tA9C9snVE3yFRfks2AKMAnAugDMAaxthEznmTshNjbBGARQAwZEjsxWyJno1eZSJJy+wwiHj0+API18lRouWV783C9ppmNHV48e2pxi540Xj1+7NwqL49IcWNhxVloeJkKJBJKiiSiIVOPdyKqkc2C5MzMRJEJGIR7kcBKAsLloltSmoArOOc+wAcYoxVQBD2G5SdOOdLASwFgOnTp3fX/En0ENwKzb3F7VPVpzRySfT4Y9PczxjZH2eM1C8LFw9zR/QPqx3aVUYV52B7TcjtUzJBJWKhU4/B/TKx76TgXx9P7hyibxOLCrABwCjG2DDGmAPAAgDLNH3ehaC1gzHWH4KZ5mDihkn0ZJTVkg43qLMyTh2ar3uOxx+Mu1hGTyHTYVXNWfIIMvIR7y6/v25SUq5LpDdRhTvn3A/gDgAfAdgD4E3O+S7G2IOMscvEbh8BaGCM7QbwOYCfc84b9K9IpBvKOqdHTnXgK0XJuS/218MfCOJPK/erklJ5fAG4eqnXR4bdCl+Ay0Jdmn+k9MXdwaj+KUFEIiZVg3O+AsAKTdt9im0O4C7xj+hjKBdUO30B3P7KZnn/qwMNWL7jOJ78pAKNHV78+lvjxXOCvdbrwyeaRk62uFFWkClnroxUTo4gUk3v/HURKaG2xS0X1YjEcUUK3FPt4alfpbB5pcmmod2bNBt1spHSIkjzen+7kPfG1UvnQ6QnJNwJQ25/ZTMWPrsW9W2eiP1a3CGf74eX7wk7LpWdk3zeJXt1mzs54frJRnoouX1BtHn8cvm+ZLlCEkRXIOFOGLKpWqhZ2hKlqIWFMQzI1U+fO25gLiQ3c5tV2JByuY8Z2L2c5GYhmZM8/oCqXF0q/PbX/Py8pN+DSA9IuBNROf+J1RGPe/xB3eAgu5UhEOSyq+TOo0Kxa0lzj5TLvSfjUmjuyujcVORVyU6SRw6RftA3hdAlnnJ1Hn8AGXYrHFYLvKIHyQOXjcf6Q6ew61iz7E2SIxabkBYge6sZQ6m5d3rDo3OTwfKfnIkPd55ImkcOkX6QcCd02XqkMea+Hn8Q2U6bLNinDS3ATXPLsfNoMzz+IB6R7PCiYuuWE2D1UuEuml/cvqDK3z2ZjB+UF3dBb6JvQ2YZQpfdx1pi7qvN8CiF4TvtFnj8wqIjoF6IBHqv66D0xuHxB2Q30J+IedEJoqdAmjuhiySALxhbggox9N0IbYbHZnEB1mmzyu6CUj8AqGsVXCd7axCTpLm/t+04JpQK+c/PHFVk5pAIIgwS7oQukl08P9Me1fTg1mR4lBZOHTaLKsBJuk67p3cH/UhvIJ/uOYlP95wEAJRSYWiih9E7VSci6UhukNlOG2pbI/u5C0nAwgW102aBL8BlM40vwBEIctk2X5yj7z7Z09Fmf7RaGErzSbgTPQsS7kQYlbWtWF0h5CuX8rHr5WyXkOqhnjdabZrQ2tgBoKHdI2vwyUqRm2y0Xj691bxEpDdkliHCUGrqQwuFgtSd3oChd4uUJ+bvN83Asm1HcfH4AQAEs4zEoDwXjjW70ekN9HpXSG2FqN46DyK9IeFOqHD7Arj+2XXyvpSb3aiiUvni5QCA9YdOwWphuHJKqKCG0g4vJdtaf+gUHvtwX9jx3kyDTj4dgjAbEu6EijqF1v7Y1afDKhbb0FtUFZKBCmw53BR2XCm8peu+vLZabjMq5EEQRPdJD9WJSBhKeXvF5FLZ5ODWKaWnVxhbiV4xDmUFI4IgkgcJd0KFUmA7bBZ50VNPc+9UtC2eNybsuLKQ8x3npVeQz/M3Tzd7CAQRERLuhApllkMgtFgopetVoswWecHYkrDjysCm4WIOdAAoK8jA5ZMHdXusZnL+mBJULbnU7GEQhCEk3AkVtS2CbTwvQ1hIlezmP1RUV5KorGuTt/USWilt7gWZoeM1jZ1pU9hiVHE2rpxSavYwCCIMWlAlVEi51l/7/uywY15/UOXe2OEJmWXyM8LrfCqFuzYatbdGp2r54KdnwZqCVL8EES+kuRMqPttbCwAozBY07TyF0L78qS9VffedCCUX08tlrvSL17o9xpNSuCdjs1rI64fokZBwJ1RImnmJWFlpVEmoWtKe4y26fY1QCnSnzYrP7j5H3h/WP0vvFIIgEgQJd0KFxxfA4H7qPClXTyvT7avnHqlEpbnbLSq7fJ6OGYcgiMRBwp1Q4faHpxnIMAivl/LNPP2dqbrHyxSZEp02iypMv6Yx3PuGIIjEQQuqhIoVO04g26n+Wkg5y7W4fUEUZNoxb+JA3eNKO7zTZlWZaaKZdAiC6B70CyNk/GIqXqlykoRRYiy3LxBz0iynXb3weNVUch8kiGRCwp2QcRskBysryJS3lflk3P5g7MJdo6kX5/bOXO4E0Vsg4U7I7Dyqn/dFucD67y1H5W2PLxA1s+P1s4YAUKciIAgi+ZDNnZBpVqQTUKKsmLT3RKieaqvbH1Vzf/jyCbjvm+Nkk8zOBy5GIErCMYIgug+pU4TMD17eZHhs6pB8ACH7O+ccXx9sQLT4HYuFqR4A2U4b8jLJDZIgkg0JdyKMNxaFpx64/7LxAICxA4SgJql4x/D+2akbGEEQMUPCvQ+zpqIO97yzAwDw8a4TcvsgnWLPOWJFptvFBGIeMYBp/CB9N0mCIMyFhHsf5rvPr8dr6w+Dc45FCpOMMvhIQrtw6hYDmJy9tMg1QaQ79MtMI8oXL5drmsbDDo2XjF4iLJsmMZikuRsVzSYIwlxIuKcJ1Q3t8nZzh77XixGX/SWU7fGxq0/X7ZOfqc7XLqUecJHmThA9EvplpgnKJF7tmmpKSjq8fpxscRsev3b6YN12bboAN2nuBNGjIeGeJigtKUdOdRj2W7h0LWb9ZiW8BtGosUKaO0H0bOiXmSYEFWkBvAFjwb2tRrCvt3uMtXsjbpozFIDg4y65QpLmThA9k5iEO2PsEsbYPsZYJWNscYR+VzHGOGOMSsOnGKUmfuNz6/HPtdUR+x+OoN0bUZInRKp6/EG4faS5E0RPJuovkzFmBfAUgHkAxgFYyBgbp9MvB8BPAaxL9CCJ6GjNLPe+uzNi/0P17RGP6yFp6R5/kDR3gujhxKJ2zQRQyTk/yDn3AngdwOU6/R4C8CgA49U6Iil8vq8WVz/zdVzn7DvZGr2TBsnX3eMPoEr0ziHNnSB6JrH8MksBHFHs14htMoyxqQAGc84jOlkzxhYxxjYyxjbW1dXFPVhCn1te2CBvSwL4skmDIp7TopMkbNawfhHPkTxmPL4gLOIKrtZFkiCInkG31S7GmAXAkwDujtaXc76Ucz6dcz69qKiou7cmdHj79rlw2S0YkOfC1wcaUN/m0e239mCDaj8vw443fjAn4rVDmnsQ22uaAAA5TkosShA9kViE+1EASufnMrFNIgfABACrGGNVAGYDWEaLquZQmO2Ay25FpzeAhc+uxTUKc00wGPKoOVAnmFUk4WyU7ldJyOYewIodQi4aiyVKWkiCIEwhFuG+AcAoxtgwxpgDwAIAy6SDnPNmznl/znk557wcwFoAl3HONyZlxEREBuS64LRZcKrdC0C9cOrR8W1/6baZAIAxYrbHSEh5ZNrc8btREgSRWqIKd865H8AdAD4CsAfAm5zzXYyxBxljlyV7gER8MMbAObCpujHsWEN7uImmKNsJAJg9vDDqtSWzzHVL13ZzlARBJJuYDKac8xUAVmja7jPoe273h0XEQ67Lhn5ZDjxz4zQAQLbLhg5PIKyftvD10MJMDO6XiRU/OQujSqLnZSe3R4LoPdBqWBqQ47Jj6tACjBkg5FY/WBcyxSizOSrzzwDAeaOLAQDjYszJHq1eKkEQPQf6taYB3kDQUPD6FYuoUlSphJ47ZCS093jsKv0MkgRBmA8J9x7KuoMNePHLQ1H7uX0B1LXquzsqefC93VigsZXrVVyKhNYs860ovvQEQZgHCfceys0vbMD97+2O2u/Fr6oAAK+tD8WZzSgvkLdHFgu29OdjeFBEQ1vYmqJTCaLnQr/OHsLx5k5sPdIk73eKJhSuyPaox7GmzrC2H503Ut7O0gkymhODZ4we2Ypr/WnhFN2KTQRB9AxIuPcQbn1xI6546suwdj3fdCV7Twg5Yq6fNURuy7CHzCceX7jXzLUzygAAM6OkG9BiVSzOFuc44zqXIIjUQt4yPYQ9x1sACFGkyqjPTm8ALoWwdvvU+xl2KzIdVjxyxQS5TXlc6/4IAFdOKcP5Y0qQl2EPOxZ1nA9egsYOb9z2eoIgUgsJ9x5AraLsndsfQKYj9N9S1+ZBQZaQnEsqfl1WkIEvfnE+NladwuoKIQGb0kSS7QqdX9PYiU5vuPbeFcEOABkOKzIcJNgJoqdDZpkewMmWkLdLh0YQ62neNY2Cnb26Qb/gxvD+Wfjzwik4d7SQnO14c7hdniCI9IY09x6AVI8UAO54dTPWHjwl77t1tG4AqKxtw91vbdM9xhjDtyYNQlOHF6v21WH9oVO6/QiCSF9Ic+8BdCoWPZWCXXtMIsthxa+XhSotSbVNtWSI5p3F7+xIxDAJguhFkObeA9CmBVDyr001eHltNTZVhRKBjRmYC6sl9Fy+Z/5Y3XOVXjMEQfQtSLj3ALRpAZR8sPNEWJvHH8BpikRfRqkHHJr28sJMTBsan/sjQRC9ExLuPYDGDm/MfUtynej0BnCiWfCwOfTb+YbBREMLM1X7H955tspNkiCI9IWEew9gz3F1sWqH1QJvQN9UU5DpwN4TrXIlpUhRooVZ6vqmDistsRBEX4F+7T0Am4VBWa0uy2msXednxu6fXpjtxMq7z5H3qSQeQfQdSLj3ADp9AQzIdcn7jR3GqXj7abTxaIwoil6EgyCI9IPMMj2ATl8AGY5wbf2qqWV4e3ONqm1wQWZYv2i8d8eZ4IicgIwgiPSCNHeTCQY5lm8/jlqdnOyXTw7Pl+4LxC+kJ5bl4fSy/K4MjyCIXgpp7iZTUSsspra6/Xjoign4+kA9rp5Whg93noBdsQB65wWjUJTjxBf76+W2+745LuXjJQiid0DC3WSsCm+XG2cPxY2zhWjT88eU4L/76+Rjd15wGgDg872htlvPHJaiURIE0dsgs4zJSGaW4UVZYcdGl+SEtV05pTTpYyIIovdDmrvJSEnDfnVpuImlONeFA7+Zr3KTvPT0gbh4/LxUDY8giF4KCXeTkfLKOA3qkVp1fNNtFIxEEEQUSEqYjFvU3CktAEEQiYSEu8l4RM3dZSPhThBE4iDhbjIeWXOn/wqCIBIHSRQT+demGvz09a0AACeZZQiCSCAk3E3kqc8r5e2ibKeJIyEIIt0gbxkTqKpvh8tuVRXp0BbWIAiC6A4k3E3g3N+tAgAUxJG+lyAIIh5IuJtIpy+Am+YMNayBShAE0VXIFpBiWt2hXO1uXxB5GXbycScIIuGQcE8x9W3qeqkunTzuBEEQ3YWEe4qpaexQ7VPwEkEQyYCEe4p54uMK1f55Y4pNGglBEOlMTMKdMXYJY2wfY6ySMbZY5/hdjLHdjLHtjLGVjLGhiR9qehDk6kpKw/qHp/olCILoLlGFO2PMCuApAPMAjAOwkDGmzU+7BcB0zvnpAP4F4LFEDzRdmDO8UN5edPZwE0dCEEQ6E4sr5EwAlZzzgwDAGHsdwOUAdksdOOefK/qvBXBDIgeZTtQ0dgIA9j50CXnJEASRNGIxy5QCOKLYrxHbjLgNwAd6BxhjixhjGxljG+vq6vS6pDWbqhuxfMdxAICTIlIJgkgiCZUwjLEbAEwH8Ljecc75Us75dM759KKiokTeuldw5FTIU4ax8CIcBEEQiSIWs8xRAIMV+2VimwrG2AUAfgngHM65JzHDSy8krZ0gCCLZxKK5bwAwijE2jDHmALAAwDJlB8bYFAB/A3AZ57w28cNMD+xW0tYJgkgNUYU759wP4A4AHwHYA+BNzvkuxtiDjLHLxG6PA8gG8BZjbCtjbJnB5fosvkAQK3acMHsYBEH0EWJKHMY5XwFghabtPsX2BQkeV9pxVPSSAYCHLh9v4kgIgugLUFbIFNHhFXK3P3PDVFwyYaDJoyEIIt0hf7wU0SkW5shw0POUIIjkQ8I9RXSKmnsGBS4RBJECSLiniAN1bQAoeIkgiNRAkibFlBZkmD0EgiD6ACTcu8i8P/4Xb2w4jFteWI8/r9wftf/KvYL7f7aTbO4EQSQfkjRdgHOOPcdb8Iu3dwAAPt9Xhx9/Y1TEc2wWIYCJkoURBJEKSHPvAnWt4dkV3KI3jBGd3gBmlBcka0gEQRAqSLh3gS1HmsLavthfH/Ectz9AWjtBECmDhHsc+ANBfL6vFpW1bWHHDtW3RzzX7QuScCcIImWQzT0Olu84jp++vlX32Gd7a/H9CJWV3D7S3AmCSB2kucdBVX0oH3tJrlN1rLHDi/o240zHbR4/XOTjThBEiiBpEwccoeLWBZkODMh1AQAmluZh74lWTH/4U7y67nD4eZyjrtUDCxXoIAgiRZBZpotkOqz4y/VTUN3QgaYOH+5+axsAoLoh3Pbu9gUBALkZ9HETBJEaSNrEwUtfVcnbGQ4rRhbnYGRxDrbXNMnt/1xbjXvmjwUAVNa24oIn18jHhvTLTNVQCYLo45BZJg78gZBZpkQ0yQBAkCv6KHY2VzfJ21kOK84dXZzU8REEQUiQcI+R2lY3Wj1+ef+icSXyttcfDG0HguBcEPCbDzfK7b+/bjIGk+ZOEESKIOEeIx/tOglASNlbmOXAqJIc+djoATlw2S04Y2QhOAc8orBvaPfKfZT9CYIgkg3Z3GNk97EWAMCW+y4M81fPy7Bj70Pz8MKXh/BlZQPaPX647FYca+rEtKEFePv2uWYMmSCIPgxp7jHy3rZjACLnY7dZhWObDzcBAGpbPfAFgob9CYIgkgUJ9zi4ZPwAsAi+6nOGFwIA2kXbvIWRhwxBEOZAwj1GPP4AhhVlRexTkGkHALy9uQaAkAmyf7Yz0ikEQRBJgYR7DPgCQfgCHJlRcsMUioJcikSlZGEEQZgFCfcYuH/ZLgBC4FI0zhhZiNUVdTjz0c/gDQSRGcM5BEEQiYaEexQ453hFzBczd0T/qP0zHYIDUk1jJwBg0uD8pI2NIAjCiD4r3Pccb8Ezqw/gYF14bnYl22qaAQA/v3g0xg3KjXpdZXATAMwa1q/rgyQIgugifVa4P/HxPiz5YC+eXnUgYr8/icWvRxVnx3TdEZp+kVwnCYIgkkWfkTyBIMfuYy3YdawZvkAQW48IGvmJFnfE8zq8fkwenI+Lxg+I6T5ThxSg4uF58n4k10mCIIhk0WciVF/+ugr3v7cbADCyOFsurHGwLnJ5vE5vAHmZjrju5bBZMKO8ABuqGqN3JgiCSAJ9Rrgfb3HDbmWwWy1yDdTTSrJVSb/06PQFMLAL7ozP3DANjR2+Lo2VIAiiu/QZs8zfVh9EpsOG/Awh0CjDbsX08n6oauhA+eLlclQpIHjIXPPMV5j9m5WoONkGlz3+j6kw24mRMdrpCYIgEk2f0Nwl7Xx4URZunluOTdWNGD8oFxNL8+WyeBUnWzFlSAEAIauj0qRSkBWfWYYgCMJs+oRwf3VdNQDgskmDcPnkUlw+uTSszxMfV6BflgMFmXbcNLdcdUwS+gRBEL2FPiHcX99wBIDgyaLluumD8cbGIzja1ImKk62obfWgvH8oh8yYATmYWJqXsrESBEEkgj4h3A/Vt+PKKaW60aKPXn06Hr36dADA6oo63PT8eqzcUwsAWHbHGTi9LPwcgiCInk7aL6hyzuHxB9Hh9UftW5qfAQD4orIegLDoShAE0RvpdcK9zePHiWY3TjS7EVBWpjbod/hUBwDEpIGPLM7GVVPL5P1YEoURBEH0RGIyyzDGLgHwRwBWAH/nnC/RHHcC+AeAaQAaAFzHOa9K7FAFXllbjd9+sBcA8O2ppXjy2slhfTq8fsx65FO0ewNyW78YPV6GK3K25zjt3RssQRCESUQV7owxK4CnAFwIoAbABsbYMs75bkW32wA0cs5HMsYWAHgUwHXJGPBZo4qQm2HHc18cwuGGDt0+DW1etHsDuGZaGaYOLYDdasG8CbGlD/jOrCEoynaiKMeJvEwS7gRB9E5i0dxnAqjknB8EAMbY6wAuB6AU7pcDuF/c/heAvzDGGOfc2G7SRcYNysW4Qbn4fG8tVlXU4cInV4f18Yp1S88fU4x5EwfGdf38TAeunTE4IWMlCIIwi1iEeymAI4r9GgCzjPpwzv2MsWYAhQDqlZ0YY4sALAKAIUOGdHHIAtfPGgKb1Tgp17ShBZhJ6XYJguijpNQVknO+FMBSAJg+fXq3tPpzRxfj3NHFCRkXQRBEuhGLt8xRAEo7RZnYptuHMWYDkAdhYZUgCIIwgViE+wYAoxhjwxhjDgALACzT9FkG4CZx+2oAnyXD3k4QBEHERlSzjGhDvwPARxBcIZ/nnO9ijD0IYCPnfBmA5wC8zBirBHAKwgOAIAiCMImYbO6c8xUAVmja7lNsuwFck9ihEQRBEF2l10WoEgRBENEh4U4QBJGGkHAnCIJIQ0i4EwRBpCHMLI9FxlgdgOount4fmujXPkJfnTfQd+dO8+5bxDLvoZzzomgXMk24dwfG2EbO+XSzx5Fq+uq8gb47d5p33yKR8yazDEEQRBpCwp0gCCIN6a3CfanZAzCJvjpvoO/Onebdt0jYvHulzZ0gCIKITG/V3AmCIIgIkHAnCIJIQ3qdcGeMXcIY28cYq2SMLTZ7PN2FMfY8Y6yWMbZT0daPMfYJY2y/+G+B2M4YY38S576dMTZVcc5NYv/9jLGb9O7Vk2CMDWaMfc4Y280Y28UY+6nYntZzZ4y5GGPrGWPbxHk/ILYPY4ytE+f3hpheG4wxp7hfKR4vV1zrHrF9H2PsYpOmFBeMMStjbAtj7H1xP+3nzRirYoztYIxtZYxtFNuS/z3nnPeaPwgphw8AGA7AAWAbgHFmj6ubczobwFQAOxVtjwFYLG4vBvCouD0fwAcAGIDZANaJ7f0AHBT/LRC3C8yeW5R5DwQwVdzOAVABYFy6z10cf7a4bQewTpzPmwAWiO3PALhd3P4hgGfE7QUA3hC3x4nffyeAYeLvwmr2/GKY/10AXgXwvrif9vMGUAWgv6Yt6d9z0yce54c0B8BHiv17ANxj9rgSMK9yjXDfB2CguD0QwD5x+28AFmr7AVgI4G+KdlW/3vAH4D8ALuxLcweQCWAzhJrE9QBsYrv8PYdQR2GOuG0T+zHtd1/Zr6f+QajithLA+QDeF+fRF+atJ9yT/j3vbWYZvWLdpSaNJZmUcM6Pi9snAJSI20bz79Wfi/jKPQWCFpv2cxdNE1sB1AL4BIL22cQ594tdlHNQFZ8HIBWf73XzBvAHAP8LICjuF6JvzJsD+JgxtokxtkhsS/r3PKUFson44Zxzxlja+qsyxrIBvA3gTs55C2NMPpauc+ecBwBMZozlA/g3gDHmjij5MMa+CaCWc76JMXauycNJNWdyzo8yxooBfMIY26s8mKzveW/T3GMp1p0OnGSMDQQA8d9asd1o/r3yc2GM2SEI9lc45++IzX1i7gDAOW8C8DkEc0Q+E4rLA+o5GBWf723zPgPAZYyxKgCvQzDN/BHpP29wzo+K/9ZCeJjPRAq+571NuMdSrDsdUBYcvwmCPVpq/664oj4bQLP4avcRgIsYYwXiqvtFYluPhQkq+nMA9nDOn1QcSuu5M8aKRI0djLEMCOsMeyAI+avFbtp56xWfXwZggehVMgzAKADrUzKJLsA5v4dzXsY5L4fwu/2Mc/4dpPm8GWNZjLEcaRvC93MnUvE9N3uxoQuLE/MheFYcAPBLs8eTgPm8BuA4AB8EO9ptEGyLKwHsB/ApgH5iXwbgKXHuOwBMV1znVgCV4t8tZs8rhnmfCcEWuR3AVvFvfrrPHcDpALaI894J4D6xfTgEIVUJ4C0ATrHdJe5XiseHK671S/Hz2Adgntlzi+MzOBchb5m0nrc4v23i3y5JZqXie07pBwiCINKQ3maWIQiCIGKAhDtBEEQaQsKdIAgiDSHhThAEkYaQcCcIgkhDSLgTBEGkISTcCYIg0pD/D5/8170CUqe7AAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# Plot the moving average of total reward across episodes here\n",
        "rm=movingaverage(reward_of_episodes, 100)\n",
        "plot(rm)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "beautiful-australian",
      "metadata": {
        "id": "beautiful-australian"
      },
      "source": [
        "#### How do your algorithms perform for different values of epsilon? Adaptive methods vs Fixed methods?\n",
        "Fixed methods don't yield good results topping at around 0.2-0.3. This is because as the agent learns it should focus more on exploiting its knowledge rather than exploring. We use exponential decay of epsilon starting from 1 (pure exploration) to 0 (pure exploitation).\n",
        "\n",
        "#### How do your algorithms perform for different values of the learning rate? Adaptive methods vs Fixed methods?\n",
        "Learning rate didn't influence the algorithms as much the value of epsilon. A modest learning rate of 0.6 was good enough for good performance. A high learning rate gave really bad performance. Adaptive learning rate didn't help much either.\n",
        "\n",
        "#### With a fixed epsilon and under identical learning rate rules and gamma value, compare the performance of Q-learning and SARSA. \n",
        "Q-learning performs better than SARSA under identical hyperparameters due to absence of penalizing rewards in this case. Using adaptive methods, the results were very similar converging to the optimal value as we expected."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.9 (default, Oct 26 2021, 07:25:54) \n[Clang 13.0.0 (clang-1300.0.29.30)]"
    },
    "vscode": {
      "interpreter": {
        "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}