notebook/3-dim-example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7cf87dfd",
   "metadata": {},
   "source": [
    "##  3-dim grid example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b1720590",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Load packages\n",
    "from NashQLearn import Player, Grid, NashQLearning\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cf74196",
   "metadata": {},
   "source": [
    "This notebook applies the Nash Q Learning algorithm to the following multiagent problem : two robots placed on a grid need to reach the reward. Robots are allowed to move up, down, to the left, and to the right, or to stay at their current position. \n",
    "Robots are not allowed to be on the same tile unless it is the reward tile.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14455cf6",
   "metadata": {},
   "source": [
    "### Prepare the game environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b6a6e69e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Initialize the two players\n",
    "player1 = Player([0,0])\n",
    "player2 = Player([2,0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3b91d86b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Initialize the grid\n",
    "grid = Grid(length = 3,\n",
    "            width = 3,\n",
    "            players = [player1,player2],\n",
    "           obstacle_coordinates = [[1,1]], #A single obstacle in the middle of the grid\n",
    "           reward_coordinates = [1,2],\n",
    "           reward_value = 20,\n",
    "           collision_penalty = -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "abdc9942",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Available joint states : 57\n",
      "[[[0, 0], [0, 1]], [[0, 0], [0, 2]], [[0, 0], [1, 0]], [[0, 0], [1, 2]], [[0, 0], [2, 0]], [[0, 0], [2, 1]], [[0, 0], [2, 2]], [[0, 1], [0, 0]], [[0, 1], [0, 2]], [[0, 1], [1, 0]], [[0, 1], [1, 2]], [[0, 1], [2, 0]], [[0, 1], [2, 1]], [[0, 1], [2, 2]], [[0, 2], [0, 0]], [[0, 2], [0, 1]], [[0, 2], [1, 0]], [[0, 2], [1, 2]], [[0, 2], [2, 0]], [[0, 2], [2, 1]], [[0, 2], [2, 2]], [[1, 0], [0, 0]], [[1, 0], [0, 1]], [[1, 0], [0, 2]], [[1, 0], [1, 2]], [[1, 0], [2, 0]], [[1, 0], [2, 1]], [[1, 0], [2, 2]], [[1, 2], [0, 0]], [[1, 2], [0, 1]], [[1, 2], [0, 2]], [[1, 2], [1, 0]], [[1, 2], [2, 0]], [[1, 2], [2, 1]], [[1, 2], [2, 2]], [[2, 0], [0, 0]], [[2, 0], [0, 1]], [[2, 0], [0, 2]], [[2, 0], [1, 0]], [[2, 0], [1, 2]], [[2, 0], [2, 1]], [[2, 0], [2, 2]], [[2, 1], [0, 0]], [[2, 1], [0, 1]], [[2, 1], [0, 2]], [[2, 1], [1, 0]], [[2, 1], [1, 2]], [[2, 1], [2, 0]], [[2, 1], [2, 2]], [[2, 2], [0, 0]], [[2, 2], [0, 1]], [[2, 2], [0, 2]], [[2, 2], [1, 0]], [[2, 2], [1, 2]], [[2, 2], [2, 0]], [[2, 2], [2, 1]], [[1, 2], [1, 2]]]\n"
     ]
    }
   ],
   "source": [
    "joint_states = grid.joint_states()\n",
    "print('Available joint states : %s'%len(joint_states))\n",
    "print(joint_states)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "10282bf9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['left', [0, 0]],\n",
       " ['down', [0, 0]],\n",
       " ['left', [0, 1]],\n",
       " ['right', [0, 1]],\n",
       " ['left', [0, 2]],\n",
       " ['up', [0, 2]],\n",
       " ['up', [1, 0]],\n",
       " ['down', [1, 0]],\n",
       " ['up', [1, 2]],\n",
       " ['down', [1, 2]],\n",
       " ['right', [2, 0]],\n",
       " ['down', [2, 0]],\n",
       " ['left', [2, 1]],\n",
       " ['right', [2, 1]],\n",
       " ['right', [2, 2]],\n",
       " ['up', [2, 2]]]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "walls = grid.identify_walls()\n",
    "walls"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4807410f",
   "metadata": {},
   "source": [
    "### Run the Nash Q Learning algorithm\n",
    "\n",
    "The efficiency of the algorithm depends on a set of parameters (the max number of iterations, the discount factor, the learning rate, ...) which may require some tuning. In general, the epsilon-greedy decision stragegy outperforms both the random and greedy strategies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "28406b6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "nashQ = NashQLearning(grid, \n",
    "                      max_iter = 2000,\n",
    "                      discount_factor = 0.7,\n",
    "                      learning_rate = 0.7,\n",
    "                      epsilon = 0.5,\n",
    "                     decision_strategy = 'epsilon-greedy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c7c49ebe",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████| 57/57 [00:00<00:00, 2716.77it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████████| 2000/2000 [08:18<00:00,  4.01it/s]\n"
     ]
    }
   ],
   "source": [
    "#Retrieve the updated Q matrix after fitting the algorithm\n",
    "Q0, Q1 = nashQ.fit(return_history = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8ae26f6e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0, 0], [2, 0]]\n",
      "[[0, 1], [2, 1]]\n",
      "[[0, 2], [2, 2]]\n",
      "[[1, 2], [1, 2]]\n"
     ]
    }
   ],
   "source": [
    "#Best path followed by each player given the values in the q tables\n",
    "p0, p1 = nashQ.get_best_policy(Q0,Q1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e576d4ff",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Player 0 follows the  policy : up-up-right of length 3\n",
      "Player 1 follows the  policy : up-up-left of length 3\n"
     ]
    }
   ],
   "source": [
    "print('Player 0 follows the  policy : %s of length %s' %('-'.join(p0),len(p0)))\n",
    "print('Player 1 follows the  policy : %s of length %s'%('-'.join(p1),len(p1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c4b401a",
   "metadata": {},
   "source": [
    "In this experiment, the two players have successfully identified the optimal path to the reward.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}