report_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Judy_Bot_T3 -- virtually 6th place NetHack agent comparing to [NetHack Challenge at NeurIPS 2021](https://www.aicrowd.com/challenges/neurips-2021-the-nethack-challenge) results\n",
    "\n",
    "Version: 1.1.9 - AIF Exam Update\n",
    "\n",
    "## Introduction\n",
    "Released in 1987, [NetHack](https://nethackwiki.com/wiki/NetHack) is still one of the most difficult video games ever published, with the first victory only achieved by the most determined players after years of experience. Over time, NetHack has attracted the attention of researchers and AI enthusiasts, as it embodies a feature-rich environment that is relevant to the scientific advancement of the state of the art in AI.\n",
    "\n",
    "Within the game, the player will have to face fifty procedurally generated levels, plus five extra floors, characterised by an even more extreme difficulty than the previous ones. The numerous monsters, the hidden traps and the constant need to feed the protagonist will be just some of the elements that can lead the player to \"game over\", which means that the whole game has to be restarted.\n",
    "\n",
    "Starting from the analysis of the bachelor's degree thesis project Judy_Bot_T3 (02/12/2022), which aimed to create an open source bot in Python for the video game NetHack, the report addresses the main characteristics of the developed agent and its evolution in relation to what has been learnt during the Artificial Intelligence Fundamentals course at the University of Pisa.\n",
    "\n",
    "Starting with version 1.1.7, the first public version released in correspondence with the thesis project, the agent was able to achieve an average score of 744 and a median score of 645. Scores obtained by playing the game using the \"NetHackChallenge-v0\" task through [NetHack Learning Environment](https://github.com/facebookresearch/nle) framework.\n",
    "\n",
    "Starting from version 1.1.9 (AIF Exam Update) the agent is now able to obtain an average score of 1046.96 and a median score of 817.\n",
    "\n",
    "The code produced is completely open source and has been designed with the aim of respecting the principles of configurability, modularity and extensibility. In fact, it is possible to extend the agent by implementing modules dedicated to the planning and execution of new tasks, which will be fully compatible with the main system.\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "## Related Works\n",
    "Since its first release, the Judy_Bot_T3 project has been based on several previous studies, such as the implementations of the previous historical bots TAEB, BotHack and AutoAscend, as well as on two main frameworks: NetHack Learning Environment (NLE) and MiniHack.\n",
    "\n",
    "NLE is a framework that aims to provide an interface to the NetHack terminal software. To achieve this goal, NLE formalises the game commands (inputs) in Python methods with appropriate configuration attributes, and makes the game output usable by the higher-level implementation through special observation data structures according to the canons of the [Gym](https://www.gymlibrary.dev/) interface on which it is based.\n",
    "\n",
    "MiniHack, on the other hand, is a framework dedicated to the meticulous study of specific features of the otherwise vast NetHack environment. Thanks to the \"tasks\" defined in the framework's documentation, the agent developed can deal with restricted environments characterised by specific game elements in order to study a specific behaviour by limiting the space of actions and observations available to the agent.\n",
    "\n",
    "The same scientific personalities behind the development of these frameworks are also the creators of the NetHack NeurIPS Challenge, a competition that received a great deal of attention during its 2021 edition, with the participation of more than four hundred and eighty developers and the submission of more than forty implementations.\n",
    "\n",
    "<br>\n",
    "<div>\n",
    "<center><img src=\"img/ChallengeResults.jpg\" width=\"500\"/></center>\n",
    "</div>\n",
    "<br>\n",
    "\n",
    "Thanks to the public results of the [challenge](https://www.aicrowd.com/challenges/neurips-2021-the-nethack-challenge), it was possible to compare Judy_Bot_T3's results with those of the other candidates according to the following evaluation metric (ordered by relevance): number of ascensions - median of the scores - average score. These statistics played a key role in determining the degree of success of the agent, which, with an average score of 744 and a median score of 645 in its first version (1.1.7), already outperformed the eighth competitor: `JustPaulsAI`.\n",
    "\n",
    "<br>\n",
    "<div>\n",
    "<center><img src=\"img/117Performance.jpg\" width=\"500\"/></center>\n",
    "</div>\n",
    "<br>\n",
    "\n",
    "With the aim of presenting it in the Artificial Intelligence Fundamentals exam, Judy_Bot_T3 underwent numerous refinements, both from the point of view of code cleanliness and readability (with the relocation of the component modules into separate files, the inclusion of comments describing each method, and the writing of the README.md file, now merged with this notebook report) and from the point of view of functionality (with the refinement of existing modules according to the notions of the course and new elements learned through further study of the game). The capabilities of the agent after the updates as well as the technical details of its implementation will be analysed in the rest of the report, after which a further comparison with the statistics of the NetHack NeurIPS Challenge 2021 will be made for a final assessment.\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "\n",
    "## Methodologies\n",
    "The ability to move efficiently within the NetHack map is a key implementation for an optimised agent developed in this environment. For this purpose, Judy_Bot_T3 provides the \"DungeonWalker\" class, which aims to encapsulate functions useful for exploring the game world, including all implementations necessary for path finding. To compute its navigation plans, Judy_Bot_T3 uses the [`A* Algorithm`](https://doi.org/10.1109/tssc.1968.300136), which is widely used at the state of the art. \n",
    "\n",
    "An important observation concerns the heuristics associated with the algorithm. NetHack assumes that the player character and other entities in the game can move in eight directions on the map grid. Due to this characteristic of the environment, it proved essential to identify a suitable admissible heuristic, with the design choice falling on the `Octile Distance Heuristic` after a first research phase.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# heuristic for the distance between two points: A(ax, ay) and B(bx, by)\n",
    "# environment with 8 directions of movement\n",
    "def h_octile_distance (self, ay, ax, by, bx):\n",
    "    x_d = abs(ax - bx)\n",
    "    y_d = abs(ay - by)\n",
    "    return (1.414 * min(x_d , y_d)) + abs(x_d - y_d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Octile Distance Heuristic` values a diagonal move as more costly than a four-way move, but gives it less weight than two four-way moves. This is a crucial property in environments such as NetHack, where the ability to use the least number of steps brings optimisations in resource consumption. As an example, below is a comparison of the identified heuristic with the `Manhattan Distance Heuristic` studied in the course, a classic heuristic for searching in four-way environments.\n",
    "\n",
    "Given the existence of a point A (0,0) and a point B (4,4), and given that the cost of moving in the four directions is 1, the `Manhattan Distance Heuristic` would predict that the cost of moving from A to B is 8. Using the same premises, but looking at the case of the `Octile Distance Heuristic`, which predicts √2 as the cost of a diagonal move, this would instead produce as result 5.66.\n",
    "\n",
    "<br>\n",
    "<div>\n",
    "<center><img src=\"img/Heuristics.jpg\" width=\"500\"/></center>\n",
    "</div>\n",
    "<br>\n",
    "\n",
    "It is therefore clear that the `Manhattan Distance Heuristic` applied to the environment under consideration would not lead to an approximation less than or equal to the true distance between the two points, resulting in a not admissible heuristic and consequently damaging the properties of the `A* Algorithm`.\n",
    "\n",
    "The last improvements of the agent are located in the Elbereth, Eating and Fighting modules of the system, in addition to the various small improvements made to some minor sections of the code. Thanks to these additions, mainly derived from the in-depth study (facilitated by the MiniHack framework) of some specific game mechanics, the agent is now able to distinguish between healthy and unhealthy foods with an almost certain success rate, discarding suspect foods and thus avoiding potential traps and hazards usually hidden within these resources. The Agent is also now able to consistently protect itself with the [Elbereth](https://nethackwiki.com/wiki/Elbereth) engraving, a game mechanics that consists in engraving the name of the fictional goddess Elbereth on the ground. This activity requires the agent to perform a complex sequence of actions in order to aquire a temporary protection that can only be obtained if certain game conditions are met (e.g. the agent does not perform any attack or movement actions, and enemies in its vicinity respect the goddess name). The bot will now avoid engraving in situations where it would not benefit from it.\n",
    "\n",
    "Thanks to the implementation of an effective turn counter and the infusion of knowledge into the agent related to the in-game [Prayer](https://nethackwiki.com/wiki/Prayer) action, the agent is now able to gain benefits and resources without suffering any side effects. This ability greatly increases Judy_Bot_T3's Survival skills, reducing its hunger and regenerating its health in deadly situations.\n",
    "\n",
    "Enhancements to the combat module now allow the agent to use specific strategies against specific enemies such as Floating Eyes and Werefoos, creatures that were previously one of the Agent's main killers.\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "\n",
    "## Agent Assessment\n",
    "The empirical evaluation of the agent was strongly guided by the comparison with the parameters and results of the NetHack NeurIPS Challenge 2021. Specifically, the code was run on a sample of 1000 games, all generated according to the parameters of the challenge and through\n",
    "the use of “NetHackChallenge-v0” environment, available on the latest version of NLE (0.8.1). This environment comes as close as possible to playing the real game of NetHack with a random character to start, and a full keyboard of actions to take.\n",
    "\n",
    "Collecting the outcomes of the games, the agent was then evaluated according to the three metrics: number of ascensions (wins) achieved, median and mean score between games. Having thus calculated the same metrics used in the challenge for the evaluation of the competitors, it was then possible to assess the quality of Judy_Bot_T3 in comparison to the latters. As the agent never achieved an ascension, as well as the other bots competing in the 2021\n",
    "edition of the challenge, the pivotal parameter for the evaluation was the median of the scores between the games.\n",
    "\n",
    "Thanks to the changes in the last few updates, the evaluation phase of Judy_Bot_T3 ended with the results of 1046.96 as the mean score and 817 as the median score, statistics that virtually rank the agent with a solid sixth position on the NetHack NeurIPS Challenge 2021 ranking list.\n",
    "\n",
    "<br>\n",
    "<div>\n",
    "<center><img src=\"img/119Performance.jpg\" width=\"500\"/></center>\n",
    "</div>\n",
    "<br>\n",
    "\n",
    "\n",
    "## Conclusions\n",
    "The results achieved by Judy_Bot_T3 are certainly encouraging, as is the relative speed of improvement compared to previous thesis project results (around fifty to sixty hours of work). The collected data continue to demonstrate the very high potential of NetHack Learning Environment\n",
    "and MiniHack frameworks in driving scientific research around the NetHack game.\n",
    "\n",
    "It is clear, however that the journey towards the cration of a bot capable of “solving” the problem of the NetHack game is still incredibly long. Despite the challenge winners themselves have a huge difference in scores from what achieved by Judy_Bot_T3 (the winner, [AutoAscend](https://github.com/maciej-sypetkowski/autoascend), boasts an unbeaten record of 5336.5 as a median score), the very same statistics of the podium agents, however, are largely insufficient to consider the NetHack case of study as scientifically saturated.\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "## Team Contributions\n",
    "Being the result of the evolution of the previous Thesis project, Judy_Bot_T3 has not seen the participation of a team for its realization and is presented for examination as an individual project.\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "## GitHub Metrics\n",
    "As an individual work, the distribution of GitHub commits per group member is ignored in the document. However, the entire project is open-source and available on [GitHub](https://github.com/SimoneMarzeddu/Judy_Bot_T3/tree/AIF-Project).\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "## Relationship with the Course\n",
    "The design phase of the agent exploited what is presented in AIMA Chapter 2 (second lesson of the course) to properly define the task environment and the agent performance measure. The technical core of the entire agent is certainly the implementation of the `A* Algorithm` supported by the `Octile Distance Heuristic`, developed according to what was learned from the fourth lesson of the course (AIMA Chapter 3).\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "## Environment and Software Dependencies\n",
    "All Judy_Bot_T3 software dependencies are limited to the [NLE](https://github.com/facebookresearch/nle) installation.\n",
    "\n",
    "The version of [NLE](https://github.com/facebookresearch/nle) used in the programming phase is 0.8.1.\n",
    "Other software dependencies are closely related to the requirements for installing [NLE](https://github.com/facebookresearch/nle).\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "\n",
    "\n",
    "## How to run\n",
    "To run the agent, simply use the shell command `python -m main` to start the code flow.\n",
    "\n",
    "Thanks to the `config.json` configuration file it is possible to determine some aspects of the behavior of the software and the agent:\n",
    "\n",
    "* The `task_prio_list` key allows you to define a list of tasks in order of priority (with their symbolic names).\n",
    "The agent will consider the order given as a trace to administer his own logic, reserving the right to dynamically modify the established priorities in relation to the different game situations.\n",
    "Currently, the agent has the ability to plan and execute 14 different tasks, which find their implementation in specific Python classes (see next section).\n",
    "Below is a list of the symbolic names of the tasks accompanied by a brief description of their behavior. This specific order corresponds to the strategy that led Judy_Bot_T3 to the previously stated results:\n",
    "\n",
    "  * `pray` -> Where the agent's prayer is planned, considering the requirements for safe prayer and the agent's needs,\n",
    "  \n",
    "  * `engrave_elbereth` -> Which allows the agent to engrave Elbereth on the ground, thus defending himself from some malevolent creatures,\n",
    "  \n",
    "  * `run_for_your_life` -> Which allows the agent to escape from unpleasant situations, fleeing from danger,\n",
    "  \n",
    "  * `take_a_break` -> Which allows the agent to rest and restore their vitality,\n",
    "  \n",
    "  * `close_monster_fight` -> \n",
    "Which allows the agent to fight, employing a strategy of avoiding passive monsters and not granting enemies bonus attacks,\n",
    "  \n",
    "  * `time_of_the_lunch` -> \n",
    "Which allows the agent to feed, avoiding eating dangerous food and checking for the presence of traps in suspicious corpses,\n",
    "  \n",
    "  * `greed_of_gold` -> Which allows the agent to reach and collect gold during his adventure,\n",
    "  \n",
    "  * `stairs_descent` -> Which allows the agent to descend into the dungeon according to a \"slow descent\" logic,\n",
    "  \n",
    "  * `stairs_ascent` -> Which allows the agent to go back up in the dungeon according to a \"slow descent\" logic,\n",
    "  \n",
    "  * `reach_closest_explorable` -> Which allows the agent to reach and interact with points of interest for exploration, such as doors and corridors,\n",
    "  \n",
    "  * `reach_horizon` -> Which allows the agent to reach the frontier of exploration, expanding their knowledge of the dungeon,\n",
    "  \n",
    "  * `explore_unseen` -> Which allows the agent to reach tiles they have never walked on,\n",
    "  \n",
    "  * `search_hidden_room` -> Which allows the agent to locate secret passages in dungeon rooms,\n",
    "  \n",
    "  * `search_hidden_corridor` -> Which allows the agent to locate secret passages in dungeon corridors.\n",
    "\n",
    "    \n",
    "* The `fast_mode` key determines how the agent will run. When the configured value is `on`, Judy_Bot_T3 will play NetHack without the terminal showing the game interface, saving computational resources for the massive execution of several games, printing a simple agent performance report.\n",
    "When the configured value is `off`, the games played by Judy_Bot_T3 will be viewable through the typical game interface.\n",
    "\n",
    "\n",
    "* The `attempts` key determines the number of games the agent will play.\n",
    "<br>\n",
    "\n",
    "\n",
    "## Code structure\n",
    "The entire agent's logic is based on the modules that implement its tasks. Therefore, in order to expand the capabilities of the agent, it is necessary to extend the `Task` class or one of the other classes below it in the hierarchy, implementing the `planning()` and `execution()` methods, allowing the bot to integrate the task within its logic.\n",
    "\n",
    "The following is a brief description of the main structural components of Judy_Bot_T3:\n",
    "\n",
    "* `config.json` is the previously discussed configuration file,\n",
    "* `main.py` is the startup component of the agent. Its code allows the parsing of the configuration file and the setting in motion of the whole logic of Judy_Bot-T3,\n",
    "* `core.py` is the central component in interacting with the NLE framework and the underlying NetHack game,\n",
    "* `archetype_modules.py` encompasses the three archetype classes for task definition: `Task` (the most general model), `ReachTask` (specialized in tasks that require to reach a specific glyph without too many frills) and `HiddenTask` (specialized in finding hidden areas),\n",
    "* `reach_modules.py` includes classes that define tasks related to the `ReachTask` archetype,\n",
    "* `secret_passage_modules.py` includes classes that define tasks related to the `HiddenTask` archetype,\n",
    "* `general_modules.py` includes classes that define tasks related to the `Task` archetype, These are generic tasks and therefore not currently attributable to a more specific archetype."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example Configuration: Demo Mode\n",
    "\n",
    "The following code shows Judy_Bot_T3 execution with the `fast-mode` configuration option set to `off` according to the example configuration file `config_oneshot` which provides the execution of only one game in `demo-mode`, activating the visualisation and rendering of the game interface so that the agent's activity is understandable to the human user. The console output shows the passing of game turns, deleting and rewriting the various screens to generate a frame-by-frame rendering of the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.main import *\n",
    "\n",
    "dungeon_walker, game, logic, task_map, attempts = start_bot(\"config/config_oneshot.json\")\n",
    "\n",
    "main_logic(dungeon_walker, game, logic, task_map, attempts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example Configuration: Fast Mode\n",
    "\n",
    "The following code snippet shows how Judy_Bot_T3 can be started with the `fast-mode` option set to `on` according to the example configuration file `config_fast100` which provides the execution of 100 games in the former mode. The console output summarises the scores results of the games played by the bot, without showing the specific progress of each game (no rendering of rounds, no display of details such as tasks performed or paths found), devoting the computational resources exclusively to the massive execution of the bot.\n",
    "\n",
    "After each game's end, the console is updated to summarize the agent current `Mean` and `Median` scores, so to focus the user's attention on the core parameters of evaluation for the [NetHack Challenge at NeurIPS 2021](https://www.aicrowd.com/challenges/neurips-2021-the-nethack-challenge)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "// Mean :  1186.2307692307693 // Median:  1119  // Games:  13        ᕙ(`▿´)ᕗ                \n",
      "[54, 404, 485, 565, 572, 601, 1119, 1121, 1303, 1328, 1715, 1784, 4370]\n"
     ]
    }
   ],
   "source": [
    "from src.main import *\n",
    "\n",
    "dungeon_walker, game, logic, task_map, attempts = start_bot(\"config/config_fast100.json\")\n",
    "\n",
    "main_logic(dungeon_walker, game, logic, task_map, attempts)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}