05-Adding_Memory.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "01a8b5c0-87cb-4302-8e3c-dc809d0039fb",
   "metadata": {},
   "source": [
    "# Understanding Memory in LLMs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2f73380-6395-4e9f-9c83-3f47a5d7e292",
   "metadata": {},
   "source": [
    "In the previous Notebooks, we successfully explored how OpenAI models can enhance the results from Azure AI Search queries. \n",
    "\n",
    "However, we have yet to discover how to engage in a conversation with the LLM. With [Bing Chat](http://chat.bing.com/), for example, this is possible, as it can understand and reference the previous responses.\n",
    "\n",
    "There is a common misconception that LLMs (Large Language Models) have memory. This is not true. While they possess knowledge, they do not retain information from previous questions asked to them.\n",
    "\n",
    "In this Notebook, our goal is to illustrate how we can effectively \"endow the LLM with memory\" by employing prompts and context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "733c782e-204c-47d0-8dae-c9df7091ab23",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "from langchain_community.chat_message_histories import ChatMessageHistory, CosmosDBChatMessageHistory\n",
    "from langchain_core.chat_history import BaseChatMessageHistory\n",
    "from langchain_core.runnables import ConfigurableFieldSpec\n",
    "from langchain_core.runnables.history import RunnableWithMessageHistory\n",
    "from langchain_openai import AzureChatOpenAI\n",
    "from langchain_openai import AzureOpenAIEmbeddings\n",
    "from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from operator import itemgetter\n",
    "from typing import List\n",
    "\n",
    "from IPython.display import Markdown, HTML, display  \n",
    "\n",
    "def printmd(string):\n",
    "    display(Markdown(string))\n",
    "\n",
    "#custom libraries that we will use later in the app\n",
    "from common.utils import CustomAzureSearchRetriever, get_answer\n",
    "from common.prompts import DOCSEARCH_PROMPT\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv(\"credentials.env\")\n",
    "\n",
    "import logging\n",
    "\n",
    "# Get the root logger\n",
    "logger = logging.getLogger()\n",
    "# Set the logging level to a higher level to ignore INFO messages\n",
    "logger.setLevel(logging.WARNING)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6bc63c55-a57d-49a7-b6c7-0f18bca8199e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the ENV variables that Langchain needs to connect to Azure OpenAI\n",
    "os.environ[\"OPENAI_API_VERSION\"] = os.environ[\"AZURE_OPENAI_API_VERSION\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dc72b22-11c2-4df0-91b8-033d01829663",
   "metadata": {},
   "source": [
    "### Let's start with the basics\n",
    "Let's use a very simple example to see if the GPT model of Azure OpenAI have memory. We again will be using langchain to simplify our code "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3eef5dc9-8b80-4085-980c-865fa41fa1f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "QUESTION = \"Tell me some use cases for reinforcement learning\"\n",
    "FOLLOW_UP_QUESTION = \"What was my prior question?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a00181d5-bd76-4ce4-a256-75ac5b58c60f",
   "metadata": {},
   "outputs": [],
   "source": [
    "COMPLETION_TOKENS = 1000\n",
    "# Create an OpenAI instance\n",
    "llm = AzureChatOpenAI(deployment_name=os.environ[\"GPT4o_DEPLOYMENT_NAME\"], \n",
    "                      temperature=0.5, max_tokens=COMPLETION_TOKENS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9502d0f1-fddf-40d1-95d2-a1461dcc498a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We create a very simple prompt template, just the question as is:\n",
    "output_parser = StrOutputParser()\n",
    "prompt = ChatPromptTemplate.from_messages([\n",
    "    (\"system\", \"You are an assistant that give thorough responses to users.\"),\n",
    "    (\"user\", \"{input}\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c5c9903e-15c7-4e05-87a1-58e5a7917ba2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Reinforcement Learning (RL) is a type of machine learning where an agent learns to make decisions by taking actions in an environment to maximize some notion of cumulative reward. Here are some prominent use cases for RL across various domains:\n",
       "\n",
       "### 1. **Gaming and Simulations**\n",
       "   - **Game Playing:** RL has been famously used in games like Chess, Go, and video games. For example, DeepMind's AlphaGo and AlphaZero have demonstrated superhuman performance.\n",
       "   - **Simulations:** RL can be used to train agents in simulated environments before deploying them in the real world, such as in robotic simulations or autonomous driving simulations.\n",
       "\n",
       "### 2. **Robotics**\n",
       "   - **Robot Control:** RL can be used to teach robots complex tasks such as walking, grasping objects, and performing assembly tasks.\n",
       "   - **Autonomous Navigation:** RL helps in developing navigation systems for drones, self-driving cars, and other autonomous vehicles.\n",
       "\n",
       "### 3. **Finance**\n",
       "   - **Trading Algorithms:** RL can be applied to develop trading strategies that adapt to market conditions.\n",
       "   - **Portfolio Management:** It can help in optimizing portfolios by learning to balance risk and return over time.\n",
       "\n",
       "### 4. **Healthcare**\n",
       "   - **Personalized Medicine:** RL can be used to tailor treatment plans for individual patients based on their responses to previous treatments.\n",
       "   - **Drug Discovery:** It can help in optimizing the process of discovering new drugs by efficiently exploring the chemical space.\n",
       "\n",
       "### 5. **Energy Management**\n",
       "   - **Smart Grids:** RL can optimize the distribution of electricity in smart grids, balancing supply and demand in real-time.\n",
       "   - **Energy-efficient Buildings:** It can be used to manage heating, ventilation, and air conditioning (HVAC) systems to minimize energy consumption while maintaining comfort.\n",
       "\n",
       "### 6. **Natural Language Processing (NLP)**\n",
       "   - **Dialogue Systems:** RL can be used to train chatbots and virtual assistants to have more natural and effective conversations.\n",
       "   - **Text Summarization:** It can help in generating summaries that maximize the informativeness and coherence of the text.\n",
       "\n",
       "### 7. **Marketing and Advertising**\n",
       "   - **Personalized Recommendations:** RL can optimize recommendation systems for e-commerce platforms, streaming services, etc.\n",
       "   - **Ad Placement:** It can help in deciding the best placement and timing of ads to maximize engagement and revenue.\n",
       "\n",
       "### 8. **Manufacturing**\n",
       "   - **Process Optimization:** RL can optimize manufacturing processes to improve efficiency and reduce waste.\n",
       "   - **Predictive Maintenance:** It can help in predicting equipment failures and scheduling maintenance to minimize downtime.\n",
       "\n",
       "### 9. **Telecommunications**\n",
       "   - **Network Optimization:** RL can be used to optimize the allocation of resources in telecommunications networks to improve performance and reduce costs.\n",
       "   - **Traffic Management:** It can help in managing network traffic to avoid congestion and ensure quality of service.\n",
       "\n",
       "### 10. **Transportation**\n",
       "   - **Traffic Signal Control:** RL can optimize the timing of traffic signals to reduce congestion and improve traffic flow.\n",
       "   - **Fleet Management:** It can help in optimizing routes and schedules for delivery trucks, taxis, public transportation, etc.\n",
       "\n",
       "### 11. **Education**\n",
       "   - **Personalized Learning:** RL can be used to develop adaptive learning systems that tailor educational content to the needs of individual students.\n",
       "   - **Tutoring Systems:** It can help in creating intelligent tutoring systems that provide personalized feedback and guidance.\n",
       "\n",
       "### 12. **Resource Management**\n",
       "   - **Supply Chain Optimization:** RL can optimize various aspects of the supply chain, from inventory management to logistics.\n",
       "   - **Water Resource Management:** It can be used to optimize the allocation and use of water resources.\n",
       "\n",
       "These are just a few examples, and the potential applications of reinforcement learning are vast and continually expanding as the technology matures and evolves."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's see what the GPT model responds\n",
    "chain = prompt | llm | output_parser\n",
    "response_to_initial_question = chain.invoke({\"input\": QUESTION})\n",
    "display(Markdown(response_to_initial_question))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "99acaf3c-ce68-4b87-b24a-6065b15ff9a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "I'm sorry, but I don't have access to previous interactions or any prior questions you've asked. How can I assist you today?"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Now let's ask a follow up question\n",
    "printmd(chain.invoke({\"input\": FOLLOW_UP_QUESTION}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3e1c143-c95f-4566-a8b4-af8c42f08dd2",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "As you can see, it doesn't remember what it just responded, sometimes it responds based only on the system prompt, or just randomly. This proof that the LLM does NOT have memory and that we need to give the memory as a a conversation history as part of the prompt, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0946ce71-6285-432e-b011-9c2dc1ba7b8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_prompt = ChatPromptTemplate.from_template(\n",
    "\"\"\"\n",
    "    {history}\n",
    "    Human: {question}\n",
    "    AI:\n",
    "\"\"\"\n",
    ")\n",
    "chain = hist_prompt | llm | output_parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6d088e51-e5eb-4143-b87d-b2be429eb864",
   "metadata": {},
   "outputs": [],
   "source": [
    "Conversation_history = \"\"\"\n",
    "Human: {question}\n",
    "AI: {response}\n",
    "\"\"\".format(question=QUESTION, response=response_to_initial_question)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d99e34ad-5539-44dd-b080-3ad05efd2f01",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Your prior question was: \"Tell me some use cases for reinforcement learning.\""
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(chain.invoke({\"history\":Conversation_history, \"question\": FOLLOW_UP_QUESTION}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "045e5af6-55d6-4353-b3f6-3275c95db00a",
   "metadata": {},
   "source": [
    "**Bingo!**, so we now know how to create a chatbot using LLMs, we just need to keep the state/history of the conversation and pass it as context every time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eafd1694-0077-4aa8-bd01-e9f763ce36a3",
   "metadata": {},
   "source": [
    "## Now that we understand the concept of memory via adding history as a context, let's go back to our GPT Smart Search engine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9787ffb6-2b11-4b03-92fc-9443cd1f2ab9",
   "metadata": {},
   "source": [
    "From Langchain website:\n",
    "    \n",
    "A memory system needs to support two basic actions: reading and writing. Recall that every chain defines some core execution logic that expects certain inputs. Some of these inputs come directly from the user, but some of these inputs can come from memory. A chain will interact with its memory system twice in a given run.\n",
    "\n",
    "    AFTER receiving the initial user inputs but BEFORE executing the core logic, a chain will READ from its memory system and augment the user inputs.\n",
    "    AFTER executing the core logic but BEFORE returning the answer, a chain will WRITE the inputs and outputs of the current run to memory, so that they can be referred to in future runs.\n",
    "    \n",
    "So this process adds delays to the response, but it is a necessary delay :)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f36e8f14-e566-4ae9-a7d4-6dee7f469dad",
   "metadata": {},
   "source": [
    "![image](./images/memory_diagram.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ef9f459b-e8b8-40b9-a94d-80c079968594",
   "metadata": {},
   "outputs": [],
   "source": [
    "index1_name = \"srch-index-files\"\n",
    "index2_name = \"srch-index-csv\"\n",
    "index3_name = \"srch-index-books\"\n",
    "indexes = [index1_name, index2_name, index3_name]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "b01852c2-6192-496c-adff-4270f9380469",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize our custom retriever \n",
    "retriever = CustomAzureSearchRetriever(indexes=indexes, topK=10, reranker_threshold=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "633937e8-18e6-43f2-b4d5-fc36157a4d97",
   "metadata": {},
   "source": [
    "If you check closely in prompts.py, there is an optional variable in the `DOCSEARCH_PROMPT` called `history`. Now it is the time to use it. It is basically a place holder were we will inject the conversation in the prompt so the LLM is aware of it before it answers."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "035fa6e6-226c-400f-a504-30255385f43b",
   "metadata": {},
   "source": [
    "**Now let's add memory to it:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "3c8c9381-08d0-4808-9ab1-78156ca1be6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "store = {} # Our first memory will be a dictionary in memory\n",
    "\n",
    "# We have to define a custom function that takes a session_id and looks somewhere\n",
    "# (in this case in a dictionary in memory) for the conversation\n",
    "def get_session_history(session_id: str) -> BaseChatMessageHistory:\n",
    "    if session_id not in store:\n",
    "        store[session_id] = ChatMessageHistory()\n",
    "    return store[session_id]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "48ff51e1-2b1e-4c67-965d-1c2e2f55e005",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We use our original chain with the retriever but removing the StrOutputParser\n",
    "chain = (\n",
    "    {\n",
    "        \"context\": itemgetter(\"question\") | retriever, \n",
    "        \"question\": itemgetter(\"question\"),\n",
    "        \"history\": itemgetter(\"history\")\n",
    "    }\n",
    "    | DOCSEARCH_PROMPT\n",
    "    | llm\n",
    ")\n",
    "\n",
    "## Then we pass the above chain to another chain that adds memory to it\n",
    "\n",
    "output_parser = StrOutputParser()\n",
    "\n",
    "chain_with_history = RunnableWithMessageHistory(\n",
    "    chain,\n",
    "    get_session_history,\n",
    "    input_messages_key=\"question\",\n",
    "    history_messages_key=\"history\",\n",
    ") | output_parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "0e582915-243f-42cb-bb1e-c35a20ee0b9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is where we configure the session id\n",
    "config={\"configurable\": {\"session_id\": \"abc123\"}}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ff493b1-b133-4880-a040-e80f7460e7af",
   "metadata": {},
   "source": [
    "Notice below, that we are adding a `history` variable in the call. This variable will hold the chat historywithin the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "d91a7ff4-6148-459d-917c-37302805dd09",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Reinforcement learning (RL) has a wide range of applications across various domains due to its ability to learn optimal policies through trial and error. Here are some notable use cases:\n",
       "\n",
       "1. **Games and Simulations:**\n",
       "   - **Backgammon:** Tesauro applied the temporal difference algorithm to backgammon, using a backpropagation-based neural network to approximate the value function. This approach allowed the system to learn by self-play and achieve a high level of proficiency, even competing at the top level of international human play [[1]](https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/9605/9605103v1.pdf).\n",
       "   - **Chess:** Similar to backgammon, chess can be learned by reinforcement learning through example games presented in the form of sensible (board-state, move) sequences. This method helps the system learn legal and good moves by evaluating its own moves after several games [[2]](https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/0004/0004001v1.pdf).\n",
       "\n",
       "2. **Robotics:**\n",
       "   - **Juggling Robot:** Schaal and Atkeson developed a two-armed robot that learns to juggle a device known as a devil-stick. The robot uses a combination of dynamic programming and locally weighted regression to improve its juggling policy from experience [[1]](https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/9605/9605103v1.pdf).\n",
       "   - **Mobile Robot Navigation:** Mahadevan and Connell discussed a task where a mobile robot learns to push large boxes for extended periods, showcasing RL's applicability in physical interaction and navigation tasks [[1]](https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/9605/9605103v1.pdf).\n",
       "\n",
       "3. **Control Systems:**\n",
       "   - **Adaptive Control:** RL is used in adaptive control systems where the goal is to improve a sequence of decisions from experience. This is particularly useful in dynamic systems where states and actions are vectors, and system dynamics are smooth [[3]](https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/9605/9605103v1.pdf).\n",
       "\n",
       "4. **Epidemic Modeling:**\n",
       "   - **Microscopic Multi-Agent Epidemic Model:** RL can be used to model epidemics by simulating individual agents' decisions that affect the spread of the disease. This approach helps in predicting the spread and identifying necessary external interventions to regulate behaviors [[4]](https://arxiv.org/pdf/2004.12959v1.pdf).\n",
       "\n",
       "These examples illustrate the versatility of reinforcement learning in tackling complex problems across different fields by enabling systems to learn optimal behaviors through interactions with their environments."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(chain_with_history.invoke({\"question\": QUESTION}, config=config))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "25dfc233-450f-4671-8f1c-0b446e46f048",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Your prior question was: \"Tell me some use cases for reinforcement learning.\""
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Remembers\n",
    "printmd(chain_with_history.invoke({\"question\": FOLLOW_UP_QUESTION},config=config))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "c67073c2-9a82-4e44-a9e2-48fe868c1634",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "You're welcome! If you have any more questions in the future, feel free to ask. Goodbye!"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Remembers\n",
    "printmd(chain_with_history.invoke({\"question\": \"Thank you! Good bye\"},config=config))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87405173",
   "metadata": {},
   "source": [
    "## Using CosmosDB as persistent memory\n",
    "\n",
    "In previous cell we have added local RAM memory to our chatbot. However, it is not persistent, it gets deleted once the app user's session is terminated. It is necessary then to use a Database for persistent storage of each of the bot user conversations, not only for Analytics and Auditing, but also if we wish to provide recommendations in the future. \n",
    "\n",
    "Here we will store the conversation history into CosmosDB for future auditing purpose.\n",
    "We will use a class in LangChain use CosmosDBChatMessageHistory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "d87cc7c6-5ef1-4492-b133-9f63a392e223",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the function to retrieve the conversation\n",
    "\n",
    "def get_session_history(session_id: str, user_id: str) -> CosmosDBChatMessageHistory:\n",
    "    cosmos = CosmosDBChatMessageHistory(\n",
    "        cosmos_endpoint=os.environ['AZURE_COSMOSDB_ENDPOINT'],\n",
    "        cosmos_database=os.environ['AZURE_COSMOSDB_NAME'],\n",
    "        cosmos_container=os.environ['AZURE_COSMOSDB_CONTAINER_NAME'],\n",
    "        connection_string=os.environ['AZURE_COMOSDB_CONNECTION_STRING'],\n",
    "        session_id=session_id,\n",
    "        user_id=user_id\n",
    "        )\n",
    "\n",
    "    # prepare the cosmosdb instance\n",
    "    cosmos.prepare_cosmos()\n",
    "    return cosmos\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "94f4179b-c1c7-49da-9c80-a42c275ed4d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain_with_history = RunnableWithMessageHistory(\n",
    "    chain,\n",
    "    get_session_history,\n",
    "    input_messages_key=\"question\",\n",
    "    history_messages_key=\"history\",\n",
    "    history_factory_config=[\n",
    "        ConfigurableFieldSpec(\n",
    "            id=\"user_id\",\n",
    "            annotation=str,\n",
    "            name=\"User ID\",\n",
    "            description=\"Unique identifier for the user.\",\n",
    "            default=\"\",\n",
    "            is_shared=True,\n",
    "        ),\n",
    "        ConfigurableFieldSpec(\n",
    "            id=\"session_id\",\n",
    "            annotation=str,\n",
    "            name=\"Session ID\",\n",
    "            description=\"Unique identifier for the conversation.\",\n",
    "            default=\"\",\n",
    "            is_shared=True,\n",
    "        ),\n",
    "    ],\n",
    ") | output_parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "8cf1f1f0-6e46-4136-9f33-4e46617b7d4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is where we configure the session id and user id\n",
    "random_session_id = \"session\"+ str(random.randint(1, 1000))\n",
    "ramdom_user_id = \"user\"+ str(random.randint(1, 1000))\n",
    "\n",
    "config={\"configurable\": {\"session_id\": random_session_id, \"user_id\": ramdom_user_id}}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "0b20c00c-4098-4970-84e5-f71ea7615c65",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'configurable': {'session_id': 'session57', 'user_id': 'user779'}}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "7e3c32f4-f883-4045-91f9-ca317c2d01fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Reinforcement learning (RL) has a wide range of applications across various domains due to its ability to learn optimal policies through interaction with the environment. Here are some notable use cases:\n",
       "\n",
       "1. **Games and Simulations:**\n",
       "   - **Backgammon:** Tesauro applied the temporal difference algorithm to backgammon, creating a program called TD-Gammon. This program used a neural network as a function approximator for the value function and was trained through self-play. Despite its simplistic exploration strategy, TD-Gammon achieved a high level of play, competing at the top level of international human play [[1]](https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/9605/9605103v1.pdf).\n",
       "   - **Chess:** Chess can be learned by reinforcement learning, although the learning rate can be impractically slow due to the sparse feedback (only knowing if a move was good or bad at the end of the game). A more practical method involves presenting example games and asking the system to make its own moves based on these examples [[2]](https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/0004/0004001v1.pdf).\n",
       "\n",
       "2. **Robotics and Control:**\n",
       "   - **Juggling Robot:** Schaal and Atkeson developed a two-armed robot that learns to juggle a devil-stick, a complex non-linear control task. The robot learned from experience and used a function approximation scheme known as locally weighted regression to generalize to unvisited states, improving its policy through dynamic programming techniques [[3]](https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/9605/9605103v1.pdf).\n",
       "   - **Mobile Robots:** Mahadevan and Connell discussed tasks where a mobile robot pushes large boxes for extended periods, showcasing the use of RL in physical tasks that require continuous learning and adaptation [[3]](https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/9605/9605103v1.pdf).\n",
       "\n",
       "3. **Healthcare:**\n",
       "   - **Epidemic Modeling:** A microscopic multi-agent epidemic model uses RL to determine optimal activity levels for individuals to minimize the spread of disease. This model can predict the spread of disease based on individual decisions and highlight the need for external interventions when infected agents do not have enough incentives to protect others [[4]](https://arxiv.org/pdf/2004.12959v1.pdf).\n",
       "\n",
       "4. **Adaptive Control Systems:**\n",
       "   - **Dynamic Systems:** In adaptive control, RL is used to improve a sequence of decisions from experience, especially in dynamic systems where states and actions are vectors and system dynamics are smooth. This approach is common in systems that require robust, practical algorithms for real-world deployment [[5]](https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/9605/9605103v1.pdf).\n",
       "\n",
       "These examples illustrate the versatility of reinforcement learning in solving complex problems across various fields by learning optimal policies through interaction with the environment and feedback mechanisms.\n",
       "\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(chain_with_history.invoke({\"question\": QUESTION}, config=config))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "7e29643b-a531-4117-8e85-9c88a625cf02",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Your prior question was: \"Tell me some use cases for reinforcement learning.\""
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Remembers\n",
    "printmd(chain_with_history.invoke({\"question\": FOLLOW_UP_QUESTION},config=config))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "50146f05-5ef6-484f-a8ec-9631643054f2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "We discussed various use cases of reinforcement learning, including applications in games, robotics, healthcare, and adaptive control systems."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Remembers\n",
    "printmd(chain_with_history.invoke(\n",
    "    {\"question\": \"Can you tell me a one line summary of our conversation?\"},\n",
    "    config=config))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "8bc02369-904c-4063-93e1-fff24fe6a3ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "You're welcome! If you have any more questions, feel free to ask. Have a great day!"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    printmd(chain_with_history.invoke(\n",
    "    {\"question\": \"Thank you very much!\"},\n",
    "    config=config))\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "87d60faa-1446-4c07-8970-0f9712c33b2f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "I provided a one-line summary of our conversation because you requested it in your previous message. If you need more detailed information or have another question, feel free to ask!"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(chain_with_history.invoke(\n",
    "    {\"question\": \"I do have one more question, why did you give me a one line summary?\"},\n",
    "    config=config))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "cfe748aa-6116-4a7a-97e6-f1c680dd23ad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "I apologize for any confusion. If you would like a more detailed summary, here it is in two lines:\n",
       "\n",
       "We discussed various use cases of reinforcement learning, including applications in games like backgammon and chess, robotics such as juggling robots and mobile robots, healthcare through epidemic modeling, and adaptive control systems. These examples illustrate the versatility of reinforcement learning in solving complex problems across various fields.\n",
       "\n",
       "Feel free to ask if you need more information or have any other questions!"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(chain_with_history.invoke(\n",
    "    {\"question\": \"why not 2?\"},\n",
    "    config=config))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdc5ac98",
   "metadata": {},
   "source": [
    "#### Let's check our Azure CosmosDB to see the whole conversation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5e30694-ae2a-47bb-a5c7-db51ecdbba1e",
   "metadata": {},
   "source": [
    "![CosmosDB Memory](./images/cosmos-chathistory.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6789cada-23a3-451a-a91a-0906ceb0bd14",
   "metadata": {},
   "source": [
    "# Summary\n",
    "##### Adding memory to our application allows the user to have a conversation, however this feature is not something that comes with the LLM, but instead, memory is something that we must provide to the LLM in form of context of the question.\n",
    "\n",
    "We added persitent memory using CosmosDB.\n",
    "\n",
    "We also can notice that the current chain that we are using is smart, but not that much. Although we have given memory to it, it searches for similar docs everytime, regardless of the input. This doesn't seem efficient, but regardless, we are very close to finish our first RAG-talk to your data Bot.\n",
    "\n",
    "\n",
    "## <u>Important Note</u>:<br>\n",
    "As we proceed, while all the code will remain compatible with GPT-3.5 models (1106 or newer), we highly recommend transitioning to GPT-4. Here's why:\n",
    "\n",
    "**GPT-3.5-Turbo** can be likened to a 7-year-old child. You can provide it with concise instructions, but it struggles sometimes to follow them accurately (not too reliable). Additionally, its limited \"memory\" (token context) can make sustained conversations challenging. Its response are also simple not deep.\n",
    "\n",
    "**GPT-4** exhibits the capabilities of a 10-12-year-old child. It possesses enhanced reasoning skills, consistently adheres to instructions and its answers are beter. It has extended memory retention (larger context size) for instructions, and it excels at following them. Its responses are deep and thorough.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c629ebf4-aced-45b7-a6a2-315810d37d48",
   "metadata": {},
   "source": [
    "# NEXT\n",
    "We know now how to do a Smart Search Engine that can power a chatbot!! great!\n",
    "\n",
    "In the next notebook 6, we are going to build our first RAG bot. In order to do this we will introduce the concept of Agents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f53a8f7a-5e28-4d5f-9a33-0a3be0536b0f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK v2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}