06-First-RAG.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "76fbaf88-5952-47bf-a68c-85011e49b6de",
   "metadata": {},
   "source": [
    "# Building our First RAG bot - Skill: talk to Search Engine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "967c3b06-c8a0-45db-be9a-974c762ba4b8",
   "metadata": {},
   "source": [
    "We have now all the building blocks to build our first Bot that \"talks with my data\". These blocks are:\n",
    "\n",
    "1) A well indexed hybrid (text and vector) engine with my data in chunks -> Azure AI Search\n",
    "2) A good LLM python framework to build LLM Apps -> LangChain\n",
    "3) Quality OpenAI GPT models that understand language and follow instructions -> GPT3.5 and GPT4\n",
    "4) A persisten memory database -> CosmosDB\n",
    "\n",
    "We are missing just one thing: **Agents**.\n",
    "\n",
    "In this Notebook we introduce the concept of Agents and we use it to build or first RAG bot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b64f701d-5b9d-4c7c-b259-c2a515c75961",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import random\n",
    "import asyncio\n",
    "from typing import Dict, List\n",
    "from concurrent.futures import ThreadPoolExecutor\n",
    "from typing import Optional, Type\n",
    "\n",
    "from langchain.agents import AgentExecutor, create_openai_tools_agent\n",
    "from langchain_openai import AzureChatOpenAI\n",
    "from langchain_core.runnables import ConfigurableField, ConfigurableFieldSpec\n",
    "from langchain_core.runnables.history import RunnableWithMessageHistory\n",
    "from langchain_core.chat_history import BaseChatMessageHistory\n",
    "from langchain_community.chat_message_histories import ChatMessageHistory, CosmosDBChatMessageHistory\n",
    "from langchain.callbacks.manager import AsyncCallbackManagerForToolRun, CallbackManagerForToolRun\n",
    "from langchain.pydantic_v1 import BaseModel, Field\n",
    "from langchain.tools import BaseTool, StructuredTool, tool\n",
    "\n",
    "#custom libraries that we will use later in the app\n",
    "from common.utils import  GetDocSearchResults_Tool\n",
    "from common.prompts import AGENT_DOCSEARCH_PROMPT\n",
    "\n",
    "from IPython.display import Markdown, HTML, display  \n",
    "\n",
    "def printmd(string):\n",
    "    display(Markdown(string))\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv(\"credentials.env\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e4163af7-39d0-43b4-8dad-c13108d22a1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the ENV variables that Langchain needs to connect to Azure OpenAI\n",
    "os.environ[\"OPENAI_API_VERSION\"] = os.environ[\"AZURE_OPENAI_API_VERSION\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33836104-822e-4846-8b81-0de8e24838f1",
   "metadata": {},
   "source": [
    "## Introducing: Agents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16fc3d38-93f8-4a47-8125-d1bb9f529178",
   "metadata": {},
   "source": [
    "The implementation of Agents is inspired by two papers: the [MRKL Systems](https://arxiv.org/abs/2205.00445) paper (pronounced ‘miracle’ 😉) and the [ReAct](https://arxiv.org/abs/2210.03629) paper.\n",
    "\n",
    "Agents are a way to leverage the ability of LLMs to understand and act on prompts. In essence, an Agent is an LLM that has been given a very clever initial prompt. The prompt tells the LLM to break down the process of answering a complex query into a sequence of steps that are resolved one at a time.\n",
    "\n",
    "Agents become really cool when we combine them with ‘experts’, introduced in the MRKL paper. Simple example: an Agent might not have the inherent capability to reliably perform mathematical calculations by itself. However, we can introduce an expert - in this case a calculator, an expert at mathematical calculations. Now, when we need to perform a calculation, the Agent can call in the expert rather than trying to predict the result itself. This is actually the concept behind [ChatGPT Pluggins](https://openai.com/blog/chatgpt-plugins).\n",
    "\n",
    "In our case, in order to solve the problem \"How do I build a smart bot that talks to my data\", we need this REACT/MRKL approach, in which we need to instruct the LLM that it needs to use 'experts/tools' in order to read/load/understand/interact with a any particular source of data.\n",
    "\n",
    "Let's create then an Agent that interact with the user and uses a Tool to get the information from the Search engine."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7999a06-aff0-4d21-8be7-fe56c70082a8",
   "metadata": {},
   "source": [
    "#### 1. We start first defining the Tool/Expert\n",
    "\n",
    "Tools are functions that an agent can invoke. If you don't give the agent access to a correct set of tools, it will never be able to accomplish the objectives you give it. If you don't describe the tools well, the agent won't know how to use them properly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a862366b-ce9e-44f8-9610-84ec568653ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "index1_name = \"srch-index-files\"\n",
    "index2_name = \"srch-index-csv\"\n",
    "index3_name = \"srch-index-books\"\n",
    "indexes = [index1_name, index2_name, index3_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "077886c8-c5d0-481d-a5f9-f4becf60e0f9",
   "metadata": {},
   "source": [
    "We have to convert the Retreiver object into a Tool object (\"the expert\"). Check out the Tool `GetDocSearchResults_Tool` in `utils.py` and see how it is done."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f73c6ca7-d93b-4961-b90a-08572cad78d8",
   "metadata": {},
   "source": [
    "Declare the tools the agent will use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4a0fd3a0-527c-42e3-a092-46e03d33bd07",
   "metadata": {},
   "outputs": [],
   "source": [
    "tools = [GetDocSearchResults_Tool(indexes=indexes, k=5, reranker_th=1, sas_token=os.environ['BLOB_SAS_TOKEN'])]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f3ddf18-3f3c-44b4-8af5-1437973da010",
   "metadata": {},
   "source": [
    "#### 2. Define the LLM to use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5aaaf7f5-ef26-48d8-868d-b53aa4c4f9f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "COMPLETION_TOKENS = 1500\n",
    "llm = AzureChatOpenAI(deployment_name=os.environ[\"GPT35_DEPLOYMENT_NAME\"], \n",
    "                      temperature=0.5, max_tokens=COMPLETION_TOKENS, streaming=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d865755b-e4bb-468a-8dcc-4ac1999782b3",
   "metadata": {},
   "source": [
    "#### 3. Bind tools to the LLM"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec61b209-1c1e-48ff-957e-1ec2e375ada4",
   "metadata": {},
   "source": [
    "Newer OpenAI models (1106 and newer) have been fine-tuned to detect when one or more function(s) should be called and respond with the inputs that should be passed to the function(s). In an API call, you can describe functions and have the model intelligently choose to output a JSON object containing arguments to call these functions. The goal of the OpenAI tools APIs is to more reliably return valid and useful function calls than what can be done using a generic text completion or chat API.\n",
    "\n",
    "OpenAI termed the capability to invoke a single function as **functions**, and the capability to invoke one or more functions as [**tools**](https://platform.openai.com/docs/guides/function-calling).\n",
    "\n",
    "> OpenAI API has deprecated functions in favor of tools. The difference between the two is that the tools API allows the model to request that multiple functions be invoked at once, which can reduce response times in some architectures. It’s recommended to use the tools agent for OpenAI models.\n",
    "\n",
    "Having an LLM call multiple tools at the same time can greatly speed up agents whether there are tasks that are assisted by doing so. Thankfully, OpenAI models versions 1106 and newer support parallel function calling, which we will need to make sure our smart bot is performant.\n",
    "\n",
    "##### **From now on and for the rest of the notebooks, we are going to use OpenAI tools API tool call our experts/tools**\n",
    "\n",
    "To pass in our tools to the agent, we just need to format them to the [OpenAI tool format](https://platform.openai.com/docs/api-reference/chat/create) and pass them to our model. (By bind-ing the functions, we’re making sure that they’re passed in each time the model is invoked.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "856361f5-87b5-46f0-a0a6-ce3c1566ff48",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Bind (attach) the tools/functions we want on each LLM call\n",
    "\n",
    "llm_with_tools = llm.bind_tools(tools)\n",
    "\n",
    "# Let's also add the option to configure in real time the model we want\n",
    "\n",
    "llm_with_tools = llm_with_tools.configurable_alternatives(\n",
    "    ConfigurableField(id=\"model\"),\n",
    "    default_key=\"gpt35\",\n",
    "    gpt4=AzureChatOpenAI(deployment_name=os.environ[\"GPT4_DEPLOYMENT_NAME\"], temperature=0.5, max_tokens=COMPLETION_TOKENS, streaming=True),\n",
    "    gpt4o=AzureChatOpenAI(deployment_name=os.environ[\"GPT4o_DEPLOYMENT_NAME\"], temperature=0.5, max_tokens=COMPLETION_TOKENS, streaming=True) \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "330c64bd-89ca-494e-8c01-f948f9a3e6a7",
   "metadata": {},
   "source": [
    "#### 4. Define the System Prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30901f95-3bf9-4aaa-9eda-226edbf5ea00",
   "metadata": {},
   "source": [
    "Because OpenAI Function Calling is finetuned for tool usage, we hardly need any instructions on how to reason, or how to output format. We will just have two input variables: `question` and `agent_scratchpad`. The input variable `question` should be a string containing the user objective, and `agent_scratchpad` should be a sequence of messages that contains the previous agent tool invocations and the corresponding tool outputs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9cac295-8be5-4803-8342-6d4e48cd2294",
   "metadata": {},
   "source": [
    "Get the prompt to use `AGENT_DOCSEARCH_PROMPT` - you can modify this in `prompts.py`! Check it out!\n",
    "It looks like this:\n",
    "\n",
    "```python\n",
    "AGENT_DOCSEARCH_PROMPT = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", CUSTOM_CHATBOT_PREFIX  + DOCSEARCH_PROMPT_TEXT),\n",
    "        MessagesPlaceholder(variable_name='history', optional=True),\n",
    "        (\"human\", \"{question}\"),\n",
    "        MessagesPlaceholder(variable_name='agent_scratchpad')\n",
    "    ]\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a44f8df6-a68e-4215-99f3-10119f796c0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = AGENT_DOCSEARCH_PROMPT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "581ad422-c06b-434f-bff0-e2a3d6093932",
   "metadata": {},
   "source": [
    "#### 5. Create the agent"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3519b70c-007d-405c-9a81-18f58c5617be",
   "metadata": {},
   "source": [
    "The core idea of agents is to use a language model to choose a sequence of actions to take. In chains, a sequence of actions is hardcoded (in code). In agents, a language model is used as a reasoning engine to determine which actions to take and in which order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "16be0ef1-dc72-49fa-8aa7-cdd2153ef8b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.agents.format_scratchpad.openai_tools import format_to_openai_tool_messages\n",
    "from langchain.agents.output_parsers.openai_tools import OpenAIToolsAgentOutputParser\n",
    "\n",
    "agent = (\n",
    "    {\n",
    "        \"question\": lambda x: x[\"question\"],\n",
    "        \"agent_scratchpad\": lambda x: format_to_openai_tool_messages(x[\"intermediate_steps\"]),\n",
    "    }\n",
    "    | prompt\n",
    "    | llm_with_tools\n",
    "    | OpenAIToolsAgentOutputParser()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d87d9a8b-2a93-4250-b1dc-b124fa8c7ffa",
   "metadata": {},
   "source": [
    "Or , which is equivalent, LangChain has a class that does exactly the cell code above: `create_openai_tools_agent`\n",
    "\n",
    "```python\n",
    "agent = create_openai_tools_agent(llm, tools, prompt)\n",
    "```\n",
    "\n",
    "**Important Note: Other models like Mistral Large or Command R+ won't work with the same OpenAI Tools API, so in order to create agents with these models, try using the ReAct type instead from langchain**. Like [THIS COHERE AGENT](https://python.langchain.com/docs/integrations/providers/cohere/#react-agent) for example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "338336d9-a64a-4602-908a-742b418e4520",
   "metadata": {},
   "source": [
    "Create an agent executor by passing in the agent and tools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ad6c156f-9a17-4daa-80de-70ce2f55063b",
   "metadata": {},
   "outputs": [],
   "source": [
    "agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "252a017c-3b36-43ab-8633-78f4f005d166",
   "metadata": {},
   "source": [
    "Give it memory - since AgentExecutor is also a Runnable class, we do the same with did on Notebook 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7c013314-afe6-4218-b179-d0f7312d2670",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_session_history(session_id: str, user_id: str) -> CosmosDBChatMessageHistory:\n",
    "    cosmos = CosmosDBChatMessageHistory(\n",
    "        cosmos_endpoint=os.environ['AZURE_COSMOSDB_ENDPOINT'],\n",
    "        cosmos_database=os.environ['AZURE_COSMOSDB_NAME'],\n",
    "        cosmos_container=os.environ['AZURE_COSMOSDB_CONTAINER_NAME'],\n",
    "        connection_string=os.environ['AZURE_COMOSDB_CONNECTION_STRING'],\n",
    "        session_id=session_id,\n",
    "        user_id=user_id\n",
    "        )\n",
    "\n",
    "    # prepare the cosmosdb instance\n",
    "    cosmos.prepare_cosmos()\n",
    "    return cosmos"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13df017f-3ab7-4943-adc1-3477badf3d3e",
   "metadata": {},
   "source": [
    "Because cosmosDB needs two fields (an id and a partition), and RunnableWithMessageHistory takes by default only one identifier for memory (session_id), we need to use `history_factory_config` parameter and define the multiple keys for the memory class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "bf93758f-da3b-48fb-9882-91fe327b1751",
   "metadata": {},
   "outputs": [],
   "source": [
    "userid_spec = ConfigurableFieldSpec(\n",
    "            id=\"user_id\",\n",
    "            annotation=str,\n",
    "            name=\"User ID\",\n",
    "            description=\"Unique identifier for the user.\",\n",
    "            default=\"\",\n",
    "            is_shared=True,\n",
    "        )\n",
    "session_id = ConfigurableFieldSpec(\n",
    "            id=\"session_id\",\n",
    "            annotation=str,\n",
    "            name=\"Session ID\",\n",
    "            description=\"Unique identifier for the conversation.\",\n",
    "            default=\"\",\n",
    "            is_shared=True,\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "52d1aaa6-efca-4512-b680-896dae39a359",
   "metadata": {},
   "outputs": [],
   "source": [
    "agent_with_chat_history = RunnableWithMessageHistory(\n",
    "    agent_executor,\n",
    "    get_session_history,\n",
    "    input_messages_key=\"question\",\n",
    "    history_messages_key=\"history\",\n",
    "    history_factory_config=[userid_spec,session_id]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "05c6b489-3db9-4965-9eae-ed2790e62bd7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'configurable': {'session_id': 'session808', 'user_id': 'user139'}}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# configure the session id and user id\n",
    "random_session_id = \"session\"+ str(random.randint(1, 1000))\n",
    "ramdom_user_id = \"user\"+ str(random.randint(1, 1000))\n",
    "\n",
    "config={\"configurable\": {\"session_id\": random_session_id, \"user_id\": ramdom_user_id}}\n",
    "config"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3295c54e-a5e2-46f6-99fc-6f76453a877d",
   "metadata": {},
   "source": [
    "#### 6.Run the Agent!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "2ac81763-6bcc-4408-9daf-d047a0e2cb08",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 625 ms, sys: 10.1 ms, total: 635 ms\n",
      "Wall time: 16.6 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'question': 'I would like to know what is NLP?',\n",
       " 'history': [],\n",
       " 'output': 'Natural Language Processing (NLP) is a field of computer science that focuses on enabling computers to use human languages both as input and output. It involves creating machines that can understand and generate human language. NLP is broad and encompasses various applications, including simultaneous multi-language translation, advanced search engine development, and designing computer interfaces that can combine speech, diagrams, and other modalities simultaneously[[1]](https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/0304/0304027v1.pdf?sv=2022-11-02&ss=b&srt=sco&sp=rl&se=2026-07-09T13:52:04Z&st=2024-07-09T05:52:04Z&spr=https&sig=9Nx31tWOzf6CWylUZnGaciT9VDWVJSJ9vQulMcshm7Q%3D).\\n\\nThe field of NLP integrates ideas from various disciplines, including linguistics, psychology, information theory, mathematics, and statistics. It combines these concepts with advancements in machine learning to make computers language-enabled by acquiring linguistic information directly from language samples. This approach is known as statistical natural language processing, where computers learn language patterns and structures from data[[1]](https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/0304/0304027v1.pdf?sv=2022-11-02&ss=b&srt=sco&sp=rl&se=2026-07-09T13:52:04Z&st=2024-07-09T05:52:04Z&spr=https&sig=9Nx31tWOzf6CWylUZnGaciT9VDWVJSJ9vQulMcshm7Q%3D).\\n\\nNLP faces challenges due to the complexity of human language. Understanding human language involves dealing with ambiguity, context, and nuances that make it a difficult problem for computers. Despite advancements in technology, achieving a level of language understanding comparable to human capabilities remains a significant challenge in the field of NLP[[1]](https://datasetsgptsmartsearch.blob.core.windows.net/arxivcs/pdf/0304/0304027v1.pdf?sv=2022-11-02&ss=b&srt=sco&sp=rl&se=2026-07-09T13:52:04Z&st=2024-07-09T05:52:04Z&spr=https&sig=9Nx31tWOzf6CWylUZnGaciT9VDWVJSJ9vQulMcshm7Q%3D).\\n\\nIn the healthcare domain, NLP plays a crucial role in extracting information from free-text data sources such as electronic health records. It helps in improving documentation, quality, and efficiency of healthcare by enhancing the availability and utility of clinical information. NLP has shown promise in tasks like determining appropriate colonoscopy intervals, identifying cases of inflammatory bowel disease, and processing symptom information documented in electronic health records[[2]](https://doi.org/10.1016/j.cgh.2014.05.013) [[3]](https://doi.org/10.1093/jamia/ocy173) [[4]](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7148103/).'}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "agent_with_chat_history.invoke({\"question\": \"I would like to know what is NLP?\"}, config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "cb3fca7e-33a1-40f1-afb0-dee441a1d1d5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Markov chains are mathematical systems that transition between different states according to certain probabilistic rules. In the context of interacting Markov Chains, these chains are not mutually independent but interact with each other in some way, often when agents collectively perform tasks or aim to achieve a goal. This concept is utilized in various computational models based on distributed computation or agent systems, including machine learning, computer science, and models of economic behavior. Interacting Markov Chains provide a flexible framework that can be extended to account for various types of interactions, such as heterogenous agents, delays in communication, rational and irrational agents, among others, making it a promising conceptual framework for analyzing complex systems[[1]](https://doi.org/10.1111/ina.12056).\n",
       "\n",
       "In the medical field, one application of Markov chains is in predicting transient particle transport in enclosed environments, which is critical for reducing infection risks to occupants. A combined computational fluid dynamics (CFD) and Markov chain method has been developed to quickly predict particle transport. This method involves calculating a transition probability matrix using CFD simulations and then applying the Markov chain technique to predict transient particle concentration distributions. The Markov chain method can provide faster-than-real-time information about particle transport in enclosed environments and is particularly useful for scenarios like particle transport in clean rooms, office environments with air distribution systems, or aircraft cabins[[2]](https://www.ncbi.nlm.nih.gov/pubmed/23789964).\n",
       "\n",
       "Therefore, Markov chains have practical applications in medicine, particularly in understanding particle transport dynamics in enclosed environments, which is crucial for infection control and risk reduction."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(agent_with_chat_history.invoke(\n",
    "    {\"question\": \"What are markov chains and is there an application in medicine?\"}, \n",
    "    config=config)[\"output\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c430c456-f390-4319-a3b1-bee19da130cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "Markov chains are a valuable tool for modeling the spread of viruses, especially in understanding infectious disease dynamics. Here are some insights from the retrieved documents on the use of Markov chains in the context of virus spread:\n",
       "\n",
       "1. **Spatial Markov Chain Model**:\n",
       "   - A Spatial Markov Chain model is utilized for the spread of viruses, where nodes represent humans connected by a graph based on interpersonal contact intensity. The infectious transfer between individuals is determined by chance, and the model is extended to incorporate various lockdown scenarios[[1]](https://arxiv.org/pdf/2004.05635v1.pdf?sv=2022-11-02&ss=b&srt=sco&sp=rl&se=2026-07-09T13:52:04Z&st=2024-07-09T05:52:04Z&spr=https&sig=9Nx31tWOzf6CWylUZnGaciT9VDWVJSJ9vQulMcshm7Q%3D).\n",
       "\n",
       "2. **Continuous-Time Markov Chain (CTMC) Model**:\n",
       "   - Stochastic epidemic models with two groups are formulated using a continuous-time Markov chain (CTMC) model to study disease emergence or re-emergence. The transmission rates in the model depend on either the infectious host or the susceptible host. This model is applied to diseases like Severe Acute Respiratory Syndrome (SARS) and measles[[2]](https://doi.org/10.1080/17513758.2018.1538462; https://www.ncbi.nlm.nih.gov/pubmed/30381000/?sv=2022-11-02&ss=b&srt=sco&sp=rl&se=2026-07-09T13:52:04Z&st=2024-07-09T05:52:04Z&spr=https&sig=9Nx31tWOzf6CWylUZnGaciT9VDWVJSJ9vQulMcshm7Q%3D).\n",
       "\n",
       "3. **Nonlinear Markov Chains Model for Covid-19**:\n",
       "   - A nonlinear Markov chains model is proposed to analyze and understand the behavior of the Covid-19 pandemic. This model is used to estimate daily new Covid-19 cases in various countries and examine the correlation between daily new cases and daily deaths[[3]](http://medrxiv.org/cgi/content/short/2020.04.21.20073668v1?rss=1?sv=2022-11-02&ss=b&srt=sco&sp=rl&se=2026-07-09T13:52:04Z&st=2024-07-09T05:52:04Z&spr=https&sig=9Nx31tWOzf6CWylUZnGaciT9VDWVJSJ9vQulMcshm7Q%3D).\n",
       "\n",
       "4. **Bayesian Markov Chain Monte Carlo Simulation**:\n",
       "   - Bayesian Markov chain Monte Carlo simulation methods are used to develop models for the trend of the Coronavirus disease 2019 pandemic in Lebanon. Different models are compared in terms of their predictive ability, with a Poisson autoregressive model showing the best performance in capturing short and long-term memory effects[[4]](http://medrxiv.org/cgi/content/short/2020.04.29.20082263v1?rss=1?sv=2022-11-02&ss=b&srt=sco&sp=rl&se=2026-07-09T13:52:04Z&st=2024-07-09T05:52:04Z&spr=https&sig=9Nx31tWOzf6CWylUZnGaciT9VDWVJSJ9vQulMcshm7Q%3D).\n",
       "\n",
       "These applications demonstrate the versatility and effectiveness of Markov chains in modeling and analyzing the spread of viruses, providing valuable insights for understanding and managing infectious disease outbreaks."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "printmd(agent_with_chat_history.invoke(\n",
    "        {\"question\": \"Interesting, Tell me more about the use of markov chains, specifically in the spread of viruses\"},\n",
    "        config=config)[\"output\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9fd54f71-03c9-4332-885b-0d1df942fa88",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "You're welcome! If you have any more questions or need assistance with anything else, feel free to ask."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 168 ms, sys: 0 ns, total: 168 ms\n",
      "Wall time: 4.38 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "printmd(agent_with_chat_history.invoke({\"question\": \"Thhank you!\"}, config=config)[\"output\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "149648ba-945d-4e7d-81f7-a8bca2ac87f2",
   "metadata": {},
   "source": [
    "#### Important: there is a limitation of GPT3.5, once we start adding long prompts, and long contexts and thorough answers, or the agent makes multiple searches for multi-step questions, we run out of space (number of tokens)!\n",
    "\n",
    "You can minimize this by:\n",
    "- Shorter System Prompt\n",
    "- Smaller chunks (less than the default of 5000 characters)\n",
    "- Reducing topK to bring less relevant chunks\n",
    "\n",
    "However, you ultimately are sacrificing quality to make everything work with GPT3.5 (cheaper and faster model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41787714-73fd-4336-85f2-bec3abb41eda",
   "metadata": {},
   "source": [
    "### Let's add more things we have learned so far: dynamic LLM selection of GPT4 and asyncronous streaming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "1511d2c3-97fe-4232-a560-014d0f157008",
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = create_openai_tools_agent(llm_with_tools.with_config(configurable={\"model\": \"gpt4o\"}), tools, prompt) # We select now GPT-4o\n",
    "agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=False)\n",
    "agent_with_chat_history = RunnableWithMessageHistory(agent_executor,get_session_history,input_messages_key=\"question\", \n",
    "                                                     history_messages_key=\"history\", history_factory_config=[userid_spec,session_id])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bec5b32-6017-44b9-97e7-34ba3695e688",
   "metadata": {},
   "source": [
    "In prior notebooks with use the function `.stream()` of the runnable in order to stream the tokens. However if you need to stream individual tokens from the agent or surface steps occuring within tools, you would need to use a combination of `Callbacks` and `.astream()` OR the new `astream_events` API (beta).\n",
    "\n",
    "Let’s use here the astream_events API to stream the following events:\n",
    "\n",
    "    Agent Start with inputs\n",
    "    Tool Start with inputs\n",
    "    Tool End with outputs\n",
    "    Stream the agent final anwer token by token\n",
    "    Agent End with outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "9600a35e-8d2e-43d0-a334-092b2e8b832c",
   "metadata": {},
   "outputs": [],
   "source": [
    "QUESTION = \"Tell me more about your last answer, search again multiple times and provide a deeper explanation\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "3808fa33-05bb-4f5d-9ab9-7159f6db62a8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting agent: AgentExecutor\n",
      "--\n",
      "Starting tool: docsearch with inputs: {'query': 'Markov chains virus spread'}\n",
      "--\n",
      "Starting tool: docsearch with inputs: {'query': 'Markov chains infectious disease modeling'}\n",
      "--\n",
      "Starting tool: docsearch with inputs: {'query': 'Markov chains epidemiology'}\n",
      "Done tool: docsearch\n",
      "--\n",
      "Done tool: docsearch\n",
      "--\n",
      "Done tool: docsearch\n",
      "--\n",
      "Markov chains have several applications in modeling the spread of viruses and infectious diseases. Here are deeper insights based on the retrieved documents:\n",
      "\n",
      "### Spatial Markov Chain Model\n",
      "A Spatial Markov Chain model is used to represent the spread of viruses among humans connected by a graph. In this model, nodes represent individuals, and edges represent the intensity of interpersonal contacts. The likelihood of infection spread is determined probabilistically, and the model can incorporate various lockdown scenarios to simulate different intervention strategies[[1]](https://arxiv.org/pdf/2004.05635v1.pdf?sv=2022-11-02&ss=b&srt=sco&sp=rl&se=2026-07-09T13:52:04Z&st=2024-07-09T05:52:04Z&spr=https&sig=9Nx31tWOzf6CWylUZnGaciT9VDWVJSJ9vQulMcshm7Q%3D).\n",
      "\n",
      "### Nonlinear Markov Chains for COVID-19\n",
      "A nonlinear Markov chains model has been proposed to analyze and understand the behavior of the COVID-19 pandemic. This model uses data to estimate daily new COVID-19 cases and examines the correlation between new cases and deaths in various countries. This approach helps in understanding the progression of the pandemic and the effectiveness of interventions[[2]](http://medrxiv.org/cgi/content/short/2020.04.21.20073668v1?rss=1?sv=2022-11-02&ss=b&srt=sco&sp=rl&se=2026-07-09T13:52:04Z&st=2024-07-09T05:52:04Z&spr=https&sig=9Nx31tWOzf6CWylUZnGaciT9VDWVJSJ9vQulMcshm7Q%3D).\n",
      "\n",
      "### Combined CFD and Markov Chain Method\n",
      "To predict airborne infectious disease transmission in enclosed environments, a combined computational fluid dynamics (CFD) and Markov chain method has been developed. This method calculates a transition probability matrix using CFD simulations and applies the Markov chain technique to predict the transient particle concentration distributions. This approach provides faster-than-real-time information about particle transport, which is crucial for infection control in environments like clean rooms, offices, and aircraft cabins[[3]](https://doi.org/10.1111/ina.12056; https://www.ncbi.nlm.nih.gov/pubmed/23789964/?sv=2022-11-02&ss=b&srt=sco&sp=rl&se=2026-07-09T13:52:04Z&st=2024-07-09T05:52:04Z&spr=https&sig=9Nx31tWOzf6CWylUZnGaciT9VDWVJSJ9vQulMcshm7Q%3D).\n",
      "\n",
      "### Continuous-Time Markov Chain (CTMC) Model\n",
      "Stochastic epidemic models using a continuous-time Markov chain (CTMC) approach are applied to study emerging and re-emerging infectious diseases. These models consider the dynamics of disease transmission, including the role of superspreaders and waning immunity. The CTMC model helps estimate the probability of a major epidemic based on the initial conditions and the characteristics of the infectious and susceptible hosts[[4]](https://doi.org/10.1080/17513758.2018.1538462; https://www.ncbi.nlm.nih.gov/pubmed/30381000/?sv=2022-11-02&ss=b&srt=sco&sp=rl&se=2026-07-09T13:52:04Z&st=2024-07-09T05:52:04Z&spr=https&sig=9Nx31tWOzf6CWylUZnGaciT9VDWVJSJ9vQulMcshm7Q%3D).\n",
      "\n",
      "### Markov Chain Monte Carlo (MCMC) Methods\n",
      "Markov Chain Monte Carlo (MCMC) methods are used to estimate epidemiological parameters and model the spread of infectious diseases. For example, MCMC has been applied to the 2001 foot and mouth disease (FMD) epidemic in Great Britain to estimate transmission parameters and assess the impact of control measures. This approach provides predictive risk maps and insights into the transmission potential in different geographic areas[[5]](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1876810/?sv=2022-11-02&ss=b&srt=sco&sp=rl&se=2026-07-09T13:52:04Z&st=2024-07-09T05:52:04Z&spr=https&sig=9Nx31tWOzf6CWylUZnGaciT9VDWVJSJ9vQulMcshm7Q%3D).\n",
      "\n",
      "### Bayesian Markov Chain Monte Carlo (MCMC) Simulation\n",
      "Bayesian MCMC simulation methods are used to model the trend of pandemics, such as COVID-19. These methods allow for the estimation of model parameters and the approximation of unobserved counts from daily reported data. This approach helps in understanding the progression of the pandemic and the effectiveness of interventions[[6]](https://arxiv.org/pdf/2005.04500v1.pdf?sv=2022-11-02&ss=b&srt=sco&sp=rl&se=2026-07-09T13:52:04Z&st=2024-07-09T05:52:04Z&spr=https&sig=9Nx31tWOzf6CWylUZnGaciT9VDWVJSJ9vQulMcshm7Q%3D).\n",
      "\n",
      "### Discrete Time Markov Chain Simulator\n",
      "A discrete-time Markov chain-based simulator has been developed to model various epidemic scenarios, including the SEQIJR (susceptible, exposed, quarantined, infected, isolated, and recovered) model. This simulator is designed to test different control algorithms and is compatible with network-based epidemic simulators. It provides a computationally efficient way to reproduce different epidemic behaviors[[7]](https://doi.org/10.1109/embc.2016.7591271; https://www.ncbi.nlm.nih.gov/pubmed/28227061/?sv=2022-11-02&ss=b&srt=sco&sp=rl&se=2026-07-09T13:52:04Z&st=2024-07-09T05:52:04Z&spr=https&sig=9Nx31tWOzf6CWylUZnGaciT9VDWVJSJ9vQulMcshm7Q%3D).\n",
      "\n",
      "These applications highlight the versatility and effectiveness of Markov chains in understanding and managing the spread of infectious diseases, providing valuable insights for public health interventions and policy-making.\n",
      "--\n",
      "Done agent: AgentExecutor\n"
     ]
    }
   ],
   "source": [
    "async for event in agent_with_chat_history.astream_events(\n",
    "    {\"question\": QUESTION}, config=config, version=\"v1\",\n",
    "):\n",
    "    kind = event[\"event\"]\n",
    "    if kind == \"on_chain_start\":\n",
    "        if (event[\"name\"] == \"AgentExecutor\"):\n",
    "            print( f\"Starting agent: {event['name']}\")\n",
    "    elif kind == \"on_chain_end\":\n",
    "        if (event[\"name\"] == \"AgentExecutor\"):  \n",
    "            print()\n",
    "            print(\"--\")\n",
    "            print(f\"Done agent: {event['name']}\")\n",
    "    if kind == \"on_chat_model_stream\":\n",
    "        content = event[\"data\"][\"chunk\"].content\n",
    "        # Empty content in the context of OpenAI means that the model is asking for a tool to be invoked.\n",
    "        # So we only print non-empty content\n",
    "        if content:\n",
    "            print(content, end=\"\")\n",
    "    elif kind == \"on_tool_start\":\n",
    "        print(\"--\")\n",
    "        print(f\"Starting tool: {event['name']} with inputs: {event['data'].get('input')}\")\n",
    "    elif kind == \"on_tool_end\":\n",
    "        print(f\"Done tool: {event['name']}\")\n",
    "        # print(f\"Tool output was: {event['data'].get('output')}\")\n",
    "        print(\"--\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b41bba7-18df-4ab8-b4f6-60368160d348",
   "metadata": {},
   "source": [
    "#### Note: Try to run this last question with GPT3.5 and see how you are going to run out of token space in the LLM"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0ec64bf-fe24-42fc-8dde-4d478f0af21e",
   "metadata": {},
   "source": [
    "# Summary\n",
    "\n",
    "We just built our first RAG BOT!.\n",
    "\n",
    "- We learned that **Agents + Tools are the best way to go about building Bots**. <br>\n",
    "- We converted the Azure Search retriever into a Tool using the function `GetDocSearchResults_Tool` in `utils.py`\n",
    "- We learned about the events API (Beta), one way to stream the answer from agents\n",
    "- We learned that for comprehensive, quality answers we will run out of space with GPT3.5. GPT4 then becomes necessary.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56306506-d53d-4d43-93e2-a9300ed2a3ee",
   "metadata": {},
   "source": [
    "# NEXT\n",
    "\n",
    "Now that we have a bot with one skill (Document Search), let's build more skills!. In the next Notebook, we are going to build an agent that can understand tabular data in csv file and can execute python commands"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK v2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}