multihop_langchain_frames_benchmark.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 🦾 👩‍🦰 Multihop agent and human in Langchain with Argilla\n",
    "\n",
    "In this recipe we will use the [frames-benchmark dataset](https://huggingface.co/datasets/google/frames-benchmark) to evaluate a multi-hop agent in Langchain with Argilla. The agent will be able to answer questions that require multiple steps to be answered. We'll review the responses in the Argilla UI.\n",
    "\n",
    "The agent will be able to retrieve similar examples that have been reviewed in the Argilla UI by human reviewers. The agent will be able to use the information from these examples to answer the questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install --qqq wikipedia \\\n",
    "    langchain-community \\\n",
    "    langchain_openai \\\n",
    "    argilla"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Langchain tool for Wikipedia\n",
    "\n",
    "We will use a community tool to retrieve wikipedia pages that help our agent answer questions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.tools import WikipediaQueryRun\n",
    "from langchain_community.utilities import WikipediaAPIWrapper\n",
    "\n",
    "wikipedia = WikipediaQueryRun(api_wrapper=WikipediaAPIWrapper())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can run the wikipedia tool to look up the page for the Python programming language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Page: Python (programming language)\\nSummary: Python is a high-level, general-purpose programming language. Its design philosophy emphasizes code readability with the use of significant indentation.\\nPython is dynamically typed and garbage-collected. It supports multiple programming paradigms, including structured (particularly procedural), object-oriented and functional programming. It is often described as a \"batteries included\" language due to its comprehensive standard library.\\nGuido van Rossum began working on Python in the late 1980s as a successor to the ABC programming language and first released it in 1991 as Python 0.9.0. Python 2.0 was released in 2000. Python 3.0, released in 2008, was a major revision not completely backward-compatible with earlier versions. Python 2.7.18, released in 2020, was the last release of Python 2.\\nPython consistently ranks as one of the most popular programming languages, and has gained widespread use in the machine learning community.\\n\\n\\n\\nPage: History of Python\\nSummary: The programming language Python was conceived in the late 1980s, and its implementation was started in December 1989 by Guido van Rossum at CWI in the Netherlands as a successor to ABC capable of exception handling and interfacing with the Amoeba operating system. Van Rossum is Python\\'s principal author, and his continuing central role in deciding the direction of Python is reflected in the title given to him by the Python community, Benevolent Dictator for Life (BDFL). (However, Van Rossum stepped down as leader on July 12, 2018.). Python was named after the BBC TV show Monty Python\\'s Flying Circus.\\nPython 2.0 was released on October 16, 2000, with many major new features, including a cycle-detecting garbage collector (in addition to reference counting) for memory management and support for Unicode, along with a change to the development process itself, with a shift to a more transparent and community-backed process.\\nPython 3.0, a major, backwards-incompatible release, was released on December 3, 2008 after a long period of testing. Many of its major features have also been backported to the backwards-compatible, though now-unsupported, Python 2.6 and 2.7.\\n\\n\\n\\nPage: Mojo (programming language)\\nSummary: Mojo is a programming language in the Python family that is currently under development. It is available both in browsers via Jupyter notebooks, and locally on Linux and macOS. Mojo aims to combine the usability of higher level programming languages, specifically Python, with the performance of system programming languages like C++, Rust, and Zig. The Mojo compiler is currently closed source with an open source standard library, although Modular, the company behind Mojo, has stated their intent to eventually open source the Mojo programming language itself as it matures.\\nMojo builds upon the MLIR compiler framework instead of directly on the lower level LLVM compiler framework that many languages like Julia, Swift, clang and Rust do. MLIR is a newer compiler framework that allows Mojo to take advantage of higher level compiler passes not available in LLVM alone and allows Mojo to compile down and target more than just CPUs, including producing code that can run on GPUs, TPUs, ASICs and other accelerators. It can also often more effectively use certain types of CPU optimizations directly, like SIMD without direct intervention by the developer like in many other languages. According to Jeremy Howard of fast.ai, Mojo can be seen as \"syntax sugar for MLIR\" and for that reason Mojo is well optimized for applications like AI.'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wikipedia.run(\"Python_(programming_language)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Langchain prompt for ReAct agent\n",
    "\n",
    "\n",
    "A ReAct Agent is a type of prompting strategy that responds by using tools through trial and error. It attempts different actions, sees what happens, and adjusts its behavior based on the outcome. The “react” part refers to the agent’s ability to react to the environment’s responses, rather than just following a pre-programmed plan. By doing so, the agent can learn to achieve its goals more efficiently and adapt to changing situations, much like humans and animals do.\n",
    "\n",
    "ReAct agents were release in the paper [ReAct: A Framework for Agent-Centric Learning from Demonstrations](https://arxiv.org/abs/2210.03629) and are available in Langchain, and are implemented within [Langchain](https://python.langchain.com/v0.1/docs/modules/agents/agent_types/react/).\n",
    "\n",
    "We can also just pull a ReAct prompt from the Langchain hub and use it in our code. Here is an example of a ReAct prompt for the frames-benchmark dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ben/code/argilla-cookbook/.venv/lib/python3.12/site-packages/langsmith/client.py:323: LangSmithMissingAPIKeyWarning: API key must be provided when using hosted LangSmith API\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "from langchain import hub\n",
    "from langchain.agents import AgentExecutor, create_react_agent\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "tools = [wikipedia]\n",
    "\n",
    "# Get the prompt to use - you can modify this!\n",
    "prompt = hub.pull(\"hwchase17/react\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create a Langchain agent using Llama 3.1 and Hugging Face\n",
    "\n",
    "Here we will connect to an inference endpoint in Langchain so that we can use the Llama 3.1 model to generate responses to the frames-benchmark dataset. We will use the Hugging Face library to connect to the Langchain endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/ben/code/argilla-cookbook/.venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.\n",
      "Token is valid (permission: read).\n",
      "Your token has been saved to /Users/ben/.cache/huggingface/token\n",
      "Login successful\n"
     ]
    }
   ],
   "source": [
    "# Choose the LLM to use\n",
    "from langchain_huggingface import HuggingFaceEndpoint, ChatHuggingFace\n",
    "\n",
    "llm = ChatHuggingFace(\n",
    "    llm=HuggingFaceEndpoint(\n",
    "        repo_id=\"meta-llama/Llama-3.1-8B-Instruct\",\n",
    "        task=\"text-generation\",\n",
    "        max_new_tokens=20,\n",
    "        do_sample=False,\n",
    "        repetition_penalty=1.03,\n",
    "    ),\n",
    "    verbose=True,\n",
    ")\n",
    "\n",
    "# Create an agent executor by passing in the agent and tools\n",
    "agent_executor = AgentExecutor(\n",
    "    agent=create_react_agent(llm, tools, prompt),\n",
    "    tools=tools,\n",
    "    verbose=True,\n",
    "    handle_parsing_errors=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load a of multi-hop queries and wikipedia articles\n",
    "\n",
    "We will load the frames-benchmark dataset and the wikipedia articles that are needed to answer the questions.\n",
    "\n",
    "FRAMES (Factuality, Retrieval, And reasoning MEasurement Set) is a benchmark dataset consisting of 824 challenging multi-hop questions that require retrieving and reasoning about information from 2-15 Wikipedia articles. The dataset covers diverse topics and includes labeled reasoning types, gold answers, and relevant articles. FRAMES evaluates the performance of Retrieval-Augmented Generation (RAG) systems on factuality, retrieval accuracy, and reasoning, providing a comprehensive framework for testing and improving language model capabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "ds = load_dataset(\"google/frames-benchmark\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Review agent responses in Argilla"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import argilla as rg\n",
    "\n",
    "client = rg.Argilla()\n",
    "\n",
    "dataset = rg.Dataset(\n",
    "    name=\"google-frames-benchmark\",\n",
    "    settings=rg.Settings(\n",
    "        fields=[\n",
    "            rg.TextField(name=\"input\"),\n",
    "            rg.TextField(name=\"answer\"),\n",
    "        ],\n",
    "        questions=[rg.TextQuestion(name=\"output\")],\n",
    "    ),\n",
    ")\n",
    "\n",
    "dataset.create()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a series of Few-Shot Learning prompts\n",
    "\n",
    "We will respond to the agent predictions with comments that can be used within a Few-Shot Learning prompt. This will allow the agent to use the feedback to improve its responses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
      "\u001b[32;1m\u001b[1;3mTo determine the name of your future wife, we need to:\n",
      "\n",
      "1. Identify the first name of the 15th First Lady of the United States' mother.\n",
      "2. Identify the maiden name of the mother of the second assassinated president.\n",
      "\n",
      "Let's proceed step-by-step.\n",
      "\n",
      "### Step 1: Identify the First Name of the 15th First Lady's Mother\n",
      "\n",
      "The 15th President of the United States was James Buchanan. His niece, Harriet Lane, served as the First Lady during his presidency because Buchanan was a bachelor.\n",
      "\n",
      "Thought: I need to find out the first name of Harriet Lane's mother.\n",
      "Action: wikipedia\n",
      "Action Input: Harriet Lane\n",
      "\n",
      "### Step 2: Identify the Maiden Name of the Second Assassinated President's Mother\n",
      "\n",
      "The second U.S. president to be assassinated was James A. Garfield.\n",
      "\n",
      "Thought: I need to find out the maiden name of James A. Garfield's mother.\n",
      "Action: wikipedia\n",
      "Action Input: James A. Garfield\n",
      "\n",
      "### Observation Results\n",
      "\n",
      "#### Harriet Lane\u001b[0m\u001b[36;1m\u001b[1;3mNo good Wikipedia Search Result was found\u001b[0m\u001b[32;1m\u001b[1;3mIt seems there was an issue retrieving the information about Harriet Lane. Let's try searching specifically for James Buchanan, as Harriet Lane is often mentioned in his context.\n",
      "\n",
      "Action: wikipedia\n",
      "Action Input: James Buchanan\n",
      "\u001b[0m\u001b[36;1m\u001b[1;3mPage: James Buchanan\n",
      "Summary: James Buchanan Jr. ( bew-KAN-ən; April 23, 1791 – June 1, 1868) was the 15th president of the United States, serving from 1857 to 1861. Buchanan also served as the secretary of state from 1845 to 1849 and represented Pennsylvania in both houses of the U.S. Congress. He was an advocate for states' rights, particularly regarding slavery, and minimized the role of the federal government preceding the Civil War.\n",
      "Buchanan was a lawyer in Pennsylvania and won his first election to the state's House of Representatives as a Federalist. He was elected to the U.S. House of Representatives in 1820 and retained that post for five terms, aligning with Andrew Jackson's Democratic Party. Buchanan served as Jackson's minister to Russia in 1832. He won the election in 1834 as a U.S. senator from Pennsylvania and continued in that position for 11 years. He was appointed to serve as President James K. Polk's secretary of state in 1845, and eight years later was named as President Franklin Pierce's minister to the United Kingdom.\n",
      "Beginning in 1844, Buchanan became a regular contender for the Democratic Party's presidential nomination. He was nominated and won the 1856 presidential election. As President, Buchanan intervened to assure the Supreme Court's majority ruling in the pro-slavery decision in the Dred Scott case. He acceded to Southern attempts to engineer Kansas' entry into the Union as a slave state under the Lecompton Constitution, and angered not only Republicans but also Northern Democrats. Buchanan honored his pledge to serve only one term and supported Breckinridge's unsuccessful candidacy in the 1860 presidential election. He failed to reconcile the fractured Democratic Party amid the grudge against Stephen Douglas, leading to the election of Republican and former Congressman Abraham Lincoln.\n",
      "Buchanan's leadership during his lame duck period, before the American Civil War, has been widely criticized. He simultaneously angered the North by not stopping secession and the South by not yielding to their demands. He supported the Corwin Amendment in an effort to reconcile the country. He made an unsuccessful attempt to reinforce Fort Sumter, but otherwise refrained from preparing the military. In his personal life, Buchanan never married and was the only U.S. president to remain a lifelong bachelor, leading some historians and authors to question his sexual orientation. His failure to forestall the Civil War has been described as incompetence, and he spent his last years defending his reputation. Historians and scholars rank Buchanan as among the worst presidents in American history.\n",
      "\n",
      "Page: James Buchanan Duke\n",
      "Summary: James Buchanan Duke (December 23, 1856 – October 10, 1925) was an American tobacco and electric power industrialist best known for the introduction of modern cigarette manufacture and marketing, and his involvement with Duke University. He was the founder of the American Tobacco Company in 1890.\n",
      "\n",
      "Page: James M. Buchanan\n",
      "Summary: James McGill Buchanan Jr. (; October 3, 1919 – January 9, 2013) was an American economist known for his work on public choice theory originally outlined in his most famous work, The Calculus of Consent, co-authored with Gordon Tullock in 1962. He continued to develop the theory, eventually receiving the Nobel Memorial Prize in Economic Sciences in 1986. Buchanan's work initiated research on how politicians' and bureaucrats' self-interest, utility maximization, and other non-wealth-maximizing considerations affect their decision-making. He was a member of the Board of Advisors of The Independent Institute as well as of the Institute of Economic Affairs, a member of the Mont Pelerin Society (MPS) and MPS president from 1984 to 1986, a Distinguished Senior Fellow of the Cato Institute, and professor at George Mason University.\n",
      "\n",
      "\u001b[0m\u001b[32;1m\u001b[1;3mIt seems that the search for James Buchanan's Wikipedia page did not provide direct information about Harriet Lane's mother. Let's refine the search specifically for Harriet Lane to ensure we get the necessary details.\n",
      "\n",
      "Action: wikipedia\n",
      "Action Input: Harriet Lane\n",
      "\u001b[0m\u001b[36;1m\u001b[1;3mPage: Harriet Lane\n",
      "Summary: Harriet Rebecca Lane Johnston (May 9, 1830 – July 3, 1903) acted as first lady of the United States during the administration of her uncle, lifelong bachelor president James Buchanan, from 1857 to 1861. She has been described as the first of the modern first ladies, being a notably charming and diplomatic hostess, whose dress-styles were copied, and who promoted deserving causes. In her will, she left funds for a new school on the grounds of Washington National Cathedral. Several ships have been named in her honor, including the cutter USCGC Harriet Lane, still in service.\n",
      "\n",
      "\n",
      "\n",
      "Page: USRC Harriet Lane\n",
      "Summary: Harriet Lane was a revenue cutter of the United States Revenue Cutter Service and, on the outbreak of the American Civil War, a ship of the United States Navy and later Confederate States Navy. The craft was named after the niece of senator and later United States President, James Buchanan; during his presidency, she acted as First Lady. The cutter was christened and entered the water for the Revenue Service in 1859 out of New York City, and saw action during the Civil War at Fort Sumter, New Orleans, Galveston, Texas, and Virginia Point. The Confederates captured her in 1863, whereupon she was converted to mercantile service. Union forces recaptured her at the end of war. The U.S. Navy declared her unfit for service and sold her. New owners out of Philadelphia renamed her Elliot Ritchie. Her crew abandoned her at sea in 1881.\n",
      "\n",
      "Page: USCGC Harriet Lane\n",
      "Summary: USCGC Harriet Lane refers to three ships of the United States Coast Guard:\n",
      "\n",
      "USRC Harriet Lane (1857), a revenue cutter serving in the United States Revenue Cutter Service 1861–1881\n",
      "USCGC Harriet Lane (WSC-141), a 125-foot cutter in service with the Coast Guard 1926–1946\n",
      "USCGC Harriet Lane (WMEC-903), a medium-endurance cutter active in service with the Coast Guard, commissioned 1984\u001b[0m\u001b[32;1m\u001b[1;3mI have not yet found the information about Harriet Lane's mother. Let's try another search specifically for Harriet Lane's family background to get the first name of her mother.\n",
      "\n",
      "Action: wikipedia\n",
      "Action Input: Harriet Lane family\n",
      "\u001b[0m\u001b[36;1m\u001b[1;3mPage: Harriet Lane\n",
      "Summary: Harriet Rebecca Lane Johnston (May 9, 1830 – July 3, 1903) acted as first lady of the United States during the administration of her uncle, lifelong bachelor president James Buchanan, from 1857 to 1861. She has been described as the first of the modern first ladies, being a notably charming and diplomatic hostess, whose dress-styles were copied, and who promoted deserving causes. In her will, she left funds for a new school on the grounds of Washington National Cathedral. Several ships have been named in her honor, including the cutter USCGC Harriet Lane, still in service.\n",
      "\n",
      "\n",
      "\n",
      "Page: USRC Harriet Lane\n",
      "Summary: Harriet Lane was a revenue cutter of the United States Revenue Cutter Service and, on the outbreak of the American Civil War, a ship of the United States Navy and later Confederate States Navy. The craft was named after the niece of senator and later United States President, James Buchanan; during his presidency, she acted as First Lady. The cutter was christened and entered the water for the Revenue Service in 1859 out of New York City, and saw action during the Civil War at Fort Sumter, New Orleans, Galveston, Texas, and Virginia Point. The Confederates captured her in 1863, whereupon she was converted to mercantile service. Union forces recaptured her at the end of war. The U.S. Navy declared her unfit for service and sold her. New owners out of Philadelphia renamed her Elliot Ritchie. Her crew abandoned her at sea in 1881.\n",
      "\n",
      "Page: Harriet Beecher Stowe\n",
      "Summary: Harriet Elisabeth Beecher Stowe (; June 14, 1811 – July 1, 1896) was an American author and abolitionist. She came from the religious Beecher family and wrote the popular novel Uncle Tom's Cabin (1852), which depicts the harsh conditions experienced by enslaved African Americans. The book reached an audience of millions as a novel and play, and became influential in the United States and in Great Britain, energizing anti-slavery forces in the American North, while provoking widespread anger in the South. Stowe wrote 30 books, including novels, three travel memoirs, and collections of articles and letters. She was influential both for her writings as well as for her public stances and debates on social issues of the day.\u001b[0m\u001b[32;1m\u001b[1;3mIt appears that I have not yet found the information about Harriet Lane's mother. To solve the problem, I will need to find the first name of Harriet Lane's mother and the maiden name of James A. Garfield's mother.\n",
      "\n",
      "Let me proceed with finding the maiden name of James A. Garfield's mother.\n",
      "\n",
      "Action: wikipedia\n",
      "Action Input: James A. Garfield\n",
      "\u001b[0m\u001b[36;1m\u001b[1;3mPage: James A. Garfield\n",
      "Summary: James Abram Garfield (November 19, 1831 – September 19, 1881) was the 20th president of the United States, serving from March 1881 until his assassination in September that year. A preacher, lawyer, and Civil War general, Garfield served nine terms in the United States House of Representatives and is the only sitting member of the House to be elected president. Before his candidacy for the presidency, he had been elected to the U.S. Senate by the Ohio General Assembly—a position he declined when he became president-elect.\n",
      "Garfield was born into poverty in a log cabin and grew up in northeastern Ohio. After graduating from Williams College, he studied law and became an attorney. He was a preacher in the Stone–Campbell Movement and president of the Western Reserve Eclectic Institute, affiliated with the Disciples. Garfield was elected as a Republican member of the Ohio State Senate in 1859, serving until 1861. He opposed Confederate secession, was a major general in the Union Army during the American Civil War, and fought in the battles of Middle Creek, Shiloh, and Chickamauga. He was elected to Congress in 1862 to represent Ohio's 19th district. Throughout his congressional service, he firmly supported the gold standard and gained a reputation as a skilled orator. He initially agreed with Radical Republican views on Reconstruction but later favored a Moderate Republican–aligned approach to civil rights enforcement for freedmen. Garfield's aptitude for mathematics extended to his own proof of the Pythagorean theorem, which he published in 1876.\n",
      "At the 1880 Republican National Convention, delegates chose Garfield, who had not sought the White House, as a compromise presidential nominee on the 36th ballot. In the 1880 presidential election, he conducted a low-key front porch campaign and narrowly defeated the Democratic nominee, Winfield Scott Hancock. Garfield's accomplishments as president included his assertion of presidential authority against senatorial courtesy in executive appointments, a purge of corruption in the Post Office, and his appointment of a Supreme Court justice. He advocated for agricultural technology, an educated electorate, and civil rights for African Americans. He also proposed substantial civil service reforms, which were passed by Congress in 1883 as the Pendleton Civil Service Reform Act and signed into law by his successor, Chester A. Arthur.\n",
      "Garfield was a member of the intraparty \"Half-Breed\" faction who used the powers of the presidency to defy the powerful \"Stalwart\" Senator Roscoe Conkling from New York. He did this by appointing Blaine faction leader William H. Robertson to the lucrative post of Collector of the Port of New York. The ensuing political battle resulted in Robertson's confirmation and the resignations of Conkling and Thomas C. Platt from the Senate.\n",
      "On July 2, 1881, Charles J. Guiteau, a disappointed and delusional office seeker, shot Garfield at the Baltimore and Potomac Railroad Station in Washington. The wound was not immediately fatal, but an infection caused by his doctors' unsanitary methods in treating the wound killed Garfield on September 19. Due to his brief tenure in office, historians tend to rank Garfield as a below-average president, though he has earned praise for anti-corruption and pro-civil rights stances.\n",
      "\n",
      "Page: Assassination of James A. Garfield\n",
      "Summary: James A. Garfield, the 20th president of the United States, was shot at the Baltimore and Potomac Railroad Station in Washington, D.C., at 9:30 am on Saturday, July 2, 1881. He died in Elberon, New Jersey, two and a half months later on September 19, 1881. The shooting occurred less than four months into his term as president. Charles J. Guiteau was convicted of Garfield's murder and executed by hanging one year after the shooting.\n",
      "\n",
      "Page: James A. Garfield Memorial\n",
      "Summary: The James A. Garfield Memorial is the final resting place of assassinated President James A. Garfield, locat\u001b[0m\u001b[32;1m\u001b[1;3mTo find the name of your future wife, we need to:\n",
      "\n",
      "1. Identify the first name of Harriet Lane's mother (Harriet Lane was the niece of James Buchanan, the 15th President of the United States).\n",
      "2. Identify the maiden name of James A. Garfield's mother (James A. Garfield was the second U.S. president to be assassinated).\n",
      "\n",
      "### Step 1: Identify the First Name of Harriet Lane's Mother\n",
      "\n",
      "Harriet Lane's mother was named Jane Buchanan Lane.\n",
      "\n",
      "### Step 2: Identify the Maiden Name of James A. Garfield's Mother\n",
      "\n",
      "James A. Garfield's mother was named Eliza Ballou Garfield. Her maiden name was Ballou.\n",
      "\n",
      "Combining these:\n",
      "\n",
      "- First name: Jane (Harriet Lane's mother's first name)\n",
      "- Surname: Ballou (James A. Garfield's mother's maiden name)\n",
      "\n",
      "### Final Answer\n",
      "Your future wife's name is Jane Ballou.\u001b[0mInvalid Format: Missing 'Action:' after 'Thought:\u001b[32;1m\u001b[1;3mI now know the final answer.\n",
      "\n",
      "Final Answer: Your future wife's name is Jane Ballou.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">DatasetRecords: The provided batch size <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">256</span> was normalized. Using value <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "DatasetRecords: The provided batch size \u001b[1;36m256\u001b[0m was normalized. Using value \u001b[1;36m1\u001b[0m.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Sending records...: 100%|██████████| 1/1 [00:00<00:00,  1.78batch/s]\n"
     ]
    }
   ],
   "source": [
    "for sample in ds[\"test\"].select(range(10)):\n",
    "    prompt = sample[\"Prompt\"]\n",
    "    prompt += sample[\"wiki_links\"]\n",
    "    response = agent_executor.invoke(input={\"input\": prompt})\n",
    "    response[\"answer\"] = sample[\"Answer\"]\n",
    "    dataset.records.log([response])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collect Fewshot Examples from Argilla\n",
    "\n",
    "Here we will create a prompt that will allow the agent to learn from the feedback that we provide in Argilla. We will use the Few-Shot Learning prompt to provide the agent with examples of how to improve its responses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "fewshot_examples = \"\"\n",
    "\n",
    "for record in dataset.records(with_responses=True):\n",
    "    for response in record.responses:\n",
    "        input_example = record.fields[\"input\"]\n",
    "        response_example = response.value\n",
    "        fewshot_examples += f\"input: {input_example}\\noutput: {response_example}\\n\\n\"\n",
    "\n",
    "fewshot_examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Review final agent responses in Argilla\n",
    "\n",
    "We can now review the final responses from the agent in Argilla to see if the few shot examples have improved the agent's responses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for sample in ds[\"test\"].select(range(10, 20)):\n",
    "    prompt = sample[\"Prompt\"]\n",
    "    prompt += sample[\"wiki_links\"]\n",
    "    prompt += fewshot_examples\n",
    "    response = agent_executor.invoke(input={\"input\": prompt})\n",
    "    response[\"answer\"] = sample[\"Answer\"]\n",
    "    dataset.records.log([response])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}