few_shot_prompt_selection/few_shot_prompt_selection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "82c66e32",
   "metadata": {},
   "source": [
    "# Ensuring Reliable Few-Shot Prompt Selection for LLMs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e356214c",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/cleanlab/cleanlab-tools/blob/master/few_shot_prompt_selection/few_shot_prompt_selection.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91c42988",
   "metadata": {},
   "source": [
    "In this notebook, we prompt the Davinci LLM from OpenAI (the model underpinning GPT-3/ChatGPT) with few-shot prompts in an effort to classify the intent of customer service requests at a large bank. Following typical practice, we source the few-shot examples to include in the prompt template from an available dataset of human-labeled request examples. However, the resulting LLM predictions are unreliable — a close inspection reveals this is because real-world data is messy and error-prone.  LLM performance in this customer service intent classification task is only marginally boosted by manually modifying the prompt template to mitigate potentially noisy data. The LLM predictions become significantly more accurate if we instead use data-centric AI algorithms via Cleanlab Studio to ensure only high-quality few-shot examples are selected for inclusion in the prompt template."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d802b04f",
   "metadata": {},
   "source": [
    "# Imports and Helpers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "66f4f859",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import warnings\n",
    "import openai, os\n",
    "import string\n",
    "import random\n",
    "import tiktoken\n",
    "from tqdm import tqdm\n",
    "from sklearn.metrics import accuracy_score\n",
    "from langchain.prompts import PromptTemplate\n",
    "from langchain.prompts import FewShotPromptTemplate\n",
    "warnings.filterwarnings('ignore')\n",
    "pd.set_option('max_colwidth', None)\n",
    "\n",
    "def eval_preds(preds):\n",
    "    acc = accuracy_score(preds, test.label.values)\n",
    "    return \"Model Accuracy: \" + '{:.1%}'.format(acc)\n",
    "\n",
    "def tokens_per_prompt(prompt):\n",
    "    encoding = tiktoken.get_encoding(\"p50k_base\")\n",
    "    num_tokens = len(encoding.encode(prompt))\n",
    "    return num_tokens\n",
    "\n",
    "def cost_per_prompt(prompt):\n",
    "    cost_per_token = 0.02 / 1000\n",
    "    tokens = tokens_per_prompt(prompt)\n",
    "    cost = tokens * cost_per_token\n",
    "    return cost\n",
    "\n",
    "def cost_per_test_evaluation(examples_pool, test):\n",
    "    texts = test.text.values\n",
    "    cost = 0\n",
    "    examples = get_examples(examples_pool)\n",
    "    for text in texts:\n",
    "        prompt = get_prompt(examples_pool, text, examples)\n",
    "        prompt_cost = cost_per_prompt(prompt)\n",
    "        cost += prompt_cost\n",
    "    return \"${:,.2f}\".format(cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e25a0d0",
   "metadata": {},
   "source": [
    "# Banking Intent Dataset\n",
    "This notebook studies a 50-class variant of the [Banking-77](https://arxiv.org/abs/2003.04807) Dataset which contains online banking queries annotated with their corresponding intents (the label shown below). We evaluate models that predict this label using a fixed test dataset containing ~500 phrases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "67596bbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "test = pd.read_csv('https://s.cleanlab.ai/banking-intent-50/test.csv')\n",
    "examples_pool = pd.read_csv('https://s.cleanlab.ai/banking-intent-50/examples-pool.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "0047884f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>i moved to a new city and need to change my address</td>\n",
       "      <td>edit_personal_details</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>on my transfer there was a \"decline\" message</td>\n",
       "      <td>declined_transfer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>help!  my wallet was stolen and someone is taking money out.  i need this money!  what can i do?</td>\n",
       "      <td>card_payment_fee_charged</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>while abroad i got cash, and a wrong exchange rate was applied.</td>\n",
       "      <td>wrong_exchange_rate_for_cash_withdrawal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>why can't i get cash?</td>\n",
       "      <td>getting_spare_card</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                               text   \n",
       "0                                               i moved to a new city and need to change my address  \\\n",
       "1                                                      on my transfer there was a \"decline\" message   \n",
       "2  help!  my wallet was stolen and someone is taking money out.  i need this money!  what can i do?   \n",
       "3                                   while abroad i got cash, and a wrong exchange rate was applied.   \n",
       "4                                                                             why can't i get cash?   \n",
       "\n",
       "                                     label  \n",
       "0                    edit_personal_details  \n",
       "1                        declined_transfer  \n",
       "2                 card_payment_fee_charged  \n",
       "3  wrong_exchange_rate_for_cash_withdrawal  \n",
       "4                       getting_spare_card  "
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "examples_pool[['text', 'label']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cd34610",
   "metadata": {},
   "source": [
    "# OpenAI API Key\n",
    "Replace with your own key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "0bc467b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: OPENAI_API_KEY={your_key_here}\n"
     ]
    }
   ],
   "source": [
    "%env OPENAI_API_KEY = {your_key_here}\n",
    "openai.api_key = os.environ['OPENAI_API_KEY']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0edd6b39",
   "metadata": {},
   "source": [
    "# Building the Few-Shot Prompt\n",
    "Few-shot prompting is a technique used in natural language processing that enables pretrained foundation models to perform complex tasks without any explicit training (i.e. updates to model parameters).  In few-shot prompting (also known as in-context learning), we provide a model with a limited number of input-output pairs, as part of a prompt template that is included in the prompt used to instruct the model how to handle a particular input. \n",
    "\n",
    "Our prompt will consist of a few pieces:\n",
    "- (optional) prefix \n",
    "- list of class labels to help LLM choose a valid class\n",
    "- (optional) 50 examples, one from each class\n",
    "- target text for LLM to classify"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b16c8462",
   "metadata": {},
   "source": [
    "## Adding List of Classes for Valid Completions\n",
    "\n",
    "This text will go at the beginning of the prompt and will tell the LLM what the valid classes are so that it can consistently output a class. Without this, the LLM will not choose a valid class and output something not parsable.\n",
    "\n",
    "Here we can also add an optional `prefix` that we will use later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "ebe632ad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Beware some labels in the examples may be noisy.\n",
      "You can choose the label from: edit_personal_details,declined_transfer,card_payment_fee_charged,wrong_exchange_rate_for_cash_withdrawal,getting_spare_card,cash_withdrawal_charge,verify_top_up,transfer_timing,apple_pay_or_google_pay,card_payment_not_recognised,visa_or_mastercard,reverted_card_payment,transfer_not_received_by_recipient,country_support,wrong_amount_of_cash_received,refund_not_showing_up,card_linking,failed_transfer,exchange_via_app,fiat_currency_support,activate_my_card,direct_debit_payment_not_recognised,balance_not_updated_after_cheque_or_cash_deposit,cash_withdrawal_not_recognised,transfer_fee_charged,card_arrival,pending_top_up,extra_charge_on_statement,supported_cards_and_currencies,declined_card_payment,top_up_failed,automatic_top_up,transaction_charged_twice,disposable_card_limits,card_payment_wrong_exchange_rate,pending_transfer,declined_cash_withdrawal,balance_not_updated_after_bank_transfer,beneficiary_not_allowed,why_verify_identity,order_physical_card,request_refund,card_about_to_expire,cancel_transfer,pending_card_payment,change_pin,lost_or_stolen_phone,pending_cash_withdrawal,top_up_reverted,exchange_charge\n"
     ]
    }
   ],
   "source": [
    "# Helper to get prefix for prompt. This gives the LLM all of the labels so that it chooses more accurately.\n",
    "def get_prefix(examples_pool, prefix=\"\"):\n",
    "    s = \"\"\n",
    "    if len(prefix) != 0:\n",
    "        s += prefix\n",
    "        s += '\\n'\n",
    "    s += \"You can choose the label from: \"\n",
    "    classes = list(examples_pool.label.unique())\n",
    "    s += \",\".join(classes)\n",
    "    return s\n",
    "\n",
    "print(get_prefix(examples_pool, \"Beware some labels in the examples may be noisy.\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0e9466e",
   "metadata": {},
   "source": [
    "## Adding K-Shot Examples\n",
    "\n",
    "Here we randomly choose 50 examples, 1 from each class to build a 50-shot prompt for the LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "db135c31",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'text': 'i just got married and i need to change my name',\n",
       "  'label': 'edit_personal_details'},\n",
       " {'text': 'it declined my transfer.', 'label': 'declined_transfer'},\n",
       " {'text': \"why am i being charged for atm cash withdrawals? the only reason i use it is because it's been free! now you expect me to pay for them, and how much is that going to cost me?\",\n",
       "  'label': 'card_payment_fee_charged'},\n",
       " {'text': 'i attempted to get money using a foreign currency at an atm but the rate was highly inaccurate!',\n",
       "  'label': 'wrong_exchange_rate_for_cash_withdrawal'},\n",
       " {'text': 'tell me where i can find the auto top up feature and a little bit about it please.',\n",
       "  'label': 'getting_spare_card'},\n",
       " {'text': 'how come i got charged extra for withdrawing cash?',\n",
       "  'label': 'cash_withdrawal_charge'},\n",
       " {'text': \"i don't know where the top-up verification code is.\",\n",
       "  'label': 'verify_top_up'},\n",
       " {'text': 'how long do i have to wait for a us transfer?',\n",
       "  'label': 'transfer_timing'},\n",
       " {'text': 'am i able to do top-ups with google pay?',\n",
       "  'label': 'apple_pay_or_google_pay'},\n",
       " {'text': 'tell me why there is an extra charge when i used an atm?',\n",
       "  'label': 'card_payment_not_recognised'},\n",
       " {'text': 'do you guys accept mastercard or visa?',\n",
       "  'label': 'visa_or_mastercard'},\n",
       " {'text': 'the payment i made with my card is not showing up.',\n",
       "  'label': 'reverted_card_payment'},\n",
       " {'text': 'when i exchanged currencies the value of my money changed.',\n",
       "  'label': 'transfer_not_received_by_recipient'},\n",
       " {'text': 'what countries are you available in?', 'label': 'country_support'},\n",
       " {'text': 'is there a way to change name?',\n",
       "  'label': 'wrong_amount_of_cash_received'},\n",
       " {'text': 'i am still waiting to find out the status of my refund. i have made previous attempts to find out what the status is to no avail. will you please look into this for me?',\n",
       "  'label': 'refund_not_showing_up'},\n",
       " {'text': 'where do i need to go in the app to enter my card info?',\n",
       "  'label': 'card_linking'},\n",
       " {'text': 'what stopped my transfer, from being accepted?',\n",
       "  'label': 'failed_transfer'},\n",
       " {'text': 'if i need gbp instead of aud what do i do?',\n",
       "  'label': 'exchange_via_app'},\n",
       " {'text': 'i want to know how many different currencies my money can be in',\n",
       "  'label': 'fiat_currency_support'},\n",
       " {'text': 'how long will it take to activate my new card?',\n",
       "  'label': 'activate_my_card'},\n",
       " {'text': 'i see a direct debit charge that i did not approve',\n",
       "  'label': 'direct_debit_payment_not_recognised'},\n",
       " {'text': \"i made a cash deposit to my account but i don't see it\",\n",
       "  'label': 'balance_not_updated_after_cheque_or_cash_deposit'},\n",
       " {'text': \"there is a withdrawal in my account i didn't make.\",\n",
       "  'label': 'cash_withdrawal_not_recognised'},\n",
       " {'text': 'Curabitur dapibus tellus nec sapien condimentum, eget congue metus efficitur.',\n",
       "  'label': 'transfer_fee_charged'},\n",
       " {'text': 'still waiting on my new card', 'label': 'card_arrival'},\n",
       " {'text': \"your top-up function isn't working, it still says pending even though i know my card works.\",\n",
       "  'label': 'pending_top_up'},\n",
       " {'text': 'i was charged an extra pound for no reason. why is that?',\n",
       "  'label': 'extra_charge_on_statement'},\n",
       " {'text': \"my card payment doesn't work.\",\n",
       "  'label': 'supported_cards_and_currencies'},\n",
       " {'text': \"Did you hear about the mathematician who's afraid of negative numbers? He'll stop at nothing to avoid them!\",\n",
       "  'label': 'declined_card_payment'},\n",
       " {'text': 'why is it failing to top-up?', 'label': 'top_up_failed'},\n",
       " {'text': 'i will be traveling.  can i add money automatically?',\n",
       "  'label': 'automatic_top_up'},\n",
       " {'text': 'i got charged in double and want a refund',\n",
       "  'label': 'transaction_charged_twice'},\n",
       " {'text': 'do the disposable cards have any restrictions on them?',\n",
       "  'label': 'disposable_card_limits'},\n",
       " {'text': 'i am not receiving the correct exchange rate on my card.',\n",
       "  'label': 'card_payment_wrong_exchange_rate'},\n",
       " {'text': 'i noticed on my online statement that an atm withdrawal i made early is still pending.  is this accurate?',\n",
       "  'label': 'pending_transfer'},\n",
       " {'text': 'the cash i tried to draw from the atm was not approved.',\n",
       "  'label': 'declined_cash_withdrawal'},\n",
       " {'text': \"my balance didn't change after i transferred some money.\",\n",
       "  'label': 'balance_not_updated_after_bank_transfer'},\n",
       " {'text': 'the transfer to my beneficiary has not gone through',\n",
       "  'label': 'beneficiary_not_allowed'},\n",
       " {'text': 'am i supposed to verify my identity?',\n",
       "  'label': 'why_verify_identity'},\n",
       " {'text': 'Unaitwa nani?', 'label': 'order_physical_card'},\n",
       " {'text': 'please could you give me a refund', 'label': 'request_refund'},\n",
       " {'text': 'my transfer got declined.', 'label': 'card_about_to_expire'},\n",
       " {'text': 'Nullam vitae turpis sit amet diam luctus efficitur id id lacus.',\n",
       "  'label': 'cancel_transfer'},\n",
       " {'text': \"hello, i made a payment with my card and it's not working. it's been a while and still says its pending. can you tell when this will go through?\",\n",
       "  'label': 'pending_card_payment'},\n",
       " {'text': 'can i change my pin through the app?', 'label': 'change_pin'},\n",
       " {'text': 'lost phone, can i still access account?',\n",
       "  'label': 'lost_or_stolen_phone'},\n",
       " {'text': \"i didn't receive my money earlier and it says the transaction is still in progress. can you fix it?\",\n",
       "  'label': 'pending_cash_withdrawal'},\n",
       " {'text': 'there is an urgent thing that i need to transfer from china. approximately how long will it take?',\n",
       "  'label': 'top_up_reverted'},\n",
       " {'text': 'is there a pay structure for currency exchanges?',\n",
       "  'label': 'exchange_charge'}]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Helper method to get one example from each class for k-shot prompt.\n",
    "def get_examples(examples_pool):\n",
    "    out = []\n",
    "    unique_classes = examples_pool.label.unique()\n",
    "    for i, cls in enumerate(unique_classes):\n",
    "        temp = examples_pool[examples_pool.label==cls]\n",
    "        random.seed(0+i)\n",
    "        idx = random.choice(list(range(len(temp))))\n",
    "        text = temp.iloc[idx].text\n",
    "        label = temp.iloc[idx].label\n",
    "        d = {'text':text, 'label':label}         \n",
    "        out.append(d)\n",
    "    return out\n",
    "\n",
    "examples = get_examples(examples_pool)\n",
    "examples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04af47e3",
   "metadata": {},
   "source": [
    "## Generate Entire Prompt "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "e487377d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Beware some labels in the examples may be noisy.\n",
      "You can choose the label from: edit_personal_details,declined_transfer,card_payment_fee_charged,wrong_exchange_rate_for_cash_withdrawal,getting_spare_card,cash_withdrawal_charge,verify_top_up,transfer_timing,apple_pay_or_google_pay,card_payment_not_recognised,visa_or_mastercard,reverted_card_payment,transfer_not_received_by_recipient,country_support,wrong_amount_of_cash_received,refund_not_showing_up,card_linking,failed_transfer,exchange_via_app,fiat_currency_support,activate_my_card,direct_debit_payment_not_recognised,balance_not_updated_after_cheque_or_cash_deposit,cash_withdrawal_not_recognised,transfer_fee_charged,card_arrival,pending_top_up,extra_charge_on_statement,supported_cards_and_currencies,declined_card_payment,top_up_failed,automatic_top_up,transaction_charged_twice,disposable_card_limits,card_payment_wrong_exchange_rate,pending_transfer,declined_cash_withdrawal,balance_not_updated_after_bank_transfer,beneficiary_not_allowed,why_verify_identity,order_physical_card,request_refund,card_about_to_expire,cancel_transfer,pending_card_payment,change_pin,lost_or_stolen_phone,pending_cash_withdrawal,top_up_reverted,exchange_charge\n",
      "\n",
      "Text: i just got married and i need to change my name\n",
      "Label: edit_personal_details\n",
      "\n",
      "Text: it declined my transfer.\n",
      "Label: declined_transfer\n",
      "\n",
      "Text: why am i being charged for atm cash withdrawals? the only reason i use it is because it's been free! now you expect me to pay for them, and how much is that going to cost me?\n",
      "Label: card_payment_fee_charged\n",
      "\n",
      "Text: i attempted to get money using a foreign currency at an atm but the rate was highly inaccurate!\n",
      "Label: wrong_exchange_rate_for_cash_withdrawal\n",
      "\n",
      "Text: tell me where i can find the auto top up feature and a little bit about it please.\n",
      "Label: getting_spare_card\n",
      "\n",
      "Text: how come i got charged extra for withdrawing cash?\n",
      "Label: cash_withdrawal_charge\n",
      "\n",
      "Text: i don't know where the top-up verification code is.\n",
      "Label: verify_top_up\n",
      "\n",
      "Text: how long do i have to wait for a us transfer?\n",
      "Label: transfer_timing\n",
      "\n",
      "Text: am i able to do top-ups with google pay?\n",
      "Label: apple_pay_or_google_pay\n",
      "\n",
      "Text: tell me why there is an extra charge when i used an atm?\n",
      "Label: card_payment_not_recognised\n",
      "\n",
      "Text: do you guys accept mastercard or visa?\n",
      "Label: visa_or_mastercard\n",
      "\n",
      "Text: the payment i made with my card is not showing up.\n",
      "Label: reverted_card_payment\n",
      "\n",
      "Text: when i exchanged currencies the value of my money changed.\n",
      "Label: transfer_not_received_by_recipient\n",
      "\n",
      "Text: what countries are you available in?\n",
      "Label: country_support\n",
      "\n",
      "Text: is there a way to change name?\n",
      "Label: wrong_amount_of_cash_received\n",
      "\n",
      "Text: i am still waiting to find out the status of my refund. i have made previous attempts to find out what the status is to no avail. will you please look into this for me?\n",
      "Label: refund_not_showing_up\n",
      "\n",
      "Text: where do i need to go in the app to enter my card info?\n",
      "Label: card_linking\n",
      "\n",
      "Text: what stopped my transfer, from being accepted?\n",
      "Label: failed_transfer\n",
      "\n",
      "Text: if i need gbp instead of aud what do i do?\n",
      "Label: exchange_via_app\n",
      "\n",
      "Text: i want to know how many different currencies my money can be in\n",
      "Label: fiat_currency_support\n",
      "\n",
      "Text: how long will it take to activate my new card?\n",
      "Label: activate_my_card\n",
      "\n",
      "Text: i see a direct debit charge that i did not approve\n",
      "Label: direct_debit_payment_not_recognised\n",
      "\n",
      "Text: i made a cash deposit to my account but i don't see it\n",
      "Label: balance_not_updated_after_cheque_or_cash_deposit\n",
      "\n",
      "Text: there is a withdrawal in my account i didn't make.\n",
      "Label: cash_withdrawal_not_recognised\n",
      "\n",
      "Text: Curabitur dapibus tellus nec sapien condimentum, eget congue metus efficitur.\n",
      "Label: transfer_fee_charged\n",
      "\n",
      "Text: still waiting on my new card\n",
      "Label: card_arrival\n",
      "\n",
      "Text: your top-up function isn't working, it still says pending even though i know my card works.\n",
      "Label: pending_top_up\n",
      "\n",
      "Text: i was charged an extra pound for no reason. why is that?\n",
      "Label: extra_charge_on_statement\n",
      "\n",
      "Text: my card payment doesn't work.\n",
      "Label: supported_cards_and_currencies\n",
      "\n",
      "Text: Did you hear about the mathematician who's afraid of negative numbers? He'll stop at nothing to avoid them!\n",
      "Label: declined_card_payment\n",
      "\n",
      "Text: why is it failing to top-up?\n",
      "Label: top_up_failed\n",
      "\n",
      "Text: i will be traveling.  can i add money automatically?\n",
      "Label: automatic_top_up\n",
      "\n",
      "Text: i got charged in double and want a refund\n",
      "Label: transaction_charged_twice\n",
      "\n",
      "Text: do the disposable cards have any restrictions on them?\n",
      "Label: disposable_card_limits\n",
      "\n",
      "Text: i am not receiving the correct exchange rate on my card.\n",
      "Label: card_payment_wrong_exchange_rate\n",
      "\n",
      "Text: i noticed on my online statement that an atm withdrawal i made early is still pending.  is this accurate?\n",
      "Label: pending_transfer\n",
      "\n",
      "Text: the cash i tried to draw from the atm was not approved.\n",
      "Label: declined_cash_withdrawal\n",
      "\n",
      "Text: my balance didn't change after i transferred some money.\n",
      "Label: balance_not_updated_after_bank_transfer\n",
      "\n",
      "Text: the transfer to my beneficiary has not gone through\n",
      "Label: beneficiary_not_allowed\n",
      "\n",
      "Text: am i supposed to verify my identity?\n",
      "Label: why_verify_identity\n",
      "\n",
      "Text: Unaitwa nani?\n",
      "Label: order_physical_card\n",
      "\n",
      "Text: please could you give me a refund\n",
      "Label: request_refund\n",
      "\n",
      "Text: my transfer got declined.\n",
      "Label: card_about_to_expire\n",
      "\n",
      "Text: Nullam vitae turpis sit amet diam luctus efficitur id id lacus.\n",
      "Label: cancel_transfer\n",
      "\n",
      "Text: hello, i made a payment with my card and it's not working. it's been a while and still says its pending. can you tell when this will go through?\n",
      "Label: pending_card_payment\n",
      "\n",
      "Text: can i change my pin through the app?\n",
      "Label: change_pin\n",
      "\n",
      "Text: lost phone, can i still access account?\n",
      "Label: lost_or_stolen_phone\n",
      "\n",
      "Text: i didn't receive my money earlier and it says the transaction is still in progress. can you fix it?\n",
      "Label: pending_cash_withdrawal\n",
      "\n",
      "Text: there is an urgent thing that i need to transfer from china. approximately how long will it take?\n",
      "Label: top_up_reverted\n",
      "\n",
      "Text: is there a pay structure for currency exchanges?\n",
      "Label: exchange_charge\n",
      "\n",
      "Text: Classify this text!\n",
      "Label:\n"
     ]
    }
   ],
   "source": [
    "# Helper to format the k-shot prompt with:\n",
    "# - prefix\n",
    "# - 1 example from each class\n",
    "# - target text for classification\n",
    "def get_prompt(examples_pool, text, examples, prefix=\"\"):\n",
    "    prompt_template = PromptTemplate(\n",
    "        input_variables=[\"text\", \"label\"],\n",
    "        template=\"Text: {text}\\nLabel: {label}\",\n",
    "    )\n",
    "\n",
    "    p = FewShotPromptTemplate(\n",
    "        example_prompt = prompt_template,\n",
    "        examples = examples,\n",
    "        prefix = get_prefix(examples_pool, prefix),\n",
    "        suffix = \"Text: {text}\\nLabel:\",\n",
    "        input_variables = ['text'],\n",
    "        )\n",
    "    return p.format(text=text).strip()\n",
    "\n",
    "print(get_prompt(examples_pool, \"Classify this text!\", examples, \"Beware some labels in the examples may be noisy.\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4180a3c9",
   "metadata": {},
   "source": [
    "## Query OpenAI LLM API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "9a40952b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model classified  'How can I change my pin?'  as  change_pin\n"
     ]
    }
   ],
   "source": [
    "# Helper method to prompt OpenAI LLM and get response.\n",
    "def get_response(prompt):\n",
    "    response = openai.Completion.create(\n",
    "      model=\"text-davinci-003\",\n",
    "      prompt=prompt,\n",
    "      temperature=0,\n",
    "      max_tokens=50,\n",
    "      top_p=1,\n",
    "      frequency_penalty=0,\n",
    "      presence_penalty=0\n",
    "    )\n",
    "    \n",
    "    # Parse output to get just the label.\n",
    "    resp = response['choices'][0]['text'].split('\\n')[0].split(',')[0].strip().lower().rstrip(string.punctuation)\n",
    "    \n",
    "    # Just in case a respone is not a perfect match, we know.\n",
    "    if resp not in examples_pool.label.unique():\n",
    "        print(resp)\n",
    "    return resp\n",
    "\n",
    "text = \"\\'How can I change my pin?\\'\"\n",
    "examples = get_examples(examples_pool)\n",
    "prompt = get_prompt(examples_pool, text, examples)\n",
    "response = get_response(prompt)\n",
    "print(\"Model classified \", text, \" as \", response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b175845d",
   "metadata": {},
   "source": [
    "## Evaluate Prompt on Test Examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "212aa87b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Helper method to evaluate test set.\n",
    "def eval_prompt(examples_pool, test, prefix=\"\", use_examples = True):\n",
    "    texts = test.text.values\n",
    "    responses = []\n",
    "    examples = get_examples(examples_pool) if use_examples else []\n",
    "    for i in tqdm(range(len(texts))):\n",
    "        text = texts[i]\n",
    "        prompt = get_prompt(examples_pool, text, examples, prefix)\n",
    "        resp = get_response(prompt)\n",
    "        responses.append(resp)\n",
    "    return responses"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2a6ac09",
   "metadata": {},
   "source": [
    "## Estimate Spend\n",
    "\n",
    "Before we run our prompt evaluation script, let's estimate how much it will cost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "39981601",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'$17.85'"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cost_per_test_evaluation(examples_pool, test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a579b6a",
   "metadata": {},
   "source": [
    "# Baseline Model Performance\n",
    "Running each of the test examples through the LLM with the 50-shot prompt shown above, we achieve an accuracy of 59.6% which isn’t bad for a 50-class problem — let’s take a closer look at the examples pool and see if we can improve anything!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 454,
   "id": "e7da26fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Model Accuracy: 59.6%'"
      ]
     },
     "execution_count": 454,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "baseline_preds = eval_prompt(examples_pool, test)\n",
    "# If you'd like to save the predictions.\n",
    "np.save('baseline_preds.npy', np.array(baseline_preds))\n",
    "eval_preds(baseline_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcfe74ec",
   "metadata": {},
   "source": [
    "# Issues in Our Data\n",
    "After a closer inspection of the pool of few-shot examples, we find examples of label errors and outliers. Here are a few examples of each. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7f6262c",
   "metadata": {},
   "source": [
    "### Label Issues"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "a0238b4f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>152</th>\n",
       "      <td>how much does it cost to get more cards?</td>\n",
       "      <td>failed_transfer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>182</th>\n",
       "      <td>i may need to dispute a direct debit payment.</td>\n",
       "      <td>getting_spare_card</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1289</th>\n",
       "      <td>should i contact customer support if i can't edit details</td>\n",
       "      <td>card_about_to_expire</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                           text   \n",
       "152                    how much does it cost to get more cards?  \\\n",
       "182               i may need to dispute a direct debit payment.   \n",
       "1289  should i contact customer support if i can't edit details   \n",
       "\n",
       "                     label  \n",
       "152        failed_transfer  \n",
       "182     getting_spare_card  \n",
       "1289  card_about_to_expire  "
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_idx = [4013, 3180, 6737]\n",
    "issues = examples_pool[examples_pool.id.isin(label_idx)]\n",
    "issues[['text', 'label']]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5db31ec3",
   "metadata": {},
   "source": [
    "### Outliers\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "0cb4a0a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>not (A and B and C)</td>\n",
       "      <td>disposable_card_limits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>441</th>\n",
       "      <td>7ZFGXIBX26Q3TXS4W8VA06ZZLA9GK9I</td>\n",
       "      <td>card_arrival</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>729</th>\n",
       "      <td>Add eggs one at a time, beating well after each addition.</td>\n",
       "      <td>visa_or_mastercard</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                          text   \n",
       "72                                         not (A and B and C)  \\\n",
       "441                            7ZFGXIBX26Q3TXS4W8VA06ZZLA9GK9I   \n",
       "729  Add eggs one at a time, beating well after each addition.   \n",
       "\n",
       "                      label  \n",
       "72   disposable_card_limits  \n",
       "441            card_arrival  \n",
       "729      visa_or_mastercard  "
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_idx = [13295, 13249, 13275]\n",
    "label_idx = [13295, 13275, 13284]\n",
    "\n",
    "examples_pool[examples_pool.id.isin(label_idx)][['text', 'label']]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f59c7324",
   "metadata": {},
   "source": [
    "# Why Do These Issues Matter?\n",
    "\n",
    "As the context size for LLMs grows every day, it is becoming more and more common for prompts to include many examples from each class. In many applications, it may not be possible to hand-select which examples to use in your few-shot prompt, especially as the number of classes grows. As a result, if your examples pool contains issues that we have shown above, randomly selecting examples from each class may result in these errant examples finding their way into the prompt. The remainder of this article illustrates the decreased model performance as a result of this."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3abd932",
   "metadata": {},
   "source": [
    "# Can we tell the LLM the examples are noisy?\n",
    "\n",
    "Instead of modifying the few-shot examples, what if we just include a “disclaimer warning” in the prompt telling the LLM that the labels may be incorrect?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "808694b3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Model Accuracy: 62.0%'"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add warning to the beginning of our prompt.\n",
    "prefix = \"Beware that some labels in the examples may be noisy and have been incorrectly specified.\"\n",
    "preds_with_disclaimer = eval_prompt(examples_pool, test, prefix)\n",
    "# If you'd like to save the predictions.\n",
    "np.save('preds_with_disclaimer.npy', np.array(preds_with_disclaimer))\n",
    "eval_preds(preds_with_disclaimer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b152e7dd",
   "metadata": {},
   "source": [
    "Using the above prompt, we achieve an accuracy of 62%. Marginally better! I’m certain we can still improve."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31513ea5",
   "metadata": {},
   "source": [
    "# Can we remove the noisy examples entirely?\n",
    "\n",
    "Since we can’t trust the labels in the examples pool, what if we just remove them entirely from the prompt and only rely on the powerful LLM?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "fe7b297b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Model Accuracy: 67.4%'"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here we set use_examples to False so we don't add any of the \n",
    "# fewshot examples to the prompt.\n",
    "preds_with_no_examples = eval_prompt(examples_pool, test, use_examples=False)\n",
    "# If you'd like to save the predictions.\n",
    "np.save('preds_with_no_examples.npy', np.array(preds_with_no_examples))\n",
    "eval_preds(preds_with_no_examples)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0be957f0",
   "metadata": {},
   "source": [
    "# Can we identify and correct the noisy examples?\n",
    "\n",
    "Instead of modifying the prompt or removing the examples entirely, the smarter (yet more complex) way to improve our dataset would be to find and fix the label issues by hand. This simultaneously removes a noisy data point that is harming the model and adds an accurate one that should improve its performance via few-shot prompting, but making such corrections manually is cumbersome. Here we instead effortlessly correct the data using [Cleanlab Studio](https://cleanlab.ai/studio/), a platform that implements [Confident Learning](https://l7.curtisnorthcutt.com/confident-learning) algorithms to automatically find and fix label issues.\n",
    "\n",
    "After replacing the bad labels we spotted with more suitable ones, we re-run the original 50-shot through the LLM with each test example, except this time we use the **corrected label** which ensures we provide the LLM with 50 high-quality examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "80f59fd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load in examples pool that has been corrected using Cleanlab Studio.\n",
    "examples_pool_studio = pd.read_csv('https://s.cleanlab.ai/banking-intent-50/examples-pool-studio.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 451,
   "id": "611b68ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Model Accuracy: 72.0%'"
      ]
     },
     "execution_count": 451,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds_corrected_labels = eval_prompt(examples_pool_studio, test)\n",
    "np.save('preds_corrected_labels.npy', np.array(preds_corrected_labels))\n",
    "eval_preds(preds_corrected_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62f8bdc5",
   "metadata": {},
   "source": [
    "After doing this, we achieve an accuracy of 72% which is **quite impressive** for the 50-class problem.\n",
    "\n",
    "**We’ve now shown that noisy few-shot examples can considerably decrease LLM performance and that it is not wise to just change the prompt (via adding warnings or removing examples). To achieve the highest performance, you should also try correcting your examples using Data-centric AI via tools like Cleanlab Studio.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "418f5d76",
   "metadata": {},
   "source": [
    "## Resources\n",
    "- [Cleanlab Studio](https://cleanlab.ai)\n",
    "- [Github](https://github.com/cleanlab/cleanlab)\n",
    "- [Cleanlab Community](https://cleanlab.ai/slack)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}