Basic_Samples/Completions/completions_with_dynamic_prompt.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "278e7451",
   "metadata": {
    "tags": []
   },
   "source": [
    "<h1 align =\"center\"> Dynamic Prompting for task completion</h1>\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "805163f0-e5fe-4753-9e38-dc0f4c85bd2c",
   "metadata": {},
   "source": [
    "Recent papers such as [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://arxiv.org/abs/2109.01247) and [What Makes Good In-Context Examples for GPT-3?](https://aclanthology.org/2022.deelio-1.10.pdf) have shown that using dynamic set of examples instead of fixed set of examples help GPT-3 to perfom the task with higher accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b5655453",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# if needed, upgrade to the latest version of the OpenAI Python library\n",
    "%pip install --upgrade openai\n",
    "%pip install --upgrade torch\n",
    "%pip install --upgrade sentence_transformers\n",
    "%pip install --upgrade numpy\n",
    "%pip install --upgrade datasets\n",
    "%pip install --upgrade scikit-learn\n",
    "%pip install --upgrade transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ccbb9a99",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import os module & the OpenAI Python library for calling the OpenAI API\n",
    "# please make sure you have installed required libraries via pip install -r requirements.txt\n",
    "import os\n",
    "import dotenv\n",
    "import openai\n",
    "from sentence_transformers import SentenceTransformer, util\n",
    "import numpy as np\n",
    "from datasets import load_dataset\n",
    "from sklearn.metrics import classification_report\n",
    "from torch import cuda\n",
    "device = 'cuda' if cuda.is_available() else 'cpu'\n",
    "dotenv.load_dotenv()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6c650b9-ae37-46cb-a47a-2386f096adb7",
   "metadata": {},
   "source": [
    "## Dataset Summary\n",
    "\n",
    "The Text REtrieval Conference (TREC) Question Classification dataset is a dataset for question classification consisting of open-domain, fact-based questions divided into broad semantic categories. It contains 5500 labeled questions in training set and another 500 for test set.\n",
    "\n",
    "The dataset has 6 coarse class labels and 50 fine class labels. Average length of each sentence is 10, vocabulary size of 8700."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ceaaad07-d9e2-44e0-88aa-b72c9560ed10",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load dataset from Huggingface's dataset library\n",
    "dataset = load_dataset(\"trec\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a769332c-077c-49d1-81c0-88d6ee87daa0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['text', 'coarse_label', 'fine_label'],\n",
       "        num_rows: 5452\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['text', 'coarse_label', 'fine_label'],\n",
       "        num_rows: 500\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1cbbcf88-3d33-4c36-b4a8-399b11b8ca1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# name of the text and label column\n",
    "label_type = \"coarse_label\"\n",
    "text_key = \"text\"\n",
    "# create mapping of ids2class and class2id\n",
    "id2class = dict((i, label) for i, label in enumerate(dataset['train'].features[label_type].names))\n",
    "class2id = dict((label, i) for i, label in enumerate(dataset['train'].features[label_type].names))\n",
    "# create a dictionary with classes as key and containing all the training examples within that class\n",
    "class2TrainDataset = dict((label, []) for label in dataset['train'].features[label_type].names)\n",
    "for example in dataset['train']:\n",
    "    label = id2class[example[label_type]]\n",
    "    class2TrainDataset[label].append(example[text_key])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68e4ea26-5071-41d8-a142-3426ada87f58",
   "metadata": {},
   "source": [
    "# Task Prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0442beee-70fb-4555-9e11-654fe0fa9621",
   "metadata": {},
   "outputs": [],
   "source": [
    "# a prompt for asking LLM to perform a task\n",
    "task_prompt = \"As a Question Answering agent, your goal is to categorize questions into different semantic classes that impose constraints on potential answers, so that they can be utilized in later stages of the question answering process.\\nFollowing are the semantic classes: [\"\n",
    "task_prompt += \", \".join([label for label in class2TrainDataset]) + \"]\"\n",
    "# a prompt for asking LLM to generate the output for current task\n",
    "query_prompt = \"\\nClassify the following question into one of the above classes. Please answer in a single word.\\nquestion: \"\n",
    "answer_prompt = \"\\noutput: \""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "033c2541-4a80-4dc5-b773-bc85452aad7e",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Setup OpenAI API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "25a92251-1fab-46dc-9e98-29b7fb51a61e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setting up the deployment name\n",
    "deployment_name = os.environ['COMPLETIONS_MODEL']\n",
    "\n",
    "# The base URL for your Azure OpenAI resource. e.g. \"https://<your resource name>.openai.azure.com\"\n",
    "# This is the value of the endpoint for your Azure OpenAI resource\n",
    "azure_endpoint = os.environ['AZURE_OPENAI_ENDPOINT']\n",
    "\n",
    "# The API key for your Azure OpenAI resource.\n",
    "api_key = os.environ['AZURE_OPENAI_API_KEY']\n",
    "\n",
    "# Currently OPENAI API have the following versions available: https://learn.microsoft.com/azure/ai-services/openai/reference\n",
    "api_version = os.environ['OPENAI_API_VERSION']\n",
    "\n",
    "client = openai.AzureOpenAI(\n",
    "  api_key=api_key,  \n",
    "  azure_endpoint=azure_endpoint,\n",
    "  api_version=api_version\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a03c4f2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Text completion using GPT\n",
    "def trim_text(text):\n",
    "    return text.strip().strip('\\n').strip('\\\\n')\n",
    "\n",
    "def generate_using_gpt(prompt):\n",
    "    generated_sentence = \"\"\n",
    "    try:\n",
    "        # Create a completion for the provided prompt and parameters\n",
    "        # To know more about the parameters, checkout this documentation: https://learn.microsoft.com/en-us/azure/cognitive-services/openai/reference\n",
    "        response = client.completions.create(\n",
    "            model=deployment_name,\n",
    "            prompt=prompt, \n",
    "            max_tokens=3,\n",
    "            temperature=0,\n",
    "            top_p=1,\n",
    "            stop=None,\n",
    "            frequency_penalty=0,\n",
    "            presence_penalty=0.0)\n",
    "        \n",
    "        choices = response.choices\n",
    "        if len(choices) == 0 or not hasattr(choices[0], \"text\"):\n",
    "            print(\"Text not generated properly\")\n",
    "        generated_sentence = choices[0].text.lstrip('\\\\n').rstrip('\\\\n').lstrip('\\n\\n').rstrip('\\n\\n').lstrip('\\n').rstrip('\\n')\n",
    "\n",
    "    except openai.APITimeoutError as e:\n",
    "        # Handle request timeout\n",
    "        print(f\"Request timed out: {e}\")\n",
    "    \n",
    "    except openai.AuthenticationError as e:\n",
    "        # Handle Authentication error here, e.g. invalid API key\n",
    "        print(f\"OpenAI API returned an Authentication Error: {e}\")\n",
    "\n",
    "    except openai.APIConnectionError as e:\n",
    "        # Handle connection error here\n",
    "        print(f\"Failed to connect to OpenAI API: {e}\")\n",
    "\n",
    "    except openai.BadRequestError as e:\n",
    "        # Handle connection error here\n",
    "        print(f\"Invalid Request Error: {e}\")\n",
    "        \n",
    "    except openai.RateLimitError as e:\n",
    "        # Handle rate limit error\n",
    "        print(f\"OpenAI API request exceeded rate limit: {e}\")\n",
    "\n",
    "    except openai.InternalServerError as e:\n",
    "        # Handle Service Unavailable error\n",
    "        print(f\"Service Unavailable: {e}\")\n",
    "\n",
    "    except openai.APIError as e:\n",
    "        # Handle API error here, e.g. retry or log\n",
    "        print(f\"OpenAI API returned an API Error: {e}\")\n",
    "\n",
    "    return generated_sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "328d2126-c1b8-4942-ae60-e70f93265a3d",
   "metadata": {},
   "source": [
    "# Zero-shot Prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "057efe38-749e-4443-a565-2163506eac1d",
   "metadata": {},
   "source": [
    "### Example of the zero-shot prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8ea8ae71-68b9-41b0-a3a6-3c92a3f3dfae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "As a Question Answering agent, your goal is to categorize questions into different semantic classes that impose constraints on potential answers, so that they can be utilized in later stages of the question answering process.\n",
      "Following are the semantic classes: [ABBR, ENTY, DESC, HUM, LOC, NUM]\n",
      "Classify the following question into one of the above classes. Please answer in a single word.\n",
      "question: How far is it from Denver to Aspen ?\n",
      "output: \n"
     ]
    }
   ],
   "source": [
    "zeroshot_prompt = task_prompt +  query_prompt + dataset['test'][0][text_key] + answer_prompt\n",
    "print(zeroshot_prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "eee4a12e-3ce0-457a-ab23-c94c42071b36",
   "metadata": {},
   "outputs": [],
   "source": [
    "# prompt without any examples from the training dataset\n",
    "labels = []\n",
    "predictions = []\n",
    "for example in dataset['test']:\n",
    "    zeroshot_prompt = task_prompt +  query_prompt + example[text_key] + answer_prompt\n",
    "    pred = generate_using_gpt(zeroshot_prompt)\n",
    "    pred=trim_text(pred)\n",
    "    labels.append(example[label_type])\n",
    "    if pred not in class2id:\n",
    "        predictions.append(-1)\n",
    "    else:\n",
    "        predictions.append(class2id[pred])\n",
    "        \n",
    "report = classification_report(labels, predictions) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6aba42eb-7f20-45ea-a893-38cde49f6859",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.69      1.00      0.82         9\n",
      "           1       0.30      0.69      0.42        94\n",
      "           2       0.70      0.15      0.25       138\n",
      "           3       0.88      0.35      0.51        65\n",
      "           4       0.70      0.90      0.78        81\n",
      "           5       0.84      0.80      0.82       113\n",
      "\n",
      "    accuracy                           0.56       500\n",
      "   macro avg       0.69      0.65      0.60       500\n",
      "weighted avg       0.68      0.56      0.54       500\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(report)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "258912ca-9593-4fe0-9627-44dfcd93753a",
   "metadata": {},
   "source": [
    "# Few-shot Prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6269709e-a4e9-4a33-a4ff-96c6edc38204",
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to selection few examples in each of the classes from the training dataset\n",
    "def generateFewshotPrompt(class2TrainDataset, N=3):\n",
    "    fewshot_prompt = \"\\nFollowing are some examples.\"\n",
    "    for label in class2TrainDataset:\n",
    "        for example in class2TrainDataset[label][:N]:\n",
    "            fewshot_prompt += \"\\nquestion: \" + example\n",
    "            fewshot_prompt += \"\\noutput: \" + label\n",
    "    return fewshot_prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "635c6777-da11-4d51-9cb8-df07eaf57c7b",
   "metadata": {},
   "source": [
    "### Example of the few-shot prompt "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "68c25be0-afda-4a45-bc67-ba2962ad32de",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "As a Question Answering agent, your goal is to categorize questions into different semantic classes that impose constraints on potential answers, so that they can be utilized in later stages of the question answering process.\n",
      "Following are the semantic classes: [ABBR, ENTY, DESC, HUM, LOC, NUM]\n",
      "Following are some examples.\n",
      "question: What is the full form of .com ?\n",
      "output: ABBR\n",
      "question: What films featured the character Popeye Doyle ?\n",
      "output: ENTY\n",
      "question: How did serfdom develop in and then leave Russia ?\n",
      "output: DESC\n",
      "question: What contemptible scoundrel stole the cork from my lunch ?\n",
      "output: HUM\n",
      "question: What sprawling U.S. state boasts the most airports ?\n",
      "output: LOC\n",
      "question: When was Ozzy Osbourne born ?\n",
      "output: NUM\n",
      "Classify the following question into one of the above classes. Please answer in a single word.\n",
      "question: How far is it from Denver to Aspen ?\n",
      "output: \n"
     ]
    }
   ],
   "source": [
    "# prompt with one example in each of the classes\n",
    "fewshot_examples = generateFewshotPrompt(class2TrainDataset, N=1)\n",
    "fewshot_prompt = task_prompt +  fewshot_examples + query_prompt + dataset['test'][0][text_key] + answer_prompt\n",
    "print(fewshot_prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "38f65537-04e7-4dee-a40d-a28973e27dbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "# prompt is created by adding one example in each of the classes \n",
    "labels = []\n",
    "predictions = []\n",
    "for example in dataset['test']:\n",
    "    fewshot_prompt = task_prompt + fewshot_examples + query_prompt + example[text_key] + answer_prompt\n",
    "    pred = generate_using_gpt(fewshot_prompt)\n",
    "    pred=trim_text(pred)\n",
    "    labels.append(example[label_type])\n",
    "    if pred not in class2id:\n",
    "        predictions.append(-1)\n",
    "    else:\n",
    "        predictions.append(class2id[pred])\n",
    "        \n",
    "report = classification_report(labels, predictions) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "724a5041-ded3-4fe9-9db9-d47556b6ba10",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.75      1.00      0.86         9\n",
      "           1       0.41      0.57      0.48        94\n",
      "           2       0.80      0.51      0.62       138\n",
      "           3       0.96      0.69      0.80        65\n",
      "           4       0.93      0.93      0.93        81\n",
      "           5       0.80      1.00      0.89       113\n",
      "\n",
      "    accuracy                           0.73       500\n",
      "   macro avg       0.77      0.78      0.76       500\n",
      "weighted avg       0.77      0.73      0.73       500\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(report)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1e1d828-72fa-4540-8995-157cce2c82c8",
   "metadata": {},
   "source": [
    "# Extract Embeddings for Training dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "bf81306b-1e58-49a1-913b-2569328d609c",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# loading Sentence Transformer based model\n",
    "model = SentenceTransformer('all-mpnet-base-v2', device=device)\n",
    "\n",
    "# extract embeddings for a set of examples\n",
    "def ExtractEmbeddings(examples):\n",
    "    embedding_ls = []\n",
    "    for example in examples:\n",
    "        embedding = model.encode(example)     \n",
    "        embedding_ls.append(embedding)\n",
    "    return embedding_ls\n",
    "\n",
    "# extract embeddings for all the training examples\n",
    "class2TrainDatasetWithEmbedding = {}\n",
    "for label in class2TrainDataset:\n",
    "    embeddings = ExtractEmbeddings(class2TrainDataset[label])\n",
    "    class2TrainDatasetWithEmbedding[label] = [class2TrainDataset[label], embeddings]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab8c8ce6-60a0-4abc-8d00-5cbc82ba6c00",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Dynamic Few-shot Prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "a42cb4a5-fa35-441e-96ec-76edcd5ae4d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# extract similar queries for a given input text from each of the classes\n",
    "def getSimilarExamples(input_text, dataset, dataset_embedding):\n",
    "    input_embedding = model.encode(input_text)\n",
    "    sim_score = util.dot_score(input_embedding, dataset_embedding)[0]\n",
    "    topN_ids = np.argsort(-sim_score)\n",
    "    return [dataset[i] for i in topN_ids]\n",
    "    \n",
    "def getClasswiseSimilarExamples(input_text, class2TrainDatasetWithEmbedding):\n",
    "    classwiseSimilarExamples = {}\n",
    "    for label in class2TrainDataset:\n",
    "        similarExamples = getSimilarExamples(input_text, class2TrainDatasetWithEmbedding[label][0], class2TrainDatasetWithEmbedding[label][1])\n",
    "        classwiseSimilarExamples[label] = similarExamples\n",
    "    return classwiseSimilarExamples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "78d83dfc-fa32-4d86-a730-6a63ff3578dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate a prompt with similar examples in each of the classes\n",
    "def generateDynamicPrompt(input_text, class2TrainDatasetWithEmbedding, N=3):\n",
    "    classwiseSimilarExamples = getClasswiseSimilarExamples(input_text, class2TrainDatasetWithEmbedding)\n",
    "    dynamic_prompt = \"\\nFollowing are some examples.\"\n",
    "    for label in classwiseSimilarExamples:\n",
    "        for example in classwiseSimilarExamples[label][:N]:\n",
    "            dynamic_prompt += \"\\nquestion: \" + example\n",
    "            dynamic_prompt += \"\\noutput: \" + label\n",
    "    return dynamic_prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "456e460c-6734-47fe-9ca5-3d2f78eb144c",
   "metadata": {},
   "source": [
    "### Example of the dynamic prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c6c4482-a00f-4a0c-8809-531916ec239e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# dynamic prompt with one similar example in each of the classes\n",
    "fewshot_examples = generateDynamicPrompt(dataset['test'][0][text_key], class2TrainDatasetWithEmbedding, N=1)\n",
    "dynamic_prompt = task_prompt + fewshot_examples + query_prompt + dataset['test'][0][text_key] + answer_prompt\n",
    "print(dynamic_prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "fc17023f-678b-4386-b7e7-541282b30f04",
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = []\n",
    "predictions = []\n",
    "for example in dataset['test']:\n",
    "    fewshot_examples = generateDynamicPrompt(example[text_key], class2TrainDatasetWithEmbedding, N=1)\n",
    "    dynamic_prompt = task_prompt + fewshot_examples + query_prompt + example[text_key] + answer_prompt\n",
    "    pred = generate_using_gpt(dynamic_prompt)\n",
    "    pred=trim_text(pred)\n",
    "    labels.append(example[label_type])\n",
    "    if pred not in class2id:\n",
    "        predictions.append(-1)\n",
    "    else:\n",
    "        predictions.append(class2id[pred])\n",
    "        \n",
    "report = classification_report(labels, predictions) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "826584d8-8223-4f88-89cc-65181212a1fb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.69      1.00      0.82         9\n",
      "           1       0.66      0.76      0.70        94\n",
      "           2       0.88      0.71      0.79       138\n",
      "           3       0.95      0.91      0.93        65\n",
      "           4       0.94      0.90      0.92        81\n",
      "           5       0.88      0.99      0.93       113\n",
      "\n",
      "    accuracy                           0.84       500\n",
      "   macro avg       0.83      0.88      0.85       500\n",
      "weighted avg       0.85      0.84      0.84       500\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(report)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}