Semantic_Search_+_GPT_3_.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyPAhaJS++e17eQuFEEoZdm0",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/PradipNichite/Youtube-Tutorials/blob/main/Semantic_Search_%2B_GPT_3_.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install sentence_transformers pinecone-client openai -q"
      ],
      "metadata": {
        "id": "RU4Phx0n6TEM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import pinecone\n",
        "from sentence_transformers import SentenceTransformer,util\n",
        "import requests\n",
        "from bs4 import BeautifulSoup\n",
        "model = SentenceTransformer('all-MiniLM-L6-v2') #384 dimensional"
      ],
      "metadata": {
        "id": "gxSbuLRY6NpP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model.encode(\"Ronaldo began his senior career with Sporting CP, before signing with Manchester United in 2003, winning the FA Cup in his first season. He would also go on to win three consecutive Premier League titles, the Champions\").tolist()"
      ],
      "metadata": {
        "id": "CWsyO-1JHkMZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def get_html_content(url):\n",
        "    response = requests.get(url)\n",
        "    return response.content"
      ],
      "metadata": {
        "id": "3GUmc3uh6jP9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def get_plain_text(html_content):\n",
        "    soup = BeautifulSoup(html_content, 'html.parser')\n",
        "    for script in soup([\"script\"]):\n",
        "        script.extract()\n",
        "    return soup.get_text()"
      ],
      "metadata": {
        "id": "Ao498Caz6mHG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def split_text_into_chunks(plain_text, max_chars=2000):\n",
        "    text_chunks = []\n",
        "    current_chunk = \"\"\n",
        "    for line in plain_text.split(\"\\n\"):\n",
        "        if len(current_chunk) + len(line) + 1 <= max_chars:\n",
        "            current_chunk += line + \" \"\n",
        "        else:\n",
        "            text_chunks.append(current_chunk.strip())\n",
        "            current_chunk = line + \" \"\n",
        "    if current_chunk:\n",
        "        text_chunks.append(current_chunk.strip())\n",
        "    return text_chunks"
      ],
      "metadata": {
        "id": "R7-yjzAI6rkC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7CoOj5LCPbQl"
      },
      "outputs": [],
      "source": [
        "def scrape_text_from_url(url, max_chars=2000):\n",
        "    html_content = get_html_content(url)\n",
        "    plain_text = get_plain_text(html_content)\n",
        "    text_chunks = split_text_into_chunks(plain_text, max_chars)\n",
        "    return text_chunks"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "url = input(\"Enter the URL to scrape text from: \")\n",
        "plain_text_chunks = scrape_text_from_url(url)\n",
        "print(plain_text_chunks)"
      ],
      "metadata": {
        "id": "m_nQbQuX6y4p"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "len(plain_text_chunks)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZYRzwr0qPiIx",
        "outputId": "8d3b67d4-4c8a-4b2f-94fb-86c653d57c82"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "104"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "plain_text_chunks[10]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 261
        },
        "id": "ZjHzpxPCRB2D",
        "outputId": "d6ef4cad-c743-4a83-a7b2-6e15e67da373"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Raj Singh Dungarpur is credited for the selection of Tendulkar for the Indian tour of Pakistan in late 1989,[67] after one first class season.[68] The Indian selection committee had shown interest in selecting Tendulkar for the tour of the West Indies held earlier that year, but eventually did not select him, as they did not want him to be exposed to the dominant fast bowlers of the West Indies so early in his career. Tendulkar made his Test debut against Pakistan in Karachi in November 1989 aged 16 years and 205 days. He made 15 runs, being bowled by Waqar Younis, who also made his debut in that match, but was noted for how he handled numerous blows to his body at the hands of the Pakistani pace attack.[69] In the fourth and final Test in Sialkot, he was hit on the nose by a bouncer bowled by Younis, but he declined medical assistance and continued to bat even as he gushed blood from it.[70] In a 20-over exhibition game in Peshawar, held in parallel with the bilateral series, Tendulkar made 53 runs off 18 balls, including an over in which he scored 27 runs (6, 4, 0, 6, 6, 6) off leg-spinner Abdul Qadir.[71] This was later called \"one of the best innings I have seen\" by the then Indian captain Krishnamachari Srikkanth.[72] In all, he scored 215 runs at an average of 35.83 in the Test series, and was dismissed without scoring a run in the only One Day International (ODI) he played.[73][74] Thus Sachin Tendulkar became the youngest player to debut for India in Tests at the age of 16 years and 205 days and also the youngest player to debut for India in ODIs at the age of 16 years and 238 days.[75][76]'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pinecone.init(api_key=\"api_key\", environment=\"env\") \n",
        "index = pinecone.Index(\"indexname\")\n",
        "index"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "viHnRkPcSzeP",
        "outputId": "d3f9ad6c-8615-484d-dd39-0705c5af2ed6"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<pinecone.index.Index at 0x7f2b86cfaf10>"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "index.describe_index_stats()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0hqYAfQHfdQ9",
        "outputId": "9150a470-4d90-4697-b42c-4b58bdd17976"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'dimension': 384,\n",
              " 'index_fullness': 0.0,\n",
              " 'namespaces': {},\n",
              " 'total_vector_count': 0}"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def addData(corpusData,url):\n",
        "    id  = index.describe_index_stats()['total_vector_count']\n",
        "    for i in range(len(corpusData)):\n",
        "        chunk=corpusData[i]\n",
        "        chunkInfo=(str(id+i),\n",
        "                model.encode(chunk).tolist(),\n",
        "                {'title': url,'context': chunk})\n",
        "        index.upsert(vectors=[chunkInfo])"
      ],
      "metadata": {
        "id": "yShks1P8S10Z"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "plain_text_chunks[10]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 174
        },
        "id": "-ff-7Jb-H7bW",
        "outputId": "26ff386c-0133-4c3d-ccfd-3d201449f5e0"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Raj Singh Dungarpur is credited for the selection of Tendulkar for the Indian tour of Pakistan in late 1989,[67] after one first class season.[68] The Indian selection committee had shown interest in selecting Tendulkar for the tour of the West Indies held earlier that year, but eventually did not select him, as they did not want him to be exposed to the dominant fast bowlers of the West Indies so early in his career. Tendulkar made his Test debut against Pakistan in Karachi in November 1989 aged 16 years and 205 days. He made 15 runs, being bowled by Waqar Younis, who also made his debut in that match, but was noted for how he handled numerous blows to his body at the hands of the Pakistani pace attack.[69] In the fourth and final Test in Sialkot, he was hit on the nose by a bouncer bowled by Younis, but he declined medical assistance and continued to bat even as he gushed blood from it.[70] In a 20-over exhibition game in Peshawar, held in parallel with the bilateral series, Tendulkar made 53 runs off 18 balls, including an over in which he scored 27 runs (6, 4, 0, 6, 6, 6) off leg-spinner Abdul Qadir.[71] This was later called \"one of the best innings I have seen\" by the then Indian captain Krishnamachari Srikkanth.[72] In all, he scored 215 runs at an average of 35.83 in the Test series, and was dismissed without scoring a run in the only One Day International (ODI) he played.[73][74] Thus Sachin Tendulkar became the youngest player to debut for India in Tests at the age of 16 years and 205 days and also the youngest player to debut for India in ODIs at the age of 16 years and 238 days.[75][76]'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "addData(plain_text_chunks,url)"
      ],
      "metadata": {
        "id": "rx10Okg1fH0b"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def find_match(query,k):\n",
        "    query_em = model.encode(query).tolist()\n",
        "    result = index.query(query_em, top_k=k, includeMetadata=True)\n",
        "    \n",
        "    return [result['matches'][i]['metadata']['title'] for i in range(k)],[result['matches'][i]['metadata']['context'] for i in range(k)]"
      ],
      "metadata": {
        "id": "B51s4xGkggXQ"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "find_match(\"What awards was tendulakar awarded with?\",2)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JYkUoNDIg1Qo",
        "outputId": "d6846596-4644-450a-9f75-b5feb222c153"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(['https://en.wikipedia.org/wiki/Sachin_Tendulkar',\n",
              "  'https://en.wikipedia.org/wiki/Sachin_Tendulkar'],\n",
              " ['Tendulkar took up cricket at the age of eleven, made his Test match debut on 15 November 1989 against Pakistan in Karachi at the age of sixteen, and went on to represent Mumbai domestically and India internationally for close to twenty-four years. In 2002, halfway through his career, Wisden ranked him the second-greatest Test batsman of all time, behind Don Bradman, and the second-greatest ODI batsman of all time, behind Viv Richards.[8] Later in his career, Tendulkar was part of the Indian team that won the 2011 Cricket World Cup, his first win in six World Cup appearances for India.[9] He had previously been named \"Player of the Tournament\" at the 2003 edition of the tournament. Tendulkar received the Arjuna Award in 1994 for his outstanding sporting achievements, the Khel Ratna Award, India\\'s highest sporting honour, in 1997, and the Padma Shri and Padma Vibhushan awards in 1999 and 2008, respectively, two of India\\'s highest civilian awards.[10][11] A few hours after the end of his last match in November 2013, the Prime Minister\\'s Office announced the decision to award him the Bharat Ratna, India\\'s highest civilian award.[12][13] As of 2021, he is the youngest recipient to date and was the first sportsperson to receive the award.[14][15] In 2012, Tendulkar was nominated to the Rajya Sabha, the upper house of the Parliament of India.[16] In 2010, Time magazine included Tendulkar in its annual Time 100 list as one of the most influential people in the world.[17] Tendulkar was awarded the Sir Garfield Sobers Trophy for cricketer of the year at the 2010 ICC Awards.[18]',\n",
              "  \"An innings-by-innings breakdown of Tendulkar's Test match batting career showing runs scored (red and green bars) and the average of the last ten innings (blue line) National honours India 1994 – Arjuna Award, by the Government of India in recognition of his outstanding achievement in sports.[371] 1997–98 – Khel Ratna Award, India's highest honour given for achievement in sports.[372] 1999 –  Padma Shri, India's fourth-highest civilian award.[373] 2001 – Maharashtra Bhushan Award, Maharashtra state's highest civilian award.[374] 2008 –  Padma Vibhushan, India's second-highest civilian award.[375] 2014 –  Bharat Ratna, India's highest civilian award.[14][15] Australia 2012 –  Honorary Member of the Order of Australia, given by the Australian government.[376][377] Other honours .mw-parser-output .tmulti .multiimageinner{display:flex;flex-direction:column}.mw-parser-output .tmulti .trow{display:flex;flex-direction:row;clear:left;flex-wrap:wrap;width:100%;box-sizing:border-box}.mw-parser-output .tmulti .tsingle{margin:1px;float:left}.mw-parser-output .tmulti .theader{clear:both;font-weight:bold;text-align:center;align-self:center;background-color:transparent;width:100%}.mw-parser-output .tmulti .thumbcaption{background-color:transparent}.mw-parser-output .tmulti .text-align-left{text-align:left}.mw-parser-output .tmulti .text-align-right{text-align:right}.mw-parser-output .tmulti .text-align-center{text-align:center}@media all and (max-width:720px){.mw-parser-output .tmulti .thumbinner{width:100%!important;box-sizing:border-box;max-width:none!important;align-items:center}.mw-parser-output .tmulti .trow{justify-content:center}.mw-parser-output .tmulti .tsingle{float:none!important;max-width:100%!important;box-sizing:border-box;text-align:center}.mw-parser-output .tmulti .tsingle .thumbcaption{text-align:left}.mw-parser-output .tmulti .trow>.thumbcaption{text-align:center}}2013 Indian postage stamps commemorating Sachin Tendulkar's 200th Test Match\"])"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Prompt\n",
        "Answer the question as truthfully as possible using the provided context, and if the answer is not contained within the text and requires some latest information to be updated, print 'Sorry Not Sufficient context to answer query' \\n\"\n",
        "    return header + context + \"\\n\\n\" + query + \"\\n\""
      ],
      "metadata": {
        "id": "ZKikjxJ17-wK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import openai\n",
        "openai.api_key=\"openai key\"\n",
        "def create_prompt(context,query):\n",
        "    header = \"Answer the question as truthfully as possible using the provided context, and if the answer is not contained within the text and requires some latest information to be updated, print 'Sorry Not Sufficient context to answer query' \\n\"\n",
        "    return header + context + \"\\n\\n\" + query + \"\\n\"\n",
        "\n",
        "def generate_answer(prompt):\n",
        "    response = openai.Completion.create(\n",
        "    model=\"text-davinci-003\",\n",
        "    prompt=prompt,\n",
        "    temperature=0,\n",
        "    max_tokens=256,\n",
        "    top_p=1,\n",
        "    frequency_penalty=0,\n",
        "    presence_penalty=0,\n",
        "    stop = [' END']\n",
        "    )\n",
        "    return (response.choices[0].text).strip()"
      ],
      "metadata": {
        "id": "b0TJSLqthbtz"
      },
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "query = \"Who is Sachin?\"\n",
        "docs,res = find_match(query,1)"
      ],
      "metadata": {
        "id": "s3H-OHM4g47o"
      },
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "res"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k92EZhmBJXNJ",
        "outputId": "322d42bf-7486-49a1-81bb-ac97f1db2e19"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['^ \"Sachin Tendulkar: Who is he?\". India Today. 3 April 2012. Retrieved 30 August 2021.  ^ Ghosh, Avijit (5 January 2008). \"Now, Oz-speak about Brahmin dominance!\". The Times of India. Retrieved 30 August 2021.  ^ Thani, L.; Mishra, R. (1999). Sensational Sachin. Diamond Pocket Books. p.\\xa0113. ISBN\\xa081-288-2573-9. His mother Rajni Tendulkar worked in L.I.C.  ^ \"Ajit Dotes on Sachin, Who Could\\'ve Guessed They\\'re Half-Brothers!\". The Quint. Retrieved 2 September 2018.  ^ \"This man sacrificed his cricket, career for Sachin Tendulkar\". Asianet Newsable.  ^ \"38 special facts about Sachin Tendulkar – 3\". MSN. 16 March 2012. Archived from the original on 6 June 2013. Retrieved 18 December 2012.  ^ a b \"\\'कांबली बहुत गुस्सैल है,\\' सचिन ने KBC में अमिताभ को सुनाई थी दोस्त की कहानी; देखें Video\". Jansatta (in Hindi). Retrieved 24 December 2021.  ^ \"38 special facts about Sachin Tendulkar – 2\". MSN. 16 March 2012. Archived from the original on 6 June 2013. Retrieved 18 December 2012.  ^ \"Throwback to Sachin Tendulkar\\'s Comments About Vinod Kambli on KBC That Surprised Amitabh Bachchan\". www.news18.com. Retrieved 24 December 2021.  ^ \"Reminiscing old days: Sachin Tendulkar on late coach Ramakant Achrekar\". in.com.[dead link]  ^ Pandey, Kirti Phadtare (25 May 2017). \"Ajit Dotes on Sachin, Who Could\\'ve Guessed They\\'re Half-Brothers!\". TheQuint. Retrieved 24 December 2021.  ^ a b c d e Gulu Ezekiel. \"HTCricket.com: A special HTCricket section celebrating Sachin Tendulkar\\'s 100th Test\". Hindustan Times. Archived from the original on 25 October 2012. Retrieved 18 December 2012.  ^ Baliga, Linah (1 April 2011). \"Shivaji Park prays for famous son\". The Times of India. Times News Network. Archived from the original on 1 May 2013.  ^ Somak Sen (17 October 2008). \"Sachin Tendulkar: The last word in cricket\". Merinews. Archived from the original on 29 January 2013. Retrieved 3 June 2009.']"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "context= \"\\n\\n\".join(res)\n",
        "prompt = create_prompt(context,query)\n",
        "print(prompt)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UVJGPpt4hQ8-",
        "outputId": "e76ca4ac-9743-4130-ba79-799b5cf6b90d"
      },
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Answer the question as truthfully as possible using the provided context, and if the answer is not contained within the text and requires some latest information to be updated, print 'Sorry Not Sufficient context to answer query' \n",
            "^ \"Sachin Tendulkar: Who is he?\". India Today. 3 April 2012. Retrieved 30 August 2021.  ^ Ghosh, Avijit (5 January 2008). \"Now, Oz-speak about Brahmin dominance!\". The Times of India. Retrieved 30 August 2021.  ^ Thani, L.; Mishra, R. (1999). Sensational Sachin. Diamond Pocket Books. p. 113. ISBN 81-288-2573-9. His mother Rajni Tendulkar worked in L.I.C.  ^ \"Ajit Dotes on Sachin, Who Could've Guessed They're Half-Brothers!\". The Quint. Retrieved 2 September 2018.  ^ \"This man sacrificed his cricket, career for Sachin Tendulkar\". Asianet Newsable.  ^ \"38 special facts about Sachin Tendulkar – 3\". MSN. 16 March 2012. Archived from the original on 6 June 2013. Retrieved 18 December 2012.  ^ a b \"'कांबली बहुत गुस्सैल है,' सचिन ने KBC में अमिताभ को सुनाई थी दोस्त की कहानी; देखें Video\". Jansatta (in Hindi). Retrieved 24 December 2021.  ^ \"38 special facts about Sachin Tendulkar – 2\". MSN. 16 March 2012. Archived from the original on 6 June 2013. Retrieved 18 December 2012.  ^ \"Throwback to Sachin Tendulkar's Comments About Vinod Kambli on KBC That Surprised Amitabh Bachchan\". www.news18.com. Retrieved 24 December 2021.  ^ \"Reminiscing old days: Sachin Tendulkar on late coach Ramakant Achrekar\". in.com.[dead link]  ^ Pandey, Kirti Phadtare (25 May 2017). \"Ajit Dotes on Sachin, Who Could've Guessed They're Half-Brothers!\". TheQuint. Retrieved 24 December 2021.  ^ a b c d e Gulu Ezekiel. \"HTCricket.com: A special HTCricket section celebrating Sachin Tendulkar's 100th Test\". Hindustan Times. Archived from the original on 25 October 2012. Retrieved 18 December 2012.  ^ Baliga, Linah (1 April 2011). \"Shivaji Park prays for famous son\". The Times of India. Times News Network. Archived from the original on 1 May 2013.  ^ Somak Sen (17 October 2008). \"Sachin Tendulkar: The last word in cricket\". Merinews. Archived from the original on 29 January 2013. Retrieved 3 June 2009.\n",
            "\n",
            "Who is Sachin?\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "reply = generate_answer(prompt)\n",
        "print(reply)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wXCBNxRIiZQP",
        "outputId": "3b01ff34-aaa9-4cd0-c6f4-0f6b6373e757"
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sachin Tendulkar is an Indian cricketer widely regarded as one of the greatest batsmen in the history of cricket. He is the highest run scorer of all time in International cricket. He is the only player to have scored one hundred international centuries, the first batsman to score a double century in a One Day International, and the only player to complete more than 30,000 runs in international cricket.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "WcLMjYeZJfW6"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}