ElasticSearch_on_Colab_7th_Sep_2023.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "authorship_tag": "ABX9TyNwlRMfi4kjlBdtEp3V72zs",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/pranaya-mathur/Deep-Learning-Projects/blob/master/ElasticSearch_on_Colab_7th_Sep_2023.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "\n",
        "pip install --upgrade pip\n",
        "pip install farm-haystack[colab,preprocessing,elasticsearch,inference]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_ZqTVoNtVXss",
        "outputId": "7dd9e1db-33ef-4290-ae0b-9999aacc0cb7"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: pip in /usr/local/lib/python3.10/dist-packages (23.1.2)\n",
            "Collecting pip\n",
            "  Downloading pip-23.2.1-py3-none-any.whl (2.1 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 10.5 MB/s eta 0:00:00\n",
            "Installing collected packages: pip\n",
            "  Attempting uninstall: pip\n",
            "    Found existing installation: pip 23.1.2\n",
            "    Uninstalling pip-23.1.2:\n",
            "      Successfully uninstalled pip-23.1.2\n",
            "Successfully installed pip-23.2.1\n",
            "Collecting farm-haystack[colab,elasticsearch,inference,preprocessing]\n",
            "  Obtaining dependency information for farm-haystack[colab,elasticsearch,inference,preprocessing] from https://files.pythonhosted.org/packages/a3/e3/8a7ebb563df04300c9e2387ac31e8ffda809b21f276faa838b32cf4a74a7/farm_haystack-1.20.0-py3-none-any.whl.metadata\n",
            "  Downloading farm_haystack-1.20.0-py3-none-any.whl.metadata (25 kB)\n",
            "Collecting boilerpy3 (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Downloading boilerpy3-1.0.6-py3-none-any.whl (22 kB)\n",
            "Collecting canals==0.7.0 (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Obtaining dependency information for canals==0.7.0 from https://files.pythonhosted.org/packages/08/2f/62e9e455c4cf183ebf6672c47befabb6a36a37a8cdc86256a5632d6ace10/canals-0.7.0-py3-none-any.whl.metadata\n",
            "  Downloading canals-0.7.0-py3-none-any.whl.metadata (4.4 kB)\n",
            "Collecting events (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Obtaining dependency information for events from https://files.pythonhosted.org/packages/25/ed/e47dec0626edd468c84c04d97769e7ab4ea6457b7f54dcb3f72b17fcd876/Events-0.5-py3-none-any.whl.metadata\n",
            "  Downloading Events-0.5-py3-none-any.whl.metadata (3.9 kB)\n",
            "Requirement already satisfied: jsonschema in /usr/local/lib/python3.10/dist-packages (from farm-haystack[colab,elasticsearch,inference,preprocessing]) (4.19.0)\n",
            "Collecting lazy-imports==0.3.1 (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Downloading lazy_imports-0.3.1-py3-none-any.whl (12 kB)\n",
            "Requirement already satisfied: more-itertools in /usr/local/lib/python3.10/dist-packages (from farm-haystack[colab,elasticsearch,inference,preprocessing]) (10.1.0)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from farm-haystack[colab,elasticsearch,inference,preprocessing]) (3.1)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from farm-haystack[colab,elasticsearch,inference,preprocessing]) (1.5.3)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.10/dist-packages (from farm-haystack[colab,elasticsearch,inference,preprocessing]) (9.4.0)\n",
            "Requirement already satisfied: platformdirs in /usr/local/lib/python3.10/dist-packages (from farm-haystack[colab,elasticsearch,inference,preprocessing]) (3.10.0)\n",
            "Collecting posthog (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Obtaining dependency information for posthog from https://files.pythonhosted.org/packages/a7/73/35758818228c70348be4c3c66a76653c62e894e0e3c3461453c5341ca926/posthog-3.0.2-py2.py3-none-any.whl.metadata\n",
            "  Downloading posthog-3.0.2-py2.py3-none-any.whl.metadata (2.0 kB)\n",
            "Collecting prompthub-py==4.0.0 (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Obtaining dependency information for prompthub-py==4.0.0 from https://files.pythonhosted.org/packages/27/5f/8c4939e290ff93af79364b88ffe3902d29c234f94e8227cf0b7fce3c887f/prompthub_py-4.0.0-py3-none-any.whl.metadata\n",
            "  Downloading prompthub_py-4.0.0-py3-none-any.whl.metadata (2.2 kB)\n",
            "Collecting pydantic<2 (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Obtaining dependency information for pydantic<2 from https://files.pythonhosted.org/packages/bc/e0/0371e9b6c910afe502e5fe18cc94562bfd9399617c7b4f5b6e13c29115b3/pydantic-1.10.12-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata\n",
            "  Downloading pydantic-1.10.12-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (149 kB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 149.3/149.3 kB 6.8 MB/s eta 0:00:00\n",
            "Collecting quantulum3 (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Downloading quantulum3-0.9.0-py3-none-any.whl (10.7 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.7/10.7 MB 100.5 MB/s eta 0:00:00\n",
            "Collecting rank-bm25 (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Downloading rank_bm25-0.2.2-py3-none-any.whl (8.6 kB)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from farm-haystack[colab,elasticsearch,inference,preprocessing]) (2.31.0)\n",
            "Collecting requests-cache<1.0.0 (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Downloading requests_cache-0.9.8-py3-none-any.whl (48 kB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.7/48.7 kB 6.8 MB/s eta 0:00:00\n",
            "Collecting scikit-learn>=1.3.0 (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Obtaining dependency information for scikit-learn>=1.3.0 from https://files.pythonhosted.org/packages/5c/e9/ee572691a3fb05555bcde41826faad29ae4bc1fb07982e7f53d54a176879/scikit_learn-1.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata\n",
            "  Downloading scikit_learn-1.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)\n",
            "Collecting sseclient-py (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Obtaining dependency information for sseclient-py from https://files.pythonhosted.org/packages/49/58/97655efdfeb5b4eeab85b1fc5d3fa1023661246c2ab2a26ea8e47402d4f2/sseclient_py-1.8.0-py2.py3-none-any.whl.metadata\n",
            "  Downloading sseclient_py-1.8.0-py2.py3-none-any.whl.metadata (2.0 kB)\n",
            "Requirement already satisfied: tenacity in /usr/local/lib/python3.10/dist-packages (from farm-haystack[colab,elasticsearch,inference,preprocessing]) (8.2.3)\n",
            "Collecting tiktoken>=0.3.2 (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Downloading tiktoken-0.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 85.0 MB/s eta 0:00:00\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from farm-haystack[colab,elasticsearch,inference,preprocessing]) (4.66.1)\n",
            "Collecting transformers==4.32.1 (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Obtaining dependency information for transformers==4.32.1 from https://files.pythonhosted.org/packages/83/8d/f65f8138365462ace54458a9e164f4b28ce1141361970190eef36bdef986/transformers-4.32.1-py3-none-any.whl.metadata\n",
            "  Downloading transformers-4.32.1-py3-none-any.whl.metadata (118 kB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.5/118.5 kB 14.7 MB/s eta 0:00:00\n",
            "Collecting pillow (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Downloading Pillow-9.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.3/4.3 MB 97.3 MB/s eta 0:00:00\n",
            "Collecting langdetect (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Downloading langdetect-1.0.9.tar.gz (981 kB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 981.5/981.5 kB 77.3 MB/s eta 0:00:00\n",
            "  Preparing metadata (setup.py): started\n",
            "  Preparing metadata (setup.py): finished with status 'done'\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from farm-haystack[colab,elasticsearch,inference,preprocessing]) (3.8.1)\n",
            "Collecting huggingface-hub>=0.5.0 (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Obtaining dependency information for huggingface-hub>=0.5.0 from https://files.pythonhosted.org/packages/7f/c4/adcbe9a696c135578cabcbdd7331332daad4d49b7c43688bc2d36b3a47d2/huggingface_hub-0.16.4-py3-none-any.whl.metadata\n",
            "  Downloading huggingface_hub-0.16.4-py3-none-any.whl.metadata (12 kB)\n",
            "Collecting sentence-transformers>=2.2.0 (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Downloading sentence-transformers-2.2.2.tar.gz (85 kB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.0/86.0 kB 11.9 MB/s eta 0:00:00\n",
            "  Preparing metadata (setup.py): started\n",
            "  Preparing metadata (setup.py): finished with status 'done'\n",
            "Requirement already satisfied: pyyaml<7.0,>=6.0 in /usr/local/lib/python3.10/dist-packages (from prompthub-py==4.0.0->farm-haystack[colab,elasticsearch,inference,preprocessing]) (6.0.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.32.1->farm-haystack[colab,elasticsearch,inference,preprocessing]) (3.12.2)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.32.1->farm-haystack[colab,elasticsearch,inference,preprocessing]) (1.23.5)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.32.1->farm-haystack[colab,elasticsearch,inference,preprocessing]) (23.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.32.1->farm-haystack[colab,elasticsearch,inference,preprocessing]) (2023.6.3)\n",
            "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers==4.32.1->farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 120.3 MB/s eta 0:00:00\n",
            "Collecting safetensors>=0.3.1 (from transformers==4.32.1->farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Obtaining dependency information for safetensors>=0.3.1 from https://files.pythonhosted.org/packages/6c/f0/c17bbdb1e5f9dab29d44cade445135789f75f8f08ea2728d04493ea8412b/safetensors-0.3.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata\n",
            "  Downloading safetensors-0.3.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.7 kB)\n",
            "Requirement already satisfied: torch!=1.12.0,>=1.9 in /usr/local/lib/python3.10/dist-packages (from transformers==4.32.1->farm-haystack[colab,elasticsearch,inference,preprocessing]) (2.0.1+cu118)\n",
            "Collecting accelerate>=0.20.3 (from transformers==4.32.1->farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Obtaining dependency information for accelerate>=0.20.3 from https://files.pythonhosted.org/packages/4d/a7/05c67003d659a0035f2b3a8cf389c1d9645865aee84a73ce99ddab16682f/accelerate-0.22.0-py3-none-any.whl.metadata\n",
            "  Downloading accelerate-0.22.0-py3-none-any.whl.metadata (17 kB)\n",
            "Collecting sentencepiece!=0.1.92,>=0.1.91 (from transformers==4.32.1->farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Downloading sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 68.1 MB/s eta 0:00:00\n",
            "Requirement already satisfied: protobuf in /usr/local/lib/python3.10/dist-packages (from transformers==4.32.1->farm-haystack[colab,elasticsearch,inference,preprocessing]) (3.20.3)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.5.0->farm-haystack[colab,elasticsearch,inference,preprocessing]) (2023.6.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.5.0->farm-haystack[colab,elasticsearch,inference,preprocessing]) (4.7.1)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->farm-haystack[colab,elasticsearch,inference,preprocessing]) (3.2.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->farm-haystack[colab,elasticsearch,inference,preprocessing]) (3.4)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->farm-haystack[colab,elasticsearch,inference,preprocessing]) (2.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->farm-haystack[colab,elasticsearch,inference,preprocessing]) (2023.7.22)\n",
            "Requirement already satisfied: appdirs>=1.4.4 in /usr/local/lib/python3.10/dist-packages (from requests-cache<1.0.0->farm-haystack[colab,elasticsearch,inference,preprocessing]) (1.4.4)\n",
            "Requirement already satisfied: attrs>=21.2 in /usr/local/lib/python3.10/dist-packages (from requests-cache<1.0.0->farm-haystack[colab,elasticsearch,inference,preprocessing]) (23.1.0)\n",
            "Collecting cattrs>=22.2 (from requests-cache<1.0.0->farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Obtaining dependency information for cattrs>=22.2 from https://files.pythonhosted.org/packages/3a/ba/05df14efaa0624fac6b1510e87f5ce446208d2f6ce50270a89b6268aebfe/cattrs-23.1.2-py3-none-any.whl.metadata\n",
            "  Downloading cattrs-23.1.2-py3-none-any.whl.metadata (9.3 kB)\n",
            "Collecting url-normalize>=1.4 (from requests-cache<1.0.0->farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Downloading url_normalize-1.4.3-py2.py3-none-any.whl (6.8 kB)\n",
            "Requirement already satisfied: scipy>=1.5.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=1.3.0->farm-haystack[colab,elasticsearch,inference,preprocessing]) (1.10.1)\n",
            "Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=1.3.0->farm-haystack[colab,elasticsearch,inference,preprocessing]) (1.3.2)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=1.3.0->farm-haystack[colab,elasticsearch,inference,preprocessing]) (3.2.0)\n",
            "Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (from sentence-transformers>=2.2.0->farm-haystack[colab,elasticsearch,inference,preprocessing]) (0.15.2+cu118)\n",
            "Collecting elastic-transport<8 (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Downloading elastic_transport-7.16.0-py2.py3-none-any.whl (35 kB)\n",
            "Collecting elasticsearch<8,>=7.17 (from farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Downloading elasticsearch-7.17.9-py2.py3-none-any.whl (385 kB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 386.0/386.0 kB 34.9 MB/s eta 0:00:00\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema->farm-haystack[colab,elasticsearch,inference,preprocessing]) (2023.7.1)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema->farm-haystack[colab,elasticsearch,inference,preprocessing]) (0.30.2)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema->farm-haystack[colab,elasticsearch,inference,preprocessing]) (0.9.2)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.10/dist-packages (from langdetect->farm-haystack[colab,elasticsearch,inference,preprocessing]) (1.16.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk->farm-haystack[colab,elasticsearch,inference,preprocessing]) (8.1.7)\n",
            "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->farm-haystack[colab,elasticsearch,inference,preprocessing]) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->farm-haystack[colab,elasticsearch,inference,preprocessing]) (2023.3)\n",
            "Collecting monotonic>=1.5 (from posthog->farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Downloading monotonic-1.6-py2.py3-none-any.whl (8.2 kB)\n",
            "Collecting backoff>=1.10.0 (from posthog->farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Downloading backoff-2.2.1-py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: inflect in /usr/local/lib/python3.10/dist-packages (from quantulum3->farm-haystack[colab,elasticsearch,inference,preprocessing]) (7.0.0)\n",
            "Collecting num2words (from quantulum3->farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Downloading num2words-0.5.12-py3-none-any.whl (125 kB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.2/125.2 kB 16.3 MB/s eta 0:00:00\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.20.3->transformers==4.32.1->farm-haystack[colab,elasticsearch,inference,preprocessing]) (5.9.5)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from cattrs>=22.2->requests-cache<1.0.0->farm-haystack[colab,elasticsearch,inference,preprocessing]) (1.1.3)\n",
            "Collecting urllib3<3,>=1.21.1 (from requests->farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Obtaining dependency information for urllib3<3,>=1.21.1 from https://files.pythonhosted.org/packages/c5/05/c214b32d21c0b465506f95c4f28ccbcba15022e000b043b72b3df7728471/urllib3-1.26.16-py2.py3-none-any.whl.metadata\n",
            "  Downloading urllib3-1.26.16-py2.py3-none-any.whl.metadata (48 kB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.4/48.4 kB 5.8 MB/s eta 0:00:00\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch!=1.12.0,>=1.9->transformers==4.32.1->farm-haystack[colab,elasticsearch,inference,preprocessing]) (1.12)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch!=1.12.0,>=1.9->transformers==4.32.1->farm-haystack[colab,elasticsearch,inference,preprocessing]) (3.1.2)\n",
            "Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch!=1.12.0,>=1.9->transformers==4.32.1->farm-haystack[colab,elasticsearch,inference,preprocessing]) (2.0.0)\n",
            "Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch!=1.12.0,>=1.9->transformers==4.32.1->farm-haystack[colab,elasticsearch,inference,preprocessing]) (3.27.2)\n",
            "Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch!=1.12.0,>=1.9->transformers==4.32.1->farm-haystack[colab,elasticsearch,inference,preprocessing]) (16.0.6)\n",
            "Collecting docopt>=0.6.2 (from num2words->quantulum3->farm-haystack[colab,elasticsearch,inference,preprocessing])\n",
            "  Downloading docopt-0.6.2.tar.gz (25 kB)\n",
            "  Preparing metadata (setup.py): started\n",
            "  Preparing metadata (setup.py): finished with status 'done'\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch!=1.12.0,>=1.9->transformers==4.32.1->farm-haystack[colab,elasticsearch,inference,preprocessing]) (2.1.3)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch!=1.12.0,>=1.9->transformers==4.32.1->farm-haystack[colab,elasticsearch,inference,preprocessing]) (1.3.0)\n",
            "Downloading canals-0.7.0-py3-none-any.whl (32 kB)\n",
            "Downloading prompthub_py-4.0.0-py3-none-any.whl (6.9 kB)\n",
            "Downloading transformers-4.32.1-py3-none-any.whl (7.5 MB)\n",
            "   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.5/7.5 MB 115.6 MB/s eta 0:00:00\n",
            "Downloading huggingface_hub-0.16.4-py3-none-any.whl (268 kB)\n",
            "   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 268.8/268.8 kB 31.2 MB/s eta 0:00:00\n",
            "Downloading pydantic-1.10.12-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)\n",
            "   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 105.5 MB/s eta 0:00:00\n",
            "Downloading scikit_learn-1.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.8 MB)\n",
            "   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.8/10.8 MB 115.4 MB/s eta 0:00:00\n",
            "Downloading Events-0.5-py3-none-any.whl (6.8 kB)\n",
            "Downloading farm_haystack-1.20.0-py3-none-any.whl (789 kB)\n",
            "   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 789.9/789.9 kB 69.2 MB/s eta 0:00:00\n",
            "Downloading posthog-3.0.2-py2.py3-none-any.whl (37 kB)\n",
            "Downloading sseclient_py-1.8.0-py2.py3-none-any.whl (8.8 kB)\n",
            "Downloading accelerate-0.22.0-py3-none-any.whl (251 kB)\n",
            "   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 251.2/251.2 kB 29.1 MB/s eta 0:00:00\n",
            "Downloading cattrs-23.1.2-py3-none-any.whl (50 kB)\n",
            "   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.8/50.8 kB 6.3 MB/s eta 0:00:00\n",
            "Downloading safetensors-0.3.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 88.6 MB/s eta 0:00:00\n",
            "Downloading urllib3-1.26.16-py2.py3-none-any.whl (143 kB)\n",
            "   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 143.1/143.1 kB 19.5 MB/s eta 0:00:00\n",
            "Building wheels for collected packages: sentence-transformers, langdetect, docopt\n",
            "  Building wheel for sentence-transformers (setup.py): started\n",
            "  Building wheel for sentence-transformers (setup.py): finished with status 'done'\n",
            "  Created wheel for sentence-transformers: filename=sentence_transformers-2.2.2-py3-none-any.whl size=125923 sha256=93d4d25d65f5902a6ab8cd2c75c6650fafcbbad4f8d1dc6cadf0ada4e651c448\n",
            "  Stored in directory: /root/.cache/pip/wheels/62/f2/10/1e606fd5f02395388f74e7462910fe851042f97238cbbd902f\n",
            "  Building wheel for langdetect (setup.py): started\n",
            "  Building wheel for langdetect (setup.py): finished with status 'done'\n",
            "  Created wheel for langdetect: filename=langdetect-1.0.9-py3-none-any.whl size=993224 sha256=b9b18afa5bffde8f1c81feac2fb748d45bef54fca0a15e03229c38f924707431\n",
            "  Stored in directory: /root/.cache/pip/wheels/95/03/7d/59ea870c70ce4e5a370638b5462a7711ab78fba2f655d05106\n",
            "  Building wheel for docopt (setup.py): started\n",
            "  Building wheel for docopt (setup.py): finished with status 'done'\n",
            "  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13705 sha256=37372a392fb0ddd731e9c84a5cf891fd85b7b9f85b786c14f8e306c593f7591f\n",
            "  Stored in directory: /root/.cache/pip/wheels/fc/ab/d4/5da2067ac95b36618c629a5f93f809425700506f72c9732fac\n",
            "Successfully built sentence-transformers langdetect docopt\n",
            "Installing collected packages: tokenizers, sseclient-py, sentencepiece, safetensors, monotonic, events, docopt, urllib3, url-normalize, rank-bm25, pydantic, pillow, num2words, lazy-imports, langdetect, cattrs, canals, boilerpy3, backoff, scikit-learn, elasticsearch, elastic-transport, tiktoken, requests-cache, quantulum3, prompthub-py, posthog, huggingface-hub, transformers, farm-haystack, accelerate, sentence-transformers\n",
            "  Attempting uninstall: urllib3\n",
            "    Found existing installation: urllib3 2.0.4\n",
            "    Uninstalling urllib3-2.0.4:\n",
            "      Successfully uninstalled urllib3-2.0.4\n",
            "  Attempting uninstall: pydantic\n",
            "    Found existing installation: pydantic 2.2.1\n",
            "    Uninstalling pydantic-2.2.1:\n",
            "      Successfully uninstalled pydantic-2.2.1\n",
            "  Attempting uninstall: pillow\n",
            "    Found existing installation: Pillow 9.4.0\n",
            "    Uninstalling Pillow-9.4.0:\n",
            "      Successfully uninstalled Pillow-9.4.0\n",
            "  Attempting uninstall: scikit-learn\n",
            "    Found existing installation: scikit-learn 1.2.2\n",
            "    Uninstalling scikit-learn-1.2.2:\n",
            "      Successfully uninstalled scikit-learn-1.2.2\n",
            "Successfully installed accelerate-0.22.0 backoff-2.2.1 boilerpy3-1.0.6 canals-0.7.0 cattrs-23.1.2 docopt-0.6.2 elastic-transport-7.16.0 elasticsearch-7.17.9 events-0.5 farm-haystack-1.20.0 huggingface-hub-0.16.4 langdetect-1.0.9 lazy-imports-0.3.1 monotonic-1.6 num2words-0.5.12 pillow-9.0.0 posthog-3.0.2 prompthub-py-4.0.0 pydantic-1.10.12 quantulum3-0.9.0 rank-bm25-0.2.2 requests-cache-0.9.8 safetensors-0.3.3 scikit-learn-1.3.0 sentence-transformers-2.2.2 sentencepiece-0.1.99 sseclient-py-1.8.0 tiktoken-0.4.0 tokenizers-0.13.3 transformers-4.32.1 url-normalize-1.4.3 urllib3-1.26.16\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "\n",
        "wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q\n",
        "tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz\n",
        "chown -R daemon:daemon elasticsearch-7.9.2"
      ],
      "metadata": {
        "id": "ynMLRFilVd_V"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash --bg\n",
        "\n",
        "sudo -u daemon -- elasticsearch-7.9.2/bin/elasticsearch"
      ],
      "metadata": {
        "id": "LFKDIulfV0ew"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import time\n",
        "\n",
        "time.sleep(30)"
      ],
      "metadata": {
        "id": "zFmMLOWxV3VT"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "from haystack.document_stores import ElasticsearchDocumentStore\n",
        "\n",
        "# Get the host where Elasticsearch is running, default to localhost\n",
        "host = os.environ.get(\"ELASTICSEARCH_HOST\", \"localhost\")\n",
        "\n",
        "document_store = ElasticsearchDocumentStore(host=host, username=\"\", password=\"\", index=\"document\")"
      ],
      "metadata": {
        "id": "1JFhUUygV7Iq"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "res = requests.get('http://localhost:9200/_cluster/health')\n",
        "\n",
        "res.json()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rIG5YiIjWJdr",
        "outputId": "f32f39a8-9ac5-412c-bb67-4b84d9e8477f"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'cluster_name': 'elasticsearch',\n",
              " 'status': 'yellow',\n",
              " 'timed_out': False,\n",
              " 'number_of_nodes': 1,\n",
              " 'number_of_data_nodes': 1,\n",
              " 'active_primary_shards': 2,\n",
              " 'active_shards': 2,\n",
              " 'relocating_shards': 0,\n",
              " 'initializing_shards': 0,\n",
              " 'unassigned_shards': 2,\n",
              " 'delayed_unassigned_shards': 0,\n",
              " 'number_of_pending_tasks': 0,\n",
              " 'number_of_in_flight_fetch': 0,\n",
              " 'task_max_waiting_in_queue_millis': 0,\n",
              " 'active_shards_percent_as_number': 50.0}"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "\n",
        "with open('dev.json', 'r') as f:\n",
        "    squad = json.load(f)"
      ],
      "metadata": {
        "id": "36Cl25FJWQKW"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "squad"
      ],
      "metadata": {
        "id": "0F2F5exJal9L"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "squad_docs = []\n",
        "\n",
        "for sample in squad:\n",
        "    squad_docs.append({\n",
        "        'content': sample['context']\n",
        "    })"
      ],
      "metadata": {
        "id": "nz8D6hPuaval"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "document_store.write_documents(squad_docs)"
      ],
      "metadata": {
        "id": "CS1TrpyjYRLj"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from haystack.nodes import TfidfRetriever\n",
        "\n",
        "retriever = TfidfRetriever(document_store)"
      ],
      "metadata": {
        "id": "pwMPKWzAYvF8"
      },
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "len(squad)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "H1JzDQiIbooX",
        "outputId": "7bba0e0c-26e1-443d-a50e-2151cf164694"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "16209"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "query = \"Physics is a very abstract subject\"\n",
        "\n",
        "retriever.retrieve(query)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yPpIBYH5cF4g",
        "outputId": "fd1cc71e-cc02-4126-bf5d-c11800a752ad"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<Document: {'content': 'Even though some proofs of complexity-theoretic theorems regularly assume some concrete choice of input encoding, one tries to keep the discussion abstract enough to be independent of the choice of encoding. This can be achieved by ensuring that different representations can be transformed into each other efficiently.', 'content_type': 'text', 'score': None, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': 'fdafa6182e072da1f6005c49f858c175'}>,\n",
              " <Document: {'content': 'With modern insights into quantum mechanics and technology that can accelerate particles close to the speed of light, particle physics has devised a Standard Model to describe forces between particles smaller than atoms. The Standard Model predicts that exchanged particles called gauge bosons are the fundamental means by which forces are emitted and absorbed. Only four main interactions are known: in order of decreasing strength, they are: strong, electromagnetic, weak, and gravitational.:2–10:79 High-energy particle physics observations made during the 1970s and 1980s confirmed that the weak and electromagnetic forces are expressions of a more fundamental electroweak interaction.', 'content_type': 'text', 'score': None, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': '828c018bed6be4ea72243b7af5130cb8'}>,\n",
              " <Document: {'content': 'Subject Committees are established at the beginning of each parliamentary session, and again the members on each committee reflect the balance of parties across Parliament. Typically each committee corresponds with one (or more) of the departments (or ministries) of the Scottish Government. The current Subject Committees in the fourth Session are: Economy, Energy and Tourism; Education and Culture; Health and Sport; Justice; Local Government and Regeneration; Rural Affairs, Climate Change and Environment; Welfare Reform; and Infrastructure and Capital Investment.', 'content_type': 'text', 'score': None, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': 'c11276654d094f408ab71779b84a2a21'}>,\n",
              " <Document: {'content': 'A computational problem can be viewed as an infinite collection of instances together with a solution for every instance. The input string for a computational problem is referred to as a problem instance, and should not be confused with the problem itself. In computational complexity theory, a problem refers to the abstract question to be solved. In contrast, an instance of this problem is a rather concrete utterance, which can serve as the input for a decision problem. For example, consider the problem of primality testing. The instance is a number (e.g. 15) and the solution is \"yes\" if the number is prime and \"no\" otherwise (in this case \"no\"). Stated another way, the instance is a particular input to the problem, and the solution is the output corresponding to the given input.', 'content_type': 'text', 'score': None, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': '6475dc896c098df79cbbe73c2f4b183d'}>,\n",
              " <Document: {'content': 'However, some computational problems are easier to analyze in terms of more unusual resources. For example, a non-deterministic Turing machine is a computational model that is allowed to branch out to check many different possibilities at once. The non-deterministic Turing machine has very little to do with how we physically want to compute algorithms, but its branching exactly captures many of the mathematical models we want to analyze, so that non-deterministic time is a very important resource in analyzing computational problems.', 'content_type': 'text', 'score': None, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': '41356176d5b1625db5fea90286c3cf'}>,\n",
              " <Document: {'content': 'The notion \"force\" keeps its meaning in quantum mechanics, though one is now dealing with operators instead of classical variables and though the physics is now described by the Schrödinger equation instead of Newtonian equations. This has the consequence that the results of a measurement are now sometimes \"quantized\", i.e. they appear in discrete portions. This is, of course, difficult to imagine in the context of \"forces\". However, the potentials V(x,y,z) or fields, from which the forces generally can be derived, are treated similar to classical position variables, i.e., .', 'content_type': 'text', 'score': None, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': 'cac0630683bf55c6f3a0968e92e79710'}>,\n",
              " <Document: {'content': 'The development of fundamental theories for forces proceeded along the lines of unification of disparate ideas. For example, Isaac Newton unified the force responsible for objects falling at the surface of the Earth with the force responsible for the orbits of celestial mechanics in his universal theory of gravitation. Michael Faraday and James Clerk Maxwell demonstrated that electric and magnetic forces were unified through one consistent theory of electromagnetism. In the 20th century, the development of quantum mechanics led to a modern understanding that the first three fundamental forces (all except gravity) are manifestations of matter (fermions) interacting by exchanging virtual particles called gauge bosons. This standard model of particle physics posits a similarity between the forces and led scientists to predict the unification of the weak and electromagnetic forces in electroweak theory subsequently confirmed by observation. The complete formulation of the standard model predicts an as yet unobserved Higgs mechanism, but observations such as neutrino oscillations indicate that the standard model is incomplete. A Grand Unified Theory allowing for the combination of the electroweak interaction with the strong force is held out as a possibility with candidate theories such as supersymmetry proposed to accommodate some of the outstanding unsolved problems in physics. Physicists are still attempting to develop self-consistent unification models that would combine all four fundamental interactions into a theory of everything. Einstein tried and failed at this endeavor, but currently the most popular approach to answering this question is string theory.:212–219', 'content_type': 'text', 'score': None, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': 'd75f0925dca88d80bc176c8e6ac99de'}>,\n",
              " <Document: {'content': \"Trioxygen (O\\n3) is usually known as ozone and is a very reactive allotrope of oxygen that is damaging to lung tissue. Ozone is produced in the upper atmosphere when O\\n2 combines with atomic oxygen made by the splitting of O\\n2 by ultraviolet (UV) radiation. Since ozone absorbs strongly in the UV region of the spectrum, the ozone layer of the upper atmosphere functions as a protective radiation shield for the planet. Near the Earth's surface, it is a pollutant formed as a by-product of automobile exhaust. The metastable molecule tetraoxygen (O\\n4) was discovered in 2001, and was assumed to exist in one of the six phases of solid oxygen. It was proven in 2006 that this phase, created by pressurizing O\\n2 to 20 GPa, is in fact a rhombohedral O\\n8 cluster. This cluster has the potential to be a much more powerful oxidizer than either O\\n2 or O\\n3 and may therefore be used in rocket fuel. A metallic phase was discovered in 1990 when solid oxygen is subjected to a pressure of above 96 GPa and it was shown in 1998 that at very low temperatures, this phase becomes superconducting.\", 'content_type': 'text', 'score': None, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': '5d1e17240677586a41b878993c195cd8'}>,\n",
              " <Document: {'content': \"Newton's Second Law asserts the direct proportionality of acceleration to force and the inverse proportionality of acceleration to mass. Accelerations can be defined through kinematic measurements. However, while kinematics are well-described through reference frame analysis in advanced physics, there are still deep questions that remain as to what is the proper definition of mass. General relativity offers an equivalence between space-time and mass, but lacking a coherent theory of quantum gravity, it is unclear as to how or whether this connection is relevant on microscales. With some justification, Newton's second law can be taken as a quantitative definition of mass by writing the law as an equality; the relative units of force and mass then are fixed.\", 'content_type': 'text', 'score': None, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': '979ba4d49a5db02ba94de24fbb62a4d6'}>,\n",
              " <Document: {'content': 'A deterministic Turing machine is the most basic Turing machine, which uses a fixed set of rules to determine its future actions. A probabilistic Turing machine is a deterministic Turing machine with an extra supply of random bits. The ability to make probabilistic decisions often helps algorithms solve problems more efficiently. Algorithms that use random bits are called randomized algorithms. A non-deterministic Turing machine is a deterministic Turing machine with an added feature of non-determinism, which allows a Turing machine to have multiple possible future actions from a given state. One way to view non-determinism is that the Turing machine branches into many possible computational paths at each step, and if it solves the problem in any of these branches, it is said to have solved the problem. Clearly, this model is not meant to be a physically realizable model, it is just a theoretically interesting abstract machine that gives rise to particularly interesting complexity classes. For examples, see non-deterministic algorithm.', 'content_type': 'text', 'score': None, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': '7548b7fce7050990a134ce44445026ce'}>]"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "res = requests.post('http://localhost:9200/document/_delete_by_query',\n",
        "                    json={\n",
        "                        'query': {\n",
        "                            'match_all': {}\n",
        "                        }\n",
        "                    })\n",
        "\n",
        "res.json()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hEXanth_coRh",
        "outputId": "3aa8b87d-cc65-43cf-a60a-7ed7938d5387"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'took': 301,\n",
              " 'timed_out': False,\n",
              " 'total': 1204,\n",
              " 'deleted': 1204,\n",
              " 'batches': 2,\n",
              " 'version_conflicts': 0,\n",
              " 'noops': 0,\n",
              " 'retries': {'bulk': 0, 'search': 0},\n",
              " 'throttled_millis': 0,\n",
              " 'requests_per_second': -1.0,\n",
              " 'throttled_until_millis': 0,\n",
              " 'failures': []}"
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "res = requests.get('http://localhost:9200/document/_count')\n",
        "\n",
        "res.json()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PTVbUH3TcpTI",
        "outputId": "8e8c6881-47f9-4cfe-edff-ca2b46d53f65"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'count': 0,\n",
              " '_shards': {'total': 1, 'successful': 1, 'skipped': 0, 'failed': 0}}"
            ]
          },
          "metadata": {},
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# create list of contexts (we cannot do this using current dictionary format)\n",
        "contexts = [sample['context'] for sample in squad]\n",
        "\n",
        "# convert to set to remove duplicates, then back to list\n",
        "contexts = list(set(contexts))\n",
        "\n",
        "# convert back to dictionary format we need\n",
        "squad_docs = [{'content': sample} for sample in contexts]"
      ],
      "metadata": {
        "id": "WyxZLPbGczzS"
      },
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "document_store.write_documents(squad_docs)"
      ],
      "metadata": {
        "id": "gec61WC7c5pD"
      },
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "retriever = TfidfRetriever(document_store)"
      ],
      "metadata": {
        "id": "miE7QhL8c8Qi"
      },
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "retriever.retrieve(query)"
      ],
      "metadata": {
        "id": "SVAxivUac_KT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# import BM25 retriever\n",
        "from haystack.nodes import BM25Retriever\n",
        "\n",
        "# intialize\n",
        "retriever = BM25Retriever(document_store)\n",
        "\n",
        "# and query\n",
        "retriever.retrieve(query)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "r0ngZG4VdB9E",
        "outputId": "1618e673-d42c-4854-933c-72be5c72d762"
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<Document: {'content': 'Even though some proofs of complexity-theoretic theorems regularly assume some concrete choice of input encoding, one tries to keep the discussion abstract enough to be independent of the choice of encoding. This can be achieved by ensuring that different representations can be transformed into each other efficiently.', 'content_type': 'text', 'score': 0.7329360380503047, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': 'fdafa6182e072da1f6005c49f858c175'}>,\n",
              " <Document: {'content': 'A computational problem can be viewed as an infinite collection of instances together with a solution for every instance. The input string for a computational problem is referred to as a problem instance, and should not be confused with the problem itself. In computational complexity theory, a problem refers to the abstract question to be solved. In contrast, an instance of this problem is a rather concrete utterance, which can serve as the input for a decision problem. For example, consider the problem of primality testing. The instance is a number (e.g. 15) and the solution is \"yes\" if the number is prime and \"no\" otherwise (in this case \"no\"). Stated another way, the instance is a particular input to the problem, and the solution is the output corresponding to the given input.', 'content_type': 'text', 'score': 0.7168560824162143, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': '6475dc896c098df79cbbe73c2f4b183d'}>,\n",
              " <Document: {'content': 'A deterministic Turing machine is the most basic Turing machine, which uses a fixed set of rules to determine its future actions. A probabilistic Turing machine is a deterministic Turing machine with an extra supply of random bits. The ability to make probabilistic decisions often helps algorithms solve problems more efficiently. Algorithms that use random bits are called randomized algorithms. A non-deterministic Turing machine is a deterministic Turing machine with an added feature of non-determinism, which allows a Turing machine to have multiple possible future actions from a given state. One way to view non-determinism is that the Turing machine branches into many possible computational paths at each step, and if it solves the problem in any of these branches, it is said to have solved the problem. Clearly, this model is not meant to be a physically realizable model, it is just a theoretically interesting abstract machine that gives rise to particularly interesting complexity classes. For examples, see non-deterministic algorithm.', 'content_type': 'text', 'score': 0.705369954591001, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': '7548b7fce7050990a134ce44445026ce'}>,\n",
              " <Document: {'content': 'With modern insights into quantum mechanics and technology that can accelerate particles close to the speed of light, particle physics has devised a Standard Model to describe forces between particles smaller than atoms. The Standard Model predicts that exchanged particles called gauge bosons are the fundamental means by which forces are emitted and absorbed. Only four main interactions are known: in order of decreasing strength, they are: strong, electromagnetic, weak, and gravitational.:2–10:79 High-energy particle physics observations made during the 1970s and 1980s confirmed that the weak and electromagnetic forces are expressions of a more fundamental electroweak interaction.', 'content_type': 'text', 'score': 0.7052212742359505, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': '828c018bed6be4ea72243b7af5130cb8'}>,\n",
              " <Document: {'content': 'Dynamic equilibrium was first described by Galileo who noticed that certain assumptions of Aristotelian physics were contradicted by observations and logic. Galileo realized that simple velocity addition demands that the concept of an \"absolute rest frame\" did not exist. Galileo concluded that motion in a constant velocity was completely equivalent to rest. This was contrary to Aristotle\\'s notion of a \"natural state\" of rest that objects with mass naturally approached. Simple experiments showed that Galileo\\'s understanding of the equivalence of constant velocity and rest were correct. For example, if a mariner dropped a cannonball from the crow\\'s nest of a ship moving at a constant velocity, Aristotelian physics would have the cannonball fall straight down while the ship moved beneath it. Thus, in an Aristotelian universe, the falling cannonball would land behind the foot of the mast of a moving ship. However, when this experiment is actually conducted, the cannonball always falls at the foot of the mast, as if the cannonball knows to travel with the ship despite being separated from it. Since there is no forward horizontal force being applied on the cannonball as it falls, the only conclusion left is that the cannonball continues to move with the same velocity as the boat as it falls. Thus, no force is required to keep the cannonball moving at the constant forward velocity.', 'content_type': 'text', 'score': 0.68968973013168, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': 'e9b4a45fed9e32bcb1eb5aa7cb1cd837'}>,\n",
              " <Document: {'content': 'The notion \"force\" keeps its meaning in quantum mechanics, though one is now dealing with operators instead of classical variables and though the physics is now described by the Schrödinger equation instead of Newtonian equations. This has the consequence that the results of a measurement are now sometimes \"quantized\", i.e. they appear in discrete portions. This is, of course, difficult to imagine in the context of \"forces\". However, the potentials V(x,y,z) or fields, from which the forces generally can be derived, are treated similar to classical position variables, i.e., .', 'content_type': 'text', 'score': 0.6881510924281012, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': 'cac0630683bf55c6f3a0968e92e79710'}>,\n",
              " <Document: {'content': 'The development of fundamental theories for forces proceeded along the lines of unification of disparate ideas. For example, Isaac Newton unified the force responsible for objects falling at the surface of the Earth with the force responsible for the orbits of celestial mechanics in his universal theory of gravitation. Michael Faraday and James Clerk Maxwell demonstrated that electric and magnetic forces were unified through one consistent theory of electromagnetism. In the 20th century, the development of quantum mechanics led to a modern understanding that the first three fundamental forces (all except gravity) are manifestations of matter (fermions) interacting by exchanging virtual particles called gauge bosons. This standard model of particle physics posits a similarity between the forces and led scientists to predict the unification of the weak and electromagnetic forces in electroweak theory subsequently confirmed by observation. The complete formulation of the standard model predicts an as yet unobserved Higgs mechanism, but observations such as neutrino oscillations indicate that the standard model is incomplete. A Grand Unified Theory allowing for the combination of the electroweak interaction with the strong force is held out as a possibility with candidate theories such as supersymmetry proposed to accommodate some of the outstanding unsolved problems in physics. Physicists are still attempting to develop self-consistent unification models that would combine all four fundamental interactions into a theory of everything. Einstein tried and failed at this endeavor, but currently the most popular approach to answering this question is string theory.:212–219', 'content_type': 'text', 'score': 0.6823407772868847, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': 'd75f0925dca88d80bc176c8e6ac99de'}>,\n",
              " <Document: {'content': 'Subject Committees are established at the beginning of each parliamentary session, and again the members on each committee reflect the balance of parties across Parliament. Typically each committee corresponds with one (or more) of the departments (or ministries) of the Scottish Government. The current Subject Committees in the fourth Session are: Economy, Energy and Tourism; Education and Culture; Health and Sport; Justice; Local Government and Regeneration; Rural Affairs, Climate Change and Environment; Welfare Reform; and Infrastructure and Capital Investment.', 'content_type': 'text', 'score': 0.6817404079690633, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': 'c11276654d094f408ab71779b84a2a21'}>,\n",
              " <Document: {'content': 'However, some computational problems are easier to analyze in terms of more unusual resources. For example, a non-deterministic Turing machine is a computational model that is allowed to branch out to check many different possibilities at once. The non-deterministic Turing machine has very little to do with how we physically want to compute algorithms, but its branching exactly captures many of the mathematical models we want to analyze, so that non-deterministic time is a very important resource in analyzing computational problems.', 'content_type': 'text', 'score': 0.6775114617437179, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': '41356176d5b1625db5fea90286c3cf'}>,\n",
              " <Document: {'content': \"Newton's Second Law asserts the direct proportionality of acceleration to force and the inverse proportionality of acceleration to mass. Accelerations can be defined through kinematic measurements. However, while kinematics are well-described through reference frame analysis in advanced physics, there are still deep questions that remain as to what is the proper definition of mass. General relativity offers an equivalence between space-time and mass, but lacking a coherent theory of quantum gravity, it is unclear as to how or whether this connection is relevant on microscales. With some justification, Newton's second law can be taken as a quantitative definition of mass by writing the law as an equality; the relative units of force and mass then are fixed.\", 'content_type': 'text', 'score': 0.6773744944452036, 'meta': {}, 'id_hash_keys': ['content'], 'embedding': None, 'id': '979ba4d49a5db02ba94de24fbb62a4d6'}>]"
            ]
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "FBkSfaOxdGJX"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}