Application de la RI pour la liaison référentielle.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/SelmaDM/Pyterrier/blob/master/Application%20de%20la%20RI%20pour%20la%20liaison%20r%C3%A9f%C3%A9rentielle.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jzpZPYGi5E0O"
      },
      "source": [
        "# Université Paul Sabatier\n",
        "\n",
        "M1 IAFA - Recherche d'information\n",
        "\n",
        "**TP 2**\n",
        "\n",
        "Enseignant : José G. Moreno\n",
        "\n",
        "2023"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QVVgNtmF5b31"
      },
      "source": [
        "## TP 2. Application de la RI pour la liaison référentielle "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mDoEiTn25rWo"
      },
      "source": [
        "Une application des méthodes de RI est la liaison référentielle. Cette tâche consiste à trouver le lien entre une portion de texte et la page Wikipédia qui le représentent le mieux. Dans ce cas d’application :\n",
        "* Comme requêtes nous allons utiliser chaque mot dans une phrase (ou chaque mot important).\n",
        "* Le corpus sont les pages Wikipédia. Donc, chaque page Wikipédia est un document à retrouver.\n",
        "* La liste de résultats sont les candidats pour la liaison référentielle.\n",
        "\n",
        "Considère par exemple la chaîne de caractères « ***Thomas and Mario are strikers playing in Munich*** ». Il existe trois sous-chaînes (***Thomas***, ***Mario*** et ***Munich***) ou mots importants qui sont susceptibles d'être identifiées dans une base de connaissances telle que Wikipédia. Une recherche de ces mots-clés dans les pages Wikipédia produirait de nombreux concepts candidats pour ces sous-chaînes. Donc un bon système doit retrouver comme premier résultats les liens aux pages Wikipédia : [Thomas Müller](https://en.wikipedia.org/wiki/Thomas_M%C3%BCller), [Mario Gómez](https://en.wikipedia.org/wiki/Mario_G%C3%B3mez) (ou Mario Götze) et FC [Bayern Munich](https://en.wikipedia.org/wiki/FC_Bayern_Munich). Cette tâche est particulièrement complexe et les méthodes de recherche d’information sont particulièrement utiles pour la résoudre."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "odNyknkI8_Bu"
      },
      "source": [
        "> Plusieurs méthodes de liaison référentielle (Entity Linking en anglais) sont disponibles sur internet. Les plus connus sont : \n",
        "\n",
        ">\t* Spotlight → https://demo.dbpedia-spotlight.org/\n",
        ">\t* Babelfy → http://babelfy.org/\n",
        ">\t* TagMe → https://tagme.d4science.org/tagme/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bynRMtE99ENL"
      },
      "source": [
        "### 1) Liaison avec le texte\n",
        "En utilisant l’indexe du champ **text** de Wikipédia (comme dans le TP1) et la requête « Thomas and Mario are strikers playing in Munich » vérifiez si vous obtenez les pages Wikipédia respectives (comme le ferai Spotlight, Babelfy ou TagMe) :\n",
        "* Thomas Müller\n",
        "* Mario Gómez ou Mario Götze\n",
        "* FC Bayern Munich\n",
        "\n",
        "Voici un example:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aCh7sGOuj--_",
        "outputId": "7f0794d4-a2be-43f5-b459-fb6e904cb148"
      },
      "source": [
        "!wget \"https://drive.google.com/uc?id=16rd8rFNR5qtjaX_vtZ__nqopq7Q6pSFu&confirm=t&uuid=0cc61c9f-387b-4218-b1e1-6421fb83d11e&at=ALgDtsx9J1qAF9DesqbqTEOfuLsR:1675797919874\" -O pd_index.zip\n",
        "!unzip pd_index.zip"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-03-16 12:52:13--  https://drive.google.com/uc?id=16rd8rFNR5qtjaX_vtZ__nqopq7Q6pSFu&confirm=t&uuid=0cc61c9f-387b-4218-b1e1-6421fb83d11e&at=ALgDtsx9J1qAF9DesqbqTEOfuLsR:1675797919874\n",
            "Resolving drive.google.com (drive.google.com)... 173.194.210.138, 173.194.210.113, 173.194.210.102, ...\n",
            "Connecting to drive.google.com (drive.google.com)|173.194.210.138|:443... connected.\n",
            "HTTP request sent, awaiting response... 303 See Other\n",
            "Location: https://doc-14-9s-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/fijisa9onoi2hehmlfl1ln1tdqsibvf9/1678971075000/12754483704616509995/*/16rd8rFNR5qtjaX_vtZ__nqopq7Q6pSFu?uuid=0cc61c9f-387b-4218-b1e1-6421fb83d11e [following]\n",
            "Warning: wildcards not supported in HTTP.\n",
            "--2023-03-16 12:52:13--  https://doc-14-9s-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/fijisa9onoi2hehmlfl1ln1tdqsibvf9/1678971075000/12754483704616509995/*/16rd8rFNR5qtjaX_vtZ__nqopq7Q6pSFu?uuid=0cc61c9f-387b-4218-b1e1-6421fb83d11e\n",
            "Resolving doc-14-9s-docs.googleusercontent.com (doc-14-9s-docs.googleusercontent.com)... 172.253.123.132, 2607:f8b0:400c:c16::84\n",
            "Connecting to doc-14-9s-docs.googleusercontent.com (doc-14-9s-docs.googleusercontent.com)|172.253.123.132|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 56607479 (54M) [application/zip]\n",
            "Saving to: ‘pd_index.zip’\n",
            "\n",
            "pd_index.zip        100%[===================>]  53.98M   152MB/s    in 0.4s    \n",
            "\n",
            "2023-03-16 12:52:14 (152 MB/s) - ‘pd_index.zip’ saved [56607479/56607479]\n",
            "\n",
            "Archive:  pd_index.zip\n",
            "   creating: pd_index/\n",
            "  inflating: pd_index/data.meta.idx  \n",
            "  inflating: pd_index/data.inverted.bf  \n",
            "  inflating: pd_index/data.lexicon.fsomaphash  \n",
            "  inflating: pd_index/data.meta-0.fsomapfile  \n",
            "  inflating: pd_index/data.lexicon.fsomapfile  \n",
            "  inflating: pd_index/data.direct.bf  \n",
            "  inflating: pd_index/data.lexicon.fsomapid  \n",
            "  inflating: pd_index/data.properties  \n",
            "  inflating: pd_index/data.meta.zdata  \n",
            "  inflating: pd_index/data.document.fsarrayfile  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w-SfcjQe5FeS"
      },
      "source": [
        "# déclaration de la variable JAVA_HOME\n",
        "import os\n",
        "os.environ['JAVA_HOME'] = '/usr/lib/jvm/java-11-openjdk-amd64'\n",
        "!export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8CWv901sil2K",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "68eb3e4e-47c5-40ed-8a6e-157aac761f81"
      },
      "source": [
        "#installation de pyterrier avec pip\n",
        "!pip install --upgrade git+https://github.com/terrier-org/pyterrier.git#egg=python-terrier"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting python-terrier\n",
            "  Cloning https://github.com/terrier-org/pyterrier.git to /tmp/pip-install-8v7qj0su/python-terrier_a9250885dc1042e59f641f15c9ddb51b\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/terrier-org/pyterrier.git /tmp/pip-install-8v7qj0su/python-terrier_a9250885dc1042e59f641f15c9ddb51b\n",
            "  Resolved https://github.com/terrier-org/pyterrier.git to commit dc7997ed4bb4bbaf78f639a511bfe92fcd290c02\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.9/dist-packages (from python-terrier) (1.22.4)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.9/dist-packages (from python-terrier) (1.4.4)\n",
            "Collecting wget\n",
            "  Downloading wget-3.2.zip (10 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.9/dist-packages (from python-terrier) (4.65.0)\n",
            "Collecting pyjnius>=1.4.2\n",
            "  Downloading pyjnius-1.4.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.5/1.5 MB\u001b[0m \u001b[31m10.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting matchpy\n",
            "  Downloading matchpy-0.5.5-py3-none-any.whl (69 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m69.6/69.6 KB\u001b[0m \u001b[31m4.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: scikit-learn in /usr/local/lib/python3.9/dist-packages (from python-terrier) (1.2.2)\n",
            "Collecting deprecated\n",
            "  Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)\n",
            "Collecting chest\n",
            "  Downloading chest-0.2.3.tar.gz (9.6 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.9/dist-packages (from python-terrier) (1.10.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.9/dist-packages (from python-terrier) (2.25.1)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.9/dist-packages (from python-terrier) (1.1.1)\n",
            "Collecting nptyping==1.4.4\n",
            "  Downloading nptyping-1.4.4-py3-none-any.whl (31 kB)\n",
            "Requirement already satisfied: more_itertools in /usr/local/lib/python3.9/dist-packages (from python-terrier) (9.1.0)\n",
            "Collecting ir_datasets>=0.3.2\n",
            "  Downloading ir_datasets-0.5.4-py3-none-any.whl (311 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m311.5/311.5 KB\u001b[0m \u001b[31m17.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: jinja2 in /usr/local/lib/python3.9/dist-packages (from python-terrier) (3.1.2)\n",
            "Requirement already satisfied: statsmodels in /usr/local/lib/python3.9/dist-packages (from python-terrier) (0.13.5)\n",
            "Collecting ir_measures>=0.3.1\n",
            "  Downloading ir_measures-0.3.1.tar.gz (46 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.5/46.5 KB\u001b[0m \u001b[31m2.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting dill\n",
            "  Downloading dill-0.3.6-py3-none-any.whl (110 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m110.5/110.5 KB\u001b[0m \u001b[31m8.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pytrec_eval_terrier>=0.5.3\n",
            "  Downloading pytrec_eval_terrier-0.5.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (287 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m287.2/287.2 KB\u001b[0m \u001b[31m16.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting typish>=1.7.0\n",
            "  Downloading typish-1.9.3-py3-none-any.whl (45 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.1/45.1 KB\u001b[0m \u001b[31m2.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting ijson>=3.1.3\n",
            "  Downloading ijson-3.2.0.post0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (112 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m112.5/112.5 KB\u001b[0m \u001b[31m5.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: lxml>=4.5.2 in /usr/local/lib/python3.9/dist-packages (from ir_datasets>=0.3.2->python-terrier) (4.9.2)\n",
            "Collecting warc3-wet>=0.2.3\n",
            "  Downloading warc3_wet-0.2.3-py3-none-any.whl (13 kB)\n",
            "Collecting trec-car-tools>=2.5.4\n",
            "  Downloading trec_car_tools-2.6-py3-none-any.whl (8.4 kB)\n",
            "Collecting warc3-wet-clueweb09>=0.2.5\n",
            "  Downloading warc3-wet-clueweb09-0.2.5.tar.gz (17 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting zlib-state>=0.1.3\n",
            "  Downloading zlib_state-0.1.5-cp39-cp39-manylinux2010_x86_64.whl (71 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m72.0/72.0 KB\u001b[0m \u001b[31m3.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting unlzw3>=0.2.1\n",
            "  Downloading unlzw3-0.2.2-py3-none-any.whl (6.1 kB)\n",
            "Collecting lz4>=3.1.1\n",
            "  Downloading lz4-4.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m33.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: beautifulsoup4>=4.4.1 in /usr/local/lib/python3.9/dist-packages (from ir_datasets>=0.3.2->python-terrier) (4.9.3)\n",
            "Requirement already satisfied: pyyaml>=5.3.1 in /usr/local/lib/python3.9/dist-packages (from ir_datasets>=0.3.2->python-terrier) (6.0)\n",
            "Collecting pyautocorpus>=0.1.1\n",
            "  Downloading pyautocorpus-0.1.9-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (293 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m293.3/293.3 KB\u001b[0m \u001b[31m13.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting cwl-eval>=1.0.10\n",
            "  Downloading cwl-eval-1.0.12.tar.gz (31 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: six>=1.7.0 in /usr/local/lib/python3.9/dist-packages (from pyjnius>=1.4.2->python-terrier) (1.15.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests->python-terrier) (2022.12.7)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests->python-terrier) (1.26.15)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.9/dist-packages (from requests->python-terrier) (4.0.0)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests->python-terrier) (2.10)\n",
            "Requirement already satisfied: heapdict in /usr/local/lib/python3.9/dist-packages (from chest->python-terrier) (1.0.1)\n",
            "Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.9/dist-packages (from deprecated->python-terrier) (1.15.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.9/dist-packages (from jinja2->python-terrier) (2.1.2)\n",
            "Collecting multiset<3.0,>=2.0\n",
            "  Downloading multiset-2.1.1-py2.py3-none-any.whl (8.8 kB)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.9/dist-packages (from pandas->python-terrier) (2022.7.1)\n",
            "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.9/dist-packages (from pandas->python-terrier) (2.8.2)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.9/dist-packages (from scikit-learn->python-terrier) (3.1.0)\n",
            "Requirement already satisfied: patsy>=0.5.2 in /usr/local/lib/python3.9/dist-packages (from statsmodels->python-terrier) (0.5.3)\n",
            "Requirement already satisfied: packaging>=21.3 in /usr/local/lib/python3.9/dist-packages (from statsmodels->python-terrier) (23.0)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.9/dist-packages (from beautifulsoup4>=4.4.1->ir_datasets>=0.3.2->python-terrier) (2.4)\n",
            "Collecting cbor>=1.0.0\n",
            "  Downloading cbor-1.0.0.tar.gz (20 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Building wheels for collected packages: python-terrier, ir_measures, chest, wget, cwl-eval, warc3-wet-clueweb09, cbor\n",
            "  Building wheel for python-terrier (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for python-terrier: filename=python_terrier-0.9.2-py3-none-any.whl size=112520 sha256=19f9f0eaf4ac6babc5b9aed12ce2d6d55ac8c7c7bc5f6bde60d78754f36513b0\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-54vaep22/wheels/1b/7e/7c/19df81bd2538ca44417933eead7ee6f7c8a3044901e7fc46ee\n",
            "  Building wheel for ir_measures (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for ir_measures: filename=ir_measures-0.3.1-py3-none-any.whl size=60184 sha256=bc730ffedf2733ae022dcbd3899a7f6ccd99ebe59f2a4056a7b02d6e8c130bb8\n",
            "  Stored in directory: /root/.cache/pip/wheels/00/fa/7d/0ba22fce7d11d69df68c31b7c247fbbb28fe2d4c610f8c653f\n",
            "  Building wheel for chest (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for chest: filename=chest-0.2.3-py3-none-any.whl size=7630 sha256=105dc7a9a2e8fa92f0421e1cd72bc1c53014b5c6980a84bfab31879ba51666f8\n",
            "  Stored in directory: /root/.cache/pip/wheels/ce/05/8e/2ecd3728ed5d938ee57161c8acf2ee74f55f894964bb3d31ef\n",
            "  Building wheel for wget (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for wget: filename=wget-3.2-py3-none-any.whl size=9673 sha256=dacb1e75492ceb6adf99e2442d95a66f82ab6a5cd4f4e65fc05a023adb6f4245\n",
            "  Stored in directory: /root/.cache/pip/wheels/04/5f/3e/46cc37c5d698415694d83f607f833f83f0149e49b3af9d0f38\n",
            "  Building wheel for cwl-eval (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for cwl-eval: filename=cwl_eval-1.0.12-py3-none-any.whl size=38084 sha256=21f9f0fa85f475ef5cda14af9a7f6ee5b62e9b6f7ec57e25a32b39b3fde39b87\n",
            "  Stored in directory: /root/.cache/pip/wheels/0e/52/dc/9d448ff093ccae9b248a41b265d31e2c930eea2d0e84aa29e0\n",
            "  Building wheel for warc3-wet-clueweb09 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for warc3-wet-clueweb09: filename=warc3_wet_clueweb09-0.2.5-py3-none-any.whl size=18918 sha256=6ae10d8ef3b69e70609820ef415603643730dd5da3d50035434846a2e45f606b\n",
            "  Stored in directory: /root/.cache/pip/wheels/7f/22/ed/a11944d7fdf4e94c4206a3f760d385122a4d34d8acc12f71a3\n",
            "  Building wheel for cbor (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for cbor: filename=cbor-1.0.0-cp39-cp39-linux_x86_64.whl size=57285 sha256=e65f71aaf8af17e40dcc2456fffb81d52fcc60c3dc98904064da675028743b6a\n",
            "  Stored in directory: /root/.cache/pip/wheels/ec/10/03/a281e0682ddd4b310431fb25d1a4f53987105267cf46c417f3\n",
            "Successfully built python-terrier ir_measures chest wget cwl-eval warc3-wet-clueweb09 cbor\n",
            "Installing collected packages: wget, warc3-wet-clueweb09, warc3-wet, typish, multiset, ijson, cbor, zlib-state, unlzw3, trec-car-tools, pytrec_eval_terrier, pyjnius, pyautocorpus, nptyping, matchpy, lz4, dill, deprecated, cwl-eval, chest, ir_measures, ir_datasets, python-terrier\n",
            "Successfully installed cbor-1.0.0 chest-0.2.3 cwl-eval-1.0.12 deprecated-1.2.13 dill-0.3.6 ijson-3.2.0.post0 ir_datasets-0.5.4 ir_measures-0.3.1 lz4-4.3.2 matchpy-0.5.5 multiset-2.1.1 nptyping-1.4.4 pyautocorpus-0.1.9 pyjnius-1.4.2 python-terrier-0.9.2 pytrec_eval_terrier-0.5.5 trec-car-tools-2.6 typish-1.9.3 unlzw3-0.2.2 warc3-wet-0.2.3 warc3-wet-clueweb09-0.2.5 wget-3.2 zlib-state-0.1.5\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TT8mjrpmimJW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "79c70781-078e-493c-9c62-63430a76ceff"
      },
      "source": [
        "#Initialization de JVM\n",
        "import pyterrier as pt\n",
        "if not pt.started():\n",
        "  pt.init()"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "terrier-assemblies 5.7 jar-with-dependencies not found, downloading to /root/.pyterrier...\n",
            "Done\n",
            "terrier-python-helper 0.0.7 jar not found, downloading to /root/.pyterrier...\n",
            "Done\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "PyTerrier 0.9.2 has loaded Terrier 5.7 (built by craigm on 2022-11-10 18:30) and terrier-helper 0.0.7\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "839-7wBBWqOk"
      },
      "source": [
        "indexref2 = pt.autoclass(\"org.terrier.querying.IndexRef\").of(os.path.join(\"/content/pd_index\", \"data.properties\"))"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YFKcMV9_409G",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 362
        },
        "outputId": "d45903b6-1255-49f8-dc5f-7d0304351d29"
      },
      "source": [
        "pt.BatchRetrieve(indexref2, wmodel=\"BM25\", metadata=[\"docno\",\"title\",\"url\"]).search(\"Thomas and Mario are strikers playing in Munich\").head(10)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  qid   docid   docno                   title  \\\n",
              "0   1   28330  114620  Mario Strikers Charged   \n",
              "1   1   28329  114619    Mario Smash Football   \n",
              "2   1   71587  303588           Audi Cup 2013   \n",
              "3   1   15782   60424             Donkey Kong   \n",
              "4   1   13712   50727          Princess Daisy   \n",
              "5   1  166504  771849         Mario Balotelli   \n",
              "6   1    1074    4029                   Yoshi   \n",
              "7   1   13711   50725          Princess Peach   \n",
              "8   1  113762  509811           Audi Cup 2015   \n",
              "9   1   74872  321537       Alexander Zickler   \n",
              "\n",
              "                                                 url  rank      score  \\\n",
              "0  https://simple.wikipedia.org/wiki/Mario%20Stri...     0  30.771667   \n",
              "1  https://simple.wikipedia.org/wiki/Mario%20Smas...     1  30.306808   \n",
              "2  https://simple.wikipedia.org/wiki/Audi%20Cup%2...     2  29.796582   \n",
              "3    https://simple.wikipedia.org/wiki/Donkey%20Kong     3  28.570535   \n",
              "4  https://simple.wikipedia.org/wiki/Princess%20D...     4  28.192863   \n",
              "5  https://simple.wikipedia.org/wiki/Mario%20Balo...     5  27.400834   \n",
              "6            https://simple.wikipedia.org/wiki/Yoshi     6  25.804927   \n",
              "7  https://simple.wikipedia.org/wiki/Princess%20P...     7  25.073465   \n",
              "8  https://simple.wikipedia.org/wiki/Audi%20Cup%2...     8  24.388908   \n",
              "9  https://simple.wikipedia.org/wiki/Alexander%20...     9  23.780821   \n",
              "\n",
              "                                             query  \n",
              "0  Thomas and Mario are strikers playing in Munich  \n",
              "1  Thomas and Mario are strikers playing in Munich  \n",
              "2  Thomas and Mario are strikers playing in Munich  \n",
              "3  Thomas and Mario are strikers playing in Munich  \n",
              "4  Thomas and Mario are strikers playing in Munich  \n",
              "5  Thomas and Mario are strikers playing in Munich  \n",
              "6  Thomas and Mario are strikers playing in Munich  \n",
              "7  Thomas and Mario are strikers playing in Munich  \n",
              "8  Thomas and Mario are strikers playing in Munich  \n",
              "9  Thomas and Mario are strikers playing in Munich  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-e5c8a63a-45bd-4f4f-a284-72308dfa6508\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>qid</th>\n",
              "      <th>docid</th>\n",
              "      <th>docno</th>\n",
              "      <th>title</th>\n",
              "      <th>url</th>\n",
              "      <th>rank</th>\n",
              "      <th>score</th>\n",
              "      <th>query</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>28330</td>\n",
              "      <td>114620</td>\n",
              "      <td>Mario Strikers Charged</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/Mario%20Stri...</td>\n",
              "      <td>0</td>\n",
              "      <td>30.771667</td>\n",
              "      <td>Thomas and Mario are strikers playing in Munich</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>28329</td>\n",
              "      <td>114619</td>\n",
              "      <td>Mario Smash Football</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/Mario%20Smas...</td>\n",
              "      <td>1</td>\n",
              "      <td>30.306808</td>\n",
              "      <td>Thomas and Mario are strikers playing in Munich</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>71587</td>\n",
              "      <td>303588</td>\n",
              "      <td>Audi Cup 2013</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/Audi%20Cup%2...</td>\n",
              "      <td>2</td>\n",
              "      <td>29.796582</td>\n",
              "      <td>Thomas and Mario are strikers playing in Munich</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>15782</td>\n",
              "      <td>60424</td>\n",
              "      <td>Donkey Kong</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/Donkey%20Kong</td>\n",
              "      <td>3</td>\n",
              "      <td>28.570535</td>\n",
              "      <td>Thomas and Mario are strikers playing in Munich</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1</td>\n",
              "      <td>13712</td>\n",
              "      <td>50727</td>\n",
              "      <td>Princess Daisy</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/Princess%20D...</td>\n",
              "      <td>4</td>\n",
              "      <td>28.192863</td>\n",
              "      <td>Thomas and Mario are strikers playing in Munich</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1</td>\n",
              "      <td>166504</td>\n",
              "      <td>771849</td>\n",
              "      <td>Mario Balotelli</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/Mario%20Balo...</td>\n",
              "      <td>5</td>\n",
              "      <td>27.400834</td>\n",
              "      <td>Thomas and Mario are strikers playing in Munich</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>1</td>\n",
              "      <td>1074</td>\n",
              "      <td>4029</td>\n",
              "      <td>Yoshi</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/Yoshi</td>\n",
              "      <td>6</td>\n",
              "      <td>25.804927</td>\n",
              "      <td>Thomas and Mario are strikers playing in Munich</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>1</td>\n",
              "      <td>13711</td>\n",
              "      <td>50725</td>\n",
              "      <td>Princess Peach</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/Princess%20P...</td>\n",
              "      <td>7</td>\n",
              "      <td>25.073465</td>\n",
              "      <td>Thomas and Mario are strikers playing in Munich</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>1</td>\n",
              "      <td>113762</td>\n",
              "      <td>509811</td>\n",
              "      <td>Audi Cup 2015</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/Audi%20Cup%2...</td>\n",
              "      <td>8</td>\n",
              "      <td>24.388908</td>\n",
              "      <td>Thomas and Mario are strikers playing in Munich</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>1</td>\n",
              "      <td>74872</td>\n",
              "      <td>321537</td>\n",
              "      <td>Alexander Zickler</td>\n",
              "      <td>https://simple.wikipedia.org/wiki/Alexander%20...</td>\n",
              "      <td>9</td>\n",
              "      <td>23.780821</td>\n",
              "      <td>Thomas and Mario are strikers playing in Munich</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e5c8a63a-45bd-4f4f-a284-72308dfa6508')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-e5c8a63a-45bd-4f4f-a284-72308dfa6508 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-e5c8a63a-45bd-4f4f-a284-72308dfa6508');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t9wpqw9-9Xdd"
      },
      "source": [
        "### 2) Les poids\n",
        "\n",
        "Utilisez les différents méthodes de pondération prédefinies sur pyterrier (http://terrier.org/docs/v5.2/javadoc/org/terrier/matching/models/package-summary.html) pour voir s’il existent des méthodes que vous donnent de meilleurs résultats, c’est-à-dire inspectez les 10 premiers résultats et vérifiez si les pages d’intérêt (Thomas Müller, Mario Gómez ou Mario Götze et FC Bayern Munich) font partie des résultats, puis les 20 premiers, et finalement les 50 premiers. Existent-ils de différences importantes ?"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "wmodel_list = [\"BB2\",\"BM25\",\"DFIC\",\"DFIZ\",\"DFR_BM25\",\"DFRee\",\"DFReeKLIM\",\n",
        "               \"DirichletLM\",\"Dl\",\"DLH\",\"DLH13\",\"DPH\",\"Hiemstra_LM\",\"IFB2\",\n",
        "               \"In_expB2\",\"In_expC2\",\"InB2\",\"InL2\",\"Js_KLs\",\"LemurTF_IDF\",\"LGD\",\n",
        "               \"PL2\",\"Tf\",\"TF_IDF\",\"XSqrA_M\"]\n",
        "elem_of_interest_list = [(\"Thomas Muller\",\"Thomas M\"),\n",
        "                         (\"Mario Gomez (ou Gotze)\",\"Mario G\"),\n",
        "                         (\"Bayern Munich\",\"Bayern Munich\")]\n",
        "\n",
        "data_wmodel = {'wmodel' : [],\n",
        "               'Thomas Muller' : [],\n",
        "               'Mario Gomez (ou Gotze)' : [],\n",
        "               'Bayern Munich' : []}\n",
        "for weighting_model in wmodel_list:\n",
        "  data_wmodel[\"wmodel\"].append(weighting_model)\n",
        "  result = pt.BatchRetrieve(indexref2, wmodel=weighting_model, metadata=[\"docno\",\"title\",\"url\"]).search(\"Thomas and Mario are strikers playing in Munich\").head(50)\n",
        "  for elem_of_interest in elem_of_interest_list:\n",
        "    nb_occ = len(result[result['title'].str.contains(elem_of_interest[1])])\n",
        "    index_list = result[result['title'].str.contains(elem_of_interest[1])].index\n",
        "\n",
        "    data_wmodel[elem_of_interest[0]].append((nb_occ, index_list))"
      ],
      "metadata": {
        "id": "2h2ei3uBl7YB"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "\n",
        "df = pd.DataFrame(data_wmodel)\n",
        "df\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 833
        },
        "id": "bp57QudUmHWq",
        "outputId": "6d310d98-8d55-441a-857b-948a22918d84"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "         wmodel  Thomas Muller Mario Gomez (ou Gotze) Bayern Munich\n",
              "0           BB2        (0, [])              (1, [31])       (0, [])\n",
              "1          BM25      (1, [35])                (0, [])       (0, [])\n",
              "2          DFIC        (0, [])                (0, [])       (0, [])\n",
              "3          DFIZ      (1, [33])              (1, [46])       (0, [])\n",
              "4      DFR_BM25      (1, [37])                (0, [])       (0, [])\n",
              "5         DFRee  (2, [22, 33])              (1, [26])       (0, [])\n",
              "6     DFReeKLIM      (1, [33])                (0, [])       (0, [])\n",
              "7   DirichletLM        (0, [])                (0, [])       (0, [])\n",
              "8            Dl        (0, [])                (0, [])       (0, [])\n",
              "9           DLH      (1, [32])                (0, [])       (0, [])\n",
              "10        DLH13      (1, [26])              (1, [36])       (0, [])\n",
              "11          DPH      (1, [40])                (0, [])       (0, [])\n",
              "12  Hiemstra_LM        (0, [])                (0, [])       (0, [])\n",
              "13         IFB2        (0, [])              (1, [31])       (0, [])\n",
              "14     In_expB2        (0, [])              (1, [31])       (0, [])\n",
              "15     In_expC2        (0, [])              (1, [30])       (0, [])\n",
              "16         InB2        (0, [])              (1, [31])       (0, [])\n",
              "17         InL2      (1, [22])                (0, [])       (0, [])\n",
              "18       Js_KLs  (2, [24, 37])              (1, [30])       (0, [])\n",
              "19  LemurTF_IDF        (0, [])                (0, [])       (0, [])\n",
              "20          LGD  (2, [25, 26])          (2, [22, 30])       (0, [])\n",
              "21          PL2        (0, [])                (0, [])       (0, [])\n",
              "22           Tf        (0, [])                (0, [])       (0, [])\n",
              "23       TF_IDF      (1, [33])                (0, [])       (0, [])\n",
              "24      XSqrA_M      (1, [39])              (1, [48])       (0, [])"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-91db4c41-2f02-4550-b388-318c116ce9aa\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>wmodel</th>\n",
              "      <th>Thomas Muller</th>\n",
              "      <th>Mario Gomez (ou Gotze)</th>\n",
              "      <th>Bayern Munich</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>BB2</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(1, [31])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>BM25</td>\n",
              "      <td>(1, [35])</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>DFIC</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>DFIZ</td>\n",
              "      <td>(1, [33])</td>\n",
              "      <td>(1, [46])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>DFR_BM25</td>\n",
              "      <td>(1, [37])</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>DFRee</td>\n",
              "      <td>(2, [22, 33])</td>\n",
              "      <td>(1, [26])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>DFReeKLIM</td>\n",
              "      <td>(1, [33])</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>DirichletLM</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>Dl</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>DLH</td>\n",
              "      <td>(1, [32])</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>DLH13</td>\n",
              "      <td>(1, [26])</td>\n",
              "      <td>(1, [36])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>DPH</td>\n",
              "      <td>(1, [40])</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>Hiemstra_LM</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>IFB2</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(1, [31])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>In_expB2</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(1, [31])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>In_expC2</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(1, [30])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>InB2</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(1, [31])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>InL2</td>\n",
              "      <td>(1, [22])</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>Js_KLs</td>\n",
              "      <td>(2, [24, 37])</td>\n",
              "      <td>(1, [30])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>LemurTF_IDF</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>LGD</td>\n",
              "      <td>(2, [25, 26])</td>\n",
              "      <td>(2, [22, 30])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>PL2</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>Tf</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>TF_IDF</td>\n",
              "      <td>(1, [33])</td>\n",
              "      <td>(0, [])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>XSqrA_M</td>\n",
              "      <td>(1, [39])</td>\n",
              "      <td>(1, [48])</td>\n",
              "      <td>(0, [])</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-91db4c41-2f02-4550-b388-318c116ce9aa')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-91db4c41-2f02-4550-b388-318c116ce9aa button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-91db4c41-2f02-4550-b388-318c116ce9aa');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "ipEP-kZ7mNih"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "**REPONSE :** La plupart des modéles semblent Être équivalents (tout ceux qui ont (1,[0]) pour Thomas Muller et (1,[1]) pour Bayern Munich.\n",
        "\n",
        "Cependant on peut remarquer que certains models modifient \"grandement\" le placement des documents d'intérêt,et  ne selectionnent pas le document ayant \"Bayern Munich\" dans son titre."
      ],
      "metadata": {
        "id": "fVgcg30L1LNE"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-C9-nZU19cHp"
      },
      "source": [
        "### 3) Utilisation de termes\n",
        "Une stratégie consiste à utiliser individuellement chaque terme de la requête pour voir si vous arrivez à trouver les pages d’intérêt. Un code exemple est donné ci-dessous, cependant il n'est pas suffisant pour faire la tâche. Une fois fini votre code, réalisez la même vérification de résultats que dans le point précédent.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AnRLQ7ol_0N3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ad67438e-bf81-4259-8615-2c54d8e546b9"
      },
      "source": [
        "#utiliser individuellement chaque terme de la requête \n",
        "query = \"Thomas\"\n",
        "#query = \"Mario\"\n",
        "#query = \"Munich\"\n",
        "#query = \"Thomas and Mario are strikers playing in Munich\"\n",
        "termToSearch = [query]\n",
        "print(termToSearch)\n",
        "for term, page in zip(termToSearch,['Thomas Muller','Mario Gtze', 'FC Bayern Munich']):\n",
        "  dfres = pt.BatchRetrieve(indexref2, wmodel=\"BM25\", metadata=[\"docno\",\"title\",\"url\"]).search(term)[['title','score']]\n",
        "  print(dfres.head(10))\n",
        "  print(dfres[dfres['title']=='Wikipedia: '+page])\n",
        "\n"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['Thomas']\n",
            "                         title      score\n",
            "0             Thomas (surname)  11.700841\n",
            "1                 Saint Thomas  11.597729\n",
            "2           Thomas the Apostle  11.447421\n",
            "3                Thomas County  11.387148\n",
            "4           Thomas Wedgwood IV  11.210921\n",
            "5                 Sarah Thomas  11.152142\n",
            "6                 Helen Thomas  11.145990\n",
            "7             Gospel of Thomas  11.121265\n",
            "8      Thomas White (merchant)  11.026214\n",
            "9  Mallappally Marthoma Church  11.013052\n",
            "Empty DataFrame\n",
            "Columns: [title, score]\n",
            "Index: []\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KyLR-z7J9vQ4"
      },
      "source": [
        "### 4) Le titre\n",
        "Maintenant utilisez le champ ***title*** avec la requête complète et avec chaque terme. Expliquez  pourquoi les résultats sont différents."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Réponse 04\n",
        "\n",
        "#terme = ['Thomas Muller']\n",
        "#terme = ['Mario Gtze']\n",
        "terme = ['FC Bayern Munich']\n",
        "\n",
        "#4.1\n",
        "query = \"Thomas and Mario are strikers playing in Munich\"\n",
        "termToSearch = [query]\n",
        "print(termToSearch)\n",
        "for term, page in zip(termToSearch,terme):\n",
        "  dfres = pt.BatchRetrieve(indexref2, wmodel=\"BM25\", metadata=[\"docno\",\"title\",\"url\"]).search(term)[['title','score']]\n",
        "  print(dfres.head(10))\n",
        "  print(dfres[dfres['title']=='Wikipedia:'+page])\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KSDHjgG0OS-h",
        "outputId": "d396a53b-7f83-4340-e08c-4598b876b9ff"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['Thomas and Mario are strikers playing in Munich']\n",
            "                    title      score\n",
            "0  Mario Strikers Charged  30.771667\n",
            "1    Mario Smash Football  30.306808\n",
            "2           Audi Cup 2013  29.796582\n",
            "3             Donkey Kong  28.570535\n",
            "4          Princess Daisy  28.192863\n",
            "5         Mario Balotelli  27.400834\n",
            "6                   Yoshi  25.804927\n",
            "7          Princess Peach  25.073465\n",
            "8           Audi Cup 2015  24.388908\n",
            "9       Alexander Zickler  23.780821\n",
            "Empty DataFrame\n",
            "Columns: [title, score]\n",
            "Index: []\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Le résultat est différent car avec la deuxième méthode, le champs de recherche est réduit ( chercher un seul terme à la fois ).\n",
        "Ce qui peut offrir des meilleurs résultats."
      ],
      "metadata": {
        "id": "wKyMm_F_dcy4"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FGpUzpcc920z"
      },
      "source": [
        "### 5) L'outil\n",
        "Développez un outil/méthode/système capable de recevoir comme entrée un paragraphe de texte et comme sortie une liste ordonné de pages Wikipédia pour chaque unité lexical (token ou mot) dans le paragraphe."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#RÉPONSE 05 \n",
        "query = \"Thomas and Mario are strikers playing in Munich\"\n",
        "termToSearch = [x for x in query.split() if x[0]==x[0].upper()]\n",
        "print(\"termToSearch : \", termToSearch ,\"\\n\")\n",
        "for term, page in zip(termToSearch,term):\n",
        "  dfres = pt.BatchRetrieve(indexref2, wmodel=\"BM25\", metadata=[\"docno\",\"title\",\"url\"]).search(term)[['title','score']]\n",
        "  print (\"Token:\", term, \"\\n\")\n",
        "  print(dfres.head(10),\"\\n\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X2tOkP35TC9t",
        "outputId": "51d18e10-bb9b-4143-9f69-cd860ae0b241"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "termToSearch :  ['Thomas', 'Mario', 'Munich'] \n",
            "\n",
            "Token: Thomas \n",
            "\n",
            "                         title      score\n",
            "0             Thomas (surname)  11.700841\n",
            "1                 Saint Thomas  11.597729\n",
            "2           Thomas the Apostle  11.447421\n",
            "3                Thomas County  11.387148\n",
            "4           Thomas Wedgwood IV  11.210921\n",
            "5                 Sarah Thomas  11.152142\n",
            "6                 Helen Thomas  11.145990\n",
            "7             Gospel of Thomas  11.121265\n",
            "8      Thomas White (merchant)  11.026214\n",
            "9  Mallappally Marthoma Church  11.013052 \n",
            "\n",
            "Token: Mario \n",
            "\n",
            "                          title      score\n",
            "0                Princess Peach  16.478378\n",
            "1        Mario (disambiguation)  16.451796\n",
            "2  Super Mario (disambiguation)  16.419316\n",
            "3                         Mario  16.395732\n",
            "4              Charles Martinet  16.349001\n",
            "5                    Bowser Jr.  16.214048\n",
            "6                         Wario  16.177717\n",
            "7                         Luigi  16.132403\n",
            "8                   Mario Bros.  15.990315\n",
            "9                Princess Daisy  15.961688 \n",
            "\n",
            "Token: Munich \n",
            "\n",
            "                         title      score\n",
            "0                   Bundesliga  15.502280\n",
            "1      Olympiastadion (Munich)  15.192906\n",
            "2                       Munich  15.082408\n",
            "3                Audi Cup 2015  14.993887\n",
            "4  Technical University Munich  14.951113\n",
            "5                    Munich RE  14.887408\n",
            "6          Stephan Beckenbauer  14.885528\n",
            "7               Altstadt-Lehel  14.833179\n",
            "8         University of Munich  14.782431\n",
            "9                         Laim  14.782431 \n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cR-Dumjy99Yj"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n",
        "En plus à l’information du TP, vous pouvez utiliser les ressources/idées suivants pour améliorer les résultats du système à développer :\n",
        "- La combinaison de plusieurs méthodes de pondération c’est possible en utilisant les indications sur https://pyterrier.readthedocs.io/en/latest/pipeline_examples.html .\n",
        "- Comme mentionné dans le TP1, il existe plusieurs type de champs. On a utilisé celui de la classe TextField. Renseignez vous sur les autres (StringField par exemple) et ses différences.\n",
        "- Les distances d’édition sont largement utilisés par les moteurs de recherche. Par exemple si vous tapez « paris sain germain » sur Google il va vous suggérer la requête « paris saint germain » l’équipe de football. Vous pouvez utiliser les distances d’édition pour trouver des entités avec une orthographe différent que celle sur Wikipédia. Une de distance très performantes est la distance Levenshtein implémentée dans la classe LuceneLevenshteinDistance du paquet lucene.search.spell.\n",
        "- Les moteurs de recherche peuvent être utilisés pour calculer la similarité entre les mots ou entre les entités. Par exemple, sur Google si vous tapez « orange » et « carrot » il y a plus de résultats que si vous tapez « purple » et « carrot ». Vous pouvez utiliser cette information pour décider si deux pages Wikipédia font partie d’un même sujet.\n",
        "- Utilisez le fichier [anchor_text_en.ttl](http://downloads.dbpedia.org/2015-10/core-i18n/en/anchor_text_en.ttl.bz2) qui contient les anchor texts  de la Wikipédia en anglais. Le format du fichier est décrit [ici](https://iai-group.github.io/DBpedia-Entity/index_details.html). Transformez-le dans ce format au autre similaire pour faire l'indexation:\n",
        "\n",
        "|             Frequency                 \t|       Wikipedia Page        \t| Anchor Text \t|\n",
        "|------------------------------\t|---------------\t|-----------\t|\n",
        "|162| \\<Saturn_Award_for_Best_Supporting_Actor> |\"Best Supporting Actor\"|\n",
        "|14 |\\<André_Golke>| \"Golke\"|\n",
        "|13 |\\<Charles_Pinckney_National_Historic_Site> |\"Charles Pinckney National Historic Site\"|\n",
        "|25 |\\<Thomas_Manfredini>| \"Manfredini\"|\n",
        "|13 |<Marek_Zieńczuk> |\"Marek Zieńczuk\"|\n",
        "|14 |\\<Sigurd_Ulveseth>| \"Sigurd Ulveseth\"|\n",
        "|104| <Pole_Position_(video_game)>| \"Pole Position\"|\n",
        "|10| <Bang_(band)>| \"Bang\"|\n",
        "|44| <George_Susce_(catcher)>| \"George Susce\"|\n",
        "|11| \\<Kyle_Rankin>| \"Kyle Rankin\"|\n",
        "|…|…|…|\n",
        "\n",
        "Donc, la premier colonne correspond au nombre de fois que la chaîne de caractères de la troisième colonne a été utilisé pour faire allusion à la page Wikipédia de la deuxième colonne. C’est-à-dire que \"Best Supporting Actor\" a été utilisé 162 fois dans Wikipédia pour faire allusion à la page Wikipédia https://en.wikipedia.org/wiki/Saturn_Award_for_Best_Supporting_Actor . Le fichier anchor_text_en.tll  contient les informations de la Wikipédia en anglais et pas de la Wikipédia simple, donc certain titres n’ont pas de correspondance."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gKcDjgEknZE1"
      },
      "source": [],
      "execution_count": 11,
      "outputs": []
    }
  ]
}