strip_common_prefixes_from_questions.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/harmonydata/harmony_examples/blob/main/strip_common_prefixes_from_questions.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e6a01f42",
      "metadata": {
        "id": "e6a01f42"
      },
      "source": [
        "![The Harmony Project logo](https://raw.githubusercontent.com/harmonydata/brand/main/Logo/PNG/%D0%BB%D0%BE%D0%B3%D0%BE%20%D1%84%D1%83%D0%BB-05.png)\n",
        "\n",
        "<a href=\"https://harmonydata.ac.uk\"><span align=\"left\">🌐 harmonydata.ac.uk</span></a>\n",
        "<a href=\"https://github.com/harmonydata/harmony\"><img align=\"left\" src=\"https://raw.githubusercontent.com//harmonydata/.github/main/profile/github-mark.svg\" alt=\"Harmony Python package | Github\" width=\"21px\"/></a>\n",
        "<a href=\"https://www.linkedin.com/company/harmonydata\"><img align=\"left\" src=\"https://raw.githubusercontent.com//harmonydata/.github/main/profile/linkedin.svg\" alt=\"Harmony | LinkedIn\" width=\"21px\"/></a>\n",
        "<a href=\"https://twitter.com/harmony_data\"><img align=\"left\" src=\"https://raw.githubusercontent.com//harmonydata/.github/main/profile/x.svg\" alt=\"Harmony | X\" width=\"21px\"/></a>\n",
        "<a href=\"https://www.instagram.com/harmonydata/\"><img align=\"left\" src=\"https://raw.githubusercontent.com//harmonydata/.github/main/profile/instagram.svg\" alt=\"Harmony | Instagram\" width=\"21px\"/></a>\n",
        "<a href=\"https://www.facebook.com/people/Harmony-Project/100086772661697/\"><img align=\"left\" src=\"https://raw.githubusercontent.com//harmonydata/.github/main/profile/fb.svg\" alt=\"Harmony | Facebook\" width=\"21px\"/></a>\n",
        "<a href=\"https://www.youtube.com/channel/UCraLlfBr0jXwap41oQ763OQ\"><img align=\"left\" src=\"https://raw.githubusercontent.com//harmonydata/.github/main/profile/yt.svg\" alt=\"Harmony | YouTube\" width=\"21px\"/></a>\n",
        "\n",
        "# Harmony example: stripping common prefixes from questions\n",
        "\n",
        "Questions often start with a default list of prefixes such as\n",
        "\n",
        "        \"Have you ever\",\n",
        "        \"Did you ever\",\n",
        "        \"Do you\",\n",
        "        \"Is it true that\",\n",
        "        \"Would you say\",\n",
        "        \"Can you\",\n",
        "        \"Are you aware that\",\n",
        "        \"Do you think\",\n",
        "\n",
        "Harmony comes with a utility function to strip these prefixes. You can customise it to your own list.\n",
        "\n",
        "## Install Harmony"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "41dcffbd",
      "metadata": {
        "id": "41dcffbd",
        "outputId": "9d40fe8a-7cc6-4cb2-fa43-ee6f2bb2ba0f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Cloning into 'harmony'...\n",
            "remote: Enumerating objects: 5197, done.\u001b[K\n",
            "remote: Counting objects: 100% (362/362), done.\u001b[K\n",
            "remote: Compressing objects: 100% (140/140), done.\u001b[K\n",
            "remote: Total 5197 (delta 235), reused 286 (delta 201), pack-reused 4835 (from 1)\u001b[K\n",
            "Receiving objects: 100% (5197/5197), 23.75 MiB | 2.94 MiB/s, done.\n",
            "Resolving deltas: 100% (2241/2241), done.\n"
          ]
        }
      ],
      "source": [
        "!git clone https://github.com/harmonydata/harmony.git"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "04c80b01",
      "metadata": {
        "id": "04c80b01",
        "outputId": "c58299a3-7982-4711-b4f3-fcac2d583bfc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Processing /home/thomas/projects_client/wellcome/harmony_examples/harmony\n",
            "  Installing build dependencies ... \u001b[?25ldone\n",
            "\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\n",
            "\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n",
            "\u001b[?25hCollecting pydantic==2.8.2 (from harmonydata==1.0.0)\n",
            "  Using cached pydantic-2.8.2-py3-none-any.whl (423 kB)\n",
            "Collecting pandas==2.2.2 (from harmonydata==1.0.0)\n",
            "  Using cached pandas-2.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.0 MB)\n",
            "Requirement already satisfied: tika==2.6.0 in /home/thomas/anaconda3/lib/python3.11/site-packages (from harmonydata==1.0.0) (2.6.0)\n",
            "Requirement already satisfied: lxml==4.9.2 in /home/thomas/anaconda3/lib/python3.11/site-packages (from harmonydata==1.0.0) (4.9.2)\n",
            "Requirement already satisfied: langdetect==1.0.9 in /home/thomas/anaconda3/lib/python3.11/site-packages (from harmonydata==1.0.0) (1.0.9)\n",
            "Requirement already satisfied: XlsxWriter==3.0.9 in /home/thomas/anaconda3/lib/python3.11/site-packages (from harmonydata==1.0.0) (3.0.9)\n",
            "Requirement already satisfied: openpyxl==3.1.2 in /home/thomas/anaconda3/lib/python3.11/site-packages (from harmonydata==1.0.0) (3.1.2)\n",
            "Requirement already satisfied: wget==3.2 in /home/thomas/anaconda3/lib/python3.11/site-packages (from harmonydata==1.0.0) (3.2)\n",
            "Requirement already satisfied: sentence-transformers==2.2.2 in /home/thomas/anaconda3/lib/python3.11/site-packages (from harmonydata==1.0.0) (2.2.2)\n",
            "Collecting numpy==2.1.0 (from harmonydata==1.0.0)\n",
            "  Using cached numpy-2.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.3 MB)\n",
            "Requirement already satisfied: sklearn-crfsuite==0.5.0 in /home/thomas/anaconda3/lib/python3.11/site-packages (from harmonydata==1.0.0) (0.5.0)\n",
            "Requirement already satisfied: scikit-learn in /home/thomas/anaconda3/lib/python3.11/site-packages (from harmonydata==1.0.0) (1.4.2)\n",
            "Requirement already satisfied: scipy==1.14.1 in /home/thomas/anaconda3/lib/python3.11/site-packages (from harmonydata==1.0.0) (1.14.1)\n",
            "Collecting huggingface-hub==0.25.0 (from harmonydata==1.0.0)\n",
            "  Using cached huggingface_hub-0.25.0-py3-none-any.whl (436 kB)\n",
            "Requirement already satisfied: filelock in /home/thomas/.local/lib/python3.11/site-packages (from huggingface-hub==0.25.0->harmonydata==1.0.0) (3.13.1)\n",
            "Requirement already satisfied: fsspec>=2023.5.0 in /home/thomas/anaconda3/lib/python3.11/site-packages (from huggingface-hub==0.25.0->harmonydata==1.0.0) (2024.3.1)\n",
            "Requirement already satisfied: packaging>=20.9 in /home/thomas/anaconda3/lib/python3.11/site-packages (from huggingface-hub==0.25.0->harmonydata==1.0.0) (23.2)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /home/thomas/anaconda3/lib/python3.11/site-packages (from huggingface-hub==0.25.0->harmonydata==1.0.0) (6.0)\n",
            "Requirement already satisfied: requests in /home/thomas/anaconda3/lib/python3.11/site-packages (from huggingface-hub==0.25.0->harmonydata==1.0.0) (2.32.3)\n",
            "Requirement already satisfied: tqdm>=4.42.1 in /home/thomas/anaconda3/lib/python3.11/site-packages (from huggingface-hub==0.25.0->harmonydata==1.0.0) (4.65.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/thomas/anaconda3/lib/python3.11/site-packages (from huggingface-hub==0.25.0->harmonydata==1.0.0) (4.11.0)\n",
            "Requirement already satisfied: six in /home/thomas/anaconda3/lib/python3.11/site-packages (from langdetect==1.0.9->harmonydata==1.0.0) (1.16.0)\n",
            "Requirement already satisfied: et-xmlfile in /home/thomas/anaconda3/lib/python3.11/site-packages (from openpyxl==3.1.2->harmonydata==1.0.0) (1.1.0)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /home/thomas/anaconda3/lib/python3.11/site-packages (from pandas==2.2.2->harmonydata==1.0.0) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /home/thomas/anaconda3/lib/python3.11/site-packages (from pandas==2.2.2->harmonydata==1.0.0) (2022.7)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /home/thomas/anaconda3/lib/python3.11/site-packages (from pandas==2.2.2->harmonydata==1.0.0) (2023.3)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /home/thomas/anaconda3/lib/python3.11/site-packages (from pydantic==2.8.2->harmonydata==1.0.0) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.20.1 in /home/thomas/anaconda3/lib/python3.11/site-packages (from pydantic==2.8.2->harmonydata==1.0.0) (2.20.1)\n",
            "Requirement already satisfied: transformers<5.0.0,>=4.6.0 in /home/thomas/anaconda3/lib/python3.11/site-packages (from sentence-transformers==2.2.2->harmonydata==1.0.0) (4.30.2)\n",
            "Requirement already satisfied: torch>=1.6.0 in /home/thomas/anaconda3/lib/python3.11/site-packages (from sentence-transformers==2.2.2->harmonydata==1.0.0) (2.0.1)\n",
            "Requirement already satisfied: torchvision in /home/thomas/anaconda3/lib/python3.11/site-packages (from sentence-transformers==2.2.2->harmonydata==1.0.0) (0.15.2)\n",
            "Requirement already satisfied: nltk in /home/thomas/anaconda3/lib/python3.11/site-packages (from sentence-transformers==2.2.2->harmonydata==1.0.0) (3.6.7)\n",
            "Requirement already satisfied: sentencepiece in /home/thomas/anaconda3/lib/python3.11/site-packages (from sentence-transformers==2.2.2->harmonydata==1.0.0) (0.1.99)\n",
            "Requirement already satisfied: python-crfsuite>=0.9.7 in /home/thomas/anaconda3/lib/python3.11/site-packages (from sklearn-crfsuite==0.5.0->harmonydata==1.0.0) (0.9.10)\n",
            "Requirement already satisfied: tabulate>=0.4.2 in /home/thomas/anaconda3/lib/python3.11/site-packages (from sklearn-crfsuite==0.5.0->harmonydata==1.0.0) (0.9.0)\n",
            "Requirement already satisfied: setuptools in /home/thomas/anaconda3/lib/python3.11/site-packages (from tika==2.6.0->harmonydata==1.0.0) (67.8.0)\n",
            "Requirement already satisfied: joblib>=1.2.0 in /home/thomas/anaconda3/lib/python3.11/site-packages (from scikit-learn->harmonydata==1.0.0) (1.2.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /home/thomas/anaconda3/lib/python3.11/site-packages (from scikit-learn->harmonydata==1.0.0) (3.5.0)\n",
            "Requirement already satisfied: sympy in /home/thomas/anaconda3/lib/python3.11/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2->harmonydata==1.0.0) (1.11.1)\n",
            "Requirement already satisfied: networkx in /home/thomas/anaconda3/lib/python3.11/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2->harmonydata==1.0.0) (2.8.4)\n",
            "Requirement already satisfied: jinja2 in /home/thomas/anaconda3/lib/python3.11/site-packages (from torch>=1.6.0->sentence-transformers==2.2.2->harmonydata==1.0.0) (3.1.2)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /home/thomas/anaconda3/lib/python3.11/site-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers==2.2.2->harmonydata==1.0.0) (2022.7.9)\n",
            "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /home/thomas/anaconda3/lib/python3.11/site-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers==2.2.2->harmonydata==1.0.0) (0.13.3)\n",
            "Requirement already satisfied: safetensors>=0.3.1 in /home/thomas/anaconda3/lib/python3.11/site-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers==2.2.2->harmonydata==1.0.0) (0.4.2)\n",
            "Requirement already satisfied: click in /home/thomas/anaconda3/lib/python3.11/site-packages (from nltk->sentence-transformers==2.2.2->harmonydata==1.0.0) (8.0.4)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /home/thomas/anaconda3/lib/python3.11/site-packages (from requests->huggingface-hub==0.25.0->harmonydata==1.0.0) (2.0.4)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /home/thomas/anaconda3/lib/python3.11/site-packages (from requests->huggingface-hub==0.25.0->harmonydata==1.0.0) (3.4)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /home/thomas/anaconda3/lib/python3.11/site-packages (from requests->huggingface-hub==0.25.0->harmonydata==1.0.0) (1.26.19)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /home/thomas/anaconda3/lib/python3.11/site-packages (from requests->huggingface-hub==0.25.0->harmonydata==1.0.0) (2023.5.7)\n",
            "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /home/thomas/anaconda3/lib/python3.11/site-packages (from torchvision->sentence-transformers==2.2.2->harmonydata==1.0.0) (10.3.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /home/thomas/anaconda3/lib/python3.11/site-packages (from jinja2->torch>=1.6.0->sentence-transformers==2.2.2->harmonydata==1.0.0) (2.1.1)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: mpmath>=0.19 in /home/thomas/anaconda3/lib/python3.11/site-packages (from sympy->torch>=1.6.0->sentence-transformers==2.2.2->harmonydata==1.0.0) (1.2.1)\n",
            "Building wheels for collected packages: harmonydata\n",
            "  Building wheel for harmonydata (pyproject.toml) ... \u001b[?25ldone\n",
            "\u001b[?25h  Created wheel for harmonydata: filename=harmonydata-1.0.0-py3-none-any.whl size=151144 sha256=62c4da40861270c6c52b542f2fee0d670fc4d301d69e434600b0789a633f9707\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-_5ich0hi/wheels/0d/9f/6b/827587d43e4247317ee80869d5b52d1399aa8cf90a0dde7646\n",
            "Successfully built harmonydata\n",
            "Installing collected packages: numpy, pydantic, pandas, huggingface-hub, harmonydata\n",
            "  Attempting uninstall: numpy\n",
            "    Found existing installation: numpy 1.23.5\n",
            "    Uninstalling numpy-1.23.5:\n",
            "      Successfully uninstalled numpy-1.23.5\n",
            "  Attempting uninstall: pydantic\n",
            "    Found existing installation: pydantic 1.10.18\n",
            "    Uninstalling pydantic-1.10.18:\n",
            "      Successfully uninstalled pydantic-1.10.18\n",
            "  Attempting uninstall: pandas\n",
            "    Found existing installation: pandas 2.2.3\n",
            "    Uninstalling pandas-2.2.3:\n",
            "      Successfully uninstalled pandas-2.2.3\n",
            "  Attempting uninstall: huggingface-hub\n",
            "    Found existing installation: huggingface-hub 0.22.2\n",
            "    Uninstalling huggingface-hub-0.22.2:\n",
            "      Successfully uninstalled huggingface-hub-0.22.2\n",
            "  Attempting uninstall: harmonydata\n",
            "    Found existing installation: harmonydata 1.0.0\n",
            "    Uninstalling harmonydata-1.0.0:\n",
            "      Successfully uninstalled harmonydata-1.0.0\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "tables 3.8.0 requires blosc2~=2.0.0, which is not installed.\n",
            "spacy 3.5.4 requires pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4, but you have pydantic 2.8.2 which is incompatible.\n",
            "confection 0.1.0 requires pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4, but you have pydantic 2.8.2 which is incompatible.\n",
            "streamlit 1.33.0 requires numpy<2,>=1.19.3, but you have numpy 2.1.0 which is incompatible.\n",
            "faststylometry 1.0.4 requires numpy<=1.24.3,>=1.18, but you have numpy 2.1.0 which is incompatible.\n",
            "faststylometry 1.0.4 requires pandas<=2.1.0,>=1.5.3, but you have pandas 2.2.2 which is incompatible.\n",
            "faststylometry 1.0.4 requires scikit-learn<=1.3.0,>=1.2.0, but you have scikit-learn 1.4.2 which is incompatible.\n",
            "numba 0.57.0 requires numpy<1.25,>=1.21, but you have numpy 2.1.0 which is incompatible.\n",
            "azureml-dataset-runtime 1.58.0 requires numpy!=1.19.3,<1.24; sys_platform == \"linux\", but you have numpy 2.1.0 which is incompatible.\n",
            "label-studio 1.12.1 requires numpy<2.0.0,>=1.24.3, but you have numpy 2.1.0 which is incompatible.\n",
            "label-studio 1.12.1 requires pydantic<=1.11.0,>=1.7.3, but you have pydantic 2.8.2 which is incompatible.\n",
            "label-studio 1.12.1 requires requests==2.31.0, but you have requests 2.32.3 which is incompatible.\n",
            "instagrapi 2.0.0 requires pydantic==1.10.9, but you have pydantic 2.8.2 which is incompatible.\n",
            "simpletransformers 0.70.0 requires transformers>=4.31.0, but you have transformers 4.30.2 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed harmonydata-1.0.0 huggingface-hub-0.25.0 numpy-2.1.0 pandas-2.2.2 pydantic-2.8.2\n"
          ]
        }
      ],
      "source": [
        "!cd harmony/; pip install ."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "eb7e8456",
      "metadata": {
        "id": "eb7e8456"
      },
      "outputs": [],
      "source": [
        "from harmony.parsing.util import strip_prefixes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9bfd3711",
      "metadata": {
        "id": "9bfd3711"
      },
      "outputs": [],
      "source": [
        "questions = [\n",
        "    \"Have you ever lived with anyone?\",\n",
        "    \"Would you say that you could talk frankly and share your feelings with him/her?\",\n",
        "    \"Please tick which of the following apply to you:\"\n",
        "]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f82a344a",
      "metadata": {
        "id": "f82a344a"
      },
      "source": [
        "## Strip prefixes with the default list"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9437be55",
      "metadata": {
        "id": "9437be55",
        "outputId": "c2886732-27a5-4ec0-fa50-dc94e1790a10"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "lived with anyone?\n",
            "that you could talk frankly and share your feelings with him/her?\n",
            "Please tick which of the following apply to you:\n"
          ]
        }
      ],
      "source": [
        "for question in questions:\n",
        "    print (strip_prefixes(question))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "edfe47ee",
      "metadata": {
        "id": "edfe47ee"
      },
      "source": [
        "## Override the default prefix list"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f464b281",
      "metadata": {
        "scrolled": true,
        "id": "f464b281",
        "outputId": "a476e319-1685-46c2-b006-728fbd47849d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Have you ever lived with anyone?\n",
            "Would you say that you could talk frankly and share your feelings with him/her?\n",
            "which of the following apply to you:\n"
          ]
        }
      ],
      "source": [
        "for question in questions:\n",
        "    print (strip_prefixes(question, [\"Please tick\"]))"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "harmonysandboxvenv",
      "language": "python",
      "name": "harmonysandboxvenv"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.3"
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}