Finetune2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Overview of Colaboratory Features",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ameasure/colab_tutorials/blob/master/Finetune2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ic5-oTjJodK_",
        "colab_type": "text"
      },
      "source": [
        "# Transfer Learning with Finetune \n",
        "Finetune is a library that creates a scikit-learn style `fit(), predict()` interface to a variety of state-of-the-art pretrained language models, making them much easier to use.\n",
        "\n",
        "# Resources:\n",
        "* [Finetune Quick Start Guide](https://finetune.indico.io/)\n",
        "* [Finetune Source Code](https://github.com/IndicoDataSolutions/finetune)\n",
        "* [GPT1 Paper](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf) (Default model in Finetune)\n",
        "* [GPT2 Paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)\n",
        "\n",
        "# Download Packages and Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aMc7WPBRob2v",
        "colab_type": "code",
        "outputId": "38c5e700-8dc7-4bc6-a59b-c1869dbb682b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 904
        }
      },
      "source": [
        "!pip install -U finetune\n",
        "!wget 'https://github.com/ameasure/autocoding-class/raw/master/msha.xlsx'"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already up-to-date: finetune in /usr/local/lib/python3.6/dist-packages (0.6.7)\n",
            "Requirement already satisfied, skipping upgrade: lxml>=4.3.3 in /usr/local/lib/python3.6/dist-packages (from finetune) (4.3.4)\n",
            "Requirement already satisfied, skipping upgrade: joblib>=0.12.0 in /usr/local/lib/python3.6/dist-packages (from finetune) (0.13.2)\n",
            "Requirement already satisfied, skipping upgrade: numpy>=1.13.0 in /usr/local/lib/python3.6/dist-packages (from finetune) (1.16.4)\n",
            "Requirement already satisfied, skipping upgrade: pandas>=0.23.1 in /usr/local/lib/python3.6/dist-packages (from finetune) (0.24.2)\n",
            "Requirement already satisfied, skipping upgrade: imblearn>=0.0 in /usr/local/lib/python3.6/dist-packages (from finetune) (0.0)\n",
            "Requirement already satisfied, skipping upgrade: tqdm>=4.0.0 in /usr/local/lib/python3.6/dist-packages (from finetune) (4.28.1)\n",
            "Requirement already satisfied, skipping upgrade: nltk>=3.2.4 in /usr/local/lib/python3.6/dist-packages (from finetune) (3.2.5)\n",
            "Requirement already satisfied, skipping upgrade: ftfy>=4.4.0 in /usr/local/lib/python3.6/dist-packages (from finetune) (5.5.1)\n",
            "Requirement already satisfied, skipping upgrade: spacy>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from finetune) (2.1.4)\n",
            "Requirement already satisfied, skipping upgrade: h5py>=2.8.0 in /usr/local/lib/python3.6/dist-packages (from finetune) (2.8.0)\n",
            "Requirement already satisfied, skipping upgrade: bs4>=0.0.1 in /usr/local/lib/python3.6/dist-packages (from finetune) (0.0.1)\n",
            "Requirement already satisfied, skipping upgrade: regex>=2019.03.12 in /usr/local/lib/python3.6/dist-packages (from finetune) (2019.6.8)\n",
            "Requirement already satisfied, skipping upgrade: scikit-learn>=0.18.0 in /usr/local/lib/python3.6/dist-packages (from finetune) (0.21.2)\n",
            "Requirement already satisfied, skipping upgrade: scipy>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from finetune) (1.3.0)\n",
            "Requirement already satisfied, skipping upgrade: python-dateutil>=2.5.0 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.23.1->finetune) (2.5.3)\n",
            "Requirement already satisfied, skipping upgrade: pytz>=2011k in /usr/local/lib/python3.6/dist-packages (from pandas>=0.23.1->finetune) (2018.9)\n",
            "Requirement already satisfied, skipping upgrade: imbalanced-learn in /usr/local/lib/python3.6/dist-packages (from imblearn>=0.0->finetune) (0.4.3)\n",
            "Requirement already satisfied, skipping upgrade: six in /usr/local/lib/python3.6/dist-packages (from nltk>=3.2.4->finetune) (1.12.0)\n",
            "Requirement already satisfied, skipping upgrade: wcwidth in /usr/local/lib/python3.6/dist-packages (from ftfy>=4.4.0->finetune) (0.1.7)\n",
            "Requirement already satisfied, skipping upgrade: jsonschema<3.1.0,>=2.6.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.0->finetune) (2.6.0)\n",
            "Requirement already satisfied, skipping upgrade: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.0->finetune) (1.0.2)\n",
            "Requirement already satisfied, skipping upgrade: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.0->finetune) (2.21.0)\n",
            "Requirement already satisfied, skipping upgrade: thinc<7.1.0,>=7.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.0->finetune) (7.0.4)\n",
            "Requirement already satisfied, skipping upgrade: plac<1.0.0,>=0.9.6 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.0->finetune) (0.9.6)\n",
            "Requirement already satisfied, skipping upgrade: preshed<2.1.0,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.0->finetune) (2.0.1)\n",
            "Requirement already satisfied, skipping upgrade: blis<0.3.0,>=0.2.2 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.0->finetune) (0.2.4)\n",
            "Requirement already satisfied, skipping upgrade: wasabi<1.1.0,>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.0->finetune) (0.2.2)\n",
            "Requirement already satisfied, skipping upgrade: srsly<1.1.0,>=0.0.5 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.0->finetune) (0.0.6)\n",
            "Requirement already satisfied, skipping upgrade: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy>=2.0.0->finetune) (2.0.2)\n",
            "Requirement already satisfied, skipping upgrade: beautifulsoup4 in /usr/local/lib/python3.6/dist-packages (from bs4>=0.0.1->finetune) (4.6.3)\n",
            "Requirement already satisfied, skipping upgrade: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.0.0->finetune) (1.24.3)\n",
            "Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.0.0->finetune) (3.0.4)\n",
            "Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.0.0->finetune) (2019.3.9)\n",
            "Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.0.0->finetune) (2.8)\n",
            "--2019-06-16 18:30:45--  https://github.com/ameasure/autocoding-class/raw/master/msha.xlsx\n",
            "Resolving github.com (github.com)... 52.74.223.119\n",
            "Connecting to github.com (github.com)|52.74.223.119|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://raw.githubusercontent.com/ameasure/autocoding-class/master/msha.xlsx [following]\n",
            "--2019-06-16 18:30:46--  https://raw.githubusercontent.com/ameasure/autocoding-class/master/msha.xlsx\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 4183086 (4.0M) [application/octet-stream]\n",
            "Saving to: ‘msha.xlsx.3’\n",
            "\n",
            "msha.xlsx.3         100%[===================>]   3.99M  --.-KB/s    in 0.03s   \n",
            "\n",
            "2019-06-16 18:30:46 (152 MB/s) - ‘msha.xlsx.3’ saved [4183086/4183086]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KdpWCKhdxUNB",
        "colab_type": "code",
        "outputId": "85f58ca4-5086-4fb9-e701-6e6ff3dd4d36",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "df = pd.read_excel('msha.xlsx')\n",
        "df['ACCIDENT_YEAR'] = df['ACCIDENT_DT'].apply(lambda x: x.year)\n",
        "df['ACCIDENT_YEAR'].value_counts()\n",
        "df_train = df[df['ACCIDENT_YEAR'].isin([2010, 2011])][:3200].copy()\n",
        "df_valid = df[df['ACCIDENT_YEAR'] == 2012][:1000].copy()\n",
        "print('training rows:', len(df_train))\n",
        "print('validation rows:', len(df_valid))"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "training rows: 3200\n",
            "validation rows: 1000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HiItWBjnhG1Q",
        "colab_type": "text"
      },
      "source": [
        "# Train Classifier\n",
        "By default finetune uses the GPT1 pretrained model. Parameters include the following:\n",
        "* `batch_size`: The number of training examples used to calculate each gradient update. Bigger batches train the model faster but take up more memory on the GPU. If it's too big you will get out-of-memory errors.\n",
        "* `max_length` - The maximum number of words that will be considered in each training example. You want this to be just big enough for your data. Longer lengths require more processing time and more GPU memory but also allow the model to read all of the words in longer narratives.\n",
        "* `n_epochs` - The number of complete passes through the training set. Too much and the model risks overfitting. Too little and it risks underfitting.\n",
        "* `val_size` - The number of examples from the training set that will be used to periodically validate the model during training.\n",
        "\n",
        "Other parameters are left at their defaults. See [configuration options](https://finetune.indico.io/#finetune-model-configuration-options) for other options."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zxOweCiZA0KU",
        "colab_type": "code",
        "outputId": "be2a9088-9eb3-4689-aeb8-5f8fdfe85e27",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "from finetune import Classifier\n",
        "\n",
        "model = Classifier(batch_size=32, \n",
        "                   max_length=90, \n",
        "                   n_epochs=4, \n",
        "                   val_size=0)\n",
        "model.fit(df_train['NARRATIVE'], df_train['INJ_BODY_PART'])"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "I0616 18:32:21.420229 139771120363392 base.py:104] Saving tensorboard output to /tmp/Finetunexxxoohxu\n",
            "I0616 18:32:21.469501 139771120363392 config.py:78]  Visible GPUs: {0: Tesla T4}\n",
            "Epoch 4/4: 100%|██████████| 3200/3200 [01:03<00:00, 22.55it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LWbjdsG29Pqb",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 394
        },
        "outputId": "cdac1950-88ce-4f57-d532-5f1d548cd1cc"
      },
      "source": [
        "# re-use the existing tensorflow graph\n",
        "with model.cached_predict():\n",
        "  # generate predictions\n",
        "  df_valid['PREDICTED_PART'] = model.predict(df_valid['NARRATIVE'].values)\n",
        "# look at a sample\n",
        "df_valid[['NARRATIVE', 'INJ_BODY_PART', 'PREDICTED_PART']].sample(5)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\rInference:   0%|          | 0/1000 [00:00<?, ?it/s]/usr/local/lib/python3.6/dist-packages/finetune/encoding/input_encoder.py:182: UserWarning: Some examples are longer than the max_length. Please trim documents or increase `max_length`. Fallback behaviour is to use the first 88 byte-pair encoded tokens\n",
            "  \"Fallback behaviour is to use the first {} byte-pair encoded tokens\".format(max_length - 2)\n",
            "/usr/local/lib/python3.6/dist-packages/finetune/encoding/input_encoder.py:135: UserWarning: Document is longer than max length allowed, trimming document to 90 tokens.\n",
            "  max_length\n",
            "Inference: 100%|██████████| 1000/1000 [00:19<00:00, 51.77it/s]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>NARRATIVE</th>\n",
              "      <th>INJ_BODY_PART</th>\n",
              "      <th>PREDICTED_PART</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2486</th>\n",
              "      <td>Coal rolled out from rib striking employee on ...</td>\n",
              "      <td>LOWER LEG/TIBIA/FIBULA</td>\n",
              "      <td>LOWER LEG/TIBIA/FIBULA</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3452</th>\n",
              "      <td>Employee was using a knife to cut a rope and s...</td>\n",
              "      <td>FINGER(S)/THUMB</td>\n",
              "      <td>FINGER(S)/THUMB</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3457</th>\n",
              "      <td>Walking on belt line said too much weight on r...</td>\n",
              "      <td>BACK (MUSCLES/SPINE/S-CORD/TAILBONE)</td>\n",
              "      <td>MULTIPLE PARTS (MORE THAN ONE MAJOR)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3106</th>\n",
              "      <td>Employee was welding and a foreign body enter ...</td>\n",
              "      <td>EYE(S) OPTIC NERVE/VISON</td>\n",
              "      <td>EYE(S) OPTIC NERVE/VISON</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1984</th>\n",
              "      <td>This accident is still under investigation and...</td>\n",
              "      <td>BACK (MUSCLES/SPINE/S-CORD/TAILBONE)</td>\n",
              "      <td>BACK (MUSCLES/SPINE/S-CORD/TAILBONE)</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                              NARRATIVE  ...                        PREDICTED_PART\n",
              "2486  Coal rolled out from rib striking employee on ...  ...                LOWER LEG/TIBIA/FIBULA\n",
              "3452  Employee was using a knife to cut a rope and s...  ...                       FINGER(S)/THUMB\n",
              "3457  Walking on belt line said too much weight on r...  ...  MULTIPLE PARTS (MORE THAN ONE MAJOR)\n",
              "3106  Employee was welding and a foreign body enter ...  ...              EYE(S) OPTIC NERVE/VISON\n",
              "1984  This accident is still under investigation and...  ...  BACK (MUSCLES/SPINE/S-CORD/TAILBONE)\n",
              "\n",
              "[5 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lmb9BX0RA9bN",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "2bd744f3-933c-4291-813f-c5338f2c54b0"
      },
      "source": [
        "# calculate the predicted probabilities\n",
        "with model.cached_predict():\n",
        "  df_valid['PROB_DICT'] = model.predict_proba(df_valid['NARRATIVE'].values)\n",
        "  \n",
        "with pd.option_context('display.max_colwidth', 500):\n",
        "  df_valid[['NARRATIVE', 'PREDICTED_PART', 'PROB_DICT']].head(2)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/finetune/encoding/input_encoder.py:182: UserWarning: Some examples are longer than the max_length. Please trim documents or increase `max_length`. Fallback behaviour is to use the first 88 byte-pair encoded tokens\n",
            "  \"Fallback behaviour is to use the first {} byte-pair encoded tokens\".format(max_length - 2)\n",
            "/usr/local/lib/python3.6/dist-packages/finetune/encoding/input_encoder.py:135: UserWarning: Document is longer than max length allowed, trimming document to 90 tokens.\n",
            "  max_length\n",
            "Inference: 100%|██████████| 1000/1000 [00:07<00:00, 126.04it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HvZ3fBw3BBWf",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 145
        },
        "outputId": "c8de4eeb-dff2-4ac2-bde1-d15e04c545c2"
      },
      "source": [
        "# function that takes a row of our dataframe and returns the predicted probability\n",
        "def get_probability(row):\n",
        "    predicted_part = row['PREDICTED_PART']\n",
        "    probability_dict = row['PROB_DICT']\n",
        "    return probability_dict[predicted_part]\n",
        "\n",
        "# apply get_probability to each row in our dataframe and store the result\n",
        "df_valid['PREDICTED_PROB'] = df_valid.apply(func=get_probability, axis=1)\n",
        "# take a peak at what we get\n",
        "df_valid[['NARRATIVE', 'INJ_BODY_PART', 'PREDICTED_PART', 'PREDICTED_PROB']].sample(2).head()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>NARRATIVE</th>\n",
              "      <th>INJ_BODY_PART</th>\n",
              "      <th>PREDICTED_PART</th>\n",
              "      <th>PREDICTED_PROB</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>421</th>\n",
              "      <td>Hearing Loss</td>\n",
              "      <td>EAR(S) INTERNAL &amp; HEARING</td>\n",
              "      <td>EAR(S) INTERNAL &amp; HEARING</td>\n",
              "      <td>0.000004</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3249</th>\n",
              "      <td>Employee was performing routine maintenance on...</td>\n",
              "      <td>BODY SYSTEMS</td>\n",
              "      <td>BODY SYSTEMS</td>\n",
              "      <td>0.000033</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                              NARRATIVE  ... PREDICTED_PROB\n",
              "421                                        Hearing Loss  ...       0.000004\n",
              "3249  Employee was performing routine maintenance on...  ...       0.000033\n",
              "\n",
              "[2 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p6Tzjh2JBD4M",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "a950f13e-c88c-4449-c0f6-df586472cc94"
      },
      "source": [
        "from sklearn.metrics import accuracy_score, f1_score\n",
        "\n",
        "mf1 = f1_score(df_valid['INJ_BODY_PART'], df_valid['PREDICTED_PART'], average='macro')\n",
        "acc = accuracy_score(df_valid['INJ_BODY_PART'], df_valid['PREDICTED_PART'])\n",
        "print('macro-f1:', mf1)\n",
        "print('accuracy:', acc)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "macro-f1: 0.6082996613119874\n",
            "accuracy: 0.805\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
            "  'precision', 'predicted', average, warn_for)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lt6ivGRTOFBq",
        "colab_type": "text"
      },
      "source": [
        "# Semi-Supervised Learning\n",
        "\n",
        "One of the benefits of language model pretraining is that it also allows us to pretrain models on unlabeled data from our town. This often improves performance even further. We illustrate this below, assuming that we only have access to 100 labeled examples, and 18,681 unlabeled examples."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SOB35qcgOCmK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "5927d8f3-c63d-4feb-b704-0d383f01db1c"
      },
      "source": [
        "df_train = df[df['ACCIDENT_YEAR'].isin([2010, 2011])].copy()\n",
        "# grab 2 examples of each INJ_BODY_PART\n",
        "df_small_train = df_train.sample(100)\n",
        "# number of stratified examples:\n",
        "print(f'labeled examples in small train: {len(df_small_train)}')           \n",
        "print(f'total \"unlabeled\" example in big train: {len(df_train)}')"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "labeled examples in small train: 100\n",
            "total \"unlabeled\" example in big train: 18681\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uAz0X8HsawSU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "85f84ad5-3ccb-4c33-cda8-e9162c562b1e"
      },
      "source": [
        "# max_length is the maximum number of words we will use from each narrative\n",
        "model = Classifier(batch_size=32, \n",
        "                   max_length=90, \n",
        "                   n_epochs=5, \n",
        "                   val_size=0)\n",
        "# finetune the language model to our narratives (note no labels are used)\n",
        "model.fit(df_small_train['NARRATIVE'], df_small_train['INJ_BODY_PART'])"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "I0616 20:18:23.854972 139771120363392 base.py:104] Saving tensorboard output to /tmp/Finetunew0zalxzs\n",
            "Epoch 5/5: 100%|██████████| 100/100 [00:01<00:00, 51.50it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Uvm6W0StbOiV",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 190
        },
        "outputId": "60b6394e-9501-4940-e5c9-a721004e848c"
      },
      "source": [
        "# re-use the existing tensorflow graph\n",
        "with model.cached_predict():\n",
        "  # generate predictions\n",
        "  preds = model.predict(df_valid['NARRATIVE'].values)\n",
        "acc = accuracy_score(y_true=df_valid['INJ_BODY_PART'], y_pred=preds)\n",
        "mf1 = f1_score(y_true=df_valid['INJ_BODY_PART'], y_pred=preds, average='macro')\n",
        "print(f'accuracy={acc}')\n",
        "print(f'macro-f1={mf1}')"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\rInference:   0%|          | 0/1000 [00:00<?, ?it/s]/usr/local/lib/python3.6/dist-packages/finetune/encoding/input_encoder.py:182: UserWarning: Some examples are longer than the max_length. Please trim documents or increase `max_length`. Fallback behaviour is to use the first 88 byte-pair encoded tokens\n",
            "  \"Fallback behaviour is to use the first {} byte-pair encoded tokens\".format(max_length - 2)\n",
            "/usr/local/lib/python3.6/dist-packages/finetune/encoding/input_encoder.py:135: UserWarning: Document is longer than max length allowed, trimming document to 90 tokens.\n",
            "  max_length\n",
            "Inference: 100%|██████████| 1000/1000 [00:20<00:00, 49.10it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "accuracy=0.179\n",
            "macro-f1=0.014154086241981731\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
            "  'precision', 'predicted', average, warn_for)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CcSsX5LhO_Hk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "c58510ce-33f1-4808-b253-432c9182abb4"
      },
      "source": [
        "# max_length is the maximum number of words we will use from each narrative\n",
        "model = Classifier(batch_size=32, \n",
        "                   max_length=90, \n",
        "                   n_epochs=5, \n",
        "                   val_size=0,\n",
        "                   dataset_size=len(df_train['NARRATIVE']))\n",
        "# finetune the language model to our narratives (no labels are used, just narratives)\n",
        "pretrain_generator = lambda: iter(df_train['NARRATIVE'])\n",
        "model.fit(pretrain_generator)"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "I0616 20:35:04.817986 139771120363392 base.py:104] Saving tensorboard output to /tmp/Finetunequs1m68v\n",
            "Epoch 1/1: 100%|██████████| 18681/18681 [07:13<00:00, 43.14it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "00cpkqnFQwoD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.config.dataset_size = len(df_small_train['NARRATIVE'])\n",
        "model.config.n_epochs=1\n",
        "model.fit(df_small_train['NARRATIVE'], df_small_train['INJ_BODY_PART'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5zSZq2gjQ_5m",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# re-use the existing tensorflow graph\n",
        "with model.cached_predict():\n",
        "  # generate predictions\n",
        "  preds = model.predict(df_valid['NARRATIVE'].values)\n",
        "acc = accuracy_score(y_true=df_valid['INJ_BODY_PART'], y_pred=preds)\n",
        "mf1 = f1_score(y_true=df_valid['INJ_BODY_PART'], y_pred=preds, average='macro')\n",
        "print(f'accuracy={acc}')\n",
        "print(f'macro-f1={mf1}')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5QwO7V84Y6ez",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "f132b1ef-7c4c-4b1d-a975-e6a7f0b7b424"
      },
      "source": [
        "model.config.lr_schedule"
      ],
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'warmup_linear'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hy3Yrp6T4wWd",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "5cb755ff-cad6-4b42-892d-14300849ece8"
      },
      "source": [
        "model.config.lr_warmup"
      ],
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.002"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 51
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xTrWUf-m45YW",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 163
        },
        "outputId": "7965ed18-cce4-4bd2-9dc8-a4786fb23e06"
      },
      "source": [
        "model.lr"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "error",
          "ename": "AttributeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-52-5e1a445137b3>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlr\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mAttributeError\u001b[0m: 'Classifier' object has no attribute 'lr'"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dXcoP7Au4679",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}