tl_nlp/tl_nlp_allennlp.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "tl_nlp_allennlp.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "v-3PsUGNNQpG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Transfer Learning in NLP\n",
        "\n",
        "In this notebook, we will go through basics of Transfer Learning in NLP using two architectures (pretrained models) ELMo and BERT and also compare the results of GloVe embeddings with ELMo and BERT on [Twitter US Airline Sentiment dataset](https://www.kaggle.com/crowdflower/twitter-airline-sentiment).\n",
        "\n",
        "### Dataset\n",
        "\n",
        "Twitter data was scraped from February of 2015 and contributors were asked to first classify positive, negative, and neutral tweets, followed by categorizing negative reasons (such as \"late flight\" or \"rude service\").  It contains whether the sentiment of the tweets in this set was positive, neutral, or negative for six US airlines\n",
        "\n",
        "---\n",
        "\n",
        "Here we will use [Allennlp](https://github.com/allenai/allennlp  \"Allennlp Githubl\").\n",
        "\n",
        "\n",
        "Everything is explained in-detail in [blog post](https://dudeperf3ct.github.io/nlp/transfer/learning/2019/02/22/Power-of-Transfer-Learning-in-NLP/). This is notebook which replicates the result of blog and runs in colab. Enjoy!\n",
        "\n",
        "\n",
        "#### Run in Colab\n",
        "\n",
        "You can run this notebook in google colab.\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dudeperf3ct/DL_notebooks/blob/master/tl_nlp/tl_nlp_allennlp.ipynb)\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "pDBy4oPBOftc",
        "colab_type": "code",
        "outputId": "4c806113-7aee-4e33-ab11-5f9e80ac825d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 3576
        }
      },
      "cell_type": "code",
      "source": [
        "! pip install allennlp"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting allennlp\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/64/32/d6d0a93a23763f366df2dbd4e007e45ce4d2ad97e6315506db9da8af7731/allennlp-0.8.2-py3-none-any.whl (5.6MB)\n",
            "\u001b[K    100% |████████████████████████████████| 5.6MB 8.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from allennlp) (1.1.0)\n",
            "Collecting flaky (from allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/02/42/cca66659a786567c8af98587d66d75e7d2b6e65662f8daab75db708ac35b/flaky-3.5.3-py2.py3-none-any.whl\n",
            "Collecting pytz==2017.3 (from allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a3/7f/e7d1acbd433b929168a4fb4182a2ff3c33653717195a26c1de099ad1ef29/pytz-2017.3-py2.py3-none-any.whl (511kB)\n",
            "\u001b[K    100% |████████████████████████████████| 512kB 25.0MB/s \n",
            "\u001b[?25hCollecting numpydoc==0.8.0 (from allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/95/a8/b4706a6270f0475541c5c1ee3373c7a3b793936ec1f517f1a1dab4f896c0/numpydoc-0.8.0.tar.gz\n",
            "Collecting parsimonious==0.8.0 (from allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/4a/89/32c55944cd30dff856f16859ee325b13c83c260d0c56c0eed511e8063c87/parsimonious-0.8.0.tar.gz\n",
            "Collecting sqlparse==0.2.4 (from allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/65/85/20bdd72f4537cf2c4d5d005368d502b2f464ede22982e724a82c86268eda/sqlparse-0.2.4-py2.py3-none-any.whl\n",
            "Requirement already satisfied: editdistance in /usr/local/lib/python3.6/dist-packages (from allennlp) (0.5.3)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from allennlp) (0.20.3)\n",
            "Requirement already satisfied: tqdm>=4.19 in /usr/local/lib/python3.6/dist-packages (from allennlp) (4.28.1)\n",
            "Collecting jsonnet==0.10.0; sys_platform != \"win32\" (from allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/3c/83/d49904ee98dd4fbba6a003938e30e76251951c4bdb49628b4f92e5009a42/jsonnet-0.10.0.tar.gz (124kB)\n",
            "\u001b[K    100% |████████████████████████████████| 133kB 29.5MB/s \n",
            "\u001b[?25hCollecting gevent==1.3.6 (from allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/43/3d/a19fece28ba1b5133cf74bd22a229d77b4d9cc4b24aa8f263cca2845c555/gevent-1.3.6-cp36-cp36m-manylinux1_x86_64.whl (4.5MB)\n",
            "\u001b[K    100% |████████████████████████████████| 4.5MB 7.7MB/s \n",
            "\u001b[?25hRequirement already satisfied: pytest in /usr/local/lib/python3.6/dist-packages (from allennlp) (3.6.4)\n",
            "Collecting responses>=0.7 (from allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/d1/5a/b887e89925f1de7890ef298a74438371ed4ed29b33def9e6d02dc6036fd8/responses-0.10.6-py2.py3-none-any.whl\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from allennlp) (2.8.0)\n",
            "Collecting matplotlib==2.2.3 (from allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/9e/59/f235ab21bbe7b7c6570c4abf17ffb893071f4fa3b9cf557b09b60359ad9a/matplotlib-2.2.3-cp36-cp36m-manylinux1_x86_64.whl (12.6MB)\n",
            "\u001b[K    100% |████████████████████████████████| 12.6MB 3.5MB/s \n",
            "\u001b[?25hRequirement already satisfied: msgpack<0.6.0,>=0.5.6 in /usr/local/lib/python3.6/dist-packages (from allennlp) (0.5.6)\n",
            "Collecting unidecode (from allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/31/39/53096f9217b057cb049fe872b7fc7ce799a1a89b76cf917d9639e7a558b5/Unidecode-1.0.23-py2.py3-none-any.whl (237kB)\n",
            "\u001b[K    100% |████████████████████████████████| 245kB 27.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: nltk in /usr/local/lib/python3.6/dist-packages (from allennlp) (3.2.5)\n",
            "Collecting pytorch-pretrained-bert>=0.6.0 (from allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/5d/3c/d5fa084dd3a82ffc645aba78c417e6072ff48552e3301b1fa3bd711e03d4/pytorch_pretrained_bert-0.6.1-py3-none-any.whl (114kB)\n",
            "\u001b[K    100% |████████████████████████████████| 122kB 32.3MB/s \n",
            "\u001b[?25hCollecting flask-cors==3.0.7 (from allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/65/cb/683f71ff8daa3aea0a5cbb276074de39f9ab66d3fbb8ad5efb5bb83e90d2/Flask_Cors-3.0.7-py2.py3-none-any.whl\n",
            "Requirement already satisfied: flask==1.0.2 in /usr/local/lib/python3.6/dist-packages (from allennlp) (1.0.2)\n",
            "Requirement already satisfied: spacy<2.1,>=2.0 in /usr/local/lib/python3.6/dist-packages (from allennlp) (2.0.18)\n",
            "Requirement already satisfied: requests>=2.18 in /usr/local/lib/python3.6/dist-packages (from allennlp) (2.18.4)\n",
            "Collecting ftfy (from allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/8f/86/df789c5834f15ae1ca53a8d4c1fc4788676c2e32112f6a786f2625d9c6e6/ftfy-5.5.1-py3-none-any.whl (43kB)\n",
            "\u001b[K    100% |████████████████████████████████| 51kB 21.3MB/s \n",
            "\u001b[?25hCollecting moto==1.3.4 (from allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ee/8f/7b36e81ff067d0e7bf90f7210b351c0cfe6657f79fa4dcb0cb4787462e05/moto-1.3.4-py2.py3-none-any.whl (548kB)\n",
            "\u001b[K    100% |████████████████████████████████| 552kB 24.7MB/s \n",
            "\u001b[?25hCollecting awscli>=1.11.91 (from allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/58/fe/1dfe7e1cf92a80835086d4e6ac6bcedd4543b74626e3085c23233cae4320/awscli-1.16.130-py2.py3-none-any.whl (1.5MB)\n",
            "\u001b[K    100% |████████████████████████████████| 1.5MB 10.9MB/s \n",
            "\u001b[?25hCollecting conllu==0.11 (from allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/d4/2c/856344d9b69baf5b374c395b4286626181a80f0c2b2f704914d18a1cea47/conllu-0.11-py2.py3-none-any.whl\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from allennlp) (1.14.6)\n",
            "Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from allennlp) (1.9.115)\n",
            "Collecting overrides (from allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/de/55/3100c6d14c1ed177492fcf8f07c4a7d2d6c996c0a7fc6a9a0a41308e7eec/overrides-1.9.tar.gz\n",
            "Requirement already satisfied: torch>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from allennlp) (1.0.1.post2)\n",
            "Collecting tensorboardX==1.2 (from allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/c5/22/43f4f0318f7c68a1000dbb700a353b745584bc2397437832d15ba69ea5f1/tensorboardX-1.2-py2.py3-none-any.whl (44kB)\n",
            "\u001b[K    100% |████████████████████████████████| 51kB 21.5MB/s \n",
            "\u001b[?25hRequirement already satisfied: sphinx>=1.2.3 in /usr/local/lib/python3.6/dist-packages (from numpydoc==0.8.0->allennlp) (1.8.5)\n",
            "Requirement already satisfied: Jinja2>=2.3 in /usr/local/lib/python3.6/dist-packages (from numpydoc==0.8.0->allennlp) (2.10)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from parsimonious==0.8.0->allennlp) (1.11.0)\n",
            "Requirement already satisfied: greenlet>=0.4.14; platform_python_implementation == \"CPython\" in /usr/local/lib/python3.6/dist-packages (from gevent==1.3.6->allennlp) (0.4.15)\n",
            "Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp) (6.0.0)\n",
            "Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp) (1.3.0)\n",
            "Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp) (19.1.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp) (40.8.0)\n",
            "Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp) (1.8.0)\n",
            "Requirement already satisfied: pluggy<0.8,>=0.5 in /usr/local/lib/python3.6/dist-packages (from pytest->allennlp) (0.7.1)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib==2.2.3->allennlp) (2.5.3)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib==2.2.3->allennlp) (2.3.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib==2.2.3->allennlp) (0.10.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib==2.2.3->allennlp) (1.0.1)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert>=0.6.0->allennlp) (2018.1.10)\n",
            "Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.6/dist-packages (from flask==1.0.2->allennlp) (1.1.0)\n",
            "Requirement already satisfied: Werkzeug>=0.14 in /usr/local/lib/python3.6/dist-packages (from flask==1.0.2->allennlp) (0.14.1)\n",
            "Requirement already satisfied: click>=5.1 in /usr/local/lib/python3.6/dist-packages (from flask==1.0.2->allennlp) (7.0)\n",
            "Requirement already satisfied: plac<1.0.0,>=0.9.6 in /usr/local/lib/python3.6/dist-packages (from spacy<2.1,>=2.0->allennlp) (0.9.6)\n",
            "Requirement already satisfied: thinc<6.13.0,>=6.12.1 in /usr/local/lib/python3.6/dist-packages (from spacy<2.1,>=2.0->allennlp) (6.12.1)\n",
            "Requirement already satisfied: dill<0.3,>=0.2 in /usr/local/lib/python3.6/dist-packages (from spacy<2.1,>=2.0->allennlp) (0.2.9)\n",
            "Requirement already satisfied: ujson>=1.35 in /usr/local/lib/python3.6/dist-packages (from spacy<2.1,>=2.0->allennlp) (1.35)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.6/dist-packages (from spacy<2.1,>=2.0->allennlp) (1.0.2)\n",
            "Requirement already satisfied: preshed<2.1.0,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from spacy<2.1,>=2.0->allennlp) (2.0.1)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy<2.1,>=2.0->allennlp) (2.0.2)\n",
            "Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.18->allennlp) (1.22)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.18->allennlp) (2019.3.9)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.18->allennlp) (3.0.4)\n",
            "Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.18->allennlp) (2.6)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/dist-packages (from ftfy->allennlp) (0.1.7)\n",
            "Requirement already satisfied: boto>=2.36.0 in /usr/local/lib/python3.6/dist-packages (from moto==1.3.4->allennlp) (2.49.0)\n",
            "Collecting cookies (from moto==1.3.4->allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/6a/60/557f84aa2db629e5124aa05408b975b1b5d0e1cec16cde0bfa06aae097d3/cookies-2.2.1-py2.py3-none-any.whl (44kB)\n",
            "\u001b[K    100% |████████████████████████████████| 51kB 21.6MB/s \n",
            "\u001b[?25hCollecting docker>=2.5.1 (from moto==1.3.4->allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/fb/d8/8242b8fb3bd3000274fbf5ac1a06cdba8a5ccbcf4e2a8c05f0ab37999fd8/docker-3.7.1-py2.py3-none-any.whl (134kB)\n",
            "\u001b[K    100% |████████████████████████████████| 143kB 31.9MB/s \n",
            "\u001b[?25hCollecting jsondiff==1.1.1 (from moto==1.3.4->allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/bd/5f/13e28a2f9abeda2ffb3f44f2f809b01b52bc02cdb63816e05b8c9cbbdfc5/jsondiff-1.1.1.tar.gz\n",
            "Collecting aws-xray-sdk<0.96,>=0.93 (from moto==1.3.4->allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a4/a5/da7887285564f9e0ae5cd25a453cca36e2cd43d8ccc9effde260b4d80904/aws_xray_sdk-0.95-py2.py3-none-any.whl (52kB)\n",
            "\u001b[K    100% |████████████████████████████████| 61kB 21.4MB/s \n",
            "\u001b[?25hCollecting xmltodict (from moto==1.3.4->allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/28/fd/30d5c1d3ac29ce229f6bdc40bbc20b28f716e8b363140c26eff19122d8a5/xmltodict-0.12.0-py2.py3-none-any.whl\n",
            "Collecting python-jose<3.0.0 (from moto==1.3.4->allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/bf/5c/5fa238c0c5b0656994b52721dd8b1d7bf52ebd8786518dde794f44de86b6/python_jose-2.0.2-py2.py3-none-any.whl\n",
            "Collecting pyaml (from moto==1.3.4->allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/c5/e1/1523fb1dab744e2c6b1f02446f2139a78726c18c062a8ddd53875abb20f8/pyaml-18.11.0-py2.py3-none-any.whl\n",
            "Requirement already satisfied: botocore>=1.9.16 in /usr/local/lib/python3.6/dist-packages (from moto==1.3.4->allennlp) (1.12.115)\n",
            "Requirement already satisfied: mock in /usr/local/lib/python3.6/dist-packages (from moto==1.3.4->allennlp) (2.0.0)\n",
            "Collecting cryptography>=2.0.0 (from moto==1.3.4->allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/5b/12/b0409a94dad366d98a8eee2a77678c7a73aafd8c0e4b835abea634ea3896/cryptography-2.6.1-cp34-abi3-manylinux1_x86_64.whl (2.3MB)\n",
            "\u001b[K    100% |████████████████████████████████| 2.3MB 13.9MB/s \n",
            "\u001b[?25hCollecting colorama<=0.3.9,>=0.2.5 (from awscli>=1.11.91->allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl\n",
            "Collecting rsa<=3.5.0,>=3.1.2 (from awscli>=1.11.91->allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl (46kB)\n",
            "\u001b[K    100% |████████████████████████████████| 51kB 21.8MB/s \n",
            "\u001b[?25hRequirement already satisfied: s3transfer<0.3.0,>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from awscli>=1.11.91->allennlp) (0.2.0)\n",
            "Requirement already satisfied: PyYAML<=3.13,>=3.10 in /usr/local/lib/python3.6/dist-packages (from awscli>=1.11.91->allennlp) (3.13)\n",
            "Requirement already satisfied: docutils>=0.10 in /usr/local/lib/python3.6/dist-packages (from awscli>=1.11.91->allennlp) (0.14)\n",
            "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->allennlp) (0.9.4)\n",
            "Requirement already satisfied: protobuf>=0.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorboardX==1.2->allennlp) (3.7.0)\n",
            "Requirement already satisfied: alabaster<0.8,>=0.7 in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.2.3->numpydoc==0.8.0->allennlp) (0.7.12)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.2.3->numpydoc==0.8.0->allennlp) (19.0)\n",
            "Requirement already satisfied: Pygments>=2.0 in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.2.3->numpydoc==0.8.0->allennlp) (2.1.3)\n",
            "Requirement already satisfied: snowballstemmer>=1.1 in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.2.3->numpydoc==0.8.0->allennlp) (1.2.1)\n",
            "Requirement already satisfied: sphinxcontrib-websupport in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.2.3->numpydoc==0.8.0->allennlp) (1.1.0)\n",
            "Requirement already satisfied: imagesize in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.2.3->numpydoc==0.8.0->allennlp) (1.1.0)\n",
            "Requirement already satisfied: babel!=2.0,>=1.3 in /usr/local/lib/python3.6/dist-packages (from sphinx>=1.2.3->numpydoc==0.8.0->allennlp) (2.6.0)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.6/dist-packages (from Jinja2>=2.3->numpydoc==0.8.0->allennlp) (1.1.1)\n",
            "Requirement already satisfied: wrapt<1.11.0,>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from thinc<6.13.0,>=6.12.1->spacy<2.1,>=2.0->allennlp) (1.10.11)\n",
            "Requirement already satisfied: msgpack-numpy<0.4.4 in /usr/local/lib/python3.6/dist-packages (from thinc<6.13.0,>=6.12.1->spacy<2.1,>=2.0->allennlp) (0.4.3.2)\n",
            "Requirement already satisfied: cytoolz<0.10,>=0.9.0 in /usr/local/lib/python3.6/dist-packages (from thinc<6.13.0,>=6.12.1->spacy<2.1,>=2.0->allennlp) (0.9.0.1)\n",
            "Collecting docker-pycreds>=0.4.0 (from docker>=2.5.1->moto==1.3.4->allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/f5/e8/f6bd1eee09314e7e6dee49cbe2c5e22314ccdb38db16c9fc72d2fa80d054/docker_pycreds-0.4.0-py2.py3-none-any.whl\n",
            "Collecting websocket-client>=0.32.0 (from docker>=2.5.1->moto==1.3.4->allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/29/19/44753eab1fdb50770ac69605527e8859468f3c0fd7dc5a76dd9c4dbd7906/websocket_client-0.56.0-py2.py3-none-any.whl (200kB)\n",
            "\u001b[K    100% |████████████████████████████████| 204kB 31.8MB/s \n",
            "\u001b[?25hCollecting jsonpickle (from aws-xray-sdk<0.96,>=0.93->moto==1.3.4->allennlp)\n",
            "  Downloading https://files.pythonhosted.org/packages/dc/12/8c44eabb501e2bc0aec0dd152b328074d98a50968d3a02be28f6037f0c6a/jsonpickle-1.1-py2.py3-none-any.whl\n",
            "Collecting ecdsa<1.0 (from python-jose<3.0.0->moto==1.3.4->allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/63/f4/73669d51825516ce8c43b816c0a6b64cd6eb71d08b99820c00792cb42222/ecdsa-0.13-py2.py3-none-any.whl (86kB)\n",
            "\u001b[K    100% |████████████████████████████████| 92kB 26.7MB/s \n",
            "\u001b[?25hCollecting pycryptodome<4.0.0,>=3.3.1 (from python-jose<3.0.0->moto==1.3.4->allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/4e/b4/7d007370568d98822c833e0d0e804417620865710dc8a5830bbed58328d1/pycryptodome-3.8.0-cp36-cp36m-manylinux1_x86_64.whl (9.7MB)\n",
            "\u001b[K    100% |████████████████████████████████| 9.7MB 3.7MB/s \n",
            "\u001b[?25hRequirement already satisfied: future<1.0 in /usr/local/lib/python3.6/dist-packages (from python-jose<3.0.0->moto==1.3.4->allennlp) (0.16.0)\n",
            "Requirement already satisfied: pbr>=0.11 in /usr/local/lib/python3.6/dist-packages (from mock->moto==1.3.4->allennlp) (5.1.3)\n",
            "Collecting asn1crypto>=0.21.0 (from cryptography>=2.0.0->moto==1.3.4->allennlp)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)\n",
            "\u001b[K    100% |████████████████████████████████| 102kB 28.6MB/s \n",
            "\u001b[?25hRequirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/local/lib/python3.6/dist-packages (from cryptography>=2.0.0->moto==1.3.4->allennlp) (1.12.2)\n",
            "Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python3.6/dist-packages (from rsa<=3.5.0,>=3.1.2->awscli>=1.11.91->allennlp) (0.4.5)\n",
            "Requirement already satisfied: toolz>=0.8.0 in /usr/local/lib/python3.6/dist-packages (from cytoolz<0.10,>=0.9.0->thinc<6.13.0,>=6.12.1->spacy<2.1,>=2.0->allennlp) (0.9.0)\n",
            "Requirement already satisfied: pycparser in /usr/local/lib/python3.6/dist-packages (from cffi!=1.11.3,>=1.8->cryptography>=2.0.0->moto==1.3.4->allennlp) (2.19)\n",
            "Building wheels for collected packages: numpydoc, parsimonious, jsonnet, overrides, jsondiff\n",
            "  Building wheel for numpydoc (setup.py) ... \u001b[?25ldone\n",
            "\u001b[?25h  Stored in directory: /root/.cache/pip/wheels/ea/55/7f/3e25d754760ccd62d6796e5b2cfe25629346f52ea00753d549\n",
            "  Building wheel for parsimonious (setup.py) ... \u001b[?25ldone\n",
            "\u001b[?25h  Stored in directory: /root/.cache/pip/wheels/bb/51/82/ae9b22a790f11e7be918939d01aa397c545ebb3723453c5fb4\n",
            "  Building wheel for jsonnet (setup.py) ... \u001b[?25ldone\n",
            "\u001b[?25h  Stored in directory: /root/.cache/pip/wheels/a0/aa/f0/b4ab8854cf00f922a87787425cfbb789aac01ab2c2cd1b4ca4\n",
            "  Building wheel for overrides (setup.py) ... \u001b[?25ldone\n",
            "\u001b[?25h  Stored in directory: /root/.cache/pip/wheels/8d/52/86/e5a83b1797e7d263b458d2334edd2704c78508b3eea9323718\n",
            "  Building wheel for jsondiff (setup.py) ... \u001b[?25ldone\n",
            "\u001b[?25h  Stored in directory: /root/.cache/pip/wheels/68/08/07/69d839606fb7fdc778fa86476abc0a864693d45969a0c1936c\n",
            "Successfully built numpydoc parsimonious jsonnet overrides jsondiff\n",
            "\u001b[31mimgaug 0.2.8 has requirement numpy>=1.15.0, but you'll have numpy 1.14.6 which is incompatible.\u001b[0m\n",
            "\u001b[31mfeaturetools 0.4.1 has requirement pandas>=0.23.0, but you'll have pandas 0.22.0 which is incompatible.\u001b[0m\n",
            "\u001b[31mfastai 1.0.49 has requirement numpy>=1.15, but you'll have numpy 1.14.6 which is incompatible.\u001b[0m\n",
            "\u001b[31malbumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.8 which is incompatible.\u001b[0m\n",
            "\u001b[31mawscli 1.16.130 has requirement botocore==1.12.120, but you'll have botocore 1.12.115 which is incompatible.\u001b[0m\n",
            "Installing collected packages: flaky, pytz, numpydoc, parsimonious, sqlparse, jsonnet, gevent, responses, matplotlib, unidecode, pytorch-pretrained-bert, flask-cors, ftfy, cookies, docker-pycreds, websocket-client, docker, jsondiff, jsonpickle, aws-xray-sdk, xmltodict, ecdsa, pycryptodome, python-jose, pyaml, asn1crypto, cryptography, moto, colorama, rsa, awscli, conllu, overrides, tensorboardX, allennlp\n",
            "  Found existing installation: pytz 2018.9\n",
            "    Uninstalling pytz-2018.9:\n",
            "      Successfully uninstalled pytz-2018.9\n",
            "  Found existing installation: sqlparse 0.3.0\n",
            "    Uninstalling sqlparse-0.3.0:\n",
            "      Successfully uninstalled sqlparse-0.3.0\n",
            "  Found existing installation: gevent 1.4.0\n",
            "    Uninstalling gevent-1.4.0:\n",
            "      Successfully uninstalled gevent-1.4.0\n",
            "  Found existing installation: matplotlib 3.0.3\n",
            "    Uninstalling matplotlib-3.0.3:\n",
            "      Successfully uninstalled matplotlib-3.0.3\n",
            "  Found existing installation: rsa 4.0\n",
            "    Uninstalling rsa-4.0:\n",
            "      Successfully uninstalled rsa-4.0\n",
            "Successfully installed allennlp-0.8.2 asn1crypto-0.24.0 aws-xray-sdk-0.95 awscli-1.16.130 colorama-0.3.9 conllu-0.11 cookies-2.2.1 cryptography-2.6.1 docker-3.7.1 docker-pycreds-0.4.0 ecdsa-0.13 flaky-3.5.3 flask-cors-3.0.7 ftfy-5.5.1 gevent-1.3.6 jsondiff-1.1.1 jsonnet-0.10.0 jsonpickle-1.1 matplotlib-2.2.3 moto-1.3.4 numpydoc-0.8.0 overrides-1.9 parsimonious-0.8.0 pyaml-18.11.0 pycryptodome-3.8.0 python-jose-2.0.2 pytorch-pretrained-bert-0.6.1 pytz-2017.3 responses-0.10.6 rsa-3.4.2 sqlparse-0.2.4 tensorboardX-1.2 unidecode-1.0.23 websocket-client-0.56.0 xmltodict-0.12.0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "matplotlib",
                  "mpl_toolkits",
                  "pytz",
                  "rsa"
                ]
              }
            }
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "UrHyuxNgRwXV",
        "colab_type": "code",
        "outputId": "8782cf63-d353-4474-8d94-163442b747c6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 323
        }
      },
      "cell_type": "code",
      "source": [
        "! python -m spacy download en\n",
        "! python -m spacy download en_core_web_md"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: en_core_web_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz#egg=en_core_web_sm==2.0.0 in /usr/local/lib/python3.6/dist-packages (2.0.0)\n",
            "\n",
            "\u001b[93m    Linking successful\u001b[0m\n",
            "    /usr/local/lib/python3.6/dist-packages/en_core_web_sm -->\n",
            "    /usr/local/lib/python3.6/dist-packages/spacy/data/en\n",
            "\n",
            "    You can now load the model via spacy.load('en')\n",
            "\n",
            "Requirement already satisfied: en_core_web_md==2.0.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_md-2.0.0/en_core_web_md-2.0.0.tar.gz#egg=en_core_web_md==2.0.0 in /usr/local/lib/python3.6/dist-packages (2.0.0)\n",
            "\n",
            "\u001b[93m    Linking successful\u001b[0m\n",
            "    /usr/local/lib/python3.6/dist-packages/en_core_web_md -->\n",
            "    /usr/local/lib/python3.6/dist-packages/spacy/data/en_core_web_md\n",
            "\n",
            "    You can now load the model via spacy.load('en_core_web_md')\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "CrfKWMPPLjfc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Twitter Sentiment Data\n",
        "\n",
        "Code Adapted from : [Link](https://github.com/keitakurita/Practical_NLP_in_PyTorch)\n",
        "\n",
        "Paper ELMo : [Link](https://arxiv.org/pdf/1802.05365.pdf)\n",
        "\n",
        "Paper BERT: [Link](https://arxiv.org/pdf/1810.04805.pdf)\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "nmROOTrsNK3R",
        "colab_type": "code",
        "outputId": "95e196b0-472b-4686-8d5b-4e48fdb2c247",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "from pathlib import Path\n",
        "from typing import *\n",
        "import os\n",
        "import torch\n",
        "import torch.optim as optim\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from functools import partial\n",
        "from overrides import overrides\n",
        "\n",
        "from allennlp.data import Instance\n",
        "from allennlp.data.token_indexers import TokenIndexer\n",
        "from allennlp.data.tokenizers import Token\n",
        "from allennlp.nn import util as nn_util\n",
        "from allennlp.common.checks import ConfigurationError\n",
        "\n",
        "USE_GPU = torch.cuda.is_available()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "tocRrPaaUzTr",
        "outputId": "95195ea6-45c6-4a85-a2b0-b380ae8719d9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "cell_type": "code",
      "source": [
        "df = pd.read_csv('https://query.data.world/s/hus7zihvuo5vt65cnv4fcfn2ppfj6y', encoding = \"ISO-8859-1\")\n",
        "df = df[[\"airline_sentiment\", \"text\"]]\n",
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>airline_sentiment</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>neutral</td>\n",
              "      <td>@VirginAmerica What @dhepburn said.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>positive</td>\n",
              "      <td>@VirginAmerica plus you've added commercials t...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>neutral</td>\n",
              "      <td>@VirginAmerica I didn't today... Must mean I n...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>negative</td>\n",
              "      <td>@VirginAmerica it's really aggressive to blast...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>negative</td>\n",
              "      <td>@VirginAmerica and it's a really big bad thing...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  airline_sentiment                                               text\n",
              "0           neutral                @VirginAmerica What @dhepburn said.\n",
              "1          positive  @VirginAmerica plus you've added commercials t...\n",
              "2           neutral  @VirginAmerica I didn't today... Must mean I n...\n",
              "3          negative  @VirginAmerica it's really aggressive to blast...\n",
              "4          negative  @VirginAmerica and it's a really big bad thing..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "metadata": {
        "id": "mb1y-2T0Vzm7",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "df['positive'] = df.apply(lambda row: 1 if row['airline_sentiment'] == 'positive' else 0, axis=1)\n",
        "df['negative'] = df.apply(lambda row: 1 if row['airline_sentiment'] == 'negative' else 0, axis=1)\n",
        "df['neutral'] = df.apply(lambda row: 1 if row['airline_sentiment'] == 'neutral' else 0, axis=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "nY_zhIFZVzsE",
        "colab_type": "code",
        "outputId": "817350cc-2563-4ef8-86c5-6cbe1c2e50f0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "cell_type": "code",
      "source": [
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>airline_sentiment</th>\n",
              "      <th>text</th>\n",
              "      <th>positive</th>\n",
              "      <th>negative</th>\n",
              "      <th>neutral</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>neutral</td>\n",
              "      <td>@VirginAmerica What @dhepburn said.</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>positive</td>\n",
              "      <td>@VirginAmerica plus you've added commercials t...</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>neutral</td>\n",
              "      <td>@VirginAmerica I didn't today... Must mean I n...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>negative</td>\n",
              "      <td>@VirginAmerica it's really aggressive to blast...</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>negative</td>\n",
              "      <td>@VirginAmerica and it's a really big bad thing...</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  airline_sentiment                                               text  \\\n",
              "0           neutral                @VirginAmerica What @dhepburn said.   \n",
              "1          positive  @VirginAmerica plus you've added commercials t...   \n",
              "2           neutral  @VirginAmerica I didn't today... Must mean I n...   \n",
              "3          negative  @VirginAmerica it's really aggressive to blast...   \n",
              "4          negative  @VirginAmerica and it's a really big bad thing...   \n",
              "\n",
              "   positive  negative  neutral  \n",
              "0         0         0        1  \n",
              "1         1         0        0  \n",
              "2         0         0        1  \n",
              "3         0         1        0  \n",
              "4         0         1        0  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "metadata": {
        "id": "cZ6GJQP2amGE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#os.mkdir('data/')\n",
        "df.to_csv('data/train.csv', index=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "xWJVOsTHNO74",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class Config(dict):\n",
        "    def __init__(self, **kwargs):\n",
        "        super().__init__(**kwargs)\n",
        "        for k, v in kwargs.items():\n",
        "            setattr(self, k, v)\n",
        "    \n",
        "    def set(self, key, val):\n",
        "        self[key] = val\n",
        "        setattr(self, key, val)\n",
        "        \n",
        "config = Config(\n",
        "    testing=False,\n",
        "    seed=1,\n",
        "    batch_size=64,\n",
        "    lr=3e-4,\n",
        "    epochs=10,\n",
        "    hidden_sz=64,\n",
        "    max_seq_len=100, # necessary to limit memory usage\n",
        "    max_vocab_size=10000,\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "AWEsfzKbNPEw",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "torch.manual_seed(config.seed)\n",
        "DATA_ROOT = Path(\"data\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qe6bDJo-PPcN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Load Data"
      ]
    },
    {
      "metadata": {
        "id": "1ux5_-oyTld1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Prepare Dataset"
      ]
    },
    {
      "metadata": {
        "id": "97kO5UV5NPKK",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from allennlp.data.vocabulary import Vocabulary\n",
        "from allennlp.data.dataset_readers import DatasetReader"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "gEFPAZf7NPM1",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "label_cols = [\"negative\", \"neutral\", \"positive\"]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "LgTrDjOkPcED",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from allennlp.data.fields import TextField, MetadataField, ArrayField\n",
        "\n",
        "class SentimentDatasetReader(DatasetReader):\n",
        "    def __init__(self, tokenizer: Callable[[str], List[str]]=lambda x: x.split(),\n",
        "                 token_indexers: Dict[str, TokenIndexer] = None,\n",
        "                 max_seq_len: Optional[int]=config.max_seq_len) -> None:\n",
        "        super().__init__(lazy=False)\n",
        "        self.tokenizer = tokenizer\n",
        "        self.token_indexers = token_indexers or {\"tokens\": SingleIdTokenIndexer()}\n",
        "        self.max_seq_len = max_seq_len\n",
        "\n",
        "    @overrides\n",
        "    def text_to_instance(self, tokens: List[Token], id: str=None, labels: np.ndarray=None) -> Instance:\n",
        "        sentence_field = TextField(tokens, self.token_indexers)\n",
        "        fields = {\"tokens\": sentence_field}\n",
        "        \n",
        "        id_field = MetadataField(id)\n",
        "        fields[\"id\"] = id_field\n",
        "        \n",
        "        if labels is None:\n",
        "            labels = np.zeros(len(label_cols))\n",
        "        label_field = ArrayField(array=labels)\n",
        "        fields[\"label\"] = label_field\n",
        "\n",
        "        return Instance(fields)\n",
        "    \n",
        "    @overrides\n",
        "    def _read(self, file_path: str) -> Iterator[Instance]:\n",
        "        df = pd.read_csv(file_path)\n",
        "        if config.testing: df = df.head(1000)\n",
        "        for i, row in df.iterrows():\n",
        "            yield self.text_to_instance([Token(x) for x in self.tokenizer(row[\"text\"])], None, row[label_cols].values)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "vmHy2QCLPdHS",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "or3jp4-UhL5Y",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## GloVe"
      ]
    },
    {
      "metadata": {
        "id": "qAgVb-eBPdMr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from allennlp.data.tokenizers.word_splitter import SpacyWordSplitter\n",
        "from allennlp.data.token_indexers import SingleIdTokenIndexer\n",
        "\n",
        "# the token indexer is responsible for mapping tokens to integers\n",
        "token_indexer = SingleIdTokenIndexer()\n",
        "\n",
        "def tokenizer(x: str):\n",
        "    return [w.text for w in SpacyWordSplitter(language='en_core_web_sm', pos_tags=False).split_words(x)[:config.max_seq_len]]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "YdAHm3opPdPt",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "reader = SentimentDatasetReader(\n",
        "    tokenizer=tokenizer,\n",
        "    token_indexers={\"tokens\": token_indexer}\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "7-8ytnhiPdKL",
        "colab_type": "code",
        "outputId": "31a1642c-1bf3-480d-f20c-95c22d555fcd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "train_ds = reader.read(DATA_ROOT / \"train.csv\")\n",
        "val_ds = None"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "14640it [00:19, 735.51it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "VSeQ66dnSEPn",
        "colab_type": "code",
        "outputId": "dce5e3c2-a953-4d72-847e-2153ccde8710",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "len(train_ds)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "14640"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "metadata": {
        "id": "SDXp5oW1TPq6",
        "colab_type": "code",
        "outputId": "bdc60489-a1a4-4ecc-9c81-ce5832138999",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        }
      },
      "cell_type": "code",
      "source": [
        "vars(train_ds[0].fields[\"tokens\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'_indexed_tokens': None,\n",
              " '_indexer_name_to_indexed_token': None,\n",
              " '_token_indexers': {'tokens': <allennlp.data.token_indexers.single_id_token_indexer.SingleIdTokenIndexer at 0x7f42d587d7b8>},\n",
              " 'tokens': [@VirginAmerica, What, @dhepburn, said, .]}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "metadata": {
        "id": "SWRDeRR6cH9t",
        "colab_type": "code",
        "outputId": "97d69a40-5952-40bd-ba0a-b48ebf82c851",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "vars(train_ds[0].fields[\"label\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'array': array([0, 1, 0], dtype=object), 'padding_value': 0}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "metadata": {
        "id": "RKMzs8OxToJB",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Prepare Vocabulary"
      ]
    },
    {
      "metadata": {
        "id": "2Ew--iSZTqEs",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "vocab = Vocabulary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "z4v8onskTrrR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Prepare Iterator"
      ]
    },
    {
      "metadata": {
        "id": "R03ami5GT5L_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from allennlp.data.iterators import BucketIterator"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "03eh_PJbTqsY",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "iterator = BucketIterator(batch_size=config.batch_size, \n",
        "                          sorting_keys=[(\"tokens\", \"num_tokens\")],\n",
        "                         )"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "1PRSF-foTqx-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "iterator.index_with(vocab)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FdMktcvUTq0y",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "iIkCbmOeT9Rj",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Read Sample"
      ]
    },
    {
      "metadata": {
        "id": "940qxbRkT-hv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "batch = next(iter(iterator(train_ds)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "zG8fAYMsT_AL",
        "colab_type": "code",
        "outputId": "addf1566-6d05-4380-d907-feb6d0015454",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 2429
        }
      },
      "cell_type": "code",
      "source": [
        "batch"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'id': [None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None],\n",
              " 'label': tensor([[1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 1., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 1., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [0., 1., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [0., 1., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 1., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [0., 0., 1.],\n",
              "         [0., 1., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [0., 1., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.]]),\n",
              " 'tokens': {'tokens': tensor([[1, 1, 1,  ..., 1, 1, 1],\n",
              "          [1, 1, 1,  ..., 1, 1, 0],\n",
              "          [1, 1, 1,  ..., 0, 0, 0],\n",
              "          ...,\n",
              "          [1, 1, 1,  ..., 1, 1, 1],\n",
              "          [1, 1, 1,  ..., 1, 1, 1],\n",
              "          [1, 1, 1,  ..., 0, 0, 0]])}}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "metadata": {
        "id": "Z5ISZOIGcO3i",
        "colab_type": "code",
        "outputId": "df1b1cd2-65b6-4dda-e875-c9bfa5323bac",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "batch[\"tokens\"][\"tokens\"].shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([64, 28])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "metadata": {
        "id": "9cGBIM95ceqE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Prepare Model"
      ]
    },
    {
      "metadata": {
        "id": "u748ckpWcRRY",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "\n",
        "from allennlp.modules.seq2vec_encoders import Seq2VecEncoder, PytorchSeq2VecWrapper\n",
        "from allennlp.nn.util import get_text_field_mask\n",
        "from allennlp.models import Model\n",
        "from allennlp.modules.text_field_embedders import TextFieldEmbedder"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "oYgAshAMceIH",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class BaselineModel(Model):\n",
        "    def __init__(self, word_embeddings: TextFieldEmbedder,\n",
        "                 encoder: Seq2VecEncoder,\n",
        "                 out_sz: int=len(label_cols)):\n",
        "        super().__init__(vocab)\n",
        "        self.word_embeddings = word_embeddings\n",
        "        self.encoder = encoder\n",
        "        self.projection = nn.Linear(self.encoder.get_output_dim(), out_sz)\n",
        "        self.loss = nn.BCEWithLogitsLoss()\n",
        "        \n",
        "    def forward(self, tokens: Dict[str, torch.Tensor],\n",
        "                id: Any, label: torch.Tensor) -> torch.Tensor:\n",
        "        mask = get_text_field_mask(tokens)\n",
        "        embeddings = self.word_embeddings(tokens)\n",
        "        state = self.encoder(embeddings, mask)\n",
        "        class_logits = self.projection(state)\n",
        "        \n",
        "        output = {\"class_logits\": class_logits}\n",
        "        output[\"loss\"] = self.loss(class_logits, label)\n",
        "\n",
        "        return output"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FuQDJHLxceSF",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Uq1V6_P6crac",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Prepare Embeddings"
      ]
    },
    {
      "metadata": {
        "id": "HOVPXuWGcePD",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from allennlp.modules.token_embedders import Embedding\n",
        "from allennlp.modules.text_field_embedders import BasicTextFieldEmbedder\n",
        "\n",
        "token_embedding = Embedding(num_embeddings=config.max_vocab_size + 2,\n",
        "                            embedding_dim=300, padding_index=0)\n",
        "# the embedder maps the input tokens to the appropriate embedding matrix\n",
        "word_embeddings: TextFieldEmbedder = BasicTextFieldEmbedder({\"tokens\": token_embedding})"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FCz-pkkcceMK",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from allennlp.modules.seq2vec_encoders import PytorchSeq2VecWrapper\n",
        "encoder: Seq2VecEncoder = PytorchSeq2VecWrapper(nn.LSTM(word_embeddings.get_output_dim(),\n",
        "                                                        config.hidden_sz, bidirectional=True, batch_first=True))\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "dIC_1JrsceDt",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model = BaselineModel(\n",
        "    word_embeddings, \n",
        "    encoder, \n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "LfzbD7V-c1Ox",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "if USE_GPU: model.cuda()\n",
        "else: model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "-WjNYh0BdS6w",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Train"
      ]
    },
    {
      "metadata": {
        "id": "iqaa1ZgNdIzy",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "optimizer = optim.Adam(model.parameters(), lr=config.lr)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "DmitmJTedSYt",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from allennlp.training.trainer import Trainer\n",
        "\n",
        "trainer = Trainer(\n",
        "    model=model,\n",
        "    optimizer=optimizer,\n",
        "    iterator=iterator,\n",
        "    train_dataset=train_ds,\n",
        "    cuda_device=0 if USE_GPU else -1,\n",
        "    num_epochs=config.epochs,\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "snEJd_oydSTo",
        "colab_type": "code",
        "outputId": "b1b05d81-b420-4706-fc31-d0ceaa5449a0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 196
        }
      },
      "cell_type": "code",
      "source": [
        "metrics = trainer.train()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "loss: 0.5522 ||: 100%|██████████| 229/229 [00:04<00:00, 46.38it/s]\n",
            "loss: 0.5034 ||: 100%|██████████| 229/229 [00:04<00:00, 49.46it/s]\n",
            "loss: 0.5012 ||: 100%|██████████| 229/229 [00:04<00:00, 50.00it/s]\n",
            "loss: 0.5008 ||: 100%|██████████| 229/229 [00:04<00:00, 50.04it/s]\n",
            "loss: 0.5013 ||: 100%|██████████| 229/229 [00:04<00:00, 47.44it/s]\n",
            "loss: 0.5006 ||: 100%|██████████| 229/229 [00:04<00:00, 57.80it/s]\n",
            "loss: 0.5004 ||: 100%|██████████| 229/229 [00:04<00:00, 50.30it/s]\n",
            "loss: 0.5008 ||: 100%|██████████| 229/229 [00:04<00:00, 50.60it/s]\n",
            "loss: 0.4999 ||: 100%|██████████| 229/229 [00:04<00:00, 55.95it/s]\n",
            "loss: 0.4996 ||: 100%|██████████| 229/229 [00:04<00:00, 50.54it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "2RYbGCqHdchT",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "92MOsnXxsQMp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Predictions"
      ]
    },
    {
      "metadata": {
        "id": "2VdVH92FsSRB",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from allennlp.predictors.sentence_tagger import SentenceTaggerPredictor"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "WzqYS4dEsSL9",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "tagger = SentenceTaggerPredictor(model, reader)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "05DHopXwsSFx",
        "colab_type": "code",
        "outputId": "a8469b03-efb7-47d5-fdca-493730a107e6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        }
      },
      "cell_type": "code",
      "source": [
        "tagger.predict(\"Bad Service, utter disaster!\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'class_logits': [-0.9477578997612, -0.3868182301521301, -0.7099025249481201],\n",
              " 'loss': 0.41527312994003296}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "Po9wuzw-rZ9x"
      },
      "cell_type": "markdown",
      "source": [
        "## ELMo"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "cU-g0PQ-rZ9_",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from allennlp.data.tokenizers.word_splitter import SpacyWordSplitter\n",
        "from allennlp.data.token_indexers.elmo_indexer import ELMoCharacterMapper, ELMoTokenCharactersIndexer\n",
        "\n",
        "# the token indexer is responsible for mapping tokens to integers\n",
        "token_indexer = ELMoTokenCharactersIndexer()\n",
        "\n",
        "def tokenizer(x: str):\n",
        "    return [w.text for w in SpacyWordSplitter(language='en_core_web_sm', pos_tags=False).split_words(x)[:config.max_seq_len]]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "s8p-gJ-9rZ-P",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "reader = SentimentDatasetReader(\n",
        "    tokenizer=tokenizer,\n",
        "    token_indexers={\"tokens\": token_indexer}\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "outputId": "123e18aa-02f2-4522-e766-c43414fa1410",
        "id": "fX3yQmXurZ-d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "train_ds = reader.read(DATA_ROOT / \"train.csv\")\n",
        "val_ds = None"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "14640it [00:18, 785.09it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "outputId": "4c0d69b4-9272-4067-de23-91e6a014d1b1",
        "id": "XeYQxX_YrZ-s",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "len(train_ds)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "14640"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 42
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "outputId": "a6a9a3ba-0743-450f-f1a4-3cb853af05cc",
        "id": "0KqdOBUZrZ-4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        }
      },
      "cell_type": "code",
      "source": [
        "vars(train_ds[0].fields[\"tokens\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'_indexed_tokens': None,\n",
              " '_indexer_name_to_indexed_token': None,\n",
              " '_token_indexers': {'tokens': <allennlp.data.token_indexers.elmo_indexer.ELMoTokenCharactersIndexer at 0x7f42b6afe320>},\n",
              " 'tokens': [@VirginAmerica, What, @dhepburn, said, .]}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 43
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "outputId": "200504d3-2d68-4df3-abc0-825f1dc115d1",
        "id": "N6lwAUZjrZ_C",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "vars(train_ds[0].fields[\"label\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'array': array([0, 1, 0], dtype=object), 'padding_value': 0}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "t1TFE9kvrZ_O"
      },
      "cell_type": "markdown",
      "source": [
        "### Prepare Vocabulary"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "TsN2VpKSrZ_T",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "vocab = Vocabulary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "ZSDayQsgrZ_g"
      },
      "cell_type": "markdown",
      "source": [
        "### Prepare Iterator"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "vc7kgux6rZ_p",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from allennlp.data.iterators import BucketIterator"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "CwFIZq18rZ_1",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "iterator = BucketIterator(batch_size=config.batch_size, \n",
        "                          sorting_keys=[(\"tokens\", \"num_tokens\")],\n",
        "                         )"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "wx1PbbcarZ_7",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "iterator.index_with(vocab)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "RS60JF-UraAB",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "BvVvxeRcraAN"
      },
      "cell_type": "markdown",
      "source": [
        "### Read Sample"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "0u9LFm7XraAR",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "batch = next(iter(iterator(train_ds)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "outputId": "be577ac8-6b9e-4f50-bd0f-d968aafac4cd",
        "id": "O6EwGRNCraAZ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 3179
        }
      },
      "cell_type": "code",
      "source": [
        "batch"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'id': [None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None],\n",
              " 'label': tensor([[1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [0., 1., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 1., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 1., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 1., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 1., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 1., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [0., 1., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.]]),\n",
              " 'tokens': {'tokens': tensor([[[259,  65, 118,  ..., 261, 261, 261],\n",
              "           [259, 110, 112,  ..., 261, 261, 261],\n",
              "           [259, 117, 102,  ..., 261, 261, 261],\n",
              "           ...,\n",
              "           [259, 117, 105,  ..., 261, 261, 261],\n",
              "           [  0,   0,   0,  ...,   0,   0,   0],\n",
              "           [  0,   0,   0,  ...,   0,   0,   0]],\n",
              "  \n",
              "          [[259,  65,  86,  ..., 261, 261, 261],\n",
              "           [259, 109,  98,  ..., 261, 261, 261],\n",
              "           [259,  51, 260,  ..., 261, 261, 261],\n",
              "           ...,\n",
              "           [259, 100, 105,  ..., 261, 261, 261],\n",
              "           [  0,   0,   0,  ...,   0,   0,   0],\n",
              "           [  0,   0,   0,  ...,   0,   0,   0]],\n",
              "  \n",
              "          [[259,  65,  66,  ..., 261, 261, 261],\n",
              "           [259, 103, 109,  ..., 261, 261, 261],\n",
              "           [259,  68,  98,  ..., 261, 261, 261],\n",
              "           ...,\n",
              "           [259, 112, 115,  ..., 261, 261, 261],\n",
              "           [259,  99, 118,  ..., 261, 261, 261],\n",
              "           [  0,   0,   0,  ...,   0,   0,   0]],\n",
              "  \n",
              "          ...,\n",
              "  \n",
              "          [[259,  65, 118,  ..., 261, 261, 261],\n",
              "           [259, 105,  98,  ..., 261, 261, 261],\n",
              "           [259, 117, 112,  ..., 261, 261, 261],\n",
              "           ...,\n",
              "           [259,  34, 260,  ..., 261, 261, 261],\n",
              "           [  0,   0,   0,  ...,   0,   0,   0],\n",
              "           [  0,   0,   0,  ...,   0,   0,   0]],\n",
              "  \n",
              "          [[259,  65,  84,  ..., 261, 261, 261],\n",
              "           [259, 120, 102,  ..., 261, 261, 261],\n",
              "           [259,  98, 115,  ..., 261, 261, 261],\n",
              "           ...,\n",
              "           [259,  73,  70,  ..., 261, 261, 261],\n",
              "           [  0,   0,   0,  ...,   0,   0,   0],\n",
              "           [  0,   0,   0,  ...,   0,   0,   0]],\n",
              "  \n",
              "          [[259,  65,  86,  ..., 261, 261, 261],\n",
              "           [259,  71, 106,  ..., 261, 261, 261],\n",
              "           [259, 117, 105,  ..., 261, 261, 261],\n",
              "           ...,\n",
              "           [  0,   0,   0,  ...,   0,   0,   0],\n",
              "           [  0,   0,   0,  ...,   0,   0,   0],\n",
              "           [  0,   0,   0,  ...,   0,   0,   0]]])}}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "outputId": "de42797d-ca46-4b95-d28b-913b78a892cc",
        "id": "Lv9W4xipraAl",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "batch[\"tokens\"][\"tokens\"].shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([64, 27, 50])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 51
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "aXuusjbgraA1"
      },
      "cell_type": "markdown",
      "source": [
        "### Prepare Model"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "Zl1OCBVEraA_",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "\n",
        "from allennlp.modules.seq2vec_encoders import Seq2VecEncoder, PytorchSeq2VecWrapper\n",
        "from allennlp.nn.util import get_text_field_mask\n",
        "from allennlp.models import Model\n",
        "from allennlp.modules.text_field_embedders import TextFieldEmbedder"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "QoexCIfmraBM",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class BaselineModel(Model):\n",
        "    def __init__(self, word_embeddings: TextFieldEmbedder,\n",
        "                 encoder: Seq2VecEncoder,\n",
        "                 out_sz: int=len(label_cols)):\n",
        "        super().__init__(vocab)\n",
        "        self.word_embeddings = word_embeddings\n",
        "        self.encoder = encoder\n",
        "        self.projection = nn.Linear(self.encoder.get_output_dim(), out_sz)\n",
        "        self.loss = nn.BCEWithLogitsLoss()\n",
        "        \n",
        "    def forward(self, tokens: Dict[str, torch.Tensor],\n",
        "                id: Any, label: torch.Tensor) -> torch.Tensor:\n",
        "        mask = get_text_field_mask(tokens)\n",
        "        embeddings = self.word_embeddings(tokens)\n",
        "        state = self.encoder(embeddings, mask)\n",
        "        class_logits = self.projection(state)\n",
        "        \n",
        "        output = {\"class_logits\": class_logits}\n",
        "        output[\"loss\"] = self.loss(class_logits, label)\n",
        "\n",
        "        return output"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "MAvUsveeraBU",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "t-RIhCDvraBb"
      },
      "cell_type": "markdown",
      "source": [
        "### Prepare Embeddings"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "outputId": "b6e89797-3f6c-4255-c20f-76031ad6e8b4",
        "id": "ng3lzsNMraBf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        }
      },
      "cell_type": "code",
      "source": [
        "from allennlp.modules.text_field_embedders import BasicTextFieldEmbedder\n",
        "from allennlp.modules.token_embedders import ElmoTokenEmbedder\n",
        "\n",
        "options_file = 'https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x1024_128_2048cnn_1xhighway/elmo_2x1024_128_2048cnn_1xhighway_options.json'\n",
        "weight_file = 'https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x1024_128_2048cnn_1xhighway/elmo_2x1024_128_2048cnn_1xhighway_weights.hdf5'\n",
        "\n",
        "elmo_embedder = ElmoTokenEmbedder(options_file, weight_file)\n",
        "word_embeddings = BasicTextFieldEmbedder({\"tokens\": elmo_embedder})"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 336/336 [00:00<00:00, 245306.55B/s]\n",
            "100%|██████████| 54402456/54402456 [00:02<00:00, 25637890.34B/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "ZAbX3qNtraBx",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from allennlp.modules.seq2vec_encoders import PytorchSeq2VecWrapper\n",
        "encoder: Seq2VecEncoder = PytorchSeq2VecWrapper(nn.LSTM(word_embeddings.get_output_dim(), config.hidden_sz, bidirectional=True, batch_first=True))\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "QsgdLD-zraCB",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model = BaselineModel(\n",
        "    word_embeddings, \n",
        "    encoder, \n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "wCyB0d0iraCG",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "if USE_GPU: model.cuda()\n",
        "else: model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "c9h0Vmo0raCO"
      },
      "cell_type": "markdown",
      "source": [
        "### Train"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "0lBrE6U_raCP",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "optimizer = optim.Adam(model.parameters(), lr=config.lr)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "t31y1Lh-raCZ",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from allennlp.training.trainer import Trainer\n",
        "\n",
        "trainer = Trainer(\n",
        "    model=model,\n",
        "    optimizer=optimizer,\n",
        "    iterator=iterator,\n",
        "    train_dataset=train_ds,\n",
        "    cuda_device=0 if USE_GPU else -1,\n",
        "    num_epochs=config.epochs,\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "outputId": "a299ed49-7e36-4926-d46a-48001ec40a51",
        "id": "fB45IOSfraCg",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 196
        }
      },
      "cell_type": "code",
      "source": [
        "metrics = trainer.train()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "loss: 0.4940 ||: 100%|██████████| 229/229 [00:40<00:00,  5.54it/s]\n",
            "loss: 0.3640 ||: 100%|██████████| 229/229 [00:40<00:00,  5.94it/s]\n",
            "loss: 0.3417 ||: 100%|██████████| 229/229 [00:40<00:00,  5.63it/s]\n",
            "loss: 0.3316 ||: 100%|██████████| 229/229 [00:40<00:00,  5.54it/s]\n",
            "loss: 0.3232 ||: 100%|██████████| 229/229 [00:40<00:00,  5.67it/s]\n",
            "loss: 0.3173 ||: 100%|██████████| 229/229 [00:40<00:00,  5.73it/s]\n",
            "loss: 0.3101 ||: 100%|██████████| 229/229 [00:40<00:00,  6.72it/s]\n",
            "loss: 0.3064 ||: 100%|██████████| 229/229 [00:40<00:00,  6.03it/s]\n",
            "loss: 0.2999 ||: 100%|██████████| 229/229 [00:40<00:00,  5.61it/s]\n",
            "loss: 0.2932 ||: 100%|██████████| 229/229 [00:40<00:00,  6.17it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "OAjZoFkwraCv",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "MQF-AbNGsr_j"
      },
      "cell_type": "markdown",
      "source": [
        "### Predictions"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "94YpjLPisr_0",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from allennlp.predictors.sentence_tagger import SentenceTaggerPredictor"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "7A1js83hssAC",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "tagger = SentenceTaggerPredictor(model, reader)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "_nLYjz8BssAM",
        "outputId": "316c65f9-494a-40b7-aefd-2997cbc1205d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        }
      },
      "cell_type": "code",
      "source": [
        "tagger.predict(\"Bad Service, utter disaster!\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'class_logits': [1.424069881439209, -2.3454854488372803, -1.7706302404403687],\n",
              " 'loss': 0.6294845342636108}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 63
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "oWlfHBd5rgK8"
      },
      "cell_type": "markdown",
      "source": [
        "## BERT"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "Qb-UmjK6rgLG",
        "outputId": "5f91cc54-95db-4622-ebcb-4b01d08c0409",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "from allennlp.data.token_indexers import PretrainedBertIndexer\n",
        "\n",
        "token_indexer = PretrainedBertIndexer(\n",
        "    pretrained_model=\"bert-base-uncased\",\n",
        "    max_pieces=config.max_seq_len,\n",
        "    do_lowercase=True,\n",
        " )\n",
        "# apparently we need to truncate the sequence here, which is a stupid design decision\n",
        "def tokenizer(s: str):\n",
        "    return token_indexer.wordpiece_tokenizer(s)[:config.max_seq_len - 2]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 231508/231508 [00:00<00:00, 5636031.33B/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "DoAOHkEmrgLU",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "reader = SentimentDatasetReader(\n",
        "    tokenizer=tokenizer,\n",
        "    token_indexers={\"tokens\": token_indexer}\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "outputId": "c65c35f1-8f7e-4f37-82a4-692cf44187f1",
        "id": "trtiLbT-rgLh",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "train_ds = reader.read(DATA_ROOT / \"train.csv\")\n",
        "val_ds = None"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "14640it [00:15, 937.78it/s] \n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "outputId": "8dd8ea14-50ee-45f6-bb37-4153b700ba65",
        "id": "axghVEtfrgLx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "len(train_ds)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "14640"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 67
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "outputId": "9aa7f973-2365-4ce4-9ea0-42024d88669d",
        "id": "OPME5WthrgL7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        }
      },
      "cell_type": "code",
      "source": [
        "vars(train_ds[0].fields[\"tokens\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'_indexed_tokens': None,\n",
              " '_indexer_name_to_indexed_token': None,\n",
              " '_token_indexers': {'tokens': <allennlp.data.token_indexers.wordpiece_indexer.PretrainedBertIndexer at 0x7f42b0c8e198>},\n",
              " 'tokens': [[UNK], [UNK], @, ##dh, ##ep, ##burn, said, ##.]}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 68
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "outputId": "5fd32d10-60f4-43e2-9b49-1838034f5347",
        "id": "F8MkrB3ArgMH",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "vars(train_ds[0].fields[\"label\"])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'array': array([0, 1, 0], dtype=object), 'padding_value': 0}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 69
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "JMwGeb7VrgMZ"
      },
      "cell_type": "markdown",
      "source": [
        "### Prepare Vocabulary"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "NMy1AirnrgMd",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "vocab = Vocabulary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "hTgW8f_BrgMu"
      },
      "cell_type": "markdown",
      "source": [
        "### Prepare Iterator"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "JsrdDhEhrgMz",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from allennlp.data.iterators import BucketIterator"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "cbe-TugxrgND",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "iterator = BucketIterator(batch_size=config.batch_size, \n",
        "                          sorting_keys=[(\"tokens\", \"num_tokens\")],\n",
        "                         )"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "9sYwq92UrgNJ",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "iterator.index_with(vocab)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "lmhGnzrXrgNP",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "VstgcGgQrgNc"
      },
      "cell_type": "markdown",
      "source": [
        "### Read Sample"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "sTO_Fw8PrgNd",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "batch = next(iter(iterator(train_ds)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "outputId": "c983199f-86a9-4b04-a363-a93d59489317",
        "id": "lXPWwdk5rgNo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 2804
        }
      },
      "cell_type": "code",
      "source": [
        "batch"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'id': [None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None,\n",
              "  None],\n",
              " 'label': tensor([[1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 1., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 1., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 1., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 1., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 1., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.],\n",
              "         [0., 0., 1.],\n",
              "         [1., 0., 0.],\n",
              "         [1., 0., 0.]]),\n",
              " 'tokens': {'mask': tensor([[1, 1, 1,  ..., 0, 0, 0],\n",
              "          [1, 1, 1,  ..., 1, 0, 0],\n",
              "          [1, 1, 1,  ..., 1, 1, 0],\n",
              "          ...,\n",
              "          [1, 1, 1,  ..., 0, 0, 0],\n",
              "          [1, 1, 1,  ..., 0, 0, 0],\n",
              "          [1, 1, 1,  ..., 0, 0, 0]]),\n",
              "  'tokens': tensor([[  101,   100,   100,  ...,     0,     0,     0],\n",
              "          [  101,   100,  2017,  ...,   102,     0,     0],\n",
              "          [  101,   100,  7864,  ...,   100,   102,     0],\n",
              "          ...,\n",
              "          [  101,  1030, 19496,  ...,     0,     0,     0],\n",
              "          [  101,   100,  1011,  ...,     0,     0,     0],\n",
              "          [  101,  1030, 19496,  ...,     0,     0,     0]]),\n",
              "  'tokens-offsets': tensor([[ 1,  2,  3,  ...,  0,  0,  0],\n",
              "          [ 1,  2,  3,  ..., 24,  0,  0],\n",
              "          [ 1,  2,  3,  ..., 24, 25,  0],\n",
              "          ...,\n",
              "          [ 1,  2,  3,  ...,  0,  0,  0],\n",
              "          [ 1,  2,  3,  ...,  0,  0,  0],\n",
              "          [ 1,  2,  3,  ...,  0,  0,  0]]),\n",
              "  'tokens-type-ids': tensor([[0, 0, 0,  ..., 0, 0, 0],\n",
              "          [0, 0, 0,  ..., 0, 0, 0],\n",
              "          [0, 0, 0,  ..., 0, 0, 0],\n",
              "          ...,\n",
              "          [0, 0, 0,  ..., 0, 0, 0],\n",
              "          [0, 0, 0,  ..., 0, 0, 0],\n",
              "          [0, 0, 0,  ..., 0, 0, 0]])}}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 75
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "outputId": "2c0ef1a2-8b4f-4031-ade1-1bfe1d8aa9e5",
        "id": "CKEBQUuZrgNy",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "batch[\"tokens\"][\"tokens\"].shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([64, 28])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 76
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "EttN8opjrgOA"
      },
      "cell_type": "markdown",
      "source": [
        "### Prepare Model"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "4-6CIoSJrgOL",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "\n",
        "from allennlp.modules.seq2vec_encoders import Seq2VecEncoder, PytorchSeq2VecWrapper\n",
        "from allennlp.nn.util import get_text_field_mask\n",
        "from allennlp.models import Model\n",
        "from allennlp.modules.text_field_embedders import TextFieldEmbedder"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "tKgrwnC3rgOU",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class BaselineModel(Model):\n",
        "    def __init__(self, word_embeddings: TextFieldEmbedder,\n",
        "                 encoder: Seq2VecEncoder,\n",
        "                 out_sz: int=len(label_cols)):\n",
        "        super().__init__(vocab)\n",
        "        self.word_embeddings = word_embeddings\n",
        "        self.encoder = encoder\n",
        "        self.projection = nn.Linear(self.encoder.get_output_dim(), out_sz)\n",
        "        self.loss = nn.BCEWithLogitsLoss()\n",
        "        \n",
        "    def forward(self, tokens: Dict[str, torch.Tensor],\n",
        "                id: Any, label: torch.Tensor) -> torch.Tensor:\n",
        "        mask = get_text_field_mask(tokens)\n",
        "        embeddings = self.word_embeddings(tokens)\n",
        "        state = self.encoder(embeddings, mask)\n",
        "        class_logits = self.projection(state)\n",
        "        \n",
        "        output = {\"class_logits\": class_logits}\n",
        "        output[\"loss\"] = self.loss(class_logits, label)\n",
        "\n",
        "        return output"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "TJ1zuv-krgOa",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "amRwVOjNrgOi"
      },
      "cell_type": "markdown",
      "source": [
        "### Prepare Embeddings"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "outputId": "b7431037-3a89-429e-86b0-0a6317a8bacb",
        "id": "uP4NsbI2rgOp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "from allennlp.modules.text_field_embedders import BasicTextFieldEmbedder\n",
        "from allennlp.modules.token_embedders.bert_token_embedder import PretrainedBertEmbedder\n",
        "\n",
        "bert_embedder = PretrainedBertEmbedder(\n",
        "        pretrained_model=\"bert-base-uncased\",\n",
        "        top_layer_only=True, # conserve memory\n",
        ")\n",
        "word_embeddings: TextFieldEmbedder = BasicTextFieldEmbedder({\"tokens\": bert_embedder},\n",
        "                                                            # we'll be ignoring masks so we'll need to set this to True\n",
        "                                                           allow_unmatched_keys = True)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 407873900/407873900 [00:08<00:00, 47933335.81B/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "lAuVNfRbrgPB",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "BERT_DIM = word_embeddings.get_output_dim()\n",
        "\n",
        "class BertSentencePooler(Seq2VecEncoder):\n",
        "    def forward(self, embs: torch.tensor, \n",
        "                mask: torch.tensor=None) -> torch.tensor:\n",
        "        # extract first token tensor\n",
        "        return embs[:, 0]\n",
        "    \n",
        "    @overrides\n",
        "    def get_output_dim(self) -> int:\n",
        "        return BERT_DIM\n",
        "    \n",
        "encoder = BertSentencePooler(vocab)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "05D8FyTOrgPV",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model = BaselineModel(\n",
        "    word_embeddings, \n",
        "    encoder, \n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "rsPwXJHBrgPh",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "if USE_GPU: model.cuda()\n",
        "else: model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "G7K1Cp1TrgPt"
      },
      "cell_type": "markdown",
      "source": [
        "### Train"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "erYlvJ2BrgPy",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "optimizer = optim.Adam(model.parameters(), lr=config.lr)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "gL-nfX9urgQC",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from allennlp.training.trainer import Trainer\n",
        "\n",
        "trainer = Trainer(\n",
        "    model=model,\n",
        "    optimizer=optimizer,\n",
        "    iterator=iterator,\n",
        "    train_dataset=train_ds,\n",
        "    cuda_device=0 if USE_GPU else -1,\n",
        "    num_epochs=config.epochs,\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "outputId": "2c5529a2-e28e-4877-fbe6-dffcabb77b28",
        "id": "gAMaK-UtrgQJ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 196
        }
      },
      "cell_type": "code",
      "source": [
        "metrics = trainer.train()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "loss: 0.5332 ||: 100%|██████████| 229/229 [00:54<00:00,  5.84it/s]\n",
            "loss: 0.4862 ||: 100%|██████████| 229/229 [00:54<00:00,  4.54it/s]\n",
            "loss: 0.4623 ||: 100%|██████████| 229/229 [00:54<00:00,  4.65it/s]\n",
            "loss: 0.4470 ||: 100%|██████████| 229/229 [00:54<00:00,  4.20it/s]\n",
            "loss: 0.4341 ||: 100%|██████████| 229/229 [00:56<00:00,  3.62it/s]\n",
            "loss: 0.4245 ||: 100%|██████████| 229/229 [00:54<00:00,  4.71it/s]\n",
            "loss: 0.4178 ||: 100%|██████████| 229/229 [00:54<00:00,  4.30it/s]\n",
            "loss: 0.4107 ||: 100%|██████████| 229/229 [00:54<00:00,  3.92it/s]\n",
            "loss: 0.4076 ||: 100%|██████████| 229/229 [00:54<00:00,  4.22it/s]\n",
            "loss: 0.4033 ||: 100%|██████████| 229/229 [00:54<00:00,  4.81it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "_swYsU_OrgQa",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "DXUfl9AXsttT"
      },
      "cell_type": "markdown",
      "source": [
        "### Predictions"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "3IO8mTRgsttl",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from allennlp.predictors.sentence_tagger import SentenceTaggerPredictor"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "jTVx6jpnstt1",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "tagger = SentenceTaggerPredictor(model, reader)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "_l1dff8ostuC",
        "outputId": "7d0da65f-b6ba-4876-f298-e9aafdd5cdf2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        }
      },
      "cell_type": "code",
      "source": [
        "tagger.predict(\"Bad Service, utter disaster!\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'class_logits': [0.7946805357933044,\n",
              "  -2.3529469966888428,\n",
              "  -1.6355410814285278],\n",
              " 'loss': 0.4787622392177582}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 88
        }
      ]
    },
    {
      "metadata": {
        "id": "FSqPuPO_uDAd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}