1_bilstm.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.6"
    },
    "colab": {
      "name": "1 - BiLSTM for PoS Tagging.ipynb",
      "provenance": [],
      "machine_shape": "hm"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "scyE4k8gpwl0"
      },
      "source": [
        "# 1 - BiLSTM for PoS Tagging\n",
        "\n",
        "## Introduction\n",
        "\n",
        "In this series we'll be building a machine learning model that produces an output for every element in an input sequence, using PyTorch and TorchText. Specifically, we will be inputting a sequence of text and the model will output a part-of-speech (PoS) tag for each token in the input text. This can also be used for named entity recognition (NER), where the output for each token will be what type of entity, if any, the token is.\n",
        "\n",
        "In this notebook, we'll be implementing a multi-layer bi-directional LSTM (BiLSTM) to predict PoS tags using the Universal Dependencies English Web Treebank (UDPOS) dataset.\n",
        "\n",
        "## Preparing Data\n",
        "\n",
        "First, let's import the necessary Python modules."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9osUMmompwl2"
      },
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "\n",
        "from torchtext.legacy import data\n",
        "from torchtext.legacy import datasets\n",
        "\n",
        "import spacy\n",
        "import numpy as np\n",
        "\n",
        "import time\n",
        "import random"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-5OZxOjBpwl3"
      },
      "source": [
        "Next, we'll set the random seeds for reproducability."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "egSkC0DDpwl4"
      },
      "source": [
        "SEED = 1234\n",
        "\n",
        "random.seed(SEED)\n",
        "np.random.seed(SEED)\n",
        "torch.manual_seed(SEED)\n",
        "torch.backends.cudnn.deterministic = True"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oTuoKelVpwl4"
      },
      "source": [
        "One of the key parts of TorchText is the `Field`. The `Field` handles how your dataset is processed.\n",
        "\n",
        "Our `TEXT` field handles how the text that we need to tag is dealt with. All we do here is set `lower = True` which lowercases all of the text.\n",
        "\n",
        "Next we'll define the `Fields` for the tags. This dataset actually has two different sets of tags, [universal dependency (UD) tags](https://universaldependencies.org/u/pos/) and [Penn Treebank (PTB) tags](https://www.sketchengine.eu/penn-treebank-tagset/). We'll only train our model on the UD tags, but will load the PTB tags to show how they could be used instead.\n",
        "\n",
        "`UD_TAGS` handles how the UD tags should be handled. Our `TEXT` vocabulary - which we'll build later - will have *unknown* tokens in it, i.e. tokens that are not within our vocabulary. However, we won't have unknown tags as we are dealing with a finite set of possible tags. TorchText `Fields` initialize a default unknown token, `<unk>`, which we remove by setting `unk_token = None`.\n",
        "\n",
        "`PTB_TAGS` does the same as `UD_TAGS`, but handles the PTB tags instead."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fidoBBJfpwl5"
      },
      "source": [
        "TEXT = data.Field(lower = True)\n",
        "UD_TAGS = data.Field(unk_token = None)\n",
        "PTB_TAGS = data.Field(unk_token = None)"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Te5ZU7FPpwl5"
      },
      "source": [
        "We then define `fields`, which handles passing our fields to the dataset.\n",
        "\n",
        "Note that order matters, if you only wanted to load the PTB tags your field would be:\n",
        "\n",
        "```\n",
        "fields = ((\"text\", TEXT), (None, None), (\"ptbtags\", PTB_TAGS))\n",
        "```\n",
        "\n",
        "Where `None` tells TorchText to not load those tags."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Iqya4Ki-pwl6"
      },
      "source": [
        "fields = ((\"text\", TEXT), (\"udtags\", UD_TAGS), (\"ptbtags\", PTB_TAGS))"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6AHzDh46pwl7"
      },
      "source": [
        "Next, we load the UDPOS dataset using our defined fields."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eI_RLyzzpwl7"
      },
      "source": [
        "train_data, valid_data, test_data = datasets.UDPOS.splits(fields)"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6cHlSTlEpwl7"
      },
      "source": [
        "We can check how many examples are in each section of the dataset by checking their length."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qCznT-c3pwl8",
        "outputId": "01c721a3-0051-43c0-c14a-d1c06c316fbc"
      },
      "source": [
        "print(f\"Number of training examples: {len(train_data)}\")\n",
        "print(f\"Number of validation examples: {len(valid_data)}\")\n",
        "print(f\"Number of testing examples: {len(test_data)}\")"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of training examples: 12543\n",
            "Number of validation examples: 2002\n",
            "Number of testing examples: 2077\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZIY61Z5Cpwl9"
      },
      "source": [
        "Let's print out an example:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MpadMDOCpwl9",
        "outputId": "704ee46f-a052-4b93-bbb2-e4fe299bd39e"
      },
      "source": [
        "print(vars(train_data.examples[0]))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'text': ['al', '-', 'zaman', ':', 'american', 'forces', 'killed', 'shaikh', 'abdullah', 'al', '-', 'ani', ',', 'the', 'preacher', 'at', 'the', 'mosque', 'in', 'the', 'town', 'of', 'qaim', ',', 'near', 'the', 'syrian', 'border', '.'], 'udtags': ['PROPN', 'PUNCT', 'PROPN', 'PUNCT', 'ADJ', 'NOUN', 'VERB', 'PROPN', 'PROPN', 'PROPN', 'PUNCT', 'PROPN', 'PUNCT', 'DET', 'NOUN', 'ADP', 'DET', 'NOUN', 'ADP', 'DET', 'NOUN', 'ADP', 'PROPN', 'PUNCT', 'ADP', 'DET', 'ADJ', 'NOUN', 'PUNCT'], 'ptbtags': ['NNP', 'HYPH', 'NNP', ':', 'JJ', 'NNS', 'VBD', 'NNP', 'NNP', 'NNP', 'HYPH', 'NNP', ',', 'DT', 'NN', 'IN', 'DT', 'NN', 'IN', 'DT', 'NN', 'IN', 'NNP', ',', 'IN', 'DT', 'JJ', 'NN', '.']}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S39xsVtxpwl9"
      },
      "source": [
        "We can also view the text and tags separately:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UULauMEBpwl-",
        "outputId": "b200c814-28a5-4495-f595-c8b1c9845b26"
      },
      "source": [
        "print(vars(train_data.examples[0])['text'])"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['al', '-', 'zaman', ':', 'american', 'forces', 'killed', 'shaikh', 'abdullah', 'al', '-', 'ani', ',', 'the', 'preacher', 'at', 'the', 'mosque', 'in', 'the', 'town', 'of', 'qaim', ',', 'near', 'the', 'syrian', 'border', '.']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ACWgCsPqpwl-",
        "outputId": "55f4e5ca-30bb-4308-cf71-fd6cdfe09847"
      },
      "source": [
        "print(vars(train_data.examples[0])['udtags'])"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['PROPN', 'PUNCT', 'PROPN', 'PUNCT', 'ADJ', 'NOUN', 'VERB', 'PROPN', 'PROPN', 'PROPN', 'PUNCT', 'PROPN', 'PUNCT', 'DET', 'NOUN', 'ADP', 'DET', 'NOUN', 'ADP', 'DET', 'NOUN', 'ADP', 'PROPN', 'PUNCT', 'ADP', 'DET', 'ADJ', 'NOUN', 'PUNCT']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ko_hI-Fgpwl-",
        "outputId": "e351c398-01b1-4289-de33-537be2359941"
      },
      "source": [
        "print(vars(train_data.examples[0])['ptbtags'])"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['NNP', 'HYPH', 'NNP', ':', 'JJ', 'NNS', 'VBD', 'NNP', 'NNP', 'NNP', 'HYPH', 'NNP', ',', 'DT', 'NN', 'IN', 'DT', 'NN', 'IN', 'DT', 'NN', 'IN', 'NNP', ',', 'IN', 'DT', 'JJ', 'NN', '.']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0TYUko-Bpwl-"
      },
      "source": [
        "Next, we'll build the vocabulary - a mapping of tokens to integers. \n",
        "\n",
        "We want some unknown tokens within our dataset in order to replicate how this model would be used in real life, so we set the `min_freq` to 2 which means only tokens that appear twice in the training set will be added to the vocabulary and the rest will be replaced by `<unk>` tokens.\n",
        "\n",
        "We also load the [GloVe](https://nlp.stanford.edu/projects/glove/) pre-trained token embeddings. Specifically, the 100-dimensional embeddings that have been trained on 6 billion tokens. Using pre-trained embeddings usually leads to improved performance - although admittedly the dataset used in this tutorial is too small to take advantage of the pre-trained embeddings. \n",
        "\n",
        "`unk_init` is used to initialize the token embeddings which are not in the pre-trained embedding vocabulary. By default this sets those embeddings to zeros, however it is better to not have them all initialized to the same value, so we initialize them from a Normal/Gaussian distribution.\n",
        "\n",
        "These pre-trained vectors are now loaded into our vocabulary and we will initialize our model with these values later."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MGKCnORWpwl_"
      },
      "source": [
        "MIN_FREQ = 2\n",
        "\n",
        "TEXT.build_vocab(train_data, \n",
        "                 min_freq = MIN_FREQ,\n",
        "                 vectors = \"glove.6B.100d\",\n",
        "                 unk_init = torch.Tensor.normal_)\n",
        "\n",
        "\n",
        "UD_TAGS.build_vocab(train_data)\n",
        "PTB_TAGS.build_vocab(train_data)"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lt76JMgPpwmA"
      },
      "source": [
        "We can check how many tokens and tags are in our vocabulary by getting their length:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Y0meSOCKpwmB",
        "outputId": "d8440999-07a7-4c38-9f60-ca0167915f0b"
      },
      "source": [
        "print(f\"Unique tokens in TEXT vocabulary: {len(TEXT.vocab)}\")\n",
        "print(f\"Unique tokens in UD_TAG vocabulary: {len(UD_TAGS.vocab)}\")\n",
        "print(f\"Unique tokens in PTB_TAG vocabulary: {len(PTB_TAGS.vocab)}\")"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Unique tokens in TEXT vocabulary: 8866\n",
            "Unique tokens in UD_TAG vocabulary: 18\n",
            "Unique tokens in PTB_TAG vocabulary: 51\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W-jeBfsSpwmG"
      },
      "source": [
        "Exploring the vocabulary, we can check the most common tokens within our texts:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jFHRIvpMpwmH",
        "outputId": "5fddecae-d778-4754-89d5-77d3a2e70db5"
      },
      "source": [
        "print(TEXT.vocab.freqs.most_common(20))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[('the', 9076), ('.', 8640), (',', 7021), ('to', 5137), ('and', 5002), ('a', 3782), ('of', 3622), ('i', 3379), ('in', 3112), ('is', 2239), ('you', 2156), ('that', 2036), ('it', 1850), ('for', 1842), ('-', 1426), ('have', 1359), ('\"', 1296), ('on', 1273), ('was', 1244), ('with', 1216)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dJQYwhqgpwmH"
      },
      "source": [
        "We can see the vocabularies for both of our tags:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_8y_7x1epwmH",
        "outputId": "b3e6c42f-280f-4c25-ebc6-3f84f0c27686"
      },
      "source": [
        "print(UD_TAGS.vocab.itos)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['<pad>', 'NOUN', 'PUNCT', 'VERB', 'PRON', 'ADP', 'DET', 'PROPN', 'ADJ', 'AUX', 'ADV', 'CCONJ', 'PART', 'NUM', 'SCONJ', 'X', 'INTJ', 'SYM']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lNoqu4mlpwmI",
        "outputId": "a6612011-ac28-44ac-eb64-204f63be5ff1"
      },
      "source": [
        "print(PTB_TAGS.vocab.itos)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['<pad>', 'NN', 'IN', 'DT', 'NNP', 'PRP', 'JJ', 'RB', '.', 'VB', 'NNS', ',', 'CC', 'VBD', 'VBP', 'VBZ', 'CD', 'VBN', 'VBG', 'MD', 'TO', 'PRP$', '-RRB-', '-LRB-', 'WDT', 'WRB', ':', '``', \"''\", 'WP', 'RP', 'UH', 'POS', 'HYPH', 'JJR', 'NNPS', 'JJS', 'EX', 'NFP', 'GW', 'ADD', 'RBR', '$', 'PDT', 'RBS', 'SYM', 'LS', 'FW', 'AFX', 'WP$', 'XX']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1e4zYvDgpwmI"
      },
      "source": [
        "We can also see how many of each tag are in our vocabulary:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o9tF5JAdpwmI",
        "outputId": "299f754d-1f60-4131-dfce-c9ea2c7186dd"
      },
      "source": [
        "print(UD_TAGS.vocab.freqs.most_common())"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[('NOUN', 34781), ('PUNCT', 23679), ('VERB', 23081), ('PRON', 18577), ('ADP', 17638), ('DET', 16285), ('PROPN', 12946), ('ADJ', 12477), ('AUX', 12343), ('ADV', 10548), ('CCONJ', 6707), ('PART', 5567), ('NUM', 3999), ('SCONJ', 3843), ('X', 847), ('INTJ', 688), ('SYM', 599)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "i5qAQz76pwmI",
        "outputId": "1a16db14-7a85-4923-b49b-16eada6b4c70"
      },
      "source": [
        "print(PTB_TAGS.vocab.freqs.most_common())"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[('NN', 26915), ('IN', 20724), ('DT', 16817), ('NNP', 12449), ('PRP', 12193), ('JJ', 11591), ('RB', 10831), ('.', 10317), ('VB', 9476), ('NNS', 8438), (',', 8062), ('CC', 6706), ('VBD', 5402), ('VBP', 5374), ('VBZ', 4578), ('CD', 3998), ('VBN', 3967), ('VBG', 3330), ('MD', 3294), ('TO', 3286), ('PRP$', 3068), ('-RRB-', 1008), ('-LRB-', 973), ('WDT', 948), ('WRB', 869), (':', 866), ('``', 813), (\"''\", 785), ('WP', 760), ('RP', 755), ('UH', 689), ('POS', 684), ('HYPH', 664), ('JJR', 503), ('NNPS', 498), ('JJS', 383), ('EX', 359), ('NFP', 338), ('GW', 294), ('ADD', 292), ('RBR', 276), ('$', 258), ('PDT', 175), ('RBS', 169), ('SYM', 156), ('LS', 117), ('FW', 93), ('AFX', 48), ('WP$', 15), ('XX', 1)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5pj_pOsqpwmJ"
      },
      "source": [
        "We can also view how common each of the tags are within the training set:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ee-Hm4E-pwmJ"
      },
      "source": [
        "def tag_percentage(tag_counts):\n",
        "    \n",
        "    total_count = sum([count for tag, count in tag_counts])\n",
        "    \n",
        "    tag_counts_percentages = [(tag, count, count/total_count) for tag, count in tag_counts]\n",
        "        \n",
        "    return tag_counts_percentages"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "u2sFxAAHpwmJ",
        "outputId": "ffa2bf90-f758-4e41-e883-9bc48fb2e44f"
      },
      "source": [
        "print(\"Tag\\t\\tCount\\t\\tPercentage\\n\")\n",
        "\n",
        "for tag, count, percent in tag_percentage(UD_TAGS.vocab.freqs.most_common()):\n",
        "    print(f\"{tag}\\t\\t{count}\\t\\t{percent*100:4.1f}%\")"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tag\t\tCount\t\tPercentage\n",
            "\n",
            "NOUN\t\t34781\t\t17.0%\n",
            "PUNCT\t\t23679\t\t11.6%\n",
            "VERB\t\t23081\t\t11.3%\n",
            "PRON\t\t18577\t\t 9.1%\n",
            "ADP\t\t17638\t\t 8.6%\n",
            "DET\t\t16285\t\t 8.0%\n",
            "PROPN\t\t12946\t\t 6.3%\n",
            "ADJ\t\t12477\t\t 6.1%\n",
            "AUX\t\t12343\t\t 6.0%\n",
            "ADV\t\t10548\t\t 5.2%\n",
            "CCONJ\t\t6707\t\t 3.3%\n",
            "PART\t\t5567\t\t 2.7%\n",
            "NUM\t\t3999\t\t 2.0%\n",
            "SCONJ\t\t3843\t\t 1.9%\n",
            "X\t\t847\t\t 0.4%\n",
            "INTJ\t\t688\t\t 0.3%\n",
            "SYM\t\t599\t\t 0.3%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GfhpCJILpwmK",
        "outputId": "1af76429-a08e-48b2-bc6d-e19d3df6ae52"
      },
      "source": [
        "print(\"Tag\\t\\tCount\\t\\tPercentage\\n\")\n",
        "\n",
        "for tag, count, percent in tag_percentage(PTB_TAGS.vocab.freqs.most_common()):\n",
        "    print(f\"{tag}\\t\\t{count}\\t\\t{percent*100:4.1f}%\")"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tag\t\tCount\t\tPercentage\n",
            "\n",
            "NN\t\t26915\t\t13.2%\n",
            "IN\t\t20724\t\t10.1%\n",
            "DT\t\t16817\t\t 8.2%\n",
            "NNP\t\t12449\t\t 6.1%\n",
            "PRP\t\t12193\t\t 6.0%\n",
            "JJ\t\t11591\t\t 5.7%\n",
            "RB\t\t10831\t\t 5.3%\n",
            ".\t\t10317\t\t 5.0%\n",
            "VB\t\t9476\t\t 4.6%\n",
            "NNS\t\t8438\t\t 4.1%\n",
            ",\t\t8062\t\t 3.9%\n",
            "CC\t\t6706\t\t 3.3%\n",
            "VBD\t\t5402\t\t 2.6%\n",
            "VBP\t\t5374\t\t 2.6%\n",
            "VBZ\t\t4578\t\t 2.2%\n",
            "CD\t\t3998\t\t 2.0%\n",
            "VBN\t\t3967\t\t 1.9%\n",
            "VBG\t\t3330\t\t 1.6%\n",
            "MD\t\t3294\t\t 1.6%\n",
            "TO\t\t3286\t\t 1.6%\n",
            "PRP$\t\t3068\t\t 1.5%\n",
            "-RRB-\t\t1008\t\t 0.5%\n",
            "-LRB-\t\t973\t\t 0.5%\n",
            "WDT\t\t948\t\t 0.5%\n",
            "WRB\t\t869\t\t 0.4%\n",
            ":\t\t866\t\t 0.4%\n",
            "``\t\t813\t\t 0.4%\n",
            "''\t\t785\t\t 0.4%\n",
            "WP\t\t760\t\t 0.4%\n",
            "RP\t\t755\t\t 0.4%\n",
            "UH\t\t689\t\t 0.3%\n",
            "POS\t\t684\t\t 0.3%\n",
            "HYPH\t\t664\t\t 0.3%\n",
            "JJR\t\t503\t\t 0.2%\n",
            "NNPS\t\t498\t\t 0.2%\n",
            "JJS\t\t383\t\t 0.2%\n",
            "EX\t\t359\t\t 0.2%\n",
            "NFP\t\t338\t\t 0.2%\n",
            "GW\t\t294\t\t 0.1%\n",
            "ADD\t\t292\t\t 0.1%\n",
            "RBR\t\t276\t\t 0.1%\n",
            "$\t\t258\t\t 0.1%\n",
            "PDT\t\t175\t\t 0.1%\n",
            "RBS\t\t169\t\t 0.1%\n",
            "SYM\t\t156\t\t 0.1%\n",
            "LS\t\t117\t\t 0.1%\n",
            "FW\t\t93\t\t 0.0%\n",
            "AFX\t\t48\t\t 0.0%\n",
            "WP$\t\t15\t\t 0.0%\n",
            "XX\t\t1\t\t 0.0%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rOu9Uv6ZpwmL"
      },
      "source": [
        "The final part of data preparation is handling the iterator. \n",
        "\n",
        "This will be iterated over to return batches of data to process. Here, we set the batch size and the `device` - which is used to place the batches of tensors on our GPU, if we have one. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "78DjTNiJpwmL"
      },
      "source": [
        "BATCH_SIZE = 128\n",
        "\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "\n",
        "train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(\n",
        "    (train_data, valid_data, test_data), \n",
        "    batch_size = BATCH_SIZE,\n",
        "    device = device)"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X-KCF_thpwmO"
      },
      "source": [
        "## Building the Model\n",
        "\n",
        "Next up, we define our model - a multi-layer bi-directional LSTM. The image below shows a simplified version of the model with only one LSTM layer and omitting the LSTM's cell state for clarity.\n",
        "\n",
        "![](https://github.com/bentrevett/pytorch-pos-tagging/blob/master/assets/pos-bidirectional-lstm.png?raw=1)\n",
        "\n",
        "The model takes in a sequence of tokens, $X = \\{x_1, x_2,...,x_T\\}$, passes them through an embedding layer, $e$, to get the token embeddings, $e(X) = \\{e(x_1), e(x_2), ..., e(x_T)\\}$.\n",
        "\n",
        "These embeddings are processed - one per time-step - by the forward and backward LSTMs. The forward LSTM processes the sequence from left-to-right, whilst the backward LSTM processes the sequence right-to-left, i.e. the first input to the forward LSTM is $x_1$ and the first input to the backward LSTM is $x_T$. \n",
        "\n",
        "The LSTMs also take in the the hidden, $h$, and cell, $c$, states from the previous time-step\n",
        "\n",
        "$$h^{\\rightarrow}_t = \\text{LSTM}^{\\rightarrow}(e(x^{\\rightarrow}_t), h^{\\rightarrow}_{t-1}, c^{\\rightarrow}_{t-1})$$\n",
        "$$h^{\\leftarrow}_t=\\text{LSTM}^{\\leftarrow}(e(x^{\\leftarrow}_t), h^{\\leftarrow}_{t-1}, c^{\\leftarrow}_{t-1})$$\n",
        "\n",
        "After the whole sequence has been processed, the hidden and cell states are then passed to the next layer of the LSTM.\n",
        "\n",
        "The initial hidden and cell states, $h_0$ and $c_0$, for each direction and layer are initialized to a tensor full of zeros.\n",
        "\n",
        "We then concatenate both the forward and backward hidden states from the final layer of the LSTM, $H = \\{h_1, h_2, ... h_T\\}$, where $h_1 = [h^{\\rightarrow}_1;h^{\\leftarrow}_T]$, $h_2 = [h^{\\rightarrow}_2;h^{\\leftarrow}_{T-1}]$, etc. and pass them through a linear layer, $f$, which is used to make the prediction of which tag applies to this token, $\\hat{y}_t = f(h_t)$.\n",
        "\n",
        "When training the model, we will compare our predicted tags, $\\hat{Y}$ against the actual tags, $Y$, to calculate a loss, the gradients w.r.t. that loss, and then update our parameters.\n",
        "\n",
        "We implement the model detailed above in the `BiLSTMPOSTagger` class.\n",
        "\n",
        "`nn.Embedding` is an embedding layer and the input dimension should be the size of the input (text) vocabulary. We tell it what the index of the padding token is so it does not update the padding token's embedding entry.\n",
        "\n",
        "`nn.LSTM` is the LSTM. We apply dropout as regularization between the layers, if we are using more than one.\n",
        "\n",
        "`nn.Linear` defines the linear layer to make predictions using the LSTM outputs. We double the size of the input if we are using a bi-directional LSTM. The output dimensions should be the size of the tag vocabulary.\n",
        "\n",
        "We also define a dropout layer with `nn.Dropout`, which we use in the `forward` method to apply dropout to the embeddings and the outputs of the final layer of the LSTM."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w-Wa3Tp5pwmO"
      },
      "source": [
        "class BiLSTMPOSTagger(nn.Module):\n",
        "    def __init__(self, \n",
        "                 input_dim, \n",
        "                 embedding_dim, \n",
        "                 hidden_dim, \n",
        "                 output_dim, \n",
        "                 n_layers, \n",
        "                 bidirectional, \n",
        "                 dropout, \n",
        "                 pad_idx):\n",
        "        \n",
        "        super().__init__()\n",
        "        \n",
        "        self.embedding = nn.Embedding(input_dim, embedding_dim, padding_idx = pad_idx)\n",
        "        \n",
        "        self.lstm = nn.LSTM(embedding_dim, \n",
        "                            hidden_dim, \n",
        "                            num_layers = n_layers, \n",
        "                            bidirectional = bidirectional,\n",
        "                            dropout = dropout if n_layers > 1 else 0)\n",
        "        \n",
        "        self.fc = nn.Linear(hidden_dim * 2 if bidirectional else hidden_dim, output_dim)\n",
        "        \n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "        \n",
        "    def forward(self, text):\n",
        "\n",
        "        #text = [sent len, batch size]\n",
        "        \n",
        "        #pass text through embedding layer\n",
        "        embedded = self.dropout(self.embedding(text))\n",
        "        \n",
        "        #embedded = [sent len, batch size, emb dim]\n",
        "        \n",
        "        #pass embeddings into LSTM\n",
        "        outputs, (hidden, cell) = self.lstm(embedded)\n",
        "        \n",
        "        #outputs holds the backward and forward hidden states in the final layer\n",
        "        #hidden and cell are the backward and forward hidden and cell states at the final time-step\n",
        "        \n",
        "        #output = [sent len, batch size, hid dim * n directions]\n",
        "        #hidden/cell = [n layers * n directions, batch size, hid dim]\n",
        "        \n",
        "        #we use our outputs to make a prediction of what the tag should be\n",
        "        predictions = self.fc(self.dropout(outputs))\n",
        "        \n",
        "        #predictions = [sent len, batch size, output dim]\n",
        "        \n",
        "        return predictions"
      ],
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JTjQq-Eqpwmi"
      },
      "source": [
        "## Training the Model\n",
        "\n",
        "Next, we instantiate the model. We need to ensure the embedding dimensions matches that of the GloVe embeddings we loaded earlier.\n",
        "\n",
        "The rest of the hyperparmeters have been chosen as sensible defaults, though there may be a combination that performs better on this model and dataset.\n",
        "\n",
        "The input and output dimensions are taken directly from the lengths of the respective vocabularies. The padding index is obtained using the vocabulary and the `Field` of the text."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "crDf8hTHpwmi"
      },
      "source": [
        "INPUT_DIM = len(TEXT.vocab)\n",
        "EMBEDDING_DIM = 100\n",
        "HIDDEN_DIM = 128\n",
        "OUTPUT_DIM = len(UD_TAGS.vocab)\n",
        "N_LAYERS = 2\n",
        "BIDIRECTIONAL = True\n",
        "DROPOUT = 0.25\n",
        "PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]\n",
        "\n",
        "model = BiLSTMPOSTagger(INPUT_DIM, \n",
        "                        EMBEDDING_DIM, \n",
        "                        HIDDEN_DIM, \n",
        "                        OUTPUT_DIM, \n",
        "                        N_LAYERS, \n",
        "                        BIDIRECTIONAL, \n",
        "                        DROPOUT, \n",
        "                        PAD_IDX)"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Oc0xJFAApwmi"
      },
      "source": [
        "We initialize the weights from a simple Normal distribution. Again, there may be a better initialization scheme for this model and dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qEq91x0Epwmj",
        "outputId": "b75ee1e3-e792-4cad-eab8-6dec2028fa79"
      },
      "source": [
        "def init_weights(m):\n",
        "    for name, param in m.named_parameters():\n",
        "        nn.init.normal_(param.data, mean = 0, std = 0.1)\n",
        "        \n",
        "model.apply(init_weights)"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "BiLSTMPOSTagger(\n",
              "  (embedding): Embedding(8866, 100, padding_idx=1)\n",
              "  (lstm): LSTM(100, 128, num_layers=2, dropout=0.25, bidirectional=True)\n",
              "  (fc): Linear(in_features=256, out_features=18, bias=True)\n",
              "  (dropout): Dropout(p=0.25, inplace=False)\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZQmNFG_wpwmj"
      },
      "source": [
        "Next, a small function to tell us how many parameters are in our model. Useful for comparing different models."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rfPyt7TTpwmk",
        "outputId": "0b3cdffe-31d0-472f-e5c5-1dd418b4b677"
      },
      "source": [
        "def count_parameters(model):\n",
        "    return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
        "\n",
        "print(f'The model has {count_parameters(model):,} trainable parameters')"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The model has 1,522,010 trainable parameters\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wTVuLKdFpwmk"
      },
      "source": [
        "We'll now initialize our model's embedding layer with the pre-trained embedding values we loaded earlier.\n",
        "\n",
        "This is done by getting them from the vocab's `.vectors` attribute and then performing a `.copy` to overwrite the embedding layer's current weights."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "P-H-ucDCpwml",
        "outputId": "30fd96b7-89cb-43f7-8810-703a2d26cd0a"
      },
      "source": [
        "pretrained_embeddings = TEXT.vocab.vectors\n",
        "\n",
        "print(pretrained_embeddings.shape)"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([8866, 100])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OxIg6uHVpwml",
        "outputId": "d11eda09-2ee4-44c0-ca01-93cd03127796"
      },
      "source": [
        "model.embedding.weight.data.copy_(pretrained_embeddings)"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[-0.1117, -0.4966,  0.1631,  ...,  1.2647, -0.2753, -0.1325],\n",
              "        [-0.8555, -0.7208,  1.3755,  ...,  0.0825, -1.1314,  0.3997],\n",
              "        [-0.0382, -0.2449,  0.7281,  ..., -0.1459,  0.8278,  0.2706],\n",
              "        ...,\n",
              "        [ 0.9261,  2.3049,  0.5502,  ..., -0.3492, -0.5298, -0.1577],\n",
              "        [-0.5972,  0.0471, -0.2406,  ..., -0.9446, -0.1126, -0.2260],\n",
              "        [-0.4809,  2.5629,  0.9530,  ...,  0.5278, -0.4588,  0.7294]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uAF8-bTJpwmm"
      },
      "source": [
        "It's common to initialize the embedding of the pad token to all zeros. This, along with setting the `padding_idx` in the model's embedding layer, means that the embedding should always output a tensor full of zeros when a pad token is input."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7pbjDSgNpwmm",
        "outputId": "d895eebe-3fda-42a3-b78f-f589c107c9da"
      },
      "source": [
        "model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)\n",
        "\n",
        "print(model.embedding.weight.data)"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[-0.1117, -0.4966,  0.1631,  ...,  1.2647, -0.2753, -0.1325],\n",
            "        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
            "        [-0.0382, -0.2449,  0.7281,  ..., -0.1459,  0.8278,  0.2706],\n",
            "        ...,\n",
            "        [ 0.9261,  2.3049,  0.5502,  ..., -0.3492, -0.5298, -0.1577],\n",
            "        [-0.5972,  0.0471, -0.2406,  ..., -0.9446, -0.1126, -0.2260],\n",
            "        [-0.4809,  2.5629,  0.9530,  ...,  0.5278, -0.4588,  0.7294]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hO51ESn5pwmn"
      },
      "source": [
        "We then define our optimizer, used to update our parameters w.r.t. their gradients. We use Adam with the default learning rate."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JpCGPGTkpwmn"
      },
      "source": [
        "optimizer = optim.Adam(model.parameters())"
      ],
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-8EQh0BGpwmn"
      },
      "source": [
        "Next, we define our loss function, cross-entropy loss.\n",
        "\n",
        "Even though we have no `<unk>` tokens within our tag vocab, we still have `<pad>` tokens. This is because all sentences within a batch need to be the same size. However, we don't want to calculate the loss when the target is a `<pad>` token as we aren't training our model to recognize padding tokens.\n",
        "\n",
        "We handle this by setting the `ignore_index` in our loss function to the index of the padding token in our tag vocabulary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fOhdfyscpwmo"
      },
      "source": [
        "TAG_PAD_IDX = UD_TAGS.vocab.stoi[UD_TAGS.pad_token]\n",
        "\n",
        "criterion = nn.CrossEntropyLoss(ignore_index = TAG_PAD_IDX)"
      ],
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VmiF0b31pwmo"
      },
      "source": [
        "We then place our model and loss function on our GPU, if we have one."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BHLdYmqhpwmp"
      },
      "source": [
        "model = model.to(device)\n",
        "criterion = criterion.to(device)"
      ],
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eIgmzjvEpwmq"
      },
      "source": [
        "We will be using the loss value between our predicted and actual tags to train the network, but ideally we'd like a more interpretable way to see how well our model is doing - accuracy.\n",
        "\n",
        "The issue is that we don't want to calculate accuracy over the `<pad>` tokens as we aren't interested in predicting them.\n",
        "\n",
        "The function below only calculates accuracy over non-padded tokens. `non_pad_elements` is a tensor containing the indices of the non-pad tokens within an input batch. We then compare the predictions of those elements with the labels to get a count of how many predictions were correct. We then divide this by the number of non-pad elements to get our accuracy value over the batch."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l6FMQ9Pppwmq"
      },
      "source": [
        "def categorical_accuracy(preds, y, tag_pad_idx):\n",
        "    \"\"\"\n",
        "    Returns accuracy per batch, i.e. if you get 8/10 right, this returns 0.8, NOT 8\n",
        "    \"\"\"\n",
        "    max_preds = preds.argmax(dim = 1, keepdim = True) # get the index of the max probability\n",
        "    non_pad_elements = (y != tag_pad_idx).nonzero()\n",
        "    correct = max_preds[non_pad_elements].squeeze(1).eq(y[non_pad_elements])\n",
        "    return correct.sum() / y[non_pad_elements].shape[0]"
      ],
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MzbRWYkIpwmq"
      },
      "source": [
        "Next is the function that handles training our model.\n",
        "\n",
        "We first set the model to `train` mode to turn on dropout/batch-norm/etc. (if used). Then we iterate over our iterator, which returns a batch of examples. \n",
        "\n",
        "For each batch: \n",
        "- we zero the gradients over the parameters from the last gradient calculation\n",
        "- insert the batch of text into the model to get predictions\n",
        "- as PyTorch loss functions cannot handle 3-dimensional predictions we reshape our predictions\n",
        "- calculate the loss and accuracy between the predicted tags and actual tags\n",
        "- call `backward` to calculate the gradients of the parameters w.r.t. the loss\n",
        "- take an optimizer `step` to update the parameters\n",
        "- add to the running total of loss and accuracy"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_QGjrEYppwmr"
      },
      "source": [
        "def train(model, iterator, optimizer, criterion, tag_pad_idx):\n",
        "    \n",
        "    epoch_loss = 0\n",
        "    epoch_acc = 0\n",
        "    \n",
        "    model.train()\n",
        "    \n",
        "    for batch in iterator:\n",
        "        \n",
        "        text = batch.text\n",
        "        tags = batch.udtags\n",
        "        \n",
        "        optimizer.zero_grad()\n",
        "        \n",
        "        #text = [sent len, batch size]\n",
        "        \n",
        "        predictions = model(text)\n",
        "        \n",
        "        #predictions = [sent len, batch size, output dim]\n",
        "        #tags = [sent len, batch size]\n",
        "        \n",
        "        predictions = predictions.view(-1, predictions.shape[-1])\n",
        "        tags = tags.view(-1)\n",
        "        \n",
        "        #predictions = [sent len * batch size, output dim]\n",
        "        #tags = [sent len * batch size]\n",
        "        \n",
        "        loss = criterion(predictions, tags)\n",
        "                \n",
        "        acc = categorical_accuracy(predictions, tags, tag_pad_idx)\n",
        "        \n",
        "        loss.backward()\n",
        "        \n",
        "        optimizer.step()\n",
        "        \n",
        "        epoch_loss += loss.item()\n",
        "        epoch_acc += acc.item()\n",
        "        \n",
        "    return epoch_loss / len(iterator), epoch_acc / len(iterator)"
      ],
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jkdJ4J-Gpwms"
      },
      "source": [
        "The `evaluate` function is similar to the `train` function, except with changes made so we don't update the model's parameters.\n",
        "\n",
        "`model.eval()` is used to put the model in evaluation mode, so dropout/batch-norm/etc. are turned off. \n",
        "\n",
        "The iteration loop is also wrapped in `torch.no_grad` to ensure we don't calculate any gradients. We also don't need to call `optimizer.zero_grad()` and `optimizer.step()`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QoxU16Sbpwms"
      },
      "source": [
        "def evaluate(model, iterator, criterion, tag_pad_idx):\n",
        "    \n",
        "    epoch_loss = 0\n",
        "    epoch_acc = 0\n",
        "    \n",
        "    model.eval()\n",
        "    \n",
        "    with torch.no_grad():\n",
        "    \n",
        "        for batch in iterator:\n",
        "\n",
        "            text = batch.text\n",
        "            tags = batch.udtags\n",
        "            \n",
        "            predictions = model(text)\n",
        "            \n",
        "            predictions = predictions.view(-1, predictions.shape[-1])\n",
        "            tags = tags.view(-1)\n",
        "            \n",
        "            loss = criterion(predictions, tags)\n",
        "            \n",
        "            acc = categorical_accuracy(predictions, tags, tag_pad_idx)\n",
        "\n",
        "            epoch_loss += loss.item()\n",
        "            epoch_acc += acc.item()\n",
        "        \n",
        "    return epoch_loss / len(iterator), epoch_acc / len(iterator)"
      ],
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UUnSxXfmpwmt"
      },
      "source": [
        "Next, we have a small function that tells us how long an epoch takes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aM2Mfyohpwmt"
      },
      "source": [
        "def epoch_time(start_time, end_time):\n",
        "    elapsed_time = end_time - start_time\n",
        "    elapsed_mins = int(elapsed_time / 60)\n",
        "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
        "    return elapsed_mins, elapsed_secs"
      ],
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HlWPogO8pwmv"
      },
      "source": [
        "Finally, we train our model!\n",
        "\n",
        "After each epoch we check if our model has achieved the best validation loss so far. If it has then we save the parameters of this model and we will use these \"best\" parameters to calculate performance over our test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W08Crfvzpwmw",
        "outputId": "b84a3cfe-8f92-423a-a26f-af3b6043800a"
      },
      "source": [
        "N_EPOCHS = 10\n",
        "\n",
        "best_valid_loss = float('inf')\n",
        "\n",
        "for epoch in range(N_EPOCHS):\n",
        "\n",
        "    start_time = time.time()\n",
        "    \n",
        "    train_loss, train_acc = train(model, train_iterator, optimizer, criterion, TAG_PAD_IDX)\n",
        "    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion, TAG_PAD_IDX)\n",
        "    \n",
        "    end_time = time.time()\n",
        "\n",
        "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
        "    \n",
        "    if valid_loss < best_valid_loss:\n",
        "        best_valid_loss = valid_loss\n",
        "        torch.save(model.state_dict(), 'tut1-model.pt')\n",
        "    \n",
        "    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
        "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
        "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch: 01 | Epoch Time: 0m 3s\n",
            "\tTrain Loss: 1.328 | Train Acc: 58.49%\n",
            "\t Val. Loss: 0.666 |  Val. Acc: 78.83%\n",
            "Epoch: 02 | Epoch Time: 0m 3s\n",
            "\tTrain Loss: 0.472 | Train Acc: 85.22%\n",
            "\t Val. Loss: 0.503 |  Val. Acc: 83.60%\n",
            "Epoch: 03 | Epoch Time: 0m 3s\n",
            "\tTrain Loss: 0.344 | Train Acc: 89.15%\n",
            "\t Val. Loss: 0.436 |  Val. Acc: 85.41%\n",
            "Epoch: 04 | Epoch Time: 0m 3s\n",
            "\tTrain Loss: 0.283 | Train Acc: 91.00%\n",
            "\t Val. Loss: 0.402 |  Val. Acc: 86.43%\n",
            "Epoch: 05 | Epoch Time: 0m 3s\n",
            "\tTrain Loss: 0.248 | Train Acc: 92.17%\n",
            "\t Val. Loss: 0.395 |  Val. Acc: 86.66%\n",
            "Epoch: 06 | Epoch Time: 0m 3s\n",
            "\tTrain Loss: 0.222 | Train Acc: 92.90%\n",
            "\t Val. Loss: 0.378 |  Val. Acc: 87.03%\n",
            "Epoch: 07 | Epoch Time: 0m 3s\n",
            "\tTrain Loss: 0.202 | Train Acc: 93.51%\n",
            "\t Val. Loss: 0.363 |  Val. Acc: 88.54%\n",
            "Epoch: 08 | Epoch Time: 0m 3s\n",
            "\tTrain Loss: 0.186 | Train Acc: 94.00%\n",
            "\t Val. Loss: 0.361 |  Val. Acc: 87.53%\n",
            "Epoch: 09 | Epoch Time: 0m 3s\n",
            "\tTrain Loss: 0.174 | Train Acc: 94.40%\n",
            "\t Val. Loss: 0.349 |  Val. Acc: 88.98%\n",
            "Epoch: 10 | Epoch Time: 0m 3s\n",
            "\tTrain Loss: 0.163 | Train Acc: 94.70%\n",
            "\t Val. Loss: 0.353 |  Val. Acc: 88.85%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QMKuvg9apwmw"
      },
      "source": [
        "We then load our \"best\" parameters and evaluate performance on the test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lWVYuxHqpwmx",
        "outputId": "e23c434d-6ce6-4233-df42-1794e329e0a1"
      },
      "source": [
        "model.load_state_dict(torch.load('tut1-model.pt'))\n",
        "\n",
        "test_loss, test_acc = evaluate(model, test_iterator, criterion, TAG_PAD_IDX)\n",
        "\n",
        "print(f'Test Loss: {test_loss:.3f} |  Test Acc: {test_acc*100:.2f}%')"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Test Loss: 0.366 |  Test Acc: 88.26%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FoRcN30cpwmx"
      },
      "source": [
        "## Inference\n",
        "\n",
        "88% accuracy looks pretty good, but let's see our model tag some actual sentences.\n",
        "\n",
        "We define a `tag_sentence` function that will:\n",
        "- put the model into evaluation mode\n",
        "- tokenize the sentence with spaCy if it is not a list\n",
        "- lowercase the tokens if the `Field` did\n",
        "- numericalize the tokens using the vocabulary\n",
        "- find out which tokens are not in the vocabulary, i.e. are `<unk>` tokens\n",
        "- convert the numericalized tokens into a tensor and add a batch dimension\n",
        "- feed the tensor into the model\n",
        "- get the predictions over the sentence\n",
        "- convert the predictions into readable tags\n",
        "\n",
        "As well as returning the tokens and tags, it also returns which tokens were `<unk>` tokens."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "czlCsTpApwmx"
      },
      "source": [
        "def tag_sentence(model, device, sentence, text_field, tag_field):\n",
        "    \n",
        "    model.eval()\n",
        "    \n",
        "    if isinstance(sentence, str):\n",
        "        nlp = spacy.load('en_core_web_sm')\n",
        "        tokens = [token.text for token in nlp(sentence)]\n",
        "    else:\n",
        "        tokens = [token for token in sentence]\n",
        "\n",
        "    if text_field.lower:\n",
        "        tokens = [t.lower() for t in tokens]\n",
        "        \n",
        "    numericalized_tokens = [text_field.vocab.stoi[t] for t in tokens]\n",
        "\n",
        "    unk_idx = text_field.vocab.stoi[text_field.unk_token]\n",
        "    \n",
        "    unks = [t for t, n in zip(tokens, numericalized_tokens) if n == unk_idx]\n",
        "    \n",
        "    token_tensor = torch.LongTensor(numericalized_tokens)\n",
        "    \n",
        "    token_tensor = token_tensor.unsqueeze(-1).to(device)\n",
        "         \n",
        "    predictions = model(token_tensor)\n",
        "    \n",
        "    top_predictions = predictions.argmax(-1)\n",
        "    \n",
        "    predicted_tags = [tag_field.vocab.itos[t.item()] for t in top_predictions]\n",
        "    \n",
        "    return tokens, predicted_tags, unks"
      ],
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3VG1mjbEpwmy"
      },
      "source": [
        "We'll get an already tokenized example from the training set and test our model's performance."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JX9dYDOhpwmy",
        "outputId": "ee9f7d6f-f558-43f2-ffcd-330e33f8cdc5"
      },
      "source": [
        "example_index = 1\n",
        "\n",
        "sentence = vars(train_data.examples[example_index])['text']\n",
        "actual_tags = vars(train_data.examples[example_index])['udtags']\n",
        "\n",
        "print(sentence)"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['[', 'this', 'killing', 'of', 'a', 'respected', 'cleric', 'will', 'be', 'causing', 'us', 'trouble', 'for', 'years', 'to', 'come', '.', ']']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1eAGWiI-pwmy"
      },
      "source": [
        "We can then use our `tag_sentence` function to get the tags. Notice how the tokens referring to subject of the sentence, the \"respected cleric\", are both `<unk>` tokens!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EJmb8kVTpwmy",
        "outputId": "da9fd84d-cc6c-430a-cb25-8f69a2f4c597"
      },
      "source": [
        "tokens, pred_tags, unks = tag_sentence(model, \n",
        "                                       device, \n",
        "                                       sentence, \n",
        "                                       TEXT, \n",
        "                                       UD_TAGS)\n",
        "\n",
        "print(unks)"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['respected', 'cleric']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2zAXE1KTpwmy"
      },
      "source": [
        "We can then check how well it did. Surprisingly, it got every token correct, including the two that were unknown tokens!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hixzgb6fpwmz",
        "outputId": "2e9849f7-009d-41b7-8951-0c931e1522b3"
      },
      "source": [
        "print(\"Pred. Tag\\tActual Tag\\tCorrect?\\tToken\\n\")\n",
        "\n",
        "for token, pred_tag, actual_tag in zip(tokens, pred_tags, actual_tags):\n",
        "    correct = '✔' if pred_tag == actual_tag else '✘'\n",
        "    print(f\"{pred_tag}\\t\\t{actual_tag}\\t\\t{correct}\\t\\t{token}\")"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Pred. Tag\tActual Tag\tCorrect?\tToken\n",
            "\n",
            "PUNCT\t\tPUNCT\t\t✔\t\t[\n",
            "DET\t\tDET\t\t✔\t\tthis\n",
            "NOUN\t\tNOUN\t\t✔\t\tkilling\n",
            "ADP\t\tADP\t\t✔\t\tof\n",
            "DET\t\tDET\t\t✔\t\ta\n",
            "ADJ\t\tADJ\t\t✔\t\trespected\n",
            "NOUN\t\tNOUN\t\t✔\t\tcleric\n",
            "AUX\t\tAUX\t\t✔\t\twill\n",
            "AUX\t\tAUX\t\t✔\t\tbe\n",
            "VERB\t\tVERB\t\t✔\t\tcausing\n",
            "PRON\t\tPRON\t\t✔\t\tus\n",
            "NOUN\t\tNOUN\t\t✔\t\ttrouble\n",
            "ADP\t\tADP\t\t✔\t\tfor\n",
            "NOUN\t\tNOUN\t\t✔\t\tyears\n",
            "PART\t\tPART\t\t✔\t\tto\n",
            "VERB\t\tVERB\t\t✔\t\tcome\n",
            "PUNCT\t\tPUNCT\t\t✔\t\t.\n",
            "PUNCT\t\tPUNCT\t\t✔\t\t]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o2o_BtQspwm2"
      },
      "source": [
        "Let's now make up our own sentence and see how well the model does.\n",
        "\n",
        "Our example sentence below has every token within the model's vocabulary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qEO3Wy48pwm5",
        "outputId": "f0191cff-3064-4738-f5a8-6f1197657087"
      },
      "source": [
        "sentence = 'The Queen will deliver a speech about the conflict in North Korea at 1pm tomorrow.'\n",
        "\n",
        "tokens, tags, unks = tag_sentence(model, \n",
        "                                  device, \n",
        "                                  sentence, \n",
        "                                  TEXT, \n",
        "                                  UD_TAGS)\n",
        "\n",
        "print(unks)"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oMXAdlzCpwm5"
      },
      "source": [
        "Looking at the sentence it seems like it gave sensible tags to every token!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qwieKTFxpwm6",
        "outputId": "5d09b568-038d-4f88-830e-8883549f977e"
      },
      "source": [
        "print(\"Pred. Tag\\tToken\\n\")\n",
        "\n",
        "for token, tag in zip(tokens, tags):\n",
        "    print(f\"{tag}\\t\\t{token}\")"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Pred. Tag\tToken\n",
            "\n",
            "DET\t\tthe\n",
            "NOUN\t\tqueen\n",
            "AUX\t\twill\n",
            "VERB\t\tdeliver\n",
            "DET\t\ta\n",
            "NOUN\t\tspeech\n",
            "ADP\t\tabout\n",
            "DET\t\tthe\n",
            "NOUN\t\tconflict\n",
            "ADP\t\tin\n",
            "PROPN\t\tnorth\n",
            "PROPN\t\tkorea\n",
            "ADP\t\tat\n",
            "NUM\t\t1\n",
            "NOUN\t\tpm\n",
            "NOUN\t\ttomorrow\n",
            "PUNCT\t\t.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KajaGMzPpwm6"
      },
      "source": [
        "We've now seen how to implement PoS tagging with PyTorch and TorchText! \n",
        "\n",
        "The BiLSTM isn't a state-of-the-art model, in terms of performance, but is a strong baseline for PoS tasks and is a good tool to have in your arsenal."
      ]
    }
  ]
}