lstm_and_gru/lstm_and_gru_flair.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "lstm_and_gru_flair.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "ZIC0nEEjEjGG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# LSTM\n",
        "\n",
        "In this notebook, we will go through basics of LSTM and Sentiment Analyser API of flair using pretrained embeddings like word2vec and flair embeddings on IMDB dataset. \n",
        "\n",
        "Here we will use [Flair](https://github.com/zalandoresearch/flair  \"flair\").\n",
        "\n",
        "\n",
        "Everything is explained in-detail in [blog post](https://dudeperf3ct.github.io/lstm/gru/nlp/2019/01/28/Force-of-LSTM-and-GRU/). This is notebook which replicates the result of blog and runs in colab. Enjoy!\n",
        "\n",
        "\n",
        "Let's see at [nlpprogess](http://nlpprogress.com/english/sentiment_analysis.html) what is the current state-of-the-art in sentiment analysis.\n",
        "\n",
        "\n",
        "Model | Accuracy | Paper |\n",
        "----- | -------- | ------|\n",
        "ULMFit| 95.4     | [Universal Language Model Fine-tuning for Text Classification](https://arxiv.org/abs/1801.06146)|\n",
        "Block-sparse LSTM| 94.99 | [GPU Kernels for Block-Sparse Weights](https://s3-us-west-2.amazonaws.com/openai-assets/blocksparse/blocksparsepaper.pdf)|\n",
        "oh-LSTM | 94.1 | [Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddings](https://arxiv.org/abs/1602.02373) |\n",
        "Virtual adversarial training  | 94.1 | [Adversarial Training Methods for Semi-Supervised Text Classification](https://arxiv.org/abs/1605.07725) |\n",
        "BCN+Char+CoVe | 91.8 | [Learned in Translation: Contextualized Word Vectors](https://arxiv.org/abs/1708.00107) |\n",
        "\n",
        "\n",
        "#### Run in Colab\n",
        "\n",
        "You can run this notebook in google colab.\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dudeperf3ct/DL_notebooks/blob/master/lstm_and_gru/lstm_and_gru_flair.ipynb)\n"
      ]
    },
    {
      "metadata": {
        "id": "oYAs-iIiT5wG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Download libraries"
      ]
    },
    {
      "metadata": {
        "id": "pZRBa3ZSyJ6I",
        "colab_type": "code",
        "outputId": "389e1e1a-854e-4579-e325-f8d5b0dd12be",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1216
        }
      },
      "cell_type": "code",
      "source": [
        "! pip install flair"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting flair\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/44/54/76374f9a448ca765446502e7f2bb53c976e9c055102290fe6f8b0b038b37/flair-0.4.1.tar.gz (78kB)\n",
            "\u001b[K    100% |████████████████████████████████| 81kB 3.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: torch>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from flair) (1.0.1.post2)\n",
            "Requirement already satisfied: gensim>=3.4.0 in /usr/local/lib/python3.6/dist-packages (from flair) (3.6.0)\n",
            "Requirement already satisfied: tqdm>=4.26.0 in /usr/local/lib/python3.6/dist-packages (from flair) (4.28.1)\n",
            "Collecting segtok>=1.5.7 (from flair)\n",
            "  Downloading https://files.pythonhosted.org/packages/1d/59/6ed78856ab99d2da04084b59e7da797972baa0efecb71546b16d48e49d9b/segtok-1.5.7.tar.gz\n",
            "Requirement already satisfied: matplotlib>=2.2.3 in /usr/local/lib/python3.6/dist-packages (from flair) (3.0.2)\n",
            "Collecting mpld3>=0.3 (from flair)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/91/95/a52d3a83d0a29ba0d6898f6727e9858fe7a43f6c2ce81a5fe7e05f0f4912/mpld3-0.3.tar.gz (788kB)\n",
            "\u001b[K    100% |████████████████████████████████| 798kB 8.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: sklearn in /usr/local/lib/python3.6/dist-packages (from flair) (0.0)\n",
            "Collecting sqlitedict>=1.6.0 (from flair)\n",
            "  Downloading https://files.pythonhosted.org/packages/0f/1c/c757b93147a219cf1e25cef7e1ad9b595b7f802159493c45ce116521caff/sqlitedict-1.6.0.tar.gz\n",
            "Collecting deprecated>=1.2.4 (from flair)\n",
            "  Downloading https://files.pythonhosted.org/packages/9f/7a/003fa432f1e45625626549726c2fbb7a29baa764e9d1fdb2323a5d779f8a/Deprecated-1.2.5-py2.py3-none-any.whl\n",
            "Requirement already satisfied: hyperopt>=0.1.1 in /usr/local/lib/python3.6/dist-packages (from flair) (0.1.2)\n",
            "Collecting pytorch-pretrained-bert>=0.6.1 (from flair)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/5d/3c/d5fa084dd3a82ffc645aba78c417e6072ff48552e3301b1fa3bd711e03d4/pytorch_pretrained_bert-0.6.1-py3-none-any.whl (114kB)\n",
            "\u001b[K    100% |████████████████████████████████| 122kB 20.5MB/s \n",
            "\u001b[?25hCollecting bpemb>=0.2.9 (from flair)\n",
            "  Downloading https://files.pythonhosted.org/packages/fe/d5/229f4d1a8de7a08a34d3b205bf82f6487f3b624c665d3de58e797fba2a9f/bpemb-0.2.11-py3-none-any.whl\n",
            "Requirement already satisfied: regex==2018.1.10 in /usr/local/lib/python3.6/dist-packages (from flair) (2018.1.10)\n",
            "Requirement already satisfied: numpy>=1.11.3 in /usr/local/lib/python3.6/dist-packages (from gensim>=3.4.0->flair) (1.14.6)\n",
            "Requirement already satisfied: six>=1.5.0 in /usr/local/lib/python3.6/dist-packages (from gensim>=3.4.0->flair) (1.11.0)\n",
            "Requirement already satisfied: scipy>=0.18.1 in /usr/local/lib/python3.6/dist-packages (from gensim>=3.4.0->flair) (1.1.0)\n",
            "Requirement already satisfied: smart-open>=1.2.1 in /usr/local/lib/python3.6/dist-packages (from gensim>=3.4.0->flair) (1.8.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2.3->flair) (0.10.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2.3->flair) (1.0.1)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2.3->flair) (2.3.1)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.2.3->flair) (2.5.3)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from sklearn->flair) (0.20.2)\n",
            "Requirement already satisfied: wrapt<2,>=1 in /usr/local/lib/python3.6/dist-packages (from deprecated>=1.2.4->flair) (1.11.1)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from hyperopt>=0.1.1->flair) (0.16.0)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.6/dist-packages (from hyperopt>=0.1.1->flair) (2.2)\n",
            "Requirement already satisfied: pymongo in /usr/local/lib/python3.6/dist-packages (from hyperopt>=0.1.1->flair) (3.7.2)\n",
            "Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert>=0.6.1->flair) (1.9.103)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert>=0.6.1->flair) (2.18.4)\n",
            "Collecting sentencepiece (from bpemb>=0.2.9->flair)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ed/86/aeb647d3ccb924997ea0d05b457d82648a1da57c471688ffbbd69650d9bc/sentencepiece-0.1.8-cp36-cp36m-manylinux1_x86_64.whl (1.0MB)\n",
            "\u001b[K    100% |████████████████████████████████| 1.0MB 3.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: boto>=2.32 in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim>=3.4.0->flair) (2.49.0)\n",
            "Requirement already satisfied: bz2file in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim>=3.4.0->flair) (0.98)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from kiwisolver>=1.0.1->matplotlib>=2.2.3->flair) (40.8.0)\n",
            "Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx->hyperopt>=0.1.1->flair) (4.3.2)\n",
            "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch-pretrained-bert>=0.6.1->flair) (0.9.4)\n",
            "Requirement already satisfied: s3transfer<0.3.0,>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch-pretrained-bert>=0.6.1->flair) (0.2.0)\n",
            "Requirement already satisfied: botocore<1.13.0,>=1.12.103 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch-pretrained-bert>=0.6.1->flair) (1.12.103)\n",
            "Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert>=0.6.1->flair) (1.22)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert>=0.6.1->flair) (2018.11.29)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert>=0.6.1->flair) (3.0.4)\n",
            "Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert>=0.6.1->flair) (2.6)\n",
            "Requirement already satisfied: docutils>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.13.0,>=1.12.103->boto3->pytorch-pretrained-bert>=0.6.1->flair) (0.14)\n",
            "Building wheels for collected packages: flair, segtok, mpld3, sqlitedict\n",
            "  Building wheel for flair (setup.py) ... \u001b[?25ldone\n",
            "\u001b[?25h  Stored in directory: /root/.cache/pip/wheels/70/55/6b/c12cf58209b8346f653a04f37dd8f607ab0e85a26238a23420\n",
            "  Building wheel for segtok (setup.py) ... \u001b[?25ldone\n",
            "\u001b[?25h  Stored in directory: /root/.cache/pip/wheels/15/ee/a8/6112173f1386d33eebedb3f73429cfa41a4c3084556bcee254\n",
            "  Building wheel for mpld3 (setup.py) ... \u001b[?25ldone\n",
            "\u001b[?25h  Stored in directory: /root/.cache/pip/wheels/c0/47/fb/8a64f89aecfe0059830479308ad42d62e898a3e3cefdf6ba28\n",
            "  Building wheel for sqlitedict (setup.py) ... \u001b[?25ldone\n",
            "\u001b[?25h  Stored in directory: /root/.cache/pip/wheels/bd/57/d3/907c3ee02d35e66f674ad0106e61f06eeeb98f6ee66a6cc3fe\n",
            "Successfully built flair segtok mpld3 sqlitedict\n",
            "Installing collected packages: segtok, mpld3, sqlitedict, deprecated, pytorch-pretrained-bert, sentencepiece, bpemb, flair\n",
            "Successfully installed bpemb-0.2.11 deprecated-1.2.5 flair-0.4.1 mpld3-0.3 pytorch-pretrained-bert-0.6.1 segtok-1.5.7 sentencepiece-0.1.8 sqlitedict-1.6.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "CPX0rk2CUG-f",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## IMDB data\n",
        "\n",
        "Code Adapted from: [link](https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/1%20-%20Simple%20Sentiment%20Analysis.ipynb)"
      ]
    },
    {
      "metadata": {
        "id": "pNkINkerEmzY",
        "colab_type": "code",
        "outputId": "20dd6aef-3f27-4e7c-c4ce-e29541704628",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "from flair.data_fetcher import NLPTaskDataFetcher\n",
        "from flair.embeddings import WordEmbeddings, FlairEmbeddings, DocumentRNNEmbeddings\n",
        "from flair.models import TextClassifier\n",
        "from flair.trainers import ModelTrainer\n",
        "from flair.data_fetcher import NLPTaskDataFetcher, NLPTask\n",
        "from pathlib import Path\n",
        "from flair.data import Sentence\n",
        "import os"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "XN-r2zOZn80b",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Flair"
      ]
    },
    {
      "metadata": {
        "id": "S2YEh-3IpU14",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Data Preprocessing Step\n",
        "\n",
        "Flair’s classification dataset format is based on the Facebook’s FastText format. The format requires one or multiple labels to be defined at the beginning of each line starting with the prefix ` __label__`. \n",
        "\n",
        "The format is as follows:\n",
        "\n",
        "\n",
        "```\n",
        "__label__<class_1> <text>\n",
        "__label__<class_2> <text>\n",
        "```\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "-bvCi18CYSt0",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# data = pd.read_csv(csv_filename)\n",
        "\n",
        "# data = data[['v1', 'v2']].rename(columns={\"v1\":\"label\", \"v2\":\"text\"})\n",
        " \n",
        "# data['label'] = '__label__' + data['label'].astype(str)\n",
        "\n",
        "# data.iloc[0:int(len(data)*0.8)].to_csv('train.csv', sep='\\t', index = False, header = False)\n",
        "# data.iloc[int(len(data)*0.8):int(len(data)*0.9)].to_csv('test.csv', sep='\\t', index = False, header = False)\n",
        "# data.iloc[int(len(data)*0.9):].to_csv('dev.csv', sep='\\t', index = False, header = False)\n",
        "\n",
        "#corpus = NLPTaskDataFetcher.load_classification_corpus(Path('./'), test_file='test.csv', dev_file='dev.csv', train_file='train.csv')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "w-oFaWUnp1mX",
        "colab_type": "code",
        "outputId": "5fb88d90-05ed-4fcc-ca56-af7f5b603683",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        }
      },
      "cell_type": "code",
      "source": [
        "# we will take a shortcut and download preprocessed dataset of imdb in required format\n",
        "# We will downsample data: 70% of original --> NOT ENOUGH RAM\n",
        "\n",
        "corpus = NLPTaskDataFetcher.load_corpus(NLPTask.IMDB).downsample(0.7)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2019-02-23 00:21:33,175 Reading data from /root/.flair/datasets/imdb\n",
            "2019-02-23 00:21:33,177 Train: /root/.flair/datasets/imdb/train.txt\n",
            "2019-02-23 00:21:33,179 Dev: None\n",
            "2019-02-23 00:21:33,180 Test: /root/.flair/datasets/imdb/test.txt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "e9xw7fCIfuBj",
        "colab_type": "code",
        "outputId": "e1c6b590-6b33-404e-b2eb-5126c4302e3d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 880
        }
      },
      "cell_type": "code",
      "source": [
        "stats = corpus.obtain_statistics()\n",
        "print(stats)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{\n",
            "    \"TRAIN\": {\n",
            "        \"dataset\": \"TRAIN\",\n",
            "        \"total_number_of_documents\": 2250,\n",
            "        \"number_of_documents_per_class\": {\n",
            "            \"pos\": 1130,\n",
            "            \"neg\": 1120\n",
            "        },\n",
            "        \"number_of_tokens_per_tag\": {},\n",
            "        \"number_of_tokens\": {\n",
            "            \"total\": 609753,\n",
            "            \"min\": 18,\n",
            "            \"max\": 1269,\n",
            "            \"avg\": 271.0013333333333\n",
            "        }\n",
            "    },\n",
            "    \"TEST\": {\n",
            "        \"dataset\": \"TEST\",\n",
            "        \"total_number_of_documents\": 2500,\n",
            "        \"number_of_documents_per_class\": {\n",
            "            \"pos\": 1250,\n",
            "            \"neg\": 1250\n",
            "        },\n",
            "        \"number_of_tokens_per_tag\": {},\n",
            "        \"number_of_tokens\": {\n",
            "            \"total\": 687743,\n",
            "            \"min\": 23,\n",
            "            \"max\": 1640,\n",
            "            \"avg\": 275.0972\n",
            "        }\n",
            "    },\n",
            "    \"DEV\": {\n",
            "        \"dataset\": \"DEV\",\n",
            "        \"total_number_of_documents\": 250,\n",
            "        \"number_of_documents_per_class\": {\n",
            "            \"neg\": 130,\n",
            "            \"pos\": 120\n",
            "        },\n",
            "        \"number_of_tokens_per_tag\": {},\n",
            "        \"number_of_tokens\": {\n",
            "            \"total\": 65695,\n",
            "            \"min\": 33,\n",
            "            \"max\": 1168,\n",
            "            \"avg\": 262.78\n",
            "        }\n",
            "    }\n",
            "}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "mn9Zt9CbYXcd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Glove"
      ]
    },
    {
      "metadata": {
        "id": "mAA35t8in-hE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# init embedding\n",
        "glove_embedding = [WordEmbeddings('glove')]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "fRTdcgBDn-o6",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "document_embeddings = DocumentRNNEmbeddings(glove_embedding, hidden_size=128, rnn_layers=1,\n",
        "                                            reproject_words=True, reproject_words_dimension=256,\n",
        "                                            bidirectional=False, dropout=0.5, word_dropout=0.2,\n",
        "                                            rnn_type='LSTM')\n",
        "\n",
        "classifier = TextClassifier(document_embeddings, label_dictionary=corpus.make_label_dictionary(), multi_label=False)\n",
        "\n",
        "trainer = ModelTrainer(classifier, corpus)\n",
        "\n",
        "trainer.train('./', max_epochs=10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "E-8PPXm7n-w-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "classifier = TextClassifier.load_from_file('./best-model.pt')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pwraILQAn-l1",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "sentence = Sentence(\"This film is terrible\")\n",
        "classifier.predict(sentence)\n",
        "\n",
        "# print sentence with predicted labels\n",
        "print('Sentence above is: ', sentence.labels)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "RYr7-czHTmFp",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "sentence = Sentence(\"This film is great\")\n",
        "classifier.predict(sentence)\n",
        "\n",
        "# print sentence with predicted labels\n",
        "print('Sentence above is: ', sentence.labels)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ly6dUZj6RTCc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Custom Classifier\n",
        "\n",
        "\n",
        "To train a custom text classifier we will first need a labelled dataset. Flair’s classification dataset format is based on the Facebook’s FastText format. The format requires one or multiple labels to be defined at the beginning of each line starting with the prefix ` __label__`. The format is as follows:\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "__label__<class_1> <text>\n",
        "__label__<class_2> <text>\n",
        "```\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "QYDRKguXV63l",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "word_embeddings = [WordEmbeddings('glove'), FlairEmbeddings('news-forward-fast'), FlairEmbeddings('news-backward-fast')]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "uLg13uguRYB8",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "document_embeddings = DocumentRNNEmbeddings(word_embeddings, hidden_size=512, rnn_layers=1,\n",
        "                                            reproject_words=True, reproject_words_dimension=256,\n",
        "                                            bidirectional=False, dropout=0.5, word_dropout=0.2,\n",
        "                                            rnn_type='LSTM')\n",
        "\n",
        "classifier = TextClassifier(document_embeddings, label_dictionary=corpus.make_label_dictionary(), multi_label=False)\n",
        "\n",
        "trainer = ModelTrainer(classifier, corpus)\n",
        "\n",
        "trainer.train('./', max_epochs=10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "dvuIVBPpRYGF",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "classifier = TextClassifier.load_from_file('./best-model.pt')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_m8ZDf5DRX83",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "sentence = Sentence(\"This film is terrible\")\n",
        "classifier.predict(sentence)\n",
        "\n",
        "# print sentence with predicted labels\n",
        "print('Sentence above is: ', sentence.labels)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "fpfGzf91RX_0",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "sentence = Sentence(\"This film is great\")\n",
        "classifier.predict(sentence)\n",
        "\n",
        "# print sentence with predicted labels\n",
        "print('Sentence above is: ', sentence.labels)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qwZF5eY1pWuW",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Classifier API"
      ]
    },
    {
      "metadata": {
        "id": "7Iq_M9XJn-d6",
        "colab_type": "code",
        "outputId": "9b8ab579-2790-443d-b542-133b880b45cf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 162
        }
      },
      "cell_type": "code",
      "source": [
        "from flair.models import TextClassifier\n",
        "from flair.data import Sentence\n",
        "\n",
        "classifier = TextClassifier.load('en-sentiment')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2019-03-03 19:03:52,956 https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/models-v0.4/TEXT-CLASSIFICATION_imdb/imdb.pt not found in cache, downloading to /tmp/tmpwckm7dng\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 2794252905/2794252905 [06:35<00:00, 7068826.37B/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "2019-03-03 19:10:29,481 copying /tmp/tmpwckm7dng to cache at /root/.flair/models/imdb.pt\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "2019-03-03 19:10:50,402 removing temp file /tmp/tmpwckm7dng\n",
            "2019-03-03 19:10:50,409 loading file /root/.flair/models/imdb.pt\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/torch/serialization.py:542: DeprecationWarning: Call to deprecated class DocumentLSTMEmbeddings. (The functionality of this class is moved to 'DocumentRNNEmbeddings') -- Deprecated since version 0.4.\n",
            "  result = unpickler.load()\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "aVEPzdazphq2",
        "colab_type": "code",
        "outputId": "7478149c-193c-43f3-f53e-fe9a40c35886",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "sentence = Sentence(\"This has to be one of the biggest misfires ever...the script was nice and could have ended a lot better.the actors should have played better and maybe then i would have given this movie a slightly better grade. maybe Hollywood should remake this movie with some little better actors and better director.sorry guys for disappointment but the movie is bad.<br /><br />If i had to re-watch it it would be like torture. I don't want to spoil everyone's opinion with mine so..my advice is watch the movie first..see if u like it and after vote(do not vote before you watch it ! ) and by the way... Have fun watching it ! Don't just peek...watch it 'till the end :))))))))) !!\")\n",
        "classifier.predict(sentence)\n",
        "\n",
        "# print sentence with predicted labels\n",
        "print('Sentence above is: ', sentence.labels)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sentence above is:  [NEGATIVE (1.0)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "-0DW8NALphx9",
        "colab_type": "code",
        "outputId": "1644a558-f2ad-4800-844e-bda5705896d6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "sentence = Sentence(\"Five medical students (Kevin Bacon, David Labraccio; William Baldwin, Dr. Joe Hurley; Oliver Platt, Randy Steckle; Julia Roberts, Dr. Rachel Mannus; Kiefer Sutherland, Nelson) experiment with clandestine near death & afterlife experiences, (re)searching for medical & personal enlightenment. One by one, each medical student's heart is stopped, then revived.<br /><br />Under temporary death spells each experiences bizarre visions, including forgotten childhood memories. Their flashbacks are like children's nightmares. The revived students are disturbed by remembering regretful acts they had committed or had done against them. As they experience afterlife, they bring real life experiences back into the present. As they continue to experiment, their remembrances dramatically intensify; so much so, some are physically overcome. Thus, they probe & transcend deeper into the death-afterlife experiences attempting to find a cure.<br /><br />Even though the DVD was released in 2007, this motion picture was released in 1990. Therefore, Kevin Bacon, William Baldwin, Julia Roberts & Kiefer Sutherland were in the early stages of their adult acting careers. Besides the plot being extremely intriguing, the suspense building to a dramatic climax & the script being tight & convincing, all of the young actors make \\\"Flatliners,\\\" what is now an all-star cult semi-sci-fi suspense. Who knew 17 years ago that the film careers of this young group of actors would skyrocket? I suspect that director Joel Schumacher did.\")\n",
        "classifier.predict(sentence)\n",
        "\n",
        "# print sentence with predicted labels\n",
        "print('Sentence above is: ', sentence.labels)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sentence above is:  [POSITIVE (1.0)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "lggSrB_fV_nR",
        "colab_type": "code",
        "outputId": "91f2316d-b955-487b-aefd-e331e9dcdd3b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "sentence = Sentence(\"A very accurate depiction of small time mob life filmed in New Jersey. The story, characters and script are believable but the acting drops the ball. Still, it's worth watching, especially for the strong images, some still with me even though I first viewed this 25 years ago.<br /><br />A young hood steps up and starts doing bigger things (tries to) but these things keep going wrong, leading the local boss to suspect that his end is being skimmed off, not a good place to be if you enjoy your health, or life.<br /><br />This is the film that introduced Joe Pesce to Martin Scorsese. Also present is that perennial screen wise guy, Frank Vincent. Strong on characterizations and visuals. Sound muddled and much of the acting is amateurish, but a great story.\")\n",
        "classifier.predict(sentence)\n",
        "\n",
        "# print sentence with predicted labels\n",
        "print('Sentence above is: ', sentence.labels)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sentence above is:  [POSITIVE (1.0)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "SrV1iAIaJbHK",
        "colab_type": "code",
        "outputId": "820eb832-123a-410f-f3e6-8d4c45f9b27d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "sentence = Sentence(\"Afraid of the Dark left me with the impression that several different screenplays were written, all too short for a feature length film, then spliced together clumsily into this Frankenstein's monster.<br /><br />At his best, the protagonist, Lucas, is creepy. As hard as it is to draw a bead on the secondary characters, they're far more sympathetic.<br /><br />Afraid of the Dark could have achieved mediocrity had it taken just one approach and seen it through -- and had it made Lucas simply psychotic and confused instead of ghoulish and off-putting. I wanted to see him packed off into an asylum so the rest of the characters could have a normal life.\")\n",
        "classifier.predict(sentence)\n",
        "\n",
        "# print sentence with predicted labels\n",
        "print('Sentence above is: ', sentence.labels)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sentence above is:  [NEGATIVE (1.0)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "EJg7wE9OJbly",
        "colab_type": "code",
        "outputId": "8bec5317-7646-4e34-f575-ef7905fb8de5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "sentence = Sentence(\"This movie is a disaster within a disaster film. It is full of great action scenes, which are only meaningful if you throw away all sense of reality. Let's see, word to the wise, lava burns you; steam burns you. You can't stand next to lava. Diverting a minor lava flow is difficult, let alone a significant one. Scares me to think that some might actually believe what they saw in this movie.<br /><br />Even worse is the significant amount of talent that went into making this film. I mean the acting is actually very good. The effects are above average. Hard to believe somebody read the scripts for this and allowed all this talent to be wasted. I guess my suggestion would be that if this movie is about to start on TV ... look away! It is like a train wreck: it is so awful that once you know what is coming, you just have to watch. Look away and spend your time on more meaningful content.\")\n",
        "classifier.predict(sentence)\n",
        "\n",
        "# print sentence with predicted labels\n",
        "print('Sentence above is: ', sentence.labels)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sentence above is:  [NEGATIVE (1.0)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "GwC4xIOLJktw",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}