labs/lab5_toxic/toxic_solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "nbgrader": {
     "grade": false,
     "locked": false,
     "solution": false
    }
   },
   "source": [
    "# Lab assignment: detecting toxic comments with Deep Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "locked": false,
     "solution": false
    }
   },
   "source": [
    "<img src=\"img/trolling.gif\" style=\"width:600px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "locked": false,
     "solution": false
    }
   },
   "source": [
    "In this assignment we will analyze text comments appearing in social media, trying to automatically detect those of toxic content: insults, threats, obscene comments, ... to do so we will make use of word embeddings and convolutional and recurrent neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "locked": false,
     "solution": false
    }
   },
   "source": [
    "## Guidelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "locked": false,
     "solution": false
    }
   },
   "source": [
    "Throughout this notebook you will find empty cells that you will need to fill with your own code. Follow the instructions in the notebook and pay special attention to the following symbols.\n",
    "\n",
    "<table>\n",
    " <tr><td width=\"80\"><img src=\"img/question.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">You will need to solve a question by writing your own code or answer in the cell immediately below, or in a different file as instructed.</td></tr>\n",
    " <tr><td width=\"80\"><img src=\"img/exclamation.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">This is a hint or useful observation that can help you solve this assignment.</td></tr>\n",
    " <tr><td width=\"80\"><img src=\"img/pro.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">This is an advanced and voluntary exercise that can help you gain a deeper knowledge into the topic. Good luck!</td></tr>\n",
    "</table>\n",
    "\n",
    "To avoid missing packages and compatibility issues you should run this notebook under one of the [recommended Deep Learning environment files](https://github.com/albarji/teaching-environments/tree/master/deeplearning)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code will embed any plots into the notebook instead of generating a new window:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, if you need any help on the usage of a Python function you can place the writing cursor over its name and press Caps+Shift to produce a pop-out with related documentation. This will only work inside code cells. \n",
    "\n",
    "Let's go!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Keras library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab we will make use of the [keras](http://keras.io/) Deep Learning library for Python. This library allows building several kinds of shallow and deep networks, following either a sequential or a graph architecture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data loading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data is provided as two separate files, one with texts for training the model and another one for testing. Both files are available in compressed form under the *data* folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    " <tr><td width=\"80\"><img src=\"img/question.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">\n",
    "     Load the data into two <a href=https://pandas.pydata.org/>pandas</a> Dataframes, <b>train</b> for the training data and <b>test</b> for the test data.\n",
    " </td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "####### INSERT YOUR CODE HERE\n",
    "import pandas as pd\n",
    "train = pd.read_csv(\"data/toxic_train.csv.zip\", index_col=\"id\")\n",
    "test = pd.read_csv(\"data/toxic_test.csv.zip\", index_col=\"id\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have loaded the data properly, you should be able to visualize the first rows of each data set as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>comment_text</th>\n",
       "      <th>toxic</th>\n",
       "      <th>severe_toxic</th>\n",
       "      <th>obscene</th>\n",
       "      <th>threat</th>\n",
       "      <th>insult</th>\n",
       "      <th>identity_hate</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>e0fdfd98c66fb643</th>\n",
       "      <td>\"\\n\\n Huggle not working \\n\\nHi Gurch. There i...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1864753b5fb6c9a3</th>\n",
       "      <td>Mossad actually.  I know where you live.</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ce1db53fb22d399c</th>\n",
       "      <td>REDIRECT Talk:UFC Fight Night: Belfort vs. Hen...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fed4f08d59399398</th>\n",
       "      <td>\"\\n\\nUPA IRC\\nWhat about 19:00 UTC?  e  | ταλκ \"</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>06e7f93938ad9e72</th>\n",
       "      <td>\"\\nI've re-added your information, together wi...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                       comment_text  toxic  \\\n",
       "id                                                                           \n",
       "e0fdfd98c66fb643  \"\\n\\n Huggle not working \\n\\nHi Gurch. There i...      0   \n",
       "1864753b5fb6c9a3           Mossad actually.  I know where you live.      0   \n",
       "ce1db53fb22d399c  REDIRECT Talk:UFC Fight Night: Belfort vs. Hen...      0   \n",
       "fed4f08d59399398   \"\\n\\nUPA IRC\\nWhat about 19:00 UTC?  e  | ταλκ \"      0   \n",
       "06e7f93938ad9e72  \"\\nI've re-added your information, together wi...      0   \n",
       "\n",
       "                  severe_toxic  obscene  threat  insult  identity_hate  \n",
       "id                                                                      \n",
       "e0fdfd98c66fb643             0        0       0       0              0  \n",
       "1864753b5fb6c9a3             0        0       0       0              0  \n",
       "ce1db53fb22d399c             0        0       0       0              0  \n",
       "fed4f08d59399398             0        0       0       0              0  \n",
       "06e7f93938ad9e72             0        0       0       0              0  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>comment_text</th>\n",
       "      <th>toxic</th>\n",
       "      <th>severe_toxic</th>\n",
       "      <th>obscene</th>\n",
       "      <th>threat</th>\n",
       "      <th>insult</th>\n",
       "      <th>identity_hate</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dd2dcd01f0536e53</th>\n",
       "      <td>\"\\nEdit:  Please stated the basis for your acc...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b20b1e8381e306f8</th>\n",
       "      <td>Wikipedia is a repetition of the old joke that...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54a933831401b9b2</th>\n",
       "      <td>And the same in Serbian:ђе and Croatian:če and...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e7a4575dba88d9f3</th>\n",
       "      <td>\"\\n\\n Congrats... You gave an awesome answer i...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>452d5cc2ccd1d611</th>\n",
       "      <td>Wiki users controlling information \\n\\nDeliber...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                       comment_text  toxic  \\\n",
       "id                                                                           \n",
       "dd2dcd01f0536e53  \"\\nEdit:  Please stated the basis for your acc...      0   \n",
       "b20b1e8381e306f8  Wikipedia is a repetition of the old joke that...      0   \n",
       "54a933831401b9b2  And the same in Serbian:ђе and Croatian:če and...      0   \n",
       "e7a4575dba88d9f3  \"\\n\\n Congrats... You gave an awesome answer i...      0   \n",
       "452d5cc2ccd1d611  Wiki users controlling information \\n\\nDeliber...      0   \n",
       "\n",
       "                  severe_toxic  obscene  threat  insult  identity_hate  \n",
       "id                                                                      \n",
       "dd2dcd01f0536e53             0        0       0       0              0  \n",
       "b20b1e8381e306f8             0        0       0       0              0  \n",
       "54a933831401b9b2             0        0       0       0              0  \n",
       "e7a4575dba88d9f3             0        0       0       0              0  \n",
       "452d5cc2ccd1d611             0        0       0       0              0  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the data files include a column *comment_text* with the text we must classify, and 6 additional columns with the kinds of toxicity that might be present in a comment: *toxic*, *severe_toxic*, *obscene*, *threat*, *insult* and *identity_hate*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can't introduce text directly into the network, so we will have to tranform it to a vector representation. To do so, we will first **tokenize** the text into words (or tokens), and assign a unique identifier to each word found in the text. Doing this will allow us to perform the encoding. We can do this easily by making use of the **Tokenizer** class in keras:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Tokenizer offers convenient methods to split texts down to tokens. At construction time we need to supply the Tokenizer the maximum number of different words we are willing to represent. If our texts have greater word variety than this number, the least frequent words will be discarded. We will choose a number large enough for our purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "maxwords = 15000\n",
    "tokenizer = Tokenizer(num_words = maxwords)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now need to **fit** the Tokenizer to the training texts, which can be done as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.fit_on_texts(train[\"comment_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After fitting the Tokenizer we can see a report of the number of times each word has been found in the input texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('huggle', 51),\n",
       "             ('not', 70441),\n",
       "             ('working', 1506),\n",
       "             ('hi', 5976),\n",
       "             ('gurch', 16),\n",
       "             ('there', 23945),\n",
       "             ('is', 132304),\n",
       "             ('a', 162211),\n",
       "             ('discussion', 6812),\n",
       "             ('over', 6172),\n",
       "             ('at', 29842),\n",
       "             ('wp', 8058),\n",
       "             ('hg', 11),\n",
       "             ('f', 1095),\n",
       "             ('regarding', 2274),\n",
       "             ('something', 5549),\n",
       "             ('that', 116212),\n",
       "             ('has', 23136),\n",
       "             ('gone', 735),\n",
       "             ('wrong', 3967),\n",
       "             ('with', 44830),\n",
       "             ('preventing', 78),\n",
       "             ('us', 4617),\n",
       "             ('from', 31088),\n",
       "             ('using', 4615),\n",
       "             ('it', 97688),\n",
       "             ('if', 44113),\n",
       "             ('you', 155213),\n",
       "             ('have', 54392),\n",
       "             ('any', 18814),\n",
       "             ('ideas', 692),\n",
       "             ('or', 40131),\n",
       "             ('suggestions', 564),\n",
       "             ('might', 4924),\n",
       "             ('want', 8692),\n",
       "             ('to', 223771),\n",
       "             ('weigh', 83),\n",
       "             ('in', 109287),\n",
       "             ('here', 16282),\n",
       "             ('thanks', 10360),\n",
       "             ('—', 2850),\n",
       "             ('toronto', 118),\n",
       "             ('mossad', 14),\n",
       "             ('actually', 4508),\n",
       "             ('i', 154649),\n",
       "             ('know', 14419),\n",
       "             ('where', 7395),\n",
       "             ('live', 1855),\n",
       "             ('redirect', 1969),\n",
       "             ('talk', 27388),\n",
       "             ('ufc', 79),\n",
       "             ('fight', 614),\n",
       "             ('night', 603),\n",
       "             ('belfort', 4),\n",
       "             ('vs', 605),\n",
       "             ('henderson', 12),\n",
       "             ('2', 5541),\n",
       "             ('upa', 24),\n",
       "             ('irc', 217),\n",
       "             ('what', 25738),\n",
       "             ('about', 28212),\n",
       "             ('19', 1112),\n",
       "             ('00', 604),\n",
       "             ('utc', 5079),\n",
       "             ('e', 3039),\n",
       "             ('ταλκ', 4),\n",
       "             (\"i've\", 6543),\n",
       "             ('re', 2889),\n",
       "             ('added', 5223),\n",
       "             ('your', 47849),\n",
       "             ('information', 8915),\n",
       "             ('together', 1179),\n",
       "             ('the', 373435),\n",
       "             ('reference', 3448),\n",
       "             ('thank', 7489),\n",
       "             ('·', 677),\n",
       "             ('hail', 60),\n",
       "             (\"i'm\", 13450),\n",
       "             ('an', 34108),\n",
       "             ('elitist', 27),\n",
       "             ('just', 20834),\n",
       "             ('spreading', 112),\n",
       "             ('joy', 75),\n",
       "             ('of', 169027),\n",
       "             ('toilet', 59),\n",
       "             ('paper', 654),\n",
       "             ('t', 1903),\n",
       "             ('c', 2481),\n",
       "             ('listed', 2366),\n",
       "             ('on', 67543),\n",
       "             ('this', 73524),\n",
       "             ('european', 668),\n",
       "             ('list', 5764),\n",
       "             ('as', 58279),\n",
       "             ('still', 5897),\n",
       "             ('survives', 13),\n",
       "             ('some', 16983),\n",
       "             ('parts', 607),\n",
       "             ('europe', 629),\n",
       "             ('which', 19057),\n",
       "             ('british', 1453),\n",
       "             ('section', 7670),\n",
       "             ('do', 30326),\n",
       "             ('mean', 3344),\n",
       "             ('extinct', 53),\n",
       "             ('animals', 216),\n",
       "             ('isles', 147),\n",
       "             ('extinction', 38),\n",
       "             ('date', 2296),\n",
       "             ('and', 168198),\n",
       "             ('correct', 2708),\n",
       "             ('one', 21632),\n",
       "             ('are', 54624),\n",
       "             ('free', 4720),\n",
       "             ('change', 3979),\n",
       "             ('course', 2175),\n",
       "             ('but', 38504),\n",
       "             ('please', 22311),\n",
       "             ('use', 12532),\n",
       "             ('references', 3389),\n",
       "             ('avoid', 1136),\n",
       "             ('reverts', 587),\n",
       "             ('keep', 4037),\n",
       "             ('mind', 1941),\n",
       "             ('uk', 1253),\n",
       "             ('also', 15564),\n",
       "             ('ireland', 489),\n",
       "             ('etc', 3044),\n",
       "             ('made', 7262),\n",
       "             ('mistake', 848),\n",
       "             ('ass', 1939),\n",
       "             ('lol', 945),\n",
       "             ('dynamic', 137),\n",
       "             ('ip', 3097),\n",
       "             ('try', 3337),\n",
       "             ('stop', 5427),\n",
       "             ('me', 27987),\n",
       "             ('82', 303),\n",
       "             ('132', 119),\n",
       "             ('214', 105),\n",
       "             ('222', 128),\n",
       "             ('for', 77142),\n",
       "             ('trying', 3779),\n",
       "             ('fix', 1012),\n",
       "             ('neil', 54),\n",
       "             ('steinberg', 5),\n",
       "             ('removed', 5021),\n",
       "             ('whole', 2157),\n",
       "             ('we', 17671),\n",
       "             ('generally', 1296),\n",
       "             ('consider', 2563),\n",
       "             ('unencyclopedic', 82),\n",
       "             ('again', 7501),\n",
       "             ('брайен', 5),\n",
       "             ('modern', 929),\n",
       "             ('examples', 799),\n",
       "             ('now', 11160),\n",
       "             ('supported', 641),\n",
       "             ('numbers', 890),\n",
       "             ('0', 1809),\n",
       "             ('1', 5444),\n",
       "             ('3', 4021),\n",
       "             ('4', 2641),\n",
       "             ('5', 2644),\n",
       "             ('6', 1742),\n",
       "             ('7', 1673),\n",
       "             ('8', 1526),\n",
       "             ('9', 1331),\n",
       "             ('class', 1297),\n",
       "             ('wikitable', 35),\n",
       "             ('style', 3578),\n",
       "             ('text', 3359),\n",
       "             ('align', 889),\n",
       "             ('center', 521),\n",
       "             ('latin', 369),\n",
       "             ('alphabet', 100),\n",
       "             ('b', 1543),\n",
       "             ('d', 1714),\n",
       "             ('g', 1285),\n",
       "             ('h', 493),\n",
       "             ('j', 631),\n",
       "             ('k', 633),\n",
       "             ('l', 479),\n",
       "             ('m', 1017),\n",
       "             ('n', 865),\n",
       "             ('ñ', 3),\n",
       "             ('o', 805),\n",
       "             ('p', 1614),\n",
       "             ('q', 376),\n",
       "             ('r', 993),\n",
       "             ('s', 2802),\n",
       "             ('response', 1582),\n",
       "             ('hey', 2051),\n",
       "             ('he', 20276),\n",
       "             ('started', 1901),\n",
       "             ('right', 6491),\n",
       "             ('opinion', 3047),\n",
       "             ('so', 27415),\n",
       "             (\"let's\", 1035),\n",
       "             ('get', 10141),\n",
       "             ('straight', 532),\n",
       "             ('rocky', 54),\n",
       "             ('awww', 17),\n",
       "             ('when', 12522),\n",
       "             ('ever', 2712),\n",
       "             ('waited', 55),\n",
       "             ('official', 1730),\n",
       "             (\"►''''''holla\", 17),\n",
       "             ('vandalism', 4750),\n",
       "             (\"lookin'\", 5),\n",
       "             ('out', 12931),\n",
       "             ('naw', 5),\n",
       "             ('going', 5589),\n",
       "             ('bother', 509),\n",
       "             ('saving', 110),\n",
       "             (\"it's\", 13232),\n",
       "             ('creative', 218),\n",
       "             ('enough', 3009),\n",
       "             ('bores', 2),\n",
       "             ('besides', 521),\n",
       "             ('hell', 1053),\n",
       "             ('nigger', 1956),\n",
       "             ('jew', 929),\n",
       "             ('whatever', 1512),\n",
       "             ('she', 4499),\n",
       "             ('called', 2994),\n",
       "             ('makes', 2385),\n",
       "             ('no', 22012),\n",
       "             ('sense', 2071),\n",
       "             ('unless', 1665),\n",
       "             ('falasha', 5),\n",
       "             ('beta', 77),\n",
       "             ('israel', 826),\n",
       "             ('hackneyed', 12),\n",
       "             ('stupid', 1653),\n",
       "             ('seem', 2702),\n",
       "             ('better', 4602),\n",
       "             ('case', 4778),\n",
       "             ('mentioned', 2181),\n",
       "             ('by', 31089),\n",
       "             ('few', 3674),\n",
       "             ('people', 13389),\n",
       "             ('was', 40971),\n",
       "             ('mentioning', 442),\n",
       "             ('signs', 141),\n",
       "             ('disruptive', 916),\n",
       "             ('editing', 7799),\n",
       "             ('referring', 647),\n",
       "             ('edit', 13839),\n",
       "             ('war', 3087),\n",
       "             ('both', 4335),\n",
       "             ('needs', 2833),\n",
       "             ('fact', 5768),\n",
       "             ('got', 3201),\n",
       "             ('email', 1227),\n",
       "             ('asking', 961),\n",
       "             ('y', 517),\n",
       "             ('reported', 867),\n",
       "             ('3vr', 1),\n",
       "             ('yet', 2954),\n",
       "             ('need', 7600),\n",
       "             ('refs', 270),\n",
       "             ('dates', 523),\n",
       "             ('because', 14449),\n",
       "             ('ref', 477),\n",
       "             ('say', 7592),\n",
       "             ('its', 8703),\n",
       "             ('look', 6235),\n",
       "             ('adding', 3111),\n",
       "             ('make', 9719),\n",
       "             ('note', 4934),\n",
       "             ('based', 2581),\n",
       "             ('like', 20761),\n",
       "             ('done', 4680),\n",
       "             ('problem', 3888),\n",
       "             ('my', 34247),\n",
       "             ('understanding', 806),\n",
       "             ('strictly', 172),\n",
       "             ('speaking', 683),\n",
       "             ('thesis', 86),\n",
       "             ('main', 2212),\n",
       "             ('point', 6524),\n",
       "             ('argument', 1541),\n",
       "             ('being', 10916),\n",
       "             ('postulated', 7),\n",
       "             ('discussed', 830),\n",
       "             ('dissertation', 28),\n",
       "             ('physical', 375),\n",
       "             ('bound', 122),\n",
       "             ('sheaf', 1),\n",
       "             ('written', 2675),\n",
       "             ('up', 13379),\n",
       "             ('practice', 583),\n",
       "             ('former', 753),\n",
       "             ('term', 2613),\n",
       "             ('come', 3414),\n",
       "             ('be', 62661),\n",
       "             ('widely', 441),\n",
       "             ('used', 6974),\n",
       "             ('latter', 401),\n",
       "             ('meaning', 1086),\n",
       "             ('too', 5841),\n",
       "             ('given', 2918),\n",
       "             ('usage', 690),\n",
       "             ('varies', 43),\n",
       "             ('throughout', 435),\n",
       "             ('world', 3628),\n",
       "             ('since', 6496),\n",
       "             ('wikipedia', 34501),\n",
       "             ('supposed', 947),\n",
       "             ('international', 858),\n",
       "             ('encyclopaedia', 204),\n",
       "             ('think', 15237),\n",
       "             ('terms', 1283),\n",
       "             ('should', 17879),\n",
       "             ('how', 14096),\n",
       "             ('hot', 273),\n",
       "             ('really', 6981),\n",
       "             ('did', 11186),\n",
       "             ('win', 446),\n",
       "             ('good', 9610),\n",
       "             ('looking', 1866),\n",
       "             ('awards', 321),\n",
       "             ('older', 333),\n",
       "             ('am', 13972),\n",
       "             ('wondering', 711),\n",
       "             ('her', 5081),\n",
       "             ('looks', 1633),\n",
       "             ('70', 448),\n",
       "             ('121', 144),\n",
       "             ('33', 342),\n",
       "             ('78', 228),\n",
       "             ('unblock', 1297),\n",
       "             ('reasonable', 710),\n",
       "             ('fucking', 2692),\n",
       "             ('comment', 4641),\n",
       "             ('only', 13253),\n",
       "             ('user', 10999),\n",
       "             ('area', 1221),\n",
       "             ('who', 16306),\n",
       "             ('ebyabe', 5),\n",
       "             ('comments', 4317),\n",
       "             ('appear', 1250),\n",
       "             ('first', 8329),\n",
       "             ('glance', 70),\n",
       "             ('unreasonable', 130),\n",
       "             ('fraudulent', 62),\n",
       "             ('other', 16784),\n",
       "             ('untrue', 168),\n",
       "             ('place', 4831),\n",
       "             ('could', 9007),\n",
       "             ('see', 16297),\n",
       "             ('well', 9307),\n",
       "             ('sort', 1352),\n",
       "             ('anyway', 1898),\n",
       "             ('claims', 1794),\n",
       "             ('confrontational', 32),\n",
       "             ('clearly', 2828),\n",
       "             ('visible', 165),\n",
       "             ('page', 34827),\n",
       "             ('users', 2820),\n",
       "             ('attacking', 410),\n",
       "             ('way', 8467),\n",
       "             ('around', 2983),\n",
       "             ('caused', 441),\n",
       "             ('confrontation', 32),\n",
       "             ('replied', 293),\n",
       "             ('putting', 847),\n",
       "             ('ask', 4606),\n",
       "             ('report', 1657),\n",
       "             ('him', 6042),\n",
       "             ('nobody', 797),\n",
       "             ('answered', 272),\n",
       "             ('regardless', 579),\n",
       "             ('several', 2464),\n",
       "             ('posts', 801),\n",
       "             ('himself', 1094),\n",
       "             ('would', 21849),\n",
       "             ('been', 18958),\n",
       "             (\"i'd\", 2602),\n",
       "             ('able', 1776),\n",
       "             ('leave', 3233),\n",
       "             ('message', 2804),\n",
       "             (\"stwalkerster's\", 1),\n",
       "             ('cannot', 2363),\n",
       "             ('reason', 4649),\n",
       "             ('those', 6414),\n",
       "             ('blocked', 5574),\n",
       "             ('instead', 2728),\n",
       "             ('hopefully', 619),\n",
       "             ('notifications', 30),\n",
       "             ('turned', 363),\n",
       "             ('tags', 1336),\n",
       "             ('username', 1427),\n",
       "             ('can', 25624),\n",
       "             ('calm', 187),\n",
       "             ('work', 6626),\n",
       "             ('noted', 746),\n",
       "             ('reply', 1365),\n",
       "             ('gives', 799),\n",
       "             ('had', 9376),\n",
       "             ('reverting', 1463),\n",
       "             ('arguing', 380),\n",
       "             ('hostilely', 1),\n",
       "             ('wish', 1795),\n",
       "             ('attacks', 1810),\n",
       "             ('write', 2920),\n",
       "             ('words', 2401),\n",
       "             ('compliance', 131),\n",
       "             ('revert', 2759),\n",
       "             ('edits', 7514),\n",
       "             ('they', 20591),\n",
       "             ('further', 2602),\n",
       "             ('fixed', 684),\n",
       "             ('details', 1038),\n",
       "             ('referred', 534),\n",
       "             ('separate', 1008),\n",
       "             ('paragraph', 1499),\n",
       "             ('below', 1760),\n",
       "             ('false', 1517),\n",
       "             ('against', 4157),\n",
       "             ('delete', 3055),\n",
       "             ('replies', 149),\n",
       "             ('pages', 8125),\n",
       "             ('then', 12212),\n",
       "             ('archive', 940),\n",
       "             ('after', 6312),\n",
       "             ('removing', 1915),\n",
       "             ('them', 11266),\n",
       "             ('these', 8332),\n",
       "             ('kinds', 189),\n",
       "             ('things', 4394),\n",
       "             ('anyone', 3569),\n",
       "             ('means', 2262),\n",
       "             ('act', 1101),\n",
       "             ('manner', 622),\n",
       "             ('normal', 495),\n",
       "             ('ensue', 13),\n",
       "             ('including', 1999),\n",
       "             ('myself', 1982),\n",
       "             ('merely', 836),\n",
       "             ('someone', 5884),\n",
       "             ('template', 2671),\n",
       "             ('shorter', 130),\n",
       "             ('wrote', 1785),\n",
       "             ('their', 10704),\n",
       "             ('aggressive', 193),\n",
       "             ('remarks', 363),\n",
       "             ('business', 894),\n",
       "             ('does', 8963),\n",
       "             ('writing', 1770),\n",
       "             ('nicely', 133),\n",
       "             ('falsified', 25),\n",
       "             ('writes', 245),\n",
       "             ('expect', 698),\n",
       "             ('victim', 204),\n",
       "             ('without', 5361),\n",
       "             ('friends', 852),\n",
       "             ('attacker', 23),\n",
       "             ('different', 3449),\n",
       "             ('article', 42272),\n",
       "             ('nor', 1588),\n",
       "             ('ganging', 26),\n",
       "             ('replying', 137),\n",
       "             ('places', 695),\n",
       "             ('once', 2551),\n",
       "             ('stating', 589),\n",
       "             ('bad', 2668),\n",
       "             ('times', 2485),\n",
       "             ('having', 2838),\n",
       "             ('actual', 1251),\n",
       "             ('kept', 643),\n",
       "             ('indicates', 269),\n",
       "             ('wanted', 1726),\n",
       "             ('anything', 4451),\n",
       "             ('attack', 2253),\n",
       "             ('another', 5011),\n",
       "             ('headlines', 40),\n",
       "             ('veiled', 34),\n",
       "             ('plain', 347),\n",
       "             ('light', 857),\n",
       "             ('attempt', 1030),\n",
       "             ('hidden', 239),\n",
       "             ('headline', 125),\n",
       "             ('literally', 276),\n",
       "             ('title', 2876),\n",
       "             ('truthful', 63),\n",
       "             ('falsification', 18),\n",
       "             ('blatant', 417),\n",
       "             ('expected', 314),\n",
       "             ('next', 1866),\n",
       "             ('directly', 932),\n",
       "             ('mentions', 442),\n",
       "             ('summary', 1938),\n",
       "             ('says', 3198),\n",
       "             ('differently', 143),\n",
       "             ('implied', 123),\n",
       "             ('personal', 4547),\n",
       "             ('true', 2606),\n",
       "             ('evident', 147),\n",
       "             ('administrator', 1446),\n",
       "             ('complains', 36),\n",
       "             ('signature', 390),\n",
       "             ('even', 10230),\n",
       "             ('though', 3927),\n",
       "             ('changed', 1957),\n",
       "             ('messages', 992),\n",
       "             ('posted', 1540),\n",
       "             ('before', 6389),\n",
       "             ('block', 4644),\n",
       "             ('happened', 1129),\n",
       "             ('direct', 814),\n",
       "             ('linking', 342),\n",
       "             ('contain', 452),\n",
       "             ('titles', 509),\n",
       "             ('were', 11754),\n",
       "             ('unambiguously', 35),\n",
       "             (\"can't\", 3978),\n",
       "             ('last', 3578),\n",
       "             ('yes', 3047),\n",
       "             ('somewhat', 530),\n",
       "             ('motivated', 160),\n",
       "             ('hostile', 160),\n",
       "             ('put', 4617),\n",
       "             ('useless', 351),\n",
       "             (\"there's\", 2365),\n",
       "             ('two', 6040),\n",
       "             ('notice', 2328),\n",
       "             ('while', 4531),\n",
       "             ('show', 3077),\n",
       "             ('tyranny', 37),\n",
       "             ('word', 3265),\n",
       "             ('solely', 199),\n",
       "             ('already', 4046),\n",
       "             ('talked', 199),\n",
       "             ('meant', 935),\n",
       "             ('instigation', 7),\n",
       "             ('although', 1895),\n",
       "             ('link', 5720),\n",
       "             ('jamie', 33),\n",
       "             ('appears', 1715),\n",
       "             ('political', 1525),\n",
       "             ('opinions', 1046),\n",
       "             ('making', 3439),\n",
       "             ('appropriate', 2009),\n",
       "             ('exposed', 149),\n",
       "             ('intents', 13),\n",
       "             (\"'possible'\", 2),\n",
       "             ('exception', 280),\n",
       "             ('particular', 1456),\n",
       "             ('repeatedly', 637),\n",
       "             ('annoyed', 91),\n",
       "             ('all', 23554),\n",
       "             ('reading', 1585),\n",
       "             ('optional', 54),\n",
       "             ('multiple', 1062),\n",
       "             ('harassing', 247),\n",
       "             ('werieth', 15),\n",
       "             ('deletionist', 53),\n",
       "             ('his', 15316),\n",
       "             ('idea', 2422),\n",
       "             ('contributing', 1259),\n",
       "             ('undoing', 175),\n",
       "             (\"people's\", 421),\n",
       "             ('acting', 508),\n",
       "             ('enforcer', 15),\n",
       "             ('subject', 3921),\n",
       "             ('matters', 479),\n",
       "             ('knows', 773),\n",
       "             ('nothing', 3951),\n",
       "             (\"won't\", 1608),\n",
       "             ('anywhere', 526),\n",
       "             ('dropped', 180),\n",
       "             ('head', 938),\n",
       "             ('child', 587),\n",
       "             ('kid', 310),\n",
       "             ('devil', 100),\n",
       "             ('hello', 3137),\n",
       "             ('recent', 1874),\n",
       "             (\"wikipedia's\", 1981),\n",
       "             ('reverted', 2930),\n",
       "             ('remove', 3851),\n",
       "             ('content', 5407),\n",
       "             ('regards', 1482),\n",
       "             ('ace', 52),\n",
       "             ('edited', 1620),\n",
       "             ('probably', 3151),\n",
       "             ('stan', 47),\n",
       "             ('slim', 80),\n",
       "             ('believe', 4482),\n",
       "             ('shaft', 20),\n",
       "             ('adressed', 14),\n",
       "             ('mayir', 1),\n",
       "             ('our', 5202),\n",
       "             ('school', 1911),\n",
       "             ('bulldowzed', 1),\n",
       "             ('down', 2909),\n",
       "             ('create', 2170),\n",
       "             ('mini', 70),\n",
       "             ('mall', 63),\n",
       "             ('wernt', 2),\n",
       "             ('chillum', 21),\n",
       "             ('living', 1083),\n",
       "             ('proof', 1053),\n",
       "             ('discriminate', 22),\n",
       "             ('slow', 211),\n",
       "             ('learners', 8),\n",
       "             ('let', 3515),\n",
       "             ('admin', 2709),\n",
       "             ('dont', 2223),\n",
       "             ('hit', 505),\n",
       "             ('belt', 96),\n",
       "             ('administrators', 885),\n",
       "             ('side', 1466),\n",
       "             (\"that's\", 4561),\n",
       "             ('big', 2044),\n",
       "             ('nono', 2),\n",
       "             ('faul', 1),\n",
       "             ('red', 921),\n",
       "             ('card', 202),\n",
       "             ('lee', 199),\n",
       "             ('strobel', 1),\n",
       "             ('affiliated', 128),\n",
       "             ('discovery', 129),\n",
       "             ('institute', 202),\n",
       "             ('argues', 77),\n",
       "             ('id', 635),\n",
       "             ('creator', 282),\n",
       "             (\"don't\", 17248),\n",
       "             ('decision', 858),\n",
       "             ('restore', 378),\n",
       "             ('nonregistered', 1),\n",
       "             ('post', 3076),\n",
       "             ('past', 1562),\n",
       "             ('tfds', 1),\n",
       "             ('drv', 88),\n",
       "             ('involved', 1625),\n",
       "             ('editors', 4986),\n",
       "             ('object', 493),\n",
       "             ('restored', 466),\n",
       "             ('doubt', 1013),\n",
       "             ('objections', 323),\n",
       "             ('editor', 3563),\n",
       "             (\"template's\", 25),\n",
       "             ('anon', 280),\n",
       "             ('same', 6772),\n",
       "             ('thing', 3996),\n",
       "             ('battle', 635),\n",
       "             ('white', 1359),\n",
       "             ('plains', 22),\n",
       "             ('thought', 2855),\n",
       "             ('best', 4486),\n",
       "             ('usually', 1082),\n",
       "             ('warning', 1861),\n",
       "             ('sam', 122),\n",
       "             ('join', 591),\n",
       "             ('united', 1523),\n",
       "             ('states', 2644),\n",
       "             ('air', 628),\n",
       "             ('force', 864),\n",
       "             ('year', 2407),\n",
       "             ('98', 345),\n",
       "             ('236', 82),\n",
       "             ('110', 168),\n",
       "             ('176', 143),\n",
       "             ('49', 248),\n",
       "             ('18', 1175),\n",
       "             ('september', 739),\n",
       "             ('2014', 575),\n",
       "             ('joined', 212),\n",
       "             ('treated', 320),\n",
       "             ('poorly', 248),\n",
       "             ('huh', 213),\n",
       "             (\"didn't\", 3964),\n",
       "             ('ethnic', 619),\n",
       "             ('slur', 53),\n",
       "             ('proves', 253),\n",
       "             (\"he's\", 1544),\n",
       "             ('medical', 553),\n",
       "             ('student', 395),\n",
       "             ('pakistan', 299),\n",
       "             ('https', 417),\n",
       "             ('twitter', 153),\n",
       "             ('com', 4062),\n",
       "             ('ashermadan', 3),\n",
       "             ('status', 1415),\n",
       "             ('65290935558344705', 1),\n",
       "             ('http', 5311),\n",
       "             ('www', 2963),\n",
       "             ('dailytimes', 4),\n",
       "             ('pk', 15),\n",
       "             ('default', 169),\n",
       "             ('asp', 105),\n",
       "             ('story', 1286),\n",
       "             ('17', 1185),\n",
       "             ('2004', 1215),\n",
       "             ('pg7', 1),\n",
       "             ('37', 299),\n",
       "             ('yourself', 4854),\n",
       "             ('deny', 291),\n",
       "             ('139', 152),\n",
       "             ('190', 167),\n",
       "             ('140', 131),\n",
       "             ('12', 1468),\n",
       "             ('vandalize', 1373),\n",
       "             ('continue', 3375),\n",
       "             ('will', 22993),\n",
       "             ('simple', 1248),\n",
       "             (\"you're\", 5506),\n",
       "             ('nasty', 195),\n",
       "             ('little', 3799),\n",
       "             ('peon', 3),\n",
       "             ('peter', 365),\n",
       "             ('cushing', 3),\n",
       "             ('ensa', 1),\n",
       "             ('never', 4754),\n",
       "             ('bothered', 237),\n",
       "             ('check', 3101),\n",
       "             ('sheer', 60),\n",
       "             ('temerity', 5),\n",
       "             ('reliable', 3524),\n",
       "             ('sources', 8589),\n",
       "             (\"you've\", 1818),\n",
       "             ('source', 7725),\n",
       "             ('entire', 1316),\n",
       "             ('time', 11939),\n",
       "             ('site', 3120),\n",
       "             ('tips', 262),\n",
       "             ('hat', 145),\n",
       "             ('90', 426),\n",
       "             ('rest', 1026),\n",
       "             ('referenced', 673),\n",
       "             ('either', 3033),\n",
       "             ('least', 3025),\n",
       "             ('knew', 604),\n",
       "             ('faith', 1800),\n",
       "             ('decent', 222),\n",
       "             ('bone', 61),\n",
       "             ('body', 739),\n",
       "             (\"we've\", 298),\n",
       "             ('crossed', 83),\n",
       "             ('swords', 23),\n",
       "             ('found', 3310),\n",
       "             ('deleted', 6782),\n",
       "             ('help', 8232),\n",
       "             ('pieces', 215),\n",
       "             ('hate', 2541),\n",
       "             ('piece', 1363),\n",
       "             ('proper', 1129),\n",
       "             ('censure', 22),\n",
       "             ('attacked', 359),\n",
       "             ('constitute', 188),\n",
       "             ('worth', 833),\n",
       "             ('every', 3103),\n",
       "             ('legitimate', 577),\n",
       "             ('reversed', 97),\n",
       "             ('pasted', 131),\n",
       "             ('into', 6657),\n",
       "             ('category', 1987),\n",
       "             ('spylab', 31),\n",
       "             ('unprotect', 81),\n",
       "             ('aware', 1292),\n",
       "             ('warring', 917),\n",
       "             ('blockable', 31),\n",
       "             ('offense', 213),\n",
       "             ('potentially', 205),\n",
       "             ('controversial', 760),\n",
       "             ('changes', 2389),\n",
       "             ('establishing', 70),\n",
       "             ('consensus', 3055),\n",
       "             ('\\xa0prattle\\xa0', 1),\n",
       "             ('giovanni', 21),\n",
       "             ('feel', 4494),\n",
       "             (\"person's\", 175),\n",
       "             (\"'warnings'\", 5),\n",
       "             ('–', 1406),\n",
       "             ('extreme', 360),\n",
       "             ('prejudice', 115),\n",
       "             ('why', 13265),\n",
       "             ('promotions', 17),\n",
       "             ('pls', 94),\n",
       "             ('understand', 3763),\n",
       "             ('im', 1011),\n",
       "             ('above', 3702),\n",
       "             ('hahc21', 3),\n",
       "             ('read', 6250),\n",
       "             ('version', 2544),\n",
       "             ('previous', 1213),\n",
       "             ('versions', 422),\n",
       "             (\"you'll\", 983),\n",
       "             ('phase', 136),\n",
       "             (\"doesn't\", 4767),\n",
       "             ('promotion', 322),\n",
       "             ('talks', 222),\n",
       "             ('predictions', 42),\n",
       "             ('thoughts', 576),\n",
       "             ('team', 982),\n",
       "             ('shoot', 163),\n",
       "             ('documentaries', 27),\n",
       "             ('seeing', 662),\n",
       "             ('artists', 246),\n",
       "             ('promote', 477),\n",
       "             ('wants', 790),\n",
       "             ('bollywood', 28),\n",
       "             ('actress', 116),\n",
       "             ('lipsync', 2),\n",
       "             ('actors', 147),\n",
       "             (\"hasn't\", 402),\n",
       "             ('pursued', 31),\n",
       "             ('singing', 83),\n",
       "             ('highlight', 76),\n",
       "             (\"someone's\", 171),\n",
       "             ('prediction', 31),\n",
       "             ('pioneer', 20),\n",
       "             ('include', 2173),\n",
       "             ('pr', 158),\n",
       "             ('skills', 280),\n",
       "             ('behind', 775),\n",
       "             ('scenes', 67),\n",
       "             ('video', 1068),\n",
       "             ('new', 7967),\n",
       "             ('star', 539),\n",
       "             ('releases', 173),\n",
       "             ('stuffs', 8),\n",
       "             ('nee', 10),\n",
       "             ('defend', 387),\n",
       "             ('jc', 11),\n",
       "             ('flew', 40),\n",
       "             ('handle', 267),\n",
       "             ('less', 2145),\n",
       "             ('afterwriting', 1),\n",
       "             ('perhaps', 2560),\n",
       "             ('off', 4432),\n",
       "             (\"haven't\", 1311),\n",
       "             ('seems', 4595),\n",
       "             ('strange', 389),\n",
       "             ('carry', 238),\n",
       "             ('abusing', 262),\n",
       "             ('wiki', 5995),\n",
       "             ('works', 1189),\n",
       "             ('knowledge', 1390),\n",
       "             ('topics', 608),\n",
       "             ('hand', 1200),\n",
       "             ('proficient', 16),\n",
       "             ('unfortunately', 821),\n",
       "             ('great', 3388),\n",
       "             ('downfall', 24),\n",
       "             ('mos', 289),\n",
       "             (\"qexigator's\", 1),\n",
       "             ('request', 3028),\n",
       "             ('circumstances', 236),\n",
       "             ('suggest', 2081),\n",
       "             ('viewing', 94),\n",
       "             ('museumstjohn', 1),\n",
       "             ('org', 2703),\n",
       "             ('overturned', 41),\n",
       "             ('shall', 906),\n",
       "             ('stand', 761),\n",
       "             ('bid', 49),\n",
       "             ('farewell', 14),\n",
       "             ('sadly', 157),\n",
       "             ('matter', 2913),\n",
       "             ('principle', 234),\n",
       "             ('such', 8577),\n",
       "             ('treatment', 350),\n",
       "             ('three', 2441),\n",
       "             ('rottweilers', 2),\n",
       "             ('properly', 741),\n",
       "             ('clear', 2648),\n",
       "             ('assure', 164),\n",
       "             ('mistakes', 429),\n",
       "             ('regarded', 223),\n",
       "             ('harassment', 494),\n",
       "             ('ciao', 39),\n",
       "             ('more', 17583),\n",
       "             ('substantive', 77),\n",
       "             ('annual', 106),\n",
       "             ('murder', 455),\n",
       "             ('rate', 402),\n",
       "             ('per', 1898),\n",
       "             ('million', 900),\n",
       "             ('indians', 143),\n",
       "             ('india', 959),\n",
       "             ('34', 300),\n",
       "             ('australians', 35),\n",
       "             ('oz', 38),\n",
       "             ('11', 1605),\n",
       "             ('25', 910),\n",
       "             ('explanation', 1323),\n",
       "             ('higher', 530),\n",
       "             ('compared', 361),\n",
       "             ('much', 7023),\n",
       "             ('likely', 1322),\n",
       "             ('young', 518),\n",
       "             ('male', 367),\n",
       "             ('half', 916),\n",
       "             ('violent', 118),\n",
       "             ('crime', 409),\n",
       "             ('involves', 156),\n",
       "             ('males', 75),\n",
       "             ('unofficially', 11),\n",
       "             ('eliminate', 117),\n",
       "             ('indian', 761),\n",
       "             ('murders', 58),\n",
       "             ('suddenly', 166),\n",
       "             ('difference', 971),\n",
       "             ('gets', 948),\n",
       "             ('smaller', 225),\n",
       "             ('pretty', 1621),\n",
       "             ('damn', 800),\n",
       "             ('each', 2275),\n",
       "             ('wherever', 88),\n",
       "             ('care', 2351),\n",
       "             ('raul654', 24),\n",
       "             ('wikiproject', 1412),\n",
       "             ('members', 1235),\n",
       "             (\"they're\", 1022),\n",
       "             ('fine', 1369),\n",
       "             ('future', 1588),\n",
       "             ('give', 3629),\n",
       "             ('niteshift', 4),\n",
       "             ('wake', 137),\n",
       "             ('call', 2032),\n",
       "             ('degrassi', 6),\n",
       "             ('rewrite', 355),\n",
       "             ('relation', 323),\n",
       "             ('climate', 221),\n",
       "             ('issues', 2176),\n",
       "             ('rewritten', 241),\n",
       "             ('taken', 1735),\n",
       "             ('alone', 1278),\n",
       "             ('niave', 1),\n",
       "             ('reader', 794),\n",
       "             ('may', 11755),\n",
       "             ('take', 6102),\n",
       "             ('issue', 3696),\n",
       "             ('creates', 134),\n",
       "             ('significant', 916),\n",
       "             ('credible', 362),\n",
       "             ('evidence', 2735),\n",
       "             ('anthropogenic', 10),\n",
       "             ('global', 476),\n",
       "             ('warming', 194),\n",
       "             ('undue', 322),\n",
       "             ('rewording', 43),\n",
       "             ('arguments', 869),\n",
       "             ('hypothesis', 184),\n",
       "             ('acadmic', 1),\n",
       "             ('debate', 1070),\n",
       "             ('impact', 326),\n",
       "             ('general', 2235),\n",
       "             ('scientific', 819),\n",
       "             ('man', 2157),\n",
       "             ('gobal', 2),\n",
       "             ('welcome', 4930),\n",
       "             ('75', 424),\n",
       "             ('30', 1035),\n",
       "             ('178', 117),\n",
       "             ('224', 102),\n",
       "             ('contributions', 2990),\n",
       "             ('hope', 4207),\n",
       "             ('decide', 1292),\n",
       "             ('stay', 1597),\n",
       "             ('conformed', 18),\n",
       "             ('neutral', 1976),\n",
       "             ('view', 3000),\n",
       "             ('policy', 4276),\n",
       "             ('articles', 12618),\n",
       "             ('refer', 966),\n",
       "             ('facts', 2303),\n",
       "             ('interpretations', 153),\n",
       "             ('stated', 1363),\n",
       "             ('print', 254),\n",
       "             ('reputable', 262),\n",
       "             ('websites', 589),\n",
       "             ('forms', 310),\n",
       "             ('media', 2673),\n",
       "             ('npov', 1418),\n",
       "             ('effectively', 168),\n",
       "             ('disparate', 29),\n",
       "             ('points', 1313),\n",
       "             ('compromising', 28),\n",
       "             ('stuck', 258),\n",
       "             (\"contributors'\", 180),\n",
       "             ('experienced', 426),\n",
       "             ('wikipedians', 712),\n",
       "             ('answer', 1894),\n",
       "             ('queries', 113),\n",
       "             ('type', 1285),\n",
       "             ('helpme', 578),\n",
       "             ('shortly', 475),\n",
       "             ('questions', 4052),\n",
       "             ('links', 4441),\n",
       "             ('newcomers', 262),\n",
       "             ('five', 1460),\n",
       "             ('pillars', 940),\n",
       "             ('tutorial', 879),\n",
       "             ('manual', 1115),\n",
       "             ('enjoy', 1146),\n",
       "             ('wikipedian', 956),\n",
       "             ...])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.word_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the we have trained the tokenizer we can use it to vectorize the texts. In particular, we would like to transform the texts to sequences of word indexes. We can do this through the **text_to_sequences** method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = tokenizer.texts_to_sequences(train[\"comment_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see now how a text has been transformed to a list of word indexes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[8034,\n",
       " 14,\n",
       " 619,\n",
       " 158,\n",
       " 41,\n",
       " 8,\n",
       " 5,\n",
       " 136,\n",
       " 152,\n",
       " 34,\n",
       " 116,\n",
       " 820,\n",
       " 425,\n",
       " 168,\n",
       " 9,\n",
       " 43,\n",
       " 1158,\n",
       " 238,\n",
       " 21,\n",
       " 6162,\n",
       " 202,\n",
       " 32,\n",
       " 204,\n",
       " 11,\n",
       " 22,\n",
       " 6,\n",
       " 19,\n",
       " 55,\n",
       " 1208,\n",
       " 25,\n",
       " 1463,\n",
       " 6,\n",
       " 188,\n",
       " 106,\n",
       " 2,\n",
       " 5909,\n",
       " 10,\n",
       " 64,\n",
       " 94,\n",
       " 344,\n",
       " 4683]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do these indexes mean? Each number represents a word recognized by the trained Tokenizer. To check what word each index represents, we can take a look at the **word_index** attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'the': 1,\n",
       " 'to': 2,\n",
       " 'of': 3,\n",
       " 'and': 4,\n",
       " 'a': 5,\n",
       " 'you': 6,\n",
       " 'i': 7,\n",
       " 'is': 8,\n",
       " 'that': 9,\n",
       " 'in': 10,\n",
       " 'it': 11,\n",
       " 'for': 12,\n",
       " 'this': 13,\n",
       " 'not': 14,\n",
       " 'on': 15,\n",
       " 'be': 16,\n",
       " 'as': 17,\n",
       " 'are': 18,\n",
       " 'have': 19,\n",
       " 'your': 20,\n",
       " 'with': 21,\n",
       " 'if': 22,\n",
       " 'article': 23,\n",
       " 'was': 24,\n",
       " 'or': 25,\n",
       " 'but': 26,\n",
       " 'page': 27,\n",
       " 'wikipedia': 28,\n",
       " 'my': 29,\n",
       " 'an': 30,\n",
       " 'by': 31,\n",
       " 'from': 32,\n",
       " 'do': 33,\n",
       " 'at': 34,\n",
       " 'about': 35,\n",
       " 'me': 36,\n",
       " 'so': 37,\n",
       " 'talk': 38,\n",
       " 'what': 39,\n",
       " 'can': 40,\n",
       " 'there': 41,\n",
       " 'all': 42,\n",
       " 'has': 43,\n",
       " 'will': 44,\n",
       " 'please': 45,\n",
       " 'no': 46,\n",
       " 'would': 47,\n",
       " 'one': 48,\n",
       " 'just': 49,\n",
       " 'like': 50,\n",
       " 'they': 51,\n",
       " 'he': 52,\n",
       " 'which': 53,\n",
       " 'been': 54,\n",
       " 'any': 55,\n",
       " 'should': 56,\n",
       " 'we': 57,\n",
       " 'more': 58,\n",
       " \"don't\": 59,\n",
       " 'some': 60,\n",
       " 'other': 61,\n",
       " 'who': 62,\n",
       " 'see': 63,\n",
       " 'here': 64,\n",
       " 'also': 65,\n",
       " 'his': 66,\n",
       " 'think': 67,\n",
       " 'because': 68,\n",
       " 'know': 69,\n",
       " 'how': 70,\n",
       " 'am': 71,\n",
       " 'edit': 72,\n",
       " \"i'm\": 73,\n",
       " 'people': 74,\n",
       " 'up': 75,\n",
       " 'why': 76,\n",
       " 'only': 77,\n",
       " \"it's\": 78,\n",
       " 'out': 79,\n",
       " 'articles': 80,\n",
       " 'use': 81,\n",
       " 'when': 82,\n",
       " 'then': 83,\n",
       " 'time': 84,\n",
       " 'may': 85,\n",
       " 'were': 86,\n",
       " 'them': 87,\n",
       " 'did': 88,\n",
       " 'now': 89,\n",
       " 'user': 90,\n",
       " 'being': 91,\n",
       " 'their': 92,\n",
       " 'than': 93,\n",
       " 'thanks': 94,\n",
       " 'even': 95,\n",
       " 'get': 96,\n",
       " 'make': 97,\n",
       " 'good': 98,\n",
       " 'had': 99,\n",
       " 'very': 100,\n",
       " 'well': 101,\n",
       " 'could': 102,\n",
       " 'does': 103,\n",
       " 'information': 104,\n",
       " 'its': 105,\n",
       " 'want': 106,\n",
       " 'deletion': 107,\n",
       " 'sources': 108,\n",
       " 'such': 109,\n",
       " 'name': 110,\n",
       " 'way': 111,\n",
       " 'these': 112,\n",
       " 'first': 113,\n",
       " 'help': 114,\n",
       " 'pages': 115,\n",
       " 'wp': 116,\n",
       " 'go': 117,\n",
       " 'new': 118,\n",
       " 'image': 119,\n",
       " 'editing': 120,\n",
       " 'fuck': 121,\n",
       " 'source': 122,\n",
       " 'section': 123,\n",
       " 'need': 124,\n",
       " 'say': 125,\n",
       " 'edits': 126,\n",
       " 'again': 127,\n",
       " 'thank': 128,\n",
       " 'where': 129,\n",
       " 'made': 130,\n",
       " 'many': 131,\n",
       " 'much': 132,\n",
       " 'really': 133,\n",
       " 'used': 134,\n",
       " 'most': 135,\n",
       " 'discussion': 136,\n",
       " 'deleted': 137,\n",
       " 'same': 138,\n",
       " 'find': 139,\n",
       " 'into': 140,\n",
       " 'work': 141,\n",
       " \"i've\": 142,\n",
       " 'point': 143,\n",
       " 'since': 144,\n",
       " 'right': 145,\n",
       " 'those': 146,\n",
       " 'before': 147,\n",
       " 'after': 148,\n",
       " 'read': 149,\n",
       " 'add': 150,\n",
       " 'look': 151,\n",
       " 'over': 152,\n",
       " 'take': 153,\n",
       " 'him': 154,\n",
       " 'two': 155,\n",
       " 'wiki': 156,\n",
       " 'back': 157,\n",
       " 'hi': 158,\n",
       " 'still': 159,\n",
       " 'someone': 160,\n",
       " 'too': 161,\n",
       " 'fact': 162,\n",
       " 'list': 163,\n",
       " 'link': 164,\n",
       " 'own': 165,\n",
       " 'going': 166,\n",
       " 'blocked': 167,\n",
       " 'something': 168,\n",
       " 'said': 169,\n",
       " '2': 170,\n",
       " \"you're\": 171,\n",
       " '1': 172,\n",
       " 'stop': 173,\n",
       " 'content': 174,\n",
       " 'without': 175,\n",
       " 'http': 176,\n",
       " 'history': 177,\n",
       " 'added': 178,\n",
       " 'under': 179,\n",
       " 'our': 180,\n",
       " 'her': 181,\n",
       " 'utc': 182,\n",
       " 'removed': 183,\n",
       " 'another': 184,\n",
       " 'editors': 185,\n",
       " 'note': 186,\n",
       " 'welcome': 187,\n",
       " 'might': 188,\n",
       " 'yourself': 189,\n",
       " 'place': 190,\n",
       " 'however': 191,\n",
       " 'case': 192,\n",
       " \"doesn't\": 193,\n",
       " 'sure': 194,\n",
       " 'never': 195,\n",
       " 'vandalism': 196,\n",
       " 'free': 197,\n",
       " 'done': 198,\n",
       " 'reason': 199,\n",
       " 'block': 200,\n",
       " 'comment': 201,\n",
       " 'us': 202,\n",
       " 'put': 203,\n",
       " 'using': 204,\n",
       " 'ask': 205,\n",
       " 'better': 206,\n",
       " 'seems': 207,\n",
       " \"that's\": 208,\n",
       " 'question': 209,\n",
       " 'personal': 210,\n",
       " 'while': 211,\n",
       " 'actually': 212,\n",
       " 'she': 213,\n",
       " 'feel': 214,\n",
       " 'best': 215,\n",
       " 'believe': 216,\n",
       " 'anything': 217,\n",
       " 'links': 218,\n",
       " 'off': 219,\n",
       " 'person': 220,\n",
       " 'things': 221,\n",
       " 'both': 222,\n",
       " 'comments': 223,\n",
       " 'part': 224,\n",
       " 'policy': 225,\n",
       " 'hope': 226,\n",
       " 'against': 227,\n",
       " 'u': 228,\n",
       " '•': 229,\n",
       " 'com': 230,\n",
       " 'questions': 231,\n",
       " 'already': 232,\n",
       " 'keep': 233,\n",
       " '3': 234,\n",
       " 'thing': 235,\n",
       " 'change': 236,\n",
       " \"can't\": 237,\n",
       " 'wrong': 238,\n",
       " \"didn't\": 239,\n",
       " 'nothing': 240,\n",
       " 'though': 241,\n",
       " 'subject': 242,\n",
       " \"i'll\": 243,\n",
       " 'problem': 244,\n",
       " 'remove': 245,\n",
       " 'copyright': 246,\n",
       " 'little': 247,\n",
       " 'tag': 248,\n",
       " 'trying': 249,\n",
       " 'understand': 250,\n",
       " 'long': 251,\n",
       " 'above': 252,\n",
       " 'issue': 253,\n",
       " 'few': 254,\n",
       " 'speedy': 255,\n",
       " 'give': 256,\n",
       " 'world': 257,\n",
       " 'sorry': 258,\n",
       " 'style': 259,\n",
       " 'last': 260,\n",
       " 'anyone': 261,\n",
       " 'editor': 262,\n",
       " 'others': 263,\n",
       " 'must': 264,\n",
       " 'rather': 265,\n",
       " 'reliable': 266,\n",
       " 'agree': 267,\n",
       " 'let': 268,\n",
       " 'english': 269,\n",
       " 'fair': 270,\n",
       " 'years': 271,\n",
       " 'different': 272,\n",
       " 'reference': 273,\n",
       " 'making': 274,\n",
       " 'come': 275,\n",
       " 'non': 276,\n",
       " 'references': 277,\n",
       " 'great': 278,\n",
       " 'doing': 279,\n",
       " 'continue': 280,\n",
       " 'shit': 281,\n",
       " 'text': 282,\n",
       " 'mean': 283,\n",
       " 'try': 284,\n",
       " 'found': 285,\n",
       " 'word': 286,\n",
       " 'leave': 287,\n",
       " 'original': 288,\n",
       " 'got': 289,\n",
       " 'says': 290,\n",
       " 'simply': 291,\n",
       " 'state': 292,\n",
       " \"isn't\": 293,\n",
       " 'top': 294,\n",
       " 'probably': 295,\n",
       " 'hello': 296,\n",
       " 'created': 297,\n",
       " 'site': 298,\n",
       " 'adding': 299,\n",
       " 'every': 300,\n",
       " 'check': 301,\n",
       " 'life': 302,\n",
       " 'ip': 303,\n",
       " 'war': 304,\n",
       " 'day': 305,\n",
       " 'show': 306,\n",
       " 'post': 307,\n",
       " 'else': 308,\n",
       " 'delete': 309,\n",
       " 'consensus': 310,\n",
       " 'opinion': 311,\n",
       " 'yes': 312,\n",
       " 'etc': 313,\n",
       " 'far': 314,\n",
       " 'e': 315,\n",
       " 'either': 316,\n",
       " 'notable': 317,\n",
       " 'request': 318,\n",
       " 'least': 319,\n",
       " 'enough': 320,\n",
       " 'view': 321,\n",
       " 'called': 322,\n",
       " 'contributions': 323,\n",
       " 'example': 324,\n",
       " 'around': 325,\n",
       " 'www': 326,\n",
       " 'yet': 327,\n",
       " 'through': 328,\n",
       " 'between': 329,\n",
       " 'old': 330,\n",
       " 'real': 331,\n",
       " 'reverted': 332,\n",
       " 'book': 333,\n",
       " 'write': 334,\n",
       " 'given': 335,\n",
       " 'matter': 336,\n",
       " 'down': 337,\n",
       " 're': 338,\n",
       " 'oh': 339,\n",
       " 'title': 340,\n",
       " 'account': 341,\n",
       " 'thought': 342,\n",
       " 'material': 343,\n",
       " '—': 344,\n",
       " 'having': 345,\n",
       " 'needs': 346,\n",
       " 'clearly': 347,\n",
       " 'images': 348,\n",
       " 'encyclopedia': 349,\n",
       " 'users': 350,\n",
       " 'message': 351,\n",
       " 's': 352,\n",
       " 'support': 353,\n",
       " 'maybe': 354,\n",
       " 'lot': 355,\n",
       " 'revert': 356,\n",
       " 'evidence': 357,\n",
       " 'number': 358,\n",
       " 'instead': 359,\n",
       " 'language': 360,\n",
       " 'tell': 361,\n",
       " 'ever': 362,\n",
       " 'admin': 363,\n",
       " 'correct': 364,\n",
       " 'org': 365,\n",
       " 'seem': 366,\n",
       " 'pov': 367,\n",
       " 'fucking': 368,\n",
       " 'saying': 369,\n",
       " 'important': 370,\n",
       " 'written': 371,\n",
       " 'media': 372,\n",
       " 'template': 373,\n",
       " 'bad': 374,\n",
       " 'until': 375,\n",
       " 'always': 376,\n",
       " 'clear': 377,\n",
       " '5': 378,\n",
       " 'states': 379,\n",
       " '4': 380,\n",
       " 'bit': 381,\n",
       " 'term': 382,\n",
       " 'true': 383,\n",
       " \"i'd\": 384,\n",
       " 'further': 385,\n",
       " 'quite': 386,\n",
       " 'review': 387,\n",
       " 'whether': 388,\n",
       " 'claim': 389,\n",
       " 'based': 390,\n",
       " 'guidelines': 391,\n",
       " 'consider': 392,\n",
       " 'research': 393,\n",
       " 'perhaps': 394,\n",
       " 'once': 395,\n",
       " 'version': 396,\n",
       " 'hate': 397,\n",
       " 'criteria': 398,\n",
       " 'getting': 399,\n",
       " 'mention': 400,\n",
       " 'website': 401,\n",
       " 'times': 402,\n",
       " 'c': 403,\n",
       " 'several': 404,\n",
       " 'three': 405,\n",
       " 'idea': 406,\n",
       " 'year': 407,\n",
       " 'considered': 408,\n",
       " 'words': 409,\n",
       " 'changes': 410,\n",
       " 'makes': 411,\n",
       " 'listed': 412,\n",
       " \"there's\": 413,\n",
       " 'cannot': 414,\n",
       " 'left': 415,\n",
       " 'following': 416,\n",
       " 'address': 417,\n",
       " 'care': 418,\n",
       " 'rules': 419,\n",
       " 'notice': 420,\n",
       " 'group': 421,\n",
       " 'facts': 422,\n",
       " 'date': 423,\n",
       " 'each': 424,\n",
       " 'regarding': 425,\n",
       " 'current': 426,\n",
       " 'means': 427,\n",
       " 'second': 428,\n",
       " 'attack': 429,\n",
       " 'gay': 430,\n",
       " 'general': 431,\n",
       " 'american': 432,\n",
       " 'possible': 433,\n",
       " 'dont': 434,\n",
       " 'topic': 435,\n",
       " 'main': 436,\n",
       " 'start': 437,\n",
       " 'en': 438,\n",
       " 'mentioned': 439,\n",
       " 'issues': 440,\n",
       " 'course': 441,\n",
       " 'include': 442,\n",
       " 'jpg': 443,\n",
       " 'create': 444,\n",
       " 'end': 445,\n",
       " '10': 446,\n",
       " 'whole': 447,\n",
       " 'man': 448,\n",
       " 'seen': 449,\n",
       " 'kind': 450,\n",
       " 'known': 451,\n",
       " 'less': 452,\n",
       " 'statement': 453,\n",
       " 'suggest': 454,\n",
       " 'related': 455,\n",
       " 'sense': 456,\n",
       " '2005': 457,\n",
       " 'ok': 458,\n",
       " 'info': 459,\n",
       " 'happy': 460,\n",
       " 'hey': 461,\n",
       " 'big': 462,\n",
       " 'call': 463,\n",
       " 'notability': 464,\n",
       " 'appropriate': 465,\n",
       " 'four': 466,\n",
       " 'including': 467,\n",
       " 'category': 468,\n",
       " 'myself': 469,\n",
       " 'provide': 470,\n",
       " \"wikipedia's\": 471,\n",
       " 'move': 472,\n",
       " 'neutral': 473,\n",
       " 'days': 474,\n",
       " 'line': 475,\n",
       " 'redirect': 476,\n",
       " 'sentence': 477,\n",
       " 'changed': 478,\n",
       " 'nigger': 479,\n",
       " 'news': 480,\n",
       " 'explain': 481,\n",
       " 'included': 482,\n",
       " 'mind': 483,\n",
       " 'contribs': 484,\n",
       " 'ass': 485,\n",
       " 'summary': 486,\n",
       " 'community': 487,\n",
       " 'removing': 488,\n",
       " 'project': 489,\n",
       " 'school': 490,\n",
       " 't': 491,\n",
       " 'started': 492,\n",
       " 'anyway': 493,\n",
       " 'per': 494,\n",
       " 'although': 495,\n",
       " 'answer': 496,\n",
       " 'interest': 497,\n",
       " 'away': 498,\n",
       " 'specific': 499,\n",
       " 'relevant': 500,\n",
       " 'order': 501,\n",
       " 'sign': 502,\n",
       " 'recent': 503,\n",
       " 'looking': 504,\n",
       " 'next': 505,\n",
       " 'moron': 506,\n",
       " 'warning': 507,\n",
       " 'picture': 508,\n",
       " 'love': 509,\n",
       " 'live': 510,\n",
       " 'full': 511,\n",
       " 'later': 512,\n",
       " 'file': 513,\n",
       " 'color': 514,\n",
       " 'discuss': 515,\n",
       " \"you've\": 516,\n",
       " 'attacks': 517,\n",
       " '0': 518,\n",
       " 'faith': 519,\n",
       " 'public': 520,\n",
       " 'wish': 521,\n",
       " 'claims': 522,\n",
       " 'background': 523,\n",
       " 'policies': 524,\n",
       " 'interested': 525,\n",
       " 'wrote': 526,\n",
       " 'especially': 527,\n",
       " 'currently': 528,\n",
       " 'able': 529,\n",
       " 'writing': 530,\n",
       " 'below': 531,\n",
       " '6': 532,\n",
       " 'taken': 533,\n",
       " 'official': 534,\n",
       " 'self': 535,\n",
       " 'wanted': 536,\n",
       " 'nice': 537,\n",
       " 'during': 538,\n",
       " 'web': 539,\n",
       " 'single': 540,\n",
       " 'appears': 541,\n",
       " 'd': 542,\n",
       " 'stuff': 543,\n",
       " 'names': 544,\n",
       " 'itself': 545,\n",
       " 'position': 546,\n",
       " 'certainly': 547,\n",
       " 'everyone': 548,\n",
       " 'country': 549,\n",
       " 'within': 550,\n",
       " 'according': 551,\n",
       " 'high': 552,\n",
       " '2006': 553,\n",
       " 'lead': 554,\n",
       " 'published': 555,\n",
       " '7': 556,\n",
       " 'completely': 557,\n",
       " 'everything': 558,\n",
       " 'books': 559,\n",
       " 'unless': 560,\n",
       " 'report': 561,\n",
       " '20': 562,\n",
       " 'stupid': 563,\n",
       " 'common': 564,\n",
       " 'anti': 565,\n",
       " 'due': 566,\n",
       " '24': 567,\n",
       " 'looks': 568,\n",
       " 'involved': 569,\n",
       " 'god': 570,\n",
       " 'suck': 571,\n",
       " 'pretty': 572,\n",
       " 'edited': 573,\n",
       " 'hard': 574,\n",
       " 'process': 575,\n",
       " 'p': 576,\n",
       " \"won't\": 577,\n",
       " '11': 578,\n",
       " 'truth': 579,\n",
       " 'learn': 580,\n",
       " 'stay': 581,\n",
       " 'came': 582,\n",
       " 'obviously': 583,\n",
       " 'nor': 584,\n",
       " 'future': 585,\n",
       " '100': 586,\n",
       " 'reading': 587,\n",
       " 'response': 588,\n",
       " 'therefore': 589,\n",
       " 'admins': 590,\n",
       " 'party': 591,\n",
       " 'remember': 592,\n",
       " 'talking': 593,\n",
       " 'past': 594,\n",
       " 'sandbox': 595,\n",
       " 'asked': 596,\n",
       " 'city': 597,\n",
       " 'entry': 598,\n",
       " 'ago': 599,\n",
       " \"he's\": 600,\n",
       " 'b': 601,\n",
       " 'argument': 602,\n",
       " 'posted': 603,\n",
       " 'quote': 604,\n",
       " \"wasn't\": 605,\n",
       " 'power': 606,\n",
       " 'game': 607,\n",
       " 'exactly': 608,\n",
       " '8': 609,\n",
       " 'political': 610,\n",
       " 'similar': 611,\n",
       " 'united': 612,\n",
       " '2007': 613,\n",
       " 'national': 614,\n",
       " 'false': 615,\n",
       " 'system': 616,\n",
       " 'whatever': 617,\n",
       " 'dispute': 618,\n",
       " 'working': 619,\n",
       " 'placed': 620,\n",
       " 'paragraph': 621,\n",
       " 'today': 622,\n",
       " 'useful': 623,\n",
       " 'took': 624,\n",
       " 'search': 625,\n",
       " 'guy': 626,\n",
       " 'noticed': 627,\n",
       " 'government': 628,\n",
       " 'regards': 629,\n",
       " 'google': 630,\n",
       " '12': 631,\n",
       " 'reasons': 632,\n",
       " 'side': 633,\n",
       " 'reverting': 634,\n",
       " 'five': 635,\n",
       " 'particular': 636,\n",
       " 'british': 637,\n",
       " 'appreciate': 638,\n",
       " 'deleting': 639,\n",
       " 'administrator': 640,\n",
       " '2008': 641,\n",
       " 'username': 642,\n",
       " 'form': 643,\n",
       " '15': 644,\n",
       " 'problems': 645,\n",
       " 'npov': 646,\n",
       " 'status': 647,\n",
       " 'wikiproject': 648,\n",
       " 'major': 649,\n",
       " 'provided': 650,\n",
       " 'guess': 651,\n",
       " 'cunt': 652,\n",
       " '–': 653,\n",
       " 'law': 654,\n",
       " 'tried': 655,\n",
       " 'bitch': 656,\n",
       " 'rule': 657,\n",
       " '000': 658,\n",
       " 'become': 659,\n",
       " 'often': 660,\n",
       " 'knowledge': 661,\n",
       " 'needed': 662,\n",
       " 'cheers': 663,\n",
       " 'open': 664,\n",
       " 'vandalize': 665,\n",
       " 'fine': 666,\n",
       " 'company': 667,\n",
       " 'almost': 668,\n",
       " 'reply': 669,\n",
       " 'piece': 670,\n",
       " 'stated': 671,\n",
       " 'white': 672,\n",
       " 'along': 673,\n",
       " 'film': 674,\n",
       " 'taking': 675,\n",
       " 'sort': 676,\n",
       " 'banned': 677,\n",
       " 'ban': 678,\n",
       " 'tags': 679,\n",
       " 'certain': 680,\n",
       " 'fat': 681,\n",
       " '9': 682,\n",
       " 'present': 683,\n",
       " 'recently': 684,\n",
       " 'follow': 685,\n",
       " 'uploaded': 686,\n",
       " 'music': 687,\n",
       " 'explanation': 688,\n",
       " 'likely': 689,\n",
       " 'entire': 690,\n",
       " 'shows': 691,\n",
       " 'points': 692,\n",
       " 'definition': 693,\n",
       " \"haven't\": 694,\n",
       " 'description': 695,\n",
       " 'otherwise': 696,\n",
       " '14': 697,\n",
       " 'saw': 698,\n",
       " 'cited': 699,\n",
       " 'citation': 700,\n",
       " 'class': 701,\n",
       " 'unblock': 702,\n",
       " 'generally': 703,\n",
       " 'short': 704,\n",
       " 'aware': 705,\n",
       " 'decide': 706,\n",
       " 'story': 707,\n",
       " 'g': 708,\n",
       " 'type': 709,\n",
       " 'soon': 710,\n",
       " 'family': 711,\n",
       " 'terms': 712,\n",
       " 'set': 713,\n",
       " 'interesting': 714,\n",
       " 'alone': 715,\n",
       " 'sucks': 716,\n",
       " 'small': 717,\n",
       " 'band': 718,\n",
       " 'internet': 719,\n",
       " 'improve': 720,\n",
       " 'conflict': 721,\n",
       " 'theory': 722,\n",
       " 'external': 723,\n",
       " 'contributing': 724,\n",
       " 'uk': 725,\n",
       " 'actual': 726,\n",
       " 'appear': 727,\n",
       " 'simple': 728,\n",
       " 'indeed': 729,\n",
       " 'sex': 730,\n",
       " 'attention': 731,\n",
       " 'contact': 732,\n",
       " 'views': 733,\n",
       " 'week': 734,\n",
       " 'members': 735,\n",
       " 'result': 736,\n",
       " 'test': 737,\n",
       " 'mr': 738,\n",
       " 'moved': 739,\n",
       " 'email': 740,\n",
       " '16': 741,\n",
       " 'told': 742,\n",
       " 'area': 743,\n",
       " 'obvious': 744,\n",
       " 'john': 745,\n",
       " 'black': 746,\n",
       " '2004': 747,\n",
       " 'themselves': 748,\n",
       " 'allowed': 749,\n",
       " 'previous': 750,\n",
       " 'various': 751,\n",
       " 'proposed': 752,\n",
       " 'cite': 753,\n",
       " \"article's\": 754,\n",
       " 'sourced': 755,\n",
       " 'copy': 756,\n",
       " 'hand': 757,\n",
       " 'disagree': 758,\n",
       " 'bias': 759,\n",
       " 'author': 760,\n",
       " 'thus': 761,\n",
       " 'went': 762,\n",
       " 'works': 763,\n",
       " 'sites': 764,\n",
       " 'jewish': 765,\n",
       " '17': 766,\n",
       " 'third': 767,\n",
       " 'ones': 768,\n",
       " 'together': 769,\n",
       " '18': 770,\n",
       " 'citations': 771,\n",
       " 'nonsense': 772,\n",
       " 'death': 773,\n",
       " 'actions': 774,\n",
       " 'bullshit': 775,\n",
       " \"aren't\": 776,\n",
       " 'hours': 777,\n",
       " 'context': 778,\n",
       " 'addition': 779,\n",
       " 'automatically': 780,\n",
       " 'fish': 781,\n",
       " 'longer': 782,\n",
       " 'guys': 783,\n",
       " 'german': 784,\n",
       " 'action': 785,\n",
       " 'enjoy': 786,\n",
       " 'large': 787,\n",
       " 'biased': 788,\n",
       " 'university': 789,\n",
       " 'avoid': 790,\n",
       " 'creating': 791,\n",
       " 'worked': 792,\n",
       " '13': 793,\n",
       " 'happened': 794,\n",
       " 'proper': 795,\n",
       " \"wouldn't\": 796,\n",
       " 'level': 797,\n",
       " 'human': 798,\n",
       " 'goes': 799,\n",
       " 'de': 800,\n",
       " 'valid': 801,\n",
       " 'job': 802,\n",
       " 'science': 803,\n",
       " 'deal': 804,\n",
       " 'manual': 805,\n",
       " '23': 806,\n",
       " \"what's\": 807,\n",
       " '21': 808,\n",
       " '19': 809,\n",
       " 'series': 810,\n",
       " 'seriously': 811,\n",
       " 'width': 812,\n",
       " 'accepted': 813,\n",
       " 'standard': 814,\n",
       " 'act': 815,\n",
       " 'available': 816,\n",
       " 'helpful': 817,\n",
       " 'accept': 818,\n",
       " 'personally': 819,\n",
       " 'f': 820,\n",
       " 'himself': 821,\n",
       " 'tildes': 822,\n",
       " 'afd': 823,\n",
       " '2010': 824,\n",
       " '\\xa0': 825,\n",
       " 'exist': 826,\n",
       " 'meaning': 827,\n",
       " 'july': 828,\n",
       " 'assume': 829,\n",
       " '22': 830,\n",
       " 'living': 831,\n",
       " 'comes': 832,\n",
       " 'usually': 833,\n",
       " 'respect': 834,\n",
       " '2009': 835,\n",
       " 'faggot': 836,\n",
       " 'greek': 837,\n",
       " \"shouldn't\": 838,\n",
       " 'criticism': 839,\n",
       " 'huge': 840,\n",
       " 'upon': 841,\n",
       " 'rights': 842,\n",
       " 'indicate': 843,\n",
       " 'debate': 844,\n",
       " 'necessary': 845,\n",
       " 'video': 846,\n",
       " 'access': 847,\n",
       " 'multiple': 848,\n",
       " 'play': 849,\n",
       " 'yeah': 850,\n",
       " 'sections': 851,\n",
       " 'violation': 852,\n",
       " 'months': 853,\n",
       " 'hell': 854,\n",
       " 'proof': 855,\n",
       " 'calling': 856,\n",
       " 'tagged': 857,\n",
       " 'pro': 858,\n",
       " 'opinions': 859,\n",
       " 'south': 860,\n",
       " 'statements': 861,\n",
       " 'historical': 862,\n",
       " 'details': 863,\n",
       " \"let's\": 864,\n",
       " '30': 865,\n",
       " 'accurate': 866,\n",
       " 'attempt': 867,\n",
       " 'church': 868,\n",
       " 'rest': 869,\n",
       " 'speak': 870,\n",
       " 'balls': 871,\n",
       " 'situation': 872,\n",
       " \"they're\": 873,\n",
       " 'blocking': 874,\n",
       " 'm': 875,\n",
       " 'v': 876,\n",
       " 'record': 877,\n",
       " 'doubt': 878,\n",
       " 'bark': 879,\n",
       " 'fix': 880,\n",
       " 'complete': 881,\n",
       " 'w': 882,\n",
       " 'im': 883,\n",
       " 'okay': 884,\n",
       " 'serious': 885,\n",
       " 'separate': 886,\n",
       " 'march': 887,\n",
       " 'run': 888,\n",
       " 'kill': 889,\n",
       " 'space': 890,\n",
       " 'explaining': 891,\n",
       " 'none': 892,\n",
       " 'online': 893,\n",
       " 'august': 894,\n",
       " 'behavior': 895,\n",
       " 'anonymous': 896,\n",
       " 'r': 897,\n",
       " 'heard': 898,\n",
       " 'messages': 899,\n",
       " 'cause': 900,\n",
       " 'rationale': 901,\n",
       " 'prove': 902,\n",
       " 'quality': 903,\n",
       " \"you'll\": 904,\n",
       " 'apparently': 905,\n",
       " 'team': 906,\n",
       " 'changing': 907,\n",
       " 'html': 908,\n",
       " 'lack': 909,\n",
       " 'military': 910,\n",
       " 'period': 911,\n",
       " 'difference': 912,\n",
       " 'close': 913,\n",
       " 'data': 914,\n",
       " 'jews': 915,\n",
       " 'dick': 916,\n",
       " 'refer': 917,\n",
       " 'penis': 918,\n",
       " 'special': 919,\n",
       " 'legal': 920,\n",
       " 'culture': 921,\n",
       " 'asking': 922,\n",
       " 'early': 923,\n",
       " 'india': 924,\n",
       " 'wikipedian': 925,\n",
       " 'contribute': 926,\n",
       " 'gets': 927,\n",
       " 'sock': 928,\n",
       " 'supposed': 929,\n",
       " 'couple': 930,\n",
       " 'tv': 931,\n",
       " 'lol': 932,\n",
       " 'box': 933,\n",
       " 'among': 934,\n",
       " 'existing': 935,\n",
       " 'countries': 936,\n",
       " 'archive': 937,\n",
       " 'pillars': 938,\n",
       " 'head': 939,\n",
       " 'meant': 940,\n",
       " 'except': 941,\n",
       " 'table': 942,\n",
       " 'pictures': 943,\n",
       " 'asshole': 944,\n",
       " 'directly': 945,\n",
       " 'born': 946,\n",
       " 'specifically': 947,\n",
       " 'modern': 948,\n",
       " 'jew': 949,\n",
       " 'watch': 950,\n",
       " 'uses': 951,\n",
       " 'incorrect': 952,\n",
       " 'described': 953,\n",
       " 'border': 954,\n",
       " 'idiot': 955,\n",
       " 'field': 956,\n",
       " 'outside': 957,\n",
       " 'purpose': 958,\n",
       " 'red': 959,\n",
       " 'civil': 960,\n",
       " 'warring': 961,\n",
       " 'disruptive': 962,\n",
       " 'half': 963,\n",
       " 'significant': 964,\n",
       " 'produce': 965,\n",
       " '50': 966,\n",
       " 'cases': 967,\n",
       " '25': 968,\n",
       " 'error': 969,\n",
       " 'photo': 970,\n",
       " 'shall': 971,\n",
       " 'primary': 972,\n",
       " 'christian': 973,\n",
       " 'million': 974,\n",
       " 'inclusion': 975,\n",
       " 'song': 976,\n",
       " 'aids': 977,\n",
       " '2012': 978,\n",
       " 'character': 979,\n",
       " 'release': 980,\n",
       " 'business': 981,\n",
       " 'sometimes': 982,\n",
       " 'x': 983,\n",
       " 'numbers': 984,\n",
       " 'vote': 985,\n",
       " 'align': 986,\n",
       " 'january': 987,\n",
       " 'towards': 988,\n",
       " 'control': 989,\n",
       " 'home': 990,\n",
       " 'administrators': 991,\n",
       " 'friend': 992,\n",
       " 'june': 993,\n",
       " 'bring': 994,\n",
       " 'linked': 995,\n",
       " 'thinking': 996,\n",
       " 'tutorial': 997,\n",
       " 'particularly': 998,\n",
       " 'house': 999,\n",
       " 'organization': 1000,\n",
       " ...}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.word_index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual, we will need to repeat this transformation to indexes for the test data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    " <tr><td width=\"80\"><img src=\"img/question.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">\n",
    "Repeat the transformation to sequences of indexes for the test data. Save the result of the transformation into an <b>X_test</b> variable..\n",
    " </td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "####### INSERT YOUR CODE HERE\n",
    "X_test = tokenizer.texts_to_sequences(test[\"comment_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is enough to train a Sequential Network. However, for efficiency reasons it is recommended that all sequences in the data have the same number of elements. Since this is not the case for our data, as each text has a different number of words, we should **pad** the sequences to ensure the same length. The padding procedure adds a special *null* symbol to short sequences, and clips out parts of long sequences, thus enforcing a common size.\n",
    "\n",
    "This procedure can be performed easily by again resorting to keras tools. To choose an appropriate padding length we can have a look at a boxplot of the number of tokens in our training texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAD8CAYAAABthzNFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAADx1JREFUeJzt3X+MZWV9x/H3pyBaahGWnRDKD3cRSmJbrWZVfhijUlNUKjT+qNbaLdJs0hDFH1UQm6BNTTWxKhJrui3W1ShK0QRqG41F0FqUuKvAImjcgOhSkDWyK/0RC/LtH/dMmdBnZs7OnXvPzOz7ldzce55zzjzf7B/7yTnPOc+TqkKSpEf7haELkCStTAaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0HD13AONavX18bNmwYugxJWlV27Njx46qaWey4VR0QGzZsYPv27UOXIUmrSpK7+hznLSZJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmlb1i3LStCSZSj+uEa+VxICQetjf/7iT+J+9Vj1vMUmSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNU0sIJJ8JMl9SW6d07YuyReTfK/7PqJrT5IPJtmV5JYkT59UXZKkfiZ5BfFR4MxHtV0EXFtVJwHXdtsALwRO6j5bgA9PsC5JUg8TC4iq+grwk0c1nw1s635vA86Z0/6xGvk6cHiSoydVmyRpcdMegziqqu7pft8LHNX9Pgb44ZzjdndtkqSBDDZIXaP1GPd7TcYkW5JsT7J9z549E6hMkgTTD4gfzd466r7v69rvBo6bc9yxXdv/U1Vbq2pTVW2amZmZaLGSdCCbdkBcA2zufm8Grp7T/ofd00ynAPvm3IqSJA3g4En94SRXAM8F1ifZDVwCvBu4Msl5wF3AK7rD/xl4EbAL+C/g3EnVJUnqZ2IBUVWvmmfXGY1jCzh/UrVIkvafb1JLkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1GRCSpCYDQpLUZEBIkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqSmQQIiyRuTfDvJrUmuSPK4JBuT3JhkV5JPJzlkiNokSSNTD4gkxwCvBzZV1a8DBwGvBN4DvL+qTgTuB86bdm2SpEcsGhBJTu/Ttp8OBn4xycHAocA9wPOBq7r924BzxuxDkjSGPlcQl/Vs66Wq7gbeC/yAUTDsA3YAe6vqoe6w3cAxS+1DkjS+g+fbkeRU4DRgJsmb5uw6jNFtoSVJcgRwNrAR2Av8A3Dmfpy/BdgCcPzxxy+1DEnSIha6gjgEeDyjEPnlOZ+fAi8bo8/fAu6sqj1V9SDwWeB04PDulhPAscDdrZOramtVbaqqTTMzM2OUIUlayLxXEFX1ZeDLST5dVd+Zuy/J+jH6/AFwSpJDgf8GzgC2A9cxCp5PAZuBq8foQ5I0pj5jEFcmOWV2I8lLgRuW2mFV3choMPqbwM6uhq3AhcCbkuwCjgQuX2ofkqTxzXsFMcergY8kuR74FUb/eT9/nE6r6hLgkkc13wE8c5y/K0laPosGRFXtTPIu4OPAA8Bzqmr3xCuTJA1q0YBIcjnwJOApwK8Cn0tyWVV9aNLFSZKG02cMYifwvKq6s6q+ADwLePpky5IkDW3RgKiqDwCPS3Jyt72vqpwGQ5LWuD5TbfwOcBPw+W77N5NcM+nCJEnD6nOL6R2Mni7aC1BVNwEnTLAmSdIK0CcgHqyqfY9qe3gSxUiSVo4+70F8O8nvAwclOYnRVN1LflFOkrQ69LmCeB3wa8DPgE8ymn31gkkWJU3SunXrSDLRDzDxPpKwbt26gf81tZb1uYJ4cVW9HXj7bEOSlzOahVVade6//36qaugylsVsGEmT0OcK4m092yRJa8hC60G8EHgRcEySD87ZdRjwUPssSdJasdAtpn9nNA33Sxit+DbrAeCNkyxKkjS8hdaDuBm4Ocknu4V9JEkHkD5TbRgOknQA6jNILUk6ABkQkqSmPutBzDBaDvTJwONm26tqrFXlJEkrW58riE8AtwMbgXcC3we+McGaJEkrQJ+AOLKqLmc0ad+Xq+q1jLkmtSRp5esz1cbsU0z3JHkxo/cjnABGkta4PgHxF0meALwZuIzRm9S+KCdJa9yiAVFVn+t+7gOeN9lyJEkrhY+5SpKaDAhJUpMBIUlq6vOi3GOBlwIb5h5fVX8+ubIkSUPr8xTT1YwGqHcwWnZUknQA6BMQx1bVmROvRJK0ovQZg7ghyW9MvBJJ0orSJyCeDexI8t0ktyTZmeSWcTpNcniSq5J8J8ntSU5Nsi7JF5N8r/s+Ypw+JEnj6XOL6YUT6PdS4PNV9bIkhwCHAhcD11bVu5NcBFzEaBZZSdIA5r2CSHJY9/OBeT5L0k3b8RzgcoCq+p+q2gucDWzrDtsGnLPUPiRJ41voCuKTwFmMnl4qIHP2FXDCEvvcCOwB/j7JU7u/fwFwVFXd0x1zL3DUEv++JGkZzBsQVXVW971xAn0+HXhdVd2Y5FJGt5Pm9l1JqnVyki3AFoDjjz9+mUuTJM0a4k3q3cDuqrqx276KUWD8KMnRAN33fa2Tq2prVW2qqk0zMzNTKViSDkRTD4iquhf4YZKTu6YzgNuAa4DNXdtmRi/oSZIG0ucppkl4HfCJ7gmmO4BzGYXVlUnOA+4CXjFQbZIk+s3F9CRGt4R+luS5wFOAj3VPHi1JVd0EbGrsOmOpf1OStLz63GL6DPDzJCcCW4HjGD3hJElaw/oExMNV9RDwu8BlVfUW4OjJliVJGlqfgHgwyasYDRzPLj/6mMmVJElaCfoExLnAqcC7qurOJBuBj0+2LEnS0BYdpK6q24DXz9m+E3jPJIuSJA1v3oBIspPRlBpNVfWUiVQkSVoRFrqCOKv7Pr/7nr2t9AcsEBySpLVhobmY7gJI8oKqetqcXRcm+SaPmj9JkrS29BmkTpLT52yc1vM8SdIq1meqjdcympr7Cd323q5NkrSGLRgQSX4BOLGqnjobEFW1byqVSZIGteCtoqp6GHhr93uf4SBJB44+Ywn/kuRPkxyXZN3sZ+KVSZIG1WcM4ve67/PntI2z5KgkaRXo8yb1ci85KklaBfqsB/EY4E+A53RN1wN/U1UPTrAuSdLA+txi+jCj2Vv/utt+Tdf2x5MqSpI0vD4B8Yyqeuqc7S8luXlSBUmSVoY+TzH9vFt2FIAkJwA/n1xJkqSVoM8VxFuA65LcAQR4IqM1IiRJa1ifp5iuTXIScHLX9N2q+tlky5IkDW2h9SBuBv6t+9xQVbdMrSpJ0uAWGoN4NXAT8ALgC0nuTnJVkjcmedZ0ypMkDWWh9SBuBW4FtgIkWQ+8EngD8F7goGkUKEkaxkK3mA4CngacBpwOPAm4G/g74GtTqU6SNJiFBqkfAG4DPgRcVFV3TqckSdJKsFBAnAecyuiN6XOTfIPRlcPXquruaRQnSRrOQmMQVwBXACQ5FHgmo9tNf5nkkKp64nRKlCQNYbEV5X4JeBaPjEM8A/gho0dfJUlr2EKD1N8CjgN2MAqEvwK+XlX/MaXaJEkDWugKYjOws6pqWsVIklaOeV+Uq6pbqqqS/NlsW5LHLlfHSQ5K8q0kn+u2Nya5McmuJJ9Ocshy9SVJ2n/zBkSSC5OcCrxsTvNyvv9wAXD7nO33AO+vqhOB+xk9RSVJGshCU218B3g5cEKSf03yt8CRSU5e4JxekhwLvJjRS3ckCfB84KrukG3AOeP2I0lauoUCYi9wMbALeC5wadd+UZIbxuz3A8BbgYe77SOBvVX1ULe9GzhmzD4kSWNYKCB+G/gnRlNsvI/R467/WVXnVtVpS+0wyVnAfVW1Y4nnb0myPcn2PXv2LLUMSdIiFhqkvriqzgC+D3yc0eR8M0m+muQfx+jzdOAlSb4PfIrRraVLgcOTzD5VdSyjeZ9adW2tqk1VtWlmZmaMMiRJC+mz5OgXqmp7VW0FdlfVsxljRbmqeltVHVtVGxjNDvulqno1cB2PDIhvBq5eah+SpPEtGhBV9dY5m3/Utf14ArVcCLwpyS5GYxKXT6APSVJPfdak/j9VdfNydl5V1wPXd7/vYDTfkyRpBehzi0mSdAAyICRJTQaEJKnJgJAkNRkQkqQmA0KS1LRfj7lKa0Fdchi84wlDl7Es6pLDhi5Ba5gBoQNO3vlT1so6WEmodwxdhdYqbzFJkpoMCElSkwEhSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1TT0gkhyX5LoktyX5dpILuvZ1Sb6Y5Hvd9xHTrk2S9IghriAeAt5cVU8GTgHOT/Jk4CLg2qo6Cbi225YkDWTqAVFV91TVN7vfDwC3A8cAZwPbusO2AedMuzZJ0iMGHYNIsgF4GnAjcFRV3dPtuhc4aqCyJEkMGBBJHg98BnhDVf107r6qKqDmOW9Lku1Jtu/Zs2cKlUrSgWmQgEjyGEbh8Imq+mzX/KMkR3f7jwbua51bVVuralNVbZqZmZlOwZJ0ABriKaYAlwO3V9X75uy6Btjc/d4MXD3t2iRJjzh4gD5PB14D7ExyU9d2MfBu4Mok5wF3Aa8YoDZJUmfqAVFVXwUyz+4zplmLJGl+vkktSWoyICRJTQaEJKlpiEFqaXCjh+lWvyOOcMoyTY4BoQPO6D3MyUoylX6kSfIWkySpyYCQJDUZEJKkJgNCktRkQEiSmgwISVKTASFJajIgJElNBoQkqcmAkCQ1OdWG1MNS5m5ayjlOz6GVxICQevA/bh2IvMUkSWoyICRJTQaEJKnJgJAkNRkQkqQmA0KS1GRASJKaDAhJUlNW8wtASfYAdw1dh9SwHvjx0EVI83hiVc0sdtCqDghppUqyvao2DV2HNA5vMUmSmgwISVKTASFNxtahC5DG5RiEJKnJKwhJUpMBIS2jJB9Jcl+SW4euRRqXASEtr48CZw5dhLQcDAhpGVXVV4CfDF2HtBwMCElSkwEhSWoyICRJTQaEJKnJgJCWUZIrgK8BJyfZneS8oWuSlso3qSVJTV5BSJKaDAhJUpMBIUlqMiAkSU0GhCSpyYCQJDUZEJKkJgNCktT0vwOzd89fflrVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.boxplot([len(text) for text in X_train], whis=[15, 85], showfliers=False)\n",
    "plt.ylabel(\"#Words in a text\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will pad all sequences to have a maximum of 120 indexes (words), which is enough to represent 85% of our texts without dropping tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.preprocessing.sequence import pad_sequences \n",
    "maxsequence = 120\n",
    "X_train = pad_sequences(X_train, maxlen=maxsequence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    " <tr><td width=\"80\"><img src=\"img/question.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">\n",
    "     Repeat the process for the test data, saving the result into a variable <b>X_test</b>.\n",
    " </td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "####### INSERT YOUR CODE HERE\n",
    "X_test = pad_sequences(X_test, maxlen=maxsequence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The texts are now ready to serve as inputs for the model. We must also process the toxicity labels to serve as outputs. To do so, just create Dataframe or numpy arrays **y_train** and **y_test** containing the 6 columns with the toxicity indicators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "####### INSERT YOUR CODE HERE\n",
    "y_train = train.drop(\"comment_text\", axis=1).values\n",
    "y_test = test.drop(\"comment_text\", axis=1).values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Continuous Bag of Words model (CBoW)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/CBoW.png\" style=\"width:800px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start with a simple, yet effective model to deal with sequences of words: the Continuous Bag of Words (CBoW) model. This model is comprised of:\n",
    "\n",
    "* An [Embedding](https://keras.io/layers/embeddings/) layer that transforms word indexes to a vector representation that is learned with the model. \n",
    "* A  **GlobalAveragePooling1D** layer that obtains a representation of the whole document as the average of all its word vector representations\n",
    "* A final layer taking the decision of the network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by creating a keras Sequential model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "\n",
    "model = Sequential()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first layer in a network that works with indexes should be the Embedding layer. We will transform each index (word) into an embedding vector of 64 elements, which will be learned by the network. This can be done as"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers.embeddings import Embedding\n",
    "\n",
    "model.add(Embedding(input_dim=maxwords, input_length=maxsequence, output_dim=64))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how the Embedding layer requires three parameters: the number of different words in our problem (input_dim), the length of the input sequences (input_length) and the size of the embedding vectors to be created (output_dim)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the embedding we will add the averaging layer to obtain a single vector of values representing the whole document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import GlobalAveragePooling1D\n",
    "\n",
    "model.add(GlobalAveragePooling1D())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After this we will add the output layer. The problem we are trying to solve is a **multilabel** problem: there are 6 possible toxic classes for each text, but a text might include several of these classes, or none of them. Therefore, the output layer should contain 6 neurons followed by a sigmoid activation. This way each class might activate independently of the rest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers.core import Dense, Activation\n",
    "\n",
    "model.add(Dense(6))\n",
    "model.add(Activation('sigmoid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And with this the design of the CBoW is finished! Let's check what we have built"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 120, 64)           960000    \n",
      "_________________________________________________________________\n",
      "global_average_pooling1d_1 ( (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 6)                 390       \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 6)                 0         \n",
      "=================================================================\n",
      "Total params: 960,390\n",
      "Trainable params: 960,390\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how most of the model parameters are located in the Embedding layer. That is, the focus of the network is on finding good word representations that allow classifying the sentiment of the sentence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now follow the usual procedure for compiling the network. Note that since we have a multiple but independent output neurons, we select **binary_crossentropy** as the network loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we proceed on to train the network. Since the CBoW model is very simple, training should be reasonably fast."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "119678/119678 [==============================] - 21s 174us/step - loss: 0.1319 - acc: 0.9635\n",
      "Epoch 2/10\n",
      "119678/119678 [==============================] - 14s 119us/step - loss: 0.0881 - acc: 0.9694\n",
      "Epoch 3/10\n",
      "119678/119678 [==============================] - 15s 122us/step - loss: 0.0750 - acc: 0.9749\n",
      "Epoch 4/10\n",
      "119678/119678 [==============================] - 14s 121us/step - loss: 0.0674 - acc: 0.9779\n",
      "Epoch 5/10\n",
      "119678/119678 [==============================] - 15s 129us/step - loss: 0.0627 - acc: 0.9797\n",
      "Epoch 6/10\n",
      "119678/119678 [==============================] - 15s 129us/step - loss: 0.0592 - acc: 0.9808\n",
      "Epoch 7/10\n",
      "119678/119678 [==============================] - 17s 139us/step - loss: 0.0566 - acc: 0.9815\n",
      "Epoch 8/10\n",
      "119678/119678 [==============================] - 17s 142us/step - loss: 0.0541 - acc: 0.9823\n",
      "Epoch 9/10\n",
      "119678/119678 [==============================] - 21s 171us/step - loss: 0.0520 - acc: 0.9828\n",
      "Epoch 10/10\n",
      "119678/119678 [==============================] - 19s 160us/step - loss: 0.0500 - acc: 0.9835\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f94e1a0dcc0>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train, batch_size=32, epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After training we can measure the performance over the test set as usual"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "39893/39893 [==============================] - 2s 54us/step\n",
      "Test loss 0.07346271173992797\n",
      "Test accuracy 0.9794617149619075\n"
     ]
    }
   ],
   "source": [
    "score = model.evaluate(X_test, y_test)\n",
    "print(\"Test loss\", score[0])\n",
    "print(\"Test accuracy\", score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note however that accuracy might not be the best metric for this problem: fortunately most of the comments in social media are not toxic, and so it's easy to obtain good accuracy by just classifying everything as non-toxic. In this kind of unbalanced problems it is better to make use of the [ROC AUC score](https://scikit-learn.org/stable/modules/model_evaluation.html#roc-metrics)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC score 0.9477635620452846\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "y_pred = model.predict(X_test)\n",
    "print(\"AUC score\", roc_auc_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolutional mixing model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the average of word embeddings is not optimal, since we are throwing away information about word ordering. A simple method to take neighborhoods of words into account is to add a small convolutional layer after the Embeddings layer. This way the embeddings of each word gets mixed with the embeddings of their neighbouring words, thus producing a more contextualized representation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    " <tr><td width=\"80\"><img src=\"img/question.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">\n",
    "     Create a new model that follows the same architecture as the CBoW model above, but including a <a href=\"https://keras.io/layers/convolutional/\">1-dimensional convolution</a> between the Embedding and the GlobalAveragePooling1D layers. You can use 64 filters with kernel size 9: this will mix the embeddings of length 64 of each word into new embeddings again of size 64, that now contain also information from close words. Try also with a small number of epochs, in the range 1-3. Do you obtain better results with this model?\n",
    " </td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_11 (Embedding)     (None, 120, 64)           960000    \n",
      "_________________________________________________________________\n",
      "conv1d_2 (Conv1D)            (None, 120, 64)           36928     \n",
      "_________________________________________________________________\n",
      "global_average_pooling1d_5 ( (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "dense_10 (Dense)             (None, 6)                 390       \n",
      "_________________________________________________________________\n",
      "activation_8 (Activation)    (None, 6)                 0         \n",
      "=================================================================\n",
      "Total params: 997,318\n",
      "Trainable params: 997,318\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Train on 119678 samples, validate on 39893 samples\n",
      "Epoch 1/2\n",
      "119678/119678 [==============================] - 22s 185us/step - loss: 0.0817 - acc: 0.9759 - val_loss: 0.0652 - val_acc: 0.9810\n",
      "Epoch 2/2\n",
      "119678/119678 [==============================] - 27s 229us/step - loss: 0.0583 - acc: 0.9816 - val_loss: 0.0619 - val_acc: 0.9806\n",
      "AUC score 0.9552490512365361\n"
     ]
    }
   ],
   "source": [
    "####### INSERT YOUR CODE HERE\n",
    "from keras.layers.convolutional import Conv1D\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(maxwords, 64, input_length=maxsequence))\n",
    "model.add(Conv1D(filters=64, kernel_size=9, padding='same', activation='relu'))\n",
    "model.add(GlobalAveragePooling1D())\n",
    "model.add(Dense(6))\n",
    "model.add(Activation('sigmoid'))\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[\"accuracy\"])\n",
    "model.summary()\n",
    "\n",
    "model.fit(X_train, y_train, batch_size=32, epochs=2, validation_data=(X_test, y_test))\n",
    "\n",
    "y_pred = model.predict(X_test)\n",
    "print(\"AUC score\", roc_auc_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LSTM model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/LSTM.png\" style=\"width:600px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even if a simple convolutional mixing of neighbouring words might help, we are still missing something important. Language makes use of relationships between distant words to produce meaningful sentences, and we will need to make use of a more powerful mixing model to improve our results. To do so, we will make use of a Long-short Term Memory layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An LSTM layer that creates a document-level representation as a vector of 32 values can be created with Keras as follows "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers.recurrent import LSTM\n",
    "\n",
    "lstm_layer = LSTM(32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    " <tr><td width=\"80\"><img src=\"img/exclamation.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">\n",
    "     If you are training your networks in an nVidia GPU, <b>don't use the LSTM layer</b>. Instead, use the <a href=\"https://keras.io/layers/recurrent/\">CuDNNLSTM layer</a>. This is a different implementation of the LSTM neurons that is highly optimized for nVidia GPUs, and can make your model train 10x times faster than using the standard LSTM implementation.\n",
    " </td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If no other parameters are given the LSTM will process the whole input sequence, and forward the computed document-level representation after processing the last element of the sequence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    " <tr><td width=\"80\"><img src=\"img/question.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">\n",
    "Create an LSTM model by copying the design from the CBoW model, but replacing the averaging layer by an LSTM layer of 32 units. Can you obtain better test results with this model?\n",
    " </td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    " <tr><td width=\"80\"><img src=\"img/exclamation.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">\n",
    "LSTM layers are expensive to compute, due to their recurrent nature. Don't worry if training takes much longer than with the CBoW model. Also, LSTM networks are prone to overfitting. For now you can control this by limiting the number if epochs of training in the model. Try with a very small number of epochs, in the range 1-3.\n",
    " </td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_3 (Embedding)      (None, 120, 64)           960000    \n",
      "_________________________________________________________________\n",
      "cu_dnnlstm_1 (CuDNNLSTM)     (None, 32)                12544     \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 6)                 198       \n",
      "_________________________________________________________________\n",
      "activation_3 (Activation)    (None, 6)                 0         \n",
      "=================================================================\n",
      "Total params: 972,742\n",
      "Trainable params: 972,742\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Train on 119678 samples, validate on 39893 samples\n",
      "Epoch 1/2\n",
      "119678/119678 [==============================] - 47s 396us/step - loss: 0.0724 - acc: 0.9770 - val_loss: 0.0512 - val_acc: 0.9818\n",
      "Epoch 2/2\n",
      "119678/119678 [==============================] - 45s 378us/step - loss: 0.0473 - acc: 0.9826 - val_loss: 0.0501 - val_acc: 0.9821\n",
      "AUC score 0.9722477489743971\n"
     ]
    }
   ],
   "source": [
    "####### INSERT YOUR CODE HERE\n",
    "from keras.layers import CuDNNLSTM\n",
    "model = Sequential()\n",
    "model.add(Embedding(input_dim=maxwords, input_length=maxsequence, output_dim=64))\n",
    "model.add(CuDNNLSTM(32))\n",
    "model.add(Dense(6))\n",
    "model.add(Activation('sigmoid'))\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[\"accuracy\"])\n",
    "model.summary()\n",
    "\n",
    "model.fit(X_train, y_train, batch_size=32, epochs=2, validation_data=(X_test, y_test))\n",
    "\n",
    "y_pred = model.predict(X_test)\n",
    "print(\"AUC score\", roc_auc_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GRU model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes it is more effective to use a simplified version of the LSTM layer. This would be the Gated Recurrent Unit layer, which can be created in Keras as the GRU layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import GRU\n",
    "\n",
    "gru_layer = GRU(32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    " <tr><td width=\"80\"><img src=\"img/exclamation.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">\n",
    "     Similarly to the LSTM layer, if you are training your GRU network in an nVidia GPU, it is highly recommended that you use the <a href=\"https://keras.io/layers/recurrent/\">CuDNNGRU layer</a> instead.\n",
    " </td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    " <tr><td width=\"80\"><img src=\"img/question.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">\n",
    "Repeat the model above, replacing the LSTM layer by a GRU layer. Can you obtain better results with this simplified recurrent network?\n",
    " </td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_4 (Embedding)      (None, 120, 64)           960000    \n",
      "_________________________________________________________________\n",
      "cu_dnngru_1 (CuDNNGRU)       (None, 32)                9408      \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 6)                 198       \n",
      "_________________________________________________________________\n",
      "activation_4 (Activation)    (None, 6)                 0         \n",
      "=================================================================\n",
      "Total params: 969,606\n",
      "Trainable params: 969,606\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Train on 119678 samples, validate on 39893 samples\n",
      "Epoch 1/2\n",
      "119678/119678 [==============================] - 43s 358us/step - loss: 0.0683 - acc: 0.9780 - val_loss: 0.0497 - val_acc: 0.9822\n",
      "Epoch 2/2\n",
      "119678/119678 [==============================] - 41s 345us/step - loss: 0.0454 - acc: 0.9832 - val_loss: 0.0469 - val_acc: 0.9831\n",
      "AUC score 0.9762819317866679\n"
     ]
    }
   ],
   "source": [
    "####### INSERT YOUR CODE HERE\n",
    "from keras.layers import CuDNNGRU\n",
    "model = Sequential()\n",
    "model.add(Embedding(input_dim=maxwords, input_length=maxsequence, output_dim=64))\n",
    "model.add(CuDNNGRU(32))\n",
    "model.add(Dense(6))\n",
    "model.add(Activation('sigmoid'))\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[\"accuracy\"])\n",
    "model.summary()\n",
    "\n",
    "model.fit(X_train, y_train, batch_size=32, epochs=2, validation_data=(X_test, y_test))\n",
    "\n",
    "y_pred = model.predict(X_test)\n",
    "print(\"AUC score\", roc_auc_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Global MaxPooling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An alternative way to summarize the sequence of inputs in the LSTM/GRU layer is to use a different aggregation operation instead of returning the layer outputs at the last value of the sequence. To do this we will tell the LSTM/GRU to return a sequence, which means the layer will produce an output vector for each element in the sequence, hence generating a transformed sequence instead. For instance, in the LSTM layer this is done by defining it as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<keras.layers.recurrent.LSTM at 0x7f94e1c385c0>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LSTM(32, return_sequences=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the **return_sequences** parameter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We still need some mechanism to summarize the sequence returned by the LSTM into a single vector. A popular method inspired by convolutional networks is to use MaxPooling. To apply this over a sequence we will use the **GlobalMaxPool1D** layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<keras.layers.pooling.GlobalMaxPooling1D at 0x7f94e1c38898>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras.layers import GlobalMaxPool1D\n",
    "GlobalMaxPool1D()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    " <tr><td width=\"80\"><img src=\"img/question.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">\n",
    "Define a new Embeddings + LSTM/GRU where the recurrent layer returns a sequence, and a GlobalMaxPool1D layer is used as a summary after that. Are you able to obtain better results in test with this architecture?\n",
    " </td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_5 (Embedding)      (None, 120, 64)           960000    \n",
      "_________________________________________________________________\n",
      "cu_dnngru_2 (CuDNNGRU)       (None, 120, 32)           9408      \n",
      "_________________________________________________________________\n",
      "global_max_pooling1d_2 (Glob (None, 32)                0         \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 6)                 198       \n",
      "_________________________________________________________________\n",
      "activation_5 (Activation)    (None, 6)                 0         \n",
      "=================================================================\n",
      "Total params: 969,606\n",
      "Trainable params: 969,606\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Train on 119678 samples, validate on 39893 samples\n",
      "Epoch 1/3\n",
      "119678/119678 [==============================] - 42s 354us/step - loss: 0.0801 - acc: 0.9759 - val_loss: 0.0503 - val_acc: 0.9824\n",
      "Epoch 2/3\n",
      "119678/119678 [==============================] - 42s 351us/step - loss: 0.0448 - acc: 0.9835 - val_loss: 0.0473 - val_acc: 0.9827\n",
      "Epoch 3/3\n",
      "119678/119678 [==============================] - 42s 350us/step - loss: 0.0393 - acc: 0.9851 - val_loss: 0.0474 - val_acc: 0.9828\n",
      "AUC score 0.976563375285517\n"
     ]
    }
   ],
   "source": [
    "####### INSERT YOUR CODE HERE\n",
    "from keras.layers import GlobalMaxPool1D\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(input_dim=maxwords, input_length=maxsequence, output_dim=64))\n",
    "model.add(CuDNNGRU(32, return_sequences=True))\n",
    "model.add(GlobalMaxPool1D())\n",
    "model.add(Dense(6))\n",
    "model.add(Activation('sigmoid'))\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[\"accuracy\"])\n",
    "model.summary()\n",
    "\n",
    "model.fit(X_train, y_train, batch_size=32, epochs=3, validation_data=(X_test, y_test))\n",
    "\n",
    "y_pred = model.predict(X_test)\n",
    "print(\"AUC score\", roc_auc_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bidirectional mixing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/bidi.png\" style=\"width:500px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another improvement to the recurrent mixing architecture is to able it to read the input sequence in both directions: from left to right (standard) and from right to left. This can be implemented easily by adding a **Bidirectional** wrapper over the LSTM/GRU definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<keras.layers.wrappers.Bidirectional at 0x7f94e1c38ef0>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras.layers import Bidirectional\n",
    "\n",
    "Bidirectional(LSTM(32, return_sequences=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This effectively creates two LSTM layers at the same level, one for each reading direction. The outputs of these two layers are concatenated and forwarded to the next layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    " <tr><td width=\"80\"><img src=\"img/question.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">\n",
    "Take the best network design you created so far and make the LSTM/GRU layer bidirectional. Do you obtain better test AUC now?\n",
    " </td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_6 (Embedding)      (None, 120, 64)           960000    \n",
      "_________________________________________________________________\n",
      "bidirectional_2 (Bidirection (None, 64)                18816     \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 6)                 390       \n",
      "_________________________________________________________________\n",
      "activation_6 (Activation)    (None, 6)                 0         \n",
      "=================================================================\n",
      "Total params: 979,206\n",
      "Trainable params: 979,206\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Train on 119678 samples, validate on 39893 samples\n",
      "Epoch 1/3\n",
      "119678/119678 [==============================] - 75s 628us/step - loss: 0.0659 - acc: 0.9790 - val_loss: 0.0497 - val_acc: 0.9823\n",
      "Epoch 2/3\n",
      "119678/119678 [==============================] - 75s 627us/step - loss: 0.0452 - acc: 0.9833 - val_loss: 0.0467 - val_acc: 0.9832\n",
      "Epoch 3/3\n",
      "119678/119678 [==============================] - 75s 624us/step - loss: 0.0389 - acc: 0.9850 - val_loss: 0.0484 - val_acc: 0.9829\n",
      "AUC score 0.974464947623828\n"
     ]
    }
   ],
   "source": [
    "####### INSERT YOUR CODE HERE\n",
    "from keras.layers import Bidirectional\n",
    "model = Sequential()\n",
    "model.add(Embedding(input_dim=maxwords, input_length=maxsequence, output_dim=64))\n",
    "model.add(Bidirectional(CuDNNGRU(32)))\n",
    "model.add(Dense(6))\n",
    "model.add(Activation('sigmoid'))\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[\"accuracy\"])\n",
    "model.summary()\n",
    "\n",
    "model.fit(X_train, y_train, batch_size=32, epochs=3, validation_data=(X_test, y_test))\n",
    "\n",
    "y_pred = model.predict(X_test)\n",
    "print(\"AUC score\", roc_auc_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stacked mixing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/stacked.png\" style=\"width:500px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much like other neural layers, LSTM/GRU layers can be stacked on top of each other to produce more complex models. To do this we should configure each layer to return sequences until the last LSTM/GRU layer in the network, which should either return only the last value in the transformed sequence or be followed by some summarizing layer (like GlobalMaxPool1D). For instance, the following would be a valid structure (note we are not adding layers to any model, this is just an example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<keras.layers.pooling.GlobalMaxPooling1D at 0x7f94430b1f98>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LSTM(32, return_sequences=True)\n",
    "LSTM(32, return_sequences=True)\n",
    "GlobalMaxPool1D()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note also that when using several LSTM/GRU layers, it is usually not useful to the Bidirectional wrapper in the layers after the first."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    " <tr><td width=\"80\"><img src=\"img/question.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">\n",
    "Repeat the training of the previous network, but using 2 LSTM/GRU layers. Make sure to configure the first of those layers in a way that it outputs a whole sequence for the next layer. Are you able to improve the results?\n",
    " </td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_7 (Embedding)      (None, 120, 64)           960000    \n",
      "_________________________________________________________________\n",
      "bidirectional_3 (Bidirection (None, 120, 64)           18816     \n",
      "_________________________________________________________________\n",
      "cu_dnngru_5 (CuDNNGRU)       (None, 32)                9408      \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 6)                 198       \n",
      "_________________________________________________________________\n",
      "activation_7 (Activation)    (None, 6)                 0         \n",
      "=================================================================\n",
      "Total params: 988,422\n",
      "Trainable params: 988,422\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Train on 119678 samples, validate on 39893 samples\n",
      "Epoch 1/3\n",
      "119678/119678 [==============================] - 109s 909us/step - loss: 0.0647 - acc: 0.9791 - val_loss: 0.0495 - val_acc: 0.9825\n",
      "Epoch 2/3\n",
      "119678/119678 [==============================] - 107s 898us/step - loss: 0.0446 - acc: 0.9833 - val_loss: 0.0463 - val_acc: 0.9832\n",
      "Epoch 3/3\n",
      "119678/119678 [==============================] - 105s 880us/step - loss: 0.0392 - acc: 0.9848 - val_loss: 0.0482 - val_acc: 0.9828\n",
      "AUC score 0.9748682934632549\n"
     ]
    }
   ],
   "source": [
    "####### INSERT YOUR CODE HERE\n",
    "model = Sequential()\n",
    "model.add(Embedding(input_dim=maxwords, input_length=maxsequence, output_dim=64))\n",
    "model.add(Bidirectional(CuDNNGRU(32, return_sequences=True)))\n",
    "model.add(CuDNNGRU(32))\n",
    "model.add(Dense(6))\n",
    "model.add(Activation('sigmoid'))\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[\"accuracy\"])\n",
    "model.summary()\n",
    "\n",
    "model.fit(X_train, y_train, batch_size=32, epochs=3, validation_data=(X_test, y_test))\n",
    "\n",
    "y_pred = model.predict(X_test)\n",
    "print(\"AUC score\", roc_auc_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine tuning the network desing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can mix the components seen so far to improve the network further. Fine tuning their parameters can give better results:\n",
    "\n",
    "* Embeddings: size of the embedding.\n",
    "* LSTM/GRU layers: size of the layer, number of layers.\n",
    "* Bidirectional wrapper.\n",
    "* Max Pooling or getting the last element of the LSTM/GRU layer. Alternatively, you can also make use of [GlobalAveragePooling1D](https://keras.io/layers/pooling/), as we did in the CBoW model.\n",
    "\n",
    "Dropout can also be used to prevent overfitting. Beyond the standard Dropout layers, other forms of Dropout have been found useful in text processing:\n",
    "\n",
    "* Dropout parameters inside LSTM/GRU layers: parameters **dropout** and **recurrent_dropout** of these layers allow introducing dropout inside the blocks that process the inputs and the recurrent states.\n",
    "* [SpatialDropout1D](https://keras.io/layers/core/): useful for performing dropout on the embeddings layer, removes entire channels (embeddings features) for all words. This makes more sense since values within the same channel are usually highly correlated across words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    " <tr><td width=\"80\"><img src=\"img/pro.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">\n",
    "Fine-tune the network design, experimenting with the layers and parameters listed above. How much AUC can you achieve?\n",
    " </td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            (None, 120)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_8 (Embedding)         (None, 120, 300)     4500000     input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "spatial_dropout1d_1 (SpatialDro (None, 120, 300)     0           embedding_8[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_4 (Bidirectional) (None, 120, 160)     183360      spatial_dropout1d_1[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "global_average_pooling1d_3 (Glo (None, 160)          0           bidirectional_4[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_4 (GlobalM (None, 160)          0           bidirectional_4[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 320)          0           global_average_pooling1d_3[0][0] \n",
      "                                                                 global_max_pooling1d_4[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dense_8 (Dense)                 (None, 6)            1926        concatenate_1[0][0]              \n",
      "==================================================================================================\n",
      "Total params: 4,685,286\n",
      "Trainable params: 4,685,286\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n",
      "Train on 119678 samples, validate on 39893 samples\n",
      "Epoch 1/2\n",
      "119678/119678 [==============================] - 123s 1ms/step - loss: 0.0589 - acc: 0.9802 - val_loss: 0.0469 - val_acc: 0.9832\n",
      "Epoch 2/2\n",
      "119678/119678 [==============================] - 122s 1ms/step - loss: 0.0415 - acc: 0.9842 - val_loss: 0.0449 - val_acc: 0.9836\n",
      "AUC score 0.9805490841311806\n"
     ]
    }
   ],
   "source": [
    "####### INSERT YOUR CODE HERE\n",
    "from keras.models import Model\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.layers import Bidirectional, Dense\n",
    "from keras.layers import GlobalMaxPool1D, GlobalAveragePooling1D, concatenate, CuDNNGRU, Input, SpatialDropout1D\n",
    "\n",
    "inp = Input(shape=(maxsequence, ))\n",
    "x = Embedding(input_dim=maxwords, input_length=maxsequence, output_dim=300)(inp)\n",
    "x = SpatialDropout1D(0.2)(x)\n",
    "x = Bidirectional(CuDNNGRU(80, return_sequences=True))(x)\n",
    "avg_pool = GlobalAveragePooling1D()(x)\n",
    "max_pool = GlobalMaxPool1D()(x)\n",
    "conc = concatenate([avg_pool, max_pool])\n",
    "outp = Dense(6, activation=\"sigmoid\")(conc)\n",
    "\n",
    "model = Model(inputs=inp, outputs=outp)\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[\"accuracy\"])\n",
    "model.summary()\n",
    "\n",
    "model.fit(X_train, y_train, batch_size=32, epochs=2, validation_data=(X_test, y_test))\n",
    "\n",
    "y_pred = model.predict(X_test)\n",
    "print(\"AUC score\", roc_auc_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transfer learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results so far can be further improved if we make use of a Language Model. This involves introducing in our model one or more pre-trained layers, that have been learned using a much larger, unsupervised dataset.\n",
    "\n",
    "To try this idea we will make use of the [FastText embeddings](https://fasttext.cc/docs/en/english-vectors.html) to initialize our Embedding layer. The following function loads from the FastText file a dictionary of embeddings for those words that have appeared in a given dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def load_vectors_words(fname, words):\n",
    "    \"\"\"Loads embeddings from a FastText file. Only loads embeddings for the given dictionary of words\"\"\"\n",
    "    data = {}\n",
    "    with open(fname) as fin:\n",
    "        next(fin)  # Skip first line, just contains embeddings size data\n",
    "        for line in fin:\n",
    "            tokens = line.rstrip().split(' ')\n",
    "            word = tokens[0]\n",
    "            if word in words:\n",
    "                data[word] = np.array(list(map(float, tokens[1:])))\n",
    "    return data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    " <tr><td width=\"80\"><img src=\"img/question.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">\n",
    "Download and unzip any of the embedding files for english available at the <a href=\"https://fasttext.cc/docs/en/english-vectors.html\">FastText website</a>. Note that larger embedding files will likely produce better results, but will also require more hard drive space. Use the provided function to load the dictionary of the embeddings for the tokenizer words. You can access the words used by the tokenizer through the <i>word_index</i> attribute. Store the computed embeddings dictionary into a variable named <b>embeddings</b>.\n",
    " </td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "####### INSERT YOUR CODE HERE\n",
    "FASTTEXT_FILE = \"/home/alvaro/datasets/embeddings/crawl-300d-2M.vec\"\n",
    "embeddings = load_vectors_words(FASTTEXT_FILE, tokenizer.word_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can query the loaded dictionary to check what is the embedding for a given word. For instance, the embedding for  the word \"stupid\" would be:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1.715e-01,  1.417e-01,  2.173e-01, -2.352e-01,  1.500e-03,\n",
       "       -3.144e-01, -7.780e-02,  1.297e-01,  9.020e-02, -2.155e-01,\n",
       "        3.902e-01,  1.627e-01,  2.430e-02, -7.740e-02,  2.720e-02,\n",
       "       -1.431e-01,  1.027e-01,  3.000e-04,  1.195e-01,  1.096e-01,\n",
       "       -1.551e-01,  2.808e-01,  1.381e-01, -6.120e-02, -3.300e-03,\n",
       "       -2.120e-01, -5.850e-02, -5.060e-02, -8.530e-02,  5.583e-01,\n",
       "        3.440e-02,  1.107e-01,  4.930e-02, -1.849e-01,  9.830e-02,\n",
       "        2.560e-02, -2.620e-02, -1.875e-01,  7.000e-03,  6.500e-03,\n",
       "       -2.430e-02,  1.189e-01, -6.420e-02,  9.930e-02,  3.929e-01,\n",
       "       -1.016e-01,  4.455e-01, -1.900e-01,  1.294e-01, -6.059e-01,\n",
       "        2.300e-02, -8.930e-02,  8.440e-02,  1.560e-02,  3.176e-01,\n",
       "       -1.407e-01, -6.540e-02,  1.387e-01,  5.400e-03, -1.995e-01,\n",
       "        7.160e-02,  2.506e-01, -8.500e-03, -5.110e-02,  2.054e-01,\n",
       "        9.810e-02,  2.717e-01, -9.970e-02, -2.478e-01, -1.761e-01,\n",
       "        2.060e-02, -6.340e-02, -8.680e-02,  3.817e-01, -1.176e-01,\n",
       "       -1.923e-01, -6.455e-01, -5.105e-01,  4.172e-01,  1.562e-01,\n",
       "        1.981e-01, -2.989e-01, -5.900e-02,  9.990e-02, -1.936e-01,\n",
       "        8.250e-02, -6.250e-02,  2.840e-02, -2.417e-01,  3.353e-01,\n",
       "       -1.760e-02, -3.660e-02, -1.500e-02, -1.387e-01, -1.569e-01,\n",
       "       -2.296e-01,  1.256e-01, -4.007e-01, -1.831e-01,  1.000e-03,\n",
       "       -2.223e-01,  7.000e-03, -7.240e-02, -7.670e-02,  8.600e-02,\n",
       "       -6.600e-03,  1.580e-02, -1.912e-01, -2.010e-02,  1.330e-02,\n",
       "       -1.263e-01,  2.080e-02,  1.400e-02,  6.270e-02, -6.061e-01,\n",
       "        2.975e-01, -1.011e-01,  2.970e-02, -1.310e-02, -2.685e-01,\n",
       "       -3.920e-02,  4.104e-01,  2.195e-01,  1.074e-01,  3.949e-01,\n",
       "       -1.777e-01,  1.838e-01,  6.000e-02,  2.220e-01,  2.110e-02,\n",
       "        2.616e-01,  1.719e-01,  1.742e-01, -8.270e-02, -2.016e-01,\n",
       "       -1.315e-01,  2.034e-01, -3.297e-01,  1.112e-01, -4.684e-01,\n",
       "       -9.890e-02, -4.147e-01,  1.166e-01,  3.843e-01, -2.957e-01,\n",
       "       -1.737e-01,  4.770e-02,  5.810e-02, -1.600e-03, -1.886e-01,\n",
       "       -1.110e-02,  1.990e-02, -1.149e-01, -7.450e-02,  1.343e-01,\n",
       "        1.233e-01,  1.248e-01, -3.256e-01,  9.140e-02, -1.480e-01,\n",
       "        1.039e-01, -2.689e-01, -2.870e-02,  1.748e-01,  9.370e-02,\n",
       "       -2.474e-01,  3.836e-01, -6.510e-02,  8.220e-02, -1.026e-01,\n",
       "       -2.222e-01,  2.889e-01,  2.830e-02, -1.109e-01, -8.430e-02,\n",
       "        3.000e-04, -3.202e-01,  1.955e-01,  1.060e-01, -1.158e-01,\n",
       "       -3.722e-01,  2.630e-02,  1.010e-01,  5.680e-02, -1.840e-02,\n",
       "        2.925e-01,  3.135e-01, -2.823e-01, -3.009e-01,  5.420e-02,\n",
       "       -2.260e-02,  1.751e-01, -9.100e-02,  9.000e-04,  1.168e-01,\n",
       "        2.822e-01,  3.120e-02,  2.880e-02, -7.573e-01,  3.590e-02,\n",
       "        1.586e-01, -1.259e-01,  8.640e-02,  6.160e-02, -1.873e-01,\n",
       "        7.150e-02,  5.220e-02, -5.180e-02,  7.180e-02, -2.866e-01,\n",
       "       -2.590e-02,  4.968e-01, -1.175e-01,  1.770e-02,  9.860e-02,\n",
       "       -7.260e-02,  1.216e-01,  3.178e-01,  6.950e-02,  2.193e-01,\n",
       "       -9.740e-02,  5.353e-01, -4.860e-02, -1.570e-01,  6.420e-02,\n",
       "        3.244e-01, -8.940e-02,  1.460e-01,  6.640e-02,  2.344e-01,\n",
       "        2.059e-01, -3.760e-02, -1.453e-01,  4.122e-01, -1.474e-01,\n",
       "       -6.977e-01, -1.300e-03, -7.200e-03,  1.675e-01,  1.718e-01,\n",
       "       -1.811e-01, -7.810e-02, -8.710e-02,  2.400e-01, -3.744e-01,\n",
       "        1.464e-01,  2.360e-01, -1.119e-01,  2.942e-01,  4.640e-02,\n",
       "        5.000e-02,  1.579e-01,  1.936e-01, -3.247e-01,  1.259e-01,\n",
       "        1.248e-01, -2.702e-01,  2.262e-01, -1.810e-02,  2.400e-02,\n",
       "        4.570e-02,  9.210e-02, -1.616e-01, -2.388e-01, -3.708e-01,\n",
       "        1.310e-02,  1.325e-01,  1.123e-01, -2.225e-01, -1.343e-01,\n",
       "        3.350e-02,  9.360e-02, -1.190e-01,  2.887e-01, -2.318e-01,\n",
       "        9.490e-02,  8.410e-02,  1.271e-01,  1.003e-01, -2.845e-01,\n",
       "        1.128e-01,  7.990e-02, -1.078e-01, -8.870e-02, -2.450e-02,\n",
       "       -3.917e-01, -4.880e-02,  8.650e-02,  1.271e-01,  1.849e-01,\n",
       "        8.800e-02,  1.152e-01, -1.380e-02,  5.610e-02, -2.151e-01,\n",
       "       -2.174e-01,  6.560e-02,  1.567e-01, -2.289e-01, -4.880e-02])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embeddings[\"stupid\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To insert these pre-trained embeddings into a Keras network, we need to create a matrix of weights that we can use to initialize the Embeddings layer. This should be done according to the indexes the Tokenizer has assigned to each word: the i-th row of the embedding matrix must correspond to the embedding learned for the i-th word in the Tokenizer. If some word found in our texts does not appear in the embeddings file we loaded, it is a good practice to initialize it following a random distribution following the mean and standard deviation of the rest of embeddings. The following function does all this work for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_embedding_matrix(embeddings, tokenizer):\n",
    "    \"\"\"Creates a weight matrix for an Embedding layer using an embeddings dictionary and a Tokenizer\"\"\"\n",
    "    \n",
    "    # Compute mean and standard deviation for embeddings\n",
    "    all_embs = np.stack(embeddings.values())\n",
    "    emb_mean,emb_std = all_embs.mean(), all_embs.std()\n",
    "    \n",
    "    embedding_size = len(next(iter(embeddings.values())))\n",
    "    \n",
    "    embedding_matrix = np.random.normal(emb_mean, emb_std, (tokenizer.num_words, embedding_size))\n",
    "    for word, i in tokenizer.word_index.items():\n",
    "        if i >= tokenizer.num_words: break\n",
    "        embedding_vector = embeddings.get(word)\n",
    "        if embedding_vector is not None: \n",
    "            embedding_matrix[i] = embedding_vector\n",
    "            \n",
    "    return embedding_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    " <tr><td width=\"80\"><img src=\"img/question.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">\n",
    "Create the embedding weights matrix using the function above, storing it in a variable named <b>embedding_matrix</b>\n",
    " </td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "####### INSERT YOUR CODE HERE\n",
    "embedding_matrix = create_embedding_matrix(embeddings, tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now check the matrix shape to ensure it has been build correctly. It must have a number of rows equal to the number of words in our tokenizer (15000), and a number of columns equal to the size of the FastText embedding (300)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(15000, 300)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_matrix.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this matrix we can create our transfer learning network! It is designed the same way as previous networks, the only difference being when creating the Embedding layer. To initialize this layer with the pre-trained embeddings, use the **embedding_matrix** you created as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "pretrained = Embedding(maxwords, embedding_matrix.shape[1], weights=[embedding_matrix], trainable=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This creates an Embedding layer with the pre-trained FastText embeddings, provided as the *weights* parameter. Note also the pameter `trainable=False`: this tells Keras that it should not change the weights of this layer. This avoids a common problem in deep learning known as [catastrophic forgetting](https://en.wikipedia.org/wiki/Catastrophic_interference), where adjusting the network for our specific dataset might make it forget about all the pre-training. In some settings, however, it can be useful to fine-tune the weights of a pre-trained layer for our specific problem, though great care must be taken not to learn too much."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    " <tr><td width=\"80\"><img src=\"img/question.png\" style=\"width:auto;height:auto\"></td><td style=\"text-align:left\">\n",
    "Create a network using a pretrained Embedding layer and what you learned from previous models. Has pretraining helped in improving AUC?\n",
    " </td></tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_2 (InputLayer)            (None, 120)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_10 (Embedding)        (None, 120, 300)     4500000     input_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "spatial_dropout1d_2 (SpatialDro (None, 120, 300)     0           embedding_10[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_5 (Bidirectional) (None, 120, 160)     183360      spatial_dropout1d_2[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "global_average_pooling1d_4 (Glo (None, 160)          0           bidirectional_5[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_5 (GlobalM (None, 160)          0           bidirectional_5[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_2 (Concatenate)     (None, 320)          0           global_average_pooling1d_4[0][0] \n",
      "                                                                 global_max_pooling1d_5[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dense_9 (Dense)                 (None, 6)            1926        concatenate_2[0][0]              \n",
      "==================================================================================================\n",
      "Total params: 4,685,286\n",
      "Trainable params: 185,286\n",
      "Non-trainable params: 4,500,000\n",
      "__________________________________________________________________________________________________\n",
      "Train on 119678 samples, validate on 39893 samples\n",
      "Epoch 1/2\n",
      "119678/119678 [==============================] - 89s 747us/step - loss: 0.0552 - acc: 0.9804 - val_loss: 0.0464 - val_acc: 0.9834\n",
      "Epoch 2/2\n",
      "119678/119678 [==============================] - 88s 736us/step - loss: 0.0444 - acc: 0.9834 - val_loss: 0.0426 - val_acc: 0.9842\n",
      "AUC score 0.9850270956227395\n"
     ]
    }
   ],
   "source": [
    "####### INSERT YOUR CODE HERE\n",
    "from keras.models import Model\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.layers import Bidirectional, Dense\n",
    "from keras.layers import GlobalMaxPool1D, GlobalAveragePooling1D, concatenate, CuDNNGRU, Input, SpatialDropout1D\n",
    "\n",
    "inp = Input(shape=(maxsequence, ))\n",
    "x = Embedding(maxwords, embedding_matrix.shape[1], weights=[embedding_matrix], trainable=False)(inp)\n",
    "x = SpatialDropout1D(0.2)(x)\n",
    "x = Bidirectional(CuDNNGRU(80, return_sequences=True))(x)\n",
    "avg_pool = GlobalAveragePooling1D()(x)\n",
    "max_pool = GlobalMaxPool1D()(x)\n",
    "conc = concatenate([avg_pool, max_pool])\n",
    "outp = Dense(6, activation=\"sigmoid\")(conc)\n",
    "\n",
    "model = Model(inputs=inp, outputs=outp)\n",
    "\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[\"accuracy\"])\n",
    "model.summary()\n",
    "\n",
    "model.fit(X_train, y_train, batch_size=32, epochs=2, validation_data=(X_test, y_test))\n",
    "\n",
    "y_pred = model.predict(X_test)\n",
    "print(\"AUC score\", roc_auc_score(y_test, y_pred))\n",
    "\n",
    "# no trainable: 0.984186395761243 AUC\n",
    "# trainable:    0.983964730975434 AUC"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}