CNN_Text_Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional Neural Networks\n",
    "---\n",
    "In this notebook, I'll train a **CNN** to classify the sentiment of movie reviews in a corpus of text. The approach will be as follows:\n",
    "* Pre-process movie reviews and their corresponding sentiment labels (positive = 1, negative = 0).\n",
    "* Load in a **pre-trained** Word2Vec model, and use it to tokenize the reviews.\n",
    "* Create training/validation/test sets of data.\n",
    "* Define a `SentimentCNN` model that has a pre-trained embedding layer, convolutional layers, and a final, fully-connected, classification layer.\n",
    "* Train and evaluate the model.\n",
    "\n",
    "An example of a positive and negative review are shown below.\n",
    "\n",
    "<img src='notebook_ims/reviews_ex.png' width=30% height=70% />\n",
    "\n",
    "The task of text classification has typically been done with an RNN, which accepts a sequence of words as input and has a hidden state that is dependent on that sequence and acts as a kind of memory. You can see an example that classifies this same review dataset using an RNN in [this Github repository](https://github.com/udacity/deep-learning-v2-pytorch/blob/master/sentiment-rnn/Sentiment_RNN_Solution.ipynb). \n",
    "\n",
    "\n",
    "## Resources\n",
    "\n",
    "This example shows how you can utilize convolutional layers to find patterns in sequences of word embeddings and create an effective text classifier using a CNN-based approach.\n",
    "\n",
    "**1. Original paper**\n",
    "* The code follows the structure outlined in the paper, [Convolutional Neural Networks for Sentence Classification](https://arxiv.org/abs/1408.5882) by Yoon Kim (2014). \n",
    "\n",
    "**2. Pre-trained Word2Vec model**\n",
    "\n",
    "* The key to this approach is convolving over word embeddings, for which I will use a pre-trained [Word2Vec](https://en.wikipedia.org/wiki/Word2vec) model. \n",
    "* I am specifically using a \"slim\"-version of a model that was trained on part of a Google News dataset (about 100 billion words). The [original model](https://code.google.com/archive/p/word2vec/) contains 300-dimensional vectors for 3 million words and phrases.\n",
    "* The \"slim\" model is cut to 300k English words, as described in [this Github repository](https://github.com/eyaler/word2vec-slim).\n",
    "\n",
    "You should be able to modify this code slightly to make it compatible with a Word2Vec model of your choosing.\n",
    "\n",
    "**3. Movie reviews data **\n",
    "\n",
    "The dataset holds 25000 movie reviews, which were obtained from the movie review site, IMDb.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Load in and Visualize the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# read data from text files\n",
    "with open('data/reviews.txt', 'r') as f:\n",
    "    reviews = f.read()\n",
    "with open('data/labels.txt', 'r') as f:\n",
    "    labels = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bromwell high is a cartoon comedy . it ran at the same time as some other programs about school life  such as  teachers  . my   years in the teaching profession lead me to believe that bromwell high  s satire is much closer to reality than is  teachers  . the scramble to survive financially  the insightful students who can see right through their pathetic teachers  pomp  the pettiness of the whole situation  all remind me of the schools i knew and their students . when i saw the episode in which a student repeatedly tried to burn down the school  i immediately recalled . . . . . . . . . at . . . . . . . . . . high . a classic line inspector i  m here to sack one of your teachers . student welcome to bromwell high . i expect that many adults of my age think that bromwell high is far fetched . what a pity that it isn  t   \n",
      "story of a man who has unnatural feelings for a pig . starts out with a opening scene that is a terrific example of absurd comedy . a formal orchestra audience is turn\n",
      "\n",
      "positive\n",
      "negative\n",
      "po\n"
     ]
    }
   ],
   "source": [
    "# print some example review/sentiment text\n",
    "print(reviews[:1000])\n",
    "print()\n",
    "print(labels[:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Data Pre-processing\n",
    "\n",
    "The first step, when building a neural network, is getting the data into the proper form to feed into the network. Since I'm planning to use a word-embedding layer, I know that I'll need to encode each word in a reviews as an integer, and encode each sentiment label as 1 (positive) or 0 (negative). \n",
    "\n",
    "I'll first want to clean up the reviews by removing punctuation and converting them to lowercase. You can see an example of the reviews data, above. Here are the processing steps, I'll want to take:\n",
    ">* Get rid of any extraneous punctuation.\n",
    "* You might notice that the reviews are delimited with newline characters `\\n`. To deal with those, I'm going to split the text into each review using `\\n` as the delimiter. \n",
    "* Then I can combined all the reviews back together into one big string to get all of my text data.\n",
    "\n",
    "First, let's remove all punctuation. Then get all the text without the newlines and split it into individual words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from string import punctuation\n",
    "\n",
    "# get rid of punctuation\n",
    "reviews = reviews.lower() # lowercase, standardize\n",
    "all_text = ''.join([c for c in reviews if c not in punctuation])\n",
    "\n",
    "# split by new lines and spaces\n",
    "reviews_split = all_text.split('\\n')\n",
    "\n",
    "all_text = ' '.join(reviews_split)\n",
    "\n",
    "# create a list of all words\n",
    "all_words = all_text.split()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding the Labels\n",
    "\n",
    "The review labels are \"positive\" or \"negative\". To use these labels in a neural network, I need to convert them to numerical values, 1 (positive) and 0 (negative)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1=positive, 0=negative label conversion\n",
    "labels_split = labels.split('\\n')\n",
    "encoded_labels = np.array([1 if label == 'positive' else 0 for label in labels_split])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing Outliers\n",
    "\n",
    "As an additional pre-processing step, I want to make sure that the reviews are in good shape for standard processing. That is, I'll want to shape the reviews into a specific, consistent length for ease of processing and comparison. I'll approach this task in two main steps:\n",
    "\n",
    "1. Getting rid of extremely long or short reviews; the outliers\n",
    "2. Padding/truncating the remaining data so that we have reviews of the same length.\n",
    "\n",
    "Before I pad the review text, below, I am checking for reviews of extremely short or long lengths; outliers that may mess with training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Zero-length reviews: 1\n",
      "Maximum review length: 2514\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "# Build a dictionary that maps indices to review lengths\n",
    "counts = Counter(all_words)\n",
    "\n",
    "# outlier review stats\n",
    "# counting words in each review\n",
    "review_lens = Counter([len(x.split()) for x in reviews_split])\n",
    "print(\"Zero-length reviews: {}\".format(review_lens[0]))\n",
    "print(\"Maximum review length: {}\".format(max(review_lens)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, a couple issues here. I seem to have one review with zero length. And, the maximum review length is really long. I'm going to remove any super short reviews and truncate super long reviews. This removes outliers and should allow our model to train more efficiently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of reviews before removing outliers:  25001\n",
      "Number of reviews after removing outliers:  25000\n"
     ]
    }
   ],
   "source": [
    "print('Number of reviews before removing outliers: ', len(reviews_split))\n",
    "\n",
    "## remove any reviews/labels with zero length from the reviews_ints list.\n",
    "\n",
    "# get indices of any reviews with length 0\n",
    "non_zero_idx = [ii for ii, review in enumerate(reviews_split) if len(review.split()) != 0]\n",
    "\n",
    "# remove 0-length reviews and their labels\n",
    "reviews_split = [reviews_split[ii] for ii in non_zero_idx]\n",
    "encoded_labels = np.array([encoded_labels[ii] for ii in non_zero_idx])\n",
    "\n",
    "print('Number of reviews after removing outliers: ', len(reviews_split))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Using a Pre-Trained Embedding Layer\n",
    "\n",
    "Next, I'll want to tokenize my reviews; turning the list of words that make up a given review into a list of tokenized integers that represent those words. Typically, this is done by creating a dictionary that maps each unique word in a vocabulary to a specific integer value.\n",
    "\n",
    "In this example, I'll actually want to use a mapping that already exists, in a pre-trained embedding layer. Below, I am loading in a pre-trained embedding model, and I'll explore its traits.\n",
    "\n",
    "> This code assumes I have a downloaded model `GoogleNews-vectors-negative300-SLIM.bin.gz` in the same directory as this notebook, in a folder, `word2vec_model`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Load a pretrained word2vec model (only need to run code, once)\n",
    "# ! gzip -d word2vec_model/GoogleNews-vectors-negative300-SLIM.bin.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import Word2Vec loading capabilities\n",
    "from gensim.models import KeyedVectors\n",
    "\n",
    "# Creating the model\n",
    "embed_lookup = KeyedVectors.load_word2vec_format('word2vec_model/GoogleNews-vectors-negative300-SLIM.bin', \n",
    "                                                 binary=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Embedding Layer\n",
    "\n",
    "You can think of an embedding layer as a lookup table, where the rows are indexed by word token and the columns hold the embedding values. For example, row 958 is the embedding vector for the word that maps to the integer value 958.\n",
    "\n",
    "<img src='notebook_ims/embedding_lookup_table.png' width=40% />\n",
    "\n",
    "In the below cells, I am storing the words in the pre-trained vocabulary, and printing out the size of the vocabulary and word embeddings. \n",
    "> The embedding dimension from the pret-rained model is 300."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# store pretrained vocab\n",
    "pretrained_words = []\n",
    "for word in embed_lookup.vocab:\n",
    "    pretrained_words.append(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of Vocab: 299567\n",
      "\n",
      "Word in vocab: for\n",
      "\n",
      "Length of embedding: 300\n",
      "\n"
     ]
    }
   ],
   "source": [
    "row_idx = 1\n",
    "\n",
    "# get word/embedding in that row\n",
    "word = pretrained_words[row_idx] # get words by index\n",
    "embedding = embed_lookup[word] # embeddings by word\n",
    "\n",
    "# vocab and embedding info\n",
    "print(\"Size of Vocab: {}\\n\".format(len(pretrained_words)))\n",
    "print('Word in vocab: {}\\n'.format(word))\n",
    "print('Length of embedding: {}\\n'.format(len(embedding)))\n",
    "#print('Associated embedding: \\n', embedding)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "in\n",
      "for\n",
      "that\n",
      "is\n",
      "on\n"
     ]
    }
   ],
   "source": [
    "# print a few common words\n",
    "for i in range(5):\n",
    "    print(pretrained_words[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cosine Similarity\n",
    "\n",
    "The pre-trained embedding model has learned to represent semantic relationships between words in vector space. Specifically, words that appear in similar contexts should point in roughly the same direction. To measure whether two vectors are colinear, we can use [**cosine similarity**](https://en.wikipedia.org/wiki/Cosine_similarity), which computes the dot product of two vectors. This dot product is largest when the angle between two vectors is 0 (cos(0) = 1) and cosine is at a maximum, so cosine similarity is larger for aligned vectors.\n",
    "\n",
    "<img src='notebook_ims/two_vectors.png' width=30% />\n",
    "\n",
    "### Embedded Bias\n",
    "\n",
    "Word2Vec, in addition to learning useful similarities and semantic relationships between words, also learns to represent problematic relationships between words. For example, a paper on [Debiasing Word Embeddings](https://papers.nips.cc/paper/6228-man-is-to-computer-programmer-as-woman-is-to-homemaker-debiasing-word-embeddings.pdf) by Bolukbasi et al. (2016), found that the vector-relationship between \"man\" and \"woman\" was similar to the relationship between \"physician\" and \"registered nurse\" or \"shopkeeper\" and \"housewife\" in the trained, Google News Word2Vec model, **which I am using in this notebook**.\n",
    "\n",
    ">*\"In this paper, we quantitatively demonstrate that word-embeddings contain biases in their geometry that reflect gender stereotypes present in broader society. Due to their wide-spread usage as basic\n",
    "features, word embeddings not only reflect such stereotypes but can also amplify them. This poses a\n",
    "significant risk and challenge for machine learning and its applications.\"*\n",
    "\n",
    "As such, it is important to note that this example is using a Word2Vec model that has been shown to encapsulate gender stereotypes.\n",
    "\n",
    "You can explore similarities and relationships between word embeddings using code. The code below finds words with the highest cosine similarity when compared to the word `find_similar_to`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similar words to fabulous: \n",
      "\n",
      "Word: wonderful, Similarity: 0.761\n",
      "Word: fantastic, Similarity: 0.761\n",
      "Word: marvelous, Similarity: 0.730\n",
      "Word: gorgeous, Similarity: 0.714\n",
      "Word: lovely, Similarity: 0.713\n",
      "Word: terrific, Similarity: 0.694\n",
      "Word: amazing, Similarity: 0.693\n",
      "Word: beautiful, Similarity: 0.670\n",
      "Word: magnificent, Similarity: 0.667\n",
      "Word: splendid, Similarity: 0.645\n"
     ]
    }
   ],
   "source": [
    "# Pick a word \n",
    "find_similar_to = 'fabulous'\n",
    "\n",
    "print('Similar words to '+find_similar_to+': \\n')\n",
    "\n",
    "# Find similar words, using cosine similarity\n",
    "# by default shows top 10 similar words\n",
    "for similar_word in embed_lookup.similar_by_word(find_similar_to):\n",
    "    print(\"Word: {0}, Similarity: {1:.3f}\".format(\n",
    "        similar_word[0], similar_word[1]\n",
    "    ))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenize reviews\n",
    "\n",
    "The pre-trained embedding layer already has tokens associated with each word in the dictionary. I want to use that same mapping to tokenize all the reviews in the movie review corpus. I will encode any unknown words (words that appear in the reviews but not in the pre-trained vocabulary) as the whitespace token, 0; this should be fine for the purpose of sentiment classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert reviews to tokens\n",
    "def tokenize_all_reviews(embed_lookup, reviews_split):\n",
    "    # split each review into a list of words\n",
    "    reviews_words = [review.split() for review in reviews_split]\n",
    "\n",
    "    tokenized_reviews = []\n",
    "    for review in reviews_words:\n",
    "        ints = []\n",
    "        for word in review:\n",
    "            try:\n",
    "                idx = embed_lookup.vocab[word].index\n",
    "            except: \n",
    "                idx = 0\n",
    "            ints.append(idx)\n",
    "        tokenized_reviews.append(ints)\n",
    "    \n",
    "    return tokenized_reviews\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_reviews = tokenize_all_reviews(embed_lookup, reviews_split)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 137, 3, 0, 11620, 3799, 13, 1215, 10, 9, 194, 54, 12, 73, 61, 685, 41, 183, 243, 129, 12, 1663, 119, 72, 0, 9, 2989, 7334, 242, 159, 0, 453, 2, 0, 137, 1239, 19951, 3, 141, 1980, 0, 1898, 55, 3, 1663, 9, 11124, 0, 3857, 6663, 9, 20401, 295, 28, 45, 148, 157, 102, 27, 15452, 1663, 30714, 9, 65172, 0, 9, 844, 737, 47, 6585, 159, 0, 9, 668, 4365, 1003, 0, 27, 295, 56, 4365, 622, 9, 3832, 0, 43, 0, 897, 3187, 907, 0, 5396, 113, 9, 183, 4365, 1009, 3165, 10, 137, 0, 3288, 296, 10314, 4365, 6638, 213, 0, 8810, 40, 0, 116, 1663, 897, 2059, 0, 0, 137, 4365, 830, 2, 124, 2216, 0, 119, 782, 144, 2, 0, 137, 3, 330, 23046, 78, 0, 16915, 2, 13, 85275, 7451]\n"
     ]
    }
   ],
   "source": [
    "# testing code and printing a tokenized review\n",
    "print(tokenized_reviews[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Padding sequences\n",
    "\n",
    "To deal with both short and very long reviews, I'll pad or truncate all the reviews to a specific length. For reviews shorter than some `seq_length`, I'll left-pad with 0s. For reviews longer than `seq_length`, I'll truncate them to the first `seq_length` words. A good `seq_length`, in this case, is about 200.\n",
    "\n",
    "> The function `pad_features` returns an array that contains padded, tokenized reviews, of a standard size, that we'll pass to the network. \n",
    "\n",
    "\n",
    "As a small example, if the `seq_length=10` and an input, tokenized review is: \n",
    "```\n",
    "[117, 18, 128]\n",
    "```\n",
    "The resultant, padded sequence should be: \n",
    "\n",
    "```\n",
    "[0, 0, 0, 0, 0, 0, 0, 117, 18, 128]\n",
    "```\n",
    "\n",
    "**Your final `features` array should be a 2D array, with as many rows as there are reviews, and as many columns as the specified `seq_length`.**\n",
    "\n",
    "This isn't trivial and there are a bunch of ways to do this. But, if you're going to be building your own deep learning networks, you're going to have to get used to preparing your data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pad_features(tokenized_reviews, seq_length):\n",
    "    ''' Return features of tokenized_reviews, where each review is padded with 0's \n",
    "        or truncated to the input seq_length.\n",
    "    '''\n",
    "    \n",
    "    # getting the correct rows x cols shape\n",
    "    features = np.zeros((len(tokenized_reviews), seq_length), dtype=int)\n",
    "\n",
    "    # for each review, I grab that review and \n",
    "    for i, row in enumerate(tokenized_reviews):\n",
    "        features[i, -len(row):] = np.array(row)[:seq_length]\n",
    "    \n",
    "    return features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[     0      0      0      0      0      0      0      0]\n",
      " [     0      0      0      0      0      0      0      0]\n",
      " [ 16483     26      0     12 106210      0   1698     22]\n",
      " [  1935   1326     12      0   1403     60   3921   2019]\n",
      " [     0      0      0      0      0      0      0      0]\n",
      " [     0      0      0      0      0      0      0      0]\n",
      " [     0      0      0      0      0      0      0      0]\n",
      " [     0      0      0      0      0      0      0      0]\n",
      " [     0      0      0      0      0      0      0      0]\n",
      " [    56   4365      8    270    119    756    247    159]\n",
      " [     0      0      0      0      0      0      0      0]\n",
      " [     0      0      0      0      0      0      0      0]\n",
      " [     0      0      0      0      0      0      0      0]\n",
      " [     9    104   1428     16      0     60  65033   9622]\n",
      " [     0     25  13619  11902   7445  10397    179      4]\n",
      " [     0      0      0      0      0      0      0      0]\n",
      " [     9    208  18994  66850 121241 212263      0  87397]\n",
      " [     0      0      0      0      0      0      0      0]\n",
      " [    38    165  66850 121241  13241  25231     88      3]\n",
      " [     9    661      3    675     67      3     81     61]]\n"
     ]
    }
   ],
   "source": [
    "# Test your implementation!\n",
    "\n",
    "seq_length = 200\n",
    "\n",
    "features = pad_features(tokenized_reviews, seq_length=seq_length)\n",
    "\n",
    "## test statements - do not change - ##\n",
    "assert len(features)==len(tokenized_reviews), \"Features should have as many rows as reviews.\"\n",
    "assert len(features[0])==seq_length, \"Each feature row should contain seq_length values.\"\n",
    "\n",
    "# print first 8 values of the first 20 batches \n",
    "print(features[:20,:8])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Training, Validation, and Test Data\n",
    "\n",
    "With the data in nice shape, I'll split it into training, validation, and test sets.\n",
    "\n",
    "In the below code, I am creating features (x) and labels (y). \n",
    "* The split fraction, `split_frac` defines the fraction of data to **keep** in the training set. Usually this is set to 0.8 or 0.9. \n",
    "* Whatever data is left is split in half to create the validation and test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t\t\tFeature Shapes:\n",
      "Train set: \t\t(20000, 200) \n",
      "Validation set: \t(2500, 200) \n",
      "Test set: \t\t(2500, 200)\n"
     ]
    }
   ],
   "source": [
    "split_frac = 0.8\n",
    "\n",
    "## split data into training, validation, and test data (features and labels, x and y)\n",
    "\n",
    "split_idx = int(len(features)*split_frac)\n",
    "train_x, remaining_x = features[:split_idx], features[split_idx:]\n",
    "train_y, remaining_y = encoded_labels[:split_idx], encoded_labels[split_idx:]\n",
    "\n",
    "test_idx = int(len(remaining_x)*0.5)\n",
    "val_x, test_x = remaining_x[:test_idx], remaining_x[test_idx:]\n",
    "val_y, test_y = remaining_y[:test_idx], remaining_y[test_idx:]\n",
    "\n",
    "## print out the shapes of your resultant feature data\n",
    "print(\"\\t\\t\\tFeature Shapes:\")\n",
    "print(\"Train set: \\t\\t{}\".format(train_x.shape), \n",
    "      \"\\nValidation set: \\t{}\".format(val_x.shape),\n",
    "      \"\\nTest set: \\t\\t{}\".format(test_x.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Check your work**\n",
    "\n",
    "With train, validation, and test fractions equal to 0.8, 0.1, 0.1, respectively, the final, feature data shapes should look like:\n",
    "```\n",
    "                    Feature Shapes:\n",
    "Train set: \t\t (20000, 200) \n",
    "Validation set: \t(2500, 200) \n",
    "Test set: \t\t  (2500, 200)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DataLoaders and Batching\n",
    "\n",
    "After creating training, test, and validation data, I can create DataLoaders for this data by following two steps:\n",
    "1. Create a known format for accessing our data, using [TensorDataset](https://pytorch.org/docs/stable/data.html#) which takes in an input set of data and a target set of data with the same first dimension, and creates a dataset.\n",
    "2. Create DataLoaders and batch our training, validation, and test Tensor datasets.\n",
    "\n",
    "```\n",
    "train_data = TensorDataset(torch.from_numpy(train_x), torch.from_numpy(train_y))\n",
    "train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size)\n",
    "```\n",
    "\n",
    "This is an alternative to creating a generator function for batching our data into full batches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import TensorDataset, DataLoader\n",
    "\n",
    "# create Tensor datasets\n",
    "train_data = TensorDataset(torch.from_numpy(train_x), torch.from_numpy(train_y))\n",
    "valid_data = TensorDataset(torch.from_numpy(val_x), torch.from_numpy(val_y))\n",
    "test_data = TensorDataset(torch.from_numpy(test_x), torch.from_numpy(test_y))\n",
    "\n",
    "# dataloaders\n",
    "batch_size = 50\n",
    "\n",
    "# shuffling and batching data\n",
    "train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size)\n",
    "valid_loader = DataLoader(valid_data, shuffle=True, batch_size=batch_size)\n",
    "test_loader = DataLoader(test_data, shuffle=True, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Sentiment Network with PyTorch\n",
    "\n",
    "The complete model is made of a few layers:\n",
    "\n",
    "**1. An [embedding layer](https://pytorch.org/docs/stable/nn.html#embedding)**\n",
    "* This converts our word tokens (integers) into embedded vectors of a specific size.\n",
    "* In this case, the vectors/weights of this layer will come from a **pretrained** lookup table. \n",
    "\n",
    "**2. A few [convolutional layers](https://pytorch.org/docs/stable/nn.html#conv1d)**\n",
    "* These are defined by an input size, number of filters/feature maps to output, and a kernel size.\n",
    "* The output of these layers will go through a ReLu activation function and pooling layer in the `forward` function.\n",
    "\n",
    "**3. A fully-connected, output layer**\n",
    "* This maps the convolutional layer outputs to a desired output_size (1 sentiment class).\n",
    "\n",
    "**4. A sigmoid activation layer**\n",
    "* This turns the output logit into a value 0-1; a class score.\n",
    "\n",
    "There is also a dropout layer, which will prevent overfitting, placed between the convolutional outputs and the final, fully-connected layer.\n",
    "\n",
    "<img src=\"notebook_ims/complete_embedding_CNN.png\" width=60%>\n",
    "\n",
    "*Image from the original paper, [Convolutional Neural Networks for Sentence Classification](https://arxiv.org/pdf/1408.5882.pdf).*\n",
    "\n",
    "### The Embedding Layer\n",
    "\n",
    "The embedding layer comes from our pre-trained `embed_lookup` model. By default, the weights of this layer are set to the vectors from the pre-trained model and frozen, so it will just be used as a lookup table. You could train your own embedding layer here, but it will speed up the training process to use a pre-trained model.\n",
    "\n",
    "### The Convolutional Layer(s)\n",
    "\n",
    "I am creating three convolutional layers, which will have kernel_sizes of (3, 300), (4, 300), and (5, 300); to look at 3-, 4-, and 5- sequences of word embeddings at a time. Each of these three layers will produce  100 filtered outputs. This is following the layer conventions in the paper, [CNNs for Sentence Classification](https://arxiv.org/abs/1408.5882).\n",
    "\n",
    "> The kernels only move in one dimension: down a sequence of word embeddings. In other words, these kernels move along a sequence of words, in time!\n",
    "\n",
    "### Maxpooling Layers\n",
    "\n",
    "In the `forward` function, I am applying a ReLu activation to the outputs of all convolutional layers and a maxpooling layer over the input sequence dimension. The maxpooling layer will get us an indication of whether some high-level text feature was found. \n",
    "\n",
    "> After moving through 3 convolutional layers with 100 filtered outputs each, these layers will output 300 values that can be sent to a final, fully-connected, classification layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No GPU available, training on CPU.\n"
     ]
    }
   ],
   "source": [
    "# First checking if GPU is available\n",
    "train_on_gpu=torch.cuda.is_available()\n",
    "\n",
    "if(train_on_gpu):\n",
    "    print('Training on GPU.')\n",
    "else:\n",
    "    print('No GPU available, training on CPU.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class SentimentCNN(nn.Module):\n",
    "    \"\"\"\n",
    "    The embedding layer + CNN model that will be used to perform sentiment analysis.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, embed_model, vocab_size, output_size, embedding_dim,\n",
    "                 num_filters=100, kernel_sizes=[3, 4, 5], freeze_embeddings=True, drop_prob=0.5):\n",
    "        \"\"\"\n",
    "        Initialize the model by setting up the layers.\n",
    "        \"\"\"\n",
    "        super(SentimentCNN, self).__init__()\n",
    "\n",
    "        # set class vars\n",
    "        self.num_filters = num_filters\n",
    "        self.embedding_dim = embedding_dim\n",
    "        \n",
    "        # 1. embedding layer\n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim)\n",
    "        # set weights to pre-trained\n",
    "        self.embedding.weight = nn.Parameter(torch.from_numpy(embed_model.vectors)) # all vectors\n",
    "        # (optional) freeze embedding weights\n",
    "        if freeze_embeddings:\n",
    "            self.embedding.requires_grad = False\n",
    "        \n",
    "        # 2. convolutional layers\n",
    "        self.convs_1d = nn.ModuleList([\n",
    "            nn.Conv2d(1, num_filters, (k, embedding_dim), padding=(k-2,0)) \n",
    "            for k in kernel_sizes])\n",
    "        \n",
    "        # 3. final, fully-connected layer for classification\n",
    "        self.fc = nn.Linear(len(kernel_sizes) * num_filters, output_size) \n",
    "        \n",
    "        # 4. dropout and sigmoid layers\n",
    "        self.dropout = nn.Dropout(drop_prob)\n",
    "        self.sig = nn.Sigmoid()\n",
    "        \n",
    "    \n",
    "    def conv_and_pool(self, x, conv):\n",
    "        \"\"\"\n",
    "        Convolutional + max pooling layer\n",
    "        \"\"\"\n",
    "        # squeeze last dim to get size: (batch_size, num_filters, conv_seq_length)\n",
    "        # conv_seq_length will be ~ 200\n",
    "        x = F.relu(conv(x)).squeeze(3)\n",
    "        \n",
    "        # 1D pool over conv_seq_length\n",
    "        # squeeze to get size: (batch_size, num_filters)\n",
    "        x_max = F.max_pool1d(x, x.size(2)).squeeze(2)\n",
    "        return x_max\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        Defines how a batch of inputs, x, passes through the model layers.\n",
    "        Returns a single, sigmoid-activated class score as output.\n",
    "        \"\"\"\n",
    "        # embedded vectors\n",
    "        embeds = self.embedding(x) # (batch_size, seq_length, embedding_dim)\n",
    "        # embeds.unsqueeze(1) creates a channel dimension that conv layers expect\n",
    "        embeds = embeds.unsqueeze(1)\n",
    "        \n",
    "        # get output of each conv-pool layer\n",
    "        conv_results = [self.conv_and_pool(embeds, conv) for conv in self.convs_1d]\n",
    "        \n",
    "        # concatenate results and add dropout\n",
    "        x = torch.cat(conv_results, 1)\n",
    "        x = self.dropout(x)\n",
    "        \n",
    "        # final logit\n",
    "        logit = self.fc(x) \n",
    "        \n",
    "        # sigmoid-activated --> a class score\n",
    "        return self.sig(logit)\n",
    "      "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instantiate the network\n",
    "\n",
    "Here, I'll instantiate the network. First up, defining the hyperparameters.\n",
    "\n",
    "* `vocab_size`: Size of our vocabulary or the range of values for our input, word tokens.\n",
    "* `output_size`: Size of our desired output; the number of class scores we want to output (pos/neg).\n",
    "* `embedding_dim`: Number of columns in the embedding lookup table; size of our embeddings.\n",
    "* `num_filters`: Number of filters that each convolutional layer produces as output.\n",
    "* `filter_sizes`: A list of kernel sizes; one convolutional layer will be created for each kernel size.\n",
    "\n",
    "Any parameters I did not list, are left as the default value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SentimentCNN(\n",
      "  (embedding): Embedding(299567, 300)\n",
      "  (convs_1d): ModuleList(\n",
      "    (0): Conv2d(1, 100, kernel_size=(3, 300), stride=(1, 1), padding=(1, 0))\n",
      "    (1): Conv2d(1, 100, kernel_size=(4, 300), stride=(1, 1), padding=(2, 0))\n",
      "    (2): Conv2d(1, 100, kernel_size=(5, 300), stride=(1, 1), padding=(3, 0))\n",
      "  )\n",
      "  (fc): Linear(in_features=300, out_features=1, bias=True)\n",
      "  (dropout): Dropout(p=0.5)\n",
      "  (sig): Sigmoid()\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Instantiate the model w/ hyperparams\n",
    "\n",
    "vocab_size = len(pretrained_words)\n",
    "output_size = 1 # binary class (1 or 0)\n",
    "embedding_dim = len(embed_lookup[pretrained_words[0]]) # 300-dim vectors\n",
    "num_filters = 100\n",
    "kernel_sizes = [3, 4, 5]\n",
    "\n",
    "net = SentimentCNN(embed_lookup, vocab_size, output_size, embedding_dim,\n",
    "                   num_filters, kernel_sizes)\n",
    "\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Training\n",
    "\n",
    "Below is some training code, which iterates over all of the training data, records some loss statistics and performs backpropagation + optimization steps to update the weights of this network.\n",
    "\n",
    ">I'll also be using a binary cross entropy loss, which is designed to work with a single Sigmoid output. [BCELoss](https://pytorch.org/docs/stable/nn.html#bceloss), or **Binary Cross Entropy Loss**, applies cross entropy loss to a single value between 0 and 1.\n",
    "\n",
    "I also have some training hyperparameters:\n",
    "\n",
    "* `lr`: Learning rate for the optimizer.\n",
    "* `epochs`: Number of times to iterate through the training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# loss and optimization functions\n",
    "lr=0.001\n",
    "\n",
    "criterion = nn.BCELoss()\n",
    "optimizer = torch.optim.Adam(net.parameters(), lr=lr)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# training loop\n",
    "def train(net, train_loader, epochs, print_every=100):\n",
    "\n",
    "    # move model to GPU, if available\n",
    "    if(train_on_gpu):\n",
    "        net.cuda()\n",
    "\n",
    "    counter = 0 # for printing\n",
    "    \n",
    "    # train for some number of epochs\n",
    "    net.train()\n",
    "    for e in range(epochs):\n",
    "\n",
    "        # batch loop\n",
    "        for inputs, labels in train_loader:\n",
    "            counter += 1\n",
    "\n",
    "            if(train_on_gpu):\n",
    "                inputs, labels = inputs.cuda(), labels.cuda()\n",
    "\n",
    "            # zero accumulated gradients\n",
    "            net.zero_grad()\n",
    "\n",
    "            # get the output from the model\n",
    "            output = net(inputs)\n",
    "\n",
    "            # calculate the loss and perform backprop\n",
    "            loss = criterion(output.squeeze(), labels.float())\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "            # loss stats\n",
    "            if counter % print_every == 0:\n",
    "                # Get validation loss\n",
    "                val_losses = []\n",
    "                net.eval()\n",
    "                for inputs, labels in valid_loader:\n",
    "\n",
    "                    if(train_on_gpu):\n",
    "                        inputs, labels = inputs.cuda(), labels.cuda()\n",
    "\n",
    "                    output = net(inputs)\n",
    "                    val_loss = criterion(output.squeeze(), labels.float())\n",
    "\n",
    "                    val_losses.append(val_loss.item())\n",
    "\n",
    "                net.train()\n",
    "                print(\"Epoch: {}/{}...\".format(e+1, epochs),\n",
    "                      \"Step: {}...\".format(counter),\n",
    "                      \"Loss: {:.6f}...\".format(loss.item()),\n",
    "                      \"Val Loss: {:.6f}\".format(np.mean(val_losses)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/2... Step: 100... Loss: 0.451722... Val Loss: 0.446736\n",
      "Epoch: 1/2... Step: 200... Loss: 0.435447... Val Loss: 0.365078\n",
      "Epoch: 1/2... Step: 300... Loss: 0.333672... Val Loss: 0.344555\n",
      "Epoch: 1/2... Step: 400... Loss: 0.319042... Val Loss: 0.328191\n",
      "Epoch: 2/2... Step: 500... Loss: 0.287158... Val Loss: 0.343141\n",
      "Epoch: 2/2... Step: 600... Loss: 0.300172... Val Loss: 0.364031\n",
      "Epoch: 2/2... Step: 700... Loss: 0.183973... Val Loss: 0.353891\n",
      "Epoch: 2/2... Step: 800... Loss: 0.162030... Val Loss: 0.354852\n"
     ]
    }
   ],
   "source": [
    "# training params\n",
    "\n",
    "epochs = 2 # this is approx where I noticed the validation loss stop decreasing\n",
    "print_every = 100\n",
    "\n",
    "train(net, train_loader, epochs, print_every=print_every)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Testing\n",
    "\n",
    "There are a few ways to test this network.\n",
    "\n",
    "* **Test data performance:** First, I'll see how our trained model performs on all of the defined test_data, above; I'll calculate the average loss and accuracy over the test data.\n",
    "\n",
    "* **Inference on user-generated data:** Second, I'll see if I can input just one example review at a time (without a label), and see what the trained model predicts. Looking at new, user input data like this, and predicting an output label, is called **inference**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.376\n",
      "Test accuracy: 0.840\n"
     ]
    }
   ],
   "source": [
    "# Get test data loss and accuracy\n",
    "\n",
    "test_losses = [] # track loss\n",
    "num_correct = 0\n",
    "\n",
    "\n",
    "net.eval()\n",
    "# iterate over test data\n",
    "for inputs, labels in test_loader:\n",
    "\n",
    "    if(train_on_gpu):\n",
    "        inputs, labels = inputs.cuda(), labels.cuda()\n",
    "    \n",
    "    # get predicted outputs\n",
    "    output = net(inputs)\n",
    "    \n",
    "    # calculate loss\n",
    "    test_loss = criterion(output.squeeze(), labels.float())\n",
    "    test_losses.append(test_loss.item())\n",
    "    \n",
    "    # convert output probabilities to predicted class (0 or 1)\n",
    "    pred = torch.round(output.squeeze())  # rounds to the nearest integer\n",
    "    \n",
    "    # compare predictions to true label\n",
    "    correct_tensor = pred.eq(labels.float().view_as(pred))\n",
    "    correct = np.squeeze(correct_tensor.numpy()) if not train_on_gpu else np.squeeze(correct_tensor.cpu().numpy())\n",
    "    num_correct += np.sum(correct)\n",
    "\n",
    "\n",
    "# -- stats! -- ##\n",
    "# avg test loss\n",
    "print(\"Test loss: {:.3f}\".format(np.mean(test_losses)))\n",
    "\n",
    "# accuracy over all test data\n",
    "test_acc = num_correct/len(test_loader.dataset)\n",
    "print(\"Test accuracy: {:.3f}\".format(test_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference on a test review\n",
    "\n",
    "You can change this test_review to any text that you want. Read it and think: is it pos or neg? Then see if your model predicts correctly!\n",
    "\n",
    "> The below `predict` code, takes in a trained `embed_lookup` table, a trained net, a plain text_review, and a sequence length, and prints out a custom statement for a positive or negative review!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from string import punctuation\n",
    "\n",
    "# helper function to process and tokenize a single review\n",
    "def tokenize_review(embed_lookup, test_review):\n",
    "    test_review = test_review.lower() # lowercase\n",
    "    # get rid of punctuation\n",
    "    test_text = ''.join([c for c in test_review if c not in punctuation])\n",
    "\n",
    "    # splitting by spaces\n",
    "    test_words = test_text.split()\n",
    "\n",
    "    # tokens\n",
    "    tokenized_review = []\n",
    "    for word in test_words:\n",
    "        try:\n",
    "            idx = embed_lookup.vocab[word].index\n",
    "        except: \n",
    "            idx = 0\n",
    "        tokenized_review.append(idx)\n",
    "\n",
    "    return tokenized_review\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(embed_lookup, net, test_review, sequence_length=200):\n",
    "    \"\"\"\n",
    "    Predict whether a given test_review has negative or positive sentiment.\n",
    "    \"\"\"\n",
    "    \n",
    "    net.eval()\n",
    "    \n",
    "    # tokenize review\n",
    "    test_ints = tokenize_review(embed_lookup, test_review)\n",
    "    \n",
    "    # pad tokenized sequence\n",
    "    seq_length=sequence_length\n",
    "    features = pad_features([test_ints], seq_length)\n",
    "    \n",
    "    # convert to tensor to pass into your model\n",
    "    feature_tensor = torch.from_numpy(features)\n",
    "    \n",
    "    batch_size = feature_tensor.size(0)\n",
    "    \n",
    "    if(train_on_gpu):\n",
    "        feature_tensor = feature_tensor.cuda()\n",
    "    \n",
    "    # get the output from the model\n",
    "    output = net(feature_tensor)\n",
    "    \n",
    "    # convert output probabilities to predicted class (0 or 1)\n",
    "    pred = torch.round(output.squeeze()) \n",
    "    # printing output value, before rounding\n",
    "    print('Prediction value, pre-rounding: {:.6f}'.format(output.item()))\n",
    "    \n",
    "    # print custom response\n",
    "    if(pred.item()==1):\n",
    "        print(\"Positive review detected!\")\n",
    "    else:\n",
    "        print(\"Negative review detected.\")\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test on pos/neg reviews\n",
    "\n",
    "Below, I test my code on both positive and negative reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set hyperparams\n",
    "seq_length=200 # good to use the length that was trained on\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction value, pre-rounding: 0.000775\n",
      "Negative review detected.\n"
     ]
    }
   ],
   "source": [
    "# negative test review\n",
    "test_review_neg = 'The worst movie I have seen; acting was terrible and I want my money back. This movie had bad acting and the dialogue was slow.'\n",
    "\n",
    "# test negative review\n",
    "predict(embed_lookup, net, test_review_neg, seq_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction value, pre-rounding: 0.992333\n",
      "Positive review detected!\n"
     ]
    }
   ],
   "source": [
    "# positive test review\n",
    "test_review_pos = 'This movie had the best acting and the dialogue was so good. I loved it.'\n",
    "\n",
    "predict(embed_lookup, net, test_review_pos, seq_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Try out test reviews of your own!\n",
    "\n",
    "Now that you have a trained model and a predict function, you can pass in _any_ kind of text and this model will predict whether the text has a positive or negative sentiment.\n",
    "\n",
    "---\n",
    "## Further reading\n",
    "\n",
    "More than text classification, CNNs are used to analyze sequential data in a number of ways! Here are a couple of papers and applications that I find really interesting:\n",
    "* CNN for semantic representations and **search query retrieval**, [paper (Microsoft)](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/www2014_cdssm_p07.pdf).\n",
    "* CNN for **genetic mutation detection**, [paper (Nature)](https://www.nature.com/articles/s41467-019-09027-x).\n",
    "* CNN for classifying [whale sounds](https://ai.googleblog.com/2018/10/acoustic-detection-of-humpback-whales.html) via spectogram and for [**audio classification**, generally (Google AI)](https://ai.google/research/pubs/pub45611)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}