CSE6521_PyTorch_Tutorial - DAN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **CSE6521(AU22): PyTorch Tutorial**\n",
    "\n",
    "### Author: Zexin (Jason) Xu\n",
    "\n",
    "This notebook will serve as a basic introduction to PyTorch. This notebook will include 3 sessions: tensor, neural network, and a sample NLP task. After finishing this notebook, you will be able to build a fully-connected feed-forward neural network with PyTorch.\n",
    "\n",
    "### Credit\n",
    "* [\"Word Window Classification\" tutorial notebook](https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1204/materials/ww_classifier.ipynb) by Matt Lamm, from Winter 2020 offering of CS224N\n",
    "* CSE224N: PyTorch Tutorial (Winter '22) by Dilara Soylu, Ethan Chi\n",
    "* Official PyTorch Documentation on [Deep Learning with PyTorch: A 60 Minute Blitz](https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html) by Soumith Chintala\n",
    "* PyTorch Tutorial Notebook, [Build Basic Generative Adversarial Networks (GANs) | Coursera](https://www.coursera.org/learn/build-basic-generative-adversarial-networks-gans) by Sharon Zhou, offered on Coursera\n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sample\n",
    "\n",
    "Sentiment analysis\n",
    "\n",
    "Deep Averaging Network..(DAN)\n",
    "\n",
    "0. Dataset https://www.kaggle.com/datasets/jp797498e/twitter-entity-sentiment-analysis\n",
    "1. Preprocessing\n",
    "2. Introduce dataloader\n",
    "3. Word embeddings\n",
    "4. Batch\n",
    "5. GPU\n",
    "6. Training\n",
    "7. Evaluation\n",
    "8. Confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(40000, 2)\n",
      "                                                text  label\n",
      "0  I grew up (b. 1965) watching and loving the Th...      0\n",
      "1  When I put this movie in my DVD player, and sa...      0\n",
      "2  Why do people who do not know what a particula...      0\n",
      "3  Even though I have great interest in Biblical ...      0\n",
      "4  Im a die hard Dads Army fan and nothing will e...      1\n",
      "-------------------------\n",
      "['I grew up (b. 1965) watching and loving the Thunderbirds. All my mates at school watched. We played \"Thunderbirds\" before school, during lunch and after school. We all wanted to be Virgil or Scott. No one wanted to be Alan. Counting down from 5 became an art form. I took my children to see the movie hoping they would get a glimpse of what I loved as a child. How bitterly disappointing. The only high point was the snappy theme tune. Not that it could compare with the original score of the Thunderbirds. Thankfully early Saturday mornings one television channel still plays reruns of the series Gerry Anderson and his wife created. Jonatha Frakes should hand in his directors chair, his version was completely hopeless. A waste of film. Utter rubbish. A CGI remake may be acceptable but replacing marionettes with Homo sapiens subsp. sapiens was a huge error of judgment.'\n",
      " \"When I put this movie in my DVD player, and sat down with a coke and some chips, I had some expectations. I was hoping that this movie would contain some of the strong-points of the first movie: Awsome animation, good flowing story, excellent voice cast, funny comedy and a kick-ass soundtrack. But, to my disappointment, not any of this is to be found in Atlantis: Milo's Return. Had I read some reviews first, I might not have been so let down. The following paragraph will be directed to those who have seen the first movie, and who enjoyed it primarily for the points mentioned.<br /><br />When the first scene appears, your in for a shock if you just picked Atlantis: Milo's Return from the display-case at your local videoshop (or whatever), and had the expectations I had. The music feels as a bad imitation of the first movie, and the voice cast has been replaced by a not so fitting one. (With the exception of a few characters, like the voice of Sweet). The actual drawings isnt that bad, but the animation in particular is a sad sight. The storyline is also pretty weak, as its more like three episodes of Schooby-Doo than the single adventurous story we got the last time. But dont misunderstand, it's not very good Schooby-Doo episodes. I didnt laugh a single time, although I might have sniggered once or twice.<br /><br />To the audience who haven't seen the first movie, or don't especially care for a similar sequel, here is a fast review of this movie as a stand-alone product: If you liked schooby-doo, you might like this movie. If you didn't, you could still enjoy this movie if you have nothing else to do. And I suspect it might be a good kids movie, but I wouldn't know. It might have been better if Milo's Return had been a three-episode series on a cartoon channel, or on breakfast TV.\"]\n",
      "[0 0]\n"
     ]
    }
   ],
   "source": [
    "dataframe = pd.read_csv('Data/movie.csv')\n",
    "print(dataframe.shape)\n",
    "print(dataframe.head())\n",
    "print('-' * 25)\n",
    "\n",
    "corpus = dataframe['text'].values\n",
    "labels = dataframe['label'].values\n",
    "print(corpus[:2])\n",
    "print(labels[:2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Max length: 2470\n",
      "(32000,)\n",
      "(8000,)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Getting max length of a sentence\n",
    "max_len = -1\n",
    "for sent in corpus:\n",
    "    max_len = max(max_len, len(sent.split()))\n",
    "print(f'Max length: {max_len}')\n",
    "\n",
    "# Splitting the data\n",
    "corpus_train, corpus_test, labels_train, labels_test = train_test_split(corpus, labels, test_size=0.2, random_state=42)\n",
    "\n",
    "print(corpus_train.shape)\n",
    "print(corpus_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First off, let's create a indexer!\n",
    "# Credit: ...\n",
    "class Indexer(object):\n",
    "    \"\"\"\n",
    "    Bijection between objects and integers starting at 0. Useful for mapping\n",
    "    labels, features, etc. into coordinates of a vector space.\n",
    "    \"\"\"\n",
    "    def __init__(self):\n",
    "        self.objs_to_idx = {}\n",
    "        self.idx_to_objs = {}\n",
    "        # Adding <unk> and <pad>\n",
    "        self.objs_to_idx['<unk>'] = 0\n",
    "        self.objs_to_idx['<pad>'] = 1\n",
    "        self.idx_to_objs[0] = '<unk>'\n",
    "        self.idx_to_objs[1] = '<pad>'\n",
    "\n",
    "    def __repr__(self):\n",
    "        return str(self.objs_to_idx)\n",
    "\n",
    "    def __str__(self):\n",
    "        return self.__repr__()\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.objs_to_idx)\n",
    "\n",
    "    def get_object(self, index):\n",
    "        \"\"\"\n",
    "        :param index: integer index to look up\n",
    "        :return: Returns the object corresponding to the particular index or <unk> if not found\n",
    "        \"\"\"\n",
    "        if (index not in self.idx_to_objs):\n",
    "            return self.idx_to_objs[0]\n",
    "        else:\n",
    "            return self.idx_to_objs[index]\n",
    "\n",
    "    def contains(self, object):\n",
    "        \"\"\"\n",
    "        :param object: object to look up\n",
    "        :return: Returns True if it is in the Indexer, False otherwise\n",
    "        \"\"\"\n",
    "        return self.index_of(object) != 0\n",
    "\n",
    "    def index_of(self, object):\n",
    "        \"\"\"\n",
    "        :param object: object to look up\n",
    "        :return: Returns -1 if the object isn't present, index otherwise\n",
    "        \"\"\"\n",
    "        if (object not in self.objs_to_idx):\n",
    "            return self.objs_to_idx['<unk>']\n",
    "        else:\n",
    "            return self.objs_to_idx[object]\n",
    "\n",
    "    def add_and_get_index(self, object, add=True):\n",
    "        \"\"\"\n",
    "        Adds the object to the index if it isn't present, always returns a nonnegative index\n",
    "        :param object: object to look up or add\n",
    "        :param add: True by default, False if we shouldn't add the object. If False, equivalent to index_of.\n",
    "        :return: The index of the object\n",
    "        \"\"\"\n",
    "        if not add:\n",
    "            return self.index_of(object)\n",
    "        if (object not in self.objs_to_idx):\n",
    "            new_idx = len(self.objs_to_idx)\n",
    "            self.objs_to_idx[object] = new_idx\n",
    "            self.idx_to_objs[new_idx] = object\n",
    "        return self.objs_to_idx[object]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Indexer: 295808\n",
      "[(0, '<unk>'), (1, '<pad>'), (2, 'i'), (3, 'watched'), (4, 'it'), (5, 'last'), (6, 'night'), (7, 'and'), (8, 'again'), (9, 'this'), (10, 'morning'), (11, '-'), (12, \"that's\"), (13, 'how'), (14, 'much'), (15, 'liked'), (16, 'it.'), (17, 'there'), (18, 'is'), (19, 'something'), (20, 'about'), (21, 'movie...'), (22, 'when'), (23, 'the'), (24, 'movie'), (25, 'was'), (26, 'almost'), (27, 'over,'), (28, 'to'), (29, 'cry.'), (30, 'would'), (31, 'strongly'), (32, 'recommend'), (33, '\"latter'), (34, 'days\"'), (35, 'my'), (36, 'friends'), (37, \"it's\"), (38, 'definitely'), (39, 'worth'), (40, 'seeing!'), (41, 'agree'), (42, 'with'), (43, 'those'), (44, 'who'), (45, 'say'), (46, 'that'), (47, 'some'), (48, 'parts'), (49, 'of')]\n"
     ]
    }
   ],
   "source": [
    "indexer = Indexer()\n",
    "\n",
    "for corpus in corpus_train:\n",
    "    for word in corpus.split():\n",
    "        indexer.add_and_get_index(word.lower())\n",
    "print(f'Indexer: {len(indexer)}')\n",
    "print(list(indexer.idx_to_objs.items())[:50]) # print first 50 items in the indexer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Parameter containing:\n",
      "tensor([[-1.5301, -1.2222,  0.8139, -1.4456, -0.0746],\n",
      "        [ 1.8849, -1.8677,  0.6735, -0.3937,  0.6189],\n",
      "        [ 0.4110,  0.0099, -0.1404,  0.5565, -0.3442],\n",
      "        ...,\n",
      "        [-0.3361, -2.7709,  1.5129, -0.3075, -1.2698],\n",
      "        [ 1.2058,  1.6777, -1.7956, -1.6128,  0.0434],\n",
      "        [ 2.7031,  0.4356, -1.6229,  2.1715, -2.8551]], requires_grad=True)]\n",
      "torch.Size([295808, 5])\n"
     ]
    }
   ],
   "source": [
    "# Creating word embeddings\n",
    "embedding_dim = 5 # 50, 100\n",
    "embedding = nn.Embedding(len(indexer), embedding_dim)\n",
    "print(list(embedding.parameters()))\n",
    "print(list(embedding.parameters())[0].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2, 1271, 9, 10610]\n",
      "tensor([[ 0.4110,  0.0099, -0.1404,  0.5565, -0.3442],\n",
      "        [-1.6840, -0.5460,  0.8224, -0.8703, -0.1626],\n",
      "        [ 0.3520, -0.3347,  1.8257, -1.1186,  0.2296],\n",
      "        [-1.1989,  0.5747, -0.1248,  1.2426, -0.5887]],\n",
      "       grad_fn=<EmbeddingBackward0>)\n"
     ]
    }
   ],
   "source": [
    "sentence = \"I hate this movie!\"\n",
    "\n",
    "idx = [indexer.index_of(word.lower()) for word in sentence.split()]\n",
    "print(idx)\n",
    "\n",
    "# Remember in the tensor indexing, we can do tensor[tensor]? Let's utilize it!\n",
    "idx_tensor = torch.tensor(idx, dtype=torch.long)\n",
    "print(embedding(idx_tensor))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now, let's batch stuff!\n",
    "\n",
    "we are going to use data loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n",
      "Batched Input:\n",
      "('I also attended the RI International Horror Film Festival and I can easily see why this film won best of show.<br /><br />SEA OF DUST is a wild romp of Horror, Comedy and beautiful scenery. A back in time tale of strange goings on. An increasingly wide spread illness with an overwhelmingly irritating side effect of people\\'s heads exploding, brings a young Professor\\'s apprentice; Stefan, to investigate. Along his travels, he decides to briefly detour and once again ask for his long time love\\'s hand in marriage, only to once again be sent packing by her extremely stubborn father\\x85 Along the way \"out of town\" he comes across an ill girl in the road and delivers her to Dr. Maitland, (brilliantly played by up and coming Vincent Price like actor: Edward X Young.) Who fills Stefan in on the Evils a foot. Only the Dr. is insulted that he had called for the Professor and only received a boy in training\\x85None the less, Stefan turns out to be much more than a common bystander. Horror Icon; Tom Savini portrays the ultimate religious torment monger; Prester John. Scream Queen; Ingrid Pitt comes out of retirement to give a stellar performance as Anna. Many beautiful and talented supporting actors seamlessly held the story together and helped to effectively move it along to the climax.<br /><br />Dark Religion and over the top, but fun and sometimes very original, gore scenes play heavily in this Hammer tribute flick. This stylish movie goes back and forth between flashbacks, surreal worlds, dreams and the character\\'s reality.<br /><br />Horror and Gore aside; This is also a very Funny movie! Slapstick, tongue and cheek humor and dark comedy raise their heads among the dark story line. Like others have stated; this really is like three great movies in one. Very Entertaining and Original.', \"It is amazing to me what passes for entertainment today. maybe I am a dinosaur from the fifties, and I am out of touch with todays movie going generation, and apparently that is the case with regards to this movie, since so many people loved it. I found it foul and vulgar. I haven't said that about many movies in my life but this one fits the bill. The humor is sophomoric and crude. I am not a politically correct person, and even I found the gay jokes, not only not funny but downright offensive ( I'm not gay). The main character in the movie is not even a likable person, just pathetic. When the movie was finally over i heard a number of people comment on how disappointed they were in what they had just pay good money to see.\", \"It's hard to comment on this movie. It's one of the few movies Dimension actually has not shelved (it's hard to come up with a reason why) and it was rushed into a an unimpressive 500 theaters it's opening day. Maybe Dimension was afraid of how people would respond to a swamp creature using his tow truck to pull a house apart piece by piece.<br /><br />Ray Sawyer is just a tow truck driver, until he rescues a Voodoo priestess from a bad car accident, and in return, he gets attacked by a bag full of snakes and drowns. At the morgue, Ray comes back to life, and stalks a group of teenagers who witnessed the awful crash occur. <br /><br />What brings this movie down is it's paper thin characters. I didn't care for one moment about any of them. Also, the dialog was less than ho-hum. Also, it was very predictable. Characters did the typical stupid horror movie character things, like check creaky noises, call out people's names, and trip on a rock while being chased. I also could immediately pick out who the final girl would be. And why did the camera have those quick white flashes whenever somebody died or whenever the killer was shown?<br /><br />What's good? Well, there is an impressive suspense scene where the killer walks underneath swamp waters to get to his victims and a tense sequence where the final girl must camouflage herself with bunch of other dead bodies while the killer looks on.<br /><br />But other than that, It's another August/September disappointment. I was looking forward to it, but I did not get what was expected.\", \"This film is about a young Indian guy who comes home one day and finds himself getting engaged to a woman. The problem is that he is gay. In order to stop the wedding without telling his parents that he is gay, one lie leads to another until it spirals out of control.<br /><br />This film is hilarious and got me laughing many times. Sally Bankes' acting is superb and she plays this strong woman who does what she wants convincingly. The plot is outstanding as well. I find the plot very realistic, and I can completely identify with Jimi's feeling of being terrified, worried and upset. On the other hand, Jimi's boyfriend, Jack, is given much less attention in the film. I would have liked him to be given more lines in the film, and have more character development. However, as I guess the director wants to make this a more mainstream film, the love between Jimi and Jack was not developed in the film.<br /><br />It is great to watch a gay affirmative film. Furthermore, this film preaches us to be accepting to other people's difference, be it sexuality, culture and the way of life. This film makes viewers think hard.<br /><br />We need films like this to give us a boost. Thanks for making this film!\")\n",
      "Batched Labels:\n",
      "tensor([1, 0, 0, 1])\n"
     ]
    }
   ],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "train_data = list(zip(corpus_train, labels_train))\n",
    "batch_size = 4\n",
    "shuffle = True\n",
    "\n",
    "# Things to add:\n",
    "# Collate function customization\n",
    "# collate_fn = partial(custom_collate_fn, window_size=window_size, word_to_ix=word_to_ix)\n",
    "\n",
    "dataloader = DataLoader(train_data, batch_size=batch_size, shuffle=shuffle)\n",
    "counter = 0\n",
    "batched_corpus, batched_labels = next(iter(dataloader))\n",
    "print(len(batched_corpus))\n",
    "print(\"Batched Input:\")\n",
    "print(batched_corpus)\n",
    "print(\"Batched Labels:\")\n",
    "print(batched_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Averaging Network\n",
    "\n",
    "Time to put everything together!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from torch.utils.data import DataLoader\n",
    "import torch.optim as optim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Indexer(object):\n",
    "    def __init__(self):\n",
    "        self.objs_to_idx = {}\n",
    "        self.idx_to_objs = {}\n",
    "        # Adding <unk> and <pad>\n",
    "        self.objs_to_idx['<unk>'] = 0\n",
    "        self.objs_to_idx['<pad>'] = 1\n",
    "        self.idx_to_objs[0] = '<unk>'\n",
    "        self.idx_to_objs[1] = '<pad>'\n",
    "\n",
    "    def __repr__(self):\n",
    "        return str(self.objs_to_idx)\n",
    "\n",
    "    def __str__(self):\n",
    "        return self.__repr__()\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.objs_to_idx)\n",
    "\n",
    "    def get_object(self, index):\n",
    "        if (index not in self.idx_to_objs):\n",
    "            return self.idx_to_objs[0]\n",
    "        else:\n",
    "            return self.idx_to_objs[index]\n",
    "        \n",
    "    def pad(self, index):\n",
    "        return self.objs_to_idx.index_of['<pad>']\n",
    "\n",
    "    def contains(self, object):\n",
    "        return self.index_of(object) != 0\n",
    "\n",
    "    def index_of(self, object):\n",
    "        if (object not in self.objs_to_idx):\n",
    "            return self.objs_to_idx['<unk>']\n",
    "        else:\n",
    "            return self.objs_to_idx[object]\n",
    "\n",
    "    def add_and_get_index(self, object, add=True):\n",
    "        if not add:\n",
    "            return self.index_of(object)\n",
    "        if (object not in self.objs_to_idx):\n",
    "            new_idx = len(self.objs_to_idx)\n",
    "            self.objs_to_idx[object] = new_idx\n",
    "            self.idx_to_objs[new_idx] = object\n",
    "        return self.objs_to_idx[object]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DeepAveragingNetwork(nn.Module):\n",
    "    \n",
    "    def __init__(self, hyper_params):\n",
    "        super(DeepAveragingNetwork, self).__init__()\n",
    "        self.hyper_params = hyper_params\n",
    "        self.embedding = nn.Embedding(len(indexer), hyper_params['embedding_dim'])\n",
    "        # nn.Embedding.from_pretrained()\n",
    "        \n",
    "        # model\n",
    "        self.model = nn.Sequential(\n",
    "            nn.Linear(hyper_params['embedding_dim'], hyper_params['hidden_dim']),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(hyper_params['hidden_dim'], hyper_params['output_dim']),\n",
    "            nn.LogSoftmax(dim=0)\n",
    "        )\n",
    "    \n",
    "    def average_embeddings(self, x):\n",
    "        embedding_tensor = []\n",
    "        for i, x_idx in enumerate(x):\n",
    "            embedding = torch.mean(self.embedding(torch.LongTensor(x_idx)), dim=0)\n",
    "            embedding_tensor.append(embedding.detach().numpy())\n",
    "        embedding_tensor = torch.Tensor(embedding_tensor)\n",
    "        return embedding_tensor\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = self.average_embeddings(x)\n",
    "        # print(self.model(x))\n",
    "        return self.model(x)\n",
    "    \n",
    "    def predict(self, x):\n",
    "        x = self.average_embeddings(x)\n",
    "        return torch.argmax(self.model(x), dim=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda\n",
      "Epoch: 0, Loss: 0.01617091754451394\n",
      "Epoch: 1, Loss: 0.016123207518830895\n",
      "Epoch: 2, Loss: 0.016105429036542773\n",
      "Epoch: 3, Loss: 0.016099855303764343\n",
      "Epoch: 4, Loss: 0.016094488324597478\n",
      "Epoch: 5, Loss: 0.016090859891846776\n",
      "Epoch: 6, Loss: 0.016086806543171406\n",
      "Epoch: 7, Loss: 0.016083694295957685\n",
      "Epoch: 8, Loss: 0.016080213710665703\n",
      "Epoch: 9, Loss: 0.016077284468337893\n",
      "Epoch: 10, Loss: 0.016075460705906153\n",
      "Epoch: 11, Loss: 0.016073307488113642\n",
      "Epoch: 12, Loss: 0.016071424121037126\n",
      "Epoch: 13, Loss: 0.01606796868145466\n",
      "Epoch: 14, Loss: 0.016064456896856427\n",
      "Epoch: 15, Loss: 0.016062066424638033\n",
      "Epoch: 16, Loss: 0.016059661051258445\n",
      "Epoch: 17, Loss: 0.016057911328971386\n",
      "Epoch: 18, Loss: 0.01605471852235496\n",
      "Epoch: 19, Loss: 0.016052857507020235\n",
      "Epoch: 20, Loss: 0.016050246078521013\n",
      "Epoch: 21, Loss: 0.016047833021730185\n",
      "Epoch: 22, Loss: 0.016046809731051326\n",
      "Epoch: 23, Loss: 0.016045585507526994\n",
      "Epoch: 24, Loss: 0.01604228955693543\n",
      "Epoch: 25, Loss: 0.016039262525737286\n",
      "Epoch: 26, Loss: 0.016038277186453342\n",
      "Epoch: 27, Loss: 0.01603733585216105\n",
      "Epoch: 28, Loss: 0.016036245971918106\n",
      "Epoch: 29, Loss: 0.016034454572945833\n"
     ]
    }
   ],
   "source": [
    "print(torch.device('cuda' if torch.cuda.is_available() else 'cpu'))\n",
    "\n",
    "indexer = Indexer()\n",
    "for corpus in corpus_train:\n",
    "    for word in corpus.split():\n",
    "        indexer.add_and_get_index(word.lower())\n",
    "        \n",
    "hyper_params = {\n",
    "    \"batch_size\": 4096,\n",
    "    \"vocab_size\" : len(indexer),\n",
    "    \"embedding_dim\": 50,\n",
    "    \"hidden_dim\": 50,\n",
    "    \"output_dim\": 2,\n",
    "    \"learning_rate\": 0.01,\n",
    "    \"num_epochs\": 30,\n",
    "}\n",
    "train_data = list(zip(corpus_train, labels_train))\n",
    "dataloader = DataLoader(train_data, batch_size=hyper_params['batch_size'], shuffle=shuffle)\n",
    "\n",
    "DAN = DeepAveragingNetwork(hyper_params)\n",
    "optimizer = optim.Adam(DAN.parameters(), lr=hyper_params['learning_rate'])\n",
    "loss_function = nn.NLLLoss()\n",
    "\n",
    "# Training\n",
    "for epoch in range(hyper_params['num_epochs']):\n",
    "    total_loss = 0\n",
    "    for batch in dataloader:\n",
    "        batched_corpus, batched_labels = batch\n",
    "        # print(batched_labels)\n",
    "        # Zero the grads!\n",
    "        optimizer.zero_grad()\n",
    "        # corpus -> indexes\n",
    "        idx = [[indexer.index_of(word.lower()) for word in corpus.split()] for corpus in batched_corpus]\n",
    "        # labels -> tensor\n",
    "        pred = DAN(idx)\n",
    "        loss = loss_function(pred, batched_labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        total_loss += loss.item()\n",
    "    print(f'Epoch: {epoch}, Loss: {total_loss/hyper_params[\"batch_size\"]}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Evaluation\n",
    "\n",
    "Confusion matrix explained and testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 5109 / 8000 = 0.638625;\n",
      "Precision (fraction of predicted positives that are correct): 2544 / 3945 = 0.644867;\n",
      "Recall (fraction of true positives predicted correctly): 2544 / 4034 = 0.630640;\n",
      "F1 (harmonic mean of precision and recall): 0.637674;\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_idx = [[indexer.index_of(word.lower()) for word in corpus.split()] for corpus in corpus_test]\n",
    "pred_test = DAN.predict(test_idx)\n",
    "\n",
    "def print_evaluation(golds, predictions):\n",
    "    \"\"\"\n",
    "    Prints evaluation statistics comparing golds and predictions, each of which is a sequence of 0/1 labels.\n",
    "    Prints accuracy as well as precision/recall/F1 of the positive class, which can sometimes be informative if either\n",
    "    the golds or predictions are highly biased.\n",
    "\n",
    "    :param golds: gold labels\n",
    "    :param predictions: pred labels\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    num_correct = 0\n",
    "    num_pos_correct = 0\n",
    "    num_pred = 0\n",
    "    num_gold = 0\n",
    "    num_total = 0\n",
    "    if len(golds) != len(predictions):\n",
    "        raise Exception(\"Mismatched gold/pred lengths: %i / %i\" % (len(golds), len(predictions)))\n",
    "    for idx in range(0, len(golds)):\n",
    "        gold = golds[idx]\n",
    "        prediction = predictions[idx]\n",
    "        if prediction == gold:\n",
    "            num_correct += 1\n",
    "        if prediction == 1:\n",
    "            num_pred += 1\n",
    "        if gold == 1:\n",
    "            num_gold += 1\n",
    "        if prediction == 1 and gold == 1:\n",
    "            num_pos_correct += 1\n",
    "        num_total += 1\n",
    "    acc = float(num_correct) / num_total\n",
    "    output_str = \"Accuracy: %i / %i = %f\" % (num_correct, num_total, acc)\n",
    "    prec = float(num_pos_correct) / num_pred if num_pred > 0 else 0.0\n",
    "    rec = float(num_pos_correct) / num_gold if num_gold > 0 else 0.0\n",
    "    f1 = 2 * prec * rec / (prec + rec) if prec > 0 and rec > 0 else 0.0\n",
    "    output_str += \";\\nPrecision (fraction of predicted positives that are correct): %i / %i = %f\" % (num_pos_correct, num_pred, prec)\n",
    "    output_str += \";\\nRecall (fraction of true positives predicted correctly): %i / %i = %f\" % (num_pos_correct, num_gold, rec)\n",
    "    output_str += \";\\nF1 (harmonic mean of precision and recall): %f;\\n\" % f1\n",
    "    print(output_str)\n",
    "    # return acc, f1, output_str\n",
    "\n",
    "print_evaluation(labels_test, pred_test.tolist())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build your own!\n",
    "\n",
    "Couple of stuff that can improve:\n",
    "1. Data cleaning\n",
    "2. More layers?\n",
    "3. Activation functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.10 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "75273f6ed8af91899ebc591bf6ae2fd0716c5db2515d7097a000123632f4e53a"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}