CNN.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "CNN.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [
        "8S-g-OYCVmiN"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/telecombcn-dl/2018-dlai-team10/blob/master/CNN.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "eYhLQOn3YqGA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#**Convolutional Neural Network**\n",
        "The problem we are trying to solve here is to classify grayscale images of handwritten objects (28 pixels by 28 pixels), into 10 categories (apple, banana, fork...). The dataset we will use is extracted from the Kaggle competition: **Quick Draw! Doodle Recognition Challenge**. \n",
        "\n",
        "In this notebook, we will approach this task by implementing a **Convolutional Neural Network**. For this project we have also implemented other two approaches (Multilayer Perceptron and Long-Short Term Memory Network), that also have a corresponding self-contained notebooks. \n",
        "\n",
        "Our motivation to tackle this problem of image classification using a CNN (Convolutional Neural Network) is quite obvious, because it is a specialized kind of neural network for processing data that has a known grid-like topology that leverages the ideas of local connectivity, parameter sharing and pooling/subsampling hidden units. \n",
        "\n",
        "*The basic idea behind a CNN is that the network learns hierarchical representations  of the data with increasing levels of abstraciton.*\n",
        "\n",
        "\n",
        "\n",
        "*For more details about out project please visit: https://telecombcn-dl.github.io/2018-dlai-team10/ *"
      ]
    },
    {
      "metadata": {
        "id": "dwTJ68cmcdlr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#**1. Notebook Setting**\n",
        "\n",
        "In this section we will import Pytorch and some relevant Python libraries (Numpy, Matplotlib...) that will later be used. Additionally, we will set the notebook environment to train on the GPU to obtain faster results. "
      ]
    },
    {
      "metadata": {
        "id": "b_k3w0d1hxzB",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from os.path import exists\n",
        "from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag\n",
        "platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())\n",
        "cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\\.\\([0-9]*\\)\\.\\([0-9]*\\)$/cu\\1\\2/'\n",
        "accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'\n",
        "!pip install -q http://download.pytorch.org/whl/{accelerator}/torch-0.4.1-{platform}-linux_x86_64.whl torchvision\n",
        "  \n",
        "import numpy as np\n",
        "import os\n",
        "import sys\n",
        "import torch\n",
        "import torchvision\n",
        "import random\n",
        "import codecs\n",
        "import torch.utils.data\n",
        "import torch.optim as optim\n",
        "import torchvision.datasets as datasets\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "from matplotlib import pyplot as plt\n",
        "from PIL import Image\n",
        "from tqdm import tqdm\n",
        "from torchvision import datasets, transforms\n",
        "\n",
        "#Training on the GPU\n",
        "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "print(device)\n",
        "\n",
        "print('Done!')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "1euniB3GOHmk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# **2. Dataset Preparation**\n",
        "\n",
        "In this section we will download a part of the original dataset, we will reduce the number of samples, distribute them in training, validation and test, reshape them into images and organize them in a structured way. "
      ]
    },
    {
      "metadata": {
        "id": "E1ww9hWadKaR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## **2.1 Dataset Download**\n",
        "\n",
        "The dataset is downloaded from the Google APIs and it comes in the form of a set of Numpy arrays. The Quick! Draw challenge dataset actually contains more than 300 classes, however we will only use 10 of them for our project, for a simplification purpose. We have manually selected the classes we will work with in order to have some interesting inter-class variability (wheeel and pizza are very similar while apple is very different...).\n",
        "\n",
        "The classes we will try to classify are the following ones:\n",
        "\n",
        "\n",
        "![clases](https://user-images.githubusercontent.com/43316350/50059288-43b93480-0185-11e9-8d9f-76695e781a8b.JPG) "
      ]
    },
    {
      "metadata": {
        "id": "cfmjdhj88dfN",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "  urls = [\n",
        "        'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/key.npy',\n",
        "        'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/banana.npy',\n",
        "        'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/ladder.npy',\n",
        "        'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/tennis%20racquet.npy',\n",
        "        'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/pizza.npy',\n",
        "        'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/stop%20sign.npy',\n",
        "        'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/wheel.npy',\n",
        "        'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/fork.npy',\n",
        "        'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/book.npy',\n",
        "        'https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/apple.npy',\n",
        "    ]\n",
        "  \n",
        "  class_name = ['key', 'banana', 'ladder', 'tennis_racquet', 'pizza', 'stop_sign', 'wheel', 'fork', 'book', 'apple']\n",
        "   \n",
        "  def createDir(path):\n",
        "    if not os.path.exists(path):\n",
        "        os.makedirs(path)\n",
        "    \n",
        "  def gen_bar_updater(pbar):\n",
        "    def bar_update(count, block_size, total_size):\n",
        "        if pbar.total is None and total_size:\n",
        "            pbar.total = total_size\n",
        "        progress_bytes = count * block_size\n",
        "        pbar.update(progress_bytes - pbar.n)\n",
        "    return bar_update   \n",
        "    \n",
        "  def download_url(url, root, filename):\n",
        "      from six.moves import urllib\n",
        "      root = os.path.expanduser(root)\n",
        "      fpath = os.path.join(root, filename + \".npy\")\n",
        "\n",
        "      createDir(root)\n",
        "\n",
        "      # downloads file\n",
        "      if os.path.isfile(fpath):\n",
        "          a = 1\n",
        "          #print('Using downloaded and verified file: ' + fpath)\n",
        "      else:\n",
        "          try:\n",
        "              print('Downloading ' + url + ' to ' + fpath)\n",
        "              urllib.request.urlretrieve(\n",
        "                  url, fpath,\n",
        "                  reporthook = gen_bar_updater(tqdm(unit='B', unit_scale=True))\n",
        "              )\n",
        "          except OSError:\n",
        "              if url[:5] == 'https':\n",
        "                  url = url.replace('https:', 'http:')\n",
        "                  print('Failed download. Trying https -> http instead.'\n",
        "                        ' Downloading ' + url + ' to ' + fpath)\n",
        "                  urllib.request.urlretrieve(\n",
        "                      url, fpath,\n",
        "                      reporthook = gen_bar_updater(tqdm(unit='B', unit_scale=True))\n",
        "                  )\n",
        "                  \n",
        "                  \n",
        "                  \n",
        "  for i in range(0, len(urls)):\n",
        "    download_url(urls[i], \"data\", class_name[i])\n",
        "    \n",
        "    \n",
        "  print(\"Done!\")   "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jPNS_XT5nCIN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## **2.2 Dataset Reduction, Reshaping and Reorganization**\n",
        "As we are implementing a CNN (we are willing to exploit the local connectivity of the data), we want to have the data as images. Furthermore, we have decided to work with a reduced dataset, so the number of samples per class will be *max_length*. We also split the data into training, validation and test by the percentages defined by *percen* and place each sample in its corresponding folder. \n",
        "\n",
        "![datapercentage](https://user-images.githubusercontent.com/43316350/50059513-20dc4f80-0188-11e9-9c09-5e798d97cd96.JPG)\n"
      ]
    },
    {
      "metadata": {
        "id": "C7wbAlMHZ1kr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class_name = ['apple', 'banana', 'book', 'fork', 'key', 'ladder', 'pizza', 'stop_sign', 'tennis_racquet', 'wheel']\n",
        "step = ['train', 'validation', 'test']\n",
        "\n",
        "dire = r'data/'\n",
        "\n",
        "max_length = 10000 # Maximum number of files (drawings) per class\n",
        "percen=[0.6, 0.3, 0.1] # Percentage of training, validation and testing\n",
        "\n",
        "begin = [0, int(max_length * percen[0]), int(max_length * (percen[0] + percen[1])) ]\n",
        "end = [int(max_length * (percen[0])), int(max_length * (percen[0] + percen[1])) , max_length-10]\n",
        "\n",
        "for c in range(0, len(class_name)):\n",
        "  print('Class ' + str(c+1) + ' out of ' + str(len(class_name)))\n",
        "  filename = dire + str(class_name[c]) + '.npy'\n",
        "  data = np.load(filename)\n",
        "  \n",
        "  for s in range(0, len(step)):\n",
        "    dire_step = str(dire) + str(step[s])\n",
        "    if not os.path.exists(dire_step):\n",
        "      os.makedirs(dire_step)\n",
        "    \n",
        "    for i in range(begin[s], end[s]):\n",
        "      dire_class = str(dire_step) + '/' + str(class_name[c])\n",
        "      if not os.path.exists(dire_class):\n",
        "        os.makedirs(dire_class)\n",
        "      \n",
        "      # Reshape the raw data into 28x28 images\n",
        "      data_sample = data[i,:].reshape((28, 28))\n",
        "      sample_name = class_name[c] + '_' + str(step[s]) + '_' + str(i)\n",
        "      np.save(os.path.join(dire_class, sample_name), data_sample)\n",
        "\n",
        "print('Done!')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "HhoE0veRfHoT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## **2.3 Data Visualization**\n",
        "\n",
        "An interesting experiment (and validation step) we can do is to randomly visualize an image corresponding to the training set of images of the selected class. "
      ]
    },
    {
      "metadata": {
        "id": "3Hb-DgpT72zX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "drawing_class = 0  # 0-apple, 1-banana, 2-book, 3-fork, 4-key, 5-ladder, 6-pizza, 7-stop_sign, 8-tennis_racquet, 9-wheel\n",
        "image_number=random.randint(1,max_length*percen[0])\n",
        "dire = r'data/train/' + str(class_name[drawing_class]) + '/' + str(class_name[drawing_class]) + '_' + 'train' + '_' + str(image_number) +'.npy'\n",
        "data = np.load(dire)\n",
        "plt.imshow(data)\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "SM8HHsj3Xw9p",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# **3. Network Definition**\n",
        "\n",
        "In this section we will define mini-batchs, will set the architecture of the network and the forward pass, and will also define the loss function and the optimizer. "
      ]
    },
    {
      "metadata": {
        "id": "wbwgqtNKU8ZT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##**3.1 Mini-Batch Definition**\n",
        "\n",
        "We define a mini-batch of size *bs*. This sample subsets of data is what is going to be forward propagated through the network. We use a mini-batch instead of the whole batch because it would be very expensive to use the complete training set. "
      ]
    },
    {
      "metadata": {
        "id": "ZTTQPubmvVjj",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def load_sample(x):\n",
        "\treturn np.load(x)\n",
        "\n",
        "bs = 30 #To perfectly fit in the data\n",
        "train_dir = r\"data/train\"\n",
        "val_dir = r\"data/validation\"\n",
        "test_dir = r\"data/test\"\n",
        "\n",
        "transform = transforms.Compose(\n",
        "    [transforms.ToTensor(),\n",
        "     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\n",
        "\n",
        "train_dataset = datasets.DatasetFolder(train_dir, extensions = ['.npy'], loader = load_sample)\n",
        "train_loader = torch.utils.data.DataLoader(train_dataset, batch_size = bs, shuffle = True, num_workers = 2)\n",
        "train_iter = iter(train_loader)\n",
        "\n",
        "valid_dataset = datasets.DatasetFolder(val_dir, extensions = ['.npy'], loader = load_sample)\n",
        "valid_loader = torch.utils.data.DataLoader(valid_dataset, batch_size = bs, shuffle = True, num_workers = 2)\n",
        "valid_iter = iter(valid_loader)\n",
        "\n",
        "test_dataset = datasets.DatasetFolder(test_dir, extensions = ['.npy'], loader = load_sample)\n",
        "test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = bs, shuffle = True, num_workers = 2)\n",
        "test_iter = iter(test_loader)\n",
        "\n",
        "batch, labels = train_iter.next()\n",
        "\n",
        "print('Done!')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9Dbh7DIbgvgc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## **3.2 CNN Definition and Forward Pass**\n",
        "We started creating the basic CNN architectures and testing how they performed but, as it was very shallow it gave very poor results. In fact, most of the times it got stuck very soon in a local minimum, so the results were awful. \n",
        "\n",
        "With the purpose of improving the performance of the CNN, we deepened the network, so the probability of finding a *bad* local minimum decreased. We came up with the following structure that resulted to be excellent in terms of performance. \n",
        "\n",
        "This final architecture, which will be followingly explained, consists basically on alternating 5 convolutional layers (followed by a non-linearity and a batch normalization layer) with 2 max-pooling layers and, ending with 3 fully connected layers also followed by non-linearity. \n",
        "\n",
        "![arquitecturacnn3](https://user-images.githubusercontent.com/43316350/50046302-c963b400-00a1-11e9-90e4-769db06d6ec9.JPG)\n",
        "\n",
        "The **Convolutional Layers**  transform 3D input volume to a 3D output volume of neuron activations performing convolutions on a 2D grid. For the final architecture we have used 5 convolutional with a kernel size of 3x3 and of stride=1 each. They differ in the number of filters though, passing from 6 filters in the first layers to 16 and ending with 32 filters. These last characteristics (filter spatial extent, stride and number of filters) have been set as hyperparameters, which means that they their value is the one that has proven to give a better performance to the network after trying different ones. \n",
        "\n",
        "The **Non-liniarity Layers** that we have used are ReLU (Rectified Linear Unit) Layers, which can be seen as simple range transforms that perform a simple pixel-based mapping that sets the negative values of the image to zero. \n",
        "\n",
        "We also introduced **Batch Normalization** layers (normalize the activations of each channel by subbstracting the mean and dividing by the standard deviation), with the objective of simplifying, speeding up the training and reducing the sensitivity to network initialization. \n",
        "\n",
        "The network also contains two **Pooling Layers**, which are in charge of the down-sampling of the image and therefore reducing the number of activations, as well as providing invariance to small local changes. Four our architecture we have chosen to get the maximum values of 2x2 pixel rectangular regions around the input locations (that is, Max-Pooling with stride 2,2). It must be noted that we have just used two of this layer because the original size of our input data was already quite small (28x28 pixel images), so if we wanted a deep network, we could not afford adding pooling layers after each convolutional because we would have lost too much information about precise position of things. \n",
        "\n",
        "The **Fully-connected Layers** are the classic layers in which every neuron in the previous layer is connected to every neuron in the next layer and activation is computed as matrix multiplication plus bias. Here, the output of the last convolutional layer is flattened to a single vector which is input to a fully connected layer. \n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "6xNq_vmpwdTz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def weight_init(m):\n",
        "  if isinstance(m,nn.Conv2d):\n",
        "    torch.nn.init.xavier_uniform_(m.weight.data)        \n",
        "  \n",
        "class Net(nn.Module):\n",
        "  \n",
        "    def __init__(self):\n",
        "        super(Net, self).__init__()\n",
        "        self.conv1 = nn.Conv2d(1, 6, 3, padding = 1)\n",
        "        torch.nn.init.xavier_uniform_(self.conv1.weight)\n",
        "        \n",
        "        self.bn1 = nn.BatchNorm2d(6)\n",
        "        \n",
        "        self.conv2 = nn.Conv2d(6, 16, 3, padding = 1)\n",
        "        torch.nn.init.xavier_uniform_(self.conv2.weight)\n",
        "        \n",
        "        self.pool = nn.MaxPool2d(2, 2)\n",
        "        \n",
        "        self.bn2 = nn.BatchNorm2d(16)\n",
        "        \n",
        "        self.conv3 = nn.Conv2d(16, 16, 3, padding = 1)\n",
        "        torch.nn.init.xavier_uniform_(self.conv3.weight)\n",
        "        \n",
        "        self.bn3 = nn.BatchNorm2d(16)\n",
        "        \n",
        "        self.conv4 = nn.Conv2d(16, 32, 3, padding = 1)\n",
        "        torch.nn.init.xavier_uniform_(self.conv4.weight)\n",
        "        \n",
        "        self.bn4 = nn.BatchNorm2d(32)\n",
        "        \n",
        "        self.conv5 = nn.Conv2d(32, 32, 3, padding = 1)\n",
        "        torch.nn.init.xavier_uniform_(self.conv5.weight)\n",
        "        \n",
        "        self.bn5 = nn.BatchNorm2d(32)\n",
        "        \n",
        "        self.fc1 = nn.Linear(32 * 7 * 7, 120)\n",
        "        self.fc2 = nn.Linear(120, 84)\n",
        "        self.fc3 = nn.Linear(84, 10)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = F.relu(self.bn1(self.conv1(x)))\n",
        "        x = F.relu(self.bn2(self.conv2(x)))\n",
        "        x = self.pool(x)\n",
        "        x = F.relu(self.bn3(self.conv3(x)))              \n",
        "        x = F.relu(self.bn4(self.conv4(x)))\n",
        "        x = self.pool(x)\n",
        "        x = F.relu(self.bn5(self.conv5(x)))\n",
        "        x = F.relu((self.conv5(x)))\n",
        "        x = x.view(-1, 32 * 7 * 7)\n",
        "        x = self.fc1(x)\n",
        "        x = F.relu(x)\n",
        "        x = self.fc2(x)\n",
        "        x = F.relu(x)\n",
        "        x = self.fc3(x)\n",
        "        return x\n",
        "\n",
        "net = Net()\n",
        "net.apply(weight_init)\n",
        "net.to(device)\n",
        "print(net)\n",
        "\n",
        "print('Done!')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "YAA_NJoVYejA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## **3.3 Loss Function and Optimizer Definition**\n",
        "\n",
        "As we are working on a classification task, we have chosen to use the Cross Entropy Loss. For the optimizer we will use Adaptive Moments - **ADAM** (having previously seen that it gives better results than the Gradient Descent, where the network got stuck very often). \n",
        "\n",
        "As an hyperparameter, we worked with different **learning rates**,\n",
        "\n",
        "To prevent the overfitting that appeared on our model, we decided to implement **loss regularization**, though we could have used many other techniques such as early stopping, dropout, or data augmentation among others. We decided to add the L2 Regularization (or weight decay) to our cross-entropy loss. The L2 penalizes the complexity of the classifier by measuring the number of zeros in the weight vector. The resulting total loss is the following. \n",
        "\n",
        "![loss](https://user-images.githubusercontent.com/43316350/50059425-ee7e2280-0186-11e9-8973-6bcbf4670a88.JPG)\n",
        "\n",
        "Where *lambda* is the regularization hyperparameter (experimentally decided value).\n"
      ]
    },
    {
      "metadata": {
        "id": "2_X0YAHg316V",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "criterion = nn.CrossEntropyLoss()\n",
        "optimizer = optim.Adam(net.parameters(), lr=0.00001,weight_decay=0.001)\n",
        "print('Done!')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "esOsTmQft5fZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# **4. Network Training**\n",
        "\n",
        "In this section we will train our model and validate it with the validation data. At the end of the training, we will plot the lossses and the accuracies obtained for each epoch both for the training and the validation data. "
      ]
    },
    {
      "metadata": {
        "id": "lqBWarLoV4Z1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 4.1 Training and Validation\n",
        "\n",
        "Now that we have our model properly built. let's train the model. The process that is performed each epoch is the following one:\n",
        "\n",
        "1. Get a  batch from the DataLoader\n",
        "\n",
        "2. Forward Pass that batch through the network\n",
        "\n",
        "3. Get the ouputs of the propagated batch\n",
        "\n",
        "4. Compute the loss and accuracy with respect these outputs\n",
        "\n",
        "5. Propagate backwards to compute the parameters update\n",
        "\n",
        "6. At the end of each epoch, we compute a forward pass and its corresponding loss and accuracy in the entire validation set"
      ]
    },
    {
      "metadata": {
        "id": "eKoLV6eSt85B",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# To plot the results\n",
        "training_loss_list = []\n",
        "training_accuracy_list = []\n",
        "validation_loss_list = []\n",
        "validation_accuracy_list = []\n",
        "\n",
        "for epoch in range(3):  # loop over the dataset multiple times\n",
        "\n",
        "    running_loss = 0.0\n",
        "    training_accuracy = 0.0\n",
        "    training_total = 0.0\n",
        "    training_correct = 0.0\n",
        "    \n",
        "    for i, data in enumerate(train_loader, 0):\n",
        "      \n",
        "        # get the inputs\n",
        "        inputs, labels = data\n",
        "        inputs, labels = inputs.to(device), labels.to(device)\n",
        "        \n",
        "        # zero the parameter gradients\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "        # forward + backward + optimize\n",
        "        inputs = inputs.view(bs,1,28,28).float()\n",
        "        inputs = inputs.to(device)\n",
        "        outputs = net(inputs)\n",
        "        outputs = outputs.to(device)\n",
        "        labels = labels.to(device)\n",
        "        \n",
        "        _, predicted = torch.max(outputs.data, 1) #gets the index of the maximum predicted value\n",
        "        training_total = training_total + labels.size(0)\n",
        "        training_correct = training_correct + (predicted == labels).sum().item() #accumulate correct\n",
        "        \n",
        "        loss = criterion(outputs, labels)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "\n",
        "        # print statistics\n",
        "        running_loss += loss.item()\n",
        "        if i % 200 == 199:    # print every 2000 mini-batches\n",
        "            training_accuracy=training_correct/training_total\n",
        "            training_accuracy_list.append(training_accuracy)\n",
        "            print('[%d, %5d] Training Loss: %.3f - Training Accuracy: %.3f' %\n",
        "                  (epoch + 1, i + 1, running_loss / 200,training_accuracy))\n",
        "            training_loss_list.append(running_loss/200)\n",
        "            running_loss = 0.0\n",
        "            total=0.0\n",
        "            correct=0.0\n",
        "            \n",
        "    with torch.no_grad():\n",
        "      \n",
        "      running_validation_loss=0.0\n",
        "      validation_accuracy=0.0\n",
        "      validation_total=0.0\n",
        "      validation_correct=0.0\n",
        "      \n",
        "      for j, valid_data in enumerate(valid_loader,0):     \n",
        "        valid_inputs, valid_labels = valid_data\n",
        "        valid_inputs = valid_inputs.view(bs, 1, 28, 28).float()\n",
        "        valid_inputs = valid_inputs.to(device)\n",
        "        valid_labels = valid_labels.to(device)\n",
        "        valid_outputs = net(valid_inputs)\n",
        "        valid_loss = criterion(valid_outputs, valid_labels)\n",
        "        running_validation_loss += valid_loss.item()\n",
        "        \n",
        "        _,predicted=torch.max(valid_outputs.data,1)\n",
        "        validation_total=validation_total+valid_labels.size(0)\n",
        "        validation_correct=validation_correct + (predicted == valid_labels).sum().item()\n",
        "        \n",
        "      validation_accuracy=validation_correct/validation_total\n",
        "      validation_accuracy_list.append(validation_accuracy)\n",
        "      print('[%d] Validation Loss: %.3f - Validation Accuracy: %.3f' %\n",
        "          (epoch + 1, running_validation_loss/len(valid_loader), validation_accuracy))\n",
        "      validation_loss_list.append(running_validation_loss/len(valid_loader))\n",
        "\n",
        "\n",
        "print('Finished Training')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8S-g-OYCVmiN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 4.2 Results Visualization\n",
        "\n",
        "We will now visualize how did the training go... Both for the training and validation set we will plot the progress of the loss and the accuracy throughout all the epochs. Remember that we have been using the cross entropy loss with weight decay regularization. For the accuracy, it is computed as proportion between the number of correct outputs and the total number of outputs. \n",
        "\n",
        "*As future work, it would be interesting that the network created the losses and accuracy plots in real time.*"
      ]
    },
    {
      "metadata": {
        "id": "ooU4g0RWHAxB",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "training_examples = 6e4\n",
        "plot_every = 200 #batches\n",
        "training_loss_np = np.asarray(training_loss_list)\n",
        "validation_loss_np = np.asarray(validation_loss_list)\n",
        "training_accuracy_np = np.asarray(training_accuracy_list)\n",
        "validation_accuracy_np = np.asarray(validation_accuracy_list)\n",
        "\n",
        "x_axis_train = np.arange(1, len(training_loss_list)+1)\n",
        "x_axis_validation = np.arange(1, len(validation_loss_list)+1)\n",
        "\n",
        "p1=plt.plot(x_axis_train * bs * plot_every / training_examples, training_loss_np)\n",
        "p2=plt.plot(x_axis_validation, validation_loss_np,color=\"r\")\n",
        "\n",
        "plt.xlabel('epochs')\n",
        "plt.title('Loss')\n",
        "plt.legend((p1[0], p2[0]), ('Training Loss', 'Validation Loss'))\n",
        "plt.show(),\n",
        "\n",
        "p3=plt.plot(x_axis_train * bs * plot_every / training_examples, training_accuracy_np)\n",
        "p4=plt.plot(x_axis_validation, validation_accuracy_np,color=\"r\")\n",
        "plt.xlabel('epochs')\n",
        "plt.title('Accuracy')\n",
        "plt.legend((p3[0], p4[0]), ('Training Accuracy', 'Validation Accuracy'))\n",
        "plt.show()\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "VYGMvzBa1mky",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# **5. Network Testing**\n",
        "In this section, we will comput the test accuracy and the test loss, we will plot the confusion matrix to see which classess performed better and we will do a little performance demo. "
      ]
    },
    {
      "metadata": {
        "id": "2bBNj1FWgr58",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 5.1 Test Accuracy and Loss Computation\n",
        "Let's evaluate the model on the test data. To do so, we will pass to the network mini-batches of test data and compare their results with the ground truth to compute its loss and accuracy.\n",
        "\n",
        "Additionally, to see how well the network performs on different categories, we have created a plot that shows the accuracy for each class. It can be noted that classes that were very similar (wheel and pizza for example) have lower accuracy than the others, while very different and clear objects such as apple, have a very high accuracy. "
      ]
    },
    {
      "metadata": {
        "id": "GuiC_DNl1q_n",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class_correct = list(0. for i in range(10))\n",
        "class_total = list(0. for i in range(10))\n",
        "running_test_loss=0.0\n",
        "test_total=0.0\n",
        "test_correct=0.0\n",
        "with torch.no_grad():\n",
        "    for data in test_loader:\n",
        "        test_inputs, test_labels = data\n",
        "        test_inputs = test_inputs.view(bs, 1, 28, 28).float()\n",
        "        test_inputs = test_inputs.to(device)\n",
        "        test_labels = test_labels.to(device)\n",
        "        test_outputs = net(test_inputs)\n",
        "        test_loss = criterion(test_outputs, test_labels)\n",
        "        running_test_loss += test_loss.item()\n",
        "        _, predicted = torch.max(test_outputs.data,1)\n",
        "        c = (predicted == test_labels).squeeze()\n",
        "        test_total=test_total+test_labels.size(0)\n",
        "        test_correct=test_correct + (predicted == test_labels).sum().item()\n",
        "        for i in range(bs):\n",
        "            label = test_labels[i]\n",
        "            class_correct[label] += c[i].item()\n",
        "            class_total[label] += 1\n",
        "            \n",
        "test_accuracy=test_correct/test_total\n",
        "\n",
        "print('Test Loss: %.3f - Test Accuracy: %.3f' %\n",
        "         (running_test_loss/len(test_loader), test_accuracy))\n",
        "print()\n",
        "x=np.arange(len(class_name))\n",
        "plt.barh(x, class_correct, align='center', alpha=0.5)\n",
        "plt.yticks(x, class_name)\n",
        "plt.xlabel('Accuracy')\n",
        "plt.title('Accuracy by Class')\n",
        " \n",
        "plt.show()\n",
        "\n",
        "torch.save(net.state_dict(), 'data/weights_state_at_epoch_100')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "OaQBGjTvfeoU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 5.2 Performance Demo\n",
        "\n",
        "Finally, in this little demo we can see how the network performs for a random image of the last batch of the test set. An interesting experiment to do is to first try to classify the image by ourselfs and then looking to the predicted class and the ground true value to see if the network performed better than a human..."
      ]
    },
    {
      "metadata": {
        "id": "Z_hhvSIRpzvp",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "test_inputs, test_labels = data\n",
        "test_inputs = test_inputs.view(bs, 1, 28, 28).float()\n",
        "test_inputs = test_inputs.to(device)\n",
        "test_labels = test_labels.to(device)\n",
        "test_outputs = net(test_inputs)\n",
        "_, predicted = torch.max(test_outputs.data,1)\n",
        "image_number=random.randint(0,bs-1)\n",
        "a=test_inputs[image_number].cpu().numpy()\n",
        "plt.imshow(a[0, :, :])\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "v6FAvjg2_wl-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Let's see the network's prediction and the ground truth!"
      ]
    },
    {
      "metadata": {
        "id": "ODMvUg8T_5Z2",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "print('PREDICTED: It is a/an: %s!' % class_name[predicted[image_number]])\n",
        "print('GROUND TRUTH: %s' % class_name[test_labels[image_number]])"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}