Data_Augmentation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"http://cocl.us/pytorch_link_top\">\n",
    "    <img src=\"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DL0110EN/notebook_images%20/Pytochtop.png\" width=\"750\" alt=\"IBM Product \">\n",
    "</a> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkCV0101ENCoursera872-2023-01-01\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  />\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Data Augmentation </h1> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Table of Contents</h2>\n",
    "\n",
    "In this lab, we train a Convolutional Neural Network with Regular data and Augmented data. The purpose of this lab is to show that the Augmented data improves generalization performance.\n",
    "\n",
    "Think of a scenario where a drone has to take a picture of an object. The drone is moving and the object can also possibly be moving. When an image is taken we arent always going to get perfect images. The subject may not be perfectly centered in the image or the subject may be rotated in the image. In this case, a model trained on perfectly centered or rotated images won't perform well. This is why we train a model on rotated data so it can perform well on imperfect images.\n",
    "\n",
    "In this assignment, we will use a dataset of digit images. We will have two models one trained on non rotated digits and one trained on rotated images and then we will test the models on a rotated testing dataset which will be more realistic and robust in terms of our scenario above.\n",
    "\n",
    "<ul>\n",
    "<li><a href=\"#Makeup_Data\">Get Some Data</a></li>\n",
    "<li><a href=\"#CNN\">Convolutional Neural Network</a></li>\n",
    "<li><a href=\"#R_training_data\">Rotated Training Data</a></li>\n",
    "\n",
    "</ul>\n",
    "<p>Estimated Time Needed: <strong>25 min</strong> 14 min to train model </p>\n",
    "\n",
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Preparation</h2>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2023-04-13 12:38:25--  https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/meet_up/12.02.2020/normal.pt\n",
      "Resolving s3-api.us-geo.objectstorage.softlayer.net (s3-api.us-geo.objectstorage.softlayer.net)... 67.228.254.196\n",
      "Connecting to s3-api.us-geo.objectstorage.softlayer.net (s3-api.us-geo.objectstorage.softlayer.net)|67.228.254.196|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 75134 (73K) [application/octet-stream]\n",
      "Saving to: ‘normal.pt’\n",
      "\n",
      "normal.pt           100%[===================>]  73.37K  --.-KB/s    in 0.002s  \n",
      "\n",
      "2023-04-13 12:38:25 (33.2 MB/s) - ‘normal.pt’ saved [75134/75134]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Download a Pretrained Model because training takes a long time\n",
    "!wget https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/meet_up/12.02.2020/normal.pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2023-04-13 12:38:30--  https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/meet_up/12.02.2020/rotated_data.pt\n",
      "Resolving s3-api.us-geo.objectstorage.softlayer.net (s3-api.us-geo.objectstorage.softlayer.net)... 67.228.254.196\n",
      "Connecting to s3-api.us-geo.objectstorage.softlayer.net (s3-api.us-geo.objectstorage.softlayer.net)|67.228.254.196|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 75134 (73K) [application/octet-stream]\n",
      "Saving to: ‘rotated_data.pt’\n",
      "\n",
      "rotated_data.pt     100%[===================>]  73.37K  --.-KB/s    in 0.04s   \n",
      "\n",
      "2023-04-13 12:38:30 (1.78 MB/s) - ‘rotated_data.pt’ saved [75134/75134]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Download a Pretrained Model Trained on Augmented Data because training takes a long time\n",
    "!wget https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/meet_up/12.02.2020/rotated_data.pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting Pillow==6.2.2\n",
      "  Downloading Pillow-6.2.2-cp37-cp37m-manylinux1_x86_64.whl (2.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m23.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: Pillow\n",
      "  Attempting uninstall: Pillow\n",
      "    Found existing installation: Pillow 8.1.0\n",
      "    Uninstalling Pillow-8.1.0:\n",
      "      Successfully uninstalled Pillow-8.1.0\n",
      "Successfully installed Pillow-6.2.2\n"
     ]
    }
   ],
   "source": [
    "# Library to Show Images\n",
    "!pip install Pillow==6.2.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Restart the kernel***\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the libraries we need to use in this lab\n",
    "# Using the following line code to install the torchvision library\n",
    "# !conda install -y torchvision\n",
    "\n",
    "# PyTorch Library\n",
    "import torch \n",
    "# PyTorch Neural Network Library\n",
    "import torch.nn as nn\n",
    "# Allows us to transform data\n",
    "import torchvision.transforms as transforms\n",
    "# Used to graph data and loss curves\n",
    "import matplotlib.pylab as plt\n",
    "# Allows us to use arrays to manipulate and store data\n",
    "import numpy as np\n",
    "# Allows us to download the dataset\n",
    "import torchvision.datasets as dsets\n",
    "# Allows us to access the filesystem\n",
    "import os  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some useful functions for plotting \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot Cost and Accuracy vs Epoch Graph\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_cost_accuracy(checkpoint):\n",
    "\n",
    "# Plot the cost and accuracy\n",
    "\n",
    "    fig, ax1 = plt.subplots()\n",
    "    color = 'tab:red'\n",
    "    ax1.plot(checkpoint['cost'], color=color)\n",
    "    ax1.set_xlabel('epoch', color=color)\n",
    "    ax1.set_ylabel('Cost', color=color)\n",
    "    ax1.tick_params(axis='y', color=color)\n",
    "    \n",
    "    ax2 = ax1.twinx()  \n",
    "    color = 'tab:blue'\n",
    "    ax2.set_ylabel('accuracy', color=color) \n",
    "    ax2.set_xlabel('epoch', color=color)\n",
    "    ax2.plot( checkpoint['accuracy'], color=color)\n",
    "    ax2.tick_params(axis='y', color=color)\n",
    "    fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the function <code>show_data</code> to plot out data samples as images.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_data(data_sample):\n",
    "    plt.imshow(data_sample[0].numpy().reshape(IMAGE_SIZE, IMAGE_SIZE), cmap='gray')\n",
    "    plt.title('y = '+ str(data_sample[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot first 5 misclassified samples \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_mis_classified(model, dataset):\n",
    "    count=0\n",
    "    for x, y in torch.utils.data.DataLoader(dataset=dataset, batch_size=1):\n",
    "        z = model(x)\n",
    "        _, yhat = torch.max(z, 1)\n",
    "        if yhat != y:\n",
    "            show_data((x, y))\n",
    "            plt.show()\n",
    "            count += 1\n",
    "        if count >= 5:\n",
    "            break "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"Makeup_Data\">Load Data</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a transform object  <code>compose</code> one will resize the image and convert it to a tensor, the second will also rotate the image Randomly rotate the image.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Size of the images are 16 by 16\n",
    "IMAGE_SIZE = 16\n",
    "\n",
    "# Creating a group of transformations to created a rotated dataset\n",
    "# Resizes the images, randomly rotates it, and then converts it to a tensor\n",
    "compose_rotate = transforms.Compose([transforms.Resize((IMAGE_SIZE, IMAGE_SIZE)),transforms.RandomAffine(45), transforms.ToTensor()])\n",
    "\n",
    "# Creating a group of transformations to created a non rotated dataset\n",
    "# Resizes the images then converts it to a tensor\n",
    "compose = transforms.Compose([transforms.Resize((IMAGE_SIZE, IMAGE_SIZE)), transforms.ToTensor()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the training dataset by setting the parameters <code>train</code> to <code>True</code>. We use the transform defined above, one with rotated data one without.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The transform parameters is set to the corresponding compose\n",
    "train_dataset_rotate = dsets.MNIST(root='./data', train=True, download=True, transform=compose_rotate)\n",
    "train_dataset = dsets.MNIST(root='./data', train=True, download=True, transform=compose)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the testing dataset by setting the parameters train to <code>False</code>, where the data is <b>ALL</b> rotated.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the testing dataset\n",
    "validation_dataset = dsets.MNIST(root='./data', train=False, download=True, transform=compose_rotate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each element in the rectangular tensor corresponds to a number representing a pixel intensity as demonstrated by the following image.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DL0110EN/notebook_images%20/chapter%206/6.2.1imagenet.png\" width=\"550\" alt=\"MNIST data image\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the first sample \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAGxCAYAAADLfglZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAlgUlEQVR4nO3de3RU9b3//9eQkCFwkoEEkzBKQrDURO4CuhApoQiegFjrUYpoRF3rCCXcRDFQtEZ7SAStBwsSDp5TxKNYrEtT6qqXFMJtCRISgtZjQSRCKk2zRDrhIiGXz++P749Zjgkhgdl8ZsLzsdb+Y/b+7PfnnSyGVz4ze/a4jDFGAABY0MF2AwCAyxchBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBLTCkSNHlJubq/LyctutBN0XX3wht9utHTt2+Pfl5ubK5XI12Tp16hRw7rFjx9S1a1cVFhZe4q7RXkTabgAIB0eOHNFTTz2lXr16adCgQbbbCapHH31UY8eO1fDhw5sce++99+TxePyPO3QI/Lu1W7duevjhhzV//nyNHz9eUVFRjveL9oUQAi4zxhidPn1a0dHR+uyzz1RYWKj33nuv2bFDhgxR9+7dW6w3ffp0/cd//IfefPNNTZkyxYmW0Y7xchxC1rZt2+RyufT66683OfbKK6/I5XKppKTE8T42b96sYcOGSZIeeOAB/0tTubm5/jG7d+/Wbbfdpri4OHXq1EmDBw/WG2+8EVDn5ZdflsvlUnFxsX7+85+re/fuio+P1x133KEjR44EjN20aZMyMjIUHx+v6OhoJScn69/+7d906tQp/5hvvvlGM2bM0JVXXqmoqCj17t1bixYtUm1tbUAtl8ulmTNnatWqVUpPT5fb7dbatWslSQUFBUpKStLYsWMv+PeTmJiosWPHatWqVRdcA5cxA4SwwYMHmxEjRjTZP2zYMDNs2LDznl9XV9eqrbGx8Zw1fD6fWbNmjZFkHn/8cbNjxw6zY8cOU1lZaYwxZtOmTSYqKsqMHDnSrF+/3rz33nvm/vvvN5LMmjVr/HXO1ujdu7eZNWuWef/9981///d/m27dupnRo0f7x1VUVJhOnTqZsWPHmsLCQrN582bz2muvmaysLHPs2DFjjDHffvutGTBggOnSpYt57rnnzAcffGCeeOIJExkZacaPHx/QvyRz5ZVXmgEDBph169aZTZs2mb/85S/GGGN69+5tJk2a1ORnfvLJJ40kk5SUZDp06GASEhJMVlaWOXToULO/oyVLlpgOHTr4+wNaixBCSDv7H/eePXv8+3bt2mUkmbVr17Z4bkVFhZHUqq24uLjFWiUlJU1C5ay0tDQzePBgU1dXF7D/1ltvNT169DANDQ0BP8uMGTMCxi1dutRIMn//+9+NMca8+eabRpIpLy8/Zz+rVq0ykswbb7wRsH/JkiVGkvnggw/8+yQZj8djvvnmm4Cx//jHP4wk88wzzzSp/8orr5jFixebP/3pT2bTpk3mmWeeMXFxcSYxMdH87W9/azK+qKjISDLvvvvuOXsGmsN7Qghpd999t3JycvTiiy/qpZdekiQtX75cV1xxhX72s5+1eK7X6231y3XXXHPNBfV34MAB/fWvf9Vzzz0nSaqvr/cfGz9+vN555x3t27dP6enp/v233XZbQI0BAwZIkg4dOqSkpCQNGjRIUVFReuihhzRjxgyNHDlSvXv3Djhn06ZN6tKli+68886A/ffff79ycnK0cePGgJfYfvzjH6tbt24BY8++BJiQkNDk58rKygp4PHr0aI0ePVrDhw/X0qVL9cILLwQcP1vjq6++alILaAkhhJDmdrs1bdo0/frXv9azzz6ruro6vfHGG5o3b57cbneL50ZFRbX6SraIiIgL6u8f//iHpP93hdmjjz7a7Jivv/464HF8fHzA47M/x7fffitJuvrqq/XnP/9ZS5cuVXZ2tk6ePKnevXtr9uzZmjNnjiTp6NGjSkpKksvlCqiVkJCgyMhIHT16NGB/jx49mvR1dr7vX3Z9Ltdff71++MMfaufOnU2Ona1xtibQWoQQQt7Pf/5zPfPMM/rtb3+r06dPq76+XtOnTz/veV9++aVSU1NbNUdxcbEyMjLa3NvZK8cWLlyoO+64o9kxF7LKGjlypEaOHKmGhgbt3r1by5cv19y5c5WYmKjJkycrPj5eH330kYwxAUFUXV2t+vr6Jle0fT+svtv7N9980+q+jDFNLtP+bo3zXUkHfB8hhJDXo0cP3XXXXVq5cqXOnDmjiRMnKjk5+bznBfPluO+vVr57Xp8+fbR3717l5eW1aq62iIiI0A033KC0tDS99tprKisr0+TJkzVmzBi98cYbKiws1E9/+lP/+FdeeUWSNGbMmPPWTklJUXR0tL744otW9bJz5059/vnnmj17dpNjBw8elCRde+21raoFnEUIISzMmTNHN9xwgyRpzZo1rTonKipKQ4cODcr8V199taKjo/Xaa68pPT1d//Iv/yKv1yuv16v/+q//UmZmpm655Rbdf//9uvLKK/XNN9/os88+U1lZmX7/+9+3aa5Vq1Zp06ZNmjBhgpKTk3X69Gn99re/lSTdfPPNkqT77rtPL774oqZOnaovv/xS/fv31/bt25WXl6fx48f7x7UkKipKw4cPb/bltYEDB+ree+9Venq6OnXqpF27dunZZ59VUlKSHnvssSbjd+7cqfj4ePXv379NPyvA1XEIG7169TLp6enW5n/99ddNWlqa6dixo5FknnzySf+xvXv3mkmTJpmEhATTsWNHk5SUZH784x+bVatW+cecvTqupKQkoG5xcXHAFXo7duwwP/3pT01KSopxu90mPj7ejBo1ymzYsCHgvKNHj5rp06ebHj16mMjISJOSkmIWLlxoTp8+HTBOksnOzm72Z/qf//kfExERYY4cORKwf/LkyeYHP/iB6dKli+nYsaNJSUkx06dPbzLOGGMaGxtNSkqKmTVr1nl/h8D3uYwxxm4MAuf38ccfa+DAgXrxxRc1Y8YM2+20G6dPn1ZycrIeeeQR5eTkXFCNjRs3aty4cfr000+VlpYW5A7R3hFCCGlffPGFDh06pF/84hc6fPiwDhw4oM6dO9tuq10pKChQbm6uDh48qC5durT5/NGjR+sHP/iB/xJ6oC14Twgh7Ve/+pX+93//V+np6fr9739PADngoYce0j//+U8dPHiwze/pHDt2TKNGjWJ1igvGSggAYA03MAUAWEMIAQCsIYQAANaE3IUJjY2NOnLkiGJiYpq91QgAILQZY3T8+HF5vd5mb/P0XSEXQkeOHFHPnj1ttwEAuEiVlZW66qqrWhwTci/HxcTE2G4BABAErfn/PORCiJfgAKB9aM3/5yEXQgCAywchBACwhhACAFhDCAEArHEshFauXKnU1FR16tRJQ4YM0bZt25yaCgAQphwJofXr12vu3LlatGiR9uzZo5EjRyozM1OHDx92YjoAQJhy5C7aN9xwg6677joVFBT496Wnp+v2229Xfn5+i+fW1NTI4/EEuyUAwCXm8/kUGxvb4pigr4TOnDmj0tJSjRs3LmD/uHHj9OGHHzYZX1tbq5qamoANAHB5CHoIff3112poaFBiYmLA/sTERFVVVTUZn5+fL4/H49+4ZQ8AXD4cuzDh+5+UNcY0++nZhQsXyufz+bfKykqnWgIAhJig38C0e/fuioiIaLLqqa6ubrI6kiS32y232x3sNgAAYSDoK6GoqCgNGTJERUVFAfuLiop04403Bns6AEAYc+SrHObNm6esrCwNHTpUw4cP1+rVq3X48GFNnz7diekAAGHKkRD62c9+pqNHj+rpp5/W3//+d/Xr109/+tOflJKS4sR0AIAw5cjnhC4GnxMCgPbByueEAABoLUIIAGANIQQAsIYQAgBYQwgBAKwhhAAA1hBCAABrCCEAgDWEEADAGkIIAGANIQQAsIYQAgBYQwgBAKwhhAAA1hBCAABrCCEAgDWEEADAGkIIAGANIQQAsIYQAgBYQwgBAKwhhAAA1hBCAABrCCEAgDWEEADAGkIIAGANIQQAsCbSdgNAqOrQwbm/0SIiIhyr3bVrV8dqu91ux2o3NjY6VjshIcGx2pJUVVUVlrVDASshAIA1hBAAwBpCCABgDSEEALCGEAIAWEMIAQCsIYQAANYQQgAAa4IeQvn5+Ro2bJhiYmKUkJCg22+/Xfv27Qv2NACAdiDoIbRlyxZlZ2dr586dKioqUn19vcaNG6eTJ08GeyoAQJgL+m173nvvvYDHa9asUUJCgkpLS/WjH/2oyfja2lrV1tb6H9fU1AS7JQBAiHL8PSGfzydJiouLa/Z4fn6+PB6Pf+vZs6fTLQEAQoTLGGOcKm6M0U9+8hMdO3ZM27Zta3ZMcyshggihgBuYNsUNTJvHDUyb5/P5FBsb2+IYR++iPXPmTH388cfavn37Oce43W5H/2EDAEKXYyE0a9YsbdiwQVu3btVVV13l1DQAgDAW9BAyxmjWrFl6++23tXnzZqWmpgZ7CgBAOxH0EMrOzta6dev0hz/8QTExMf7XMz0ej6Kjo4M9HQAgjAX9ndeCggL5fD5lZGSoR48e/m39+vXBngoAEOYceTkOAIDW4N5xAABrCCEAgDWEEADAGkc/rIrgcblcjtXu3r27Y7Ul6eqrr3asdr9+/Ryrfe211zpW+4orrnCsdq9evRyrffToUcdqO3mT4x/+8IeO1ZakxYsXO1a7sLDQsdqhgJUQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWRNpuAK3ToYNzfy/cfvvtjtWWpF/+8peO1e7evbtjtd1ut2O1v/76a8dqx8TEOFb76aefdqz2zp07Hat9/Phxx2pL0r59+xyt356xEgIAWEMIAQCsIYQAANYQQgAAawghAIA1hBAAwBpCCABgDSEEALDG8RDKz8+Xy+XS3LlznZ4KABBmHA2hkpISrV69WgMGDHByGgBAmHIshE6cOKF77rlHL730krp16+bUNACAMOZYCGVnZ2vChAm6+eabWxxXW1urmpqagA0AcHlw5Aamv/vd71RWVqaSkpLzjs3Pz9dTTz3lRBsAgBAX9JVQZWWl5syZo1dffVWdOnU67/iFCxfK5/P5t8rKymC3BAAIUUFfCZWWlqq6ulpDhgzx72toaNDWrVu1YsUK1dbWKiIiwn/M7XY7est8AEDoCnoIjRkzRp988knAvgceeEBpaWnKyckJCCAAwOUt6CEUExOjfv36Bezr0qWL4uPjm+wHAFzeuGMCAMCaS/L13ps3b74U0wAAwgwrIQCANYQQAMAaQggAYA0hBACw5pJcmICL19DQ4Fjtbdu2OVZbklauXOlY7bvuusux2vX19Y7VfvTRRx2rfcUVVzhWe+/evY7VPnDggGO1EbpYCQEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYE2k7QZg31//+ldH6y9ZssSx2qdPn3as9i233OJY7S+//NKx2lu3bnWsNhBsrIQAANYQQgAAawghAIA1hBAAwBpCCABgDSEEALCGEAIAWONICH311Ve69957FR8fr86dO2vQoEEqLS11YioAQBgL+odVjx07phEjRmj06NF69913lZCQoC+++EJdu3YN9lQAgDAX9BBasmSJevbsqTVr1vj39erVK9jTAADagaC/HLdhwwYNHTpUd911lxISEjR48GC99NJL5xxfW1urmpqagA0AcHkIeggdPHhQBQUF6tOnj95//31Nnz5ds2fP1iuvvNLs+Pz8fHk8Hv/Ws2fPYLcEAAhRQQ+hxsZGXXfddcrLy9PgwYM1bdo0/fu//7sKCgqaHb9w4UL5fD7/VllZGeyWAAAhKugh1KNHD1177bUB+9LT03X48OFmx7vdbsXGxgZsAIDLQ9BDaMSIEdq3b1/Avv379yslJSXYUwEAwlzQQ+jhhx/Wzp07lZeXpwMHDmjdunVavXq1srOzgz0VACDMBT2Ehg0bprfffluvv/66+vXrp1/96ldatmyZ7rnnnmBPBQAIc458s+qtt96qW2+91YnSAIB2hHvHAQCsIYQAANYQQgAAawghAIA1LmOMsd3Ed9XU1Mjj8dhuA2HC6/U6Vvu5555zrPY///lPx2oXFhY6Vnvz5s2O1T5z5oxjtWGHz+c77w0IWAkBAKwhhAAA1hBCAABrCCEAgDWEEADAGkIIAGANIQQAsIYQAgBYQwgBAKwhhAAA1hBCAABrCCEAgDWEEADAGkIIAGANIQQAsIYQAgBYQwgBAKwhhAAA1hBCAABrCCEAgDWEEADAGkIIAGCNyxhjbDfxXTU1NfJ4PLbbQJhwuVyO1R42bJhjtRcsWOBY7bS0NMdq5+XlOVb7rbfecqz2qVOnHKuNc/P5fIqNjW1xDCshAIA1hBAAwBpCCABgDSEEALCGEAIAWEMIAQCsIYQAANYEPYTq6+v1+OOPKzU1VdHR0erdu7eefvppNTY2BnsqAECYiwx2wSVLlmjVqlVau3at+vbtq927d+uBBx6Qx+PRnDlzgj0dACCMBT2EduzYoZ/85CeaMGGCJKlXr156/fXXtXv37mBPBQAIc0F/Oe6mm27Sxo0btX//fknS3r17tX37do0fP77Z8bW1taqpqQnYAACXh6CvhHJycuTz+ZSWlqaIiAg1NDRo8eLFuvvuu5sdn5+fr6eeeirYbQAAwkDQV0Lr16/Xq6++qnXr1qmsrExr167Vc889p7Vr1zY7fuHChfL5fP6tsrIy2C0BAEJU0FdC8+fP14IFCzR58mRJUv/+/XXo0CHl5+dr6tSpTca73W653e5gtwEACANBXwmdOnVKHToElo2IiOASbQBAE0FfCU2cOFGLFy9WcnKy+vbtqz179uj555/Xgw8+GOypAABhLughtHz5cj3xxBOaMWOGqqur5fV6NW3aNP3yl78M9lQAgDAX9BCKiYnRsmXLtGzZsmCXBgC0M9w7DgBgDSEEALCGEAIAWEMIAQCsCfqFCcClZIxxrHZJSYljtWfOnOlY7dmzZztW28kLjs6cOeNY7TfffNOx2pL4HORFYCUEALCGEAIAWEMIAQCsIYQAANYQQgAAawghAIA1hBAAwBpCCABgDSEEALCGEAIAWEMIAQCsIYQAANYQQgAAawghAIA1hBAAwBpCCABgDSEEALCGEAIAWEMIAQCsIYQAANYQQgAAawghAIA1kbYbAC5Ghw7O/R2VlJTkWO1Ro0Y5Vjs9Pd2x2vHx8Y7V7tatm2O1EbpYCQEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBACwps0htHXrVk2cOFFer1cul0uFhYUBx40xys3NldfrVXR0tDIyMvTpp58Gq18AQDvS5hA6efKkBg4cqBUrVjR7fOnSpXr++ee1YsUKlZSUKCkpSWPHjtXx48cvulkAQPvS5jsmZGZmKjMzs9ljxhgtW7ZMixYt0h133CFJWrt2rRITE7Vu3TpNmzbt4roFALQrQX1PqKKiQlVVVRo3bpx/n9vt1qhRo/Thhx82e05tba1qamoCNgDA5SGoIVRVVSVJSkxMDNifmJjoP/Z9+fn58ng8/q1nz57BbAkAEMIcuTrO5XIFPDbGNNl31sKFC+Xz+fxbZWWlEy0BAEJQUO+iffauw1VVVerRo4d/f3V1dZPV0Vlut1tutzuYbQAAwkRQV0KpqalKSkpSUVGRf9+ZM2e0ZcsW3XjjjcGcCgDQDrR5JXTixAkdOHDA/7iiokLl5eWKi4tTcnKy5s6dq7y8PPXp00d9+vRRXl6eOnfurClTpgS1cQBA+GtzCO3evVujR4/2P543b54kaerUqXr55Zf12GOP6dtvv9WMGTN07Ngx3XDDDfrggw8UExMTvK4BAO1Cm0MoIyNDxphzHne5XMrNzVVubu7F9AUAuAxw7zgAgDWEEADAGkIIAGANIQQAsCaoH1ZFeDrX3SyCxePxOFb7lltucaz2Qw895Fjt66+/3rHaJ06ccKx2QUGBY7U3btzoWO3GxkbHauPisBICAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMCaSNsNtCcRERGO1e7WrZtjtUeNGuVYbUn613/9V8dq33nnnY7Vrq+vd6z2H//4R8dqr1271rHaH374oWO1jx8/7lhthC5WQgAAawghAIA1hBAAwBpCCABgDSEEALCGEAIAWEMIAQCsIYQAANa0OYS2bt2qiRMnyuv1yuVyqbCw0H+srq5OOTk56t+/v7p06SKv16v77rtPR44cCWbPAIB2os0hdPLkSQ0cOFArVqxocuzUqVMqKyvTE088obKyMr311lvav3+/brvttqA0CwBoX9p8257MzExlZmY2e8zj8aioqChg3/Lly3X99dfr8OHDSk5ObnJObW2tamtr/Y9ramra2hIAIEw5/p6Qz+eTy+VS165dmz2en58vj8fj33r27Ol0SwCAEOFoCJ0+fVoLFizQlClTFBsb2+yYhQsXyufz+bfKykonWwIAhBDH7qJdV1enyZMnq7GxUStXrjznOLfbLbfb7VQbAIAQ5kgI1dXVadKkSaqoqNCmTZvOuQoCAFzegh5CZwPo888/V3FxseLj44M9BQCgnWhzCJ04cUIHDhzwP66oqFB5ebni4uLk9Xp15513qqysTO+8844aGhpUVVUlSYqLi1NUVFTwOgcAhL02h9Du3bs1evRo/+N58+ZJkqZOnarc3Fxt2LBBkjRo0KCA84qLi5WRkXHhnQIA2p02h1BGRoaMMec83tIxAAC+i3vHAQCsIYQAANYQQgAAawghAIA1jt0x4XI0ZswYx2o/9thjjtUeOnSoY7UlBdygNthKS0sdq/2b3/zGsdrFxcWO1T5+/LhjtYFgYyUEALCGEAIAWEMIAQCsIYQAANYQQgAAawghAIA1hBAAwBpCCABgDSEEALCGEAIAWEMIAQCsIYQAANYQQgAAawghAIA1hBAAwBpCCABgDSEEALCGEAIAWEMIAQCsIYQAANYQQgAAawghAIA1LmOMsd3Ed9XU1Mjj8dhu44JcffXVjtXu06ePY7Xr6uocqy1JVVVVjtU+fPiwY7WPHz/uWG3gcuDz+RQbG9viGFZCAABrCCEAgDWEEADAGkIIAGANIQQAsIYQAgBYQwgBAKxpcwht3bpVEydOlNfrlcvlUmFh4TnHTps2TS6XS8uWLbuIFgEA7VWbQ+jkyZMaOHCgVqxY0eK4wsJCffTRR/J6vRfcHACgfYts6wmZmZnKzMxsccxXX32lmTNn6v3339eECRMuuDkAQPvW5hA6n8bGRmVlZWn+/Pnq27fvecfX1taqtrbW/7impibYLQEAQlTQL0xYsmSJIiMjNXv27FaNz8/Pl8fj8W89e/YMdksAgBAV1BAqLS3VCy+8oJdfflkul6tV5yxcuFA+n8+/VVZWBrMlAEAIC2oIbdu2TdXV1UpOTlZkZKQiIyN16NAhPfLII+rVq1ez57jdbsXGxgZsAIDLQ1DfE8rKytLNN98csO+WW25RVlaWHnjggWBOBQBoB9ocQidOnNCBAwf8jysqKlReXq64uDglJycrPj4+YHzHjh2VlJSka6655uK7BQC0K20Ood27d2v06NH+x/PmzZMkTZ06VS+//HLQGgMAtH9tDqGMjAy15ctYv/zyy7ZOAQC4THDvOACANYQQAMAaQggAYA0hBACwxmXacpXBJVBTUyOPx2O7DQDARfL5fOe9AQErIQCANYQQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBACwhhACAFgTciFkjLHdAgAgCFrz/3nIhdDx48dttwAACILW/H/uMiG29GhsbNSRI0cUExMjl8t13vE1NTXq2bOnKisrFRsbewk6DA76vvTCtXf6vrTo++IZY3T8+HF5vV516NDyWifyEvXUah06dNBVV13V5vNiY2Ot/+IvBH1feuHaO31fWvR9cTweT6vGhdzLcQCAywchBACwJuxDyO1268knn5Tb7bbdSpvQ96UXrr3T96VF35dWyF2YAAC4fIT9SggAEL4IIQCANYQQAMAaQggAYA0hBACwJqxDaOXKlUpNTVWnTp00ZMgQbdu2zXZL55Wfn69hw4YpJiZGCQkJuv3227Vv3z7bbbVZfn6+XC6X5s6da7uV8/rqq6907733Kj4+Xp07d9agQYNUWlpqu60W1dfX6/HHH1dqaqqio6PVu3dvPf3002psbLTdWhNbt27VxIkT5fV65XK5VFhYGHDcGKPc3Fx5vV5FR0crIyNDn376qZ1mv6Olvuvq6pSTk6P+/furS5cu8nq9uu+++3TkyBF7Df//zvf7/q5p06bJ5XJp2bJll6y/tgrbEFq/fr3mzp2rRYsWac+ePRo5cqQyMzN1+PBh2621aMuWLcrOztbOnTtVVFSk+vp6jRs3TidPnrTdWquVlJRo9erVGjBggO1WzuvYsWMaMWKEOnbsqHfffVf/93//p1//+tfq2rWr7dZatGTJEq1atUorVqzQZ599pqVLl+rZZ5/V8uXLbbfWxMmTJzVw4ECtWLGi2eNLly7V888/rxUrVqikpERJSUkaO3as9ZsVt9T3qVOnVFZWpieeeEJlZWV66623tH//ft12220WOg10vt/3WYWFhfroo4/k9XovUWcXyISp66+/3kyfPj1gX1pamlmwYIGlji5MdXW1kWS2bNliu5VWOX78uOnTp48pKioyo0aNMnPmzLHdUotycnLMTTfdZLuNNpswYYJ58MEHA/bdcccd5t5777XUUetIMm+//bb/cWNjo0lKSjLPPPOMf9/p06eNx+Mxq1atstBh877fd3N27dplJJlDhw5dmqZa4Vx9/+1vfzNXXnml+ctf/mJSUlLMf/7nf17y3lorLFdCZ86cUWlpqcaNGxewf9y4cfrwww8tdXVhfD6fJCkuLs5yJ62TnZ2tCRMm6Oabb7bdSqts2LBBQ4cO1V133aWEhAQNHjxYL730ku22zuumm27Sxo0btX//fknS3r17tX37do0fP95yZ21TUVGhqqqqgOeq2+3WqFGjwvK56nK5Qn4V3djYqKysLM2fP199+/a13c55hdxdtFvj66+/VkNDgxITEwP2JyYmqqqqylJXbWeM0bx583TTTTepX79+tts5r9/97ncqKytTSUmJ7VZa7eDBgyooKNC8efP0i1/8Qrt27dLs2bPldrt133332W7vnHJycuTz+ZSWlqaIiAg1NDRo8eLFuvvuu2231iZnn4/NPVcPHTpko6ULcvr0aS1YsEBTpkwJiTtUt2TJkiWKjIzU7NmzbbfSKmEZQmd9//uGjDGt+g6iUDFz5kx9/PHH2r59u+1WzquyslJz5szRBx98oE6dOtlup9UaGxs1dOhQ5eXlSZIGDx6sTz/9VAUFBSEdQuvXr9err76qdevWqW/fviovL9fcuXPl9Xo1depU2+21WTg/V+vq6jR58mQ1NjZq5cqVtttpUWlpqV544QWVlZWFze83LF+O6969uyIiIpqseqqrq5v8xRWqZs2apQ0bNqi4uPiCvj/pUistLVV1dbWGDBmiyMhIRUZGasuWLfrNb36jyMhINTQ02G6xWT169NC1114bsC89PT3kL2CZP3++FixYoMmTJ6t///7KysrSww8/rPz8fNuttUlSUpIkhe1zta6uTpMmTVJFRYWKiopCfhW0bds2VVdXKzk52f88PXTokB555BH16tXLdnvNCssQioqK0pAhQ1RUVBSwv6ioSDfeeKOlrlrHGKOZM2fqrbfe0qZNm5Sammq7pVYZM2aMPvnkE5WXl/u3oUOH6p577lF5ebkiIiJst9isESNGNLkEfv/+/UpJSbHUUeucOnWqyTdSRkREhOQl2i1JTU1VUlJSwHP1zJkz2rJlS8g/V88G0Oeff64///nPio+Pt93SeWVlZenjjz8OeJ56vV7Nnz9f77//vu32mhW2L8fNmzdPWVlZGjp0qIYPH67Vq1fr8OHDmj59uu3WWpSdna1169bpD3/4g2JiYvx/IXo8HkVHR1vu7txiYmKavG/VpUsXxcfHh/T7WQ8//LBuvPFG5eXladKkSdq1a5dWr16t1atX226tRRMnTtTixYuVnJysvn37as+ePXr++ef14IMP2m6tiRMnTujAgQP+xxUVFSovL1dcXJySk5M1d+5c5eXlqU+fPurTp4/y8vLUuXNnTZkyxWLXLfft9Xp15513qqysTO+8844aGhr8z9W4uDhFRUXZavu8v+/vh2XHjh2VlJSka6655lK32jp2L867OC+++KJJSUkxUVFR5rrrrguLy5wlNbutWbPGdmttFg6XaBtjzB//+EfTr18/43a7TVpamlm9erXtls6rpqbGzJkzxyQnJ5tOnTqZ3r17m0WLFpna2lrbrTVRXFzc7L/pqVOnGmP+32XaTz75pElKSjJut9v86Ec/Mp988ondpk3LfVdUVJzzuVpcXByyfTcn1C/R5vuEAADWhOV7QgCA9oEQAgBYQwgBAKwhhAAA1hBCAABrCCEAgDWEEADAGkIIAGANIQQAsIYQAgBYQwgBAKz5/wDT2HoiyiiJyAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# The image for the first data sample\n",
    "show_data(train_dataset[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print out the first label, as we can see this digit is a five\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(5)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The label for the first data element\n",
    "train_dataset[0][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's plot the first sample of the rotated training dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAGxCAYAAADLfglZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAljElEQVR4nO3df3RU9Z3/8deQkElgw2iiSRglIbSsifwW0INAAYX0BMRVVymiEfV0C/LbWBpSdEW7JILW4oLAwm4RqyjWoynrqWBqwq8jSEhArWtBJEI0pVmFTgKYQJLP94/9MscxIT9wLp+Z8Hycc/+Yez/3fd/JYXjlM3PnMy5jjBEAABZ0st0AAODSRQgBAKwhhAAA1hBCAABrCCEAgDWEEADAGkIIAGANIQQAsIYQAgBYQwgBbVBZWalFixZp//79tlsJus8++0xut1u7du3y71u0aJFcLleTLTo6OuDcEydO6LLLLlNBQcFF7hodRaTtBoBwUFlZqSeeeEI9e/bUwIEDbbcTVD//+c81btw4DRs2rMmxzZs3y+Px+B936hT4d+vll1+uhx9+WPPnz9f48eMVFRXleL/oWAgh4BJjjFFtba1iYmL0ySefqKCgQJs3b2527ODBg3XFFVe0WG/69On6t3/7N73++uuaMmWKEy2jA+PlOISsHTt2yOVy6ZVXXmly7MUXX5TL5VJJSYnjfWzdulVDhw6VJD3wwAP+l6YWLVrkH7N3717deuutiouLU3R0tAYNGqTXXnstoM4LL7wgl8ul4uJiPfTQQ7riiisUHx+vO+64Q5WVlQFji4qKNHr0aMXHxysmJkbJycn653/+Z50+fdo/5vjx45oxY4auuuoqRUVFqVevXlq4cKHq6uoCarlcLs2aNUurV69Wenq63G631q9fL0latWqVkpKSNG7cuAv+/SQmJmrcuHFavXr1BdfAJcwAIWzQoEFm+PDhTfYPHTrUDB06tNXzz54926atsbHxvDV8Pp9Zt26dkWQeffRRs2vXLrNr1y5TUVFhjDGmqKjIREVFmZEjR5qNGzeazZs3m/vvv99IMuvWrfPXOVejV69eZvbs2WbLli3mP//zP83ll19uxowZ4x9XXl5uoqOjzbhx40xBQYHZunWrefnll01WVpY5ceKEMcaYb775xvTv39907drVPPPMM+add94xjz32mImMjDTjx48P6F+Sueqqq0z//v3Nhg0bTFFRkfnzn/9sjDGmV69eZtKkSU1+5scff9xIMklJSaZTp04mISHBZGVlmSNHjjT7O1qyZInp1KmTvz+grQghhLRz/3Hv27fPv2/Pnj1Gklm/fn2L55aXlxtJbdqKi4tbrFVSUtIkVM5JS0szgwYNMmfPng3Yf8stt5ju3bubhoaGgJ9lxowZAeOWLl1qJJm//vWvxhhjXn/9dSPJ7N+//7z9rF692kgyr732WsD+JUuWGEnmnXfe8e+TZDwejzl+/HjA2L/97W9Gknnqqaea1H/xxRfN4sWLzR//+EdTVFRknnrqKRMXF2cSExPNF1980WR8YWGhkWTefvvt8/YMNIf3hBDS7r77buXk5Oj555/X2rVrJUnLly/XlVdeqZ/85Cctnuv1etv8ct0111xzQf0dOnRIf/nLX/TMM89Ikurr6/3Hxo8fr7feeksHDhxQenq6f/+tt94aUKN///6SpCNHjigpKUkDBw5UVFSUfvazn2nGjBkaOXKkevXqFXBOUVGRunbtqjvvvDNg//3336+cnBy9++67AS+x3XTTTbr88ssDxp57CTAhIaHJz5WVlRXweMyYMRozZoyGDRumpUuX6rnnngs4fq7Gl19+2aQW0BJCCCHN7XZr2rRp+vWvf62nn35aZ8+e1Wuvvabs7Gy53e4Wz42KimrznWwREREX1N/f/vY3Sf93h9nPf/7zZsd89dVXAY/j4+MDHp/7Ob755htJ0g9+8AP96U9/0tKlSzVz5kydOnVKvXr10pw5czR37lxJ0tdff62kpCS5XK6AWgkJCYqMjNTXX38dsL979+5N+jp3ve/edn0+119/vf7xH/9Ru3fvbnLsXI1zNYG2IoQQ8h566CE99dRT+u1vf6va2lrV19dr+vTprZ73+eefKzU1tU3XKC4u1ujRo9vd27k7x3Jzc3XHHXc0O+ZCZlkjR47UyJEj1dDQoL1792r58uWaN2+eEhMTNXnyZMXHx+v999+XMSYgiKqqqlRfX9/kjrbvhtW3ez9+/Hib+zLGNLlN+9s1WruTDvguQgghr3v37rrrrru0cuVKnTlzRhMnTlRycnKr5wXz5bjvzla+fV7v3r31wQcfKC8vr03Xao+IiAjdcMMNSktL08svv6yysjJNnjxZN998s1577TUVFBTo9ttv949/8cUXJUk333xzq7VTUlIUExOjzz77rE297N69W59++qnmzJnT5Njhw4clSddee22bagHnEEIIC3PnztUNN9wgSVq3bl2bzomKitKQIUOCcv0f/OAHiomJ0csvv6z09HT9wz/8g7xer7xer/7jP/5DmZmZ+vGPf6z7779fV111lY4fP65PPvlEZWVl+v3vf9+ua61evVpFRUWaMGGCkpOTVVtbq9/+9reSpLFjx0qS7rvvPj3//POaOnWqPv/8c/Xr1087d+5UXl6exo8f7x/XkqioKA0bNqzZl9cGDBige++9V+np6YqOjtaePXv09NNPKykpSb/4xS+ajN+9e7fi4+PVr1+/dv2sAHfHIWz07NnTpKenW7v+K6+8YtLS0kznzp2NJPP444/7j33wwQdm0qRJJiEhwXTu3NkkJSWZm266yaxevdo/5tzdcSUlJQF1i4uLA+7Q27Vrl7n99ttNSkqKcbvdJj4+3owaNcps2rQp4Lyvv/7aTJ8+3XTv3t1ERkaalJQUk5uba2prawPGSTIzZ85s9mf6r//6LxMREWEqKysD9k+ePNn88Ic/NF27djWdO3c2KSkpZvr06U3GGWNMY2OjSUlJMbNnz271dwh8l8sYY+zGINC6Dz/8UAMGDNDzzz+vGTNm2G6nw6itrVVycrIeeeQR5eTkXFCNd999VxkZGfr444+VlpYW5A7R0RFCCGmfffaZjhw5ol/+8pc6evSoDh06pC5duthuq0NZtWqVFi1apMOHD6tr167tPn/MmDH64Q9/6L+FHmgP3hNCSPvVr36l3/3ud0pPT9fvf/97AsgBP/vZz/T3v/9dhw8fbvd7OidOnNCoUaOYneKCMRMCAFjDAqYAAGsIIQCANYQQAMCakLsxobGxUZWVlYqNjW12qREAQGgzxqimpkZer7fZZZ6+LeRCqLKyUj169LDdBgDge6qoqNDVV1/d4piQC6HY2FjbLQCOa+7rE4KlqqrKsdpAe7Tl//OQCyFegsOloLWXKICOoC3/n/NMAABYQwgBAKwhhAAA1hBCAABrHAuhlStXKjU1VdHR0Ro8eLB27Njh1KUAAGHKkRDauHGj5s2bp4ULF2rfvn0aOXKkMjMzdfToUScuBwAIU46son3DDTfouuuu06pVq/z70tPTddtttyk/P7/Fc6urq+XxeILdEhBSkpKSHKt97Ngxx2oD7eHz+dStW7cWxwR9JnTmzBmVlpYqIyMjYH9GRobee++9JuPr6upUXV0dsAEALg1BD6GvvvpKDQ0NSkxMDNifmJjY7F9o+fn58ng8/o0lewDg0uHYjQnf/aSsMabZT8/m5ubK5/P5t4qKCqdaAgCEmKAv23PFFVcoIiKiyaynqqqqyexIktxut9xud7DbAACEgaDPhKKiojR48GAVFhYG7C8sLNSNN94Y7MsBAMKYIwuYZmdnKysrS0OGDNGwYcO0Zs0aHT16VNOnT3ficgCAMOVICP3kJz/R119/rSeffFJ//etf1bdvX/3xj39USkqKE5cDAIQpRz4n9H3wOSFcCvicEC4FVj4nBABAWxFCAABrCCEAgDWEEADAGkfujgMuFq/X61jthIQEx2r37NnTsdoFBQWO1QaCjZkQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWRNpuAB3f1Vdf7VjtxsZGx2rPnz/fsdq/+c1vHKsdGxvrWO2amhrHauPSxEwIAGANIQQAsIYQAgBYQwgBAKwhhAAA1hBCAABrCCEAgDWEEADAmqCHUH5+voYOHarY2FglJCTotttu04EDB4J9GQBABxD0ENq2bZtmzpyp3bt3q7CwUPX19crIyNCpU6eCfSkAQJgL+rI9mzdvDni8bt06JSQkqLS0VD/60Y+ajK+rq1NdXZ3/cXV1dbBbAgCEKMffE/L5fJKkuLi4Zo/n5+fL4/H4tx49ejjdEgAgRDgaQsYYZWdna8SIEerbt2+zY3Jzc+Xz+fxbRUWFky0BAEKIo6toz5o1Sx9++KF27tx53jFut1tut9vJNgAAIcqxEJo9e7Y2bdqk7du3O7qUPwAgfAU9hIwxmj17tt58801t3bpVqampwb4EAKCDCHoIzZw5Uxs2bNAf/vAHxcbG6tixY5Ikj8ejmJiYYF8OABDGgn5jwqpVq+Tz+TR69Gh1797dv23cuDHYlwIAhDlHXo4DAKAtWDsOAGANIQQAsIYQAgBY4+iHVREerrzySkfrf3ttwGBbu3atY7Wfe+45x2rHxsY6Vrumpsax2kCwMRMCAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMCaSNsNoG06dXLu74W///3vjtWWpKKiIsdqDxkyxLHa77//vmO18/PzHasNhBNmQgAAawghAIA1hBAAwBpCCABgDSEEALCGEAIAWEMIAQCsIYQAANY4HkL5+flyuVyaN2+e05cCAIQZR0OopKREa9asUf/+/Z28DAAgTDkWQidPntQ999yjtWvX6vLLL3fqMgCAMOZYCM2cOVMTJkzQ2LFjWxxXV1en6urqgA0AcGlwZAHTV199VWVlZSopKWl1bH5+vp544gkn2gAAhLigz4QqKio0d+5cvfTSS4qOjm51fG5urnw+n3+rqKgIdksAgBAV9JlQaWmpqqqqNHjwYP++hoYGbd++XStWrFBdXZ0iIiL8x9xut9xud7DbAACEgaCH0M0336yPPvooYN8DDzygtLQ05eTkBAQQAODSFvQQio2NVd++fQP2de3aVfHx8U32AwAubayYAACw5qJ8vffWrVsvxmUAAGGGmRAAwBpCCABgDSEEALCGEAIAWHNRbkzA99fY2OhY7Weeecax2pJ0zTXXOFrfKQcPHrTdAtDhMRMCAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMCaSNsNoG1++tOfOlb7oYcecqy2JLndbsdqjx492rHa27dvd6x2RkaGY7W3bt3qWO3ISOf+y6itrXWsttMaGxtttxC2mAkBAKwhhAAA1hBCAABrCCEAgDWEEADAGkIIAGANIQQAsMaREPryyy917733Kj4+Xl26dNHAgQNVWlrqxKUAAGEs6J88O3HihIYPH64xY8bo7bffVkJCgj777DNddtllwb4UACDMBT2ElixZoh49emjdunX+fT179gz2ZQAAHUDQX47btGmThgwZorvuuksJCQkaNGiQ1q5de97xdXV1qq6uDtgAAJeGoIfQ4cOHtWrVKvXu3VtbtmzR9OnTNWfOHL344ovNjs/Pz5fH4/FvPXr0CHZLAIAQFfQQamxs1HXXXae8vDwNGjRI06ZN07/8y79o1apVzY7Pzc2Vz+fzbxUVFcFuCQAQooIeQt27d9e1114bsC89PV1Hjx5tdrzb7Va3bt0CNgDApSHoITR8+HAdOHAgYN/BgweVkpIS7EsBAMJc0EPo4Ycf1u7du5WXl6dDhw5pw4YNWrNmjWbOnBnsSwEAwlzQQ2jo0KF688039corr6hv37761a9+pWXLlumee+4J9qUAAGHOka9JvOWWW3TLLbc4URoA0IGwdhwAwBpCCABgDSEEALCGEAIAWOMyxhjbTXxbdXW1PB6P7TYuyJVXXulY7bKyMsdq/+///q9jtSWpvr7esdrHjx93rPbhw4cdq11QUOBY7WXLljlWOyEhwbHaM2bMcKz266+/7lht6f9WikFTPp+v1QUImAkBAKwhhAAA1hBCAABrCCEAgDWEEADAGkIIAGANIQQAsIYQAgBYQwgBAKwhhAAA1hBCAABrCCEAgDWEEADAGkIIAGANIQQAsIYQAgBYQwgBAKwhhAAA1hBCAABrCCEAgDWEEADAGkIIAGCNyxhjbDfxbdXV1fJ4PI7Vd7lcjtX+6U9/6ljtHTt2OFb79ttvd6y2JNXW1jpWe+jQoY7VdlJ0dLRjtQ8ePOhYbSfl5OQ4Vrt3796O1ZakQ4cOOVo/XPl8PnXr1q3FMcyEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBACwhhACAFgT9BCqr6/Xo48+qtTUVMXExKhXr1568skn1djYGOxLAQDCXGSwCy5ZskSrV6/W+vXr1adPH+3du1cPPPCAPB6P5s6dG+zLAQDCWNBDaNeuXfqnf/onTZgwQZLUs2dPvfLKK9q7d2+wLwUACHNBfzluxIgRevfdd/1Lh3zwwQfauXOnxo8f3+z4uro6VVdXB2wAgEtD0GdCOTk58vl8SktLU0REhBoaGrR48WLdfffdzY7Pz8/XE088Eew2AABhIOgzoY0bN+qll17Shg0bVFZWpvXr1+uZZ57R+vXrmx2fm5srn8/n3yoqKoLdEgAgRAV9JjR//nwtWLBAkydPliT169dPR44cUX5+vqZOndpkvNvtltvtDnYbAIAwEPSZ0OnTp9WpU2DZiIgIbtEGADQR9JnQxIkTtXjxYiUnJ6tPnz7at2+fnn32WT344IPBvhQAIMwFPYSWL1+uxx57TDNmzFBVVZW8Xq+mTZumf/3Xfw32pQAAYS7oIRQbG6tly5Zp2bJlwS4NAOhgWDsOAGANIQQAsIYQAgBYQwgBAKxxGWOM7Sa+rbq6Wh6Px3YbISctLc2x2n/5y18cq+00r9frWO2rr77asdolJSWO1e7evbtjtVetWuVY7euvv96x2m+++aZjtSVpxowZjtYPVz6fT926dWtxDDMhAIA1hBAAwBpCCABgDSEEALCGEAIAWEMIAQCsIYQAANYQQgAAawghAIA1hBAAwBpCCABgDSEEALCGEAIAWEMIAQCsIYQAANYQQgAAawghAIA1hBAAwBpCCABgDSEEALCGEAIAWEMIAQCscRljjO0mvq26uloej8d2G4CjXC6XY7W7d+/uWO0DBw44VvvOO+90rPZ7773nWG1JqqmpcbR+uPL5fOrWrVuLY5gJAQCsIYQAANYQQgAAawghAIA1hBAAwBpCCABgDSEEALCm3SG0fft2TZw4UV6vVy6XSwUFBQHHjTFatGiRvF6vYmJiNHr0aH388cfB6hcA0IG0O4ROnTqlAQMGaMWKFc0eX7p0qZ599lmtWLFCJSUlSkpK0rhx4/gwFwCgicj2npCZmanMzMxmjxljtGzZMi1cuFB33HGHJGn9+vVKTEzUhg0bNG3atO/XLQCgQwnqe0Ll5eU6duyYMjIy/PvcbrdGjRp13mUz6urqVF1dHbABAC4NQQ2hY8eOSZISExMD9icmJvqPfVd+fr48Ho9/69GjRzBbAgCEMEfujvvu4ozGmPMu2Jibmyufz+ffKioqnGgJABCC2v2eUEuSkpIk/d+M6Nsr+VZVVTWZHZ3jdrvldruD2QYAIEwEdSaUmpqqpKQkFRYW+vedOXNG27Zt04033hjMSwEAOoB2z4ROnjypQ4cO+R+Xl5dr//79iouLU3JysubNm6e8vDz17t1bvXv3Vl5enrp06aIpU6YEtXEAQPhrdwjt3btXY8aM8T/Ozs6WJE2dOlUvvPCCfvGLX+ibb77RjBkzdOLECd1www165513FBsbG7yuAQAdAt+sCljAN6s2xTerdjx8syoAIKQRQgAAawghAIA1hBAAwJqgflgVQNs4eWPC7373O8dq19bWOlZ76tSpjtXesmWLY7Xx/TATAgBYQwgBAKwhhAAA1hBCAABrCCEAgDWEEADAGkIIAGANIQQAsIYQAgBYQwgBAKwhhAAA1hBCAABrCCEAgDWEEADAGkIIAGANIQQAsIYQAgBYQwgBAKwhhAAA1hBCAABrCCEAgDWEEADAmkjbDQCXom7dujlW+7rrrnOsdn19vWO1X331VcdqI3QxEwIAWEMIAQCsIYQAANYQQgAAawghAIA1hBAAwBpCCABgDSEEALCm3SG0fft2TZw4UV6vVy6XSwUFBf5jZ8+eVU5Ojvr166euXbvK6/XqvvvuU2VlZTB7BgB0EO0OoVOnTmnAgAFasWJFk2OnT59WWVmZHnvsMZWVlemNN97QwYMHdeuttwalWQBAx9LuZXsyMzOVmZnZ7DGPx6PCwsKAfcuXL9f111+vo0ePKjk5uck5dXV1qqur8z+urq5ub0sAgDDl+HtCPp9PLpdLl112WbPH8/Pz5fF4/FuPHj2cbgkAECIcDaHa2lotWLBAU6ZMOe+Cjbm5ufL5fP6toqLCyZYAACHEsVW0z549q8mTJ6uxsVErV6487zi32y232+1UGwCAEOZICJ09e1aTJk1SeXm5ioqKHF22HgAQvoIeQucC6NNPP1VxcbHi4+ODfQkAQAfR7hA6efKkDh065H9cXl6u/fv3Ky4uTl6vV3feeafKysr01ltvqaGhQceOHZMkxcXFKSoqKnidAwDCXrtDaO/evRozZoz/cXZ2tiRp6tSpWrRokTZt2iRJGjhwYMB5xcXFGj169IV3CgDocNodQqNHj5Yx5rzHWzoGAMC3sXYcAMAaQggAYA0hBACwhhACAFjjMiF2J0F1dbU8Ho/tNgC5XC7Haq9Zs8ax2k5KTU11rPbYsWMdqw07fD5fq4sVMBMCAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMCaSNsNAKEqPj7esdqbN292rPbKlSsdq33TTTc5Vjs2Ntax2jU1NY7VxvfDTAgAYA0hBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANe0Ooe3bt2vixInyer1yuVwqKCg479hp06bJ5XJp2bJl36NFAEBH1e4QOnXqlAYMGKAVK1a0OK6goEDvv/++vF7vBTcHAOjY2r1iQmZmpjIzM1sc8+WXX2rWrFnasmWLJkyYcMHNAQA6tqAv29PY2KisrCzNnz9fffr0aXV8XV2d6urq/I+rq6uD3RIAIEQF/caEJUuWKDIyUnPmzGnT+Pz8fHk8Hv/Wo0ePYLcEAAhRQQ2h0tJSPffcc3rhhRfkcrnadE5ubq58Pp9/q6ioCGZLAIAQFtQQ2rFjh6qqqpScnKzIyEhFRkbqyJEjeuSRR9SzZ89mz3G73erWrVvABgC4NAT1PaGsrCyNHTs2YN+Pf/xjZWVl6YEHHgjmpQAAHUC7Q+jkyZM6dOiQ/3F5ebn279+vuLg4JScnN/kOls6dOyspKUnXXHPN9+8WANChtDuE9u7dqzFjxvgfZ2dnS5KmTp2qF154IWiNAQA6vnaH0OjRo2WMafP4zz//vL2XAABcIlg7DgBgDSEEALCGEAIAWEMIAQCsCfraccDFFBER4VjtEydOOFb7oYcecqz2lClTHKvtpJqaGtstwAJmQgAAawghAIA1hBAAwBpCCABgDSEEALCGEAIAWEMIAQCsIYQAANYQQgAAawghAIA1hBAAwBpCCABgDSEEALCGEAIAWEMIAQCsIYQAANYQQgAAawghAIA1hBAAwBpCCABgDSEEALAm0nYD32WMsd0Cwki4/ns5deqUY7Xr6+sdq93Q0OBYbXQ8bXl+ukyIPYu/+OIL9ejRw3YbAIDvqaKiQldffXWLY0IuhBobG1VZWanY2Fi5XK5Wx1dXV6tHjx6qqKhQt27dLkKHwUHfF1+49k7fFxd9f3/GGNXU1Mjr9apTp5bf9Qm5l+M6derUanI2p1u3btZ/8ReCvi++cO2dvi8u+v5+PB5Pm8ZxYwIAwBpCCABgTdiHkNvt1uOPPy632227lXah74svXHun74uLvi+ukLsxAQBw6Qj7mRAAIHwRQgAAawghAIA1hBAAwBpCCABgTViH0MqVK5Wamqro6GgNHjxYO3bssN1Sq/Lz8zV06FDFxsYqISFBt912mw4cOGC7rXbLz8+Xy+XSvHnzbLfSqi+//FL33nuv4uPj1aVLFw0cOFClpaW222pRfX29Hn30UaWmpiomJka9evXSk08+qcbGRtutNbF9+3ZNnDhRXq9XLpdLBQUFAceNMVq0aJG8Xq9iYmI0evRoffzxx3aa/ZaW+j579qxycnLUr18/de3aVV6vV/fdd58qKyvtNfz/tfb7/rZp06bJ5XJp2bJlF62/9grbENq4caPmzZunhQsXat++fRo5cqQyMzN19OhR2621aNu2bZo5c6Z2796twsJC1dfXKyMjw9FVlYOtpKREa9asUf/+/W230qoTJ05o+PDh6ty5s95++239z//8j37961/rsssus91ai5YsWaLVq1drxYoV+uSTT7R06VI9/fTTWr58ue3Wmjh16pQGDBigFStWNHt86dKlevbZZ7VixQqVlJQoKSlJ48aNU01NzUXuNFBLfZ8+fVplZWV67LHHVFZWpjfeeEMHDx7UrbfeaqHTQK39vs8pKCjQ+++/L6/Xe5E6u0AmTF1//fVm+vTpAfvS0tLMggULLHV0Yaqqqowks23bNtuttElNTY3p3bu3KSwsNKNGjTJz58613VKLcnJyzIgRI2y30W4TJkwwDz74YMC+O+64w9x7772WOmobSebNN9/0P25sbDRJSUnmqaee8u+rra01Ho/HrF692kKHzftu383Zs2ePkWSOHDlycZpqg/P1/cUXX5irrrrK/PnPfzYpKSnmN7/5zUXvra3CciZ05swZlZaWKiMjI2B/RkaG3nvvPUtdXRifzydJiouLs9xJ28ycOVMTJkzQ2LFjbbfSJps2bdKQIUN01113KSEhQYMGDdLatWttt9WqESNG6N1339XBgwclSR988IF27typ8ePHW+6sfcrLy3Xs2LGA56rb7daoUaPC8rnqcrlCfhbd2NiorKwszZ8/X3369LHdTqtCbhXttvjqq6/U0NCgxMTEgP2JiYk6duyYpa7azxij7OxsjRgxQn379rXdTqteffVVlZWVqaSkxHYrbXb48GGtWrVK2dnZ+uUvf6k9e/Zozpw5crvduu+++2y3d145OTny+XxKS0tTRESEGhoatHjxYt199922W2uXc8/H5p6rR44csdHSBamtrdWCBQs0ZcqUkFihuiVLlixRZGSk5syZY7uVNgnLEDrnu983ZIxp03cQhYpZs2bpww8/1M6dO2230qqKigrNnTtX77zzjqKjo22302aNjY0aMmSI8vLyJEmDBg3Sxx9/rFWrVoV0CG3cuFEvvfSSNmzYoD59+mj//v2aN2+evF6vpk6daru9dgvn5+rZs2c1efJkNTY2auXKlbbbaVFpaamee+45lZWVhc3vNyxfjrviiisUERHRZNZTVVXV5C+uUDV79mxt2rRJxcXFF/T9SRdbaWmpqqqqNHjwYEVGRioyMlLbtm3Tv//7vysyMjJkv/a5e/fuuvbaawP2paenh/wNLPPnz9eCBQs0efJk9evXT1lZWXr44YeVn59vu7V2SUpKkqSwfa6ePXtWkyZNUnl5uQoLC0N+FrRjxw5VVVUpOTnZ/zw9cuSIHnnkEfXs2dN2e80KyxCKiorS4MGDVVhYGLC/sLBQN954o6Wu2sYYo1mzZumNN95QUVGRUlNTbbfUJjfffLM++ugj7d+/378NGTJE99xzj/bv36+IiAjbLTZr+PDhTW6BP3jwoFJSUix11DanT59u8o2UERERIXmLdktSU1OVlJQU8Fw9c+aMtm3bFvLP1XMB9Omnn+pPf/qT4uPjbbfUqqysLH344YcBz1Ov16v58+dry5YttttrVti+HJedna2srCwNGTJEw4YN05o1a3T06FFNnz7ddmstmjlzpjZs2KA//OEPio2N9f+F6PF4FBMTY7m784uNjW3yvlXXrl0VHx8f0u9nPfzww7rxxhuVl5enSZMmac+ePVqzZo3WrFlju7UWTZw4UYsXL1ZycrL69Omjffv26dlnn9WDDz5ou7UmTp48qUOHDvkfl5eXa//+/YqLi1NycrLmzZunvLw89e7dW71791ZeXp66dOmiKVOmWOy65b69Xq/uvPNOlZWV6a233lJDQ4P/uRoXF6eoqChbbbf6+/5uWHbu3FlJSUm65pprLnarbWP35rzv5/nnnzcpKSkmKirKXHfddWFxm7OkZrd169bZbq3dwuEWbWOM+e///m/Tt29f43a7TVpamlmzZo3tllpVXV1t5s6da5KTk010dLTp1auXWbhwoamrq7PdWhPFxcXN/pueOnWqMeb/btN+/PHHTVJSknG73eZHP/qR+eijj+w2bVruu7y8/LzP1eLi4pDtuzmhfos23ycEALAmLN8TAgB0DIQQAMAaQggAYA0hBACwhhACAFhDCAEArCGEAADWEEIAAGsIIQCANYQQAMAaQggAYM3/A7K4isIJu/m2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_data(train_dataset_rotate[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that this is the same sample but it is rotated as we wanted\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"CNN\">Build a Convolutional Neural Network Class</h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build a Convolutional Network class with two Convolutional layers and one fully connected layer. Pre-determine the size of the final output matrix. The parameters in the constructor are the number of output channels for the first and second layers.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CNN(nn.Module):\n",
    "    \n",
    "    # Contructor\n",
    "    def __init__(self, out_1=16, out_2=32):\n",
    "        super(CNN, self).__init__()\n",
    "        \n",
    "        # The reason we start with 1 channel is because we have a single black and white image\n",
    "        # Channel Width after this layer is 16\n",
    "        self.cnn1 = nn.Conv2d(in_channels=1, out_channels=out_1, kernel_size=5, padding=2)\n",
    "        # Channel Wifth after this layer is 8\n",
    "        self.maxpool1=nn.MaxPool2d(kernel_size=2)\n",
    "        \n",
    "        # Channel Width after this layer is 8\n",
    "        self.cnn2 = nn.Conv2d(in_channels=out_1, out_channels=out_2, kernel_size=5, stride=1, padding=2)\n",
    "        # Channel Width after this layer is 4\n",
    "        self.maxpool2=nn.MaxPool2d(kernel_size=2)\n",
    "        # In total we have out_2 (32) channels which are each 4 * 4 in size based on the width calculation above. Channels are squares.\n",
    "        # The output is a value for each class\n",
    "        self.fc1 = nn.Linear(out_2 * 4 * 4, 10)\n",
    "    \n",
    "    # Prediction\n",
    "    def forward(self, x):\n",
    "        # Puts the X value through each cnn, relu, and pooling layer and it is flattened for input into the fully connected layer\n",
    "        x = self.cnn1(x)\n",
    "        x = torch.relu(x)\n",
    "        x = self.maxpool1(x)\n",
    "        x = self.cnn2(x)\n",
    "        x = torch.relu(x)\n",
    "        x = self.maxpool2(x)\n",
    "        x = x.view(x.size(0), -1)\n",
    "        x = self.fc1(x)\n",
    "        return x\n",
    "\n",
    "    # Outputs result of each stage of the CNN, relu, and pooling layers\n",
    "    def activations(self, x):\n",
    "        # Outputs activation this is not necessary\n",
    "        z1 = self.cnn1(x)\n",
    "        a1 = torch.relu(z1)\n",
    "        out = self.maxpool1(a1)\n",
    "        \n",
    "        z2 = self.cnn2(out)\n",
    "        a2 = torch.relu(z2)\n",
    "        out1 = self.maxpool2(a2)\n",
    "        out = out.view(out.size(0),-1)\n",
    "        return z1, a1, z2, a2, out1,out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"regular_data\">Regular Data</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the Convolutional Neural Network Classifier, Criterion function, Optimizer, and Train the Model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the model object to be trained on regular data using CNN class\n",
    "model = CNN(out_1=16, out_2=32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the loss function, the optimizer, and the dataset loader \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We create a criterion which will measure loss\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "learning_rate = 0.1\n",
    "# Create an optimizer that updates model parameters using the learning rate and gradient\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate)\n",
    "# Create a Data Loader for the training data with a batch size of 100 \n",
    "train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=100)\n",
    "# Create a Data Loader for the rotated validation data with a batch size of 5000 \n",
    "validation_loader = torch.utils.data.DataLoader(dataset=validation_dataset, batch_size=5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This cell will train the model, we will comment it out as it takes a long time to run. You can change the block type from Raw to Code and run it or you can load the trained model in the next cell. Notice that we are not only training and saving the model here but we are also keeping track of important data like the cost and accuracy throughout the training process.\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# Train the model\n",
    "import os  \n",
    "\n",
    "# Location to save data\n",
    "file_normal = os.path.join(os.getcwd(), 'normal.pt')\n",
    "\n",
    "# All the data we are saving\n",
    "checkpoint={\n",
    "            # Saving the number of epochs the models was trained for\n",
    "            'epoch': None,\n",
    "            # Saving the models parameters which will allow us to recreate the trained model\n",
    "            'model_state_dict': None,\n",
    "            # Saving the optimizers parameters\n",
    "            'optimizer_state_dict': None,\n",
    "            # Saving the loss on the training dataset for the last batch of the last epoch\n",
    "            'loss': None,\n",
    "            # Saving the cost on the training dataset for each epoch\n",
    "            'cost': [],\n",
    "            # Saving the accuracy for the testing dataset for each epoch\n",
    "            'accuracy': []}\n",
    "            \n",
    "# Number of epochs to train model\n",
    "n_epochs = 5\n",
    "\n",
    "# Size of the testing dataset\n",
    "N_test = len(validation_dataset)\n",
    "\n",
    "# Training for the number of epochs we want\n",
    "for epoch in range(n_epochs):\n",
    "    # Variable to keep track of cost for each epoch\n",
    "    cost = 0\n",
    "    # For each batch in the training dataset\n",
    "    for x, y in train_loader:\n",
    "        # Resets the calculated gradient value, this must be done each time as it accumulates if we do not reset\n",
    "        optimizer.zero_grad()\n",
    "        # Makes a prediction on the image\n",
    "        z = model(x)\n",
    "        # Calculate the loss between the prediction and actual class\n",
    "        loss = criterion(z, y)\n",
    "        # Calculates the gradient value with respect to each weight and bias\n",
    "        loss.backward()\n",
    "        # Updates the weight and bias according to calculated gradient value\n",
    "        optimizer.step()\n",
    "      \n",
    "        # Saves the number of epochs we trained for  \n",
    "        checkpoint['epochs'] = n_epochs\n",
    "        # Saves the models parameters\n",
    "        checkpoint['model_state_dict'] = model.state_dict()\n",
    "        # Saves the optimizers paramters\n",
    "        checkpoint['optimizer_state_dict'] = optimizer.state_dict()\n",
    "        # Saves the loss for the last batch so ultimately this will be the loss for the last batch of the last epoch\n",
    "        checkpoint['loss'] = loss\n",
    "        # Accumulates the loss\n",
    "        cost += loss.item()\n",
    "        \n",
    "     \n",
    "    # Counter for the correct number of predictions        \n",
    "    correct = 0\n",
    "        \n",
    "    # For each batch in the validation dataset\n",
    "    for x_test, y_test in validation_loader:\n",
    "        # Make a prediction\n",
    "        z = model(x_test)\n",
    "        # Get the class that has the maximum value\n",
    "        _, yhat = torch.max(z.data, 1)\n",
    "        # Counts the number of correct predictions made\n",
    "        correct += (yhat == y_test).sum().item()\n",
    " \n",
    "    accuracy = correct / N_test\n",
    "    print(accuracy)\n",
    "    # Appends the cost of the epoch to a list\n",
    "    checkpoint['cost'].append(cost) \n",
    "    # Appends the accuracy of the epoch to a list\n",
    "    checkpoint['accuracy'].append(accuracy)\n",
    "    # Saves the data in checkpoint to the file location\n",
    "    torch.save(checkpoint, file_normal) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 id=\"Result\">Analyze Results</h3> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loads the data which is saved in `normal.pt`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint_normal = torch.load(os.path.join(os.getcwd(),'normal.pt'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot Accuracy and Cost vs Epoch Graph\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using the helper function defined at the top and the cost and accuracy lists that we saved\n",
    "plot_cost_accuracy(checkpoint_normal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Five misclassified samples\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using the model parameters we saved we load them into a model to recreate the trained model\n",
    "model.load_state_dict(checkpoint_normal['model_state_dict'])\n",
    "# Setting the model to evaluation mode\n",
    "model.eval()\n",
    "# Using the helper function plot the first five misclassified samples\n",
    "plot_mis_classified(model,validation_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"R_training_data\">Rotated Training Data</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the loss and accuracy on the validation data:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the model object using CNN class\n",
    "model_r = CNN(out_1=16, out_2=32)\n",
    "# We create a criterion which will measure loss\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "learning_rate = 0.1\n",
    "# Create an optimizer that updates model parameters using the learning rate and gradient\n",
    "optimizer = torch.optim.SGD(model_r.parameters(), lr = learning_rate)\n",
    "# Create a Data Loader for the rotated training data with a batch size of 100 \n",
    "train_loader = torch.utils.data.DataLoader(dataset=train_dataset_rotate, batch_size=100)\n",
    "# Create a Data Loader for the rotated validation data with a batch size of 5000 \n",
    "validation_loader = torch.utils.data.DataLoader(dataset=validation_dataset, batch_size=5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This cell will train the model, we will comment it out as it takes a long time to run. You can change the block type from Raw to Code and run it or you can load the trained model in the next cell.\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# Location to save data\n",
    "file_rotated = os.path.join(os.getcwd(), 'rotated_data.pt')\n",
    "\n",
    "# All the data we are saving\n",
    "checkpoint={\n",
    "            # Saving the number of epochs the models was trained for\n",
    "            'epoch': None,\n",
    "            # Saving the models parameters which will allow us to recreate the trained model\n",
    "            'model_state_dict': None,\n",
    "            # Saving the optimizers parameters\n",
    "            'optimizer_state_dict': None,\n",
    "            # Saving the loss on the training dataset for the last batch of the last epoch\n",
    "            'loss': None,\n",
    "            # Saving the cost on the training dataset for each epoch\n",
    "            'cost': [],\n",
    "            # Saving the accuracy for the testing dataset for each epoch\n",
    "            'accuracy': []}\n",
    "            \n",
    "# Number of epochs to train model            \n",
    "n_epochs = 5\n",
    "\n",
    "# Size of the testing dataset\n",
    "N_test = len(validation_dataset)\n",
    "\n",
    "# Training for the number of epochs we want\n",
    "for epoch in range(n_epochs):\n",
    "    # Variable to keep track of cost for each epoch\n",
    "    cost = 0\n",
    "    # For each batch in the training dataset\n",
    "    for x, y in train_loader:\n",
    "        # Resets the calculated gradient value, this must be done each time as it accumulates if we do not reset\n",
    "        optimizer.zero_grad()\n",
    "        # Makes a prediction on the image\n",
    "        z = model_r(x)\n",
    "        # Calculate the loss between the prediction and actual class\n",
    "        loss = criterion(z, y)\n",
    "        # Calculates the gradient value with respect to each weight and bias\n",
    "        loss.backward()\n",
    "        # Updates the weight and bias according to calculated gradient value\n",
    "        optimizer.step()\n",
    "      \n",
    "        # Saves the number of epochs we trained for  \n",
    "        checkpoint['epochs'] = n_epochs\n",
    "        # Saves the models parameters\n",
    "        checkpoint['model_state_dict'] = model.state_dict()\n",
    "        # Saves the optimizers paramters\n",
    "        checkpoint['optimizer_state_dict'] = optimizer.state_dict()\n",
    "        # Saves the loss for the last batch so ultimately this will be the loss for the last batch of the last epoch\n",
    "        checkpoint['loss'] = loss\n",
    "        # Accumulates the loss\n",
    "        cost+=loss.item()\n",
    "        \n",
    "     \n",
    "    # Counter for the correct number of predictions        \n",
    "    correct = 0\n",
    "        \n",
    "    # For each batch in the validation dataset\n",
    "    for x_test, y_test in validation_loader:\n",
    "        # Make a prediction\n",
    "        z = model_r(x_test)\n",
    "        # Get the class that has the maximum value\n",
    "        _, yhat = torch.max(z.data, 1)\n",
    "        # Counts the number of correct predictions made\n",
    "        correct += (yhat == y_test).sum().item()\n",
    " \n",
    "    accuracy = correct / N_test\n",
    "    print(accuracy)\n",
    "    # Appends the cost of the epoch to a list\n",
    "    checkpoint['cost'].append(cost) \n",
    "    # Appends the accuracy of the epoch to a list\n",
    "    checkpoint['accuracy'].append(accuracy)\n",
    "    # Saves the data in checkpoint to the file location\n",
    "    torch.save(checkpoint, file_rotated) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 id=\"Result\">Analyze Results</h3> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loads the data which is saved in `rotated_data.pt`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint_rotated= torch.load(os.path.join(os.getcwd(),'rotated_data.pt'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot Accuracy and Cost vs Epoch Graph\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using the helper function defined at the top and the cost and accuracy lists that we saved\n",
    "plot_cost_accuracy(checkpoint_rotated)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Five misclassified samples\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using the model parameters we saved we load them into a model to recreate the trained model\n",
    "model_r.load_state_dict(checkpoint_rotated['model_state_dict'])\n",
    "# Setting the model to evaluation mode\n",
    "model.eval()\n",
    "# Using the helper function plot the first five misclassified samples\n",
    "plot_mis_classified(model_r,validation_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see in the two Cost and Accuracy vs Epoch Graph the model trained on rotated data performs much better on the rotated validation data, 79% vs 95%+. By creating and training on rotated data allows the model to learn from unique data and have increased exposure to real life situations because data is not always perfect or rotated correctly.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--Empty Space for separating topics-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"http://cocl.us/pytorch_link_bottom\">\n",
    "    <img src=\"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DL0110EN/notebook_images%20/notebook_bottom%20.png\" width=\"750\" alt=\"PyTorch Bottom\">\n",
    "</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>About the Authors:</h2> \n",
    "\n",
    "<a href=\"https://www.linkedin.com/in/joseph-s-50398b136/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkCV0101ENCoursera872-2023-01-01\">Joseph Santarcangelo</a> has a PhD in Electrical Engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other contributors: <a href=\"https://www.linkedin.com/in/michelleccarey/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkCV0101ENCoursera872-2023-01-01\">Michelle Carey</a>, <a href=\"www.linkedin.com/in/jiahui-mavis-zhou-a4537814a\">Mavis Zhou</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thanks to Magnus <a href=\"http://www.hvass-labs.org/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkCV0101ENCoursera872-2023-01-01\">Erik Hvass Pedersen</a> whose tutorials helped me understand Convolutional Neural Networks\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright &copy; 2018 <a href=\"cognitiveclass.ai?utm_source=bducopyrightlink&utm_medium=dswb&utm_campaign=bdu\">cognitiveclass.ai</a>. This notebook and its source code are released under the terms of the <a href=\"https://bigdatauniversity.com/mit-license/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkCV0101ENCoursera872-2023-01-01\">MIT License</a>.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}