2_mnist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression for the MNIST dataset\n",
    "\n",
    "In this notebook we will continue using Pytorch. Now, we will create a shallow model (the logistic regression), plus the full training pipeline in order to train and evaluate this model over the MNIST dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from torchvision import datasets, transforms\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will define some hyperparameters and config options"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Set the random seed, so the experiment is reproducible\n",
    "seed = 0\n",
    "torch.manual_seed(seed)\n",
    "# For the moment, we will just train on CPU, so no cuda\n",
    "use_cuda = False\n",
    "device = torch.device(\"cuda\" if use_cuda else \"cpu\")\n",
    "# We use a batch size of 32 examples while training, and 1000 while testing\n",
    "batch_size = 32\n",
    "test_batch_size = 1000\n",
    "# We will use SGD with a momentum term\n",
    "momentum = 0.5\n",
    "# The learning rate\n",
    "lr = 0.01\n",
    "# The number of epochs\n",
    "epochs = 3\n",
    "# The size of the input. MNIST are greyscale images, 28x28 pixels each\n",
    "im_size = 28*28"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are ready to define our pytorch model. This is done by creating a class that extends nn.Module, with two methods:\n",
    "\n",
    "* __init__ : used to define and initialize the parameters of the model (for the LR, this is just a weight matrix of size 28*28 x 10, since we want to project to a 10-dimensional space to predict the digit. We could have define this explicitly, but with pytorch we have nn.Linear which is a shorthand for a linear projection.\n",
    "\n",
    "* forward: used to define the computation in the model. We just want to apply the linear projection to the input, and then the softmax transformation (so that the output of the model can be interpreted as a probability distribution over the 10 digits).\n",
    "\n",
    "Two things to note:\n",
    "\n",
    "1. Since the images are tensors of size 28x28, and the Linear operator expects a 1-d array (since it is the operation Wx), we need to flatten the image using the method view (as a reshape).\n",
    "2. Since the loss function is nll_loss (see next cell), it expects the input to be in log space, thats why we use log_softmax instead of softmax (see https://pytorch.org/docs/stable/nn.functional.html#log-softmax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class LogisticRegression(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(LogisticRegression, self).__init__()\n",
    "        self.lin = nn.Linear(im_size, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x_flat = x.view(-1, im_size)\n",
    "        return F.log_softmax(self.lin(x_flat), dim=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we define the training loop.\n",
    "For each minibatch (obtained using enumerate(train_loader) (see later for how the dataset is generated), we do the following:\n",
    "\n",
    "1. Move x, y (data, target) to the GPU if necessary (using .to(device))\n",
    "2. Reset the gradients to zero (from previous iterations) using .zero_grad()\n",
    "3. Forward pass: we compute the predictions using model(data) (which is y = f(x) with f being the logistic regression)\n",
    "4. Backward pass: we compute the gradients using .backward(), and then we apply one step of the optimizer, that is performing $w \\leftarrow w - lr * \\nabla (loss(f(x), y)$ for the case of gradient descent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train(model, device, train_loader, optimizer, epoch, train_losses):\n",
    "    model.train()\n",
    "    for batch_idx, (data, target) in enumerate(train_loader):\n",
    "        data, target = data.to(device), target.to(device)\n",
    "        optimizer.zero_grad()\n",
    "        output = model(data)\n",
    "        loss = F.nll_loss(output, target)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        if batch_idx % 100 == 0:\n",
    "            print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n",
    "                epoch, batch_idx * len(data), len(train_loader.dataset),\n",
    "                100. * batch_idx / len(train_loader), loss.item()))\n",
    "            train_losses.append(loss.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also create a function to evaluate our model. It is very similar to the previous one, but the main differences are:\n",
    "    \n",
    "1. We use torch.no_grad() to prevent gradient computations and changing the weights of the model.\n",
    "2. Once we have the predicted distribution, we use .argmax to compute the most probable digit. We also compute the accuracy of our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def test(model, device, test_loader):\n",
    "    model.eval()\n",
    "    test_loss = 0\n",
    "    correct = 0\n",
    "    with torch.no_grad():\n",
    "        for data, target in test_loader:\n",
    "            data, target = data.to(device), target.to(device)\n",
    "            output = model(data)\n",
    "            test_loss += F.nll_loss(output, target, reduction='sum').item() # sum up batch loss\n",
    "            pred = output.argmax(dim=1, keepdim=True) # get the index of the max log-probability\n",
    "            correct += pred.eq(target.view_as(pred)).sum().item()\n",
    "\n",
    "    test_loss /= len(test_loader.dataset)\n",
    "\n",
    "    print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n",
    "        test_loss, correct, len(test_loader.dataset),\n",
    "        100. * correct / len(test_loader.dataset)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now, we are going to load the dataset. Since the MNIST dataset is very popular, PyTorch already has it included into datasets.MNIST, and will automatically download it the first time. We can also specify some transformations, like converting the matrix to a tensor and normalize the pixels\n",
    "\n",
    "Note that we create two data loaders: one for the training set and another one for the testing set. Since this is just a quick experiment, we are not going to validate the initial hyperparameters, so we don't create another data loader for a validation set.\n",
    "\n",
    "The main benefit of the DataLoader class is that it creates an object which can be used in a for loop, as we did in the previous train function, to iterate over the minibatches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_loader = torch.utils.data.DataLoader(\n",
    "    datasets.MNIST('../data', train=True, download=True,\n",
    "                   transform=transforms.Compose([\n",
    "                       transforms.ToTensor(),\n",
    "                       transforms.Normalize((0.1307,), (0.3081,))\n",
    "                   ])),\n",
    "    batch_size=batch_size, shuffle=True)\n",
    "\n",
    "test_loader = torch.utils.data.DataLoader(\n",
    "    datasets.MNIST('../data', train=False, transform=transforms.Compose([\n",
    "                       transforms.ToTensor(),\n",
    "                       transforms.Normalize((0.1307,), (0.3081,))\n",
    "                   ])),\n",
    "    batch_size=test_batch_size, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize some examples together with the true labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1000, 1, 28, 28])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "examples = enumerate(test_loader)\n",
    "batch_idx, (example, label) = next(examples)\n",
    "example.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAADhCAYAAAD/Ec//AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xec1MX9x/HXR8SCoojyUEBAkYcF\nK9FfJIkFFRE0GgtGTRSMWLDXRGOCqKAYa0SxEUVFowYw6g/FLvaKLcH4M4pgRxAQsYvz+2Nv5ma5\n5W5v2TK7934+Hvfgw2z5zs3tzc13qjnnEBGRyluu0hkQEZEMVcgiIolQhSwikghVyCIiiVCFLCKS\nCFXIIiKJSLZCNrOpZnZ4uV/bEqhsS0vlWzq1XrYlr5DNbKaZ9S31dQplZoea2WIzWxR99al0vvKR\netkCmFl3M5tsZl+Y2Vwzu7DSecpX6uVrZtcs8bn91sy+qHS+8lEFZTvYzKaZ2UIz+8DMLjSz5Ut9\n3WRbyGX2rHNu1ehraqUzVAvMbAXgIeBRYB1gXeCWimaqhjjnhsafW+A2YEKl81Uj2gAnAWsB2wK7\nAKeV+qIVq5DNbI26ltMcM5tfF6+7xNM2MLMXzOxzM7vbzNpHr+9tZs+Y2QIze61aWrXlkFDZHgp8\n5Jy71Dn3pXPuG+fc6wW+VzISKt84T6sA+wE3Let7VVIqZeucu9o596Rz7jvn3IfArcAvCv/O8lPJ\nFvJywDigG9AV+Bq4connDAIOAzoBPwCjAcysM3AvMBJoT+Yv1yQz67DkRcysa90Pp2sjeelVdzv9\nlpkNK8etSYmlUra9gZlmNqWufKea2ebL/N1VXirlG9sPmAM8Ucg3lJAUyxZgB2B6s7+b5nLOlfQL\nmAn0zeN5WwHzo/9PBS6I/t8T+A5oBZwOjF/i9Q8Ag6PXHp5n/roD65P5IGwOvAH8sdTl0kLK9kHg\ne2AAsALwe2AGsEKly64WyneJ93gEOLvSZVajZfs74ANgrVKXSyW7LNqY2bVmNsvMFpL5y97OzFpF\nT3s/imcBrcn06XQD9q/7C7fAzBYA2wEdm5sP59wM59y7zrkfnXP/As4FBhb6faUglbIl07p5yjk3\nxTn3HXAxsCawSQHvlYyEytfnpwuwI3Bzoe+RigTLdm/gAmCAc25uoe+Tr0remp8KbARs65z7xMy2\nAl4BLHpOlyjuSqa1NZfMD2S8c+6IEuTLLZGHapRK2b5OGfrdKiCV8vUGAc8452YU8T0rJZmyNbP+\nwFhgj7rGWsmVq4Xc2sxWir6WB9qSaUEtqOuUH57jdQebWU8za0Om5TrRObeYzEj9nma2m5m1qnvP\nPjk6/5tkZgPMbO26eGNgGHB3gd9nJSRbtnXv1dvM+ta1cE4i84vzn0K+0QpJuXy9QcCNy/D6Skm2\nbM1sZzIDefs5514o+DtsrjL1FbklvkaS6ZCfCiwC3gKOqnts+ai/ZxTwArAQ+F+iPhwyU1EeB+aR\nGcy4F+i6ZF8Rmb+gi/xjOfJ3MTAb+JJM/+a5QOtSl0tLKNu65+wLvF13nanAppUutxor35/VfXbb\nVrq8aqlsgcfIDBguir6mlLpcrO7iIiJSYVoYIiKSCFXIIiKJUIUsIpIIVcgiIolQhSwikohmLQwx\nM03JaIJzrqBFJSrbvMx1zjXYl6ApKtu8FFS2oPLNU17lqxayVJNZlc5ADVPZllZe5asKWUQkEaqQ\nRUQSoQpZRCQR1b4RuyyDVq0yOxpuskn9bphnnXUWAPvtt19IO+igg0L8j3/8o0y5E2l51EIWEUmE\nKmQRkUSoy6IFGzRoEABjx45t8Fi8C2C3bt3KlieRlkwtZBGRRKiF3MKst956IR42bNhSn3f99deH\n+PLLLy9llkQAWHfd+oM9TjzxRAC6dKk/rWnDDTcM8VZbbQXAI488EtJat24NwKuvvtrodaZPrz88\nesKECSFesGBBIdkuKrWQRUQSoQpZRCQRzTrCqZKbiLRr1y7E7du3b/S5q622GgCHHnpozsdXXnll\nIPt26K233gLgoYceCmn33ntvs/OZ+uZCr7/+eog33XRTIHsAz98CHn744SHt/ffjU9crappzbpvm\nvkib3+SloLKFZS9f//v84osvhrTOnTsD8OSTTzb62ldeeSXEvXr1avD4LrvsEuJcdd2bb74Z4iuv\nvBKAv//97yHt888/b/T6zZBX+aqFLCKSCFXIIiKJSL7LYpttMq38a665JqTlujWJmWV6DQo5Ufvt\nt98O8dNPPx3i0047DYB58+Y1+voUuywuuOCCEJ9yyikh9kun41FnX97fffddqbKzLNRlUToV67LY\neOONAdhtt91C2gMPPABkdykUYrPNNmuQ5rvqAE4//fQQb7nllgDMmDEjpF1xxRUAjB49epnygbos\nRESqSzIt5KOPPjpnvP766wPQpk2bvN+rqRbyN998A8Ctt97a4DE/vxFgpZVWCrEfcBgyZEij106p\nhdyvXz8ApkyZkvPxTz/9FMge+HjjjTeKnY1iSrqFHK9o3H777QH4+c9/HtIGDhwIwFprrRXnLcT+\n8zpmzJiQdt5554X4k08+KXKOs1SshVxJK6ywQoj33HNPAEaOHBnSunfvDsBJJ50U0q6++upCLqUW\nsohINVGFLCKSiIovnb722muB7HmvTfniiy8AePbZZ0NaPB/xww8/BGDcuHGNvs9XX33VIM0vv4Ts\n28k4rhb9+/cHlt51c8kllwDJd1MkI+42890Pv/3tb0NaPNi85pprLvV94p9Hrp/NMcccE+Idd9wx\nxFtssUUzcyxNiQevJ02aBMC7774b0h5//HEAhg4dGtIK7LLIi1rIIiKJqEgLuW/fviHef//983pN\nPO3t0ksvBeCdd94pbsaA77//vujvWSk9e/ZskDZrVv3htxMnTizKdeINi2bOnFmU90yJX/EZD7bF\nA775WrhwIVA/mLok36peY401QlqnTp1CvOqqqwKwaNGiZl9b8rfXXnuFuDmTCYpBLWQRkUSoQhYR\nSURZuyz8LVe8cmz11VfP67XnnHNOiJd2yyew3Xbbhbhjx44NHh8xYkSI8+1eiFc2+bnb8WBUPK/2\niSeeALIHab/++uu8rpMSv7kNwMUXXww0r5vCd6fFe0n7jZuWtvrsyCOPBLIHjeLuC7+SzQ8+SXH5\nrrff/e53Ic0P/B977LFlyYNayCIiiVCFLCKSiLJ2Wfhb18mTJ4c0v1R5afN8//SnPwEwd+7cEueu\nNhxxxBEhjrsavPvuu6/R1/vbtnvuuSekde3aNcRt27YFlj63+YADDgDq96SG+u6ml156qdFrpyTu\nZvHz2uPug2+//RaAhx9+OKTdfffdIb7jjjuA5s2IePDBBxt9fPDgwYC6LIopniHk90L3n3GAM888\nE4CnnnqqLPlRC1lEJBFlbSEvXrwYgLPPPjuk7bvvvkDu1hzAY489BsCPP/5Y2szViKZWF+ZK6927\nd4jjLUdzWW65zN/wpn4eu+++e4j79OkDwIABA0JauVochYq3WfX59luTAsyePRuA559/vmx5KmTu\nszR03HHHhTjejtbf1cVrI+JDVMtBLWQRkUSoQhYRSUTFNxdqir+9ePXVV0OaH1CRhnJtXBN3Q8SH\nNvo54OPHj8/5em/OnDkh9oNVSxugO+GEEwD4yU9+EtL88tO77rorpMW3hb5bKlUfffQRkD3QWQrx\nRkW5xBtoSfNdd911AAwaNCikxQerHnbYYQBMnTq1rPmKqYUsIpKIireQfct3aYN6v/nNb4D6Exig\n/hws31qTxsXbCfrTUqB+NZ0/lSUWb216/PHHhzi+U8nFD4L4DXmg/gSGdu3ahbSddtopxKm3kMtl\n7733bpAWb3bV1JRFaSi+q/BTbOPP9s477xziFDYWUwtZRCQRqpBFRBJR8S4L38Een9ThV3vF4tVi\nt912GwDnn39+SIsP6qzFPXmLxa/2Ahg2bFiDx32XQ7w50Pvvv5/3+3/88ccAjBo1KqT51Wp+wA+y\n95y97LLLAJg/f37e16kVG2+8cYhz7V8dl4k/vUJy69ChA1C/XzpkH1rsu9v86l9Io5siphayiEgi\nVCGLiCSi4l0WXjwq/+ijj4b4wgsvBLI3q/HijUHiTV78yOl7771X5FxWp/hWuF+/fiH2+xjfeOON\nIe3oo48Gsg9/XFZXXHEFkL30OJ5ze9555wHZh3u2FPHRZLmWRvtZRpJbjx49Quw3B4q7N++///4Q\n+65Qf5RWitRCFhFJRDIt5Hj13dixY0PsB5luuOGGkBbPSfbiubTPPfcckN26qOTqm3LKtbnQ1ltv\nnfO5/uSVSy65JKQVs2Xs+ZZJvHovzucqq6xS9GumLN5kqVevXg0ej+/stDqvXnwH4cvwlltuCWl+\n46u//OUvIW348OEhTm0ALxe1kEVEEqEKWUQkEcl0WSzNjBkzANh1111Dmj908pBDDglp8WGp66yz\nDgDXXnttSDvjjDNC/M9//rM0mU3AnXfeGWI/19cfLrukm266CYA33ngj7/f388Xj94xP0vADc35O\nKMDBBx8MZG9c9MUXX+TMc0sQf/7i+fdevJR8wYIFZclTNYh/n/1nyh9CCvVz5/06hWqkFrKISCJU\nIYuIJMKWdlhlzieb5f/kMthuu+1CHB8Q6Udj4+8t3st24MCBQP2RUsXknMt9WmsTSlG2/hZvyJAh\nOR/3o87xktxZs2YB2QdpxrM0/Dzm9u3bh7SOHTuGOE73/IyKuJtizJgxIY6XsjZhmnNum6af1uD6\nSXxu99hjDyD7MNR4tom//Y4/16+99lqZcldY2UJpytf/DsfdFPvss0+Dx+O9jW+//fZiZ6OY8ipf\ntZBFRBJR1S3kWLyJyMsvvwws/ah63+Jram/fQqTUQvbHmZ911lkhbejQoSH2J3k08zPQ7Nfceuut\nQPbgXdxKbIaqayGvu+66IfYHu3bp0iXnc/3hvyNGjCh5vnJIqoXsW8MTJ07M+bhf6dmcVnGrVq0A\nWHnllUNavDrUzwk/+eSTQ9onn3yS9/s3QS1kEZFqogpZRCQRyc9Dztf06dPzfq5fdlmKLouU+EG0\n3//+9yEt3rhp9913B+o3FGqO+JDTeP9pv5lLfNyQX6LdEsUDqrm6Kj788MMQT5gwoSx5StUOO+wQ\n4qaOZ/O/u3GXUHxEmNenT58Q+9/7/v37h7T4c+w3uSpiN0WzqYUsIpKI5FvIG2ywAZDdEe873+Nt\n9PzBp9D0wNNnn31W9HxWiylTpjSI40NMpTh69+4NwGmnndbo8y6//PIQv/nmmyXNU+o23HDDEPsB\nuKXxd2DxSsdOnTo1eF68ks9vVHbggQeGtHiguRTTYJtLLWQRkUSoQhYRSUSSXRbxCRdPP/00kPvE\nkKXJ1VUR7zHrN9URKZWf/vSnQP1c79gHH3wQYn0W68UDeX6wbvbs2SEtPpS3W7duQHb34+jRo4Hs\nE3DmzZsX4uYc1lspaiGLiCRCFbKISCKS7LKIR0b97cfyy9dnNd/DMONblHg/5Pi4KJFiiWcGNHY4\n6VVXXRXiuXPnljRP1STefMovI49dffXVZcxNZaiFLCKSiJrZXCgVKW0uVIOS3lxoxx13DHG8ItLz\nKxbjFXs//PBD6TOWn6Q2F6pB2lxIRKSaqEIWEUlEkoN6ItVo2rRpIX733XeB7MNg/eY2CXVTSGLU\nQhYRSYRayCJFsmjRohD36NGjgjmRaqUWsohIIlQhi4gkorldFnOBWaXISI3otgyvVdk2rdDyVdk2\nTZ/d0sqrfJu1MEREREpHXRYiIolQhSwikghVyCIiiVCFLCKSCFXIIiKJUIUsIpIIVcgiIolQhSwi\nkghVyCIiiVCFLCKSCFXIIiKJUIUsIpIIVcgiIolQhSwikghVyCIiiVCFLCKSCFXIIiKJUIUsIpII\nVcgiIolQhSwikghVyCIiiVCFLCKSCFXIIiKJUIUsIpIIVcgiIolQhSwikghVyCIiiVCFLCKSiGQr\nZDObamaHl/u1LYHKtrRUvqVT62Vb8grZzGaaWd9SX2dZmNnJZvaJmX1uZjeY2YqVzlM+Ui9bM9vM\nzB4ws7lm5iqdn+aqgvI1MxtpZh/WfXanmtmmlc5XPqqgbFc0s8vM7CMzm29mV5lZ61JfN9kWcrmY\n2W7AGcAuwHpAd+CcSuaphnwP/AMYUumM1Kj9gcOA7YH2wLPA+IrmqHacAWwDbAZsCPwE+HOpL1qx\nCtnM1jCzyWY2p+4v0GQzW3eJp21gZi/U/fW/28zaR6/vbWbPmNkCM3vNzPoUmJXBwPXOuenOufnA\nCODQAt8rCamUrXPu/5xz1wPTl+HbSU4q5QusDzzlnJvhnFsM3AL0LPC9kpBQ2e4JjHbOzXPOzQFG\nk/njV1KVbCEvB4wDugFdga+BK5d4ziAyhdAJ+IFMoWBmnYF7gZFkWganAZPMrMOSFzGzrnU/nK5L\nycemwGvR/18D1jazNQv8vlKQStnWqlTK93agh5ltWHc7PRi4fxm/t0pLpWyt7iv+/7pmtnqB31d+\nnHMl/QJmAn3zeN5WwPzo/1OBC6L/9wS+A1oBpwPjl3j9A8Dg6LWH55m/d4D+0f9bAw5Yr9RlU+tl\nG72+R+ajVvkyq6XyBVYALq/7vP4AvAusX+lyq5GyHQk8DXQA1gGeryvnjqUsl+WpEDNrA1wG9AfW\nqEtua2atXOb2C+D96CWzyFSWa5H567m/me0ZPd4aeKyArCwCVov+7+MvCnivJCRUtjUpofIdDvwP\n0AX4BDgYeNTMNnXOfVXA+1VcQmV7HtAOeBX4FhgL9AI+LeC98lbJLotTgY2AbZ1zqwE71KXHtwld\norgrmUGiuWR+IOOdc+2ir1WccxcUkI/pwJbR/7cEZjvnPivgvVKRStnWqlTKd0vgDufcB865H5xz\nN5KpxKq5HzmJsnXOfe2cO84519k51x34DJgW/VEoiXJVyK3NbKXoa3mgLZn+oQV1nfLDc7zuYDPr\nWfdX81xgoqsfvNjTzHYzs1Z179knR+d/Pm4GhtRdZw0yI6k3FvJNVkiyZWsZK5G5tabuvapiSmEk\n2fIFXiTTIlzbzJYzs0PItAjfLug7Lb9ky9bMOptZp7rPcG9g2FLyUlxl6ityS3yNJNMhP5VMl8Fb\nwFF1jy0f9feMAl4AFgL/C6wVve+2wOPAPGAOmc78rkv2FZH5C7rIP7aUPJ4CzK67zjhgxVKXS0so\nWzLTCJfM38xKl1sNle9KwBjg47rrvEw0HpLyVxWU7Q51efwK+D/gt+UoF6u7uIiIVFiLXxgiIpIK\nVcgiIolQhSwikghVyCIiiVCFLCKSiGat1LMq3EKx3Jxz1vSzGlLZ5mWuc67BvgRNUdnmpaCyBZVv\nnvIqX7WQpZrMqnQGapjKtrTyKl9VyCIiiVCFLCKSCFXIIiKJUIUsIpIIVcgiIolQhSwikghVyCIi\niajYEU5SnQ488EAArryy/tzJF198McQDBgwoe55qzcCBA0P861//OsT7778/AJMmTcr5XKl+aiGL\niCRCFbKISCKqpstilVVWCfFJJ50EQJs2bULaL37xixA//fTTANx3330h7ZVXXgnxV19V5YG8SRgy\nZAgA7du3D2k6daa4Pvus/nzdcePGhdh3T/Tq1avseZLyUAtZRCQRqpBFRBKRZJdFz549Q3zUUUcB\ncMghh4S01VdfvcFrzOp3vdx+++0BOOOMM0La3LlzQ7zHHnsA8NJLLxUpxy1HrtvlBx98sAI5qQ6d\nO3cOcb9+/QDo3r17SPOfxViuz7fktuaaa4b4+OOPB2D33XcPaVtvvXWIZ83KbLg2ceLEkHb00UeH\neM899wRg6tSpJclrPtRCFhFJRJIt5G7duoX4uOOOy+s18UDI7bffDsC0adNCmp8/C3DPPfcAsM8+\n+4S0559/vrDMtgB+IA+yB/O8eMBVMo499lgARo4cGdJWW201AB5//PGQNnny5BDHn2Fv0KBBpcpi\nVdtpp50AmDBhQkhr164dkH23HA/gL168GIBTTjkl53tus802gFrIIiKCKmQRkWQk2WWxcOHCEPtb\njnjOsTd79uwQ77zzziF+8803Gzz37rvvDvHDDz8MwJ///OeQ5jv0JWO55er/Vu+7776NPvdf//pX\nqbNTFfbee+8QX3bZZQD89a9/DWkXXnghkD3A3JTvvvsuxGPGjFnWLFadFVdcMcSjRo0K8cEHHwzU\nd1PE7rrrrhBfdNFFIfZdRvH6hNSohSwikogkW8h+pR3Af//7XwC23HLLkPbRRx8B0L9//5CWq1Uc\nmz9/fojjlrXktsUWW4Q414ZB06dPD7EfJK1V/vv3nzuA1157rcHzTjzxxBAvv3zmV+uqq64Kafm2\njP1rIbtV6Aer1l9//ZC21VZbAfDqq6/m9d7VJi7TE044ocHj8aCov+P997//nfO9xo8f3+i1Ntlk\nk0KyWFRqIYuIJEIVsohIIpLssujdu3eIe/To0eBx31Ef3zY3xzvvvAPAQQcd1OA6b7/9dkHvWWtO\nP/30Rh+P9+Stddtuuy1QvxIM6ufKL1q0KKTFmyz5uJCNl+LVY23btg2xHyjcddddQ9o333zT7Pev\nJsOGDcuZ7lfdnXrqqSHN/17HNt100xDvt99+jV4rhb281UIWEUmEKmQRkUQk02XRoUOHEMcjy35Z\nbjzH8/7771+ma91xxx0AHHPMMSFt1VVXXab3rBV+hH+zzTbL+bhf3nv99deXLU+pWLBgQYh/+OGH\nBo9//vnnDdL8fFmACy64AKhfwrukddZZB4Czzz47pMVL+v2S33jzoVzXrFbxRkB++XL8e3n55ZeH\n2O+JHvN1RXzs1d/+9rcQ+zndcZnG3aNPPPFEoVkvGrWQRUQSkUwL+YADDgjxDjvs0ODx//znPyH+\n/vvvi359v81nPKDSEvmBo3gwJOZXTsbbHr7//vulz1gF+YG7a6+9NqTlGky7+OKLQ+xXjp577rkh\nrVOnTgAMHz68wXsD3HnnnQ3eM74z9GqpVRyLt91deeWVgfp1CJC96tFbY401QuxX6G233XYhLd5g\n7I9//COQ/XOKB11T2I5XLWQRkUSoQhYRSUQyXRbxLv8xP8Axc+bMol3rhRdeyPoXsm99WrKm9t/1\nmw7FczZrddmu5weTmuoqi5f877LLLkD24OfQoUMB6NOnT0ibN29eiP185/hn8OmnnxaY69pw3XXX\nhTge4DviiCOA7P3SfTdb/Hn81a9+FeKPP/640WsVs44plFrIIiKJqHgL2Q+i9e3bN6TFK278tJVc\n04wK5acXrb322iHNr/xpiXxrDnKf/vH111+H2J/U0JJWNBYyiOwHiOKyHTduHAC//OUvQ1o8qPTe\ne+8B2VvFtiQzZswIsf/MnXzyySEtPiMz17ab/qy8eEXlnDlz8r5+fNZepaiFLCKSCFXIIiKJqEiX\nRbxhkN8vNr51i+ceNtURX4j11lsv61+A0aNHF/06qVthhRWA+sEmyD4pxItXm7WkropiiPdAvuKK\nKwDYY489Qlr8ue/SpQuQvenV2LFjS53FZMSDon61XbxqsWvXriG+5JJLALj55ptDml+r8OOPP5Yy\nmyWlFrKISCJUIYuIJKLisyxy3V489NBDJb2mn0kQ3y7GS1hbCn8LmGuf2EcffTTE9957b9nyVGvi\no8duuukmIPsIsSOPPDLE11xzDZC9tPeVV14JcQpLe8tlypQpWf8Wkz8Ka8k4BWohi4gkouItZC/e\nEu+2224r+vvHx4n70zDi4+tzbexSi1q1ahXiG264ocHj/q7hvPPOC2nxUfSSH98yjsvYD5jGc5Pj\nAzn9Cr54a8l4pV/c2pbC5TrZJRVqIYuIJEIVsohIIpLpsoj3jY0HPYrlzDPPDPH2228PwH333RfS\n4k1eall8KkO8b6zny+Sxxx4rW55qRXzKij/VJp7XfcghhwDZ3RQx300Uz/te2sktUpvUQhYRSYQq\nZBGRRCTTZVEK/hgYqO+miE2YMKGc2UmC360Ncs/B9AdxSn423njjEMdzt/3+2gMHDgxpDz74YKPv\n5bvN/DFZkN482XLz3TyQvbQ63hkuX/5A03jrBj83PBVqIYuIJKImW8h+zrHfzAVgxx13DPEf/vAH\noP5QxFoX7zV9/vnnh9jPwYwH8OL54LJ0bdq0AWDYsGEhLT749aKLLgKat7ex3+wpniue2jzZcrvx\nxhtDHB/u+sADDwDZGzE1xf/ex3fO8eZPKVALWUQkEaqQRUQSUZEuiy+//DLEH3zwAQCbbLJJSPNz\nOJsjnq956qmnAtmHRT7zzDMhfuSRRwBYuHBhs69TjfbZZ58Qx4NEfkOlwYMHh7RiHpVVy/r16wdk\n3zLH3QvxHPfG+G4KqN/7uHPnziGtpQ+yxl0Ka621Voj9fskvv/xySLv00ksBWLx4cUhr3bp1iP1A\na8oDpWohi4gkoiIt5PgUkOeeew6AkSNHhrQFCxaE2B8MGfOtkpVWWimk+RMEoP6vov+LCXDOOeeE\nuKVttdm9e/cQx62Db7/9Fqi/S5H8+Y2p4hZcPKjnW7bx5kD+8N54u9P4mHrfMp48eXJIi1ewtkTx\nqT633HJLiH25jRo1KqT57UnjQeqePXuG2E99je9kSrG957JQC1lEJBGqkEVEEmHNmedoZkWfFLnR\nRhsB2bcOHTt2DPH8+fMbvKZDhw5A9sYtvusD6rs/KnE74pwraMSgFGXr7bXXXiGeNGlSiP0t3s9+\n9rNSXbrYpjnntmnui0pZtltssUWI4wM3N99888byE+Inn3wyxH7v5PHjx4e0Mh7YWVDZQmnLN9a2\nbdsQ+7KKB6y94cOHhzjuUhozZgyQ3WUa/5zirtISyKt81UIWEUmEKmQRkURUvMvCizf8GDFiRIj9\nfMPY6NGjAXj44YdDWnwwaiWPHEqxy6KGJNdlUUOS77KI+e6Ls846K6QNHToUqF/WDrmXnsezNQ49\n9NAS5bABdVmIiFSTZFrItUIt5JJSC7l0qqqFnMuAAQMAOPvss0NafEKOF5+UE08GKDG1kEVEqokq\nZBGRRKjLosjUZVFS6rIonarvskicuixERKqJKmQRkUSoQhYRSYQqZBGRRDR3P+S5wKxSZKRGdFuG\n16psm1Zo+apsm6bPbmnlVb7DPrpMAAAAP0lEQVTNmmUhIiKloy4LEZFEqEIWEUmEKmQRkUSoQhYR\nSYQqZBGRRKhCFhFJhCpkEZFEqEIWEUmEKmQRkUT8P65qVg+ErlodAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f1564bf4438>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "for i in range(8):\n",
    "  plt.subplot(2,4,i+1)\n",
    "  plt.imshow(example[i][0], cmap='gray', interpolation='none')\n",
    "  plt.title(\"Label: {}\".format(label[i]))\n",
    "  plt.xticks([])\n",
    "  plt.yticks([])\n",
    "fig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are almost done! We just instantiate the model, and tell which optimizer are we going to use, in this case, stochastic gradient descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LogisticRegression().to(device)\n",
    "optimizer = optim.SGD(model.parameters(), lr=lr, momentum=momentum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally, a simple loop over the number of epochs. This should take over a few minutes in a simple laptop cpu.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Epoch: 1 [0/60000 (0%)]\tLoss: 2.451776\n",
      "Train Epoch: 1 [3200/60000 (5%)]\tLoss: 0.747278\n",
      "Train Epoch: 1 [6400/60000 (11%)]\tLoss: 0.607951\n",
      "Train Epoch: 1 [9600/60000 (16%)]\tLoss: 0.449380\n",
      "Train Epoch: 1 [12800/60000 (21%)]\tLoss: 0.344543\n",
      "Train Epoch: 1 [16000/60000 (27%)]\tLoss: 0.172929\n",
      "Train Epoch: 1 [19200/60000 (32%)]\tLoss: 0.434396\n",
      "Train Epoch: 1 [22400/60000 (37%)]\tLoss: 0.192195\n",
      "Train Epoch: 1 [25600/60000 (43%)]\tLoss: 0.384779\n",
      "Train Epoch: 1 [28800/60000 (48%)]\tLoss: 0.489098\n",
      "Train Epoch: 1 [32000/60000 (53%)]\tLoss: 0.262179\n",
      "Train Epoch: 1 [35200/60000 (59%)]\tLoss: 0.172913\n",
      "Train Epoch: 1 [38400/60000 (64%)]\tLoss: 0.657756\n",
      "Train Epoch: 1 [41600/60000 (69%)]\tLoss: 0.335815\n",
      "Train Epoch: 1 [44800/60000 (75%)]\tLoss: 0.254254\n",
      "Train Epoch: 1 [48000/60000 (80%)]\tLoss: 0.438054\n",
      "Train Epoch: 1 [51200/60000 (85%)]\tLoss: 0.184763\n",
      "Train Epoch: 1 [54400/60000 (91%)]\tLoss: 0.156785\n",
      "Train Epoch: 1 [57600/60000 (96%)]\tLoss: 0.219509\n",
      "\n",
      "Test set: Average loss: 0.2898, Accuracy: 9176/10000 (92%)\n",
      "\n",
      "Train Epoch: 2 [0/60000 (0%)]\tLoss: 0.183649\n",
      "Train Epoch: 2 [3200/60000 (5%)]\tLoss: 0.132637\n",
      "Train Epoch: 2 [6400/60000 (11%)]\tLoss: 0.255301\n",
      "Train Epoch: 2 [9600/60000 (16%)]\tLoss: 0.376051\n",
      "Train Epoch: 2 [12800/60000 (21%)]\tLoss: 0.222458\n",
      "Train Epoch: 2 [16000/60000 (27%)]\tLoss: 0.289201\n",
      "Train Epoch: 2 [19200/60000 (32%)]\tLoss: 0.562356\n",
      "Train Epoch: 2 [22400/60000 (37%)]\tLoss: 0.352593\n",
      "Train Epoch: 2 [25600/60000 (43%)]\tLoss: 0.555948\n",
      "Train Epoch: 2 [28800/60000 (48%)]\tLoss: 0.578471\n",
      "Train Epoch: 2 [32000/60000 (53%)]\tLoss: 0.237165\n",
      "Train Epoch: 2 [35200/60000 (59%)]\tLoss: 0.376687\n",
      "Train Epoch: 2 [38400/60000 (64%)]\tLoss: 0.560380\n",
      "Train Epoch: 2 [41600/60000 (69%)]\tLoss: 0.320704\n",
      "Train Epoch: 2 [44800/60000 (75%)]\tLoss: 0.607076\n",
      "Train Epoch: 2 [48000/60000 (80%)]\tLoss: 0.551095\n",
      "Train Epoch: 2 [51200/60000 (85%)]\tLoss: 0.164039\n",
      "Train Epoch: 2 [54400/60000 (91%)]\tLoss: 0.247533\n",
      "Train Epoch: 2 [57600/60000 (96%)]\tLoss: 0.167713\n",
      "\n",
      "Test set: Average loss: 0.2790, Accuracy: 9194/10000 (92%)\n",
      "\n",
      "Train Epoch: 3 [0/60000 (0%)]\tLoss: 0.683573\n",
      "Train Epoch: 3 [3200/60000 (5%)]\tLoss: 0.109873\n",
      "Train Epoch: 3 [6400/60000 (11%)]\tLoss: 0.430049\n",
      "Train Epoch: 3 [9600/60000 (16%)]\tLoss: 0.233136\n",
      "Train Epoch: 3 [12800/60000 (21%)]\tLoss: 0.323978\n",
      "Train Epoch: 3 [16000/60000 (27%)]\tLoss: 0.309407\n",
      "Train Epoch: 3 [19200/60000 (32%)]\tLoss: 0.179022\n",
      "Train Epoch: 3 [22400/60000 (37%)]\tLoss: 0.272989\n",
      "Train Epoch: 3 [25600/60000 (43%)]\tLoss: 0.262438\n",
      "Train Epoch: 3 [28800/60000 (48%)]\tLoss: 0.652400\n",
      "Train Epoch: 3 [32000/60000 (53%)]\tLoss: 0.294648\n",
      "Train Epoch: 3 [35200/60000 (59%)]\tLoss: 0.183302\n",
      "Train Epoch: 3 [38400/60000 (64%)]\tLoss: 0.081914\n",
      "Train Epoch: 3 [41600/60000 (69%)]\tLoss: 0.290835\n",
      "Train Epoch: 3 [44800/60000 (75%)]\tLoss: 0.106422\n",
      "Train Epoch: 3 [48000/60000 (80%)]\tLoss: 0.388546\n",
      "Train Epoch: 3 [51200/60000 (85%)]\tLoss: 0.355330\n",
      "Train Epoch: 3 [54400/60000 (91%)]\tLoss: 0.375802\n",
      "Train Epoch: 3 [57600/60000 (96%)]\tLoss: 1.042810\n",
      "\n",
      "Test set: Average loss: 0.2822, Accuracy: 9206/10000 (92%)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "train_losses = []\n",
    "for epoch in range(1, epochs + 1):\n",
    "        train(model, device, train_loader, optimizer, epoch, train_losses)\n",
    "        test(model, device, test_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize the training losses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAD8CAYAAACYebj1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl4VOXZBvD7ISQQDEJZFGVHsbLI\nGgUVFRBRcEErWorgUluK+ilWsYK1VtG2VnEp7ojgUlxQUXArorIpAgZEVllk0UhkEwgIBEKe749n\nTjJJZjkzmTNLcv+uK9fMnDnLeyYz5znvLqoKIiIiAKiW6AQQEVHyYFAgIqJiDApERFSMQYGIiIox\nKBARUTEGBSIiKsagQERExRgUiIioGIMCEREVq57oBESqQYMG2qJFi0Qng4gopSxevHiHqjYMt17K\nBYUWLVogJycn0ckgIkopIrLZzXosPiIiomKeBQURaSois0RktYisFJERAdbpKSJ7RGSp7+8er9JD\nRETheVl8VAjgdlVdIiK1ASwWkZmquqrMevNU9SIP00FERC55FhRUNQ9Anu/5XhFZDaAxgLJBgYiq\noMOHDyM3NxcHDx5MdFIqlZo1a6JJkyZIT0+Pavu4VDSLSAsAnQEsDPD26SLyDYAtAEaq6soA2w8D\nMAwAmjVr5l1CiShucnNzUbt2bbRo0QIikujkVAqqip07dyI3NxctW7aMah+eVzSLSBaAtwHcqqr5\nZd5eAqC5qnYE8ASAdwPtQ1XHq2q2qmY3bBi2RRURpYCDBw+ifv36DAgxJCKoX79+hXJfngYFEUmH\nBYTJqjq17Puqmq+q+3zPPwSQLiINvEwTESUPBoTYq+hn6mXrIwHwAoDVqvpokHUa+daDiJzmS89O\nL9KzYgXwt78B27d7sXciosrBy5zCmQCGAujt1+S0v4gMF5HhvnUGAljhq1MYB2CQejRp9Jo1wAMP\nAHl5XuydiFLNzp070alTJ3Tq1AmNGjVC48aNi18fOnTI1T6uu+46rFmzxvUxJ0yYgFtvvTXaJMeF\nl62PPgcQMh+jqk8CeNKrNPirVcse9++Px9GIKNnVr18fS5cuBQDce++9yMrKwsiRI0uto6pQVVSr\nFvj+edKkSZ6nM96qTI9mJygcOJDYdBBRclu/fj3at2+P4cOHo0uXLsjLy8OwYcOQnZ2Ndu3aYcyY\nMcXr9ujRA0uXLkVhYSHq1q2LUaNGoWPHjjj99NOxbds218f873//i1NOOQXt27fHXXfdBQAoLCzE\n0KFDi5ePGzcOAPDYY4+hbdu26NixI4YMGRLbk0cKjn0UrcxMe2ROgSj53Hor4Ltpj5lOnYDHH49u\n21WrVmHSpEl49tlnAQAPPvgg6tWrh8LCQvTq1QsDBw5E27ZtS22zZ88enHPOOXjwwQdx2223YeLE\niRg1alTYY+Xm5uLuu+9GTk4O6tSpgz59+uD9999Hw4YNsWPHDixfvhwAsHv3bgDAQw89hM2bNyMj\nI6N4WSxVuZwCgwIRhXPCCSfg1FNPLX792muvoUuXLujSpQtWr16NVavK98HNzMxEv379AABdu3bF\npk2bXB1r4cKF6N27Nxo0aID09HQMHjwYc+fOxYknnog1a9ZgxIgRmDFjBurUqQMAaNeuHYYMGYLJ\nkydH3UEtlCqTU2BQIEpe0d7Re+Woo44qfr5u3Tr85z//waJFi1C3bl0MGTIkYD+AjIyM4udpaWko\nLCx0daxgbWvq16+PZcuW4aOPPsK4cePw9ttvY/z48ZgxYwbmzJmDadOm4YEHHsCKFSuQlpYW4RkG\nV+VyCqxTIKJI5Ofno3bt2jj66KORl5eHGTNmxHT/3bt3x6xZs7Bz504UFhbi9ddfxznnnIPt27dD\nVXHFFVfgvvvuw5IlS3DkyBHk5uaid+/eePjhh7F9+3bsj/GdbpXJKbBOgYii0aVLF7Rt2xbt27dH\nq1atcOaZZ1Zofy+88ALeeuut4tc5OTkYM2YMevbsCVXFxRdfjAsvvBBLlizB9ddfD1WFiODf//43\nCgsLMXjwYOzduxdFRUW48847Ubt27YqeYiniUbcAz2RnZ2s0k+wcPgxkZAD33w/cfbcHCSOiiKxe\nvRpt2rRJdDIqpUCfrYgsVtXscNtWmeKj9HSgenXmFIiIQqkyQQGwegUGBSKi4KpcUGBFM1HySLXi\n61RQ0c+0SgWFzEzmFIiSRc2aNbFz504Ghhhy5lOoWbNm1PuoMq2PABYfESWTJk2aIDc3F9s5dHFM\nOTOvRYtBgYgSIj09PerZwcg7Var4iEGBiCi0KhcUWNFMRBRclQoKrGgmIgqtSgUFFh8REYXGoEBE\nRMWqXFBgnQIRUXBVLigwp0BEFFyVCgqZmUBhoY2YSkRE5VWpoMDZ14iIQmNQICKiYlUyKLCymYgo\nsCoVFDglJxFRaFUqKLD4iIgoNAYFIiIqViWDAusUiIgCq5JBgTkFIqLAqlRQYEUzEVFoYYOCiBwl\nItV8z08SkUtEJN37pMUecwpERKG5ySnMBVBTRBoD+BTAdQBe9DJRXmFQICIKzU1QEFXdD+A3AJ5Q\n1csAtA27kUhTEZklIqtFZKWIjAiwjojIOBFZLyLLRKRL5KfgHiuaiYhCcxUUROR0AFcB+MC3rLqL\n7QoB3K6qbQB0B3CTiJQNJv0AtPb9DQPwjKtURyk9HUhLY06BiCgYN0HhVgCjAbyjqitFpBWAWeE2\nUtU8VV3ie74XwGoAjcusNgDAy2oWAKgrIsdFdAYREOGUnEREoYS941fVOQDmAICvwnmHqt4SyUFE\npAWAzgAWlnmrMYAf/F7n+pblRbL/SHBOBSKi4Ny0PnpVRI4WkaMArAKwRkTucHsAEckC8DaAW1U1\nv+zbATbRAPsYJiI5IpKzfft2t4cOiEGBiCg4N8VHbX0X80sBfAigGYChbnbua7r6NoDJqjo1wCq5\nAJr6vW4CYEvZlVR1vKpmq2p2w4YN3Rw6KE7JSUQUnJugkO67uF8KYJqqHkaAu/myREQAvABgtao+\nGmS16QCu9rVC6g5gj6p6VnQEsE6BiCgUN62IngOwCcA3AOaKSHMAZYuBAjkTlqNYLiJLfcvuguU0\noKrPwnIe/QGsB7Af1gfCUyw+IiIKzk1F8zgA4/wWbRaRXi62+xyB6wz811EAN4XbVyzVqgXs2hXP\nIxIRpQ43Fc11RORRp6JXRB4BcFQc0uYJ1ikQEQXnpk5hIoC9AK70/eUDmORlorzE4iMiouDc1Cmc\noKqX+72+z6+OIOWwopmIKDg3OYUDItLDeSEiZwJI2QIY5hSIiIJzk1O4AcBLIlIHVnH8M4BrvUyU\nlxgUiIiCc9P6aCmAjiJytO+1m+aoSatWLeDwYaCwEKjuJiQSEVUhQS+LInJbkOUAgBAd0pKa//DZ\ntWsnNi1ERMkm1L1ypbxk+k/JyaBARFRa0KCgqvfFMyHxwtnXiIiCc9P6qFJhUCAiCq7KBgX2aiYi\nKq/KBgXmFIiIyou49ZEjVVsf+Vc0ExFRaW5aH/0awKmwuQ8A4GIAc71MlJeYUyAiCi5s6yMR+RhA\nF1Xd63t9L4A345I6D7BOgYgoODd1Cs0AHPJ7fQhAC09SEwfMKRARBedmoIdXACwSkXd8ry8F8JJ3\nSfIW6xSIiIJzM/bRP0TkIwBnweZmvk5Vv/Y8ZR5hToGIKDi3Q8IdAVAECwpF3iXHezVqACIMCkRE\ngbiZjnMEgMkAGgA4BsB/ReRmrxPmFRFOyUlEFIybnML1ALqp6i8AICL/BvAlgCe8TJiXOKcCEVFg\nblofCaz4yHHEtyxlcUpOIqLA3OQUJgFY6Gt9JAAGAHjB01R5jDkFIqLA3LQ+elREZgNw5mlO6dZH\nAIMCEVEwkbQ+UlSC1kcAK5qJiIKpcq2PAOYUiIiCqZKtjzIzgZ9+SnQqiIiST5VsfcScAhFRYJG2\nPgJs7KOUb33EOgUiovLctj6aA+BMWA6BrY+IiCopt62PlgLIc9YXkWaq+r1nqfIYgwIRUWBhg4Kv\npdHfAWxFSX2CAujgbdK8k5kJFBQAR44AaWmJTg0RUfJwk1MYAeDXqrrT68TEi//sa1lZiU0LEVEy\ncdP66AcAeyLdsYhMFJFtIrIiyPs9RWSPiCz1/d0T6TGixSk5iYgCC5pTEJHbfE83AJgtIh8AKHDe\nV9VHw+z7RQBPAng5xDrzVPUid0mNHU60Q0QUWKjio9q+x+99fxm+P1dUda6ItIg6ZR7ilJxERIEF\nDQqqel8cjn+6iHwDYAuAkaq6Mg7HZE6BiCiIUMVHj6vqrSLyHqy1USmqekkFj70EQHNV3Sci/QG8\nC6B1kLQMAzAMAJo1a1bBwzIoEBEFE6r46BXf41gvDqyq+X7PPxSRp0WkgaruCLDueADjASA7O7tc\ngIoUK5qJiAILVXy02Pc4x4sDi0gjAFtVVUXkNFhLqLg0e2VOgYgosFDFR8sRoNgIvs5rqhqy85qI\nvAagJ4AGIpIL6wCXDtv4WQADAdwgIoUADgAYpKoVzgW4wYpmIqLAQhUfVaipqKr+Lsz7T8KarMYd\ncwpERIGFKj7a7DwXkeYAWqvqJyKSGWq7VMA6BSKiwNzMvPZHAG8BeM63qAmspVDKYk6BiCgwN8Nc\n3AQbNjsfAFR1HWxazpRVs6Y9MigQEZXmJigUqOoh54WIVEfgCuiUIWKVzQwKRESluQkKc0TkLgCZ\nInIegDcBvOdtsrzHORWIiMpzExRGAdgOYDmAPwH4UFX/6mmq4oBTchIRleemFVFnVX0ewPPOAhG5\nWFVTOrfAnAIRUXlucgrPi8gpzgsR+R2Au71LUnwwKBARlecmpzAQwFsichWAHgCuBtDX01TFASua\niYjKCxsUVHWDiAyC9U34AUBfVU350vhatYBffkl0KoiIkkskYx/VA5AGYKGIINzYR8muVi1gR7nx\nWImIqjbPxj5KdqxTICIqL1RQ2KWq+SJSL26piSPWKRARlRcqKLwKyy0shhUjid97CqCVh+nyHHMK\nRETlhRol9SLfY8v4JSd+2HmNiKi8UBXNXUJtqKpLYp+c+HGCQlERUM1Nbw0ioiogVPHRIyHeUwC9\nY5yWuHKGzz54sOQ5EVFVF6r4qFc8ExJv/lNyMigQEZkqW3DCiXaIiMqr8kGBlc1ERCWqfFBgToGI\nqETYsY+CtELaA2CzqhbGPknxwaBARFSem1FSnwbQBcAyWAe29r7n9UVkuKp+7GH6PONf0UxERMZN\n8dEm2EQ72araFUBnACsA9AHwkIdp8xTrFIiIynMTFE5W1ZXOC1VdBQsSG7xLlvdYfEREVJ6b4qM1\nIvIMgNd9r38LYK2I1ABw2LOUeYxBgYioPDc5hWsBrAdwK4A/A9jgW3YYQMp2cGOdAhFReW5mXjsg\nIk8A+Bg2vMUaVXVyCPu8TJyXmFMgIirPTZPUngBeglU4C4CmInKNqs71NmnecnIKrGgmIirhpk7h\nEdi8zGsAQEROAvAagK5eJsxr1aoBNWsyp0BE5M9NnUK6ExAAQFXXAkj3Lknxw4l2iIhKc5NTyBGR\nFwC84nt9FWw2tpTHKTmJKFUUFgLV3VyxK8hNTuEGACsB3AJgBIBVAIZ7mah4YU6BiFJF/frAXXd5\nfxw3rY8KADzq+3NNRCbC5njepqrtA7wvAP4DoD+A/QCujfdsbpySk4hSwb59QH4+UKeO98cKNR3n\nclgT1IBUtUOYfb8I4EkALwd5vx+A1r6/bgCe8T3GDXMKRJQK8vLs8bjjvD9WqJzCRRXZsarOFZEW\nIVYZAOBlVVUAC0Skrogcp6p5FTluJBgUiCgVJEVQUNXNHh+7MYAf/F7n+paVCwoiMgzAMABo1qxZ\nzBKQmQn8/HPMdkdE5Il4BoVETrIjAZYFLK5S1fG+UVqzGzZsGLMEsE6BiFJBVQkKuQCa+r1uAmBL\nPBPA4iMiSgV5eUBGBlCvnvfHchUURCRTRH4d42NPB3C1mO4A9sSzPgFgUCCi1JCXBzRqBEig8pUY\nCxsURORiAEsB/M/3upOITHex3WsAvgTwaxHJFZHrRWS4iDh9HD6Ejbi6HsDzAG6M8hyixqBARKkg\nLy8+RUeAux7N9wI4DcBsAFDVpWFaFcG33u/CvK8AbnJxfM84PZpV4xOBiYiikZcHtG4dn2O5KT4q\nVNU9nqckAZzhswsKEpsOIqJQ4plTcBMUVojIYABpItLaN7fCfI/TFRecU4GIkl1BgTWdT6agcDOA\ndgAKALwKYA9sFraUx6BARMnup5/sMZnqFH6tqn8F8FevExNvnJKTiJJdPPsoAO5yCo+KyLcicr+I\ntPM8RXHEnAIRJbukCwqq2gtATwDbAYwXkeUicrfXCYsHJyiwVzMRJaukCwoAoKo/qeo42DwKSwHc\n42mq4oQ5BSJKdnl5Nn3wMcfE53huOq+1EZF7RWQFbCjs+bAhKVIegwIRJbu8PAsIaWnxOZ6biuZJ\nAF4D0FdV4zo2kddY0UxEyS6efRQAdzOvdY9HQhKBdQpElOySJiiIyBRVvTLADGwCG6Ui3MxrSY/F\nR0SU7PLygC5d4ne8UDmFEb7HCs3AlswYFIgomR05AmzbFt+cQtCKZr9hrG9U1c3+f0jAiKZeYJ0C\nESWzbduAoqIkCQp+zguwrF+sE5IIaWk2cQWDAhElo3j3UQBC1yncAMsRtBKRZX5v1QbwhdcJixdO\nyUlEySqpggJs8LuPAPwLwCi/5XtVtdJMd9+gAbByZaJTQURUXiKCQqg6hT2quklVf+erRzgAa4WU\nJSLN4pZCj11/PfDpp8DXXyc6JUREpTlBoVGj+B3T1XScIrIOwEYAcwBsguUgKoUbbgCOPhp46KFE\np4SIqLS8PKBePaBGjfgd001F8wMAugNYq6otAZyLSlSnUKcOMHw4MGUK8N13iU4NEVGJeHdcA9wF\nhcOquhNANRGppqqzAHTyOF1xdeutQPXqwNixiU4JEVGJZA0Ku0UkC8BcAJNF5D8ACr1NVnwddxxw\n7bXApEklsxwRkRk6FLjjjkSnompK1qAwAFbJ/GcA/wPwHYCLvUxUIowcCRw6BIwbl+iUECWXmTOB\n2bMTnYqqR9VuUpMuKKjqL6p6RFULVfUlVR3nK06qVFq3BgYOBJ56CtizJ9Gp8caBA0DfvsCSJYlO\nCaWKAweArVuBH35IdEoqJicn9c7h55/tRjXpgoKI7BWR/DJ/P4jIOyLSKh6JjJc77wTy84Hnnkt0\nSryxcqXd9b35ZqJTQqli82Z73LoVKChIbFqipQr07w/85S+JTklkEtFHAXA5RzOAOwA0hk2uMxLA\n8wBeBzDRu6TFX9euQJ8+wGOPAQcPJjo1see0rsrJSWw6KHVs3Fjy/McfE5eOiti6Fdi+PfX6IiVz\nULhAVZ9T1b2qmq+q4wH0V9U3APzK4/TF3ahRVo73yiuJTknsbdhgjzk5dvdEFM6mTSXPU634xeGM\nWLBuXWqNc5bMQaFIRK4UkWq+vyv93qt0l5bevYHsbOvMduRIolMTW05Q2L275DmlpiefBGbN8v44\nlSkoFBWl1pA2yRwUrgIwFMA2AFt9z4eISCaA//MwbQkhYi2R1q8HZsxIdGpi67vvrPc2ACxenNi0\nUPS2bgVuvhk47zxg/Hhvj7VxI9C4sT1P5aDgzG+8bFnodZNJXh6QlWV/8eSm9dEGVb1YVRuoakPf\n8/WqekBVP49HIuPtssuAY48Fnnkm0SmJrQ0bgPPPt+HCWa8QW0eOACtW2N2o17780h7btAH+9Cer\nQPXquJs2Ae3a2VALqRoUVq0CunWzi+s33yQ6Ne4loo8C4K710Uki8qmIrPC97iAid3uftMTJyAD+\n8Afggw9KZ59T2aFD9qM++WSgY0cGhVgoKgLmz7e79saNgVNOAd5+2/vjzp9v39EFC2zsrocfBq68\n0pvy8k2bgBYtgKZNUzMoqFpOoX17+/8wKITnpvjoeQCjARwGAFVdBmCQl4lKBsOGWVGS19nzeNm8\n2S5iJ5xgdSaLF8fnrjZVzJgB9Ozprtnltm3WfLllS+DMM4EJE4CzzgJq1iy5i4/Gv//trrnwF19Y\nS7mjjrJ+NY8+CkydCvTqZUVLsbJvn7XaadnSgsL338du3/Hy00/Arl2W2+nQwYqPUqWRRTIHhVqq\nuqjMsko1zEUgzZoBF10EvPCC3WWnOqdiuVUrCwr5+VZvQua//wXmzAE++yz8ujffDDzyiN19vvKK\nXYjffNMuOtE2e9y2DfjrX4EHHgi9XkGB5fLOPNNeiwB//rMFheXLgXPPjd1Fz+mjkMo5BadiuV07\nyyHv3p0655HMQWGHiJwAX0sjERkIIC/0JkZELhCRNSKyXkRGBXj/WhHZLiJLfX9/iCj1HrvhBvux\nTp2a6JRUnNNHwckpACxC8jdvnj1OmxZ6vQMHrFjxj3+0xyFDSirvO3cGli6N7qI8ZYrVSyxbFnr8\nrSVL7CbljDNKL7/0UgsoK1fadzYWnKJTJyjs2gX88kts9h0vq1bZo5NTAFKjCGnvXvuskzUo3ATg\nOQAni8iPAG4FcEO4jUQkDcBTsPmc2wL4nYi0DbDqG6rayfc3wX3Svde3r91ZV4YK5w0brHijUSOg\nbVt7zhZI5ocf7K44I8OCQqhitZkz7cf6m9+Uf69zZ7sTde6wIzF5MvArX6+fTz4Jvt78+fZ4+unl\n33MuerFqdul0XHOKj4DUuct2rFxpleTHHlvy+aRCC6RENUcF3Lc+6gOgIYCTVbWHqm5yse/TAKz3\nbX8I1gN6QIVSG2fVqlnrjrlzU6t9cyDffWcBrlo1Gya8UyfmFBxOLuH//s/u0r/6Kvi6U6cCdeta\n/UNZnTvbY6RFSN99Z5XGf/kLUL8+8PHHwdedP9/+j4Fm4mrXzh5XrIjs+MFs2mQ3D8cck9pBoW1b\nK2arXds+u1TIKSR1UBCRGiIyGMAIAH8WkXtE5B4X+24MwP8rlOtbVtblIrJMRN4SkaauUh1Hv/+9\nzXqU6rmFDRvsB+HIzraiiMrWQS8a8+bZBWP0aGvP/u67gdc7fBiYPh245BIgPb38+6ecYttHGhRe\nfdUuWlddZcOszJwZuAhK1SqZnfqEsho1stxGrG5gnJZHIqkZFJyWR06wBEoqm5OdExSOPz7+x3ZT\nfDQNdodfCOAXv79wJMCysl/19wC0UNUOAD4B8FLAHYkME5EcEcnZvn27i0PHToMGwBVXAC+/bK0x\nEumLLywtkQ5Mpho4KOzbB6xdG9s0pqJ58+xC26CB5QCC1SvMmWPl6oGKjgAgM9Oa/EYSFFSt6Ojs\ns+3C27ev5VYC3e1v3GiV2mXrExwiVvkdy+Kjli3teePGtv9UCgp5eVac5x8UOnZMjeEukjqnAKCJ\nqv5WVR9S1UecPxfb5QLwv/NvAmCL/wqqulNVnUvc8wC6BtqRqo5X1WxVzW7YsKGLQ8fWDTdYxc+r\nr8b90MUOHACuvhp46y0rzorE9u0WAE44oWQZK5vNzp12ET3rLHs9YACwenXgYDl1KlCrll24g+nc\nObKgsGQJsGaN5RIA66UMBC5CcuoTggUFwC6AK1fGpgWSk1MALLd87LGpFRT8K5kdHTpYnVGsiti8\nkpdnn3nduvE/tpugMF9EToli318BaC0iLUUkA9a3Ybr/CiLiHwcvAbA6iuN47vTT7cv0zDOJa+P8\nwAN2t1+9OvDhh5Ft698c1XHyyXaBq+qVzV/4Zhv3DwpA+dxCURHwzjs2BHNmZvD9depko4m6zdC+\n+qoVRQ0caK+bNrX/zcyZ5dedP99aOvlf5Mpq187ujvNctQ8MLj/fxvN3goKTtlQKCk6Oqa1f85aO\nHe0x2YuQnOaoEqi8xWNugkIPAIt9TUuXichyEQn7kapqIWxspBmwi/0UVV0pImNE5BLfareIyEoR\n+QbALQCuje40vCUC3HijNTdcsCD+x1+50gbou+Yaa4f+0UeRbe8EBf+cQlqa3dVW9ZzCvHnW6ujU\nU+11s2b2uZStV1iwwIp1ghUdOSKpbD5yBHjtNQs0v/Ibb7hvXyuqKjt8+/z5QPfuJeP4BOIEjIoW\nITnNUZ3iIyA1g4LT8sjRsmVqDHeRqD4KgLug0A9AawB9YdNwXgSX03Gq6oeqepKqnqCq//Atu0dV\np/uej1bVdqraUVV7qeq30Z2G9666yn64o0dHn1vYvdvuviJRVGQtoOrUAcaOtQvImjWRjXLq9FHw\nv+sDrAjp66+Bwgi6In78cWICo1fmzQNOO81a2TguvdR6Jvv3Dp461YLHhReG3l+nTvboJijMnm0/\nfqfoyNG3rwUEJxcD2J378uWhi46A2AeFQDmFVOkR7FQy+99tV6tmDQKSPaewZUsSBwVV3RzoLx6J\nSyZZWTYMwZw5wEsBq8PDGzzYeklHYuJEuziMHWsVof362fJIcgsbNlhFYdlij+xsq3D71mUoLigA\nfvtba7pZGfzyixWfOUVHjgED7ML33nv2WtWCQp8+JR3VgqlXD2je3HKV4UyebK2eyn4nzjnHipT8\n6xUWLrQbhHBB4Zhj7HviVVDYt89uboIZOtR6hyeaqtUpBCpq69jRcgrJHNySPadAPtdfb61URo4E\nduyIbNuDB238+wUL3G+7dStwxx12kbjmGlvWujVw4omR1Ss4fRTKirSy+aOP7ILw9dfWCifVLVhg\nuaSyQaFDB7uwO/UK33xjLXHCFR053FQ2Hzxog+ddfnn5YJ2VZRd//6Awf77d5XbrFv74TmVzRWzc\naGMrNWhQsixcs9QdOywgPPxwxY4dC4FaHjk6drR52JN1LKcDByztDAopoFo14Nln7QsV6XyvixbZ\nhUAV+PRTd9vcfrvdzT77bOkscL9+FmAOHHC3n7LNUR0nnWQXILeVza++ap9BUVHkLaCS0bx59rmW\nvfsWsSKkmTPtznjqVDvvSy4JvJ+yOne21kuhmjC//74VCZUtOnL07Wu5DWfIivnzrdgjXE4FiE0L\nJP8+Co5wQcH5Hi1blvimzoEqmR3J3rPZGeaEQSFFtG9vF+tJk6woya05c+wHlpUVuGVJWTNnWvHC\n6NHWGsVf//4WENwc/+BBaw3jX8nsqFYN6NLFXU4hP9+KU/7wByt/dzNwXLKbN8/uGuvUKf/egAFW\nXDZjhgWFs88G3LaG7tTJLsjDdp0PAAAVnUlEQVShLjqTJ1tns169Ar/vNE395BOrkP7yy/BFR452\n7ez/VZE5lf2bozrCBQX/79Fbb0V/7FjwHwivrFN8bSnjWdl85Ig1XsjPD79uIvsoAAwKUbnnHvvB\nDB/uviPZ7Nl2sTjvvOA9Vh2qNvJl69YWFMo65xy7MLupV3DGrwmUUwCsCGnpUuutG8rUqRZgrrsO\n6NEjPlNBRmLt2pIKdTcOH7bio7JFR46zzrKGBWPH2gXGbdEREL4F0q5dVvw3aFDwlkRdulj9xMyZ\ndvy9eyMLCkDF2uJv2lS65RFgF6m0tNBB4aSTrIWUmyHAvbRypQ0Zcswx5d9LxHAX775rk3e1bAn8\n61+hc5HO95hBIYXUqmXj2H/7rbvy04ICy/737GlB4fvvrVdlMMuW2Zf69ttLt4pxZGbaXNJu6hUC\nNUf1l51tF3uno08wkyfbPrp1s2MvXx670Tgrav9+u+Pu39/9HBFLlth2wYJC9epWAey0tLr0Uvfp\nadLELkjBgsKLL9pIp0OHBt9HWpo1P/7445JWSMGGtyiroi2Qdu+2v7I5hbQ0G3YhVFA49VTrdb90\naWKHZncqmYO18+/YMb7FR/Pm2e/29NOBu+6y4DB2rH0HVe33NGaM3QxcfbU1NGjePH7p88egEKX+\n/e3L/8AD4b/8Tn3COedYCxYgdBHSlCn2Awx1d9qvnx03VHABSu46QuUUgNBFSHl5Vlw0eLD9yHr3\ntuWzZ4c+dryMG2dN+NaudVc0B5QMghcsKAAlgeC000qKTtwQCV7ZfOiQzcVwzjl2AQilb187rwkT\nrKip7EU6mAYNrG1+tEEhUMsjR7C+Cj/9BOTm2vfJ6YiXqCIkZ8yjQPUJDme4i3gNBf755/Y9ev99\nKwrs0sUakbRqZQ1HOnQA7r3Xbjgfftianfv3XYknBoUKePxx64p+882h15s92y4UZ51lX4DmzYMP\nj6wKvPGGXXhDlWH372+P4YqQNmyweoxg+zrhBCtTD1XZ/MYbdgc+eLC97trVsuDJUK+wcyfw4IM2\n9/SxxwJPPOFuu3nz7H8RaLRRx/nn2+fmtPyKROfOVnxTtlju1VetrH9UudlFynPqFZYssaKjSHq3\nVqQFUqCOa45mzQIHBef7k51t63TrlrgipC1brDFIqJ7fHTqUBA+v7dtnOacePex19+5WVzVvnn1e\nJ59sMzxu2WLBY+TIwJ99vDAoVMDxx9uP+3//s8gezJw5dmdSr579sM87zy6ogTqNLVlid/dXXhn6\n2K1aWfmtm6DQqlXwC0q1anaR//jj4IOETZ5sdzZOhXf16lbxmgxB4V//ssq7sWOtk9+HH4avWygq\nsh9fqFwCYE0yt2yxsa8i1bmz5QpW+w3cUlRkfV06drSAE07z5vY/BtzXJzjatbMilGhaIDn1UMFy\nCrm55febk2PfJafz3sCB9l0O1cnSq34CoSqZHc5wF/GoV1i0yCqayxb/9ehhOQdn0qZQNyjxxKBQ\nQddea0U9L74Y+H3/+gTHeefZhSzQuP1TpthF103FZv/+VuEbasTHYH0U/N1xh/14//jH8j/UtWvt\nB1+26WTv3pb9TuSwB5s3W87gmmusVdif/mT/i6efDr3d6tXWszxcUADsfxHN+DOBKpvfe8/qoe68\n0/0+ncH33NYnONq1szvUaNrib9pkOcFAxRdNm9p3uuzYTjk5QJs2lisFwhchbdxoObtWrWz2uqee\nirx3fTCBBsIrq0WL+A138cUX9v8ONDFSMmJQqKDjjrPy/ZdfDjw3wVdfWfNR/6Bw7rn2JSlb/q1q\nQeG88yxXEU6/fvYDDVa27wyZHayS2XHBBcD991vRxuOPl37PGet/0KDSy516hUS2Qvr73y1tY8bY\n6+OPt85gEyeGLit26hPOPtu7tLVubeXDTlBQtVxNy5ZWF+XW8OHWDLhrwPGDg6tIZbPT8ihQ4ArU\nLFXVvudO/RRgF91TTw1chFRYaEWRhw5Z8PzsM+sl36WLjQpa0R7zTsujUMWv1arFb26FL76w/0ci\nRjyNBoNCDFx3nRUzBBru2L8+wVG/vv0AygaFr76yH2S4oiPH2WfbhSdYK6S8PKvgDpdTAKzp62WX\nWa7BKRZyxvrv3bv8ZB8dOljgSlRQWL7cAvEtt5SuBL75Zms5M3ly8G3nzbNg7uZziVZamn1GTlCY\nO9eGqhg50nIfbrVrBzz/fOBJfcJtB0QXFDZuDF6p7XzW/jmQH3+03vf+QQGw3EJOTklxlGPMGGvV\n9eyz1qv7xx/te//aa3aj89RTkfUBKivQmEeBOMNduG2xFsiaNaG3d/qYRJrTSyhVTam/rl27arIp\nKFBt0ED1iivKv3fuuaodO5ZfPmqUavXqqvn5Jctuu001PV111y73x77oItVWrVSLisq/N2+eKqD6\nv/+521d+vmqbNqr166tu2qS6cKFt/8ILgde//HLVZs0CH9trF16oWreu6s6dpZcXFal26qTavn3g\ndK1aZdv99rfep/GGG1SPPlr1yBHVfv1UGzZU3b/f++M6jjtO9ZprItumqEi1dm3VW24J/P7Wrfad\n+M9/Spa9844t+/LL0utu2GDLH364ZNncuarVqgVP1/79qo0aqfbqFVm6/dN/9NH22Yfz+uuWvpdf\nju5Yy5apiqhOmBB8nW++qdgxYglAjrq4xjKnEAMZGVbmPm2atYZxHDpUvj7B0aePZaOdop+iIstq\nn39+ZNnM/v2tiCjQsAKB5lEIpXZt62RTWGi5hgkTrHXV5ZcHXr93b7tjjGTE1liYM8cq50aPLl/M\nJmK5hRUryg/FsXatpblmTSsu81rnzlZ3NG2aNQgYMSL0XAyxFk0LpF27rKNcsNYvDRvad8K/+Cgn\nx3JGTuWto2VLK/ZyipB27bLfScuWwVuJZWZancusWeFzC//8J3DxxTbkzKRJlhP79lv7zEPVJziu\nuMKKuO680845UlOnWm461ACZkfYxSQpuIkcy/SVjTkFVdelSuyMYN65k2eef27J33im//oEDqjVr\nltyRffGFrfvKK5Edd+NG2+7++8u/d889dldWUBDZPt9/3+6AAMsNBLN6ta0zfnxk+6+IoiLVbt1U\nmzQJfte9f79qvXql075+vWrjxna3vnJlfNL61Vf2+TRqpJqVpfrzz/E5rmPECNVatSyn4lZOTvDv\nrOOEE0rntM4/P3BuWFX1wQdtf5s2qV55peWOFy4MnQY3uYUPP7T9NmmiWqOGPff/++yz0MdwLFhg\n6995p7v1/XXqVHK8zZsDrzN4sJ1LInLTZcFlTiHhF/lI/5I1KKiqdu5sf44HHrBPeMeOwOv37WvF\nNaoWHGrUUN2zJ/LjXnyxbbt0aenlQ4aoNm8e+f5UVceMsbRPnx58naIiK6IYNCi6Y0Tj7bc1ZJGW\n4y9/UU1LU/3+e7sgNWtmxWLLlsUnnaoW+NPSLL233x6/4zqef96O/d135d+bPl312WfLL3/rLdvm\n66+D77dnT9UzzrDnRUX2uV5/feB116+3/Z1xhj3+85/u0v7YY7b+7Nnl39u+3S607dvbZ1xYqLpu\nneq0aRaERo9WPXTI3XFUrSgrPV117Vr32zg3YzfeaI8PPhh4vebNVQcOdL9fLzEoJMC4cfaJOhfn\nPn1UO3QIvv7DD9v6339vF9dLL43uuNu22fYnn6y6b1/J8jPOqFjZ7Jo14dcbPFj1mGPicyd0+LCd\nY5s2diEIZeNGyyVdd51qy5ZWj7BkifdpLOuUU+yCk5sb/2PPnx84sC9YYGkCVF98sfR7Y8fa8lD1\nWkOHqjZtas+di+MzzwRfv3NnW6dnz/D/N0ew3EJRkeUA09PL3wRFa8sWq0e58EL32zz+uJ3T2rWq\np58e+Heem2vrPPZYbNJZUQwKCbBjh2pGhmXbCwpUMzODV9iplhQ5/f739vjaa9Ef+5NPrMjH/47t\n2GNV//CH6PfpxoQJlvYVK9ytv2eP6qJF0QWRF16wY02d6m79AQNs/aOPtmMmwuTJqk89lZhj795t\n5/+vf5Us27bNilxatLCLdEZG6Qrim26yABrKXXdZDqiwUPXNN+0YX30VfP0nn7Sblh9+iCz9gXIL\nL78c+s48Ws4N2gcfuFu/Vy/Vdu3s+RNP2LbLl5de5403bHmivntlMSgkyBVXWHZ61qzwF7AjR+wu\nG7AAsndvxY49erTt6/XXLccQSXY9Wk4LE/+6lGC2brU7KsByUaGKKMo6cMAuZqed5j6gLFpkx/vi\nC/fHqWyaNLFiRFXLafXubUWNixfbTUyrVnZH7uRkLrzQyspDeeYZ+x/+8IOVxWdkqB48GHqbSOo1\nHGVzC5s3W4Dv0cN9jsOtggLVk05Sbd06fB3cjh0WFP/6V3u9dau9Hj269Hq33GJ1OpEUZXmJQSFB\nnAqwU0/VkPUJjsGDbb1YlDseOqTavbv9cKZNs/2+8UbF9xtOixaql10Wep0ff7Sin8xM1TvusIpg\nEdWrrw5eSefvkUc0ogpEMuefX1LPNWqUfYYTJ5a8v2KFVYJnZ9tFuF278P/L99+3/cyfb02us7O9\nS7+TW5g1y4JDVpbdiHjho4/sWA89FHq9l14qnwO44AKrP/APfl27Wm4sWTAoJEhhoerxx9sne8op\n4defNMnWnTIlNsffuFG1Th0LDOGy9bHy+99bkUOwSvJNm6zFSlaW6pw5tmzXLqsMrlHD/u68s3Sf\nDX+7d1vuq29fb9Jfmd12m7Vycyrohw0rv86779p7V12letRRqn/+c+h9LltWkiOtU0d1+HBv0q5a\nklv41a/UVQODirr4YvuebtkSfJ3LLrOWbP4B4JVXLH2ff26v9+4tnZtIBgwKCeTckd18c/h1Cwrs\nCxXL7PCUKVrcVK5s5y4vfPCBHSsry87Zv4J63Tpr+VO3rlVwlrVpkxVvAJaTWL26/Dp/+5u9n5Pj\n3TlUVk49TI0adkd/4EDg9e6/v+Q7498xLZBdu2y94cPtMVTnrVhwcguXXOJ9g4Z166w4LFjnxl9+\nsdzujTeWXp6fX3r5p59amj/80Nv0RoJBIYHWrbM7rk8+SVwabrpJ9cQT43e8RYusVYrTqqVfP8tm\nH3ec3eWHa/nz6afWhyAry5pFOn76yT7LK6/0Nv2VldMO3+mlHkxRkdWHhWuG7KyblWV38P6t7bxy\n8KBV5sbjBke1pDn2e++Vf88plv344/LvDRpkIxscOqR6331WPBrJ6AReY1BIsGgq1mKpqMgqFuMt\nL89+EM4F49hj3bdM+v57q0gGrN7h8GHLeaSluWseS+UdOGBFIm7qYvbtU3366eC5CX9t2tj/qWbN\n5KlIjZWCAusD0aRJ+SLR666zIrNAldHTp2txC6a+fd0VH8cTgwIlVEGBtbzauDGy7Q4eLCmW6NHD\nch6BysEpsfr2tf9R9+6JTok3FiywO33/YqLCQssJDB4ceJuCAmtAMWiQ9Xvwsq4lGm6DAsc+Ik9k\nZNj4SW6nkHTUqAE884yNZeOMqXPPPZ4kkSrAGS217MiolUW3bjYC79NPl4xfNH8+sGNH8Pm6MzJs\nPKU33rCxlFJqvCM/DAqUlK691mbu+uwzoHHjRKeGyqrsQQGw+debN7f5LA4etMEiMzJs/pFgrrqq\nZKKqVA0KEYzsThRfbdokOgUUTOvW9titW2LT4aWsLOC55ywI/OMfFhT69LHRhIM580wLmIWFkeeS\nkwWDAhFF7Morbf5oZ97uyur884GhQ22Y7qIiG2Y7lGrVLJDs3x/dNK7JQNTJ66SI7OxszcnJSXQy\niKiK2LHDcq07d9oMi40aJTpF0RGRxaoatsCPOQUiohAaNLDK4yVLUjcgRIJBgYgojN697a8qYOsj\nIiIq5mlQEJELRGSNiKwXkVEB3q8hIm/43l8oIi28TA8REYXmWVAQkTQATwHoB6AtgN+JSNsyq10P\nYJeqngjgMQD/9io9REQUnpc5hdMArFfVDap6CMDrAAaUWWcAgJd8z98CcK5IqjbkIiJKfV4GhcYA\nfvB7netbFnAdVS0EsAdAfQ/TREREIXgZFALd8ZftFOFmHYjIMBHJEZGc7du3xyRxRERUnpdBIRdA\nU7/XTQBsCbaOiFQHUAfAz2V3pKrjVTVbVbMbNmzoUXKJiMjLoPAVgNYi0lJEMgAMAjC9zDrTAVzj\nez4QwGeaal2siYgqEU+HuRCR/gAeB5AGYKKq/kNExsDG9Z4uIjUBvAKgMyyHMEhVN4TZ53YAm6NM\nUgMAO6LcNtlV1nPjeaWeynpuqX5ezVU1bFFLyo19VBEikuNm7I9UVFnPjeeVeirruVXW8yqLPZqJ\niKgYgwIRERWrakFhfKIT4KHKem48r9RTWc+tsp5XKVWqToGIiEKrajkFIiIKocoEhXAjtqYSEZko\nIttEZIXfsnoiMlNE1vkef5XINEZDRJqKyCwRWS0iK0VkhG95Sp+biNQUkUUi8o3vvO7zLW/pGx14\nnW+04IxEpzUaIpImIl+LyPu+15XlvDaJyHIRWSoiOb5lKf1ddKNKBAWXI7amkhcBXFBm2SgAn6pq\nawCf+l6nmkIAt6tqGwDdAdzk+z+l+rkVAOitqh0BdAJwgYh0h40K/JjvvHbBRg1ORSMArPZ7XVnO\nCwB6qWonv6aoqf5dDKtKBAW4G7E1ZajqXJQfDsR/xNmXAFwa10TFgKrmqeoS3/O9sAtNY6T4uanZ\n53uZ7vtTAL1howMDKXheACAiTQBcCGCC77WgEpxXCCn9XXSjqgQFNyO2prpjVTUPsIsrgGMSnJ4K\n8U241BnAQlSCc/MVsSwFsA3ATADfAdjtGx0YSN3v5OMA/gKgyPe6PirHeQEWuD8WkcUiMsy3LOW/\ni+FUlTmaXY3GSslBRLIAvA3gVlXNrwxTbKjqEQCdRKQugHcAtAm0WnxTVTEichGAbaq6WER6OosD\nrJpS5+XnTFXdIiLHAJgpIt8mOkHxUFVyCm5GbE11W0XkOADwPW5LcHqiIiLpsIAwWVWn+hZXinMD\nAFXdDWA2rM6krm90YCA1v5NnArhERDbBimR7w3IOqX5eAABV3eJ73AYL5KehEn0Xg6kqQcHNiK2p\nzn/E2WsATEtgWqLiK49+AcBqVX3U762UPjcRaejLIUBEMgH0gdWXzIKNDgyk4Hmp6mhVbaKqLWC/\nqc9U9Sqk+HkBgIgcJSK1necA+gJYgRT/LrpRZTqvBRqxNcFJipqIvAagJ2zUxq0A/g7gXQBTADQD\n8D2AK1S13NwUyUxEegCYB2A5Ssqo74LVK6TsuYlIB1ilZBrsRmyKqo4RkVawO+x6AL4GMERVCxKX\n0uj5io9GqupFleG8fOfwju9ldQCv+kZ5ro8U/i66UWWCAhERhVdVio+IiMgFBgUiIirGoEBERMUY\nFIiIqBiDAhERFWNQICKiYgwKRERUjEGBiIiK/T9/juTov5JjbQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f155c4028d0>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "plt.plot(range(len(train_losses)), train_losses, color='blue')\n",
    "plt.legend(['Train Loss'], loc='upper right')\n",
    "plt.ylabel('negative log likelihood loss')\n",
    "fig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As our first attemp, we obtained over 92% accuracy, which is not so bad, but soon we will see how to vastly improve on this result!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Multilayer Perceptron for MNIST dataset\n",
    "\n",
    "We will try to improve the previous results by training a deeper network. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train MLP\n",
    "First of all, define the forward pass. We propose to use one hidden FC layer with 50 units, and an output layer with `log_softmax` activation, but feel free to create the network you prefer!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class MLP(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(MLP, self).__init__()\n",
    "        self.fc1 = nn.Linear(28*28, 50)\n",
    "        self.fc2 = nn.Linear(50, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x.view(-1, 28*28)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = self.fc2(x)\n",
    "        return F.log_softmax(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we initialize the network and choose the optimizer. We propose using SGD optimizer. Set the `learning_rate` to 0.01 and the `momentum` to 0.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 0.01\n",
    "momentum = 0.5\n",
    "model = MLP().to(device)\n",
    "optimizer = optim.SGD(model.parameters(), lr=learning_rate,\n",
    "                      momentum=momentum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train the model during 3 epochs. For monitoring purposes, print the training loss every 100 batches and store it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/roi/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:11: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  # This is added back by InteractiveShellApp.init_path()\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Epoch: 1 [0/60000 (0%)]\tLoss: 2.333978\n",
      "Train Epoch: 1 [3200/60000 (5%)]\tLoss: 0.702726\n",
      "Train Epoch: 1 [6400/60000 (11%)]\tLoss: 0.553314\n",
      "Train Epoch: 1 [9600/60000 (16%)]\tLoss: 0.447418\n",
      "Train Epoch: 1 [12800/60000 (21%)]\tLoss: 0.381252\n",
      "Train Epoch: 1 [16000/60000 (27%)]\tLoss: 0.444157\n",
      "Train Epoch: 1 [19200/60000 (32%)]\tLoss: 0.156283\n",
      "Train Epoch: 1 [22400/60000 (37%)]\tLoss: 0.153781\n",
      "Train Epoch: 1 [25600/60000 (43%)]\tLoss: 0.273984\n",
      "Train Epoch: 1 [28800/60000 (48%)]\tLoss: 0.191273\n",
      "Train Epoch: 1 [32000/60000 (53%)]\tLoss: 0.143926\n",
      "Train Epoch: 1 [35200/60000 (59%)]\tLoss: 0.249884\n",
      "Train Epoch: 1 [38400/60000 (64%)]\tLoss: 0.378720\n",
      "Train Epoch: 1 [41600/60000 (69%)]\tLoss: 0.334598\n",
      "Train Epoch: 1 [44800/60000 (75%)]\tLoss: 0.330522\n",
      "Train Epoch: 1 [48000/60000 (80%)]\tLoss: 0.350831\n",
      "Train Epoch: 1 [51200/60000 (85%)]\tLoss: 0.089973\n",
      "Train Epoch: 1 [54400/60000 (91%)]\tLoss: 0.144599\n",
      "Train Epoch: 1 [57600/60000 (96%)]\tLoss: 0.167678\n",
      "\n",
      "Test set: Average loss: 0.2270, Accuracy: 9338/10000 (93%)\n",
      "\n",
      "Train Epoch: 2 [0/60000 (0%)]\tLoss: 0.238733\n",
      "Train Epoch: 2 [3200/60000 (5%)]\tLoss: 0.182948\n",
      "Train Epoch: 2 [6400/60000 (11%)]\tLoss: 0.541300\n",
      "Train Epoch: 2 [9600/60000 (16%)]\tLoss: 0.174779\n",
      "Train Epoch: 2 [12800/60000 (21%)]\tLoss: 0.268026\n",
      "Train Epoch: 2 [16000/60000 (27%)]\tLoss: 0.129040\n",
      "Train Epoch: 2 [19200/60000 (32%)]\tLoss: 0.121549\n",
      "Train Epoch: 2 [22400/60000 (37%)]\tLoss: 0.157314\n",
      "Train Epoch: 2 [25600/60000 (43%)]\tLoss: 0.121449\n",
      "Train Epoch: 2 [28800/60000 (48%)]\tLoss: 0.246961\n",
      "Train Epoch: 2 [32000/60000 (53%)]\tLoss: 0.138672\n",
      "Train Epoch: 2 [35200/60000 (59%)]\tLoss: 0.158757\n",
      "Train Epoch: 2 [38400/60000 (64%)]\tLoss: 0.228017\n",
      "Train Epoch: 2 [41600/60000 (69%)]\tLoss: 0.337682\n",
      "Train Epoch: 2 [44800/60000 (75%)]\tLoss: 0.039425\n",
      "Train Epoch: 2 [48000/60000 (80%)]\tLoss: 0.180993\n",
      "Train Epoch: 2 [51200/60000 (85%)]\tLoss: 0.202001\n",
      "Train Epoch: 2 [54400/60000 (91%)]\tLoss: 0.155060\n",
      "Train Epoch: 2 [57600/60000 (96%)]\tLoss: 0.165280\n",
      "\n",
      "Test set: Average loss: 0.1617, Accuracy: 9530/10000 (95%)\n",
      "\n",
      "Train Epoch: 3 [0/60000 (0%)]\tLoss: 0.284827\n",
      "Train Epoch: 3 [3200/60000 (5%)]\tLoss: 0.183600\n",
      "Train Epoch: 3 [6400/60000 (11%)]\tLoss: 0.086945\n",
      "Train Epoch: 3 [9600/60000 (16%)]\tLoss: 0.109416\n",
      "Train Epoch: 3 [12800/60000 (21%)]\tLoss: 0.261974\n",
      "Train Epoch: 3 [16000/60000 (27%)]\tLoss: 0.176709\n",
      "Train Epoch: 3 [19200/60000 (32%)]\tLoss: 0.064136\n",
      "Train Epoch: 3 [22400/60000 (37%)]\tLoss: 0.144867\n",
      "Train Epoch: 3 [25600/60000 (43%)]\tLoss: 0.144628\n",
      "Train Epoch: 3 [28800/60000 (48%)]\tLoss: 0.064711\n",
      "Train Epoch: 3 [32000/60000 (53%)]\tLoss: 0.060300\n",
      "Train Epoch: 3 [35200/60000 (59%)]\tLoss: 0.130378\n",
      "Train Epoch: 3 [38400/60000 (64%)]\tLoss: 0.058080\n",
      "Train Epoch: 3 [41600/60000 (69%)]\tLoss: 0.433603\n",
      "Train Epoch: 3 [44800/60000 (75%)]\tLoss: 0.157228\n",
      "Train Epoch: 3 [48000/60000 (80%)]\tLoss: 0.085178\n",
      "Train Epoch: 3 [51200/60000 (85%)]\tLoss: 0.123862\n",
      "Train Epoch: 3 [54400/60000 (91%)]\tLoss: 0.086536\n",
      "Train Epoch: 3 [57600/60000 (96%)]\tLoss: 0.122730\n",
      "\n",
      "Test set: Average loss: 0.1360, Accuracy: 9592/10000 (96%)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "training_losses = []\n",
    "for epoch in range(1, epochs + 1):\n",
    "    train(model, device, train_loader, optimizer, epoch, training_losses)\n",
    "    test(model, device, test_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the training loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAD8CAYAAACYebj1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl4VOXZBvD7ISQQDEJZFGVHsbLI\nGgUVFRBRcEErWorgUluK+ilWsYK1VtG2VnEp7ojgUlxQUXArorIpAgZEVllk0UhkEwgIBEKe749n\nTjJJZjkzmTNLcv+uK9fMnDnLeyYz5znvLqoKIiIiAKiW6AQQEVHyYFAgIqJiDApERFSMQYGIiIox\nKBARUTEGBSIiKsagQERExRgUiIioGIMCEREVq57oBESqQYMG2qJFi0Qng4gopSxevHiHqjYMt17K\nBYUWLVogJycn0ckgIkopIrLZzXosPiIiomKeBQURaSois0RktYisFJERAdbpKSJ7RGSp7+8er9JD\nRETheVl8VAjgdlVdIiK1ASwWkZmquqrMevNU9SIP00FERC55FhRUNQ9Anu/5XhFZDaAxgLJBgYiq\noMOHDyM3NxcHDx5MdFIqlZo1a6JJkyZIT0+Pavu4VDSLSAsAnQEsDPD26SLyDYAtAEaq6soA2w8D\nMAwAmjVr5l1CiShucnNzUbt2bbRo0QIikujkVAqqip07dyI3NxctW7aMah+eVzSLSBaAtwHcqqr5\nZd5eAqC5qnYE8ASAdwPtQ1XHq2q2qmY3bBi2RRURpYCDBw+ifv36DAgxJCKoX79+hXJfngYFEUmH\nBYTJqjq17Puqmq+q+3zPPwSQLiINvEwTESUPBoTYq+hn6mXrIwHwAoDVqvpokHUa+daDiJzmS89O\nL9KzYgXwt78B27d7sXciosrBy5zCmQCGAujt1+S0v4gMF5HhvnUGAljhq1MYB2CQejRp9Jo1wAMP\nAHl5XuydiFLNzp070alTJ3Tq1AmNGjVC48aNi18fOnTI1T6uu+46rFmzxvUxJ0yYgFtvvTXaJMeF\nl62PPgcQMh+jqk8CeNKrNPirVcse9++Px9GIKNnVr18fS5cuBQDce++9yMrKwsiRI0uto6pQVVSr\nFvj+edKkSZ6nM96qTI9mJygcOJDYdBBRclu/fj3at2+P4cOHo0uXLsjLy8OwYcOQnZ2Ndu3aYcyY\nMcXr9ujRA0uXLkVhYSHq1q2LUaNGoWPHjjj99NOxbds218f873//i1NOOQXt27fHXXfdBQAoLCzE\n0KFDi5ePGzcOAPDYY4+hbdu26NixI4YMGRLbk0cKjn0UrcxMe2ROgSj53Hor4Ltpj5lOnYDHH49u\n21WrVmHSpEl49tlnAQAPPvgg6tWrh8LCQvTq1QsDBw5E27ZtS22zZ88enHPOOXjwwQdx2223YeLE\niRg1alTYY+Xm5uLuu+9GTk4O6tSpgz59+uD9999Hw4YNsWPHDixfvhwAsHv3bgDAQw89hM2bNyMj\nI6N4WSxVuZwCgwIRhXPCCSfg1FNPLX792muvoUuXLujSpQtWr16NVavK98HNzMxEv379AABdu3bF\npk2bXB1r4cKF6N27Nxo0aID09HQMHjwYc+fOxYknnog1a9ZgxIgRmDFjBurUqQMAaNeuHYYMGYLJ\nkydH3UEtlCqTU2BQIEpe0d7Re+Woo44qfr5u3Tr85z//waJFi1C3bl0MGTIkYD+AjIyM4udpaWko\nLCx0daxgbWvq16+PZcuW4aOPPsK4cePw9ttvY/z48ZgxYwbmzJmDadOm4YEHHsCKFSuQlpYW4RkG\nV+VyCqxTIKJI5Ofno3bt2jj66KORl5eHGTNmxHT/3bt3x6xZs7Bz504UFhbi9ddfxznnnIPt27dD\nVXHFFVfgvvvuw5IlS3DkyBHk5uaid+/eePjhh7F9+3bsj/GdbpXJKbBOgYii0aVLF7Rt2xbt27dH\nq1atcOaZZ1Zofy+88ALeeuut4tc5OTkYM2YMevbsCVXFxRdfjAsvvBBLlizB9ddfD1WFiODf//43\nCgsLMXjwYOzduxdFRUW48847Ubt27YqeYiniUbcAz2RnZ2s0k+wcPgxkZAD33w/cfbcHCSOiiKxe\nvRpt2rRJdDIqpUCfrYgsVtXscNtWmeKj9HSgenXmFIiIQqkyQQGwegUGBSKi4KpcUGBFM1HySLXi\n61RQ0c+0SgWFzEzmFIiSRc2aNbFz504Ghhhy5lOoWbNm1PuoMq2PABYfESWTJk2aIDc3F9s5dHFM\nOTOvRYtBgYgSIj09PerZwcg7Var4iEGBiCi0KhcUWNFMRBRclQoKrGgmIgqtSgUFFh8REYXGoEBE\nRMWqXFBgnQIRUXBVLigwp0BEFFyVCgqZmUBhoY2YSkRE5VWpoMDZ14iIQmNQICKiYlUyKLCymYgo\nsCoVFDglJxFRaFUqKLD4iIgoNAYFIiIqViWDAusUiIgCq5JBgTkFIqLAqlRQYEUzEVFoYYOCiBwl\nItV8z08SkUtEJN37pMUecwpERKG5ySnMBVBTRBoD+BTAdQBe9DJRXmFQICIKzU1QEFXdD+A3AJ5Q\n1csAtA27kUhTEZklIqtFZKWIjAiwjojIOBFZLyLLRKRL5KfgHiuaiYhCcxUUROR0AFcB+MC3rLqL\n7QoB3K6qbQB0B3CTiJQNJv0AtPb9DQPwjKtURyk9HUhLY06BiCgYN0HhVgCjAbyjqitFpBWAWeE2\nUtU8VV3ie74XwGoAjcusNgDAy2oWAKgrIsdFdAYREOGUnEREoYS941fVOQDmAICvwnmHqt4SyUFE\npAWAzgAWlnmrMYAf/F7n+pblRbL/SHBOBSKi4Ny0PnpVRI4WkaMArAKwRkTucHsAEckC8DaAW1U1\nv+zbATbRAPsYJiI5IpKzfft2t4cOiEGBiCg4N8VHbX0X80sBfAigGYChbnbua7r6NoDJqjo1wCq5\nAJr6vW4CYEvZlVR1vKpmq2p2w4YN3Rw6KE7JSUQUnJugkO67uF8KYJqqHkaAu/myREQAvABgtao+\nGmS16QCu9rVC6g5gj6p6VnQEsE6BiCgUN62IngOwCcA3AOaKSHMAZYuBAjkTlqNYLiJLfcvuguU0\noKrPwnIe/QGsB7Af1gfCUyw+IiIKzk1F8zgA4/wWbRaRXi62+xyB6wz811EAN4XbVyzVqgXs2hXP\nIxIRpQ43Fc11RORRp6JXRB4BcFQc0uYJ1ikQEQXnpk5hIoC9AK70/eUDmORlorzE4iMiouDc1Cmc\noKqX+72+z6+OIOWwopmIKDg3OYUDItLDeSEiZwJI2QIY5hSIiIJzk1O4AcBLIlIHVnH8M4BrvUyU\nlxgUiIiCc9P6aCmAjiJytO+1m+aoSatWLeDwYaCwEKjuJiQSEVUhQS+LInJbkOUAgBAd0pKa//DZ\ntWsnNi1ERMkm1L1ypbxk+k/JyaBARFRa0KCgqvfFMyHxwtnXiIiCc9P6qFJhUCAiCq7KBgX2aiYi\nKq/KBgXmFIiIyou49ZEjVVsf+Vc0ExFRaW5aH/0awKmwuQ8A4GIAc71MlJeYUyAiCi5s6yMR+RhA\nF1Xd63t9L4A345I6D7BOgYgoODd1Cs0AHPJ7fQhAC09SEwfMKRARBedmoIdXACwSkXd8ry8F8JJ3\nSfIW6xSIiIJzM/bRP0TkIwBnweZmvk5Vv/Y8ZR5hToGIKDi3Q8IdAVAECwpF3iXHezVqACIMCkRE\ngbiZjnMEgMkAGgA4BsB/ReRmrxPmFRFOyUlEFIybnML1ALqp6i8AICL/BvAlgCe8TJiXOKcCEVFg\nblofCaz4yHHEtyxlcUpOIqLA3OQUJgFY6Gt9JAAGAHjB01R5jDkFIqLA3LQ+elREZgNw5mlO6dZH\nAIMCEVEwkbQ+UlSC1kcAK5qJiIKpcq2PAOYUiIiCqZKtjzIzgZ9+SnQqiIiST5VsfcScAhFRYJG2\nPgJs7KOUb33EOgUiovLctj6aA+BMWA6BrY+IiCopt62PlgLIc9YXkWaq+r1nqfIYgwIRUWBhg4Kv\npdHfAWxFSX2CAujgbdK8k5kJFBQAR44AaWmJTg0RUfJwk1MYAeDXqrrT68TEi//sa1lZiU0LEVEy\ncdP66AcAeyLdsYhMFJFtIrIiyPs9RWSPiCz1/d0T6TGixSk5iYgCC5pTEJHbfE83AJgtIh8AKHDe\nV9VHw+z7RQBPAng5xDrzVPUid0mNHU60Q0QUWKjio9q+x+99fxm+P1dUda6ItIg6ZR7ilJxERIEF\nDQqqel8cjn+6iHwDYAuAkaq6Mg7HZE6BiCiIUMVHj6vqrSLyHqy1USmqekkFj70EQHNV3Sci/QG8\nC6B1kLQMAzAMAJo1a1bBwzIoEBEFE6r46BXf41gvDqyq+X7PPxSRp0WkgaruCLDueADjASA7O7tc\ngIoUK5qJiAILVXy02Pc4x4sDi0gjAFtVVUXkNFhLqLg0e2VOgYgosFDFR8sRoNgIvs5rqhqy85qI\nvAagJ4AGIpIL6wCXDtv4WQADAdwgIoUADgAYpKoVzgW4wYpmIqLAQhUfVaipqKr+Lsz7T8KarMYd\ncwpERIGFKj7a7DwXkeYAWqvqJyKSGWq7VMA6BSKiwNzMvPZHAG8BeM63qAmspVDKYk6BiCgwN8Nc\n3AQbNjsfAFR1HWxazpRVs6Y9MigQEZXmJigUqOoh54WIVEfgCuiUIWKVzQwKRESluQkKc0TkLgCZ\nInIegDcBvOdtsrzHORWIiMpzExRGAdgOYDmAPwH4UFX/6mmq4oBTchIRleemFVFnVX0ewPPOAhG5\nWFVTOrfAnAIRUXlucgrPi8gpzgsR+R2Au71LUnwwKBARlecmpzAQwFsichWAHgCuBtDX01TFASua\niYjKCxsUVHWDiAyC9U34AUBfVU350vhatYBffkl0KoiIkkskYx/VA5AGYKGIINzYR8muVi1gR7nx\nWImIqjbPxj5KdqxTICIqL1RQ2KWq+SJSL26piSPWKRARlRcqKLwKyy0shhUjid97CqCVh+nyHHMK\nRETlhRol9SLfY8v4JSd+2HmNiKi8UBXNXUJtqKpLYp+c+HGCQlERUM1Nbw0ioiogVPHRIyHeUwC9\nY5yWuHKGzz54sOQ5EVFVF6r4qFc8ExJv/lNyMigQEZkqW3DCiXaIiMqr8kGBlc1ERCWqfFBgToGI\nqETYsY+CtELaA2CzqhbGPknxwaBARFSem1FSnwbQBcAyWAe29r7n9UVkuKp+7GH6PONf0UxERMZN\n8dEm2EQ72araFUBnACsA9AHwkIdp8xTrFIiIynMTFE5W1ZXOC1VdBQsSG7xLlvdYfEREVJ6b4qM1\nIvIMgNd9r38LYK2I1ABw2LOUeYxBgYioPDc5hWsBrAdwK4A/A9jgW3YYQMp2cGOdAhFReW5mXjsg\nIk8A+Bg2vMUaVXVyCPu8TJyXmFMgIirPTZPUngBeglU4C4CmInKNqs71NmnecnIKrGgmIirhpk7h\nEdi8zGsAQEROAvAagK5eJsxr1aoBNWsyp0BE5M9NnUK6ExAAQFXXAkj3Lknxw4l2iIhKc5NTyBGR\nFwC84nt9FWw2tpTHKTmJKFUUFgLV3VyxK8hNTuEGACsB3AJgBIBVAIZ7mah4YU6BiFJF/frAXXd5\nfxw3rY8KADzq+3NNRCbC5njepqrtA7wvAP4DoD+A/QCujfdsbpySk4hSwb59QH4+UKeO98cKNR3n\nclgT1IBUtUOYfb8I4EkALwd5vx+A1r6/bgCe8T3GDXMKRJQK8vLs8bjjvD9WqJzCRRXZsarOFZEW\nIVYZAOBlVVUAC0Skrogcp6p5FTluJBgUiCgVJEVQUNXNHh+7MYAf/F7n+paVCwoiMgzAMABo1qxZ\nzBKQmQn8/HPMdkdE5Il4BoVETrIjAZYFLK5S1fG+UVqzGzZsGLMEsE6BiFJBVQkKuQCa+r1uAmBL\nPBPA4iMiSgV5eUBGBlCvnvfHchUURCRTRH4d42NPB3C1mO4A9sSzPgFgUCCi1JCXBzRqBEig8pUY\nCxsURORiAEsB/M/3upOITHex3WsAvgTwaxHJFZHrRWS4iDh9HD6Ejbi6HsDzAG6M8hyixqBARKkg\nLy8+RUeAux7N9wI4DcBsAFDVpWFaFcG33u/CvK8AbnJxfM84PZpV4xOBiYiikZcHtG4dn2O5KT4q\nVNU9nqckAZzhswsKEpsOIqJQ4plTcBMUVojIYABpItLaN7fCfI/TFRecU4GIkl1BgTWdT6agcDOA\ndgAKALwKYA9sFraUx6BARMnup5/sMZnqFH6tqn8F8FevExNvnJKTiJJdPPsoAO5yCo+KyLcicr+I\ntPM8RXHEnAIRJbukCwqq2gtATwDbAYwXkeUicrfXCYsHJyiwVzMRJaukCwoAoKo/qeo42DwKSwHc\n42mq4oQ5BSJKdnl5Nn3wMcfE53huOq+1EZF7RWQFbCjs+bAhKVIegwIRJbu8PAsIaWnxOZ6biuZJ\nAF4D0FdV4zo2kddY0UxEyS6efRQAdzOvdY9HQhKBdQpElOySJiiIyBRVvTLADGwCG6Ui3MxrSY/F\nR0SU7PLygC5d4ne8UDmFEb7HCs3AlswYFIgomR05AmzbFt+cQtCKZr9hrG9U1c3+f0jAiKZeYJ0C\nESWzbduAoqIkCQp+zguwrF+sE5IIaWk2cQWDAhElo3j3UQBC1yncAMsRtBKRZX5v1QbwhdcJixdO\nyUlEySqpggJs8LuPAPwLwCi/5XtVtdJMd9+gAbByZaJTQURUXiKCQqg6hT2quklVf+erRzgAa4WU\nJSLN4pZCj11/PfDpp8DXXyc6JUREpTlBoVGj+B3T1XScIrIOwEYAcwBsguUgKoUbbgCOPhp46KFE\np4SIqLS8PKBePaBGjfgd001F8wMAugNYq6otAZyLSlSnUKcOMHw4MGUK8N13iU4NEVGJeHdcA9wF\nhcOquhNANRGppqqzAHTyOF1xdeutQPXqwNixiU4JEVGJZA0Ku0UkC8BcAJNF5D8ACr1NVnwddxxw\n7bXApEklsxwRkRk6FLjjjkSnompK1qAwAFbJ/GcA/wPwHYCLvUxUIowcCRw6BIwbl+iUECWXmTOB\n2bMTnYqqR9VuUpMuKKjqL6p6RFULVfUlVR3nK06qVFq3BgYOBJ56CtizJ9Gp8caBA0DfvsCSJYlO\nCaWKAweArVuBH35IdEoqJicn9c7h55/tRjXpgoKI7BWR/DJ/P4jIOyLSKh6JjJc77wTy84Hnnkt0\nSryxcqXd9b35ZqJTQqli82Z73LoVKChIbFqipQr07w/85S+JTklkEtFHAXA5RzOAOwA0hk2uMxLA\n8wBeBzDRu6TFX9euQJ8+wGOPAQcPJjo1see0rsrJSWw6KHVs3Fjy/McfE5eOiti6Fdi+PfX6IiVz\nULhAVZ9T1b2qmq+q4wH0V9U3APzK4/TF3ahRVo73yiuJTknsbdhgjzk5dvdEFM6mTSXPU634xeGM\nWLBuXWqNc5bMQaFIRK4UkWq+vyv93qt0l5bevYHsbOvMduRIolMTW05Q2L275DmlpiefBGbN8v44\nlSkoFBWl1pA2yRwUrgIwFMA2AFt9z4eISCaA//MwbQkhYi2R1q8HZsxIdGpi67vvrPc2ACxenNi0\nUPS2bgVuvhk47zxg/Hhvj7VxI9C4sT1P5aDgzG+8bFnodZNJXh6QlWV/8eSm9dEGVb1YVRuoakPf\n8/WqekBVP49HIuPtssuAY48Fnnkm0SmJrQ0bgPPPt+HCWa8QW0eOACtW2N2o17780h7btAH+9Cer\nQPXquJs2Ae3a2VALqRoUVq0CunWzi+s33yQ6Ne4loo8C4K710Uki8qmIrPC97iAid3uftMTJyAD+\n8Afggw9KZ59T2aFD9qM++WSgY0cGhVgoKgLmz7e79saNgVNOAd5+2/vjzp9v39EFC2zsrocfBq68\n0pvy8k2bgBYtgKZNUzMoqFpOoX17+/8wKITnpvjoeQCjARwGAFVdBmCQl4lKBsOGWVGS19nzeNm8\n2S5iJ5xgdSaLF8fnrjZVzJgB9Ozprtnltm3WfLllS+DMM4EJE4CzzgJq1iy5i4/Gv//trrnwF19Y\nS7mjjrJ+NY8+CkydCvTqZUVLsbJvn7XaadnSgsL338du3/Hy00/Arl2W2+nQwYqPUqWRRTIHhVqq\nuqjMsko1zEUgzZoBF10EvPCC3WWnOqdiuVUrCwr5+VZvQua//wXmzAE++yz8ujffDDzyiN19vvKK\nXYjffNMuOtE2e9y2DfjrX4EHHgi9XkGB5fLOPNNeiwB//rMFheXLgXPPjd1Fz+mjkMo5BadiuV07\nyyHv3p0655HMQWGHiJwAX0sjERkIIC/0JkZELhCRNSKyXkRGBXj/WhHZLiJLfX9/iCj1HrvhBvux\nTp2a6JRUnNNHwckpACxC8jdvnj1OmxZ6vQMHrFjxj3+0xyFDSirvO3cGli6N7qI8ZYrVSyxbFnr8\nrSVL7CbljDNKL7/0UgsoK1fadzYWnKJTJyjs2gX88kts9h0vq1bZo5NTAFKjCGnvXvuskzUo3ATg\nOQAni8iPAG4FcEO4jUQkDcBTsPmc2wL4nYi0DbDqG6rayfc3wX3Svde3r91ZV4YK5w0brHijUSOg\nbVt7zhZI5ocf7K44I8OCQqhitZkz7cf6m9+Uf69zZ7sTde6wIzF5MvArX6+fTz4Jvt78+fZ4+unl\n33MuerFqdul0XHOKj4DUuct2rFxpleTHHlvy+aRCC6RENUcF3Lc+6gOgIYCTVbWHqm5yse/TAKz3\nbX8I1gN6QIVSG2fVqlnrjrlzU6t9cyDffWcBrlo1Gya8UyfmFBxOLuH//s/u0r/6Kvi6U6cCdeta\n/UNZnTvbY6RFSN99Z5XGf/kLUL8+8PHHwdedP9/+j4Fm4mrXzh5XrIjs+MFs2mQ3D8cck9pBoW1b\nK2arXds+u1TIKSR1UBCRGiIyGMAIAH8WkXtE5B4X+24MwP8rlOtbVtblIrJMRN4SkaauUh1Hv/+9\nzXqU6rmFDRvsB+HIzraiiMrWQS8a8+bZBWP0aGvP/u67gdc7fBiYPh245BIgPb38+6ecYttHGhRe\nfdUuWlddZcOszJwZuAhK1SqZnfqEsho1stxGrG5gnJZHIqkZFJyWR06wBEoqm5OdExSOPz7+x3ZT\nfDQNdodfCOAXv79wJMCysl/19wC0UNUOAD4B8FLAHYkME5EcEcnZvn27i0PHToMGwBVXAC+/bK0x\nEumLLywtkQ5Mpho4KOzbB6xdG9s0pqJ58+xC26CB5QCC1SvMmWPl6oGKjgAgM9Oa/EYSFFSt6Ojs\ns+3C27ev5VYC3e1v3GiV2mXrExwiVvkdy+Kjli3teePGtv9UCgp5eVac5x8UOnZMjeEukjqnAKCJ\nqv5WVR9S1UecPxfb5QLwv/NvAmCL/wqqulNVnUvc8wC6BtqRqo5X1WxVzW7YsKGLQ8fWDTdYxc+r\nr8b90MUOHACuvhp46y0rzorE9u0WAE44oWQZK5vNzp12ET3rLHs9YACwenXgYDl1KlCrll24g+nc\nObKgsGQJsGaN5RIA66UMBC5CcuoTggUFwC6AK1fGpgWSk1MALLd87LGpFRT8K5kdHTpYnVGsiti8\nkpdnn3nduvE/tpugMF9EToli318BaC0iLUUkA9a3Ybr/CiLiHwcvAbA6iuN47vTT7cv0zDOJa+P8\nwAN2t1+9OvDhh5Ft698c1XHyyXaBq+qVzV/4Zhv3DwpA+dxCURHwzjs2BHNmZvD9depko4m6zdC+\n+qoVRQ0caK+bNrX/zcyZ5dedP99aOvlf5Mpq187ujvNctQ8MLj/fxvN3goKTtlQKCk6Oqa1f85aO\nHe0x2YuQnOaoEqi8xWNugkIPAIt9TUuXichyEQn7kapqIWxspBmwi/0UVV0pImNE5BLfareIyEoR\n+QbALQCuje40vCUC3HijNTdcsCD+x1+50gbou+Yaa4f+0UeRbe8EBf+cQlqa3dVW9ZzCvHnW6ujU\nU+11s2b2uZStV1iwwIp1ghUdOSKpbD5yBHjtNQs0v/Ibb7hvXyuqKjt8+/z5QPfuJeP4BOIEjIoW\nITnNUZ3iIyA1g4LT8sjRsmVqDHeRqD4KgLug0A9AawB9YdNwXgSX03Gq6oeqepKqnqCq//Atu0dV\np/uej1bVdqraUVV7qeq30Z2G9666yn64o0dHn1vYvdvuviJRVGQtoOrUAcaOtQvImjWRjXLq9FHw\nv+sDrAjp66+Bwgi6In78cWICo1fmzQNOO81a2TguvdR6Jvv3Dp461YLHhReG3l+nTvboJijMnm0/\nfqfoyNG3rwUEJxcD2J378uWhi46A2AeFQDmFVOkR7FQy+99tV6tmDQKSPaewZUsSBwVV3RzoLx6J\nSyZZWTYMwZw5wEsBq8PDGzzYeklHYuJEuziMHWsVof362fJIcgsbNlhFYdlij+xsq3D71mUoLigA\nfvtba7pZGfzyixWfOUVHjgED7ML33nv2WtWCQp8+JR3VgqlXD2je3HKV4UyebK2eyn4nzjnHipT8\n6xUWLrQbhHBB4Zhj7HviVVDYt89uboIZOtR6hyeaqtUpBCpq69jRcgrJHNySPadAPtdfb61URo4E\nduyIbNuDB238+wUL3G+7dStwxx12kbjmGlvWujVw4omR1Ss4fRTKirSy+aOP7ILw9dfWCifVLVhg\nuaSyQaFDB7uwO/UK33xjLXHCFR053FQ2Hzxog+ddfnn5YJ2VZRd//6Awf77d5XbrFv74TmVzRWzc\naGMrNWhQsixcs9QdOywgPPxwxY4dC4FaHjk6drR52JN1LKcDByztDAopoFo14Nln7QsV6XyvixbZ\nhUAV+PRTd9vcfrvdzT77bOkscL9+FmAOHHC3n7LNUR0nnWQXILeVza++ap9BUVHkLaCS0bx59rmW\nvfsWsSKkmTPtznjqVDvvSy4JvJ+yOne21kuhmjC//74VCZUtOnL07Wu5DWfIivnzrdgjXE4FiE0L\nJP8+Co5wQcH5Hi1blvimzoEqmR3J3rPZGeaEQSFFtG9vF+tJk6woya05c+wHlpUVuGVJWTNnWvHC\n6NHWGsVf//4WENwc/+BBaw3jX8nsqFYN6NLFXU4hP9+KU/7wByt/dzNwXLKbN8/uGuvUKf/egAFW\nXDZjhgWFs88G3LaG7tTJLsjDdp0PAAAVnUlEQVShLjqTJ1tns169Ar/vNE395BOrkP7yy/BFR452\n7ez/VZE5lf2bozrCBQX/79Fbb0V/7FjwHwivrFN8bSnjWdl85Ig1XsjPD79uIvsoAAwKUbnnHvvB\nDB/uviPZ7Nl2sTjvvOA9Vh2qNvJl69YWFMo65xy7MLupV3DGrwmUUwCsCGnpUuutG8rUqRZgrrsO\n6NEjPlNBRmLt2pIKdTcOH7bio7JFR46zzrKGBWPH2gXGbdEREL4F0q5dVvw3aFDwlkRdulj9xMyZ\ndvy9eyMLCkDF2uJv2lS65RFgF6m0tNBB4aSTrIWUmyHAvbRypQ0Zcswx5d9LxHAX775rk3e1bAn8\n61+hc5HO95hBIYXUqmXj2H/7rbvy04ICy/737GlB4fvvrVdlMMuW2Zf69ttLt4pxZGbaXNJu6hUC\nNUf1l51tF3uno08wkyfbPrp1s2MvXx670Tgrav9+u+Pu39/9HBFLlth2wYJC9epWAey0tLr0Uvfp\nadLELkjBgsKLL9pIp0OHBt9HWpo1P/7445JWSMGGtyiroi2Qdu+2v7I5hbQ0G3YhVFA49VTrdb90\naWKHZncqmYO18+/YMb7FR/Pm2e/29NOBu+6y4DB2rH0HVe33NGaM3QxcfbU1NGjePH7p88egEKX+\n/e3L/8AD4b/8Tn3COedYCxYgdBHSlCn2Awx1d9qvnx03VHABSu46QuUUgNBFSHl5Vlw0eLD9yHr3\ntuWzZ4c+dryMG2dN+NaudVc0B5QMghcsKAAlgeC000qKTtwQCV7ZfOiQzcVwzjl2AQilb187rwkT\nrKip7EU6mAYNrG1+tEEhUMsjR7C+Cj/9BOTm2vfJ6YiXqCIkZ8yjQPUJDme4i3gNBf755/Y9ev99\nKwrs0sUakbRqZQ1HOnQA7r3Xbjgfftianfv3XYknBoUKePxx64p+882h15s92y4UZ51lX4DmzYMP\nj6wKvPGGXXhDlWH372+P4YqQNmyweoxg+zrhBCtTD1XZ/MYbdgc+eLC97trVsuDJUK+wcyfw4IM2\n9/SxxwJPPOFuu3nz7H8RaLRRx/nn2+fmtPyKROfOVnxTtlju1VetrH9UudlFynPqFZYssaKjSHq3\nVqQFUqCOa45mzQIHBef7k51t63TrlrgipC1brDFIqJ7fHTqUBA+v7dtnOacePex19+5WVzVvnn1e\nJ59sMzxu2WLBY+TIwJ99vDAoVMDxx9uP+3//s8gezJw5dmdSr579sM87zy6ogTqNLVlid/dXXhn6\n2K1aWfmtm6DQqlXwC0q1anaR//jj4IOETZ5sdzZOhXf16lbxmgxB4V//ssq7sWOtk9+HH4avWygq\nsh9fqFwCYE0yt2yxsa8i1bmz5QpW+w3cUlRkfV06drSAE07z5vY/BtzXJzjatbMilGhaIDn1UMFy\nCrm55febk2PfJafz3sCB9l0O1cnSq34CoSqZHc5wF/GoV1i0yCqayxb/9ehhOQdn0qZQNyjxxKBQ\nQddea0U9L74Y+H3/+gTHeefZhSzQuP1TpthF103FZv/+VuEbasTHYH0U/N1xh/14//jH8j/UtWvt\nB1+26WTv3pb9TuSwB5s3W87gmmusVdif/mT/i6efDr3d6tXWszxcUADsfxHN+DOBKpvfe8/qoe68\n0/0+ncH33NYnONq1szvUaNrib9pkOcFAxRdNm9p3uuzYTjk5QJs2lisFwhchbdxoObtWrWz2uqee\nirx3fTCBBsIrq0WL+A138cUX9v8ONDFSMmJQqKDjjrPy/ZdfDjw3wVdfWfNR/6Bw7rn2JSlb/q1q\nQeG88yxXEU6/fvYDDVa27wyZHayS2XHBBcD991vRxuOPl37PGet/0KDSy516hUS2Qvr73y1tY8bY\n6+OPt85gEyeGLit26hPOPtu7tLVubeXDTlBQtVxNy5ZWF+XW8OHWDLhrwPGDg6tIZbPT8ihQ4ArU\nLFXVvudO/RRgF91TTw1chFRYaEWRhw5Z8PzsM+sl36WLjQpa0R7zTsujUMWv1arFb26FL76w/0ci\nRjyNBoNCDFx3nRUzBBru2L8+wVG/vv0AygaFr76yH2S4oiPH2WfbhSdYK6S8PKvgDpdTAKzp62WX\nWa7BKRZyxvrv3bv8ZB8dOljgSlRQWL7cAvEtt5SuBL75Zms5M3ly8G3nzbNg7uZziVZamn1GTlCY\nO9eGqhg50nIfbrVrBzz/fOBJfcJtB0QXFDZuDF6p7XzW/jmQH3+03vf+QQGw3EJOTklxlGPMGGvV\n9eyz1qv7xx/te//aa3aj89RTkfUBKivQmEeBOMNduG2xFsiaNaG3d/qYRJrTSyhVTam/rl27arIp\nKFBt0ED1iivKv3fuuaodO5ZfPmqUavXqqvn5Jctuu001PV111y73x77oItVWrVSLisq/N2+eKqD6\nv/+521d+vmqbNqr166tu2qS6cKFt/8ILgde//HLVZs0CH9trF16oWreu6s6dpZcXFal26qTavn3g\ndK1aZdv99rfep/GGG1SPPlr1yBHVfv1UGzZU3b/f++M6jjtO9ZprItumqEi1dm3VW24J/P7Wrfad\n+M9/Spa9844t+/LL0utu2GDLH364ZNncuarVqgVP1/79qo0aqfbqFVm6/dN/9NH22Yfz+uuWvpdf\nju5Yy5apiqhOmBB8nW++qdgxYglAjrq4xjKnEAMZGVbmPm2atYZxHDpUvj7B0aePZaOdop+iIstq\nn39+ZNnM/v2tiCjQsAKB5lEIpXZt62RTWGi5hgkTrHXV5ZcHXr93b7tjjGTE1liYM8cq50aPLl/M\nJmK5hRUryg/FsXatpblmTSsu81rnzlZ3NG2aNQgYMSL0XAyxFk0LpF27rKNcsNYvDRvad8K/+Cgn\nx3JGTuWto2VLK/ZyipB27bLfScuWwVuJZWZancusWeFzC//8J3DxxTbkzKRJlhP79lv7zEPVJziu\nuMKKuO680845UlOnWm461ACZkfYxSQpuIkcy/SVjTkFVdelSuyMYN65k2eef27J33im//oEDqjVr\nltyRffGFrfvKK5Edd+NG2+7++8u/d889dldWUBDZPt9/3+6AAMsNBLN6ta0zfnxk+6+IoiLVbt1U\nmzQJfte9f79qvXql075+vWrjxna3vnJlfNL61Vf2+TRqpJqVpfrzz/E5rmPECNVatSyn4lZOTvDv\nrOOEE0rntM4/P3BuWFX1wQdtf5s2qV55peWOFy4MnQY3uYUPP7T9NmmiWqOGPff/++yz0MdwLFhg\n6995p7v1/XXqVHK8zZsDrzN4sJ1LInLTZcFlTiHhF/lI/5I1KKiqdu5sf44HHrBPeMeOwOv37WvF\nNaoWHGrUUN2zJ/LjXnyxbbt0aenlQ4aoNm8e+f5UVceMsbRPnx58naIiK6IYNCi6Y0Tj7bc1ZJGW\n4y9/UU1LU/3+e7sgNWtmxWLLlsUnnaoW+NPSLL233x6/4zqef96O/d135d+bPl312WfLL3/rLdvm\n66+D77dnT9UzzrDnRUX2uV5/feB116+3/Z1xhj3+85/u0v7YY7b+7Nnl39u+3S607dvbZ1xYqLpu\nneq0aRaERo9WPXTI3XFUrSgrPV117Vr32zg3YzfeaI8PPhh4vebNVQcOdL9fLzEoJMC4cfaJOhfn\nPn1UO3QIvv7DD9v6339vF9dLL43uuNu22fYnn6y6b1/J8jPOqFjZ7Jo14dcbPFj1mGPicyd0+LCd\nY5s2diEIZeNGyyVdd51qy5ZWj7BkifdpLOuUU+yCk5sb/2PPnx84sC9YYGkCVF98sfR7Y8fa8lD1\nWkOHqjZtas+di+MzzwRfv3NnW6dnz/D/N0ew3EJRkeUA09PL3wRFa8sWq0e58EL32zz+uJ3T2rWq\np58e+Heem2vrPPZYbNJZUQwKCbBjh2pGhmXbCwpUMzODV9iplhQ5/f739vjaa9Ef+5NPrMjH/47t\n2GNV//CH6PfpxoQJlvYVK9ytv2eP6qJF0QWRF16wY02d6m79AQNs/aOPtmMmwuTJqk89lZhj795t\n5/+vf5Us27bNilxatLCLdEZG6Qrim26yABrKXXdZDqiwUPXNN+0YX30VfP0nn7Sblh9+iCz9gXIL\nL78c+s48Ws4N2gcfuFu/Vy/Vdu3s+RNP2LbLl5de5403bHmivntlMSgkyBVXWHZ61qzwF7AjR+wu\nG7AAsndvxY49erTt6/XXLccQSXY9Wk4LE/+6lGC2brU7KsByUaGKKMo6cMAuZqed5j6gLFpkx/vi\nC/fHqWyaNLFiRFXLafXubUWNixfbTUyrVnZH7uRkLrzQyspDeeYZ+x/+8IOVxWdkqB48GHqbSOo1\nHGVzC5s3W4Dv0cN9jsOtggLVk05Sbd06fB3cjh0WFP/6V3u9dau9Hj269Hq33GJ1OpEUZXmJQSFB\nnAqwU0/VkPUJjsGDbb1YlDseOqTavbv9cKZNs/2+8UbF9xtOixaql10Wep0ff7Sin8xM1TvusIpg\nEdWrrw5eSefvkUc0ogpEMuefX1LPNWqUfYYTJ5a8v2KFVYJnZ9tFuF278P/L99+3/cyfb02us7O9\nS7+TW5g1y4JDVpbdiHjho4/sWA89FHq9l14qnwO44AKrP/APfl27Wm4sWTAoJEhhoerxx9sne8op\n4defNMnWnTIlNsffuFG1Th0LDOGy9bHy+99bkUOwSvJNm6zFSlaW6pw5tmzXLqsMrlHD/u68s3Sf\nDX+7d1vuq29fb9Jfmd12m7Vycyrohw0rv86779p7V12letRRqn/+c+h9LltWkiOtU0d1+HBv0q5a\nklv41a/UVQODirr4YvuebtkSfJ3LLrOWbP4B4JVXLH2ff26v9+4tnZtIBgwKCeTckd18c/h1Cwrs\nCxXL7PCUKVrcVK5s5y4vfPCBHSsry87Zv4J63Tpr+VO3rlVwlrVpkxVvAJaTWL26/Dp/+5u9n5Pj\n3TlUVk49TI0adkd/4EDg9e6/v+Q7498xLZBdu2y94cPtMVTnrVhwcguXXOJ9g4Z166w4LFjnxl9+\nsdzujTeWXp6fX3r5p59amj/80Nv0RoJBIYHWrbM7rk8+SVwabrpJ9cQT43e8RYusVYrTqqVfP8tm\nH3ec3eWHa/nz6afWhyAry5pFOn76yT7LK6/0Nv2VldMO3+mlHkxRkdWHhWuG7KyblWV38P6t7bxy\n8KBV5sbjBke1pDn2e++Vf88plv344/LvDRpkIxscOqR6331WPBrJ6AReY1BIsGgq1mKpqMgqFuMt\nL89+EM4F49hj3bdM+v57q0gGrN7h8GHLeaSluWseS+UdOGBFIm7qYvbtU3366eC5CX9t2tj/qWbN\n5KlIjZWCAusD0aRJ+SLR666zIrNAldHTp2txC6a+fd0VH8cTgwIlVEGBtbzauDGy7Q4eLCmW6NHD\nch6BysEpsfr2tf9R9+6JTok3FiywO33/YqLCQssJDB4ceJuCAmtAMWiQ9Xvwsq4lGm6DAsc+Ik9k\nZNj4SW6nkHTUqAE884yNZeOMqXPPPZ4kkSrAGS217MiolUW3bjYC79NPl4xfNH8+sGNH8Pm6MzJs\nPKU33rCxlFJqvCM/DAqUlK691mbu+uwzoHHjRKeGyqrsQQGw+debN7f5LA4etMEiMzJs/pFgrrqq\nZKKqVA0KEYzsThRfbdokOgUUTOvW9titW2LT4aWsLOC55ywI/OMfFhT69LHRhIM580wLmIWFkeeS\nkwWDAhFF7Morbf5oZ97uyur884GhQ22Y7qIiG2Y7lGrVLJDs3x/dNK7JQNTJ66SI7OxszcnJSXQy\niKiK2LHDcq07d9oMi40aJTpF0RGRxaoatsCPOQUiohAaNLDK4yVLUjcgRIJBgYgojN697a8qYOsj\nIiIq5mlQEJELRGSNiKwXkVEB3q8hIm/43l8oIi28TA8REYXmWVAQkTQATwHoB6AtgN+JSNsyq10P\nYJeqngjgMQD/9io9REQUnpc5hdMArFfVDap6CMDrAAaUWWcAgJd8z98CcK5IqjbkIiJKfV4GhcYA\nfvB7netbFnAdVS0EsAdAfQ/TREREIXgZFALd8ZftFOFmHYjIMBHJEZGc7du3xyRxRERUnpdBIRdA\nU7/XTQBsCbaOiFQHUAfAz2V3pKrjVTVbVbMbNmzoUXKJiMjLoPAVgNYi0lJEMgAMAjC9zDrTAVzj\nez4QwGeaal2siYgqEU+HuRCR/gAeB5AGYKKq/kNExsDG9Z4uIjUBvAKgMyyHMEhVN4TZ53YAm6NM\nUgMAO6LcNtlV1nPjeaWeynpuqX5ezVU1bFFLyo19VBEikuNm7I9UVFnPjeeVeirruVXW8yqLPZqJ\niKgYgwIRERWrakFhfKIT4KHKem48r9RTWc+tsp5XKVWqToGIiEKrajkFIiIKocoEhXAjtqYSEZko\nIttEZIXfsnoiMlNE1vkef5XINEZDRJqKyCwRWS0iK0VkhG95Sp+biNQUkUUi8o3vvO7zLW/pGx14\nnW+04IxEpzUaIpImIl+LyPu+15XlvDaJyHIRWSoiOb5lKf1ddKNKBAWXI7amkhcBXFBm2SgAn6pq\nawCf+l6nmkIAt6tqGwDdAdzk+z+l+rkVAOitqh0BdAJwgYh0h40K/JjvvHbBRg1ORSMArPZ7XVnO\nCwB6qWonv6aoqf5dDKtKBAW4G7E1ZajqXJQfDsR/xNmXAFwa10TFgKrmqeoS3/O9sAtNY6T4uanZ\n53uZ7vtTAL1howMDKXheACAiTQBcCGCC77WgEpxXCCn9XXSjqgQFNyO2prpjVTUPsIsrgGMSnJ4K\n8U241BnAQlSCc/MVsSwFsA3ATADfAdjtGx0YSN3v5OMA/gKgyPe6PirHeQEWuD8WkcUiMsy3LOW/\ni+FUlTmaXY3GSslBRLIAvA3gVlXNrwxTbKjqEQCdRKQugHcAtAm0WnxTVTEichGAbaq6WER6OosD\nrJpS5+XnTFXdIiLHAJgpIt8mOkHxUFVyCm5GbE11W0XkOADwPW5LcHqiIiLpsIAwWVWn+hZXinMD\nAFXdDWA2rM6krm90YCA1v5NnArhERDbBimR7w3IOqX5eAABV3eJ73AYL5KehEn0Xg6kqQcHNiK2p\nzn/E2WsATEtgWqLiK49+AcBqVX3U762UPjcRaejLIUBEMgH0gdWXzIKNDgyk4Hmp6mhVbaKqLWC/\nqc9U9Sqk+HkBgIgcJSK1necA+gJYgRT/LrpRZTqvBRqxNcFJipqIvAagJ2zUxq0A/g7gXQBTADQD\n8D2AK1S13NwUyUxEegCYB2A5Ssqo74LVK6TsuYlIB1ilZBrsRmyKqo4RkVawO+x6AL4GMERVCxKX\n0uj5io9GqupFleG8fOfwju9ldQCv+kZ5ro8U/i66UWWCAhERhVdVio+IiMgFBgUiIirGoEBERMUY\nFIiIqBiDAhERFWNQICKiYgwKRERUjEGBiIiK/T9/juTov5JjbQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f155f1a1080>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "plt.plot(range(len(train_losses)), train_losses, color='blue')\n",
    "plt.legend(['Train Loss'], loc='upper right')\n",
    "plt.ylabel('negative log likelihood loss')\n",
    "fig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have 96% acccuracy!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}