MLAS_04_NeuralNets_Detector_MNIST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center>\n",
    "    <img src=\"./images/mlfasp.png\">\n",
    "</center>\n",
    "\n",
    "#### Prof. Dr. -Ing. Gerald Schuller <br> Jupyter Notebook: Renato Profeta\n",
    "\n",
    "[Applied Media Systems Group](https://www.tu-ilmenau.de/en/applied-media-systems-group/) <br>\n",
    "[Technische Universität Ilmenau](https://www.tu-ilmenau.de/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "hide_input": true
   },
   "outputs": [],
   "source": [
    "# Settings for Google Colab\n",
    "RunningInCOLAB = 'google.colab' in str(get_ipython())\n",
    "if RunningInCOLAB:\n",
    "    !git clone https://github.com/GuitarsAI/MLfAS.git\n",
    "    !mv MLfAS/images/ ./"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hide_input": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Neural Network Detector for MNIST Digit Recognition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/9jNt0C5quL4?rel=0\" frameborder=\"0\" allow=\"accelerometer; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/9jNt0C5quL4?rel=0\" frameborder=\"0\" allow=\"accelerometer; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hide_input": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "In part 3 we saw how to use a classifier network for MNIST digit recognition. There usually we assume that the input is a digit, hence we can normalize the network classes outputs such that they represent a probability distribution which adds to one. But often in applications we would like to recognize patterns or objects, but don’t know if they are present in the input. For instance we could have a life digit recognizer, which takes a camera as input. In this case, sometime we place a paper, for instance a letter with a zip code, in front of a camera, and at other times not, which should not confuse the recognizer.\n",
    "\n",
    "In such cases it makes sense to use a detector, as in part 1, for the recognition. This has the advantage that we could train the recognizer of each digit class to output a \"1\" (or a value close to it) when it recognizes its digit, or a \"0\" (or a\n",
    "value close to it) when it is not recognizing its digit. If no digit is present, all detectors will output values close to 0,\n",
    "which is something that the classifier cannot do.\n",
    "\n",
    "For that we use the same neural network structure as in part 3 for the MNIST classifier, but with a different Loss function. Instead of the CrossEntropyLoss(), we take the MSELoss(). But the MSELoss expects a different target representation. \n",
    "\n",
    "For the CrossEntropyLoss, we need the class label (number) as the target, which means we have one integer (or float) input digit. The MSELoss expects a float number for each class, which means 10 float numbers for each input digit, where the correct class should have a \"1\", and all other classes should have a \"0\".\n",
    "\n",
    "Since the MNSIT data set is made for classification, we need to convert its target to classes. We can do that by initializing a torch array with zeros, and then set it to \"1\" at the corresponding \"correct\" classes. This is also called a \"**one-hot**\" representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/3W4c5IEKilc?rel=0\" frameborder=\"0\" allow=\"accelerometer; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/3W4c5IEKilc?rel=0\" frameborder=\"0\" allow=\"accelerometer; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "hide_input": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe src=\"https://pytorch.org/docs/stable/nn.functional.html#one-hot\" width=\"900\" height=\"600\"></iframe>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<iframe src=\"https://pytorch.org/docs/stable/nn.functional.html#one-hot\" width=\"900\" height=\"600\"></iframe>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/7wKKVnxR2Vs?rel=0\" frameborder=\"0\" allow=\"accelerometer; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/7wKKVnxR2Vs?rel=0\" frameborder=\"0\" allow=\"accelerometer; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "#from: https://machinelearningmastery.com/handwritten-digit-recognition-using-convolutional-neural-networks-python-keras/\n",
    "#https://pytorch.org/docs/stable/torchvision/datasets.html#mnist\n",
    "#Translated to PyTorch and modified to a simple fully connected network,\n",
    "#by Gerald Schuller, October 2019\n",
    "\n",
    "# Ported and modified to Jupyter Notebook by Renato Profeta, August 2020\n",
    "\n",
    "import numpy\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "import torchvision\n",
    "import torch.nn.functional as F\n",
    "import torch.nn as nn\n",
    "import time\n",
    "\n",
    "device='cpu'\n",
    "#device='cuda'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# define the neural network model\n",
    "class DenseNet(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(DenseNet, self).__init__()\n",
    "        self.layer1=nn.Sequential(nn.Linear(in_features=num_pixels, out_features=num_pixels))\n",
    "        self.layer2=nn.Sequential(nn.Linear(in_features=num_pixels, out_features=num_classes))\n",
    "      \n",
    "   \n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.layer1(x))  #First layer with activation function\n",
    "        x = self.layer2(x)   #Second layer\n",
    "        #x = torch.nn.Softmax(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "batch_size_train = 100\n",
    "batch_size_test = 10\n",
    "num_classes = 10  #10 classes according to the 10 possible digits 0,...,9\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(\n",
    "    torchvision.datasets.MNIST('~/Downloads/MNISTfiles/', \n",
    "    train=True, download=True,\n",
    "    transform=torchvision.transforms.Compose([\n",
    "    torchvision.transforms.ToTensor(),\n",
    "    torchvision.transforms.Normalize((0.1307,), (0.3081,))])),\n",
    "    batch_size=batch_size_train, shuffle=True)\n",
    "#the Normalize() arguments are mean and standard deviation.\n",
    "\n",
    "test_loader = torch.utils.data.DataLoader(\n",
    "    torchvision.datasets.MNIST('~/Downloads/MNISTfiles/', \n",
    "    train=False, download=True,\n",
    "    transform=torchvision.transforms.Compose([\n",
    "    torchvision.transforms.ToTensor(),\n",
    "    torchvision.transforms.Normalize((0.1307,), (0.3081,))])),\n",
    "    batch_size=batch_size_test, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "In the beginning it prints the one-hot vector y_train_onehot, where we can see that each row contains the values for the 10 classes, and indeed only one entry is \"1\", the others are \"0\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train.type()= torch.FloatTensor y_train.type()= torch.LongTensor\n",
      "batch_idx= 0 X_train.shape= torch.Size([100, 1, 28, 28]) y_train.shape= torch.Size([100]) y_train= tensor([9, 1, 0, 1, 1, 3, 2, 2, 1, 6, 6, 0, 1, 6, 7, 7, 8, 5, 3, 2, 1, 3, 9, 2,\n",
      "        8, 1, 3, 1, 8, 2, 4, 2, 1, 1, 3, 3, 6, 4, 6, 9, 8, 2, 4, 1, 9, 6, 8, 8,\n",
      "        9, 7, 4, 8, 4, 3, 6, 9, 4, 8, 9, 8, 6, 8, 1, 5, 6, 9, 9, 9, 3, 6, 0, 7,\n",
      "        2, 9, 6, 7, 8, 5, 0, 6, 4, 5, 6, 0, 0, 7, 9, 5, 7, 1, 0, 0, 9, 8, 6, 3,\n",
      "        1, 1, 2, 4])\n",
      "y_train_onehot= tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
      "        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
      "        [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],\n",
      "        [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
      "        [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],\n",
      "        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],\n",
      "        [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],\n",
      "        [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],\n",
      "        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
      "        [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],\n",
      "        [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],\n",
      "        [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
      "        [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],\n",
      "        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
      "        [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
      "        [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],\n",
      "        [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
      "        [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],\n",
      "        [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],\n",
      "        [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
      "        [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
      "        [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],\n",
      "        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
      "        [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],\n",
      "        [0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
      "        [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],\n",
      "        [0., 0., 0., 0., 1., 0., 0., 0., 0., 0.]])\n"
     ]
    }
   ],
   "source": [
    "examples = enumerate(train_loader)\n",
    "# Plot ad hoc mnist instances\n",
    "\n",
    "# load (downloaded if needed) the MNIST dataset\n",
    "#(X_train, y_train), (X_test, y_test) = mnist.load_data()\n",
    "#one trainig batch of 100 images:\n",
    "batch_idx, (X_train, y_train) = next(examples)\n",
    "print( \"X_train.type()=\",X_train.type(), \"y_train.type()=\",y_train.type())\n",
    "print(\"batch_idx=\", batch_idx, \"X_train.shape=\", X_train.shape, \"y_train.shape=\", y_train.shape, \"y_train=\", y_train)\n",
    "\n",
    "#Make classes with one-hot encoding from the target classes:\n",
    "#y_train_onehot=torch.zeros(batch_size_train,num_classes)\n",
    "#for spl in range(batch_size_train):\n",
    "    #y_train_onehot[spl, y_train[spl]]=1.0\n",
    "y_train_onehot=torch.nn.functional.one_hot(y_train).type(torch.FloatTensor)\n",
    "print(\"y_train_onehot=\", y_train_onehot)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhgAAAHiCAYAAACqZbXyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAAAs6UlEQVR4nO3de5wcdZnv8e+XQBRJECQQYwgEuQooQWIWN4DBC4eLCuyLgByF4C2sggLLRUSELN5YXoDiQQNBsiEcBaLcIi6IG0EElSVhI0SCGkOQkCEXghJAzEKe80fXHJowv55Oza+7OpnP+/Wa13TX0131TM3MM8/8qn5VjggBAADktFHVCQAAgA0PDQYAAMiOBgMAAGRHgwEAALKjwQAAANnRYAAAgOxoMPoJ27va/m/bq2x/vsn3hO2dWp3burD9nO235n4tACCvShoM2/vZ/pXtv9peafs+2++qIpe6nEYWf1A3rjKPFjpL0t0RMTgivr120Pbdtj+Ve6O29y/+0D9n+/liHz9X97HduqwvIgZFxMLcr22W7T1s32n7Gdt/sT3H9qFNvneR7ffnzAcAOlXb/5ja3lzSbZI+I2mGpIGS9pf093bn0s9sL+n6dm80In4paZBUa+IkPSZpi4h4ae3X2t64p+Ud5seSJkv6YPH8XZJcXToA0JmqGMHYRZIi4rqIeDki/hYRd0bEQ5JkeyPb59p+3PYy29Ntv7GIdY8yfNz2E8V/kf9s+122Hyr+o7y8fmO2P2F7fvHan9revpkkbU+z/V3btxf/ad9n+822v1Ws61Hbe9e9/mzbfyoOQTxi+8i62ADbl9heYfsx2yfXj5bYfqPtq2132X7S9ldtDyhiO9n+RTHas8L2DQ1y/rDt3xX74W7bbyuW/1zSgZIuL76WXdZ639dUa/K64/X78P22/1h8zd+x7br3ldq3de+fZPtHtv+v7WclnWB7jO1fF19Dl+3LbQ+se8//P2xTfI++Y/snxX6/3/aOidduZfvHtp+1/UCxj+8tYrb9zeLn7a/Fz9KePeQ7RNIOkq6KiNXFx30RcW/daz5oe26R/69sv6NYfq2k7ST9uNjHZ63LvgKA9U5EtPVD0uaSnpZ0jaRDJG25VvwTkhZIeqtq//neJOnaIjZSUki6QtLrJR0k6UVJt0jaRtJwScskvad4/RHFut6m2mjNuZJ+lcire90bF8+nSVohaZ9iWz9X7b/v4yUNkPRVSXfVvX+8pLeo1rQdI+l5ScOK2D9LekTStpK2lPSfa23rFklXStqs+Dr+S9KJRew6SV8q1vt6Sfsl8t+l2OYHJG2i2iGRBZIGFvG7JX2qwfflNfEix9skbaHaH8flkg5e133bYB9PkvQ/xbo2krRpsb/3LdY5UtJ8SaeuldNOdd+jlZLGFK//vqTrE6+9vvh4g6TdJT0h6d4i9r8kzSm+Thdf07Ae8rekPxb75AhJQ9eKv1O1n79/KH5GJkhaJOl1RXyRpPe3+3eODz744KOKj2o2Wivg0yQtlvSSpJndxVrSLEmfrXvtrsUfoe4/OCFpeF38aUnH1D2/sfsPkqTbJX2yLraRpBckbd9DTmv/8Zum2n+q3fHPSZpf9/ztkv7S4GucK+nw4vHPVTQMxfP3d29L0lDVDg9tWhc/VkXzImm6pCmStu1ln35Z0oy1vtYnJY0rnt+tcg3GfnXPZ0g6e133bYN9PEnSPb18XadKunmtnOobjO/VxQ6V9Ojar1Xtj/3/SNq1LvZVvdJgvFfSH1RrbDbqJZ9tJV0u6U+S1ki6R9LORWyypK+s9frf65WGd5FoMPjgg49+8lHJSZ4RMT8iToiIbSXtqdp//t8qwm+R9Hjdyx/XK3+Iuy2te/y3Hp4PKh5vL+myYrj6L6r9t2vVRjqa0ex2ZPv4uqHxvxRf15C6r+mJuvfWP95etRGHrrr3XqnaSIZUG4mwpP8qDn98IpHrq/ZbRKwpttPs15ryVN3jF5Rv33ar3xeyvYvt22w/VRw2+bpe2Y/rkl+9rVX7GerxexARP1etafiOpKW2p7h2rtBrRMTiiDg5InZUbR88r1oTqOL56d37pNgvI1T73gBAv1L5NNWIeFS1/0S7j3kvUa1Qd9tOtVGOpVp3T6g2crBF3cemEfGrvuS8tuLcg6sknSxpq4jYQtI8vXLyX5dq//l2G7FWjn+XNKQux80jYg9JioinIuLTEfEWSSdK+q57njr6qv1WnCsxQrVRjGas6211c+3btbc7WdKjqo0KbC7pHPX9JMrlqv0Mpb4HiohvR8Q+kvZQ7XDTmb2tNCKeUK0p6f7ZfULS19baJ2+IiOu639LHrwMA1httbzBs72b7dNvbFs9HqHZI4DfFS66TdJrtHWwPUu0/2Bui3OyCKyR90fYexbbeaHt837+K19hMtT8ey4vtfFyv/NGRaocWTrE93PYWkr7QHYiILkl3SrrE9uauneS6o+33FOsa372vJD1TbOflHnKYIekw2++zvYmk01VrXJr9g79UtfNemtWqfTtY0rOSnrO9m2qzjfokIl5W7VyeSbbfUKz3+O64aycJ/0Ox355X7bye1+xj21va/lfXTrzdqDjp8xN65Wf3Kkn/XKzLtjezfZjtwUV8XfcxAKy3qhjBWKXaSXD3235eteI8T7U/iJI0VdK1qh3bfky1Yv+5MhuKiJsl/Zuk64vh9nmqnViaVUQ8IukSSb9W7Y/I2yXdV/eSq1RrIh6S9N+S/kO1/6i7/4gdr9p03UdUayJ+JGlYEXuXavvqOdXOVTklIh7rIYffS/qYpP+j2smpH5L0oYhY3eSXcZmko4oZIa+5TkYP22vVvj1D0v9W7efkKknJWTPr6GRJb1TtkMq1qjWy3VOjNy+29Yxqh5melnRxD+tYrdp5JP+pWhM0r1jHCZIUEbMlfVq1wy3PqHYS7Al17/+GpHOLwydnZPq6AKAjOYJR23azfYikKyJinaZ1Ih/b/ybpzRExoepcAGBDVPk5GP2B7U1tH2p7Y9vDJZ0v6eaq8+pPikNz7ygOXYyR9EnxPQCAlqHBaA9L+lfVhs3/W7VrO5xXaUb9z2DVzsN4XrXzVS6RdGulGQHABoxDJAAAIDtGMAAAQHY0GAAAILs+3U3V9sGqTW8coNolmy/s5fUcjwFea0VEbF11Ev3RutQw6hfQo2T9Kj2C4drdPr+j2rUPdpd0rO3dy64P6Mce7/0lyI0aBmSRrF99OUQyRtKCiFhYXMzpekmH92F9ANBO1DCghfrSYAzXq28etVg93OjK9kTbs23P7sO2ACC3XmsY9Qsory/nYPR0A6rXHKOMiCmq3W6cY5gAOkmvNYz6BZTXlxGMxXr1HSm3Ve2OngCwPqCGAS3UlwbjAUk7F3c9HSjpI6rdjAsA1gfUMKCFSh8iiYiXbJ8s6aeqTfGaGhG/y5YZALQQNQxorbZeKpxjmECP5kTE6KqTQGPUL6BHyfrFlTwBAEB2NBgAACA7GgwAAJAdDQYAAMiOBgMAAGRHgwEAALKjwQAAANnRYAAAgOxoMAAAQHY0GAAAIDsaDAAAkB0NBgAAyI4GAwAAZEeDAQAAsqPBAAAA2dFgAACA7GgwAABAdjQYAAAgOxoMAACQHQ0GAADIjgYDAABkR4MBAACyo8EAAADZ0WAAAIDsaDAAAEB2NBgAACC7jfvyZtuLJK2S9LKklyJidI6kAKAdqGHotmjRomRsu+22S8Y22oj/01P61GAUDoyIFRnWAwBVoIYBLUDrBQAAsutrgxGS7rQ9x/bEHAkBQBtRw4AW6eshkrERscT2NpJ+ZvvRiLin/gXFLy2/uAA6UcMaRv0CyuvTCEZELCk+L5N0s6QxPbxmSkSM5uQpAJ2mtxpG/QLKK91g2N7M9uDux5IOkjQvV2IA0ErUMKC1+nKIZKikm213r+cHEXFHlqzQtF122SUZ+/KXv5yMfexjH0vGIqJULjNnzkzGzj///GTst7/9bantAX1EDetnPv7xjydjW2+9dTK2fPnyVqSzwSvdYETEQkl7ZcwFANqGGga0FtNUAQBAdjQYAAAgOxoMAACQHQ0GAADIjgYDAABk57JTEkttzG7fxtYzgwYNSsY+/elPJ2MXXHBBMvaGN7whGSum5vWoFT8Tq1evTsb233//ZGz27NnZc+lAc7iQU+ejfq0fGk1FPe6445Kx7bffPhn7p3/6p2SMafbp+sUIBgAAyI4GAwAAZEeDAQAAsqPBAAAA2dFgAACA7GgwAABAdjQYAAAgu77crh0Z7bvvvsnYxRdfXGqdq1atSsbGjx+fjD322GPJ2J133pmMNZpHPnDgwGTsxz/+cTJ21llnJWPXXnttMgagfzr++OOTsQMOOCAZu+KKK5IxrnVRDiMYAAAgOxoMAACQHQ0GAADIjgYDAABkR4MBAACyo8EAAADZMU21jUaOHJmMfe973yu1zuXLlydjY8eOTcb+9Kc/ldpeoylgM2bMSMaGDh1aKjZmzJhkjGmqQP+03377JWO77rprqXVedNFFZdNBAiMYAAAgOxoMAACQHQ0GAADIjgYDAABkR4MBAACy67XBsD3V9jLb8+qWvcn2z2z/sfi8ZWvTBIByqGFANZqZpjpN0uWSptctO1vSrIi40PbZxfMv5E9vw3LIIYckYyNGjEjGGt0V9aijjkrGyk5FbeTee+9Nxn7xi18kY0cffXSp7W211Val3gfUmSZq2AZl5513Tsbe/OY3J2Nz585Nxp5//vm+pIQe9DqCERH3SFq51uLDJV1TPL5G0hF50wKAPKhhQDXKnoMxNCK6JKn4vE2+lACg5ahhQIu1/EqetidKmtjq7QBAbtQvoLyyIxhLbQ+TpOLzstQLI2JKRIyOiNEltwUAuTVVw6hfQHllG4yZkiYUjydIujVPOgDQFtQwoMWamaZ6naRfS9rV9mLbn5R0oaQP2P6jpA8UzwGg41DDgGo4Itq3Mbt9G+tACxcuTMYa3Wm10fTPAw88sC8pZfX5z38+GfvWt76VjNlOxn7zm98kY+9+97ubyms9MIch+M7X3+tXJ5k/f34ytvnmmydj//iP/5iMPf74433KqR9L1i+u5AkAALKjwQAAANnRYAAAgOxoMAAAQHY0GAAAIDsaDAAAkF3LLxXe34wenZ5t2GgqaqPpwhdffHFfUsrq9a9/fTJ2/PHHJ2Nlp0OvWLGi1PsArN+OPPLIZGy33XZLxhpdDoCpqO3FCAYAAMiOBgMAAGRHgwEAALKjwQAAANnRYAAAgOxoMAAAQHZMU81s9erVydiqVauSsUGDBpV6Xytsu+22ydiZZ56ZjO29997Zc+mkKboA2ufDH/5wMtZo2vsZZ5zRinRQAiMYAAAgOxoMAACQHQ0GAADIjgYDAABkR4MBAACyo8EAAADZMU01s4ceeigZazTlctKkScnYjBkzkrFGdzC9++67k7FGLrroomTsmGOOKbXORv7whz8kYw888ED27QHoDI3uijp27NhS61y5cmXZdJAZIxgAACA7GgwAAJAdDQYAAMiOBgMAAGRHgwEAALKjwQAAANn1Ok3V9lRJH5S0LCL2LJZNkvRpScuLl50TEf/RqiQ3FDNnzkzGjj322GRs1113TcZuv/32ZGz27NnJmO1kbJ999knGyrrrrruSsUZTbV944YXsuaB/oYZVa9SoUclYo2nvO+20UzK2cOHCZGzp0qVN5YXWa2YEY5qkg3tY/s2IGFV88IsJoFNNEzUMaLteG4yIuEcSVy4BsF6ihgHV6Ms5GCfbfsj2VNtbpl5ke6Lt2bbT4/UA0H691jDqF1Be2QZjsqQdJY2S1CXpktQLI2JKRIyOiNEltwUAuTVVw6hfQHmlGoyIWBoRL0fEGklXSRqTNy0AaB1qGNB6pRoM28Pqnh4paV6edACg9ahhQOs1M031OknjJA2xvVjS+ZLG2R4lKSQtknRi61LccPz2t79Nxg444IBk7KijjkrGTjrppGTsXe96VzLWaJpqRCRjjVx55ZXJ2Gc+85lS6wT6ihrWvK233rrU+5YvX56MHXHEEcnYWWedlYw1qkM333xzMvboo48mY2ivXhuMiOjpAg1XtyAXAMiOGgZUgyt5AgCA7GgwAABAdjQYAAAgOxoMAACQHQ0GAADIrtdZJGiPFStWJGNXXHFFMnbvvfcmY42mxTayatWqZOyjH/1oMnbHHXeU2h6AzjB27NhS77vlllvyJtKLM888s63bQzmMYAAAgOxoMAAAQHY0GAAAIDsaDAAAkB0NBgAAyI4GAwAAZMc01fXAVlttlYzdcMMN2bfXaCrqbbfdln17jWy++ebJ2LPPPtvGTIANX7unmzZyxhlnVJ0C+ogRDAAAkB0NBgAAyI4GAwAAZEeDAQAAsqPBAAAA2dFgAACA7Jim2iE23jj9rZg+fXoytttuu5Xa3he/+MVkrN13RR03blwytmjRomSMaarAhmvlypVVp4A+YgQDAABkR4MBAACyo8EAAADZ0WAAAIDsaDAAAEB2NBgAACA7R0TjF9gjJE2X9GZJayRNiYjLbL9J0g2SRkpaJOnoiHiml3U13lg/dswxxyRjP/jBD0qt86677krG3v/+95daZytce+21ydgpp5ySjG1A09jmRMToqpPYEFG/2mPkyJHJ2GOPPZaMfeUrX0nGzjvvvL6khPZJ1q9mRjBeknR6RLxN0r6STrK9u6SzJc2KiJ0lzSqeA0AnoX4BFem1wYiIroh4sHi8StJ8ScMlHS7pmuJl10g6okU5AkAp1C+gOut0DobtkZL2lnS/pKER0SXVfoklbZM9OwDIhPoFtFfTlwq3PUjSjZJOjYhnbTf7vomSJpZLDwD6jvoFtF9TIxi2N1Htl/P7EXFTsXip7WFFfJikZT29NyKmRMRoTmIDUAXqF1CNXhsM11r9qyXNj4hL60IzJU0oHk+QdGv+9ACgPOoXUJ1mDpGMlXScpIdtzy2WnSPpQkkzbH9S0p8ljW9JhhuQvfbaKxmbNm1aMtZoOPfhhx9OxjppKipQEepXG5xwwgnJ2KWXXpqMff3rX29BNugUvTYYEXGvpNRfuPflTQcA8qF+AdXhSp4AACA7GgwAAJAdDQYAAMiOBgMAAGRHgwEAALKjwQAAANk1falwNGfIkCHJ2PTp05OxgQMHJmN/+9vfkrEJEyYkY+uLWbNmJWMvvfRSGzMBUMZll12WjDW6/s+LL77YinTQIRjBAAAA2dFgAACA7GgwAABAdjQYAAAgOxoMAACQHQ0GAADIjmmq62jAgAEN4+edd14y9va3vz0Zi4hk7IYbbkjG5s6d2zCf9UGjW9UD6Azjx6fvaH/aaaclY4ceemgr0sF6gBEMAACQHQ0GAADIjgYDAABkR4MBAACyo8EAAADZ0WAAAIDsmKa6jo4//viG8ZNOOikZW716dTL2wx/+MBn77Gc/23tiANBCn//855OxoUOHJmN/+ctfWpAN1geMYAAAgOxoMAAAQHY0GAAAIDsaDAAAkB0NBgAAyK7XBsP2CNt32Z5v+3e2TymWT7L9pO25xQd3tAHQUahfQHWamab6kqTTI+JB24MlzbH9syL2zYi4uHXpdZ5Gdz2VpCVLliRjt9xySzL2uc99rmxKANKoX5nsv//+VaeA9UyvDUZEdEnqKh6vsj1f0vBWJwYAfUX9AqqzTudg2B4paW9J9xeLTrb9kO2ptrfMnRwA5EL9Atqr6QbD9iBJN0o6NSKelTRZ0o6SRqn2H8IlifdNtD3b9uy+pwsA6476BbRfUw2G7U1U++X8fkTcJEkRsTQiXo6INZKukjSmp/dGxJSIGB0Ro3MlDQDNon4B1WhmFoklXS1pfkRcWrd8WN3LjpQ0L396AFAe9QuoTjOzSMZKOk7Sw7bnFsvOkXSs7VGSQtIiSSe2ID8A6AvqF1AR9zbtMuvG7PZtDFh/zGEIvvNRv4AeJesXV/IEAADZ0WAAAIDsaDAAAEB2NBgAACA7GgwAAJAdDQYAAMiOBgMAAGRHgwEAALKjwQAAANnRYAAAgOxoMAAAQHY0GAAAIDsaDAAAkF0zt2vPaYWkx4vHQ4rnnaKT8iGXtE7KJ1cu22dYB1qvvn5JG+bPYg6dlIvUWflsiLkk61dbb9f+qg3bszvpFtWdlA+5pHVSPp2UC9qvk77/5JLWSfn0t1w4RAIAALKjwQAAANlV2WBMqXDbPemkfMglrZPy6aRc0H6d9P0nl7ROyqdf5VLZORgAAGDDxSESAACQXSUNhu2Dbf/e9gLbZ1eRQ10ui2w/bHuu7dkVbH+q7WW259Ute5Ptn9n+Y/F5ywpzmWT7yWL/zLV9aJtyGWH7Ltvzbf/O9inF8rbvmwa5VLJvUK1Oql9FPpXVMOpXMpeOqV+95NPS/dP2QyS2B0j6g6QPSFos6QFJx0bEI21N5JV8FkkaHRGVzE22fYCk5yRNj4g9i2UXSVoZERcWBWzLiPhCRblMkvRcRFzc6u2vlcswScMi4kHbgyXNkXSEpBPU5n3TIJejVcG+QXU6rX4VOS1SRTWM+pXMpWPqVy/5tLSGVTGCMUbSgohYGBGrJV0v6fAK8ugIEXGPpJVrLT5c0jXF42tU+0GoKpdKRERXRDxYPF4lab6k4apg3zTIBf0P9asO9atnnVS/esmnpapoMIZLeqLu+WJVW6xD0p2259ieWGEe9YZGRJdU+8GQtE3F+Zxs+6FiCLItQ3r1bI+UtLek+1XxvlkrF6nifYO267T6JXVeDaN+1emk+tVDPlIL908VDYZ7WFblVJaxEfFOSYdIOqkYZsMrJkvaUdIoSV2SLmnnxm0PknSjpFMj4tl2bruJXCrdN6hEp9UviRrWCPWrcT4t3T9VNBiLJY2oe76tpCUV5CFJioglxedlkm5WbQi0akuLY2bdx86WVZVIRCyNiJcjYo2kq9TG/WN7E9V+Gb4fETcViyvZNz3lUuW+QWU6qn5JHVnDqF/qrPqVyqfV+6eKBuMBSTvb3sH2QEkfkTSzgjxke7PihBfZ3kzSQZLmNX5XW8yUNKF4PEHSrVUl0v3LUDhSbdo/ti3paknzI+LSulDb900ql6r2DSrVMfVL6tgaRv3qoPrVKJ9W759KLrRVTIX5lqQBkqZGxNfankQtj7eq1vFLtTvL/qDdudi+TtI41e5st1TS+ZJukTRD0naS/ixpfES0/OSlRC7jVBs+C0mLJJ3YfQyxxbnsJ+mXkh6WtKZYfI5qxw3bum8a5HKsKtg3qFan1K8il0prGPUrmUvH1K9e8mlpDeNKngAAIDuu5AkAALKjwQAAANnRYAAAgOxoMAAAQHY0GAAAIDsaDAAAkB0NBgAAyI4GAwAAZEeDAQAAsqPBAAAA2dFgAACA7GgwAABAdjQYAAAgOxoMAACQHQ0GAADIjgYDAABkR4MBAACyo8EAAADZ0WAAAIDsaDAAAEB2NBgAACA7GgwAAJAdDQYAAMhu47682fbBki6TNEDS9yLiwl5eH33ZHrCBWhERW1edRH+0LjWM+gX0KFm/So9g2B4g6TuSDpG0u6Rjbe9edn1AP/Z41Qn0R9QwIItk/erLIZIxkhZExMKIWC3pekmH92F9ANBO1DCghfrSYAyX9ETd88XFMgBYH1DDgBbqyzkY7mHZa45R2p4oaWIftgMArdBrDaN+AeX1pcFYLGlE3fNtJS1Z+0URMUXSFImTpAB0lF5rGPULKK8vh0gekLSz7R1sD5T0EUkz86QFAC1HDQNaqPQIRkS8ZPtkST9VbYrX1Ij4XbbMAKCFqGFAazmifaN+DDECPZoTEaOrTgKNUb+AHiXrF1fyBAAA2dFgAACA7GgwAABAdjQYAAAgOxoMAACQHQ0GAADIjgYDAABkR4MBAACyo8EAAADZ0WAAAIDsaDAAAEB2NBgAACA7GgwAAJAdDQYAAMiOBgMAAGRHgwEAALKjwQAAANnRYAAAgOxoMAAAQHYbV50AOs8xxxyTjJ177rnJ2O67756MnX322cnYFVdckYytWrUqGQOAta1YsSIZ23fffZOxBQsWtCKdfo0RDAAAkB0NBgAAyI4GAwAAZEeDAQAAsqPBAAAA2dFgAACA7Po0TdX2IkmrJL0s6aWIGJ0jKeSxyy67JGOnnXZaMjZx4sRkLCJKxb7xjW8kY0OGDEnGLrnkkmRs2bJlyRjQDGrY+qnRlPgXXnghGWPae3vluA7GgRGRnngMAJ2NGga0AIdIAABAdn1tMELSnbbn2E6PqwNAZ6KGAS3S10MkYyNiie1tJP3M9qMRcU/9C4pfWn5xAXSihjWM+gWU16cRjIhYUnxeJulmSWN6eM2UiBjNyVMAOk1vNYz6BZRXusGwvZntwd2PJR0kaV6uxACglahhQGv15RDJUEk32+5ezw8i4o4sWaFp++yzTzI2a9asZGzQoEGltvfcc88lY7/+9a+TsenTpydj73nPe5KxW2+9NRl797vfnYwBTaCGrac+9KEPJWPDhw9PxgYPHpyMLV26tE854bVKNxgRsVDSXhlzAYC2oYYBrcU0VQAAkB0NBgAAyI4GAwAAZEeDAQAAsqPBAAAA2eW42RlabL/99kvGJk2alIyVnYp65ZVXJmOXXnppMrZgwYJS23vHO96RjG2//fal1glgw/XOd74zGXvqqaeSsb/+9a+tSAcJjGAAAIDsaDAAAEB2NBgAACA7GgwAAJAdDQYAAMiOBgMAAGRHgwEAALLjOhgdYptttknGbr/99mRs0003LbW9RteeeOSRR0qts6ynn346Gdt6662TsaOOOqrhen/0ox+VzglA5xo/fnwy9pvf/CYZW758eSvSQQIjGAAAIDsaDAAAkB0NBgAAyI4GAwAAZEeDAQAAsqPBAAAA2TFNtUMsXbo0GVuzZk0ytnr16mTsxBNPTMbaPRW1rI03Tv+IfulLX2r4XqapAhsm28lYV1dXGzNBI4xgAACA7GgwAABAdjQYAAAgOxoMAACQHQ0GAADIjgYDAABk1+s0VdtTJX1Q0rKI2LNY9iZJN0gaKWmRpKMj4pnWpblhOOSQQ5KxRlNRIyIZ++EPf5iMTZ8+vbnEOlij/TJixIiG791zzz2TsXnz5pXOCesXatiGp1FNvOuuu9qYCRppZgRjmqSD11p2tqRZEbGzpFnFcwDoRNNEDQPartcGIyLukbRyrcWHS7qmeHyNpCPypgUAeVDDgGqUvZLn0IjokqSI6LK9TeqFtidKmlhyOwDQCk3VMOoXUF7LLxUeEVMkTZEk2+kDZwDQYahfQHllZ5EstT1MkorPy/KlBAAtRw0DWqxsgzFT0oTi8QRJt+ZJBwDaghoGtFgz01SvkzRO0hDbiyWdL+lCSTNsf1LSnyWNb2WS65O3vOUtydhXv/rVUutsNO3qtNNOK7XODcEWW2zRMD5s2LBkjGmq/Qc1DKhGrw1GRBybCL0vcy4AkB01DKgGV/IEAADZ0WAAAIDsaDAAAEB2NBgAACA7GgwAAJBdy6/k2d+8733pE9P32muvZOzFF19Mxs4///xk7Omnn24uMQBYj7zuda8r9b477rgjcyYoixEMAACQHQ0GAADIjgYDAABkR4MBAACyo8EAAADZ0WAAAIDsmKaa2T777FPqfbNmzUrGfvWrX5VNZ4O2fPnyhvHHHnusTZkAyO3MM88s9b6VK1dmzgRlMYIBAACyo8EAAADZ0WAAAIDsaDAAAEB2NBgAACA7GgwAAJAd01QzGzduXDJmOxm7/fbbW5DN+uH0009PxjbaKN0DP/XUUw3Xu2DBgtI5AWi9gQMHJmMf/vCHk7FGtRSdgxEMAACQHQ0GAADIjgYDAABkR4MBAACyo8EAAADZ0WAAAIDsep2manuqpA9KWhYRexbLJkn6tKTu21meExH/0aok1ycRUSo2efLkVqTTMY488shkbNCgQcnYmjVrkrFG+xPoRg3rXIMHD07GGt2Zmt/99UMzIxjTJB3cw/JvRsSo4oNfTACdapqoYUDb9dpgRMQ9kla2IRcAyI4aBlSjL+dgnGz7IdtTbW+ZLSMAaA9qGNBCZRuMyZJ2lDRKUpekS1IvtD3R9mzbs0tuCwBya6qGUb+A8ko1GBGxNCJejog1kq6SNKbBa6dExOiIGF02SQDIqdkaRv0CyivVYNgeVvf0SEnz8qQDAK1HDQNar5lpqtdJGidpiO3Fks6XNM72KEkhaZGkE1uXYmdpNHVKknbYYYc2ZbJ+Oe+885KxTTfdtNQ6b7zxxrLpoB+hhgHV6LXBiIhje1h8dQtyAYDsqGFANbiSJwAAyI4GAwAAZEeDAQAAsqPBAAAA2dFgAACA7HqdRYJX22qrrRrGG90ZdEP3qU99KhnbaaedSq3z6aefTsa++93vllongM5w0EEHlXrfokWLkrEXX3yxZDbIjREMAACQHQ0GAADIjgYDAABkR4MBAACyo8EAAADZ0WAAAIDsmKbaIUaPHp2MzZ49u42ZNLbHHnskY9/+9reTsYEDB5ba3uWXX56MPfPMM6XWCaAzDBkypNT7fvKTnyRjL7zwQtl0kBkjGAAAIDsaDAAAkB0NBgAAyI4GAwAAZEeDAQAAsqPBAAAA2TFNdR2tXr26Yfzll19OxjbeOL27DzvssGSs3dNUjznmmGTs3HPPTcZe97rXldreV77ylWTsggsuKLVOAJ3vox/9aDJmOxm76aabWpEOMmMEAwAAZEeDAQAAsqPBAAAA2dFgAACA7GgwAABAdjQYAAAgO0dE4xfYIyRNl/RmSWskTYmIy2y/SdINkkZKWiTp6IhoeHtL2403tgG48847k7H3vve9ydjf//73ZOyggw5Kxu67775kbJdddknGTjvttGRs4sSJyVhvPy8pt9xySzL2sY99LBl78cUXS21vPTMnItK300Vp1K/qvfWtb03G5s+fn4w1mtY/YMCAPuWErJL1q5kRjJcknR4Rb5O0r6STbO8u6WxJsyJiZ0mziucA0EmoX0BFem0wIqIrIh4sHq+SNF/ScEmHS7qmeNk1ko5oUY4AUAr1C6jOOp2DYXukpL0l3S9paER0SbVfYknbZM8OADKhfgHt1fSlwm0PknSjpFMj4tlGl3Fd630TJaUP6gNAi1G/gPZragTD9iaq/XJ+PyK6LwK/1PawIj5M0rKe3hsRUyJiNCexAagC9QuoRq8Nhmut/tWS5kfEpXWhmZImFI8nSLo1f3oAUB71C6hOM9NU95P0S0kPqzbNS5LOUe045gxJ20n6s6TxEbGyl3Vt8NO8jjvuuGTs3//935OxRkO28+bNS8a6urqSsX333TcZGzRoUKlcVq1alYxdd911ydhZZ51Vap39BNNUW4T6Vb2ddtopGfv9739fap1MU+0oyfrV6zkYEXGvpNRfnPf1JSsAaCXqF1AdruQJAACyo8EAAADZ0WAAAIDsaDAAAEB2NBgAACA7GgwAAJBd05cKR3MWLlyYjD344IPJ2D777JOM7bHHHqViZTW6RfqECROSsVtv5VpFAF7t4IMPLvW+stfIQOdgBAMAAGRHgwEAALKjwQAAANnRYAAAgOxoMAAAQHY0GAAAIDumqWZ23333JWMHHnhgMnbYYYclY1/+8peTsbe97W3NJbaWyZMnJ2MXXHBBMrZ8+fJS2wPQPz355JPJ2JIlS5Kxf/mXf2lFOmgjRjAAAEB2NBgAACA7GgwAAJAdDQYAAMiOBgMAAGRHgwEAALJzRLRvY3b7NgasP+ZExOiqk0Bj1C+gR8n6xQgGAADIjgYDAABkR4MBAACyo8EAAADZ0WAAAIDsem0wbI+wfZft+bZ/Z/uUYvkk20/anlt8HNr6dAGgedQvoDrN3E31JUmnR8SDtgdLmmP7Z0XsmxFxcevSA4A+oX4BFem1wYiILkldxeNVtudLGt7qxACgr6hfQHXW6RwM2yMl7S3p/mLRybYfsj3V9pa5kwOAXKhfQHs13WDYHiTpRkmnRsSzkiZL2lHSKNX+Q7gk8b6Jtmfbnt33dAFg3VG/gPZr6lLhtjeRdJukn0bEpT3ER0q6LSL27GU9XGoXeC0uFd5C1C+gpcpfKty2JV0taX79L6ftYXUvO1LSvL5mCQA5Ub+A6jQzi2SspOMkPWx7brHsHEnH2h4lKSQtknRiC/IDgL6gfgEV4W6qQPU4RLIeoH4BPeJuqgAAoH1oMAAAQHY0GAAAIDsaDAAAkB0NBgAAyI4GAwAAZEeDAQAAsqPBAAAA2dFgAACA7GgwAABAdjQYAAAgOxoMAACQHQ0GAADIrpnbtee0QtLjxeMhxfNO0Un5kEtaJ+WTK5ftM6wDrVdfv6QN82cxh07KReqsfDbEXJL1q623a3/Vhu3ZnXSL6k7Kh1zSOimfTsoF7ddJ339ySeukfPpbLhwiAQAA2dFgAACA7KpsMKZUuO2edFI+5JLWSfl0Ui5ov076/pNLWifl069yqewcDAAAsOHiEAkAAMiukgbD9sG2f297ge2zq8ihLpdFth+2Pdf27Aq2P9X2Mtvz6pa9yfbPbP+x+LxlhblMsv1ksX/m2j60TbmMsH2X7fm2f2f7lGJ52/dNg1wq2TeoVifVryKfymoY9SuZS8fUr17yaen+afshEtsDJP1B0gckLZb0gKRjI+KRtibySj6LJI2OiErmJts+QNJzkqZHxJ7FsoskrYyIC4sCtmVEfKGiXCZJei4iLm719tfKZZikYRHxoO3BkuZIOkLSCWrzvmmQy9GqYN+gOp1Wv4qcFqmiGkb9SubSMfWrl3xaWsOqGMEYI2lBRCyMiNWSrpd0eAV5dISIuEfSyrUWHy7pmuLxNar9IFSVSyUioisiHiwer5I0X9JwVbBvGuSC/of6VYf61bNOql+95NNSVTQYwyU9Ufd8saot1iHpTttzbE+sMI96QyOiS6r9YEjapuJ8Trb9UDEE2ZYhvXq2R0raW9L9qnjfrJWLVPG+Qdt1Wv2SOq+GUb/qdFL96iEfqYX7p4oGwz0sq3Iqy9iIeKekQySdVAyz4RWTJe0oaZSkLkmXtHPjtgdJulHSqRHxbDu33UQule4bVKLT6pdEDWuE+tU4n5bunyoajMWSRtQ931bSkgrykCRFxJLi8zJJN6s2BFq1pcUxs+5jZ8uqSiQilkbEyxGxRtJVauP+sb2Jar8M34+Im4rFleybnnKpct+gMh1Vv6SOrGHUL3VW/Url0+r9U0WD8YCknW3vYHugpI9ImllBHrK9WXHCi2xvJukgSfMav6stZkqaUDyeIOnWqhLp/mUoHKk27R/blnS1pPkRcWldqO37JpVLVfsGleqY+iV1bA2jfnVQ/WqUT6v3TyUX2iqmwnxL0gBJUyPia21PopbHW1Xr+KXanWV/0O5cbF8naZxqd7ZbKul8SbdImiFpO0l/ljQ+Ilp+8lIil3GqDZ+FpEWSTuw+htjiXPaT9EtJD0taUyw+R7Xjhm3dNw1yOVYV7BtUq1PqV5FLpTWM+pXMpWPqVy/5tLSGcSVPAACQHVfyBAAA2dFgAACA7GgwAABAdjQYAAAgOxoMAACQHQ0GAADIjgYDAABkR4MBAACy+39/T57T4MEq/AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x576 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "examples = enumerate(test_loader)\n",
    "#one test batch of batch_size_test images:\n",
    "batch_idx, (X_test, y_test) = next(examples)\n",
    "#Make classes with one-hot encoding from the target classes:\n",
    "y_test_onehot=torch.zeros(batch_size_test,num_classes)\n",
    "for spl in range(batch_size_test):\n",
    "    y_test_onehot[spl, y_test[spl]]=1.0\n",
    "\n",
    "# plot 4 images as gray scale\n",
    "plt.figure(figsize=(10,8))\n",
    "plt.subplot(221)\n",
    "plt.title(\"Some Images of the Trainigs Set\")\n",
    "plt.imshow(X_train[0,0], cmap=plt.get_cmap('gray'))\n",
    "plt.subplot(222)\n",
    "plt.imshow(X_train[1,0], cmap=plt.get_cmap('gray'))\n",
    "plt.subplot(223)\n",
    "plt.imshow(X_train[2,0], cmap=plt.get_cmap('gray'))\n",
    "plt.subplot(224)\n",
    "plt.imshow(X_train[3,0], cmap=plt.get_cmap('gray'))\n",
    "# show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train.shape= torch.Size([100, 784])\n",
      "X_test.shape= torch.Size([10, 784])\n"
     ]
    }
   ],
   "source": [
    "X_testunflat=X_test.clone() #keep original for imshow later\n",
    "# flatten 28*28 images to a 784 vector for each image\n",
    "num_pixels = X_train.shape[2] * X_train.shape[3]\n",
    "\n",
    "\n",
    "#Flatten the images:\n",
    "X_train = X_train.view(X_train.shape[0], num_pixels)\n",
    "print(\"X_train.shape=\",X_train.shape)\n",
    "X_test = X_test.reshape(X_test.shape[0], num_pixels)\n",
    "print(\"X_test.shape=\",X_test.shape)\n",
    "\n",
    "# normalize inputs from 0-255 to 0-1\n",
    "#X_train = X_train / 255\n",
    "#X_test = X_test / 255\n",
    "\n",
    "# one hot encode outputs\n",
    "#y_train = np_utils.to_categorical(y_train)\n",
    "#y_test = np_utils.to_categorical(y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generate Model:\n"
     ]
    }
   ],
   "source": [
    "# build the model\n",
    "print(\"Generate Model:\")\n",
    "model = DenseNet()#.to('cpu')\n",
    "    \n",
    "loss_fn = nn.MSELoss()\n",
    "#learning_rate = 1e-4\n",
    "optimizer = torch.optim.Adam(model.parameters())#, lr=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "It trains the network on the MSELoss, using just 10 iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 0.1498100459575653\n",
      "2 0.09042678028345108\n",
      "4 0.12411704659461975\n",
      "6 0.06358765810728073\n",
      "8 0.022911611944437027\n",
      "Duration of optimization: 0.12865567207336426\n"
     ]
    }
   ],
   "source": [
    "# Fit the model\n",
    "X_train=X_train.to(device)\n",
    "y_train_onehot=y_train_onehot.to(device)\n",
    "starttime=time.time()\n",
    "for epoch in range(10):\n",
    "    Ypred=model(X_train)\n",
    "    loss=loss_fn(Ypred, y_train_onehot)\n",
    "    if epoch%2==0:\n",
    "        print(epoch, loss.item())\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "       \n",
    "endtime=time.time()\n",
    "print(\"Duration of optimization:\", endtime-starttime)\n",
    "#model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=2, batch_size=200, verbose=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss on the test set: tensor(0.0603, grad_fn=<MseLossBackward>)\n"
     ]
    }
   ],
   "source": [
    "# Final evaluation of the model\n",
    "Ypred=model(X_test)\n",
    "loss=loss_fn(Ypred, y_test_onehot)\n",
    "#scores = model.forward(X_test, y_test, verbose=0)\n",
    "print(\"Loss on the test set:\", loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "After that it displays a test image of a digit, prints the values of the 10 classes from the neural network. Observe those numbers. Do they show a confident detection? Then it displays the number of the class with the highest value. If this highest value is below 0.4, the it is flagged as “probably not a digit”."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Picture  0\n",
      "X_testorig.shape= torch.Size([28, 28])\n",
      "sample.shape= torch.Size([784])\n",
      "Test Ypred= tensor([ 0.0233,  0.1547, -0.0467,  0.0799,  0.2158,  0.1013,  0.3038, -0.0160,\n",
      "         0.2188,  0.2137], grad_fn=<AddBackward0>)\n",
      "prediction= tensor(6)\n",
      "Probably not a digit\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZYAAAEICAYAAAB8lNKlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAAAc+ElEQVR4nO3de7gcVZnv8e8PSFASFCICIQGjgHI3alAcUePxAuLRRORiRA3ewmHEcXicUQ7HS0blpiDgOEaDMATkFo5y0aOCMIN4IWjiIARBZSCQkJAIiAFBlPCeP9baWOl01+7eWbtrh/w+z7Of3V2rqvqtVavrrbWqulsRgZmZWSmbNB2AmZk9vTixmJlZUU4sZmZWlBOLmZkV5cRiZmZFObGYmVlRjSQWScdL+kbpeW1kk/R9STObjmNDJOkiSdP79FpHSvpJ5fkjkl7Qh9e9TtIHu5x3qqRllee3Spra5bJdzzuU+QdZ11pxDxdJIWmX9V1W0rmSPp8f7yPpZ92so+vEknf6HyRt3sV8f5b0sKTVkhZJOq66XEScGBFdNaDqvJIm5Y3erOb1Z0v6Zrfb1RRJSyS9YT2WD0l/ym/6eyV9SdKmJWMsLSLeHBHzmo5jfeSDzCP5b01u6wPPjx/C+p5649bMsw/wYuCKoca9PiJibETcWTdPN+/N4RQRe0bEdb3O283xopd1t1qfA/xIExE3Aw9Jeutg83aVWCRNAl4NBPC2LhY5JiK2BMYDHwPeCXxPkrp5PevaiyNiLPBa4HDg/Q3H87SXDzJjc73/mNTWx+a/E4fpZY8CLoghfJpZiYe8rZQLSO2xVrcN7r3AAuBcoOuhjIj4U870bwNeCbwF1j1LkPReSXdLekDSp6pn8y3zXp//P5TPEF85WAz5jOHvJf0u96I+J2lnSTfkHtV8SaPzvFtL+q6k3+fe2XclTays6/mSrs/ruUbSv7Vsx36SfibpIUm/6tR9lnQ+sBPwnbwdH8/T35bPiB/KPb/du6znO4CfApMrr/E/Jd2U1/WzfNY7UPZSSf+Vt+NSSZdUz5olfUjSHZIelHSlpB0qZW+S9BtJf5T0VUk/Uh6+UB5CkXRqrr+7JL25sux1lXl/VTnTfyTvp6m91GOe9zhJ/5235deS3l4p2yXH90dJ90u6pGY9l0q6L897vaQ9u6n7lnW8X9JteduvkvS8PF2STpe0Kq//Zkl7SZoFHAF8PNfBdzqs+s3Ajyqvc6Skn0r617y+2yW9vlJ+naQTJP0UeBR4gaTdJP0w79PfSDqsMv9z8n5eLennwM4t21UdGnmmpNOU3q9/zPv7mXR4b3aqk1z2xhz7HyV9Beh44plf99y8nl8D+7aUV48Zz5Q0L897m6SPa+1hsyWS3iDpQOB44PAc8686vHbr8Wi+pPNym7tV0pQOyw3UyUBbP7xS9rHcHlZIel9l+uZK7597JK2U9LVcv+3WP1j7foPSce8PSscqVZbtuF8GcR3weg0yckVEDPoH3AH8PfAy4K/AdjXzXgd8sM3064FT8uPZwDfz4z2AR4D9gdHAqfk13tBm3kmkXtNmNa//1Pz5eQBXAs8C9gQeB64FXgA8G/g1MDPP+xzgHcAWwJbApcDllXXdkOMbneNdXYltAvAAcBApYb8xP39uhziXDGxjfv5C4E95uVHAx3O9j+6wfAC75Me7ASuAY/PzlwKrgFcAm5JOBpYAm+fY7wY+ml/nYOAvwOfzsv8DuD+vY3PgX4Hrc9k2eZsPBjbL6/jrwP4GjszPP5Rf92hgOaBB2sYs4Pa8j3qtx0OBHfK8h+c6HJ/LLgL+Ty57BrB/Tbt5f97nmwNnADd18b54anuA6Xl/7Z7r5pPAz3LZAcAiYCvSwXP3SoznDtR9h9cYk/f1cyvTjgSeAI7N+/Bw4I/AuEpc95Da+2akdr4UeF9+/tK8j/fM818MzM+vtRdwL/CTDm3t3/L6J+R9/He5zibR8t4cpE4G2tIheRuOzdu0TvvI859M6iGOA3YEFgPL2r2f8rw/ArYGJgI318w7m8rxYrD3ap7/z6T2uSlwErCgZtmn6i4/n5q387N5uw8iJf+tc/kZpOPVOFJ7/A5wUod1d2zf+XW/S2pzOwG/Bw4cbL+02d/n0tI+837bp7bOunjz7E86WGyTn99OPoAN9mZrmX4xcFbrzgQ+DVxUmW8L0oGuZGJ5VeX5IuATleenAWd0WNdk4A/58U65QWxRKf9mJbZPAOe3LH8VOWnVNdb8/FPA/MrzTUhv8Kk1DXY16UAauZFtnsvmAJ9rmf83pCGz1+T1qlL2E/6WWM4GvlApG5v3/yRSz/WGSplIB6xqYrmjZV8GsH2ntpHb1yrghUOpxzb1chMwLT8+D5gLTOxm2co6tspxP3uQ+Z7aHuD7wAda9t+jwPNIyfq3wH7AJi3rOJf6xDIhx/KMyrQjqSTsPO3nwHsqcX22UnY48OOW9X4d+Azp4PhXYLdK2Ym0SSx5mx4jDcG2xjmJdRNLXZ28l8oBObelZa3to1J+J/nAmJ/PonOyuBM4oFL2wZp5Z9N7YrmmUrYH8FjNsu0Sy2Mt9bQqtw2R3s87V8peCdzVYd0d23d+3WqimQ8cN9h+aY25XfskHT9eU1dn3QyFzQSujoj78/ML6WE4rGIC8GCb6TuQDk4ARMSjpDPUklZWHj/W5vlYAElbSPp67uavJvWytlK6KL4D8GCOb8DSyuPnAYcqDd88JOkh0kFzfJcx7kDqSQAQEU/m9U+oWealOfbDSb2TMZVYPtYSy475NXYA7o3cQtpsR2scj5D2xwTW3VdBOhhU3VcpH6irse2Cl7QjqcHPjIjfVmLvuh6VhlFvqsy7F+lsGFKvT8DP85BF22tQkjaVdLLSkNpq0oGEynq68TzgzEocD+bXnhAR/wF8hXS2v1LSXEnP6nK9D+X/W7ZMb92Hd5P2z4DWtvmKljo9AtgeeC7prLU6/920tw3pzPi/u4y9Y53Qvi0tbbeSbAe6i7HdvHXrHYr7Ko8fBZ6h3m5aeCAinmhZx1jSvtgCWFSpsx/k6e0M1r5b4xx4H9btl25syd/aZVu1iSWP7R0GvFZp/Pk+Upf1xZJe3GUQAweQl5G6sq1WkLqr1dd8TodVRYfppXwMeBHwioh4FunsHlKlrwDGSdqiMv+OlcdLSWfaW1X+xkTEyR1eq3VblpN2eHrBNB66I+nsoKNI5pOG6T5dieWElli2iIiL8nZMqI63tmxHaxxjSPvjXtbdV6o+70Xez5eTeovfrxR1XY95XPgs4BjgORGxFWmIRAARcV9EfCgidiBdcPyq2t+h8y5gGvAG0rDRpIGX6GGTlgJHtcT9zIj4WY7lyxHxMtLw1AuBf87L1bbpiPgT6UD+wpai1n24E2nfPbVoS2w/aoltbEQcTRoieYK128BOHcK5nzQMtHObsnbbUVcnK6qvWWnvnayguxgH5q22y7r1DvcxpRf3k05096zU17Mj3Siyjh7ad6vatlpH6XrraNIISEeD9VimA2tI3b3J+W93UoJ4bxdBbCHptaTbJH8OfK/NbP8XeKukv1O6iP4vdH5D/x54knR9ZDhsSdqxD0kaRxoqACAi7gYWArMljVa6OFm97e6bpO04IJ8BP0PpnvVOB96VrL0d84G3SHq9pFGkJPc40NV946Rx5VmSticdbP+XpFcoGSPpLZK2JCWgNcAxkjaTNA14eWU9FwLvkzQ5X6A7EbgxIpYA/w/YW9L0fIb2YdJZ71CcA9weEV9omd5LPQ5cf/g9QL4IutdAoaRDK8v9Ic+7ps16tiTV9QOkM8ah3N31NeB/K1/0l/RsSYfmx/vmfTGKNNTx50ocre2gne+RhjGrtgX+QdKo/Dq70/79BWms/YWS3pPnH5Vj2j0i1gDfJrXrLSTtQYcRidyLPgf4kqQd8v55ZW4n7d6bHeuE1Jb2lHRwbkv/QH1bmp/XtXXepx/pct4JpBOPTlYCkzR8d851s3+Bp+r3LOB0SdsCSJog6YB28/fQvlvV7ZfBTAX+IyIer5tpsMqcCfx7RNyTs+N9EXEfqVt/RE337yuSHiZV6hnAt0jjo0+2zhgRt5IaycWkM42HSWOO6wSeh1ZOAH6au3H7DRJ/r84Ankk6c1hA6oZWHUEa83wA+DxwyUCcEbGUdNZ7POlNtpR0Vtqpjk8CPpm3458i4jfAu0kXy+8nJa23RsRfugk8Im4hXbD854hYSLqA/hVSg7uDNC5PXt/BwAdI3dl3kw48A9txLel6z7dI+2Nn0u3i5OHQQ4Ev5DrYg5RsaxtZB+8E3q617wx7dS/1GBG/Jl0ju4HU1vYm3R03YF/gRkmPkC6IfjQi7moTy3mkoZV7STdzLOh1YyLiMuAU4GKl4bTFpLu5IN2UcBZpX9xNqrtTc9nZwB65HVzeYfVzSe+36gnXjcCupLZyAnBIRLQdQo6Ih4E3kep8OWmI5BTSRXdIB96xefq5wL/XbOo/AbcAvyANoZxCum60znuzrk4qbenkXB+7sva+a/UvpLq7C7gaOL9m3s+ShmjvAq4hnbx2aqOX5v8PSPplzTqHajYwL9fJYYPNTLrGeAewINfZNaRRlHa6bd9rGaStDuYIUmKqNXC3zoghaSzpgLdrN5XUJKXb+26PiM8MOvMIJulG4GsRUXdAabfcJqQ38BER8Z/DEpwBIOlC0s0dl0s6knSRe/+Gw9ogSDoaeGdEtPb6rAeS9gbmRsSgH/MYER+ckvTW3A0fQzqTu4W/XUAdMfLwwc6SNlG6B34a6TrBBkXSayVtn4fCZgL7sG7vrNOyB0jaKg9/HE8atuz5DN96ExHviojLm45jQyBpvKRX5ffpi0jDypc1HdeGLiJu6SapQLobZCSYRuraijS08s4YaV2pZHvSePRzSGfqR0fEfzUb0pC8iDQOPZZ0YfiQiFjR5bKvJF2HGU0aNpoeEY8NS5RmQzOadDv180mjHxcDX20yoI3NiBsKMzOzDduIGAozM7Onj5EyFLZRkuTuotkwiwh/+W2fucdSkKQDlb7g7w5JxzUdj5lZE3yNpRClr335LelLE5eR7vOfkT9r0WkZV77ZMHOPpf/cYynn5aQvYLwzfwjxYtLdbmZmGxUnlnImsPaX3S2jzZe6SZolaaGkhX2LzMysj3zxvpx23e11hroiYi7pKzo8FGZmT0vusZSzjLW/RXUia3/brJnZRsGJpZxfALsq/XzxaNIX/l3ZcExmZn3nobBCIuIJSceQfu1wU+Cc/M3NZmYbFd9u3CBfYzEbfr7duP88FGZmZkU5sZiZWVFOLGZmVpQTi5mZFeXEYmZmRTmxmJlZUU4sZmZWlBOLmZkV5cRiZmZFObGYmVlRTixmZlaUE4uZmRXlxGJmZkU5sZiZWVFOLGZmVpQTi5mZFeXEYmZmRTmxmJlZUU4sZmZWlBOLmZkV5cRiZmZFObGYmVlRTixmZlaUE4uZmRXlxGJmZkU5sZiZWVFOLGZmVpQTi5mZFeXEYmZmRW3WdABmG5IZM2bUls+ePbu2/Bvf+EbHsi9+8YtDCclsxHFiKUjSEuBhYA3wRERMaTYiM7P+c2Ip73URcX/TQZiZNcXXWMzMrCgnlrICuFrSIkmz2s0gaZakhZIW9jk2M7O+8FBYWa+KiOWStgV+KOn2iLi+OkNEzAXmAkiKJoI0MxtO7rEUFBHL8/9VwGXAy5uNyMys/5xYCpE0RtKWA4+BNwGLm43KzKz/PBRWznbAZZIg1euFEfGDZkOy0r785S/Xlm+zzTa15fvss0/JcMxGJCeWQiLiTuDFTcdhZtY0D4WZmVlRTixmZlaUE4uZmRXlxGJmZkU5sZiZWVG+K8ysYtq0abXl48aNqy1fvnx5bfnJJ5/cc0xmGxr3WMzMrCgnFjMzK8qJxczMinJiMTOzopxYzMysKCcWMzMryonFzMyK8udYbKMyefLk2vKvf/3r67X+o446qrb81ltvXa/1m20I3GMxM7OinFjMzKwoJxYzMyvKicXMzIpyYjEzs6KcWMzMrCgnFjMzK0oR0XQMGy1JrvxhsNlmnT+edfXVV9cu+7rXva62fNGiRbXlU6ZMqS23/osINR3DxsY9FjMzK8qJxczMinJiMTOzopxYzMysKCcWMzMryonFzMyKcmIxM7Oi/Hss9rRz9NFHdywb7HMqa9asqS1/xzveMaSY+mHUqFG15ZdccknHssE+z3baaafVlt9www215f683MbFPZYeSTpH0ipJiyvTxkn6oaTf5f9bNxmjmVmTnFh6dy5wYMu044BrI2JX4Nr83Mxso+TE0qOIuB54sGXyNGBefjwPmN7PmMzMRhJfYylju4hYARARKyRt22lGSbOAWX2LzMysz5xY+iwi5gJzwV9CaWZPTx4KK2OlpPEA+f+qhuMxM2uME0sZVwIz8+OZwBUNxmJm1igPhfVI0kXAVGAbScuAzwAnA/MlfQC4Bzi0uQhtzJgxQ152zpw5teV33333kNe9vsaOHVtbfv7559eWT58+vWPZ7bffXrvsYYcdVlu+YMGC2nJ/jmXj4sTSo4iY0aHo9X0NxMxshPJQmJmZFeXEYmZmRTmxmJlZUU4sZmZWlBOLmZkVJd8G2Bx/8n5oJk6cWFt+1VVXdSwb7Kvlp0yZUlu+evXq2vLBTJo0qWPZYF/pf+yxx9aW77333rXljz32WMeyGTM63eyYXHHFhvvRrIhQ0zFsbNxjMTOzopxYzMysKCcWMzMryonFzMyKcmIxM7OinFjMzKwoJxYzMyvK325sG5x99923tnyPPfboWLZmzZraZX/wgx+sV/lg3vOe93Qs22WXXdZr3TfeeGNt+ac//emOZVdfffV6vbZZlXssZmZWlBOLmZkV5cRiZmZFObGYmVlRTixmZlaUE4uZmRXlxGJmZkX591ga5N9jGZqddtqptrzu91h222230uEU88QTT9SWn3jiibXlZ555Zm35gw8+2HNMTwf+PZb+c4/FzMyKcmIxM7OinFjMzKwoJxYzMyvKicXMzIpyYjEzs6KcWMzMrCh/jqVB/hzL8Kj7nMucOXNqlz3ooINKh7OWu+66q2PZu971rtplFyxYUDqcjYI/x9J/7rH0SNI5klZJWlyZNlvSvZJuyn/De3QyMxvBnFh6dy5wYJvpp0fE5Pz3vT7HZGY2Yjix9Cgirgc2zu/GMDPrghNLOcdIujkPlW3daSZJsyQtlLSwn8GZmfWLE0sZc4CdgcnACuC0TjNGxNyImBIRU/oUm5lZXzmxFBARKyNiTUQ8CZwFvLzpmMzMmuLEUoCk8ZWnbwcWd5rXzOzpbrOmA9jQSLoImApsI2kZ8BlgqqTJQABLgKOais9g6dKlHcuuueaa2mUH+xzLYJ/7Ov3002vLP/WpT3Use/TRR2uXNdtQOLH0KCJmtJl8dt8DMTMboTwUZmZmRTmxmJlZUU4sZmZWlBOLmZkV5cRiZmZF+WvzG+SvzR8ehxxySMeySy+9dL3WfeSRR9aWz5s3b73Wb+X5a/P7zz0WMzMryonFzMyKcmIxM7OinFjMzKwoJxYzMyvKicXMzIpyYjEzs6L87ca2wZk0aVJt+UknnTTkdZ9xxhm15RdccMGQ1222sXCPxczMinJiMTOzopxYzMysKCcWMzMryonFzMyKcmIxM7OinFjMzKwo/x5Lg/x7LO2NGjWqtvzaa6+tLX/1q1/dsWzOnDm1y37kIx+pLV+zZk1tuY08/j2W/nOPxczMinJiMTOzopxYzMysKCcWMzMryonFzMyKcmIxM7OinFjMzKwo/x5LjyTtCJwHbA88CcyNiDMljQMuASYBS4DDIuIPTcU5km2ySf35zKmnnlpbXvc5FYDLLrusY9kpp5xSu6w/p2K2/txj6d0TwMciYndgP+DDkvYAjgOujYhdgWvzczOzjY4TS48iYkVE/DI/fhi4DZgATAPm5dnmAdMbCdDMrGFOLOtB0iTgJcCNwHYRsQJS8gG2bTA0M7PG+BrLEEkaC3wL+MeIWC1193VEkmYBs4YzNjOzJrnHMgSSRpGSygUR8e08eaWk8bl8PLCq3bIRMTcipkTElP5Ea2bWX04sPVLqmpwN3BYRX6oUXQnMzI9nAlf0OzYzs5HAX5vfI0n7Az8GbiHdbgxwPOk6y3xgJ+Ae4NCIeHCQdW2UlT969Oja8scff3y91v/ud7+7Y9kFF1ywXuu2DY+/Nr//fI2lRxHxE6BTQ319P2MxMxuJPBRmZmZFObGYmVlRTixmZlaUE4uZmRXlxGJmZkU5sZiZWVG+3dg2OJdffnlt+YUXXtifQMysLfdYzMysKCcWMzMryonFzMyKcmIxM7OinFjMzKwoJxYzMyvKicXMzIry51hsg7PffvvVlh988MEdyxYsWFC77L333jukmMzsb9xjMTOzopxYzMysKCcWMzMryonFzMyKcmIxM7OinFjMzKwoJxYzMytKEdF0DBstSRtl5UuqLZ81a1Zt+Sc/+cna8okTJ3YsW7x4ce2yBxxwQG358uXLa8tt5ImI+gZnxbnHYmZmRTmxmJlZUU4sZmZWlBOLmZkV5cRiZmZFObGYmVlRTixmZlaUP8fSI0k7AucB2wNPAnMj4kxJs4EPAb/Psx4fEd8bZF2ufLNh5s+x9J8TS48kjQfGR8QvJW0JLAKmA4cBj0TEqT2sy5VvNsycWPrPvyDZo4hYAazIjx+WdBswodmozMxGDl9jWQ+SJgEvAW7Mk46RdLOkcyRt3WGZWZIWSlrYrzjNzPrJQ2FDJGks8CPghIj4tqTtgPuBAD5HGi57/yDrcOWbDTMPhfWfE8sQSBoFfBe4KiK+1KZ8EvDdiNhrkPW48s2GmRNL/3korEdKX817NnBbNanki/oD3g7Uf42umdnTlHssPZK0P/Bj4BbS7cYAxwMzgMmkobAlwFH5Qn/dulz5ZsPMPZb+c2JpkBOL2fBzYuk/D4WZmVlRTixmZlaUE4uZmRXlxGJmZkU5sZiZWVFOLGZmVpQTi5mZFeXEYmZmRTmxmJlZUU4sZmZWlBOLmZkV5cRiZmZFObGYmVlRTixmZlbUZk0HsJG7H7g7P94mPx+JHNvQOLahKRnb8wqtx3rg32MZISQtjIgpTcfRjmMbGsc2NCM5NuuOh8LMzKwoJxYzMyvKiWXkmNt0ADUc29A4tqEZybFZF3yNxczMinKPxczMinJiMTOzopxYRgBJB0r6jaQ7JB3XdDxVkpZIukXSTZIWNhzLOZJWSVpcmTZO0g8l/S7/33oExTZb0r257m6SdFADce0o6T8l3SbpVkkfzdMbr7ea2BqvN1s/vsbSMEmbAr8F3ggsA34BzIiIXzcaWCZpCTAlIhr/MJ2k1wCPAOdFxF552heAByPi5JyUt46IT4yQ2GYDj0TEqf2OpxLXeGB8RPxS0pbAImA6cCQN11tNbIfRcL3Z+nGPpXkvB+6IiDsj4i/AxcC0hmMakSLieuDBlsnTgHn58TzSganvOsTWuIhYERG/zI8fBm4DJjAC6q0mNtvAObE0bwKwtPJ8GSPrzRXA1ZIWSZrVdDBtbBcRKyAdqIBtG46n1TGSbs5DZY0M0w2QNAl4CXAjI6zeWmKDEVRv1jsnluapzbSRND75qoh4KfBm4MN5yMe6MwfYGZgMrABOayoQSWOBbwH/GBGrm4qjnTaxjZh6s6FxYmneMmDHyvOJwPKGYllHRCzP/1cBl5GG7kaSlXmsfmDMflXD8TwlIlZGxJqIeBI4i4bqTtIo0oH7goj4dp48IuqtXWwjpd5s6JxYmvcLYFdJz5c0GngncGXDMQEgaUy+qIqkMcCbgMX1S/XdlcDM/HgmcEWDsaxl4MCdvZ0G6k6SgLOB2yLiS5WixuutU2wjod5s/fiusBEg3055BrApcE5EnNBsRImkF5B6KZB+YuHCJmOTdBEwlfS16iuBzwCXA/OBnYB7gEMjou8X0TvENpU0nBPAEuCogesafYxrf+DHwC3Ak3ny8aRrGY3WW01sM2i43mz9OLGYmVlRHgozM7OinFjMzKwoJxYzMyvKicXMzIpyYjEzs6KcWMzMrCgnFjMzK+r/A8LeJUpyGbnYAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Picture  1\n",
      "X_testorig.shape= torch.Size([28, 28])\n",
      "sample.shape= torch.Size([784])\n",
      "Test Ypred= tensor([ 0.5608, -0.1397,  0.1339,  0.1205, -0.4723,  0.2635, -0.4472,  0.1408,\n",
      "         0.1972,  0.0358], grad_fn=<AddBackward0>)\n",
      "prediction= tensor(0)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZYAAAEICAYAAAB8lNKlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAAAq/0lEQVR4nO3deZgU1dUG8Pc4siiDiCA7OIiALAISwBUhaiKQiKJBFExYjBiMiAZRQRQ+1BAV1xhJBkUgURZlkYgSdxajyCLKsCkgCsgiOwMoAuf7494xTTP31MxQTI/y/p6Hh+l+61bfrqru07WLqoKIiCgux6W6A0RE9NPCwkJERLFiYSEiolixsBARUaxYWIiIKFYsLEREFKuUFBYRGSAiz8Y9LBVtIvK6iHRNdT9+jERkrIhcWUiv1U1EZic8zhaR0wvhdd8Tkd/ncdjWIrI24fFiEWmdx7Z5HrYgw0eM65B+Hy0ioiJyxpG2FZFRIvKA/7uRiPw3L+PIc2HxM32biJTIw3DfisguEdkpIvNF5O7Edqr6Z1XN0wKUOKyIZPg3fbzx+oNF5F95fV+pIiKrReTSI2ivIrLbf+jXichjIpIWZx/jpqptVXV0qvtxJPyXTLb/d8Av6zmPBxRgfD98cI1hGgFoDOCVgvb7SKhquqqusobJy2fzaFLVBqr6Xn6Hzcv3RX7GnexIvuCLGlX9FMB2Ebk8atg8FRYRyQDQEoACaJ+HJreoamkAlQH0BXAtgNdERPLyepRnjVU1HUArAJ0A9Ehxf37y/JdMup/us+CW9XT/789H6WVvAvCCFuBsZnG4yZvi8gLc8mjK6wL3OwAfAhgFIM+bMlR1t6/07QGcB+BXwOG/EkTkdyLypYhsEZF7E3/NJw070/+/3f9CPC+qD/4Xw80i8rlfi7pfRGqJyAd+jWqCiBT3w5YVkVdF5Bu/dvaqiFRLGFdNEZnpx/OWiPwt6X2cKyL/FZHtIvJJaPVZRP4JoAaAf/v3cad/vr3/Rbzdr/nVy+N0XgHgfQBNEl7j1yKy0I/rv/5Xb07WVEQ+9u/jJREZn/irWURuFJEVIrJVRKaKSJWE7JcislxEdojIMyIyQ/zmC/GbUERkmJ9+X4hI24S27yUM+0nCL/1sP59a52c6+mHvFpGV/r0sEZEOCdkZvn87RGSziIw3xvOSiGzww84UkQZ5mfZJ4+ghIkv9e/+PiJzmnxcReVxENvnxfyoiDUWkJ4AuAO700+DfgVG3BTAj4XW6icj7IvJXP75lInJJQv6eiDwoIu8D2APgdBE5U0Te9PN0uYhckzB8OT+fd4rIRwBqJb2vxE0jJ4jIo+I+rzv8/D4Bgc9maJr47Be+7ztE5GkAwR+e/nVH+fEsAdA8KU/8zjhBREb7YZeKyJ1y6Gaz1SJyqYi0ATAAQCff508Cr538fTRBRMb4ZW6xiDQLtMuZJjnLeqeErK9fHtaLSPeE50uI+/x8JSIbReTvfvrmNv6o5ftScd9728R9V0lC2+B8ifAegEskYssVVDXyH4AVAG4G8DMA3wOoaAz7HoDf5/L8TAAP+b8HA/iX/7s+gGwAFwIoDmCYf41Lcxk2A26t6Xjj9X8Y3j9WAFMBnASgAYDvALwN4HQAZQAsAdDVD1sOwNUATgRQGsBLAKYkjOsD37/ivr87E/pWFcAWAO3gCvYv/ONTA/1cnfMe/eM6AHb7dsUA3Omne/FAewVwhv/7TADrAdzuHzcFsAnAOQDS4H4MrAZQwvf9SwB9/OtcBWAfgAd824sBbPbjKAHgrwBm+qy8f89XATjej+P7nPkNoJt/fKN/3V4AvgYgEctGTwDL/DzK73TsCKCKH7aTn4aVfTYWwD0+KwngQmO56eHneQkATwBYmIfPxQ/vB8CVfn7V89NmIID/+uwyAPMBnAz35VkvoY+jcqZ94DVK+Xl9asJz3QDsB3C7n4edAOwAcEpCv76CW96Ph1vO1wDo7h839fO4gR9+HIAJ/rUaAlgHYHZgWfubH39VP4/P99MsA0mfzYhpkrMs/ca/h9v9ezps+fDD/wVuDfEUANUBZAFYm9vnyQ87A0BZANUAfGoMOxgJ3xdRn1U//Ldwy2cagKEAPjTa/jDt/OPW/n0O8e+7HVzxL+vzJ+C+r06BWx7/DWBoYNzB5du/7qtwy1wNAN8AaBM1X3KZ36OQtHz6+dbInGZ5+PBcCPdlUd4/Xgb/BRb1YUt6fhyAEckzE8B9AMYmDHci3BddnIXlgoTH8wHclfD4UQBPBMbVBMA2/3cNv0CcmJD/K6FvdwH4Z1L7/8AXLWth9Y/vBTAh4fFxcB/w1sYCuxPui1T9QlbCZ8MB3J80/HK4TWYX+fFKQjYb/ysszwF4OCFL9/M/A27N9YOETOC+sBILy4qkeakAKoWWDb98bQJQpyDTMZfpshDAFf7vMQAyAVTLS9uEcZzs+10mYrgf3g+A1wHckDT/9gA4Da5YfwbgXADHJY1jFOzCUtX3pWTCc92QULD9cx8B+G1Cv4YkZJ0AzEoa7z8ADIL7cvwewJkJ2Z+RS2Hx72kv3CbY5H5m4PDCYk2T3yHhC9kvS2uTl4+EfBX8F6N/3BPhYrEKwGUJ2e+NYQcj/4XlrYSsPoC9RtvcCsvepOm0yS8bAvd5rpWQnQfgi8C4g8u3f93EQjMBwN1R8yW5z7ktn3DfHxdZ0ywvm8K6AnhDVTf7xy8iH5vDElQFsDWX56vAfTkBAFR1D9wv1DhtTPh7by6P0wFARE4UkX/41fydcGtZJ4vbKV4FwFbfvxxrEv4+DUBHcZtvtovIdrgvzcp57GMVuDUJAICqHvTjr2q0aer73glu7aRUQl/6JvWlun+NKgDWqV9Ccnkfyf3IhpsfVXH4vFK4L4NEGxLynGmVnlvnRaQ63ALfVVU/S+h7nqejuM2oCxOGbQj3axhwa30C4CO/ySLXfVAikiYifxG3SW0n3BcJEsaTF6cBeDKhH1v9a1dV1XcAPA33a3+jiGSKyEl5HO92/3/ppOeT5+GXcPMnR/KyeU7SNO0CoBKAU+F+tSYO/yVyVx7ul/HKPPY9OE2Q+7K0JreReFWQtz7mNqw13oLYkPD3HgAlJX8HLWxR1f1J40iHmxcnApifMM2m++dzE7V8J/cz53NozZe8KI3/LZe5MguL37Z3DYBW4rY/b4BbZW0sIo3z2ImcL5Cfwa3KJlsPt7qa+JrlAqPSwPNx6QugLoBzVPUkuF/3gJvo6wGcIiInJgxfPeHvNXC/tE9O+FdKVf8SeK3k9/I13Ax3L+i2h1aH+3UQpM4EuM109yX05cGkvpyoqmP9+6iauL016X0k96MU3PxYh8PnlSQ+zg8/n6fArS2+nhDleTr67cIjANwCoJyqngy3iUQAQFU3qOqNqloFbofjM5L7ETqdAVwB4FK4zUYZOS+Rj7e0BsBNSf0+QVX/6/vylKr+DG7zVB0A/Xw7c5lW1d1wX+R1kqLkeVgDbt790DSpbzOS+pauqr3gNpHsx6HLQI1AdzbDbQaqlUuW2/uwpsn6xNdMWN5D1iNvfcwZNnG5tMZ7tL9T8mMz3A/dBgnTq4y6A0UOk4/lO5m5rFrE7W8tDrcFJChqjeVKAAfgVvea+H/14ArE7/LQiRNFpBXcYZIfAXgtl8FeBnC5iJwvbif6/yH8gf4GwEG4/SNHQ2m4GbtdRE6B21QAAFDVLwHMAzBYRIqL2zmZeNjdv+Dex2X+F3BJccesh754N+LQ9zEBwK9E5BIRKQZX5L4DkKfjxuG2K/cUkUpwX7Z/EJFzxCklIr8SkdJwBegAgFtE5HgRuQJAi4TxvAigu4g08Tvo/gxgjqquBjANwFkicqX/hfZHuF+9BTESwDJVfTjp+fxMx5z9D98AgN8J2jAnFJGOCe22+WEP5DKe0nDTegvcL8aCHN31dwD9xe/0F5EyItLR/93cz4ticJs6vk3oR/JykJvX4DZjJqoA4FYRKeZfpx5y/3wBblt7HRH5rR++mO9TPVU9AGAS3HJ9oojUR2CLhF+LHgngMRGp4ufPeX45ye2zGZwmcMtSAxG5yi9Lt8Jelib4cZX187R3HoetCvfDI2QjgAw5ekfO5WX+Avhh+o4A8LiIVAAAEakqIpflNnw+lu9k1nyJ0hrAO6r6nTVQ1MTsCuB5Vf3KV8cNqroBbrW+i7H697SI7IKbqE8AmAi3ffRg8oCquhhuIRkH90tjF9w2x8M67jetPAjgfb8ad25E//PrCQAnwP1y+BBuNTRRF7htnlsAPABgfE4/VXUN3K/eAXAfsjVwv0pD03gogIH+fdyhqssBXA+3s3wzXNG6XFX35aXjqroIbodlP1WdB7cD/Wm4BW4F3HZ5+PFdBeAGuNXZ6+G+eHLex9tw+3smws2PWnCHi8NvDu0I4GE/DerDFVtzIQu4FkAHOfTIsJb5mY6qugRuH9kHcMvaWXBHx+VoDmCOiGTD7RDto6pf5NKXMXCbVtbBHczxYX7fjKpOBvAQgHHiNqdlwR3NBbiDEkbAzYsv4abdMJ89B6C+Xw6mBEafCfd5S/zBNQdAbbhl5UEAv1HVXDchq+ouAL+Em+Zfw20ieQhupzvgvnjT/fOjADxvvNU7ACwCMBduE8pDcPuNDvtsWtMkYVn6i58etXHovEv2f3DT7gsAbwD4pzHsELhNtF8AeAvux2toGX3J/79FRBYY4yyowQBG+2lyTdTAcPsYVwD40E+zt+C2ouQmr8v3ISKW1Shd4AqTKedonSJDRNLhvvBq52UipZK4w/uWqeqgyIGLMBGZA+Dvqmp9oeTW7ji4D3AXVX33qHSOAAAi8iLcwR1TRKQb3E7uC1PcrR8FEekF4FpVTV7ro3wQkbMAZKpq5GkeReLEKRG53K+Gl4L7JbcI/9uBWmT4zQe1ROQ4ccfAXwG3n+BHRURaiUglvymsK4BGOHztLNT2MhE52W/+GAC32TLfv/Apf1S1s6pOSXU/fgxEpLKIXOA/p3XhNitPTnW/fuxUdVFeigrgjgYpCq6AW7UVuE0r12pRW5VyKsFtjy4H90u9l6p+nNouFUhduO3Q6XA7hn+jquvz2PY8uP0wxeE2G12pqnuPSi+JCqY43OHUNeG2fowD8EwqO3SsKXKbwoiI6MetSGwKIyKin46isinsmFSmTBmtVCl8hGWJEvbleL755ptgtmHDhmAGAE2bNjXz3bt3m/mePXuC2fbt2822p59uH30Z1V4irmV6/PHhxfr777832x48eNiBi4eImi6lSpUyc6vvJUuWNNt+8YV9LEtUe2tZW7XKvHgxypYta+aVK9vnAWdlZZl5zZo1g9maNfb5jfXqhS+pt2bNGmzdupUXvy1k3BQWI79D/0m4y2Q8a5wcCQCoW7euZmZmBnPrwwYAw4cPD2aPPPKI2TbqC3LOnDlm/vHH4V1Lr7xiX9395ZdfNvNJkyaZeVTBPfnkk4PZ5s2bgxkA7Nq1y8znzp1r5i1atDBzq+jVqZN8DuShunXrZuZ164aOSnX69esXzK677jqz7dVXX23mgwbZB0bWrl3bzEeNGhXM+vbta7b96KOPglnbtm3xySefsLAUMm4Ki4m4y778De548PoArvMnmxERHVNYWOLTAu4CjKv8SYjj4I52IyI6prCwxKcqDr3Y3VrkclE3EekpIvNEZN6OHTsKrXNERIWFhSU+uW3HPWwHlqpmqmozVW1WpkyZQugWEVHhYmGJz1ocehXVajj0arNERMcEFpb4zAVQW9zti4vDXfBvaor7RERU6HgeS0xUdb+I3AJ3t8M0ACP9lZuDNmzYgKFDhwbzqPM5XnstdJV0oEOHDsEMsM9DAaLPFWnZsmUwizpU+frrrzdz65BcAJg5c6aZW4dCly9v37vr/PPPN3PrsFgAmDrV/i1xzz33BLPGje1bHEUdTvzee++ZuXX4+r599kW0ly1bZubr19tXBIo6HHngwIHB7M033zTbWod4b9q0yWxLRwcLS4xU9TWE74lBRHRM4KYwIiKKFQsLERHFioWFiIhixcJCRESxYmEhIqJYsbAQEVGseLhxCu3bt8+810S1atXM9sWKFQtmW7ZsMdtOmTLFzKPumWLdt6RKlSpm22HDhpn51q1bzTxKr169gtm6devMth07djTzAwcOmHnUeS7PP/98MDvzzDPNthdccIGZz5gxw8ytS+OvXLnSbGvdigAAMjIyzDw9Pd3MW7VqFcyeeuops611r5ef//znZls6OrjGQkREsWJhISKiWLGwEBFRrFhYiIgoViwsREQUKxYWIiKKlagedpNDKiT16tVT6/DU0aNHm+1POumkYPb444+bbYcPH27m1iG7ANC+fftg9vXX9v3N3n//fTNPS0sz8zp16pj5K6+8Esy6d+9utp09e7aZR90SYNWqVWY+bty4YPbwww+bbaNuN/D999+buXVo+1tvvWW2XbFihZlPnDjRzJs3b27mbdq0CWbHHWf//q1UqVIwu/jii7Fw4UL7HhAUO66xEBFRrFhYiIgoViwsREQUKxYWIiKKFQsLERHFioWFiIhixcJCRESx4mXzU2j37t348MMPg/lnn31mtl+yZEkw69q1q9m2RYsWZm5dYh0Avvzyy2DWrVs3s63VbwB4/fXXzTzqPBjrEvCzZs0y2zZr1szMo87v6dy5s5n/5z//CWZdunQx2w4YMMDMX375ZTM///zzg9m3335rts3OzjbzqHkadZ7L3XffHczefPNNs611vteGDRvMtnR0cI2FiIhixcJCRESxYmEhIqJYsbAQEVGsWFiIiChWLCxERBQrFhYiIooVz2NJobVr16J///7BfOHChWb7PXv2BLODBw+abaPucdG4cWMzX7ZsWTC74YYbzLZR9/YQsW+f0bt3bzMvXrx4MHvqqafMtlH3wImablH3TNm7d28wizo/Z//+/WaekZFh5tb4zznnHLPtvHnzzDwrK8vMo+6h8+ijjwazSy65xGxr3Yfm7bffNtvS0cHCEiMRWQ1gF4ADAParqn22HRHRTxALS/x+rqqbU90JIqJU4T4WIiKKFQtLvBTAGyIyX0R65jaAiPQUkXkiYm+0JiL6keKmsHhdoKpfi0gFAG+KyDJVnZk4gKpmAsgEgOOOO05T0UkioqOJaywxUtWv/f+bAEwGYF9CmIjoJ4iFJSYiUkpESuf8DeCXAOxjMImIfoK4KSw+FQFM9udgHA/gRVWdbjXIyMjAkCFDgvl7771nvuCkSZOC2cCBA822f/zjH8383//+t5nPnz8/mFn3/QCizyW58847zfyDDz4wc6vvc+bMMduWK1fOzGfMmGHmU6ZMMfOvvvoqmF1wwQVm2yeffNLMt27daubWeSxR9y0ZM2aMmTdt2tTMo/rWpEmTYDZz5sxgBgCff/55MIu6zwwdHSwsMVHVVQDsswqJiI4B3BRGRESxYmEhIqJYsbAQEVGsWFiIiChWLCxERBQrUeXJ36lSsmRJrV69ejCfOnWq2f7WW28NZrNmzTLbLlmyxMx3795t5pMnTy5QvwDggQceMPMaNWqYeaVKlcz84osvDmaPPfaY2Xbt2rVm3qdPHzPv1KmTmdevXz+YzZ4922xbpkwZM7///vvNvGXLlsGsTZs2ZtvmzZubeaNGjcz8Zz/7mZlb733w4MFmW+uS/l26dMGSJUvs+zBQ7LjGQkREsWJhISKiWLGwEBFRrFhYiIgoViwsREQUKxYWIiKKFQsLERHFiuexpFB6ero2bNgwmC9evNhs365du2AWdcn9EiVKmHnUJdzvuOOOYHbuueeabe+9914zb9HCvj/a888/b+bFixcPZlu2bDHbPvPMM2beo0cPM4861+SRRx4JZrfddpvZ9rXXXjPzSy+91MytZSLqHJhnn33WzKdNm2bmWVn2rYnKli0bzFasWGG2tc6L2rBhA7777juex1LIuMZCRESxYmEhIqJYsbAQEVGsWFiIiChWLCxERBQrFhYiIooVCwsREcXq+FR34FhWoUIF8/4e06dPN9u3atUqmH322Wdm2wYNGpj5oEGDCpzfcsstZttNmzaZee3atc389ttvN/PevXsHs44dO5pt3333XTOPOv9nx44dZm7dx2b9+vVm26h7mvzmN78x8wkTJgSzvXv3mm2jzh2qUqWKmS9dutTMrff+0EMPmW2t99WtWzezLR0dXGMhIqJYsbAQEVGsWFiIiChWLCxERBQrFhYiIooVCwsREcWKhYWIiGLF+7GkUIMGDXT8+PHBvFevXmb7vn37BrOo+4LUrFnTzAcPHmzm1vgzMzPNtldeeaWZV69e3czPOeccM8/IyAhmUfcFueeee8w86hwc614wADBs2LBgNnz4cLPtaaedZuaVKlUy86pVqwazGjVqmG3/+c9/mnnr1q3N/J133jHz008/PZiVK1fObDtr1qxglpWVhezsbN6PpZBxjSWfRGSkiGwSkayE504RkTdF5HP/f/iuRUREP3EsLPk3CkCbpOfuBvC2qtYG8LZ/TER0TGJhySdVnQlga9LTVwAY7f8eDeDKwuwTEVFRwsISj4qquh4A/P8VQgOKSE8RmSci87Zt21ZoHSQiKiwsLIVMVTNVtZmqNitblrtiiOinh4UlHhtFpDIA+P/tQ4eIiH7CWFjiMRVAV/93VwCvpLAvREQpxfNY8klExgJoDaA8gI0ABgGYAmACgBoAvgLQUVWTd/AfpkKFCmrdQ6Nfv35m+3bt2gWzqPMtos5LqFatmpnfeuutwSzqviDTpk0z865du5p51HkNM2bMCGZR58CsWLHCzK17vQDA2rVrzdy6P8iaNWvMthUqBHfdAQBGjhxp5q+//nowW7Zs2RG99pGe/7Nr165gNnXqVLNt+/btg1mnTp2wePFinsdSyHijr3xS1esC0SWF2hEioiKKm8KIiChWLCxERBQrFhYiIooVCwsREcWKhYWIiGLFw41TqHjx4lq+fPlgvnnzZrP98uXLg1nt2rXNtqtXrzbzevXqmfmIESOCWfPmzc221mXtgehL9k+ZMsXMN2zYEMyiDqOOuvz7I488YuYlSpQw886dOwezJk2amG2HDh1q5meeeaaZW4dpr1u3zmzbv39/Mz/hhBPMvEuXLmZeqlSpYGYd2g4ALVu2DGY9evTA0qVLebhxIeMaCxERxYqFhYiIYsXCQkREsWJhISKiWLGwEBFRrFhYiIgoViwsREQUK57HkkKnn366Pvjgg8G8e/fuZnvrcuJVqlQx2/bq1cvMn376aTO3zlto2LCh2bZRo0Zm/uqrr5r5G2+8YeZ169YNZllZWWbbqPd9ww03mLl1+XcAmDNnTjA744wzzLatWrUy85tvvtnML7/88mAWde7RV199ZebvvPOOme/bt8/MH3744WB21VVXmW0rVqwYzFauXIm9e/fyPJZCxjUWIiKKFQsLERHFioWFiIhixcJCRESxYmEhIqJYsbAQEVGsWFiIiChWx6e6A8cyEUFaWlowf//99832HTp0CGbFihUz20ad53LhhRea+dy5c4PZqaeearbt06ePmQ8ZMsTMo85j2bFjRzCLOh8jMzPTzK1zZAA3Tws6/qh5FnV/ng8++MDMp0+fHsyi5vfkyZPN3LrXCwBs27bNzF9++eVg1rdvX7Pt3//+92B28OBBsy0dHVxjISKiWLGwEBFRrFhYiIgoViwsREQUKxYWIiKKFQsLERHFioWFiIhixfNYUujAgQPYvXt3MN+/f7/Z3jpG/9FHHzXbRt0fo2bNmmZep06dYLZ48WKzbdR5KvXq1TPzqPHXqlUrmD311FNm2zFjxph5mzZtzLxMmTJmbs2zqOnSo0cPM3/yySfN/KGHHgpm1v11otoCwPXXX2/mvXv3NvOdO3cGs6hzZEaNGhXMos4roqODayz5JCIjRWSTiGQlPDdYRNaJyEL/r10q+0hElEosLPk3CkBuP1sfV9Um/t9rhdwnIqIig4Uln1R1JoCtqe4HEVFRxcISn1tE5FO/qaxsaCAR6Ski80RkXnZ2dmH2j4ioULCwxGM4gFoAmgBYDyC451xVM1W1mao2S09PL6TuEREVHhaWGKjqRlU9oKoHAYwA0CLVfSIiShUWlhiISOWEhx0AZIWGJSL6qeN5LPkkImMBtAZQXkTWAhgEoLWINAGgAFYDuCkv41q7di369esXzFu0sFd8HnnkkWDWrFkzs23UeQdNmjQx85NPPjmYtWrVymw7cOBAM7/rrrvMfOnSpWZevHjxYDZ+/Hizbf/+/c18/vz5Zn7ttdeaufX6s2fPNttOmjTJzM866ywznzFjRoHbtmtnH0EfdZ+a8uXLm3nnzp2DWadOncy2zzzzTDDr1auX2ZaODhaWfFLV63J5+rlC7wgRURHFTWFERBQrFhYiIooVCwsREcWKhYWIiGLFwkJERLESVU11H45ZVatW1ZtuCh+ZvH37drO9dUhx1OGd7du3N/PKlSub+bp164LZ1KlTzbZNmzY189tuu83Mp02bZuY1atQIZlGXzY86rHbw4MFmvnz5cjO3bmcQ9drffvutmZcsWdLMrUOdo6b5hx9+aOYLFiww8w4dOph59erVg9ncuXPNtosWLQpmF110ERYsWMBr5xcyrrEQEVGsWFiIiChWLCxERBQrFhYiIooVCwsREcWKhYWIiGLFwkJERLHi1Y1TaO/evVi8eHEw/+6778z2O3fuDGZR54rcd999Zl6tWjUzt86/+cMf/mC2LV26tJnfeeedZn722WebuXXp+kGDBpltGzZsaOZR5weddNJJZr5s2bJgZt2KAADS0tLM3LqNAgA88cQTwexPf/qT2da6ND0A3HvvvWberVs3My9bNng3b5QrV85sa90mQYSnsKQC11iIiChWLCxERBQrFhYiIooVCwsREcWKhYWIiGLFwkJERLFiYSEioljxPJYUSktLM897sI7tB4CtW7cGs7Fjx5pt69WrZ+aPP/64mY8aNSqYtWrVymz70ksvmfm+ffvM/G9/+5uZf/3118HMOm8IAFq2bGnm1rkgAPDiiy+a+WWXXRbMJk+ebLaNOo/lqquuMnNL1L1cbr/9djNv27atmffv39/MP/3002DWp08fs+2pp54azLKzs822dHRwjYWIiGLFwkJERLFiYSEiolixsBARUaxYWIiIKFYsLEREFCsWFiIiipWoaqr78KMiItUBjAFQCcBBAJmq+qSInAJgPIAMAKsBXKOq26xxpaena+PGjYN5hw4dzL5Y9x2pUKFCgdsCwLfffmvmnTt3DmZ169Y12/bs2dPMu3TpYuZLliwx81tvvTWYTZw40Wwb1feoe6ZY56kAwIknnhjMZs+ebba1zlsCgDJlypj5r371q2AWNb+3bTMXZQwZMsTMO3XqZOZnnnlmMPv1r39ttp0zZ04w2717Nw4cOMCbshQyrrHk334AfVW1HoBzAfxRROoDuBvA26paG8Db/jER0TGHhSWfVHW9qi7wf+8CsBRAVQBXABjtBxsN4MqUdJCIKMVYWI6AiGQAOBvAHAAVVXU94IoPAHtbFBHRTxSvFVZAIpIOYCKA21R1Z17vrS0iPQH0BOx7dRMR/VhxjaUARKQYXFF5QVUn+ac3ikhln1cGsCm3tqqaqarNVLVZsWLFCqfDRESFiIUln8StmjwHYKmqPpYQTQXQ1f/dFcArhd03IqKigIcb55OIXAhgFoBFcIcbA8AAuP0sEwDUAPAVgI6qah4fWrlyZe3evXswHzFihNmXHj16BLPx48ebbZ977jkzf/XVV8383nvvDWZz5841265atcrM9+zZY+ZRl4dv0KBBMKtevbrZduXKlWYetclz0aJFZn7zzTcHs0cffdRsG/W+ow7L3bhxYzAbM2aM2fbZZ5818xIlSpj51KlTzTxqebRkZWUFs169emH58uU83LiQcR9LPqnqbAChBfWSwuwLEVFRxE1hREQUKxYWIiKKFQsLERHFioWFiIhixcJCRESxYmEhIqJY8XDjFEpLS0Pp0qWDeXZ2ttm+ZMmSwWz//v1m26jLv0ddRv2jjz4KZo0aNTLbNm3a1Mzbt29v5lOmTDFz65yJffv2mW2tS+4D0X1bv369mc+YMSOYLV++3Gx79tlnm3nlypXNfNCgQcEsapp27drVzAcOHGjmLVu2NPNx48YFM+vcHwCYOXNmMNu1a5fZlo4OrrEQEVGsWFiIiChWLCxERBQrFhYiIooVCwsREcWKhYWIiGLFwkJERLHieSwptHnzZowcOTKY9+7d22y/dOnSYPbEE0+YbaPOt1iwYIGZn3HGGcGscePGZtsHH3zQzKdPn27mFStWNPNy5coFs5NOOslsG3X+zo033mjmEydONHPrfJC3337bbGu9LwD461//auaff/55MIvqd8+ePc086jyXqHvsjB49OphF3adm586dwezAgQNmWzo6uMZCRESxYmEhIqJYsbAQEVGsWFiIiChWLCxERBQrFhYiIooVCwsREcWK57GkUMWKFXHHHXcE8+bNm5vt09LSglmfPn3MtitWrDDz8uXLm7l13xLrHBcAeOGFF8y8Y8eOZm5NMwBYtGhRMNuyZYvZtnr16mbeuXNnM2/btq2Zn3LKKcEs6hyZYsWKmXlU39euXRvMRowYYbbt16+fmderV8/Mhw0bZubWuSrz5883295zzz3B7J133jHb0tHBNRYiIooVCwsREcWKhYWIiGLFwkJERLFiYSEiolixsBARUaxYWIiIKFY8jyWfRKQ6gDEAKgE4CCBTVZ8UkcEAbgTwjR90gKq+Zo2rRIkSqFmzZjAfMmSI2Rfr3IEyZcqYbV966SUz79Gjh5mPGjUqmEXdC6Zp06Zmvm7dOjPPzs42c+u8h2effdZsG3UeyowZM8y8fv36Zv7AAw8Es+eee85sO3ToUDO//fbbzfyKK64IZuPHjzfb1qlTx8w//fRTM7fONQGAzMzMYHbfffeZbceOHRvMtm7daralo4OFJf/2A+irqgtEpDSA+SLyps8eV1X7TDAiop84FpZ8UtX1ANb7v3eJyFIAVVPbKyKiooP7WI6AiGQAOBvAHP/ULSLyqYiMFJGygTY9RWSeiMzbsWNHYXWViKjQsLAUkIikA5gI4DZV3QlgOIBaAJrArdHkevEjVc1U1Waq2ixqPwgR0Y8RC0sBiEgxuKLygqpOAgBV3aiqB1T1IIARAFqkso9ERKnCwpJPIiIAngOwVFUfS3i+csJgHQBkFXbfiIiKAu68z78LAPwWwCIRWeifGwDgOhFpAkABrAZwU9SIvvvuO/Py9ZdffrnZftOmTcFs+vTpZtuzzz7bzNu3b2/mJUuWDGau9oZFHW5cunRpMz/hhBPM3Dq0ddy4cWbbAQMGmPngwYPNfNmyZWa+atWqYHbNNdeYbVeuXGnmUZeI79+/fzB79913zbZRt0L4+OOPzXz37t1mnpUV/h1WqlQps227du2C2T/+8Q+zLR0dLCz5pKqzAeT2zWmes0JEdKzgpjAiIooVCwsREcWKhYWIiGLFwkJERLFiYSEiolixsBARUaxEVVPdh2NW2bJltXXr1sE86lpin3zySTCbNm2a2XbkyJFmft5555n59ddfH8zuv/9+s23UeS7PPPOMmVvvGwBq1aoVzLZs2WK2jer71Vdfbebdu3c384yMjGD29NNPm20HDhxo5lHn0Fj2799v5lG3E1i9erWZ9+3b18zPOuusYJaenm62vfnmm4PZ2LFjsXHjRnuBo9hxjYWIiGLFwkJERLFiYSEiolixsBARUaxYWIiIKFYsLEREFCsWFiIiihXPY0khEfkGwJf+YXkAm1PYHQv7VjDsW8HE2bfTVPXUmMZFecTCUkSIyDxVbZbqfuSGfSsY9q1ginLfKG+4KYyIiGLFwkJERLFiYSk6MlPdAQP7VjDsW8EU5b5RHnAfCxERxYprLEREFCsWFiIiihULSxEgIm1EZLmIrBCRu1Pdn0QislpEFonIQhGZl+K+jBSRTSKSlfDcKSLypoh87v8vW4T6NlhE1vlpt1BE2qWgX9VF5F0RWSoii0Wkj38+5dPN6FvKpxsdGe5jSTERSQPwGYBfAFgLYC6A61R1SUo75onIagDNVDXlJ9OJyEUAsgGMUdWG/rmHAWxV1b/4olxWVe8qIn0bDCBbVYcVdn8S+lUZQGVVXSAipQHMB3AlgG5I8XQz+nYNUjzd6MhwjSX1WgBYoaqrVHUfgHEArkhxn4okVZ0JYGvS01cAGO3/Hg33xVToAn1LOVVdr6oL/N+7ACwFUBVFYLoZfaMfORaW1KsKYE3C47UoWh8uBfCGiMwXkZ6p7kwuKqrqesB9UQGokOL+JLtFRD71m8pSspkuh4hkADgbwBwUsemW1DegCE03yj8WltTL7X7cRWn75AWq2hRAWwB/9Jt8KG+GA6gFoAmA9QAeTVVHRCQdwEQAt6nqzlT1Ize59K3ITDcqGBaW1FsLoHrC42oAvk5RXw6jql/7/zcBmAy36a4o2ei31edss9+U4v78QFU3quoBVT0IYARSNO1EpBjcF/cLqjrJP10kpltufSsq040KjoUl9eYCqC0iNUWkOIBrAUxNcZ8AACJSyu9UhYiUAvBLAFl2q0I3FUBX/3dXAK+ksC+HyPni9jogBdNORATAcwCWqupjCVHKp1uob0VhutGR4VFhRYA/nPIJAGkARqrqg6ntkSMip8OtpQDA8QBeTGXfRGQsgNZwl1XfCGAQgCkAJgCoAeArAB1VtdB3ogf61hpuc44CWA3gppz9GoXYrwsBzAKwCMBB//QAuH0ZKZ1uRt+uQ4qnGx0ZFhYiIooVN4UREVGsWFiIiChWLCxERBQrFhYiIooVCwsREcWKhYWIiGLFwkJERLH6fwzbcivOFmdxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#\"Inference\", predict example digit test image:\n",
    "for pic in range(2):\n",
    "    print(\"Picture \", pic)\n",
    "    if pic==0: #take image from test set:\n",
    "        X_testorig=X_testunflat[1,0]\n",
    "        sample=X_test[1,:]\n",
    "    if pic==1: #read externa random test image, size 28x28 pixel:\n",
    "        from PIL import Image\n",
    "        image = Image.open('./images/randpic.jpg')\n",
    "        import torchvision.transforms.functional as TF\n",
    "        X_testorig = TF.to_tensor(image) \n",
    "        X_testorig = X_testorig[0,:,:]\n",
    "        sample=X_testorig.view(28*28)\n",
    "      \n",
    "    print(\"X_testorig.shape=\",X_testorig.shape)\n",
    "    plt.imshow(X_testorig, cmap=plt.get_cmap('gray'))\n",
    "    plt.title(\"A Digit Image to Regognize as a Test (predicted digit in the shell)\")\n",
    "\n",
    "    #sample=\n",
    "    print(\"sample.shape=\", sample.shape)\n",
    "    Ypred=model(sample)\n",
    "    print(\"Test Ypred=\", Ypred)\n",
    "    prediction=torch.argmax(Ypred) #use argmax as the class with the largest output\n",
    "    print(\"prediction=\", prediction)\n",
    "    #Now we have the possibility to detect the case \"probably not a digit\" if the max output is too low,\n",
    "    #since we know from the target function that perfect detection means \"1.0\" and \n",
    "    #perfect non-detection is \"0.0\",\n",
    "    #by choosing some threshold:\n",
    "    if Ypred[prediction] <0.4 : #not confident\n",
    "        print(\"Probably not a digit\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hide_input": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "It takes the image of random pixels as input, clearly not showing a digit. It still shows the digit with the maximum value, but it should also show \"Probably not a digit\"."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}