01A_Logistic_Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Logistic Regression in TensorFlow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from tqdm import trange"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MNIST Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST dataset is very popular machine learning dataset, consisting of 70000 grayscale images of handwritten digits, of dimensions 28x28. We'll be using it as our example for this section of the tutorial, with the goal being to predict which the digit is in each image.\n",
    "\n",
    "![mnist](Figures/mnist.png)\n",
    "\n",
    "Since it's such a common (and small) dataset, TensorFlow has commands for downloading and formatting the dataset conveniently baked in already:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-2-8bf8ae5a5303>:2: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "WARNING:tensorflow:From C:\\Users\\kevin_000\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please write your own downloading logic.\n",
      "WARNING:tensorflow:From C:\\Users\\kevin_000\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "WARNING:tensorflow:From C:\\Users\\kevin_000\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:From C:\\Users\\kevin_000\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.one_hot on tensors.\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:From C:\\Users\\kevin_000\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at how the data is organized:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training image data: (55000, 784)\n",
      "Validation image data: (5000, 784)\n",
      "Testing image data: (10000, 784)\n",
      "28 x 28 = 784\n",
      "\n",
      "Test Labels: (10000, 10)\n",
      "Label distribution:[(0, 980), (1, 1135), (2, 1032), (3, 1010), (4, 982), (5, 892), (6, 958), (7, 1028), (8, 974), (9, 1009)]\n",
      "\n",
      "Train image 1 is labelled one-hot as [0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x22fd28bb160>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADhNJREFUeJzt3V2MVPUZx/HfU9Eb9EJZBKKwWGOw1Qslq2kiEo0BoTEBLjS+xNC0ssZoUrQXxZeoCYKmKRa4QddIxER8CbCVGKwa0yBNGsKbUWRBjaFAISyIiRovjO7Tiz00K+75n2HmzJxZnu8nMTszz5yZp9P9cWb2mXP+5u4CEM8vqm4AQDUIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoEa18snMjK8TAk3m7lbL/Rra85vZLDPbZ2afm9miRh4LQGtZvd/tN7OzJH0qaYakQ5K2SbrD3fcktmHPDzRZK/b810r63N2/cPfvJb0maU4DjweghRoJ/0WSDg65fii77SfMrNvMtpvZ9gaeC0DJGvmD33BvLX72tt7deyT1SLztB9pJI3v+Q5ImDrl+saTDjbUDoFUaCf82SZeZ2SVmdo6k2yVtLKctAM1W99t+d//BzB6Q9I6ksyStdvdPSusMQFPVPeqr68n4zA80XUu+5ANg5CL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gqLqX6JYkM9sv6RtJP0r6wd27ymgKrdPZ2Zms33PPPcn6o48+mqynVoE2Sy8m29fXl6w/9thjyXpvb2+yHl1D4c/c6O7HS3gcAC3E234gqEbD75LeNbMdZtZdRkMAWqPRt/3XufthM7tQ0ntmttfdPxh6h+wfBf5hANpMQ3t+dz+c/eyX1Cvp2mHu0+PuXfwxEGgvdYffzEab2XknL0uaKWl3WY0BaK5G3vaPk9SbjWtGSVrr7v8opSsATWepOWzpT2bWuicLZOzYsbm1hx9+OLntXXfdlayPGTMmWS+a1Tcy5y/63Tx48GCyfs011+TWjh8/c6fT7p5+YTOM+oCgCD8QFOEHgiL8QFCEHwiK8ANBMeobAYoOm128eHFurej/32aP244dO5asp3R0dCTrkydPTtb37NmTW7viiivqaWlEYNQHIInwA0ERfiAowg8ERfiBoAg/EBThB4Jizj8CbNu2LVmfOnVqbq3ROX9qVi5JN954Y7LeyKGz06ZNS9Y3b96crKf+t48aVcaJq9sTc34ASYQfCIrwA0ERfiAowg8ERfiBoAg/EBRz/jZw+eWXJ+tFc/4vv/wyt1Z0PH3RHP7BBx9M1hcuXJisL126NLd24MCB5LZFin53BwYGcmv33Xdfctuenp66emoHzPkBJBF+ICjCDwRF+IGgCD8QFOEHgiL8QFCFc34zWy3pFkn97n5ldtsFkl6XNFnSfkm3uftXhU/GnL8uRd8DSM3qG12Kuru7O1lftWpVsp5aJnvnzp3JbefNm5esr1u3LllP/W6PHz8+ue1IXsK7zDn/S5JmnXLbIknvu/tlkt7PrgMYQQrD7+4fSDpxys1zJK3JLq+RNLfkvgA0Wb2f+ce5+xFJyn5eWF5LAFqh6ScyM7NuSekPjgBart49/1EzmyBJ2c/+vDu6e4+7d7l7V53PBaAJ6g3/Rknzs8vzJb1ZTjsAWqUw/Gb2qqR/S5piZofM7A+SnpE0w8w+kzQjuw5gBCn8zO/ud+SUbiq5F+TYu3dvZc9ddD6Affv2Jeupcw0UnStg0aL0BLlozYFmfv/hTMA3/ICgCD8QFOEHgiL8QFCEHwiK8ANBnbnrFAcyffr03FrR4cBFo7y+vr5kfcqUKcn61q1bc2tjx45Nblt0uHlR77Nnz07Wo2PPDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBMec/A9x55525tQULFiS3LTostoZTuyfrqVl+I4fkStLKlSuT9aJTg0fHnh8IivADQRF+ICjCDwRF+IGgCD8QFOEHgmLOf4YrmtNXuf2WLVuS2z700EPJOnP8xrDnB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCuf8ZrZa0i2S+t39yuy2JyUtkHTyxOmPuPumZjWJtLVr1+bWOjs7k9t2dHQk60Xn/R89enSynvL4448n68zxm6uWPf9LkmYNc/vf3P2q7D+CD4wwheF39w8knWhBLwBaqJHP/A+Y2UdmttrMzi+tIwAtUW/4V0m6VNJVko5IWpZ3RzPrNrPtZra9zucC0AR1hd/dj7r7j+4+IOkFSdcm7tvj7l3u3lVvkwDKV1f4zWzCkKvzJO0upx0ArVLLqO9VSTdI6jCzQ5KekHSDmV0lySXtl3RvE3sE0ATW6PHap/VkZq17MpSiaM7/1FNPJetz587Nre3atSu57ezZs5P1ovP6R+Xu6QURMnzDDwiK8ANBEX4gKMIPBEX4gaAIPxAUo74apZaaPnbsWG4turfffju3dvPNNye3LTp19/Lly+vq6UzHqA9AEuEHgiL8QFCEHwiK8ANBEX4gKMIPBMUS3Znp06cn68uW5Z6pTHv37k1ue/fdd9fV05lgyZIlubWZM2cmt50yZUrZ7WAI9vxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EFSYOX/qeHxJeu6555L1/v7+3FrkOX7REt3PP/98bs2spsPO0STs+YGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gqMI5v5lNlPSypPGSBiT1uPsKM7tA0uuSJkvaL+k2d/+qea02Zt68ecl60bHjmzdvLrOdEaNoie7169cn66nXtWjNiKLzJKAxtez5f5D0J3f/laTfSLrfzH4taZGk9939MknvZ9cBjBCF4Xf3I+6+M7v8jaQ+SRdJmiNpTXa3NZLmNqtJAOU7rc/8ZjZZ0tWStkoa5+5HpMF/ICRdWHZzAJqn5u/2m9m5ktZLWujuX9f6vWwz65bUXV97AJqlpj2/mZ2tweC/4u4bspuPmtmErD5B0rBHvrh7j7t3uXtXGQ0DKEdh+G1wF/+ipD53f3ZIaaOk+dnl+ZLeLL89AM1SuES3mU2TtEXSxxoc9UnSIxr83P+GpEmSDki61d1PFDxWZUt0F42s+vr6kvU9e/bk1p5++umGHnvHjh3JepHOzs7c2vXXX5/ctmgEOndu+u+4RR//Ur9fK1asSG5btEQ3hlfrEt2Fn/nd/V+S8h7sptNpCkD74Bt+QFCEHwiK8ANBEX4gKMIPBEX4gaAK5/ylPlmFc/4i69atS9ZT8+5GZt2StGvXrmS9yKRJk3JrY8aMSW7baO9F26eW6F65cmVy2+PHjyfrGF6tc372/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFHP+TNES3ps2bcqtdXWlT1I0MDCQrDdz1l607XfffZesF50+e+nSpcl6b29vso7yMecHkET4gaAIPxAU4QeCIvxAUIQfCIrwA0Ex569RR0dHbm3x4sUNPXZ3d3o1sw0bNiTrjRz3XnTufJbJHnmY8wNIIvxAUIQfCIrwA0ERfiAowg8ERfiBoArn/GY2UdLLksZLGpDU4+4rzOxJSQskHcvu+oi75x/0rpE95wdGilrn/LWEf4KkCe6+08zOk7RD0lxJt0n61t3/WmtThB9ovlrDP6qGBzoi6Uh2+Rsz65N0UWPtAajaaX3mN7PJkq6WtDW76QEz+8jMVpvZ+TnbdJvZdjPb3lCnAEpV83f7zexcSZslLXH3DWY2TtJxSS5psQY/Gvy+4DF42w80WWmf+SXJzM6W9Jakd9z92WHqkyW95e5XFjwO4QearLQDe2zw1LAvSuobGvzsD4EnzZO0+3SbBFCdWv7aP03SFkkfa3DUJ0mPSLpD0lUafNu/X9K92R8HU4/Fnh9oslLf9peF8APNx/H8AJIIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRWewLNkxyX9Z8j1juy2dtSuvbVrXxK91avM3jprvWNLj+f/2ZObbXf3rsoaSGjX3tq1L4ne6lVVb7ztB4Ii/EBQVYe/p+LnT2nX3tq1L4ne6lVJb5V+5gdQnar3/AAqUkn4zWyWme0zs8/NbFEVPeQxs/1m9rGZfVj1EmPZMmj9ZrZ7yG0XmNl7ZvZZ9nPYZdIq6u1JM/tv9tp9aGa/rai3iWb2TzPrM7NPzOyP2e2VvnaJvip53Vr+tt/MzpL0qaQZkg5J2ibpDnff09JGcpjZfkld7l75TNjMpkv6VtLLJ1dDMrO/SDrh7s9k/3Ce7+5/bpPentRprtzcpN7yVpb+nSp87cpc8boMVez5r5X0ubt/4e7fS3pN0pwK+mh77v6BpBOn3DxH0prs8hoN/vK0XE5vbcHdj7j7zuzyN5JOrixd6WuX6KsSVYT/IkkHh1w/pPZa8tslvWtmO8ysu+pmhjHu5MpI2c8LK+7nVIUrN7fSKStLt81rV8+K12WrIvzDrSbSTiOH69x9qqTZku7P3t6iNqskXarBZdyOSFpWZTPZytLrJS1096+r7GWoYfqq5HWrIvyHJE0ccv1iSYcr6GNY7n44+9kvqVeDH1PaydGTi6RmP/sr7uf/3P2ou//o7gOSXlCFr122svR6Sa+4+4bs5spfu+H6qup1qyL82yRdZmaXmNk5km6XtLGCPn7GzEZnf4iRmY2WNFPtt/rwRknzs8vzJb1ZYS8/0S4rN+etLK2KX7t2W/G6ki/5ZKOM5ZLOkrTa3Ze0vIlhmNkvNbi3lwaPeFxbZW9m9qqkGzR41NdRSU9I+rukNyRNknRA0q3u3vI/vOX0doNOc+XmJvWWt7L0VlX42pW54nUp/fANPyAmvuEHBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiCo/wGTnJDl40xJsQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Dataset statistics\n",
    "print('Training image data: {0}'.format(mnist.train.images.shape))\n",
    "print('Validation image data: {0}'.format(mnist.validation.images.shape))\n",
    "print('Testing image data: {0}'.format(mnist.test.images.shape))\n",
    "print('28 x 28 = {0}'.format(28*28))\n",
    "\n",
    "print('\\nTest Labels: {0}'.format(mnist.test.labels.shape))\n",
    "# print(mnist.test.labels[0])\n",
    "\n",
    "labels = np.arange(10)\n",
    "num_labels = np.sum(mnist.test.labels, axis=0,dtype=np.int)\n",
    "print('Label distribution:{0}'.format(list(zip(labels,num_labels))))\n",
    "\n",
    "# Example image\n",
    "print('\\nTrain image 1 is labelled one-hot as {0}'.format(mnist.train.labels[1,:]))\n",
    "image = np.reshape(mnist.train.images[1,:], [28,28])\n",
    "plt.imshow(image,cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic Regression Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the graph input: this is where we feed in our training images into the model. Since MNIST digits are pretty small and the model we're using is very simple, we'll feed them in as flat vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define input placeholder\n",
    "x = tf.placeholder(tf.float32, [None, 784])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get our predicted probabilities of each digit, let's first start with the probability of a digit being a 3 like the image above. For our simple model, we start by applying a linear transformation. That is, we multiply each value of the input vector by a weight, sum them all together, and then add a bias. In equation form:\n",
    "\n",
    "\\begin{align}\n",
    "y_3 = \\sum_i w_{i,3} x_i + b_3\n",
    "\\end{align}\n",
    "\n",
    "The magnitude of this result $y_3$, we'll take as being correlated to our belief in how likely we think the input digit was a 3. The higher the value of $y_3$, the more likely we think the input image $x$ was a 3 (ie, we'd hope we'd get a relatively large value for $y_3$ for the above image). Remember though, our original goal was to identify all 10 digits, so we also have:\n",
    "\n",
    "\\begin{align*}\n",
    "y_0 =& \\sum_i w_{i,0} x_i + b_0 \\\\\n",
    "&\\vdots \\\\\n",
    "y_9 =& \\sum_i w_{i,9} x_i + b_9\n",
    "\\end{align*}\n",
    "\n",
    "We can express this in matrix form as:\n",
    "\n",
    "\\begin{align}\n",
    "y = W x + b \n",
    "\\end{align}\n",
    "\n",
    "To put this into our graph in TensorFlow, we need to define some Variables to hold the weights and biases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define linear transformation\n",
    "W = tf.Variable(tf.zeros([784,10]))\n",
    "b = tf.Variable(tf.zeros([10]))\n",
    "\n",
    "y = tf.matmul(x, W) + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can interpret these values (aka logits) $y$ as probabilities if we normalize them to be positive and add up to 1. In logistic regression, we do this with a softmax:\n",
    "\n",
    "\\begin{align}\n",
    "p(y_i) = \\text{softmax}(y_i) = \\frac{\\text{exp}(y_i)}{\\sum_j\\text{exp}(y_j)}\n",
    "\\end{align}\n",
    "\n",
    "Notice that because the range of the exponential function is always non-negative, and since we're normalizing by the sum, the softmax achieves the desired property of producing values between 0 and 1 that sum to 1. If we look at the case with only 2 classes, we see that the softmax is the multi-class extension of the binary sigmoid function: \n",
    "\n",
    "![sigmoid](Figures/Logistic-curve.png)\n",
    "\n",
    "Computing a softmax in TensorFlow is pretty easy, sort of*:\n",
    "\n",
    "*&#42;More on this later*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Softmax to probabilities\n",
    "py = tf.nn.softmax(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That defines our forward pass of our model! We now have a graph that performs a forward pass: given an input image, the graph returns the probabilities the model thinks the input is each of the 10 classes. Are we done?\n",
    "\n",
    "Not quite. We don't know the values of $W$ and $b$ yet. We're going to learn those by defining a loss and using gradient descent to do backpropagation. Essentially, we'll be taking the derivative with respect to each of the elements in $W$ and $b$ and wiggling them in a direction that reduces our loss.\n",
    "\n",
    "The loss we commonly use in classification is cross-entropy. Cross-entropy is a concept from information theory:\n",
    "\n",
    "\\begin{align}\n",
    "H_{y'}(y)=-\\sum_i y'_i \\text{log}(y_i)\n",
    "\\end{align}\n",
    "\n",
    "Cross-entropy not only captures how *correct* (max probability corresponds to the right answer) the model's answers are, it also accounts for how *confident* (high confidence in correct answers) they are. This encourages the model to produce very high probabilities for correct answers while driving down the probabilities for the wrong answers, instead of merely be satisfied with it being the argmax. \n",
    "\n",
    "In supervised models, we need labels to learn, so we create a placeholder for the labels in our training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define labels placeholder\n",
    "y_ = tf.placeholder(tf.float32, [None, 10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cross-entropy loss is pretty easy to implement:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loss\n",
    "cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(py), reduction_indices=[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the old days, we would have to go through and derive all the gradients ourselves, then code them into our program. Nowadays, we have libraries to compute all the gradients automatically. Not only that, but TensorFlow comes with a whole suite of optimizers implementing various optimization algorithms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optimizer\n",
    "train_step = tf.train.GradientDescentOptimizer(0.05).minimize(cross_entropy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train, we simply call the optimizer op we defined above. First though, we need to start a session and initialize our variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a session object and initialize all graph variables\n",
    "sess = tf.Session()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are much cleverer ways to design a training regimen that stop training once the model is converged and before it starts overfitting, but for this demo, we'll keep it simple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████| 1000/1000 [00:01<00:00, 818.38it/s]\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "# trange is a tqdm function. It's the same as range, but adds a pretty progress bar\n",
    "for _ in trange(1000):\n",
    "    batch_xs, batch_ys = mnist.train.next_batch(100)\n",
    "    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice, because of the way the dependency links are connected in our graph, running the optimizer requires an input to both the training image placeholder `x` and the training label placeholder `y_` (as it should). The values of all variables (`W` and `b`) are updated in place automatically by the optimizer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see how we did! For every image in our test set, we run the data through the model, and take the digit in which we have the highest confidence as our answer. We then compute an accuracy by seeing how many we got correct:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.9003000259399414\n"
     ]
    }
   ],
   "source": [
    "# Test trained model\n",
    "correct_prediction = tf.equal(tf.argmax(py, 1), tf.argmax(y_, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "_acc = sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels})\n",
    "\n",
    "print(\"Test accuracy: {0}\".format(_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not bad for a simple model and a few lines of code.  Before we close the session, there's one more interesting thing we can do. Normally, it can be difficult to inspect exactly what the filters in a model are doing, but since this model is so simple, and the weights transform the data directly to their logits, we can actually visualize what the model's learning by simply plotting the weights. The results look pretty reasonable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABIEAAACBCAYAAABXearSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztnV3MHVeVptfO/49DHMdO7CR2EueH/ImkA5oZNKNRSwiBWgiuRuq+QLlAcDODGmkQnZ7hkosehFpcBzWkL1qMRuqWQKilFkLdGgYhlPAzQ0IwduLEduLESUzATgiQUHORz9VPvfZernO+851TVed9JEQdn/rq7Nprr1X7nKx3rdI0TRhjjDHGGGOMMcaYaXPBqgdgjDHGGGOMMcYYY7Ye/whkjDHGGGOMMcYYswb4RyBjjDHGGGOMMcaYNcA/AhljjDHGGGOMMcasAf4RyBhjjDHGGGOMMWYN8I9AxhhjjDHGGGOMMWuAfwQyxhhjjDHGGGOMWQM29SNQKeXDpZQDpZRDpZSHFzUos1xsx/FjG04D23H82IbTwHYcP7bhNLAdx49tOA1sx2lRmqaZ7w9LuTAifhERH4yIYxHxWET8WdM0P1vc8MxWYzuOH9twGtiO48c2nAa24/ixDaeB7Th+bMNpYDtOj4s28bf/JiIONU3zTEREKeV/RsTHIqK6GK666qpm165dm/hIMw8vv/xynDp1qlTensmOl156abNt27atGahJOXny5CtN05zLgWb2xW3btjU7duzYmoGaKidPnozTp08vxBdtw9Vx9OjRhfnilVdeaTuugJMnT8brr7++EF+8/PLLm6uvvnprBmpSXnrppYX54hVXXNFs3759awZqqrz22mvxxhtvLMQXHU9Xx7Fjxxbmi5dddpm/a6yA06dPx5tvvrkQX7QNV8err75a88UOm/kR6MaIOIrXxyLi32Z/sGvXrvjCF76wiY808/D5z38+e3smO27bti0+9KEPLWhkZha+/vWvP1d5a2Zf3LFjR3zuc59b1NBMT774xS9mb89kR9twdXz6059eqC9+5jOfWdTQTE++/OUvZ2/PZMerr746Pv7xjy9oZGYWvvSlLy3MF7dv3x6f/OQnFzW0LaeUf/2uNm9W/xD4yle+kr0983PR8XQ1fPazn12YL27bti0+8pGPLGpopiff+ta3srdn/r740Y9+dEEjM7Pwta99reaLHTZTE+hcvxSe9RQqpXyqlPJ4KeXxU6dObeLjzBZxXjvShm+++eaShmVmYGZfPH369BKGZWZkJl+0DQfJzL74+uuvL2FYZkZm8sU33nhjScMyMzCzL9qOg2QmX3Q8HSQz+6K/awwSf1+cGJvJBDoWEXvx+qaIeEFPaprmkYh4JCJi//79S/tPFX/4wx96nXfBBf1+B+t7vUXRd1wL4Lx2pA2vvfba8f7npukysy/u27dvknZUv1m2326SmXxxqjYcOTP74t69ewdhxyyTgO8x+2AW5v27FTGTL+7evXsQNjQdZvbFG264YaF2nGXN89zs7/pm/ywiM6jvNbY4C2kmXxxKPDUdZvbFnTt32o7DYyZftA2Hz2Z+aXgsIu4opdxaSrkkIv40Ir65mGGZJWI7jh/bcBrYjuPHNpwGtuP4sQ2nge04fmzDaWA7Toy5M4GapnmrlPJfIuKfIuLCiPhq0zRPLmxkZinYjuPHNpwGtuP4sQ2nge04fmzDaWA7jh/bcBrYjtNjM3KwaJrmHyPiHxc0FrMibMfxYxtOA9tx/NiG08B2HD+24TSwHcePbTgNbMdpsakfgVbNIup9vP32272ur5rnTANd03arzjt7b9E1jaZGXw36vHUoFqFxH1kNjEmg/kw/yvyZfqR2y95bV+bxj3l9qm8NjL62WTcbcs6y5xj946233qqeR7K51GdTzY/0PNtxfjL/qO0p9G/mmdd59yHrZkPOk87ZRRf963b8kksuOee/63sXX3xxe5zNpdqYz0m+9/vf/75z3u9+97v2WGMCX/NY19mYu5bNylY+F9ekFtsgyJ6ZpLa/nJcsjmb7UNs4p689F1EXse97Q7HZev6CYIwxxhhjjDHGGLNm+EcgY4wxxhhjjDHGmDVgFHKwedLssvTX7Hq19D6VmfRNEcxSyJjKqym/F1544TmPZ5GNjV0qlqUVZ7YhfVMo9bP62pDXp5303CGmAS4SXWucC13bl19+eXvM1PZLL720cx7f4/U1LZ0p67/97W87750+fbo9PnXqVPU82ji7Fx7PK2MZI1ks4ZyoDdX25/qbiK4Pq31r76l0gTbUmDCPRHes9JV8RXTnMLMBz8ueOZw/tTHXAt/jc1DfWycfy8hsSPruWfhe9kxTarFQx9RXQrsIeecQqElWs/ii/kE/4LOPx3reFVdccc7jiG4sVh+77LLLzjkOPksjIn75y1+2x6+99lrnvV/96lft8RtvvFG9hsYSMkapWF/JSCY/r/3NvP7QV147r4xlimTzXpM3Zs/F2nFEXX4Z0Z13+qnGB/osjyO6z9axf++bl777nuy7XnZe5ps1ma/6VGanVX1HXM/VYowxxhhjjDHGGLNm+EcgY4wxxhhjjDHGmDXAPwIZY4wxxhhjjDHGrAGjqAlUI9P2qR6+Vu8g0+xRd6m6bOqtVQNd04ZmGv1M59+3XskYUZ0l50F1tbU50rojV111VXvM+jNKpuGtzblejxpetS918rx+ppEfky671tI2ous773rXuzrv7dy5sz2+5ppr2uMdO3Z0ztu2bVt7nNV74TzzOCLi+PHj7fGxY8fa4xdffLFz3q9//ev2WO1Tu0+951r9myHD9ZvVjdB6E3xNn9BrcN2zDhPnOyLi9ddfb49/85vfdN6jTRkDtK4T7aZxpVZ/TbX1Wf21KaAxlL7E+XzzzTc759VqMWW1YLQ+FF/zWP2Ir/X5xutnMaFvPY5V07eGCO9Pn1W1da9xjPblNXT+GXezOjNZTSD6jl6jtmfRNcd9VVb3bwjU1tgstSRr65l17RTGMvU3zvu1117bea8W97M6RVntrr6xcsi+WCOrNaL2pc9lNda4tueptxZRf6apnfie7lH61jIZ67Mw+45Fm/S1T1bPh3sJ7nEV9WfuhTJfZ3zM9mo81piQ2XgMvjlLvcPankW/r3Ne+V62B1JqtU11f5nVaav58FbXRXQmkDHGGGOMMcYYY8wa4B+BjDHGGGOMMcYYY9aAQeoXspRIplhqKiZTuTTli+cyjVXlPZQS7dq1qz1WScuVV15ZHSM/m200eRzRTQPMUrz5nqZzZq0hs3kcCjrGrN13TcaXSb6YVqefxXQ/ylEiuutl+/bt7XEmi8nka1k6e9ZSdshwvWnaI9PPd+/e3Xmv5lc6t7WWiZoSSvkC28BHdOedNmbr24g8TbbWLjRrrTykNp06FvoE546xLyLi6quvPudxRNdWjEmaMss5V6lebUwqSWAc4HHW1lVTeWk3rlW1Ya118hDp2/a+b9vhvq1ws7ToLCbQ17nWNGWdqdX6vKNNMskMJYVDTnOvzbnOMe9Vn2M1SZCug5ocLHs26TX4rGUMUBvSvhrXuS44jldffbVz3ssvv9wea+zI1uCqyWzFtahruyZtUHlsTRqosj7Ou87tSy+91B7TpuorfJ6qDWpyvTHsOzdD3zhJu2l84nORc6zfW/hZul5ob9pQ98M1n43o+m0m78/2R0OjZh/dnzMe6rzXykGwfEFExE033dQec1973XXXdc5jPNT54/fAw4cPt8cvvPBC5zzuWXV/w2vyPnUPQ7sO+blI+vqb2pD+V3v2RdRLEWjJAvqp+jPXCPc5/O4Y0d1H636bvlkrtxDRteEivmcM55uKMcYYY4wxxhhjjNky/COQMcYYY4wxxhhjzBowGDlY3+5YWWpYJgfjNSg90JR1vq5V39fXmmLJ9yi30BRLXl/T+5iWxjS0LA1Qrz/UtM2s0wnT9vp2CVIb8vq8hqa9M91PU/9oQ6ZUato7x6G2Ycog7yvrqDN0ONbMB5gSqeuSfvvKK6+0x2rvmgxL4XkqIaQNTp482R7ruiPqN/xsxhWt7p9V9F8lmhJMKSvTVbWDDNNVGcciuveadeWi7emnmlZNe6jsgGuL9tSUXF5DZYG0Pf1UY8cyuzIskkwWTLLuGkRjGeMjpSUqM+Fa0A4pN998c3t8/fXXV8+jDTQVmvG3Jr0YMlk3oUy6QDTu0J+Zbp7FXc6Xfhb9SvdYfM34r3J5vlYpKX2f1+CzQGGXx4jVdwvr2zEqk61lkj/6n/oi7cX3ND4xbjL+RdSl6llJgaxzWK07jp43pJiafXZfabeWheA6yJ5HXOu1TlR6vawzVbbmsvcymT3JJCirfi5mHYazvSHfyzq90o9U5kX703Za+oM+oeOtfa/RuMkx6h6ptl71e/BY4JrNOnZRyqUlPfh3nBP1Wcr4brzxxvaYe5SIrg11/tlt+MSJE+0xJc0RudyM1+TeO/vuk8k2+zKcbyrGGGOMMcYYY4wxZsvwj0DGGGOMMcYYY4wxa4B/BDLGGGOMMcYYY4xZA1ZWE2iWNpI8N6sJRB2h6i5Z+6HWFjGiq6nLag5ktXioIaU2XvWZ1Deqxr2m61TN8Kq18X3pW9cpm3NqJjl3asO+rdn71unh/KvutNZeMLt+1rZxaHprhWOn1l/rAGStf9mettZmVq9Rq7kQkbf65JrJ2lxz3rXmBtcXtf3qizoHQ0HXG+MTa/3oebW6ZBFd/8vqi9BuWU0g6rSzGka8vtahIlpfhOPNap6MpW1qRhbzshpsjGWvvfZa5zxq29m69plnnumcx7ll+9yIrl0z7TrHr75YG3tW70Fj6JBiKu2h90DoO9palrUjWNdLn4usU0E7aX0D1o/R+a/FSfVZvtYaDLwGx6v+zDXIOgtDhGu2b00gpVbrR+tg8DX/Ru1NG/OZG9G1Me2v1+DzQdcdX2ctsPvW11kGHFu2ZulvWjeOdcp0bXMvxz2f7hvpi7SF1rmrfW5Ed7/BedX4zz2R7lFr3610P5zVjVo1WU0gHuvapu2yNtz6HuFzkXskrffCz9K5ZYzYsWNH9bNqcT6ivq7ZVl5f6zhW+VzUsdRqROo+lK811tLenLt9+/Z1zrvrrrva4/vvv789fuCBBzrnsXaQ1vr5wQ9+0B7/6Ec/ao9//vOfd85j7SCtG8XnP8eu6y9bj/MwLG82xhhjjDHGGGOMMVuCfwQyxhhjjDHGGGOMWQMG0yKeaHpfLb02S2dTaQbT+3isKYJ8j9fL0iM1/ZLpa1l7cUrFspajPNbUYI4rayG5amr3M8u80vZMcdU0QKbV0Z4qQeH8qPyhNnZNr8ykSDVZg6YXc43oehwanAvOn8pMOE+agsx54lzo+q2lvatEgfOXySp5/VlkdzVpia67WSSuyyRLz+e86tqmT2iLYZ7LVPcsnZ3ptHfeeWfnPaboZnGdMTOTquh65L1wjH1bpg+Bvs8I9QG+p5KjWrzVOFSTR+jaymRA27dvb48z+UjWWpnPVt6LSpoySdmyn4vZ844xhOdlkmFKdCK681yLd/pZfaWe+kyrrTP1N45fx8trck+kz0X6+tAlKITzrL7IecmuwffUF7meM0keJSmUcOpr2l8leWyTrOOt7W2HJLFVX+dcahkHrr9MKsQ4pvPFOcniDGMX/S0rLZHJQGhrlarwte5z+ezOyiP0bVW/LLLxMFbQpjp/fAZlsYf3p7GSMAYeOXKk8x7to/GwtudXOXUmp+d4+SzUvbfGiFWS7Vk4l7WyEBFdW1PyFdGdr+uuu6493rlzZ+c8+hz94cknn6x+lpYbePrpp9vj7Lskx5SVIOGa2Gr/GtaT1RhjjDHGGGOMMcZsCf4RyBhjjDHGGGOMMWYN8I9AxhhjjDHGGGOMMWvAsAuQbFDTf6pOnLpk1T7WarJonR7qRnmeaiupCVQNIDWf1G5qbYKsxTHhGLXmBsc1pBpAGVntlKw+Dl9nNQJ4HvXbd999d+c82vro0aOd96iZZ4tzbf2oNZpIVnuKDK3eQV+yWkl8rfVkuE6pvdf6E1zrXDPUP0d0a82oT/GzeJy1Y+5b02DIraeJ+lut/bDGlmPHjrXH1DxHdFsOMxZq20v633vf+972+I477uicx5ip88jx025924zrNRnXs7pCQyCrJ1Nbl1qfhejc8v7pV1o/ga1reQ3GxohunHvwwQc77917773tMW2gMZS20xoMnAPGC71nxt6szfyq4Xrjsy+rX6H1IBhDOa9ZLKQfaSzknkJrLXHO+Z6el7Vm5mvubbRWBsel/jw0ar6o8YRrMYtzWqOE0E+5ztWPWAsmq+nGuc0+V/2Idsz2ZkOqJ8Oxaf2d2t5f77vWvjqiXhNIv4/Uapno3oa+rjVPuE9h62l9TtCfNU7W6mxmNROHus85Q61WFfciEd0269xDRnTnmvOn3/X4PYH+p+c9//zz7bHGuT179rTHtIfGTT4DtJU8127fWJnFn2WQ1QSq1SdUH6B/qO/w+cc6QDr/rAP0ve9975z/HlGvqRdRX3MaT7kG1cf4OotTi96jnvdqpZSvllJOlFKewL/tKKV8u5RycOP/r8muYVaP7TgJbrENx499cRLYFyeAfXES2BcngH1xEtgXJ4B9cX3o85PSoxHxYfm3hyPiO03T3BER39l4bYbNo2E7jp1XwjacAo+G7Th27IvT4NGwHceOfXEaPBq249ixL06DR8N2XAvOKwdrmuZ/l1JukX/+WET88cbx30bEv0TEX/T5wD4pZ1nKaK2NWkSe9s10rVoKvL7HVPQTJ050zmP6paZT7969uz2+9dZb22NtTccUvkyWxnGo7CZL/SeLtuNmqKW96Wu1b00qpvIdwvTBe+65p/Meba+pf88++2x7zPnP0t419a/WVjlL7ztPqu3piDgp/7ZUG3KuKf3R+XvmmWfaY02NraWr6jU475yXvXv3ds675ZZb2mNtq8m55jxrrOBr9alaaqr6bN/2m8vwxSwFn5JGpjAzpkVEHDp0qD3WdpmUitG+Oq9Mf73tttvaY20RzxaeWRttovIHrheVPzB9l3ab14YxAF+s2Thr5ayxh/Gx9oyM6Pom51alSUxtf+CBBzrv7d+/vz3ObMUYq7anL3Kd6HiH5ItE47va4wwq5eKzirKAiK6PMdVd0955HvcXOibOscqfKWvIWltzr6NSUvo61wTXTkRXnnoeicPKfZFkexjuA7JnPe2veyS+x3nnniWi62PqR7X22Cq3oE24r9VzuZ70nvvKF7bCF7M5zuTcjKG6v+drndfavWbnZftQ7kU0xtXuTf8927PUWlHrNcb0XOQ80T80pvLZRWlYRNc/eO8qA+I1GPNUrsU4mpUxyaSBHL/G9r4yRL7WfZas/6XuURXakHJnffZxHtSGvFf+ndrwwIED5zzW87iW9LNuvPHG9pjPWfVZPjNVSs99NP0+i6eLkGbOKy67vmma4xERG/9/3XnON8PEdhw/tuE0sB3Hj204DWzH8WMbTgPbcfzYhtPAdpwgW14Fs5TyqVLK46WUx/W/KplxQBtmhZDNsKEdtQiwGQe24TSgHfW//ppxQBtqhqgZD7bj+HE8nQb+rjF+bMNxMW93sJdKKXuapjleStkTESdqJzZN80hEPBIRsX///uZMKtMslch5btZBg2l1Wfow08s01Yo/VDH1+amnnuqcx9RbTRe9+eab22NW3M86lmnleqYFcowce0Q3hWyO1LBedqQNr7322noOX0JN0pfJwc4xjvaY6bUaaJjGx/nS7mD8LE3XrUlmdN0ylVptU+tCN2+6dIW5fHHfvn1z2ZHzxHRz7a5GKZFKGOkTvIZ2luK8M+1TN+pM01RJXu09TSvNfIepmVmqf9YBrgcz+2JfG2aSGsYPdvyK6Eq+NHWVNmTsuv322zvnfeADH2iP3//+97fHjJERXd/R9UK4JtiVTMfPrh0RXb9lWnXWtWWr4mlE14579+6t2jFLmSYcq0pzuO41TZ1dUfiefu7BgwfbY/qD+hGlmSq/5TrhmtSYyrWlz3GOK+skyffmiK8z++Lu3bt7GSqTanDMKl3IOp/yPUoIVF5FuCZUjse9jcZk+hj/TiW0HL923uFrdrDSmMCYr9fv4Rdz+eINN9yw6f0N93W6LnXPSmrlDHRfQT/iPkVl14yV119/fec9rhnaSiUolIDpeuK46GP6Hydou2XsUbN4SjLZMX9I0hhU61C4MY72OOuyx/Nq8qWI7pzrNfgdgd9BdP5rHcAiuveSPfsyqVgP5vLFnTt3zhVTa/d0js9qjzW+cA4Zo3TNsNwHfUx9lq/1+wrjIWOergVeQ8fBtZH9CMpnR9aRq8LMvkgbZjE7k80y3un+he/pXoRrnfOle0Pubbj3VMkdJV9adoKlDjhGXX98nmbfA/W5S2aQZvZi3m+f34yIhzaOH4qIbyxmOGbJ2I7jxzacBrbj+LENp4HtOH5sw2lgO44f23Aa2I4TpE+L+K9HxPcj4t2llGOllE9ExF9FxAdLKQcj4oMbr82AsR0nwa1hG44e++IksC9OAPviJLAvTgD74iSwL04A++L60Kc72J9V3vpA5d/NALEdJ8Hhpmned45/tw1HhH1xEtgXJ4B9cRLYFyeAfXES2BcngH1xfdhUEYtVQC2z1tGhxk7bu9XqIqjOlnUm2Bb5hz/8Yec86gq15gY/mxpr1cZnrTn5Hq+vNU8yfe4ma80sBR1jdj/UdVJXy7oyEV3tLOeV9SoiurUPTpzoyltZA4Ua275t4CO6ml6uzVnuedWojpdrkfpZ1o+J6M6n2of+Qe2r6tWplc+0zKyLcPz48c57XAts66g1N7i2tMYXx0tbqc9mWt0eeustQz+7ViNA5z+7V7Z5Zt2C97znPZ3zWBOINbn0elwjWtOA64xt7I8cOdI5jzVKNK7zucE1PWTfi+jf4phrT7XsbGNLXXtE95nEY9Wks813zaciuuuCLXMjuvEwq9lT8/uIru34d/O2iF81tGGtPlBEXouM79GGWreAPsH9i8a7rL4Ebch1pTU1OEZ9hvD6fE5ovZy+9bCGBm2qLbn5Otuf8Rr0qYju3LJupcY81vPp64tqA17jhhtu6LzHe8nq/jCWrNqm/Hx9LvL5x/O0hkvWPp7PKu5FtM4d/Ypzp/uS++67rz1W2/DveKyxm89TfbZy/NwDaTwd2neJvt8TuHfTPT79hX4U0d0jcX+pMfXBBx9sj1mTRuMm7aPrqVYfV9cW7ap7atYD42drTOA49HO32jezOlP6vKt9h+v7PVnh3lC/q3CeaV+tUbd37972WGMy/47f+XX+aRv1qVqNQ72vTdbnOothebYxxhhjjDHGGGOM2RL8I5AxxhhjjDHGGGPMGrAyOZimQmUt42up7toSLmsVzb9jqqO2RWba+09+8pP2+PDhw53zmN6p90KZGtP0tBUr06lVilGTj2j6Fz9b5zCb01Uybwob0+KY5pmlXlL+oCmHbP2oKYLafvoM2voxkzXwPledBj0vOm6uS9ogkxJpijNfU5qpMhamVXLeNS2aqZOaastzOUZNmea9qJSU644p8Fm7Z/XfM2tjCPKj2lpU/2B81XhKG7I16l133dU5jy3jGXefe+65znlHjx5tj9WPmPrM1N2sbXRf2eyY/DJbb/QPbeXM15rGTP+jv6kfcW3wuaVtbCk7ydqc0y9V1s2/U1+sSTayVPNVw7H0lQLr2s7kBLQB51WlH9x/UKpC6bOep/5BX2d80NjB+KeyBs5BFk8Zd4cm78vWF8eqcjD6bCZhzPa5tA+Ptb37nXfe2R7TZyO6NuGxPlvpm+qnvDeuSd2PUU6z7Hirn9e3vTv9TWMQ31Mf4z6S3xm0LTX3H7w+5XcRXX/jd4mI7vzTV9Tf+kqAuP6GFD9nhXalfVSaU5PFR5w9h2egPSK6z1P6s+5luS/StcB1Qhvrd1P6d/ZdL/OxIcn6suci9281KX/E2c9CQr/S796EUi76H+VfEd29kkpjKYvnWtJnayYRZZzk+tO5WXQMHc6KMMYYY4wxxhhjjDFbhn8EMsYYY4wxxhhjjFkDViYHm0WqxHQoptlp+qum3hKmlzFFUGU/zzzzTHvMivGa1sVUM017r3WMylLWNR2RY2Q65yypYENK/SO8B72f7P64ZrJuS5QkMH1TK+rTvipP4bpgKrV+LsebpdryvaGltmf0rejPVMmIrm+qZIQ+kXXPYxo8P5dpkxHdtEqNAbVK+prGTR/LqvHXOr5F5HY9c81lpcP37RrIe9C54/xrfKJsgPFPP7fWLYVSzIhueryOg3PO+L9v377OeVwXOl76fjY3Q+sclo2B88SuGepHlINpdw3OLZ9pmmbNtcCUaU2bZ5p01pGIkhGVoHDN6HOXa5exOIvLQ7BjjZpsSiU1tI3GJz6fsq6l9LkDBw60x9oZh7ZXyRJjdyb9I5kMg9fXzxoTfZ/vWUc7+hiP9Tw+u7jONR5SCq/XYJfFrPss7arPO16zJp+JyGXxW+2bmRwsO5dzorGQvqnxiRIe+pvKUfg8YgzI1kTWsYt201jI8etejPesz8yxQhsz9mjnWNpHpXaMt9pVmHDdcy712Zd1aON3jYMHD7bHakfufXS/zb0AfVbLV2TfR5dN1uWqVnYi+76o8an2zFR5Lf2K49A9KteVyjbpY/wNgR3GI7rPZ/0ewzFyHBqTF80wfyUwxhhjjDHGGGOMMQvFPwIZY4wxxhhjjDHGrAH+EcgYY4wxxhhjjDFmDVhZTaBZqLXeUx06awuofpZ6Qdaf0DZ8tTpAqmsnWmeBekHWpNEWm9SOaxtK6gVZr2QW3e5QW8RTx6m6V77Wmga8H2qbVZ95xx13tMdcL6rFffrpp9tjXQe1Okw6pxyj3gvPpX57zG2pqU9lzQGtQ0K05ketRbzqnOlXnNtnn322cx7bM6oumFrdmk9F1NvAR3RtxxpG6vecG40Xq2wRrzUqeD+0jbY15T1ofKKGntdnq/eIeqtd1UPTV3ReGUO5RrRdK2Oj1nqjvjuL5Vm9siFp6BWue/WB7O9oB8ZHrUnDeMu2uPoMZitcjcucd/6dflZmg9p5GpezegNDgvfA+dc6JIx/WWvZQ4cOtceo9eChAAAcN0lEQVQ6J/w71i1gW+uIbp0L3UfRNxl39TzGU61DwnHV6q0NnazWDO9d9zCMvXq/fLZwr6ixl9fkefRLvb6uGd0LnUHtyGvoc4Tjom9nrddXvffJYgZf0wfUhvRNjV30Uz5btUYM4x/3Dffcc0/nPNaj0fouvIaOkdBuasO+sXEsNdYiumOlrTSm1ur5RHTjHP0qq6lUq2MT0d2PaH0oxmzun3R9ckxq71oszmrlKau0q46L+0Y+37Sua62VvF6Tvqj3yXXBZ6HuE+mb+h5f/+xnP2uPWXsvortv1u8g3LdxvFkdpEXsUZ0JZIwxxhhjjDHGGLMG+EcgY4wxxhhjjDHGmDVgZXIwbV/O1DdNcaq1QtTWaUyn0jR1pl5l6ZxM46vJJvT62prztttua48pmdHWdFmLeErWmLaoabxDlXwptbRJTWuknfTeaF/Kj1SKRHsw/VVb/h0+fLg6DpVUzEMtbS9L7xtaqq3OA32O0hz1j+x+6QdMZ2dKa0Q3/ZUpoZpqy3FoSijTL+nPeg2iaddMAeZ7eh5tl7XMXQZc95k0lj6WtWbX1FXag+nN2jKXErNaDI7orgNKiiK6bcd5XhbjFb6XxZ9VyxUydGwcO9ezpkzTPprGTDtSMpLNC/1Nn3233nprdbz0RR6rFJfj1eedrpszqMxhqHbU+F6Lk5rGT79SOewTTzzRHjM+q9/zs7kO9LOyFsOUtTDma+zg+tG9Dceh+8AaQ5NmZr6YtRdn/NJnJp+LKokljN+Mr2pH+pVK/jjGm2666ZzXi+jOs/oin3HZ8zmTKi2bLC7UJH36XOE1dP2qv5yBUr+Irm9y/3rvvfd2zqNt1J9r8l21E22jvsj1yfvXeKqvh4TalH5A2+k9UE5+zTXXVN/bv39/e3z33Xd3zuPfUQb04x//uHMe29OrFJ6xk76tMU7tTzgHvE+9Btd1dr1lo9/1GFu4tjXG0d+0nAT3r4ytug64F+F3xCzGq7yWe5YjR460x7ofpv/pPXNcffcvi3gOOhPIGGOMMcYYY4wxZg3wj0DGGGOMMcYYY4wxa4B/BDLGGGOMMcYYY4xZAwbZIr5vnRvVw2V6ylrLWK15Qi02tbqq9WX72/vvv7/z3rvf/e72mBpwHRP1gZn+kBrSWWqN9NXbL4NavQPVzGeaSdqAWlxtRUw9L/Xpzz//fOc81l1SfT7nn1rNWeZ0nppAQ0N1w1zPrA+h9blquuyIrlaXNQhUl81r0h5aA4rrRNcTa19Ql5/V+NK6CFx3Wc0N6oJr7WeXZWvGGrUN75120vN4fzruEydOtMf0o+eee65zHvXuWc0Bau11/rnO+HfarpX3ojVKaF9qwHW9DClmKqqH5+ts7dE+Ou+sH6S1hAh9gDWashpcPI6IePrpp9vjF154oT1+8cUXO+cxZmctbrOW4kOtYZHVO2T803bf/LusHgH3B1mNL/qDxl3ugbS1da1uje5tGPN1j0Wb1uqTRXT3AquuAaToWPvWIeF5un7pY6yPpzGJNmHdH601Qr/S2pfcM7Fupa6F7NnKeMHP1nHU6pWsGvVF3mu2f+G6VNvoM/QMet+07+23394es/6MXo9xPKK7t2EdII3jjAm6bmvfhYb8HFT0nmo1qPQ7HL8n3HfffZ33WJeQtWa0ZhPX0MGDB9vjX/ziF53zGLPV77nf2bt3b3usa4b7No2pfI/XH4sd1RcZa3ic+aK+x2cc50RjYW3fo7GQzz7WeIroxtpsL5Y972q/UWS/ZSyCcawQY4wxxhhjjDHGGLMp/COQMcYYY4wxxhhjzBowCjlYreWmSqOy9PBau3dNd2ZqbJa6ytaNd955Z+c9SmaIpqtx/Dp2pnoyfW0WOdgq28dnkifOg7as5GtNeWQ6HlM01Ya029GjR9tjbRHPlF+VoBDaRsc0T3rzVqf3LZKsBSpTmlV6wL/TlqWcM0oKVF5Vk/6oDZh+rqnAPJefpXZj6qj6L9cGz9O0Uqbm6zjOzMeybM3P0TRoyuJoN/VZ3p+2LOac8DyNcTVZpV6P8gTGVj2Xn6trk6+z9qcck8qNhozah3PN54XKhTjv+kygb1JuoPNHCRjXls4fbcC06IiuVPCVV14559h1jOqnHJfGgaFCu+n8834YW9QHuO71vg8fPtwe0/Y6d4xrTHVnHI/o7o84pohu7OB9Ze2M9T3Gcr6nMXNIrcXPR002lcVUjZW0F2O2+uLNN998zuupXKgmM4nori/Kq1Xyxfigrd/5eVkbco5/SPKUTA5G2+iem+9le2zaU/cUjKf79u1rj7WVPOeV+4uI7vxzHaidMjle7TvTkGR754L3pHGC649rT/dB9COW8IiIuOWWW9pj2kTX9k9/+tP2+Pvf/357rHIw7rOyMiZ79uw559gjujbRe+a90cey8gND+t6hfkQfy77z0we4p4iol/HQuMsYR/ml+mwmned3S64RLTtB1IYcbybNtBzMGGOMMcYYY4wxxsyMfwQyxhhjjDHGGGOMWQNGkU/NVDGmPWqqMtOpNGWKaV5Mk2UqZkQ39YxpspqaRzmSdvJg6h/T5TVNM0vvq0ngNL1vSCl9GbRhlkLLNLhMxkLb6xywcxHTaTWFj2l2NQmfjklT8ftWcV9mtfdF0ne9aVpxXwkd39M0TdqLnYa0mxDTObMUy5r0IqIrj6BvR3TXIdeupp9m8oVldwfj2taOJZRXZRI5xq5szdIvVfrHcXDO2ZkjotshRbsy1KQRKkWibTRtmGuJ743JF3VsvH9KqrQ7D22iEsaa1Fj9o9axS6VnXN9ZFw5eL/ObTPLFtaVrd6h2zCQonHOVONe6MkZ0pT4qeSbcl2SdVLNumFl3K8JrZucRlVqMiZpEUn3g1VdfbY9V+sP9JteFPo9ob/q67lEZ93XNMKZyf6l7JI5ffZ2fzWsMSfKV0bc7WNYJrtYFNKI7/yxlENGVYNI/9LnF9cJ9bUR3/hn/M4mu+mJN9jUWG0bkNuD9qg0YU3UtcK2zE9Szzz7bOY8dwR577LH2mHaL6PqwSoQ4Ro5JSyxk36F4zUx+TFb9XTLrmFy710yuqrGWewx+t9D75HOXMk39vs7Xus9l3Mw62NKvMgl19v3JcjBjjDHGGGOMMcYYMzPn/RGolLK3lPLPpZSnSilPllL+fOPfd5RSvl1KObjx/9ec71pmNbz11lthG06Ci23HcWNfnAz2xZFjX5wM9sWR8/bbb9sXp4F9ceTYF9eLPplAb0XEf22a5u6I+HcR8Z9LKfdExMMR8Z2mae6IiO9svDbDxTacBrbjiNlI5bQNp4HtOGLsi5PCdhw/tuE0sB1HjJ+L68V5awI1TXM8Io5vHJ8qpTwVETdGxMci4o83TvvbiPiXiPiLRQxKNW/UC2a6PKJaSGrxWBNIW7Hy77R1KqGmOqulQZ2ijimrd1BrYzxL7YMz93zRRRdF0zQ/ilieDRWOk/ettSd4ntYEqtUGUR07a7XwetncacvqWr0mtVnWyi/Tf87J71dtx5o+V3XO1MOrtpnrmbZSDS418DyP9YH0erqeqAOnjbXOAuszMD4oWfzh61rr8QsvvHDpvqjrkrbhcba2tX4FY+ORI0faY/VFrhfq3bUmEOOwriXqufksUH046yJoO12uH15P62iUpHaXsHJf5Pg4L1pbpVZbK6I7Z1ntCNaq4NyqD2Qt3Dlerju1Qe16+jprPd0n3q7CF7X2AW1FW2htkKwGIVsdM2aqbWh7ttbV8zhGjd2Mr1wjWV0OtQVtxXvWa8zwzByUL/JYn2msZ6drgeuZsfLWW2/tnMe1cP/997fHuma4tjQmsGbJyZMn2+Pnn3++cx7jptYa476I+yXdZ2W1OnnOqn0xW7O187LvGbXnbETX1lwj9MuIrm205gl9R9dZ7bM0TtbstomaQEv3xSxu8J7UBuTw4cOd1wcOHGiPOc9aE4j2YRzNaqnpvoUxlse6D+V96trlfWocIH2+k1xwwQUr/75IOE61NedSn2P0JZ6X1Sjl9XWvxHlVG9bqE2Z7at3b1N4bVE2gUsotEfFHEfGDiLh+4weiMz8UXVf5m0+VUh4vpTyuXxDM8tmsDXXxm9WwWTtmP6Ca5WAbToPN2lE392b5bNaG/KJgVoftOH4cT6eBv2uMH9tw+vT+EaiUsi0i/j4iPtM0za/Pd/4ZmqZ5pGma9zVN8z6tzm6WyyJsmP2abpbDIuyYZbuYrcc2nAaLsKNmPJrlsggbagaiWT624/hxPJ0G/q4xfmzD9aBXi/hSysXxzmL4u6Zp/mHjn18qpexpmuZ4KWVPRJyoX2E2NJWuJgHQX/yzNCmmLvPL0549e6qfTfmIptNyTJp+x/9Cz7/TVrh9Uy6zdq59r7dsGypMkeP86/3wtW7IeH+cV23VzbRlfi5bsOr1tUU855yptpqGmbWPr6UIziNdwLlLtaOmX3Jt1yRzep5KS5gay7R0tSNlJ2x9rH7PcWgrXLYb5wNJJUdZa2uuNbYOZSqwnpe1gl+GDfvKpph+rJJIrmf9wYl/Rz/SNsKE56m8jJ+ta67mz2onSmHY1jWie899JaLnY9m+qGOrpQ+rhIdrnVI4/bvas0+vyWuoDCGTHtS+YKsNGEsyKdEi5AurtiH9lL5DiaX+ncrU+aWXUqG+sTuT8uo64Gcxjmu8y/yIvsi1lMk6sni6ce5S7ahrtuZ/6h+0gfpibW+r0tZ9+/a1x4yjej0+MzUeUpbGuKnxW/e9hM913r/OTV/fXPUeleNkbNX9C/0lu1fGO419/M5Au+l3iUzaw8/m/KuvcD3qvfB1JhUa8nMx209nzyP6os4zbcI4p98T+Hrv3r3tsapeMjtyb8X9q36fyFqlN5V260P2xUx6X5NUZfIqfY/PsZqvRHR9M/u+Xnt+RtRjQt8xRXRtswWlRKr06Q5WIuJvIuKppmn+Gm99MyIe2jh+KCK+sfjhmUWwERRsw2lgO44Y++KksB1HjH1xUtiOI8a+OClsxxFjX1wv+mQC/fuI+HhE/LSU8pONf/tvEfFXEfG/SimfiIgjEfGftmaIZrNs/AptG46fbWE7jpqN/9JgG44f++LIsS9OBvviyNnIvLANx499ceRsZMDYhmtCn+5g/yciavlIH1jscMxWcNlll0XTNLbh+DltO46bSy+91L44DeyLI8e+OBnsiyPnkksusS9OA/viyLn44ovti2tEr5pAy0Z1i9TmUbOneutMx0sNJTWyWkSOuk62YtV2cdROq26a46UWtNY2OuJsfSDHuEx94KLINMXURaqdaEOdE+p0aQ/VsbM1ID9L68XQ9jpejoufpbr7Wo0KfW+MNow4e83W6l1pa9OsRgn9g3/H+kAR3RpBPNZaMFmL6VrtGoX21ntmDQuuBW2Zy/nQ+hbLhrpw7XhTq7Gjenf6h94PW7rz77RjWa0Wj1Lz7Yh6+3O9L8YBXSOrar+5SDLdPOOcNmBokpbfnHfOhRZ0rMVl9T3WN9DYXmsLr897Pqt13TF28L6GbDeS1TjiXkFbdXNOtI4h/Y+xS23D5x9rpekcZ/5B/+Nn6XMxqxvF+9S6C2Mh2y/wOaPzwtir+xbW8Dl06FB7/N3vfrdzHmtC0af0s2grrSHCc2s1VCK695K1Oc/2bUP1zb7xVONTrS6WktXioQ9ktVE4DvXTrPYIoZ362lAZcqzN6sRwnjUO0f+ylus33nhje3z77bd3zuP3Rf6N+vaxY8faY93f1NA9DF/rPWct0MnQbHeGvjWBdP/C+KTXYJ2eWr2viLqPaR0nrh/dY/EajA/qb/y7rD4Tx6ExZtHfLearpmiMMcYYY4wxxhhjRoV/BDLGGGOMMcYYY4xZA0YhB6u1otZ0uSyVi6+ZOqetimutwVXmwJQ+lbGwPaCm6JIsFbMmJRortfvRe+NrTXvjXGZthJmCx+O+7YYj8taDhH+XSeCylN8hk6X512Q6+jqTS9KH1Z/pR/Q/XRdM+9S55bkch6ZzMl1b1wLTMTMpw6olYIRjzuRgNZlURDetVdPZaz6s7aZpU8ZdldJxjCpdqMU/jcm0r/psLb14LNKFiDx+0XYqceY9qiSS62QeySrlXxH12KuvuZ4yv9HnOM8dkr9lcC51LfMemOqu95ZJCGpp6uqztTbSmm5O/8vknbSNyilqkng9d8j+lpHJF7LnEW2g8ZY25jGlJBHduc0kQZkUmuuEe2CNHfNI98di08w2nDuNcbzvTKLFY/Wxmg+oDfl3+myt+anuSxhz1IY1uw3dhhxftnfnvOg+iPamPDaia/NMMn/XXXe1xzfccEN7TGlnRMQLL7zQHr/00kvV9/hdUvfejPMab/s+u4dq1757m1l8ka+z75XZs4rwsynJjYjYtWtXe8y1pHE3G2/tO1MmOV0E4/91wRhjjDHGGGOMMcacF/8IZIwxxhhjjDHGGLMGDFIOpjAdL5OgMN1PU61q6ZdZWi+vkXVe0PRLflYmt+grB5sCNamept9lqXk1OYlW/a91kNEUTa4XtU0tvVLP65tCO9Q0zPOhaae0I+dZbUBbaRou31PpD2EqOtMq1QZZVyP6M+VlKj1jmrR2IKjZTtOQh0QWu9iRje+pv1EyoHPC1zwv656XdWCrSRwiumtL7UayrlVTk9dG5DICwnlRO9IOWdpxLabqXNL++gymr6ssoe+YNB6NjSztvZa+rqhEi9BXVD6isfEMGoMpRdLYXVsj2X5L4b1l543J1rX9QtaNSe1DWRY706p9aINszWTPxVrnqmyPqnF5rFKiM2RlJ7L9QNYxrdahWP2mJhHNpCrq99zP8HN1vWTyy7HLiCLOtmOt+3P2PYFzGVEv46HfOZ977rn2uCYhi+jaVfcwlIdlZRQYi3V8tVg5ZLuR7Ht4rWtfRFeel3WJYxzLyozQd3S90J8zORjXXyaTVn+udcrcatn7NHbDxhhjjDHGGGOMMSbFPwIZY4wxxhhjjDHGrAH+EcgYY4wxxhhjjDFmDRhMTaBMF0sy7TGvoRrcWss1Pa/22Zl+OKsTk7UEzdpq9q1bMcb6FrX6EhF5XQreK7W+qq3nXPL6qtPlezqOmma7bxv4qdC3daOuQ+pnVcdLG2d1H2pzm/nivPap1QLLPnurWzcuCtUa83XWtr1WIyGirrfWOa7VcNO6BdRfqxaemu2sjlrfZ8NU/LR2H+ofmb/xvdq6iOjahJ+ruvlsbjmurPU0NfVq47HXIVH61gap1dSLqNfTyvYltLvauu/+q8+/65jWgaxWV63WRUTXJ/rW6ur7/NSYUKuR1vc8vf4U4FzW6i4pWi+Ie9HMF2vrQO3JuKvzXxuv2oXPz8y+ZEy21bHS51inR23FGjv6HGNdtBdffLE9fuqpp6rjoO21lTnfy/YmWazsW7NpTLbrA+dE42lWY421u3icxWT6uu5Da9fWz+Y19NnKZ/csdYa3kvH9gmCMMcYYY4wxxhhjZsY/AhljjDHGGGOMMcasAYORgxFNUewr0crS6phynsmAamRyob4SlEWk045R/qX0nTtN25vHhll7zHlsuO70Xdu0naZpbtb/5m1lOtYWqFtJX9mBtjWt2VDtydjdV36ZpXf3jafrZk/erz77svc4t7S/SrRqkjK1Y5bOzs/KZDGZzHQKz78a2dqmH+na5ns8rrU51vNm8cXaeUpmpzG1ft8s8+4r5nlG6j45s8E8sr6pxdRMGsu51Oci/YqSoohu3MxKQdA2KlOqjUmlJZSMZDG+7x54Kn7J++c86/cJyi8z+e08csxMFt/X772/eQfOSd/SBhERp06dao9p++y7Cu2u59Hv5y3fwrWksu5MfraVTHdHZYwxxhhjjDHGGGNa/COQMcYYY4wxxhhjzBrgH4GMMcYYY4wxxhhj1oBB1gRS+raPr/3NuV6bYbJOmvQpYztOi6yWQMZW1hnwOjo3Oi9Z/Z1anbW+ZPVk+mI7nk1mw0WwiP2Q7bY5Fv2MXPQaWWeyOiSs66GtxUnfmkyLsFvfGl/rxqLnOWPevc4626cPfb/zK33brNfqr2V1nTLGVrvJv4wYY4wxxhhjjDHGrAH+EcgYY4wxxhhjjDFmDSjLbAtYSnk5Ip6LiJ0R8crSPvjcDGEMEcsZx81N0+xaxIUGZsOI9RrHou34eqzP3PVhjDa0L57NGO1oX+wyRhvaF89mjHa0L3YZow3ti2czRjvaF7uM0Yb2xdWMoZcdl/ojUPuhpTzeNM37lv7BAxvDkMYxK0MZt8cxP0MZs8exOYYybo9jfoYyZo9jcwxl3B7H/AxlzB7H5hjKuD2O+RnKmD2OzTGUcQ9hHEMYA7EczBhjjDHGGGOMMWYN8I9AxhhjjDHGGGOMMWvAqn4EemRFn0uGMIaI4YxjVoYybo9jfoYyZo9jcwxl3B7H/AxlzB7H5hjKuD2O+RnKmD2OzTGUcXsc8zOUMXscm2Mo4x7COIYwhpaV1AQyxhhjjDHGGGOMMcvFcjBjjDHGGGOMMcaYNWCpPwKVUj5cSjlQSjlUSnl4iZ/71VLKiVLKE/i3HaWUb5dSDm78/zVLGMfeUso/l1KeKqU8WUr581WNZTOssx1tw01/rm24IFZlw43Pth0XhH3RNtzkZ9uOC8K+aBtu8rNtxwVhX7QNN/nZtmMfmqZZyv8i4sKIeDoi9kfEJRHxfyPiniV99n+MiAcj4gn82xcj4uGN44cj4n8sYRx7IuLBjeOrIuIXEXHPKsZiO9qGtqFtaDuurx1tw/Hb0Hachh1tw/Hb0Hachh1tw/Hb0HacYYxLNMj7I+Kf8PovI+Ivl/j5t8hiOBARe2CoA0uf/IhvRMQHhzAW29E2tA1tQ9txvexoG47fhrbjNOxoG47fhrbjNOxoG47fhrZjv/8tUw52Y0QcxetjG/+2Kq5vmuZ4RMTG/1+3zA8vpdwSEX8UET9Y9VhmxHbcwDZcGLbh7AzNhhG24zwMzY624ewMzYYRtuM8DM2OtuHsDM2GEbbjPAzNjrbh7AzNhhG241ks80egco5/a5b4+YOhlLItIv4+Ij7TNM2vVz2eGbEdwzacArbhNLAdx49tOA1sx/FjG04D23H82IbTYMh2XOaPQMciYi9e3xQRLyzx85WXSil7IiI2/v/EMj60lHJxvLMY/q5pmn9Y5VjmZO3taBsuHNtwdoZmwwjbcR6GZkfbcHaGZsMI23EehmZH23B2hmbDCNtxHoZmR9twdoZmwwjb8SyW+SPQYxFxRynl1lLKJRHxpxHxzSV+vvLNiHho4/iheEert6WUUkpE/E1EPNU0zV+vciybYK3taBtuCbbh7AzNhhG24zwMzY624ewMzYYRtuM8DM2OtuHsDM2GEbbjPAzNjrbh7AzNhhG249ksuSjSn8Q71bGfjoj/vsTP/XpEHI+I38c7v05+IiKujYjvRMTBjf/fsYRx/Id4Jx3u/0XETzb+9yerGIvtaBvahrah7bj6/9kXbUPbcRj/sy/ahrbjMP5nX7QNbcet/1/ZGKgxxhhjjDHGGGOMmTDLlIMZY4wxxhhjjDHGmBXhH4GMMcYYY4wxxhhj1gD/CGSMMcYYY4wxxhizBvhHIGOMMcYYY4wxxpg1wD8CGWOMMcYYY4wxxqwB/hHIGGOMMcYYY4wxZg3wj0DGGGOMMcYYY4wxa4B/BDLGGGOMMcYYY4xZA/4/6yGS1lkD8qYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1440x144 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get weights\n",
    "weights = sess.run(W)\n",
    "\n",
    "fig, ax = plt.subplots(1, 10, figsize=(20, 2))\n",
    "\n",
    "for digit in range(10):\n",
    "    ax[digit].imshow(weights[:,digit].reshape(28,28), cmap='gray')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Close session to finish\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Full Code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The entire model, with the complete model definition, training, and evaluation (but minus the weights visualization), is below. Note the slight difference when calculating the softmax; this is done for numerical stability purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████| 1000/1000 [00:01<00:00, 738.63it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.9187999963760376\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tqdm import trange\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "# Import data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)\n",
    "\n",
    "# Create the model\n",
    "x = tf.placeholder(tf.float32, [None, 784])\n",
    "W = tf.Variable(tf.zeros([784, 10]))\n",
    "b = tf.Variable(tf.zeros([10]))\n",
    "y = tf.matmul(x, W) + b\n",
    "\n",
    "# Define loss and optimizer\n",
    "y_ = tf.placeholder(tf.float32, [None, 10])\n",
    "cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels=y_, logits=y))\n",
    "train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)\n",
    "\n",
    "# Create a Session object, initialize all variables\n",
    "sess = tf.Session()\n",
    "sess.run(tf.global_variables_initializer())\n",
    "\n",
    "# Train\n",
    "for _ in trange(1000):\n",
    "    batch_xs, batch_ys = mnist.train.next_batch(100)\n",
    "    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})\n",
    "\n",
    "# Test trained model\n",
    "correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "print('Test accuracy: {0}'.format(sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels})))\n",
    "\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: The accuracy from the full version directly above might return a slightly different test accuracy from the step-by-step version we first went through. This is because mnist.train.next_batch by default shuffles the order of the training data, so we're seeing the data in a different order. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# In-class Exercise: Multi-layer Perceptron\n",
    "\n",
    "Build a 2-layer Multi-layer Perception (MLP) for MNIST digit classfication. Feel free to play around with the model architecture and see how the training time/performance changes, but to begin, try the following:\n",
    "\n",
    "Image (784 dimensions) -> fully connected layer (500 hidden units)  -> nonlinearity (ReLU) -> fully connected layer (100 hidden units) -> nonlinearity (ReLU) -> fully connected (10 hidden units) -> softmax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "# Import data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def weight_variable(shape):\n",
    "    \"\"\"weight_variable generates a weight variable of a given shape.\"\"\"\n",
    "    initial = tf.truncated_normal(shape, stddev=0.1)\n",
    "    return tf.Variable(initial)\n",
    "\n",
    "def bias_variable(shape):\n",
    "    \"\"\"bias_variable generates a bias variable of a given shape.\"\"\"\n",
    "    initial = tf.constant(0.1, shape=shape)\n",
    "    return tf.Variable(initial)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 0, validation accuracy 0.1188\n",
      "step 250, validation accuracy 0.9132\n",
      "step 500, validation accuracy 0.9532\n",
      "step 750, validation accuracy 0.9636\n",
      "step 1000, validation accuracy 0.9696\n",
      "step 1250, validation accuracy 0.9708\n",
      "step 1500, validation accuracy 0.9674\n",
      "step 1750, validation accuracy 0.972\n",
      "step 2000, validation accuracy 0.9712\n",
      "step 2250, validation accuracy 0.9772\n",
      "step 2500, validation accuracy 0.9776\n",
      "step 2750, validation accuracy 0.976\n",
      "step 3000, validation accuracy 0.9764\n",
      "step 3250, validation accuracy 0.977\n",
      "step 3500, validation accuracy 0.9796\n",
      "step 3750, validation accuracy 0.9798\n",
      "test accuracy 0.9731\n"
     ]
    }
   ],
   "source": [
    "# Model Inputs\n",
    "x = tf.placeholder(tf.float32, [None, 784])\n",
    "y_ = tf.placeholder(tf.float32, [None, 10])\n",
    "\n",
    "# Define the graph\n",
    "# First fully connected layer\n",
    "W_fc1 = weight_variable([784, 500])\n",
    "b_fc1 = bias_variable([500])\n",
    "# h_fc1 = tf.nn.sigmoid(tf.matmul(x, W_fc1) + b_fc1)\n",
    "h_fc1 = tf.nn.relu(tf.matmul(x, W_fc1) + b_fc1)\n",
    "\n",
    "# Second fully connected layer\n",
    "W_fc2 = weight_variable([500, 10])\n",
    "b_fc2 = bias_variable([10])\n",
    "y_mlp = tf.matmul(h_fc1, W_fc2) + b_fc2\n",
    "\n",
    "# Loss \n",
    "cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels=y_, logits=y_mlp))\n",
    "\n",
    "# Optimizer\n",
    "train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)\n",
    "\n",
    "# Evaluation\n",
    "correct_prediction = tf.equal(tf.argmax(y_mlp, 1), tf.argmax(y_, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    # Initialize all variables\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    # Training regimen\n",
    "    for i in range(4000):\n",
    "        # Validate every 250th batch\n",
    "        if i % 250 == 0:\n",
    "            validation_accuracy = sess.run(accuracy, feed_dict={x: mnist.validation.images, y_: mnist.validation.labels})\n",
    "            print('step %d, validation accuracy %g' % (i, validation_accuracy))\n",
    "        \n",
    "        # Train    \n",
    "        batch = mnist.train.next_batch(50)\n",
    "        loss, _ = sess.run([cross_entropy, train_step], feed_dict={x: batch[0], y_: batch[1]})\n",
    "\n",
    "    print('test accuracy %g' % sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Questions/Exercises (Optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. How many learnable weight parameters does your MLP have? How does that compare with the logistic regression model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Your answer here]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Try making some changes to your MLP architecture. How sensitive is your MLP's accuracy to the number of layers (depth) or number of units per layer (width)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Your answer here]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Acknowledgment: Material partially adapted from the TensorFlow tutorial: https://www.tensorflow.org/get_started/*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}