4. Classify MNIST images with TensorFlow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Classify MNIST images with TensorFlow\n",
    "\n",
    "In this section, we will see how to recognize hand-written digits with a simple neural network running on TensorFlow.\n",
    "This is an adaptation of [TensorFlow Tutorial](https://www.tensorflow.org/versions/master/tutorials/layers)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/Screen%20Shot%202017-02-18%20at%206.04.53%20PM.png)\n",
    "\n",
    "### Clear all Cells\n",
    "\n",
    "Before starting, select `Clear all Cells` from the `Clear` menu at the top of Datalab, to clear all the cell result in this section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4-1. Classifying images as n-dimensional data\n",
    "\n",
    "So far, we have learned how to classify datapoints in two dimensional spaces, such as a geolocation with latitude and longitude. In this section, we will extend the same technique to classify datapoints in n-dimensional space. \n",
    "\n",
    "What do you mean by **\"classifying datapoints in n-dimensional space\"**? As an example, we will use images of handwritten text called [MNIST](http://yann.lecun.com/exdb/mnist/) dataset as the datapoints in n-dimensional space. MNIST is one of the most popular datasets used for learning neural network technology. It's like a **hello world** in neural network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading MNIST training data\n",
    "\n",
    "At first, let's download the MNIST dataset. **Run the cell below**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# import MNIST data\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)\n",
    "print (mnist.train.images.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 55,000 training Images\n",
    "\n",
    "The variable `mnist` holds 55,000 gray scale images of handwritten text. Each image has 784 values that represent each pixel in an image. So it looks like this:\n",
    "\n",
    "![mnist.train.xs](images/Screen%20Shot%202017-02-18%20at%2011.17.39%20AM.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Single image = 784 real values\n",
    "\n",
    "Each handwritten text image has 784 values of pixels from 0.0 (white) 1.0 (black). \n",
    "\n",
    "![](images/Screen%20Shot%202017-02-18%20at%206.06.25%20PM.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, if you print the values of 5th images out of the 55,000 images, it looks like this: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# check MNIST training images matrix data\n",
    "sample_img = mnist.train.images[5].reshape(28, 28)\n",
    "print(sample_img)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also plot the image with Matplotlib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.imshow(sample_img).set_cmap('Greys')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training Labels\n",
    "\n",
    "The MNISt dataset also contains the labels for each image for training.\n",
    "\n",
    "![mnist.train.ys](images/Screen%20Shot%202017-02-18%20at%2011.21.20%20AM.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Run the cell below** to check the shape and the values in the label array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# check MNIST labels shape\n",
    "print(mnist.train.labels.shape)\n",
    "\n",
    "# show MNIST label data\n",
    "print(mnist.train.labels[5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the example above, you have 1.0 at the 8th value. That means the label for the 5th image is \"8\". This is so called **one-hot vector**, a popular way of encoding the labels in machine learning classification problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Lab: Use Embedding Projector to see how N-dimensional space look like\n",
    "\n",
    "If you have 784 values in an array or a vector, it is called \"784 dimensional vector\" in the context of machine learning. It's a vector in 784 dimensional space.\n",
    "\n",
    "If you have X and Y values in 2D space, or X, Y and Z in 3D space, it's really easy for humans to imagine how they look like. For example, if you have three values in a 3D vector that represents \"you like movies so much, you also like actors, and you like music a little\", you can draw a vector in 3D space like this.\n",
    "\n",
    "![](images/3d-vector.png)\n",
    "\n",
    "Meanwhile, we can't imagine how high dimensional spaces and vectors look like, if it's higher than 3D. You can't draw a picture in your head what kind of shape a vector in 784 dimensional space would have.\n",
    "\n",
    "But there's a great tool to visualize that. Open [TensorFlow Embedding Projector](http://projector.tensorflow.org/) and follow the steps below.\n",
    "\n",
    "- Select `MNIST images` as DATA at the top left\n",
    "- Select `label` from the `Color by` menu\n",
    "- Select `T-SNE` tab at the middle of left navigation\n",
    "\n",
    "You will see the MNIST images would be slowly grouped into 10 groups for each digits.\n",
    "\n",
    "<br/>\n",
    "![](images/MNIST.png)\n",
    "<br/>\n",
    "\n",
    "What's happening here? The tool uses an algorithm called [t-SNE](https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding) to do [dimensionarity reduction](https://en.wikipedia.org/wiki/Dimensionality_reduction). That means, you can \"cast a shadow\" of the n-dimensional space into 3D or 2D space. So, what you are watching above is **a shadow** of MNIST image vectors in 784 dimensional space, casted on 3D space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 4-2. Defining a Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Single neuron can recognize a single digit\n",
    "\n",
    "So, an image in MNIST dataset is a 784 dimensional vector. And you can use a single neuron to classify each vector is an image of \"1\" or not. To do that you can do the same thing we have done with latitude and longitude: mutiplies the 784 values with weights and checks if the sum exceeds a certain threthold.\n",
    "\n",
    "![](images/Screen%20Shot%202017-02-18%20at%206.08.38%20PM.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define a neural network as a graph\n",
    "\n",
    "To classify an image to 10 digits, we need a single layer neural network (Perception) with 10 neurons. It would look like this. Here we have inputs from X1 to X784, multiplied with the bunch of weights to get 10 summation results, added to the 10 biases that work as thresholds. We'll see what is \"softmax\" later.\n",
    "\n",
    "![](images/Screen%20Shot%202017-02-18%20at%2011.22.26%20AM.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Or, as a matrix operation:\n",
    "\n",
    "There is a great way in Math to calculate this kind of \"multiplying a set of numbers agains another set of numbers\" with a single formula. That is called [Matrix operation](https://en.wikipedia.org/wiki/Matrix_multiplication). So, you can define a 784 dimensional vector for holding the image data, a 784 x 10 matrix for holding the weights, and 10 dimensional vector for holding biases. Then you can define a single layer neural network with the following formula by using a [dot product](https://en.wikipedia.org/wiki/Dot_product) between the weight matrix and the image vector.\n",
    "\n",
    "<br/>\n",
    "![](images/Screen%20Shot%202017-02-18%20at%2011.22.40%20AM.png)\n",
    "<br/>\n",
    "\n",
    "You can also write this calculation in the following formula, where **W** is the weight matrix, **x** is the input vector, and **b** is the bias vector.\n",
    "\n",
    "<br/>\n",
    "$${\\Huge y=softmax(Wx + b)}$$\n",
    "<br/>\n",
    "\n",
    "This is the reason why you see many math formulas in neural network and machine learning text books. It's so easier to use them to express the ideas in a terse math formula. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Or, as a TensorFlow graph:\n",
    "\n",
    "In TensorFlow, there is so-called **Low level API** that allows you to define the vector and matrix operations above by using **Tensor**. It looks like this. **Run the cell below** to define the network. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# define a neural network (softmax logistic regression)\n",
    "import tensorflow as tf\n",
    "x = tf.placeholder(tf.float32, [None, 784]) # a placeholder for inputting the image \n",
    "W = tf.Variable(tf.zeros([784, 10])) # weights\n",
    "b = tf.Variable(tf.zeros([10])) # biases\n",
    "y = tf.nn.softmax(tf.matmul(x, W) + b)\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tensor** is just an array. In Physics, they use the word Tensor for the complex math calculations, but you can forget about it in the context of machine learning. Tensor in TensorFlow is just a multi-dimensional array that can hold any high or low dimensional vectors and matrices of the input data, weights, biases and etc.\n",
    "\n",
    "So, the word **TensorFlow** means that you can use the Low level API to define **a flow (a graph) of calculations between vectors and matrices**. In this case, we define the following computation graph.\n",
    "\n",
    "- At line 3, it calls `tf.placeholder` method to define a Tensor `x` for accepting any number of 784 dimensional vectors. This will be used to receive the training image data\n",
    "- At line 4, it calls `tf.Variable` method to define a Variable `W` for holding the weight matrix that has 784 x 10 values\n",
    "- At line 5, it calls `tf.Variable` method to define a Variable `b` for holding the bias vector that has 10 values\n",
    "- At line 6, it calls `tf.matmul` method to define a dot product between `x` and `W`, and calls `tf.nn.softmax` method to define a **softmax** of the value. The result Tensor is named as `y`\n",
    "\n",
    "Please keep in mind that you are only **defining** the graph of calculations, not **executing** the calculations at this time. You need to pass the graph definition to `Session` to execute it. We will look how it works later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The trained weights will be \"filters\"\n",
    "\n",
    "By training the network with the 55,000 images, you will have patterns of weights like the following. The blue area has **positive** weights, and the red area has **negative** weights.\n",
    "\n",
    "![](images/Screen%20Shot%202017-02-18%20at%2011.22.14%20AM.png)\n",
    "\n",
    "You can see that the blue and red patterns would work as \"filters\" for looking at each image. The network applies those filters on each image, and see the matching. If an image matches well with a filter (weights) for \"8\" and the summation exceeds the threashold (bias) for \"8\", the network believes the images must be an image of \"8\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is Softmax?\n",
    "\n",
    "So, what does the softmax method does? Softmax is a function that converts an array of values into an array of **probabilities** (0 - 1.0).\n",
    "\n",
    "<br/>\n",
    "<br/>\n",
    "<br/>\n",
    "$${\\Huge softmax(n) = \\frac{\\exp n_i}{\\sum \\exp n_i}}$$\n",
    "<br/>\n",
    "<br/>\n",
    "<br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how it works by using 10 random values. **Run the cell below** to create 10 random values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "i = np.arange(0, 10)\n",
    "n = np.random.randn(10)\n",
    "plt.bar(i, n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, **run the cell below** to calculate the softmax values for those random values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def softmax(n):\n",
    "  return np.exp(n) / np.sum(np.exp(n))\n",
    "\n",
    "s = softmax(n)\n",
    "plt.bar(i, s)\n",
    "print('The sum of softmax: ' + str(np.sum(s)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, softmax normalizes the original values to compare them as probabilities from 0.0 to 1.0, and the summation of all values will be 1.0. So you can choose a single value closest to 1.0 as the final answer from neural network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4-3. Defining the Train Step\n",
    "\n",
    "Next, you need to define how to train the network. **Run the cell below**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# define the train step to minimize the cross entropy with SGD\n",
    "y_ = tf.placeholder(tf.float32, [None, 10]) # the training labels\n",
    "cross_entropy = -tf.reduce_sum(y_ * tf.log(y))\n",
    "train_step = tf.train.GradientDescentOptimizer(0.01).minimize(cross_entropy)\n",
    "train_step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- At line 2, it calls `tf.placeholder` method to define a Tensor `_y` for accepting any number of 10 dimensional vectors. This will be used to receive the training labels\n",
    "- At line 3, it calls `tf.log` and `tf.reduce_sum` to define a **cross entropy** calculation on the softmax result from the network. The result Tensor is named as `cross_entropy`\n",
    "- At line 4, it calls `tf.train.GradientsDescentOptimizer.minimize` method to use **Gradient Descent** algorithm to train the network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Lab: What is a loss function and Cross Entropy?\n",
    "\n",
    "When you train your model in machine learning, you define a [loss function](https://en.wikipedia.org/wiki/Loss_function) for evaluating the accuracy the model while you are training it. In neural network, one of the popular loss function is [Cross Entropy](https://en.wikipedia.org/wiki/Cross_entropy) as follows.\n",
    "\n",
    "Cross entropy function is like **a teacher** for training your model who'd measure how much error your neural network is making. \n",
    "\n",
    "<br/>\n",
    "![](images/cross_entropy.png)\n",
    "<br/>\n",
    "(Diagram quoted from: [TensorFlow and deep learning, without a PhD](https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist/#4))\n",
    "\n",
    "The formula simply means, it returns **higher value** when you have many **wrong answers**, and **lower value** when you have many **correct answers**. \n",
    "\n",
    "Let's see how it works in practice. **Run the cell below** to define a label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# label\n",
    "label = np.array([0, 0, 0, 0, 0, 0, 0, 0, 1, 0])\n",
    "plt.bar(i, label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, let's calculate a cross entropy value on the softmax values calculated at the last lab. **Run the cell below**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def cross_entropy(x, _y):\n",
    "  return -np.sum(_y * np.log(x))\n",
    "cross_entropy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This defines the method `cross_entropy`. You can pass the output from the softmax as parameter `x`, and pass the labels as `_y`. So that the method calculates cross entropy value.\n",
    "\n",
    "Now, **run the cell below** to emulate what would happen during the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# simulate the training\n",
    "cross_ent = []\n",
    "for i in range(0, 100):\n",
    "  cross_ent.append(cross_entropy(softmax(n), label))\n",
    "  n[8] += 0.1\n",
    "plt.plot(cross_ent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the code above, it manually increase the value of `n[8]` with the loop. It's faking a training to see how the cross entropy value changes as training proceeds. As you can see on the graph, cross entropy function has **a steep curb** at the initial phase of the training, means it returns much bigger value if the answer from the network is wrong. That's the reason why many people love it as a loss function to **make the neural network training faster**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Create a `Session` in TensorFlow\n",
    "\n",
    "Now, let's start the training of our network. First, you need a `Session` as a runtime for your TensorFlow graph. **Run the cell below** to define it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# supress warning messages\n",
    "tf.logging.set_verbosity(tf.logging.ERROR)\n",
    "\n",
    "# initialize variables and session\n",
    "init = tf.initialize_all_variables()\n",
    "sess = tf.Session()\n",
    "sess.run(init)\n",
    "sess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At line 6, it creates a new `Session`. Session represents a runtime of TensorFlow where you can execute various TensorFlow operations for training, testing and etc. With TensorFlow Low level API, you follow the steps below to do training.\n",
    "\n",
    "- Define a neural network\n",
    "- Define a train step with a loss function and an optimization algorithm\n",
    "- Create a `Session` for training the model\n",
    "- Run the training with the `Session`\n",
    "\n",
    "Why we need to use this tedious procedures? Because TensorFlow is designed to be **scalable and portable**. You can specify different kinds of `Sessions` that represent the runtime on your local laptop, GPUs on a PC, or CPU/GPU/[TPU](https://cloudplatform.googleblog.com/2016/05/Google-supercharges-machine-learning-tasks-with-custom-chip.html) on the cloud. So, the whole design - defining a computation graph and pass it on a runtime - allows the code independent from a particular runtime. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training with mini batches\n",
    "\n",
    "Next, start training. In this case we will use [Stochastic Gradient Decent (SGD)](https://en.wikipedia.org/wiki/Stochastic_gradient_descent). That means, when you apploy the Gradient Descent algorithm (defined with the tf.Operation `train_step`) to the training data, you will randomly pick (so it is called \"stochastic\") 100 samples out of 55,000 images and labels. This 100 random sample is called **mini batch**. SGD allows you to have the training much faster and efficient than applying gradient decent on entire training data. **Run the cell below** to try this in practice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for i in range(1000):\n",
    "  batch_xs, batch_ys = mnist.train.next_batch(100)\n",
    "  sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})\n",
    "\n",
    "print('Training Finished.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- At line 2, it randomly picks 100 sample images and labels as a mini batch\n",
    "- At line 3, it calls `Session.run` method to pass the train step definition and the mini batch.\n",
    "- At line 1, it repeats the training with a mini batch for 1000 times. With each mini batch, the optimization algorithm moves weights and biases toward the direction of less cross entropy value\n",
    "\n",
    "This is what was happening inside the `DNNClassifier` we have used in the section 2. You can define your own network and algorithm with L ow level API for sophisticated or complex applications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's use it!\n",
    "\n",
    "Now we have the network that can predict a label for each image. Let's use it for predicting labels for test images. **Run the cell below**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "predict_label = tf.argmax(y, 1)\n",
    "predicted_labels = sess.run(predict_label, {x: mnist.test.images})\n",
    "print(predicted_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- At line 1, it calls `tf.argmax` to define a graph for finding a label that has the largest softmax value\n",
    "- At line 2, it calls `Session.run` to execute the calculation with the graph definition `predict_label` where it uses `mnist.test.images` (the array of 10,000 test images) as `x` in the neural network. This returns an array with 10,000 predicted labels\n",
    "\n",
    "**Run the cell below** and try changing the parameter to see prediction results on various labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def show_image_and_predicted_label(x):\n",
    "  print(\"  Correct label: \" + str(np.argmax(mnist.test.labels[x])))\n",
    "  print(\"Predicted label: \" + str(predicted_labels[x]))\n",
    "  plt.imshow(mnist.test.images[x].reshape(28, 28)).set_cmap('Greys')\n",
    "  return x\n",
    "\n",
    "show_image_and_predicted_label(20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculating the accuracy\n",
    "\n",
    "At last, let's check the accuracy of the network you have just trained. **Run the cell below** to do that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "is_prediction_correct = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))\n",
    "calc_accuracy = tf.reduce_mean(tf.cast(is_prediction_correct, \"float\"))\n",
    "accuracy = sess.run(calc_accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels})\n",
    "print('Accuracy: ' + str(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- At line 1, it calls `tf.argmax` method on `y` to pick the prediction results from the network, and `_y` to pick the correct labels. The `tf.equal` method compares them and generate a list of boolean values\n",
    "- At line 2, it takes a mean value of how many Trues found in the `calc_accuracy`. The accuracy value will be returned from the `Session.run` method at line 3\n",
    "- At line 3, it passes the testing images to Tensor `x` and testing labels to Tensor `y`, and calls `Session.run` method to run the test calculation flow\n",
    "\n",
    "That's it!\n",
    "\n",
    "In this session you have seen how you can use TensorFlow Low level API define the single layer neural network, train it, and use the network to classify the test images at about 92% accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Lab: Try Deeper!\n",
    "\n",
    "- Try modifying the code above to define your deep neural network with 3 (or more) layers, and see how it increase the accuracy\n",
    " - Hint: you may refer to [this codelab page](https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist/#0) from Martin Gorner's great codelab material to learn how to build multi layer neural network with Low level API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Today We Learned\n",
    "\n",
    "- **Single neuron** can take an image and recognize a digit\n",
    "- **Matrix operation** (y = Wx + b) can define a neural network concisely\n",
    "- **Softmax** converts the result to probabilities (0.0 - 1.0)\n",
    "- **Cross Entropy** evaluates how much error the network makes (the loss function)\n",
    "- **Gradient Decent** moves weights and biases toward the direction with less error\n",
    "- **Mini Batch** trains the network with small batch of data\n",
    "- The word **TensorFlow** means that you can use the Low level API to define a flow of calculations between vectors and matrices\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What's Next\n",
    "\n",
    "Congrats! This is the end of this codelab. If you want to learn more about the advanced techniques you can use with TensorFlow, you may check the following page:\n",
    "\n",
    "- [TensorFlow for deep learning Research](http://web.stanford.edu/class/cs20si/syllabus.html) by Stanford Univ.\n",
    "- [Learn TensorFlow and deep learning, without a Ph. D](https://cloud.google.com/blog/big-data/2017/01/learn-tensorflow-and-deep-learning-without-a-phd) by Martin Gorner \n",
    "\n",
    "With these materials you can learn more about TensorFlow APIs, advanced neural network algorithms such as Convolutional Neural Network, and several optimization techniques to get better accuracy on training models.",
    "\n",
    "**Important Note**: Please make sure to **[Delete the Datalab](https://cloud.google.com/datalab/docs/quickstarts#clean-up)** and delete [Permanent Disk (PD)](https://console.cloud.google.com/compute/disks) after finishing the codelab. If you leave Datalab running on the cloud, it possible **you will get charged** for it."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  },
  "widgets": {
   "state": {
    "d8882490ac2a4b63bd27dd8ffdddf74a": {
     "views": [
      {
       "cell_index": 48
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}