3. Why deep neural network can get smarter?.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. How do deep neural network get smarter?\n",
    "\n",
    "In this section, we will learn how a deep neural network (DNN) obtains its power - and can then become smart enough to solve complex problems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 3-1. The power of hidden layers\n",
    "\n",
    "In the last lab, we saw that a single layer neural network can only **draw a straight line** to classify the areas on the New York map.  That means that it can never fully split the locations in Manhattan and Brooklyn, since doing that exactly requires a curved boundary.\n",
    "\n",
    "Instead, by using DNN with 4 **hidden layers**, you can draw a curved boundary between Manhattan and Brooklyn and get 99.9% accuracy. So, what are hidden layers and how can they be so powerful? \n",
    "\n",
    "There's a reason why neural networks become much smarter with hidden layers. Let's take a look at another sample dataset using the TensorFlow Playground.\n",
    "\n",
    "<br/>\n",
    "![](https://cloud.google.com/blog/big-data/2016/07/images/146798944178238/neural-networks-4.png)\n",
    "<br/>\n",
    "\n",
    "This dataset cannot be classified by a single neuron, as the two groups of data points can't be divided by a single line. This is a so-called **nonlinear classification problem**. In the real world, there's no end of non-linear and complex datasets such as this one, and the question is how can we capture these sorts of complex patterns?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Lab: Solve a non-linear problem using hidden layers\n",
    "\n",
    "- [Click this link](http://playground.tensorflow.org/#activation=sigmoid&regularization=L2&batchSize=10&dataset=circle&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=3&seed=0.84062&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&showTestData_hide=true&learningRate_hide=true&regularizationRate_hide=true&percTrainData_hide=true&numHiddenLayers_hide=true&discretize_hide=true&activation_hide=true&problem_hide=true&noise_hide=true&regularization_hide=true&dataset_hide=true&batchSize_hide=true&playButton_hide=false) to open the Playground for this problem, and click the Run button to start training\n",
    "- After network training, move your cursor over each neuron in the hidden layer and see what kind of lines they draw\n",
    "- Move your cursor over each line of weights (the dash lines in blue and orange) and see the weight values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Hidden layers \"tweak\" the data to extract features\n",
    "\n",
    "![](https://cloud.google.com/blog/big-data/2016/07/images/146798944178238/neural-networks-1.png)\n",
    "\n",
    "<br/>\n",
    "\n",
    "What's happening here? If you click each one of the neurons in the hidden layer, you'll see that they're each doing a simple, single-line classification:\n",
    "\n",
    "- The first neuron checks if a data point is on the left or right\n",
    "- The second neuron checks if it's in the top right\n",
    "- The third one checks if it's in the bottom right\n",
    "\n",
    "These three results are called features of the data. Outputs from these neurons indicate the strength of their corresponding features.\n",
    "\n",
    "Finally, the neuron on the output layer uses these features to classify the data. If you draw a three dimensional space consisting of the feature values, the final neuron can simply divide this space with a flat plane. This is an example of a transformation of the original data into an 'easier' feature space.\n",
    "\n",
    "[colah's blog](http://colah.github.io/posts/2015-01-Visualizing-Representations/) has many great visualization of this concept. You can see in those graphs that the hidden layers are **tweaking** the original data to extract the important features, such as blue or red. This enables the final neuron to draw a straight line (or a hyperplane in n-dimensional space) to classify them by looking at these new features.\n",
    "\n",
    "![](https://cloud.google.com/blog/big-data/2016/07/images/146798944178238/neural-networks-28.png)\n",
    "![](http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/img/spiral.1-2.2-2-2-2-2-2.gif)\n",
    "\n",
    "![](https://cloud.google.com/blog/big-data/2016/07/images/146798944178238/neural-networks-16.png)\n",
    "\n",
    "In the case of the Playground demo, the transformation results in a composition of multiple features corresponding to a triangular or rectangular area. If you add more neurons by clicking the \"plus\" button, you'll see that the output neuron can capture much more sophisticated polygonal shapes from the dataset.\n",
    "\n",
    "<br/>\n",
    "\n",
    "### A neural network as an office worker\n",
    "\n",
    "Think of the computer as a student or junior worker in your office. A new worker gets confused and distracted by random signals coming from e-mails, phones, the boss, customers, etc., but senior workers are very efficient about extracting the essential signal from those inputs, and organize the chaos according to a few important principles.\n",
    "\n",
    "Neural networks work the same way — trying to extract the most important features in a dataset to solve the problem. You can picture the transformation of hidden layers as extracting the insights from data, just like an experienced professional is doing in their daily work. That's why neural networks can sometimes get smart enough to handle some pretty complex tasks.\n",
    "\n",
    "![](https://cloud.google.com/blog/big-data/2016/07/images/146798944178238/neural-network-23.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Lab: How do the hidden layers work to \"tweak\" the data?\n",
    "  \n",
    "- Discuss what \"the hidden layers tweak the data to make it lineary classifiable\" means with your buddy\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 3-2. How do you train a neural network?\n",
    "\n",
    "As we have been learning, a neural network is a simple mechanism that's implemented with basic math. The only difference between traditional programming and neural networks is that you let the computer determine the parameters (weights and biases) by learning from training datasets. In other words, the trained weight pattern in our example wasn't programmed by humans.\n",
    "\n",
    "In this codelab, we won't discuss in detail how the training of parameters with algorithms such as [backpropagation](http://colah.github.io/posts/2015-08-Backprop/) and [gradient descent](https://en.wikipedia.org/wiki/Gradient_descent) works. Suffice to say that the computer tries to increase or decrease each parameter a little bit to see how it reduces the error compared with training dataset, in hopes of finding the optimal combination of parameters.\n",
    "\n",
    "<br/>\n",
    "![](https://cloud.google.com/blog/big-data/2016/07/images/146798944178238/neuralnetworks-24.png)\n",
    "<br/>\n",
    "\n",
    "Getting back to the office worker analogy: In the beginning, the computer makes many mistakes and it takes some time before it finds a practical way of solving real-world problems by minimizing the errors. The ability of the network to use the training and apply it to future problems is called **generalization**.\n",
    "\n",
    "<br/>\n",
    "![](https://cloud.google.com/blog/big-data/2016/07/images/146798944178238/neural-networks-8.png)\n",
    "<br/>\n",
    "\n",
    "For now, just accept that a neural network library such as TensorFlow encapsulates most of the necessary math for training, so that you don't have to worry too much about it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 3-3. We need to go deeper: building a hierarchy of abstractions\n",
    "\n",
    "With more neurons in a single hidden layer, you can capture more features. And having more hidden layers means more complex constructs that you can extract from the dataset. You can see how powerful this can be in the next example.\n",
    "\n",
    "<br/>\n",
    "![](https://cloud.google.com/blog/big-data/2016/07/images/146798944178238/neural-networks-9.png)\n",
    "<br/>\n",
    "\n",
    "What kind of program would you have to write to classify this dataset? Dozens of IF statements with many, many conditions and thresholds, each checking which small area a given data point is in? We wouldn't want to do that.\n",
    "\n",
    "This is where machine learning and neural networks can exceed the performance of a human programmer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Lab: See how deep neural networks works with Playground\n",
    "\n",
    "- [Click this link](http://playground.tensorflow.org/#activation=relu&batchSize=10&dataset=spiral&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=8,8,5&seed=0.53586&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&showTestData_hide=true&activation_hide=true&problem_hide=true&noise_hide=true&discretize_hide=true&regularization_hide=true&dataset_hide=true&batchSize_hide=true&learningRate_hide=true&regularizationRate_hide=true&percTrainData_hide=true&numHiddenLayers_hide=true) and see how a deep neural network on the Playground can solve the double spiral problem\n",
    "\n",
    "- Move your cursor over the neurons in the first, second and the third hidden layers and see what kind of classification each neuron is doing\n",
    "\n",
    "![](https://cloud.google.com/blog/big-data/2016/07/images/146798944178238/neural-networks-10.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### More neurons + a deeper network = more sophisticated abstraction\n",
    "\n",
    "What you just saw was the computer trying to build a hierarchy of abstraction with a deep neural network. The neurons in the first hidden layers are doing the same simple classifications, whereas the neurons in the second and third layers are composing complex features out of the simple features, eventually coming up with the double spiral pattern.\n",
    "\n",
    "More neurons + a deeper network = more sophisticated abstraction. This is how simple neurons get smarter and perform so well for certain problems such as image recognition and playing Go.\n",
    "\n",
    "<br/>\n",
    "![](https://cloud.google.com/blog/big-data/2016/07/images/146798944178238/neural-networks-5.png)\n",
    "<br/>\n",
    "\n",
    "[Some published examples of visualization](http://vision03.csail.mit.edu/cnn_art/index.html#v_single) by deep networks show how they're trained to build the hierarchy of recognized patterns, from simple edges and blobs to object parts and classes.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Lab: Try desigining your own neural network with Playground\n",
    "\n",
    "- [Click this link](http://playground.tensorflow.org/#activation=relu&batchSize=10&dataset=spiral&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=4,2&seed=0.63289&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false&regularizationRate_hide=true&noise_hide=true&batchSize_hide=true&percTrainData_hide=true&regularization_hide=true&problem_hide=true) and try solving the same problem with your own design of deep neural network\n",
    "- Try adding/removing neurons in each layer and adding/removing hidden layers and see how it affects the training speed and accuracy\n",
    "- Try adding more preprocessed feature data at the left and removing hidden layers and neurons if possible. See how the additional features at the input can affect the training speed and accuracy\n",
    "- Try changing [activation functions](http://www.kdnuggets.com/2016/08/role-activation-function-neural-network.html) and [learning rate](http://blog.datumbox.com/tuning-the-learning-rate-in-gradient-descent/) and see how it affects the training speed and quality\n",
    "- Discuss with your buddy : What are the possible ways to design a neural network to solve the double spiral problem?\n",
    "- Discuss with your buddy : What are activation functions and how do they (and the learning rate) affect the training speed?\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The burden of hyperparameter tuning\n",
    "\n",
    "As you just experienced with the Playground, it's not so easy to design a deep neural network. You have to find an optimal combination of many different configurations for the following :\n",
    "\n",
    "- Number of neurons in each hidden layer\n",
    "- Number of hidden layers\n",
    "- Activation function\n",
    "- Learning rate\n",
    "- The kind of features you choose as inputs\n",
    "\n",
    "Those are so called **hyperparameters**, the parameters that you have to find before training starts. One challenge of using neural network methods is **hyperparameter tuning**, the trial-and-error process required to find the best combination of hyperparameters. You may check the Wikipedia page for [Hyperparameter optimization](https://en.wikipedia.org/wiki/Hyperparameter_optimization) to learn more."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Two challenges: computational power and training data\n",
    "\n",
    "In this lab, we've looked at some TensorFlow Playground demos and used them to explain the mechanism and power of neural networks. As you've seen, the basics of the technology are pretty simple. Each neuron just classifies a data point into one of two kinds. And yet, by having more neurons and deep/hidden layers, a neural network can extract its own insights and complex patterns from a training dataset and build a hierarchy of abstraction.\n",
    "\n",
    "The question is then, why isn't everybody using this great technology yet? There are two big challenges for neural networks right now. The first is that training deep neural networks requires a lot of computation power, and the second is that they require large training data sets. It can take several days or even weeks for a powerful GPU server to train a deep network with a dataset of millions of images.\n",
    "\n",
    "Also, it takes a lot of trial and error to get the best training results with many combinations of different network designs and algorithms. Today, some researchers use tens of GPU servers or even supercomputers to perform large-scale distributed training.\n",
    "\n",
    "But in very near future, fully managed distributed training and prediction services such as [Google Cloud Machine Learning Engine](https://cloud.google.com/ml) with TensorFlow may solve these problems with the availability of cloud-based CPUs and GPUs at an affordable cost, and may open the power of large and deep neural networks to everyone.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What We Learned\n",
    "\n",
    "- **Hidden layers** can automatically \"tweak\" the input data to extract important **features** from it, so that making it easier to classify **non-linear** problems with linear classification\n",
    "- By adding more hidden layers and neurons in each layer, you can build **a hierarchy of abstraction** in the neural network, making it much smarter. But you also need to spend much computing power to train the network\n",
    "- The kind of **Activation functions** and **learning rate** you choose for your network can affect the training time and accuracy\n",
    "- By adding **more preprocessed feature data as input**, you can simplify the neural network and shorten the training time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What's Next\n",
    "\n",
    "In the next session, we will try classifying handwritten texts with neural network. Please proceed with [4. Classify MNIST images with TensorFlow](4.%20Classify%20MNIST%20images%20with%20TensorFlow.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}