notebooks/NN-intro-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://image.shutterstock.com/image-photo/angry-people-screaming-260nw-292800785.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Reference](http://neuralnetworksanddeeplearning.com/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Neural networks, a beautiful biologically-inspired programming paradigm which enables a computer to learn from observational data\n",
    "- Deep learning, a powerful set of techniques for learning in neural networks "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### An example:\n",
    "\n",
    "![](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcS3I7goOlpVqv9_FZ_k6w4V0c56SCziU8klbsBBgh2TTL7eg3u9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's take a very closer look and brain:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://d3i71xaburhd42.cloudfront.net/4807bf70ab59417a295d92c54954832577812bfe/4-Figure1-1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Perceptrons"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray}\n",
    "  \\mbox{output} & = & \\left\\{ \\begin{array}{ll}\n",
    "      0 & \\mbox{if } \\sum_j w_j x_j \\leq \\mbox{ threshold} \\\\\n",
    "      1 & \\mbox{if } \\sum_j w_j x_j > \\mbox{ threshold}\n",
    "      \\end{array} \\right.\n",
    "\\tag{1}\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now we can make a neural network:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](http://neuralnetworksanddeeplearning.com/images/tikz1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray}\n",
    "  \\mbox{output} = \\left\\{ \n",
    "    \\begin{array}{ll} \n",
    "      0 & \\mbox{if } w\\cdot x + b \\leq 0 \\\\\n",
    "      1 & \\mbox{if } w\\cdot x + b > 0\n",
    "    \\end{array}\n",
    "  \\right.\n",
    "\\tag{2}\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import pylab as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f5f950e2e80>]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFCtJREFUeJzt3X+s3XV9x/Hnu61AHDrAVkTa2rJVZ90PITdINFFUhoU/WhfRtYkRHbPRiftDtwyDYYZl2dRkJmZsWp1juggii7PbajoUmIsRRomKFlK51jG6MlsRWRYicM5574/zveW7b8/t/bb3nHP7/ez5SG56zvf7Od/z5ns+58Xnfs7nnm9kJpKksixb6gIkSeNnuEtSgQx3SSqQ4S5JBTLcJalAhrskFchwl6QCGe6SVCDDXZIKtGKpnnjlypW5bt26pXp6Seqke++998eZuWqhdksW7uvWrWPPnj1L9fSS1EkR8VCbdk7LSFKBDHdJKpDhLkkFMtwlqUCGuyQVaMFwj4jPRMShiPjePPsjIj4eEbMRcV9EXDD+MiVJx6PNyP1GYNMx9l8GbKh+tgN/ufiyJEmLseA698z8ekSsO0aTLcBnc3i9vrsi4oyIOCczHxlTjdLU3fvQY/zLvkNLXYYK9fqXns2vrTljos8xjj9iOhd4uHb/QLXtqHCPiO0MR/esXbt2DE8tTcaf3baPb8w+SsRSV6ISPf+5p3Ui3Ed1/5FX3c7MHcAOgJmZGa/MrZPWU70Br/yF5/H5d1601KVIJ2Qcq2UOAGtq91cDB8dwXGnJ9AbJ8mUO29Vd4wj3ncDbqlUzFwGPO9+urusb7uq4BadlIuIm4GJgZUQcAP4QeBZAZn4C2AVcDswCTwDvmFSx0rT0+skKw10d1ma1zLYF9ifwnrFVJJ0EBunIXd3mX6hKI/QGyYplvj3UXfZeaQTn3NV1hrs0Qm8wcM5dnWa4SyP0+47c1W2GuzRCb5CsWG64q7sMd2kE59zVdYa7NIKrZdR19l5pBEfu6jrDXRrB1TLqOsNdGsGRu7rOcJdGGM65G+7qLsNdahgMkkxY7geq6jB7r9TQGwyvI+M6d3WZ4S419Ktwd85dXWa4Sw29wQDAOXd1muEuNThyVwkMd6nhyJy74a4OM9ylhmdG7r491F32XqnBkbtKYLhLDf2+c+7qPsNdajiyWsZ17uoww11qcLWMSmC4Sw3OuasEhrvU4GoZlcDeKzU4clcJDHepoV99oOqcu7rMcJcaen1H7uo+w11qcLWMSmC4Sw1+n7tKYLhLDa6WUQla9d6I2BQR+yJiNiKuGbF/bUTcERHfioj7IuLy8ZcqTYerZVSCBcM9IpYDNwCXARuBbRGxsdHsg8AtmXk+sBX4i3EXKk2Lq2VUgjYj9wuB2czcn5lPATcDWxptEnhudfvngYPjK1GaLkfuKsGKFm3OBR6u3T8AvKLR5kPAP0fEe4GfAy4ZS3XSEnC1jErQZuQ+qodn4/424MbMXA1cDnwuIo46dkRsj4g9EbHn8OHDx1+tNAXPrHP3A1V1V5veewBYU7u/mqOnXa4CbgHIzG8CpwErmwfKzB2ZOZOZM6tWrTqxiqUJOzJydymkOqxNuN8DbIiI9RFxCsMPTHc22vwH8HqAiHgpw3B3aK5Ocs5dJVgw3DOzB1wN7AYeYLgqZm9EXB8Rm6tm7wfeGRHfAW4C3p6ZzakbqRNcLaMStPlAlczcBexqbLuudvt+4FXjLU1aGo7cVQI/MZIaXC2jEhjuUsMzI3ffHuoue6/U4MhdJTDcpQa/z10lMNylhv5gQAQsM9zVYYa71NAbpKN2dZ7hLjX0B+l8uzrPcJcahiN33xrqNnuw1ODIXSUw3KWG3mDgnLs6z3CXGvqDdKWMOs9wlxp6fVfLqPsMd6mhn865q/sMd6mh7zp3FcBwlxp6rpZRAQx3qaHfd527us8eLDU4clcJDHepoT8YsMKLY6vjDHepwZG7SmC4Sw2ullEJDHepwZG7SmC4Sw19vxVSBbAHSw2O3FUCw11q6PutkCqA4S419PqO3NV9hrvU0B+k69zVeYa71DC8EpNvDXWbPVhq6LnOXQUw3KUGr6GqEhjuUoPXUFUJDHepwZG7StAq3CNiU0Tsi4jZiLhmnjZviYj7I2JvRHx+vGVK0+Ocu0qwYqEGEbEcuAH4deAAcE9E7MzM+2ttNgAfAF6VmY9FxPMnVbA0af2+q2XUfW168IXAbGbuz8yngJuBLY027wRuyMzHADLz0HjLlKan5zp3FaBNuJ8LPFy7f6DaVvdi4MUR8Y2IuCsiNo06UERsj4g9EbHn8OHDJ1axNGHOuasEbcJ9VC/Pxv0VwAbgYmAb8OmIOOOoB2XuyMyZzJxZtWrV8dYqTYWrZVSCNuF+AFhTu78aODiizZcz8+nM/CGwj2HYS50yGCSDxJG7Oq9NuN8DbIiI9RFxCrAV2Nlo8/fAawEiYiXDaZr94yxUmoZ+Dn8pdeSurlsw3DOzB1wN7AYeAG7JzL0RcX1EbK6a7QYejYj7gTuA38/MRydVtDQp/cEw3F0to65bcCkkQGbuAnY1tl1Xu53A+6ofqbN6A0fuKoPDE6mm358buRvu6jbDXarpDQYArnNX5xnuUs0zc+6Gu7rNcJdqnHNXKQx3qcbVMiqFPViqceSuUhjuUk2/+kDVOXd1neEu1ThyVykMd6mm5zp3FcJwl2rmPlB1nbu6znCXanqullEh7MFSTd85dxXCcJdqeq6WUSEMd6nGkbtKYbhLNT2/W0aFMNylmrmv/F3hB6rqOHuwVOPIXaUw3KUa17mrFIa7VONqGZXCcJdqXC2jUhjuUo1z7iqF4S7VPDNy962hbrMHSzVzI3ezXV1nF5Zq+v3hB6qO3NV19mCpxjl3lcJwl2oG6WoZlcFwl2ocuasUhrtU88x3yxju6jbDXapx5K5SGO5STX+QLF8WRBju6rZW4R4RmyJiX0TMRsQ1x2h3RURkRMyMr0RpenpVuEtdt2C4R8Ry4AbgMmAjsC0iNo5o9xzgd4G7x12kNC39wcD5dhWhzcj9QmA2M/dn5lPAzcCWEe3+CPgI8LMx1idNlSN3laJNuJ8LPFy7f6DadkREnA+sycx/HGNt0tT1B+nIXUVoE+6jenoe2RmxDPgY8P4FDxSxPSL2RMSew4cPt69SmpLhyN11Buq+Nr34ALCmdn81cLB2/znALwN3RsS/AxcBO0d9qJqZOzJzJjNnVq1adeJVSxPS7ztyVxnahPs9wIaIWB8RpwBbgZ1zOzPz8cxcmZnrMnMdcBewOTP3TKRiaYKcc1cpFgz3zOwBVwO7gQeAWzJzb0RcHxGbJ12gNE39wcDrp6oIK9o0ysxdwK7GtuvmaXvx4suSloYjd5XCT46kGlfLqBSGu1TjahmVwl4s1ThyVykMd6nGOXeVwnCXavxuGZXCcJdqen1H7iqD4S7V9AfpOncVwXCXalwto1LYi6UaV8uoFIa7VONqGZXCcJdqXC2jUhjuUo0jd5XCcJdqnHNXKQx3qWa4zt23hbrPXizVOHJXKQx3qaY3SJb7R0wqgOEu1bhaRqUw3KUaV8uoFIa7VOOcu0phuEs1freMSmEvlmocuasUhrtUyUz6zrmrEIa7VOkPEsCRu4pguEuVXhXurnNXCQx3qeLIXSUx3KXKkZG7q2VUAHuxVHHkrpIY7lKlNxgAuFpGRTDcpYojd5XEcJcqvf7cnLvhru4z3KXKkZG7SyFVgFbhHhGbImJfRMxGxDUj9r8vIu6PiPsi4msR8aLxlypNlqtlVJIFe3FELAduAC4DNgLbImJjo9m3gJnM/FXgVuAj4y5UmjTn3FWSNkOUC4HZzNyfmU8BNwNb6g0y847MfKK6exewerxlSpPnahmVpE24nws8XLt/oNo2n6uAr4zaERHbI2JPROw5fPhw+yqlKXDkrpK0CfdRPT1HNox4KzADfHTU/szckZkzmTmzatWq9lVKU/DMnLvhru5b0aLNAWBN7f5q4GCzUURcAlwLvCYznxxPedL09A13FaTNyP0eYENErI+IU4CtwM56g4g4H/gksDkzD42/TGnyXOeukiwY7pnZA64GdgMPALdk5t6IuD4iNlfNPgqcDnwxIr4dETvnOZx00hrk3Jy7SyHVfW2mZcjMXcCuxrbrarcvGXNd0tQ5566SOESRKv1qKaSrZVQCw12qOOeukhjuUsXvllFJDHep0vOPmFQQw12q9P3iMBXEXixVHLmrJIa7VOn7xWEqiOEuVRy5qySGu1Txu2VUEsNdqsytc/frB1QCe7FUOTJyd527CmC4SxXn3FUSw12quFpGJTHcpcqRb4UMw13dZ7hLlf4gWRawzJG7CmC4S5XeIF0po2LYk6VKf5DOt6sYhrtU6fXTlTIqhuEuVfqDgWvcVQzDXaoM59wNd5XBcJcqzrmrJIa7VHG1jEpiT5YqjtxVEsNdqjjnrpIY7lKlPxg4clcxDHep0us7LaNyGO5SpT9IVrjOXYUw3KVKb5Asd7WMCmFPlip9P1BVQQx3qdLzA1UVpFW4R8SmiNgXEbMRcc2I/adGxBeq/XdHxLpxFypNmiN3lWTBcI+I5cANwGXARmBbRGxsNLsKeCwzfxH4GPDhcRcqTVrPP2JSQdqM3C8EZjNzf2Y+BdwMbGm02QL8TXX7VuD1EV6rTN3iyF0lWdGizbnAw7X7B4BXzNcmM3sR8TjwPODH4yiy7pZ7HuZT/7p/3IeVeOgnT/DqDSuXugxpLNqE+6ihTJ5AGyJiO7AdYO3atS2e+mhnPPtZbDj79BN6rHQsG84+nTddsHqpy5DGok24HwDW1O6vBg7O0+ZARKwAfh74SfNAmbkD2AEwMzNzVPi3cenLXsClL3vBiTxUkv7faDPnfg+wISLWR8QpwFZgZ6PNTuDK6vYVwO2ZeULhLUlavAVH7tUc+tXAbmA58JnM3BsR1wN7MnMn8FfA5yJiluGIfeski5YkHVubaRkycxewq7HtutrtnwFvHm9pkqQT5V+oSlKBDHdJKpDhLkkFMtwlqUCGuyQVKJZqOXpEHAYeOsGHr2QCX20wBtZ1fKzr+J2stVnX8VlMXS/KzFULNVqycF+MiNiTmTNLXUeTdR0f6zp+J2tt1nV8plGX0zKSVCDDXZIK1NVw37HUBczDuo6PdR2/k7U26zo+E6+rk3PukqRj6+rIXZJ0DCdtuEfEmyNib0QMImLeT5Xnu3h39RXFd0fEg9XFu08ZU11nRcRt1XFvi4gzR7R5bUR8u/bzs4h4Y7Xvxoj4YW3fy6dVV9WuX3vunbXtS3m+Xh4R36xe7/si4jdr+8Z6vhZzsfeI+EC1fV9EvGExdZxAXe+LiPur8/O1iHhRbd/I13RKdb09Ig7Xnv+3a/uurF73ByPiyuZjJ1zXx2o1fT8iflrbN8nz9ZmIOBQR35tnf0TEx6u674uIC2r7xnu+MvOk/AFeCrwEuBOYmafNcuAHwHnAKcB3gI3VvluArdXtTwDvHlNdHwGuqW5fA3x4gfZnMfwa5GdX928ErpjA+WpVF/A/82xfsvMFvBjYUN1+IfAIcMa4z9ex+kutze8An6hubwW+UN3eWLU/FVhfHWf5FOt6ba0PvXuurmO9plOq6+3An4947FnA/urfM6vbZ06rrkb79zL8qvKJnq/q2K8GLgC+N8/+y4GvMLx63UXA3ZM6XyftyD0zH8jMfQs0G3nx7ogI4HUML9YNw4t3v3FMpdUvBt7muFcAX8nMJ8b0/PM53rqOWOrzlZnfz8wHq9sHgUPAgn+kcQIWc7H3LcDNmflkZv4QmK2ON5W6MvOOWh+6i+EV0SatzfmazxuA2zLzJ5n5GHAbsGmJ6toG3DSm5z6mzPw6I65CV7MF+GwO3QWcERHnMIHzddKGe0ujLt59LsOLc/80M3uN7eNwdmY+AlD9+/wF2m/l6I71x9WvZB+LiFOnXNdpEbEnIu6amyriJDpfEXEhw9HYD2qbx3W+5usvI9tU52PuYu9tHjvJuuquYjj6mzPqNZ1mXW+qXp9bI2Lukpwnxfmqpq/WA7fXNk/qfLUxX+1jP1+tLtYxKRHxVWDUBVGvzcwvtznEiG15jO2LrqvtMarjnAP8CsOrWM35APBfDANsB/AHwPVTrGttZh6MiPOA2yPiu8B/j2i3VOfrc8CVmTmoNp/w+Rr1FCO2tb3Y+6L61AJaHzsi3grMAK+pbT7qNc3MH4x6/ATq+gfgpsx8MiLexfC3nte1fOwk65qzFbg1M/u1bZM6X21MrX8tabhn5iWLPMR8F+/+McNfd1ZUo69RF/U+oboi4kcRcU5mPlKF0aFjHOotwJcy8+nasR+pbj4ZEX8N/N4066qmPcjM/RFxJ3A+8Hcs8fmKiOcC/wR8sPp1de7YJ3y+RljMxd7bPHaSdRERlzD8H+ZrMvPJue3zvKbjCKsF68rMR2t3PwV8uPbYixuPvXMMNbWqq2Yr8J76hgmerzbmq33s56vr0zIjL96dw08o7mA43w3Di3e3+U2gjfrFwBc67lFzfVXAzc1zvxEY+an6JOqKiDPnpjUiYiXwKuD+pT5f1Wv3JYZzkV9s7Bvn+VrMxd53AltjuJpmPbAB+LdF1HJcdUXE+cAngc2Zeai2feRrOsW6zqnd3Qw8UN3eDVxa1XcmcCn/9zfYidZV1fYShh9OfrO2bZLnq42dwNuqVTMXAY9XA5jxn69JfWq82B/gNxj+3+xJ4EfA7mr7C4FdtXaXA99n+H/ea2vbz2P45psFvgicOqa6ngd8DXiw+vesavsM8Olau3XAfwLLGo+/Hfguw5D6W+D0adUFvLJ67u9U/151Mpwv4K3A08C3az8vn8T5GtVfGE7zbK5un1b9989W5+O82mOvrR63D7hszP19obq+Wr0P5s7PzoVe0ynV9SfA3ur57wB+qfbY36rO4yzwjmnWVd3/EPCnjcdN+nzdxHC119MM8+sq4F3Au6r9AdxQ1f1daisBx32+/AtVSSpQ16dlJEkjGO6SVCDDXZIKZLhLUoEMd0kqkOEuSQUy3CWpQIa7JBXofwHQlUcHmc6HNQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "z = np.linspace(-1,1,100)\n",
    "plt.plot(z,np.heaviside(z,0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Activation function needs to be differentiable!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](http://neuralnetworksanddeeplearning.com/images/tikz8.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\mathcal{A}(w \\cdot x+b)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{eqnarray} \n",
    "  \\frac{1}{1+\\exp(-\\sum_j w_j x_j-b)}.\n",
    "\\tag{4}\\end{eqnarray}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f5f9160ea90>]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xt81PWd7/HXZyY3buGWcA8Cggpewahs3fVSFIF2UXux2Hbb2p7a7q677enunmO3+3D7sN3zaO3j9Jz2rN3Wtrb2omjbtWUVUeql2lYUKPcAEhFIAiThlhBynZnP+WMGHeMEhjCT38zk/Xw88shvfvOd5J1fhje/fOc3v5+5OyIiUlhCQQcQEZHMU7mLiBQglbuISAFSuYuIFCCVu4hIAVK5i4gUIJW7iEgBUrmLiBQglbuISAEqCuobV1RU+LRp04L69iIieWn9+vWH3L3ydOMCK/dp06axbt26oL69iEheMrO96YzTtIyISAFSuYuIFCCVu4hIAVK5i4gUIJW7iEgBOm25m9mDZtZkZlv7uN/M7NtmVmtmm81sXuZjiojImUhnz/3HwKJT3L8YmJX4uBP4j7OPJSIiZ+O0x7m7+4tmNu0UQ24GfuLx6/WtMbNRZjbR3Q9kKKOIFCB3JxJzuiIxuiMxuiJReiJOdzRKd8SJxGL0RJ1INEY05vTEnGgsRjTGW5/dicWcmDvRmOMOMXdiic/+tuX45/j3TqxLLAPEb711+2TGt+5/59je49/28739h33bfQtmj+fSqlH923BpysSbmCYDdUm36xPr3lHuZnYn8b17pk6dmoFvLSJBiURjHD7RzaG2Lo6c6ObIiW6OnuimpSNCS0cPxzt7aOuKcLwzQltXhI7uKO098c8d3VE6I/HSHizM3loeV16WF+VuKdal/I25+wPAAwDV1dWD57cqkoeiMWf/sQ52HzrBG81t1B3toOFoBw3HOjjY2snhti766uZhJWFGDilmeFkRw0uLGFFWxPjyUoaVFFFWEmZIcfyjtChEaXGI0qIwxeEQJUXxj+KQURwOURROfA4ZRWEjHAoRNiMcOvkBITNCiXWhkGFAOGSYgRFfb8TL1ezk+vjjTo6xXi12cv1byyfXW9Jy8vhUNRisTJR7PVCVdHsKsD8DX1dEBkhXJMrWhlY21R1j+4FWdhw8zmuNx+mKxN4cU1YcYvKoIUwePZQ5E8sZX15KZXkZlcNLGDOslDHDShg9tJjyIcUUh3UgXtAyUe4rgLvMbDlwFdCi+XaR3NYVibJ+71F+v+sQL+8+zLaGVrqj8SKvGF7K7Ikj+Kv55zBz3HCmVwxjeuUwKoeX5uQeqqR22nI3s0eA64AKM6sH/hUoBnD37wIrgSVALdAO3JGtsCLSf62dPTy3vYmVWw7w4q5mOntihEPGZVWjuOPqacydOpp5U0cxrrws6KiSAekcLXP7ae534G8zlkhEMiYWc16qPcSja/fx25omuqMxJpSXcVt1FdfMquSqGWMYUVYcdEzJgsBO+Ssi2dPWFeHna/byk5f30nCsg9FDi/no/HN4zyUTmVs1ilBI0yuFTuUuUkCOtXfz4B/28NAf99DS0cP8GWO4e/EFLLxwPKVF4aDjyQBSuYsUgJ5ojJ+t2cv//e0uWjp6WDhnPH9z/Uwuy/Kx1JK7VO4iee6lXc3864pt7G4+wZ/PrOBL75nN7InlQceSgKncRfJUe3eE/7VyOz9bs4/pFcP44cerefcF43S4ogAqd5G8tH7vUb7w2Eb2HWnnv/35dP7xpvMpK9acurxF5S6SZx55dR/3/GYr48vLeOTT85k/Y2zQkSQHqdxF8kRPNMZXn6jhoZf3cs15lfy/2+cycoiOUZfUVO4ieaC9O8Jnfrqel3Yd4tN/MZ27F88mrGPV5RRU7iI5rq0rwid/tJZ1e49w3/sv4bYrqk7/IBn0VO4iOaylo4dP/OhVNte38O3b5/LeSyYFHUnyhMpdJEed6IrwsR++Qs2BVr7zkXncdOGEoCNJHlG5i+SgSDTG3z2ygS0NLXzvr6q5cc74oCNJnlG5i+QYd+dfV2zjuR1N/NutF6nYpV90uRSRHPO9F3fz81f28dlrz+UjV50TdBzJUyp3kRzy0q5mvr5qB++9ZCL/46bzg44jeUzlLpIjmlo7+e+PbmRm5XC+8YFLdc51OSuacxfJAdGY87nlG2nrivDwp+czpETniZGzo3IXyQH//lwtL+8+zH3vv4Tzxo8IOo4UAE3LiARsY90xvvXsa9xy2SQ+WD0l6DhSIFTuIgHqica4+1ebqRxRyr23XKRzsUvGaFpGJEAPvLibHQeP88BfXU55mc7wKJmjPXeRgOxubuNbz+5iycUTWKhTC0iGqdxFAuDufPE/t1BWFOLLSy8MOo4UIJW7SABWbNrPK28c4Z+XzGbciLKg40gBUrmLDLDOnij3rdrJnInl3Fatc7NLdqjcRQbYj/+4h4ZjHfzLe2brXaiSNSp3kQF0uK2L+5+rZcEF43jXzIqg40gBU7mLDKBvP7uL9p4oX1xyQdBRpMCp3EUGyBuHTvDzV/ax7IoqZo7TKQYku9IqdzNbZGY7zazWzO5Ocf9UM3vezDaY2WYzW5L5qCL57f7nawmHjM/dMCvoKDIInLbczSwM3A8sBuYAt5vZnF7D/gV4zN3nAsuA72Q6qEg+qzvSzuMbGvjwVVN16KMMiHT23K8Eat19t7t3A8uBm3uNcaA8sTwS2J+5iCL57zsv1BI24zPXnBt0FBkk0jm3zGSgLul2PXBVrzFfBp4xs78DhgE3ZCSdSAFoONbBL9fX86ErqpgwUnvtMjDS2XNPdSCu97p9O/Bjd58CLAF+ambv+NpmdqeZrTOzdc3NzWeeViQPfe93r+MOn71We+0ycNIp93og+W10U3jntMungMcA3P1loAx4x0G87v6Au1e7e3VlZWX/EovkkabWTpavreMDl09hyuihQceRQSSdcl8LzDKz6WZWQvwF0xW9xuwDFgCY2Wzi5a5dcxn0Hnp5Dz3RGH99nfbaZWCdttzdPQLcBTwNbCd+VMw2M7vXzJYmhv0D8Gkz2wQ8AnzC3XtP3YgMKp09UR5+ZR83zh7POWOHBR1HBpm0Ltbh7iuBlb3W3ZO0XANcndloIvnt1xsaONrewx1XTw86igxCeoeqSBa4Ow/+4Q1mTyxn/owxQceRQUjlLpIFf3z9MK81tnHH1dN0XVQJhMpdJAse/P0bjB1WwtJLJwUdRQYplbtIhu05dILndjbxkaumUlYcDjqODFIqd5EMe/jVfYTN+Oj8c4KOIoOYyl0kg7ojMX61vp4Fs8cxrlynGpDgqNxFMmh1TSOHT3Sz7MqpQUeRQU7lLpJBy9fuY/KoIVwzS6fXkGCp3EUypO5IOy/tOsQHq6cQ1oWvJWAqd5EMeWxdHWZwW3XV6QeLZJnKXSQDItEYj62r49rzKpk0akjQcURU7iKZ8LvXmmls7WLZFXohVXKDyl0kA371p3rGDithwexxQUcRAVTuImetpaOH325v4i8vnURxWP+kJDfomShylp7acoDuSIxb504OOorIm1TuImfp8Q0NzKgYxiVTRgYdReRNKneRs1B/tJ1X3jjCrXMn69S+klNU7iJn4Tcb49eKv0VTMpJjVO4i/eTuPL6hgSumjaZqzNCg44i8jcpdpJ+27W+ltqlNe+2Sk1TuIv306w0NFIeN91w8MegoIu+gchfph1jMeXLLAa6ZVcmooSVBxxF5B5W7SD9sqDvKgZZO3nup9tolN6ncRfrhic0HKCkKccPs8UFHEUlJ5S5yhmIxZ+WWA1x7XiUjyoqDjiOSkspd5Ayt23uUxtYu3nuJpmQkd6ncRc7Qk5v3U1oUYoGmZCSHqdxFzkA05qzcepB3XzCO4aVFQccR6ZPKXeQMvPrGEZqPd/EeTclIjlO5i5yBJ7fsp6w4xLsv0EU5JLep3EXSFIs5T29r5PrzxzG0RFMyktvSKnczW2RmO82s1szu7mPMbWZWY2bbzOzhzMYUCd6GuqM0H+9i0UUTgo4iclqn3f0wszBwP3AjUA+sNbMV7l6TNGYW8EXganc/amb6m1UKzqqtBykJa0pG8kM6e+5XArXuvtvdu4HlwM29xnwauN/djwK4e1NmY4oEy91Zte0gV88cqzcuSV5Ip9wnA3VJt+sT65KdB5xnZn8wszVmtijVFzKzO81snZmta25u7l9ikQDUHGil7kiHpmQkb6RT7qmuHea9bhcBs4DrgNuBH5jZqHc8yP0Bd6929+rKysozzSoSmFVbDxIydC4ZyRvplHs9UJV0ewqwP8WY37h7j7u/AewkXvYiBWHV1oNcNX0sY4eXBh1FJC3plPtaYJaZTTezEmAZsKLXmF8D1wOYWQXxaZrdmQwqEpTapjZ2NbVpSkbyymnL3d0jwF3A08B24DF332Zm95rZ0sSwp4HDZlYDPA/8k7sfzlZokYH09LaDACy8UFMykj/SeieGu68EVvZad0/SsgNfSHyIFJRnth3k0qpRTBw5JOgoImnTO1RFTuFgSyeb6lu4SXvtkmdU7iKnsHp7IwAL56jcJb+o3EVO4ZltB5lRMYxzK4cHHUXkjKjcRfrQ2tnDmt2HuXHOeMxSvd1DJHep3EX68MLOZnqirqNkJC+p3EX68My2g1QML+WyqtFBRxE5Yyp3kRS6IlFe2NnMDbPHEQ5pSkbyj8pdJIU1u4/Q1hXRlIzkLZW7SArPbDvI0JIw7zq3IugoIv2ichfpJRZzVtc0cu15lZQVh4OOI9IvKneRXrY0tNB0vIsb9cYlyWMqd5FeVtc0Eg6ZLqcneU3lLtLL6ppGrpg2mlFDS4KOItJvKneRJPsOt7Oz8Tg3ztG52yW/qdxFkjxTkzh3u+bbJc+p3EWSrK5p5IIJI6gaMzToKCJnReUuknD0RDdr9xzRUTJSEFTuIgnP7Wgi5qjcpSCo3EUSVtc0MqG8jIsnjww6ishZU7mLAJ09UV7c1cwNc8bp3O1SEFTuIsAfXz9Ee3eUhToEUgqEyl0EeGZbIyNKi5g/Y2zQUUQyQuUug1405vx2eyPXXTCOkiL9k5DCoGeyDHob9h3lUFu33rgkBUXlLoPe6ppGisPGdedXBh1FJGNU7jKouTtPbzvIn51bwYiy4qDjiGSMyl0GtdqmNvYcbteUjBQclbsMas/UNAJ6V6oUHpW7DGrP1DRyadUoxpeXBR1FJKNU7jJo7T/Wwaa6Y5qSkYKUVrmb2SIz22lmtWZ29ynGfcDM3MyqMxdRJDue2RY/d/vii/SuVCk8py13MwsD9wOLgTnA7WY2J8W4EcDfA69kOqRINqzadpDzxg9nRuXwoKOIZFw6e+5XArXuvtvdu4HlwM0pxn0FuA/ozGA+kaw43NbFq28cYdGF2muXwpROuU8G6pJu1yfWvcnM5gJV7v5EBrOJZM1vtzcSc7hJUzJSoNIp91TnP/U37zQLAf8H+IfTfiGzO81snZmta25uTj+lSIat2nqQqWOGMmdiedBRRLIinXKvB6qSbk8B9ifdHgFcBLxgZnuA+cCKVC+quvsD7l7t7tWVlXqrtwSjtbOH39ceYtFFE3TudilY6ZT7WmCWmU03sxJgGbDi5J3u3uLuFe4+zd2nAWuApe6+LiuJRc7S8zua6Ik6N2m+XQrYacvd3SPAXcDTwHbgMXffZmb3mtnSbAcUybRVWw8ybkQpc6tGBR1FJGuK0hnk7iuBlb3W3dPH2OvOPpZIdrR3R3hhZzMfuHwKoZCmZKRw6R2qMqg8v6OZjp4oSy6eGHQUkaxSucug8sTm/VSOKOXK6WOCjiKSVSp3GTROdEV4bkcTSy6aQFhTMlLgVO4yaDy7o4muSIz3XDIp6CgiWadyl0HjiU37GV9eSvU5o4OOIpJ1KncZFI539vDCa80suXiijpKRQUHlLoPCs9ub6I7EeO8lOkpGBgeVuwwKT2zez6SRZcyt0pSMDA4qdyl4Le09vPjaIRZrSkYGEZW7FLyVWw/QHY1xy2WTTz9YpECo3KXgPf6nBs6tHMZFk3V6Xxk8VO5S0OqOtPPqniO8b94Und5XBhWVuxS032xsAGDppXrjkgwuKncpWO7O4xsauHLaGKrGDA06jsiAUrlLwdrS0MLrzSe4dZ5eSJXBR+UuBevxDQ2UhEMsuUhvXJLBR+UuBSkSjfFfm/azYPY4Rg4tDjqOyIBTuUtBem5HE4faurl1rqZkZHBSuUtBenRtHZUjSrn+gnFBRxEJhMpdCs6Blg6e39nEBy+fQnFYT3EZnPTMl4Lzi3X1xBw+dEVV0FFEAqNyl4ISizmPrq3j6pljOWfssKDjiARG5S4F5aXaQzQc62DZFVODjiISKJW7FJRH1+5j9NBiFl44PugoIoFSuUvBaD7exeqaRt43bwqlReGg44gESuUuBePhV/bRE3U+fJWmZERU7lIQuiMxfvbKXq47v5JzK4cHHUckcCp3KQhPbtlP8/Eu7rh6etBRRHKCyl3ynrvz4O/3MHPccK6ZVRF0HJGcoHKXvLd+71G2NLTwiXdN09WWRBJU7pL3fvSHPYwcUsz7dN52kTelVe5mtsjMdppZrZndneL+L5hZjZltNrNnzeyczEcVeaeGYx2s2naQZVdWMbSkKOg4IjnjtOVuZmHgfmAxMAe43czm9Bq2Aah290uAXwL3ZTqoSCrf+93rhAw+/mfTgo4iklPS2XO/Eqh1993u3g0sB25OHuDuz7t7e+LmGmBKZmOKvFNjayfL19bxgcunMGnUkKDjiOSUdMp9MlCXdLs+sa4vnwKeSnWHmd1pZuvMbF1zc3P6KUVS+N7vdhONOX997cygo4jknHTKPdXhB55yoNlHgWrgG6nud/cH3L3a3asrKyvTTynSy6G2Lh5+dS+3XDaZqWOHBh1HJOek8wpUPZB8YuwpwP7eg8zsBuBLwLXu3pWZeCKpff+l3XRHYvzt9ecGHUUkJ6Wz574WmGVm082sBFgGrEgeYGZzge8BS929KfMxRd5y9EQ3P315L3956SRm6FQDIimdttzdPQLcBTwNbAcec/dtZnavmS1NDPsGMBz4hZltNLMVfXw5kbN2//O1dPREuet6zbWL9CWtA4PdfSWwste6e5KWb8hwLpGU9h4+wUMv7+G2y6uYNX5E0HFEcpbeoSp55b5VOykKhfjCwvOCjiKS01TukjfW7z3Ck1sO8JlrZzC+vCzoOCI5TeUuecHd+eqT2xk3opQ7r5kRdByRnKdyl7ywYtN+Nuw7xj8uPF/nkBFJg8pdct6x9m6+8kQNl0wZyfsv15ktRNKhXSDJef/25HaOtvfwk09eRTik87WLpEN77pLTfr/rEL9YX89nrpnBnEnlQccRyRsqd8lZHd1R/vnxLUyvGMbfL5gVdByRvKJpGclZX3tqO/uOtLP8zvmUFYeDjiOSV7TnLjlp1dYDPPTyXj559XTmzxgbdByRvKNyl5xTd6Sdf/rlZi6dMpK7F18QdByRvKRyl5zSHYlx1yMbAPj3D8+jpEhPUZH+0Jy75Ax35ytP1LCp7hj/8ZF5VI3RRThE+ku7RZIzfvj7N/jpmr3cec0MFl88Meg4InlN5S45YeWWA3z1ye0suXgCdy/SPLvI2VK5S+DW7TnC5x/dyOXnjOabt11GSO9CFTlrKncJ1No9R/jEj9YyedQQvv+xah3PLpIhKncJzB9fP8THfvgq48pLeeTT8xkzrCToSCIFQ+UugXhhZxN3/GgtU0YPYfmd85kwUhffEMkkHQopA8rd+dEf9vDVJ2s4f0I5P/vUlYwdXhp0LJGCo3KXAdMVifIvj2/lF+vrWThnPN/80GUML9VTUCQb9C9LBsTrzW184dGNbKpv4e/fPZPP33CejooRySKVu2RVLOY89PIevvbUDoaUhPnuR+ex6CK9QUkk21TukjU1+1v58n9t49U3jnD9+ZV8/f2XMK5cL5yKDASVu2Rc8/Euvrl6J8vX1jFySDFfe9/FfOiKKsw0DSMyUFTukjEHWzr5wUu7efjVfXRHYtzxrul8bsEsRg4tDjqayKCjcpez4u5saWjh52v28fiGBqLuLL10Ene9eybnVg4POp7IoKVyl35pOt7JU1sO8ujaOmoOtFJWHOKD1VP47LXn6lS9IjlA5S5pcXdeb27jd68dYtXWA6zbexR3uHBSOV+55SKWXjqJkUM0/SKSK1TuklIs5uxqauNP+46ybs9R/lB7iIOtnQBcMGEEn1swi8UXTeT8CSMCTioiqaRV7ma2CPgWEAZ+4O5f63V/KfAT4HLgMPAhd9+T2aiSDe5Oc1sXbzSf4PXmE+w42Mr2A61sP3Cctq4IAKOHFvOucyu4emYFfzGrQtMuInngtOVuZmHgfuBGoB5Ya2Yr3L0madingKPuPtPMlgFfBz6UjcCSvmjMOdrezZET3Rxq66KptYvG1k4OtHTScKyD+qMd1B9p53iixAGGlxZxwYQR3Dp3MpdVjWLeOaOZNnaoDmMUyTPp7LlfCdS6+24AM1sO3Awkl/vNwJcTy78E/t3MzN09g1nzmrsTjTnRk58TH5GYE4k6PdFYYjlGVyRGTzRGdyRGd+JzVyRGZ0+Uzp4YHT1ROrojtHdHae+O0tYVoa0zQltXhNbOHo6199DS0UNrZw+pfgPDSsJMGT2UyaOHcMW00UyvGMaMyuHMqBjGlNFDVOQiBSCdcp8M1CXdrgeu6muMu0fMrAUYCxzKRMhkj62t44GXdr95u6//P7yPGycX3T1pGU7ecudthZhqXOzNMfHlmDve63PMnVgsvhxNrM+0opAxpCTMiNIihpcVMby0iDHDSpheMYyRQ4oZNbSEscNKGDOshLHDSxhfXsb48jKdrEtkEEjnX3mq3bjeVZXOGMzsTuBOgKlTp6bxrd9p9LASzh/f60W8PnY0k1cn743am+uSl+2t8QYnb50cc/LhhhEKJZYMwmZvjgmFjFDi64RDhpkRsvhyyIxwKOnDjKKwURQywqEQRWGjOGwUhUKUFIUoCYcoDocoLQ5RWhRfN6Q4TFlxmLKiMENKwpQU6XT8IpJaOuVeD1Ql3Z4C7O9jTL2ZFQEjgSO9v5C7PwA8AFBdXd2vfdkb54znxjnj+/NQEZFBI51dv7XALDObbmYlwDJgRa8xK4CPJ5Y/ADyn+XYRkeCcds89MYd+F/A08UMhH3T3bWZ2L7DO3VcAPwR+ama1xPfYl2UztIiInFpar6y5+0pgZa919yQtdwIfzGw0ERHpL70iJyJSgFTuIiIFSOUuIlKAVO4iIgVI5S4iUoAsqMPRzawZ2NvPh1eQhVMbZIBynRnlOnO5mk25zszZ5DrH3StPNyiwcj8bZrbO3auDztGbcp0Z5TpzuZpNuc7MQOTStIyISAFSuYuIFKB8LfcHgg7QB+U6M8p15nI1m3Kdmaznyss5dxERObV83XMXEZFTyNlyN7MPmtk2M4uZWXWv+75oZrVmttPMburj8dPN7BUz22VmjyZOV5zpjI+a2cbExx4z29jHuD1mtiUxbl2mc6T4fl82s4akbEv6GLcosQ1rzezuAcj1DTPbYWabzexxMxvVx7gB2V6n+/nNrDTxO65NPJemZStL0vesMrPnzWx74vn/uRRjrjOzlqTf7z2pvlYWsp3y92Jx305sr81mNm8AMp2ftB02mlmrmX2+15gB215m9qCZNZnZ1qR1Y8xsdaKLVpvZ6D4e+/HEmF1m9vFUY86Iu+fkBzAbOB94AahOWj8H2ASUAtOB14Fwisc/BixLLH8X+Oss5/3fwD193LcHqBjAbfdl4B9PMyac2HYzgJLENp2T5VwLgaLE8teBrwe1vdL5+YG/Ab6bWF4GPDoAv7uJwLzE8gjgtRS5rgOeGKjnU7q/F2AJ8BTxC5PNB14Z4Hxh4CDx48AD2V7ANcA8YGvSuvuAuxPLd6d63gNjgN2Jz6MTy6PPJkvO7rm7+3Z335nirpuB5e7e5e5vALXEL+L9JotfU+/dxC/WDfAQcEu2sia+323AI9n6Hlnw5oXP3b0bOHnh86xx92fcPZK4uYb4Vb2Cks7PfzPx5w7En0sLLMtXD3f3A+7+p8TycWA78WsU54ObgZ943BpglJlNHMDvvwB43d37++bIs+buL/LOq9AlP4/66qKbgNXufsTdjwKrgUVnkyVny/0UUl2wu/eTfyxwLKlIUo3JpL8AGt19Vx/3O/CMma1PXEd2INyV+NP4wT7+DExnO2bTJ4nv5aUyENsrnZ//bRd+B05e+H1AJKaB5gKvpLj7z8xsk5k9ZWYXDlCk0/1egn5OLaPvHawgttdJ4939AMT/8wbGpRiT8W2X1sU6ssXMfgtMSHHXl9z9N309LMW6fl2wOx1pZrydU++1X+3u+81sHLDazHYk/ofvt1PlAv4D+Arxn/krxKeMPtn7S6R47FkfOpXO9jKzLwER4Od9fJmMb69UUVOsy9rz6EyZ2XDgV8Dn3b21191/Ij710JZ4PeXXwKwBiHW630uQ26sEWAp8McXdQW2vM5HxbRdoubv7Df14WDoX7D5E/E/CosQeV6oxGclo8QuCvw+4/BRfY3/ic5OZPU58SuCsyirdbWdm3weeSHFXOtsx47kSLxS9F1jgicnGFF8j49srhYxd+D3TzKyYeLH/3N3/s/f9yWXv7ivN7DtmVuHuWT2HShq/l6w8p9K0GPiTuzf2viOo7ZWk0cwmuvuBxDRVU4ox9cRfGzhpCvHXG/stH6dlVgDLEkcyTCf+P/CryQMSpfE88Yt1Q/zi3X39JXC2bgB2uHt9qjvNbJiZjTi5TPxFxa2pxmZKr3nOW/v4fulc+DzTuRYB/xNY6u7tfYwZqO2Vkxd+T8zp/xDY7u7f7GPMhJNz/2Z2JfF/x4eznCud38sK4GOJo2bmAy0npyMGQJ9/PQexvXpJfh711UVPAwvNbHRiGnVhYl3/DcQryP35IF5K9UAX0Ag8nXTfl4gf6bATWJy0fiUwKbE8g3jp1wK/AEqzlPPHwGd7rZsErEzKsSnxsY349ES2t91PgS3A5sQTa2LvXInbS4gfjfH6AOWqJT6vuDHx8d3euQZye6X6+YF7if/nA1CWeO7UJp5LMwZgG/058T/jkUmdAAAAk0lEQVTHNydtpyXAZ08+z4C7EttmE/EXpt81ALlS/l565TLg/sT23ELSUW5ZzjaUeFmPTFoXyPYi/h/MAaAn0V+fIv46zbPArsTnMYmx1cAPkh77ycRzrRa442yz6B2qIiIFKB+nZURE5DRU7iIiBUjlLiJSgFTuIiIFSOUuIlKAVO4iIgVI5S4iUoBU7iIiBej/A9JjS7gxYOYDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def sigmoid(z):\n",
    "    return 1/(1+np.exp(-z))\n",
    "\n",
    "z = np.linspace(-10,10,100)\n",
    "plt.plot(z,sigmoid(z))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multilayer perceptrons or MLPs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](http://neuralnetworksanddeeplearning.com/images/tikz11.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The arrows show the NN is feedforward."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Going back to the numbers problem ([MNIST](http://yann.lecun.com/exdb/mnist/))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import mnist\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28) (60000,)\n"
     ]
    }
   ],
   "source": [
    "print(x_train.shape, y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADNdJREFUeJzt3X/MneVdx/H3l+6hSDcVxELXgWwLY0N0MJ7VuS6GjUDAwAp/DNcYrMlCFx2JmMVIMHHsDw0aASEask4aumQUSDYGf7AfWI2IW5AHZAMsv8Q6Cl07BFy3uNIfX/94Tuez9jn3eXp+3af9vl9Jc865r/vc1zcn/TzXOee673NFZiKpnqPaLkBSOwy/VJThl4oy/FJRhl8qyvBLRRl+qSjDLxVl+KWi3jTOzo6OxXkMS8bZpVTKj/kRb+SuWMi+A4U/Ii4EbgYWAX+Xmdc37X8MS/i1OG+QLiU1eDg3LXjfvt/2R8Qi4G+Bi4AzgNURcUa/x5M0XoN85l8BPJ+ZL2TmG8CdwKrhlCVp1AYJ/3LgxTmPt3a2/ZSIWBsRMxExs5tdA3QnaZgGCf98XyocdH1wZq7LzOnMnJ5i8QDdSRqmQcK/FTh5zuO3AS8PVo6kcRkk/I8Ap0XE2yPiaODjwH3DKUvSqPU91ZeZeyLiKuDrzE71rc/Mp4ZWmaSRGmiePzPvB+4fUi2SxsjTe6WiDL9UlOGXijL8UlGGXyrK8EtFGX6pKMMvFWX4paIMv1SU4ZeKMvxSUYZfKsrwS0UZfqkowy8VZfilogy/VJThl4oy/FJRhl8qaqxLdOvIE4ubV2F65ub3dm179pJbG597+qYrG9tP+53HGtvVzJFfKsrwS0UZfqkowy8VZfilogy/VJThl4oaaJ4/IrYAO4G9wJ7MnB5GUTp8vPDZ9zW2P33JLV3b9vU49ubzPtfY/lHe3+MIajKMk3w+nJmvDOE4ksbIt/1SUYOGP4FvRMSjEbF2GAVJGo9B3/avzMyXI2Ip8EBEPJ2ZD87dofNHYS3AMRw7YHeShmWgkT8zX+7c7gDuAVbMs8+6zJzOzOkpmi8CkTQ+fYc/IpZExFv23wcuAJ4cVmGSRmuQt/0nAvdExP7j3JGZXxtKVZJGru/wZ+YLQPeLtXVEOOrY5u9pfnXlc2OqRMPmVJ9UlOGXijL8UlGGXyrK8EtFGX6pKH+6W412XvQrje1fecffjKzvW15798iOLUd+qSzDLxVl+KWiDL9UlOGXijL8UlGGXyrKeX41uuOmG3rsMbpfZ7rzlgsa20/gWyPruwJHfqkowy8VZfilogy/VJThl4oy/FJRhl8qynn+4p6/8QON7ae86fHG9t25t+++X9n7v43tJ331xcb2PX33LHDkl8oy/FJRhl8qyvBLRRl+qSjDLxVl+KWies7zR8R64GJgR2ae2dl2PHAXcCqwBbg8M18bXZkaldtWrWts7zWPv499fff9wa/+YWP7u158pO9jq7eFjPy3AxcesO0aYFNmngZs6jyWdBjpGf7MfBB49YDNq4ANnfsbgEuHXJekEev3M/+JmbkNoHO7dHglSRqHkZ/bHxFrgbUAx3DsqLuTtED9jvzbI2IZQOd2R7cdM3NdZk5n5vTUCH/sUdKh6Tf89wFrOvfXAPcOpxxJ49Iz/BGxEfgWcHpEbI2ITwDXA+dHxHPA+Z3Hkg4jPT/zZ+bqLk3nDbkWHYH+dMf7u7a959r/aHxu/78UoIXwDD+pKMMvFWX4paIMv1SU4ZeKMvxSUf509xHu9St+vbH9vUd/s8cRBjsr8+6Z7lN97/pvL9ltkyO/VJThl4oy/FJRhl8qyvBLRRl+qSjDLxXlPP8RbseHdze2H3vU1Ej7f+sDi0Z6fPXPkV8qyvBLRRl+qSjDLxVl+KWiDL9UlOGXinKe/wgQ5/xy17aN536u8blH9fj7PxXN8/T/8uNobD92+67GdrXHkV8qyvBLRRl+qSjDLxVl+KWiDL9UlOGXiuo5zx8R64GLgR2ZeWZn23XAlcD3O7tdm5n3j6pINXvm936ma9vZi/c1Pre5FXZnc/t0j+N/7+ru8/xv/acenWukFjLy3w5cOM/2mzLzrM4/gy8dZnqGPzMfBF4dQy2SxmiQz/xXRcR3ImJ9RBw3tIokjUW/4b8VeCdwFrANuKHbjhGxNiJmImJmN57nLU2KvsKfmdszc29m7gM+D6xo2HddZk5n5vTUgIs+ShqevsIfEcvmPLwMeHI45Ugal4VM9W0EzgVOiIitwGeAcyPiLCCBLcAnR1ijpBHoGf7MXD3P5ttGUIu6WPQLxze2rz7nX8dUycGu+M/5ZoH/3ylXvd61bc+wi9Eh8Qw/qSjDLxVl+KWiDL9UlOGXijL8UlH+dPdhYNtvvbux/Z6lN4+pkoM9e9fpje0nvvTNMVWiQ+XILxVl+KWiDL9UlOGXijL8UlGGXyrK8EtFOc8/ARb9/M81tl905UNjquRglzzz0cb2Zeu/3dje66fB1R5Hfqkowy8VZfilogy/VJThl4oy/FJRhl8qynn+CZCnLGts/+zSf+hxhNH9DX/jz09qbJ/60Usj61uj5cgvFWX4paIMv1SU4ZeKMvxSUYZfKsrwS0X1nOePiJOBLwAnMXt59rrMvDkijgfuAk4FtgCXZ+Zroyv1yPX07/9sY/u+Fq+Kn/r7R1vrW6O1kJF/D/DpzHwP8AHgUxFxBnANsCkzTwM2dR5LOkz0DH9mbsvMxzr3dwKbgeXAKmBDZ7cNwKWjKlLS8B3SZ/6IOBU4G3gYODEzt8HsHwhg6bCLkzQ6Cw5/RLwZ+BJwdWb+4BCetzYiZiJiZje7+qlR0ggsKPwRMcVs8L+YmV/ubN4eEcs67cuAHfM9NzPXZeZ0Zk5PsXgYNUsagp7hj4gAbgM2Z+aNc5ruA9Z07q8B7h1+eZJGZSGX9K4ErgCeiIjHO9uuBa4H7o6ITwDfBT42mhKPfH/xkbta6/tD//bbje3H8+yYKtG49Qx/Zj4ERJfm84ZbjqRx8Qw/qSjDLxVl+KWiDL9UlOGXijL8UlH+dPcYfO/qDza2X7ak+bLZQS7o3bhzeWP7CX+yaGR9a7I58ktFGX6pKMMvFWX4paIMv1SU4ZeKMvxSUc7zj8Hy259q3uGPBjt+0zX5Pefxv715sM512HLkl4oy/FJRhl8qyvBLRRl+qSjDLxVl+KWinOcfg72v/09j+8XLzxno+E2/re/1+OrGkV8qyvBLRRl+qSjDLxVl+KWiDL9UlOGXiuoZ/og4OSL+MSI2R8RTEfEHne3XRcRLEfF4599vjr5cScOykJN89gCfzszHIuItwKMR8UCn7abM/KvRlSdpVHqGPzO3Ads693dGxGageRkYSRPvkD7zR8SpwNnAw51NV0XEdyJifUQc1+U5ayNiJiJmdrNroGIlDc+Cwx8Rbwa+BFydmT8AbgXeCZzF7DuDG+Z7Xmauy8zpzJyeYvEQSpY0DAsKf0RMMRv8L2bmlwEyc3tm7s3MfcDngRWjK1PSsC3k2/4AbgM2Z+aNc7Yvm7PbZcCTwy9P0qgs5Nv+lcAVwBMR8Xhn27XA6og4C0hgC/DJkVQoaSQW8m3/Q0DM03T/8MuRNC6e4ScVZfilogy/VJThl4oy/FJRhl8qyvBLRRl+qSjDLxVl+KWiDL9UlOGXijL8UlGGXyoqMnN8nUV8H/ivOZtOAF4ZWwGHZlJrm9S6wNr6Nczafikzf3EhO441/Ad1HjGTmdOtFdBgUmub1LrA2vrVVm2+7ZeKMvxSUW2Hf13L/TeZ1NomtS6wtn61Ulurn/kltaftkV9SS1oJf0RcGBHPRMTzEXFNGzV0ExFbIuKJzsrDMy3Xsj4idkTEk3O2HR8RD0TEc53beZdJa6m2iVi5uWFl6VZfu0lb8Xrsb/sjYhHwLHA+sBV4BFidmf8+1kK6iIgtwHRmtj4nHBG/AfwQ+EJmntnZ9pfAq5l5fecP53GZ+ccTUtt1wA/bXrm5s6DMsrkrSwOXAr9Li69dQ12X08Lr1sbIvwJ4PjNfyMw3gDuBVS3UMfEy80Hg1QM2rwI2dO5vYPY/z9h1qW0iZOa2zHysc38nsH9l6VZfu4a6WtFG+JcDL855vJXJWvI7gW9ExKMRsbbtYuZxYmfZ9P3Lpy9tuZ4D9Vy5eZwOWFl6Yl67fla8HrY2wj/f6j+TNOWwMjPfB1wEfKrz9lYLs6CVm8dlnpWlJ0K/K14PWxvh3wqcPOfx24CXW6hjXpn5cud2B3APk7f68Pb9i6R2bne0XM9PTNLKzfOtLM0EvHaTtOJ1G+F/BDgtIt4eEUcDHwfua6GOg0TEks4XMUTEEuACJm/14fuANZ37a4B7W6zlp0zKys3dVpam5ddu0la8buUkn85Uxl8Di4D1mflnYy9iHhHxDmZHe5hdxPSONmuLiI3Aucxe9bUd+AzwFeBu4BTgu8DHMnPsX7x1qe1cZt+6/mTl5v2fscdc24eAfwaeAPZ1Nl/L7Ofr1l67hrpW08Lr5hl+UlGe4ScVZfilogy/VJThl4oy/FJRhl8qyvBLRRl+qaj/A79ChJfubvhUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ii = np.random.randint(x_train.shape[0])\n",
    "\n",
    "plt.imshow(x_train[ii])\n",
    "print(y_train[ii])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](http://neuralnetworksanddeeplearning.com/images/tikz12.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([5, 0, 4, ..., 5, 6, 8], dtype=uint8)"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "def vectorized_result(j):\n",
    "    \"\"\"Return a 10-dimensional unit vector with a 1.0 in the jth\n",
    "    position and zeroes elsewhere.  This is used to convert a digit\n",
    "    (0...9) into a corresponding desired output from the neural\n",
    "    network.\"\"\"\n",
    "    e = np.zeros((10, 1))\n",
    "    e[j] = 1.0\n",
    "    return e"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = x_train.reshape(-1,28*28)\n",
    "y_train = np.array([vectorized_result(y) for y in y_train]).reshape(-1,10)\n",
    "x_test = x_test.reshape(-1,28*28)\n",
    "y_test = np.array([vectorized_result(y) for y in y_test]).reshape(-1,10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0.])"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Activation\n",
    "from keras.optimizers import SGD\n",
    "\n",
    "model = Sequential()\n",
    "# Dense(64) is a fully-connected layer with 64 hidden units.\n",
    "# in the first layer, you must specify the expected input data shape:\n",
    "# here, 20-dimensional vectors.\n",
    "model.add(Dense(30, activation='relu', input_dim=28*28))\n",
    "model.add(Dense(10, activation='softmax'))\n",
    "\n",
    "sgd = SGD(lr=0.01)\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=sgd,\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_6\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_15 (Dense)             (None, 30)                23550     \n",
      "_________________________________________________________________\n",
      "dense_16 (Dense)             (None, 10)                310       \n",
      "=================================================================\n",
      "Total params: 23,860\n",
      "Trainable params: 23,860\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "60000/60000 [==============================] - 1s 13us/step - loss: 3.7743 - accuracy: 0.1832\n",
      "Epoch 2/20\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 2.0739 - accuracy: 0.2028\n",
      "Epoch 3/20\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 1.9838 - accuracy: 0.2420\n",
      "Epoch 4/20\n",
      "60000/60000 [==============================] - 1s 13us/step - loss: 1.8870 - accuracy: 0.2756\n",
      "Epoch 5/20\n",
      "60000/60000 [==============================] - 1s 15us/step - loss: 1.8629 - accuracy: 0.2797\n",
      "Epoch 6/20\n",
      "60000/60000 [==============================] - 1s 13us/step - loss: 1.8261 - accuracy: 0.2842\n",
      "Epoch 7/20\n",
      "60000/60000 [==============================] - 1s 15us/step - loss: 1.8118 - accuracy: 0.2886\n",
      "Epoch 8/20\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 1.9157 - accuracy: 0.2519\n",
      "Epoch 9/20\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 2.0178 - accuracy: 0.2118\n",
      "Epoch 10/20\n",
      "60000/60000 [==============================] - 1s 13us/step - loss: 2.0067 - accuracy: 0.2130\n",
      "Epoch 11/20\n",
      "60000/60000 [==============================] - 1s 13us/step - loss: 2.0037 - accuracy: 0.2114\n",
      "Epoch 12/20\n",
      "60000/60000 [==============================] - 1s 13us/step - loss: 1.9997 - accuracy: 0.2115\n",
      "Epoch 13/20\n",
      "60000/60000 [==============================] - 1s 14us/step - loss: 1.9902 - accuracy: 0.2134\n",
      "Epoch 14/20\n",
      "60000/60000 [==============================] - 1s 14us/step - loss: 2.0014 - accuracy: 0.2122\n",
      "Epoch 15/20\n",
      "60000/60000 [==============================] - 1s 14us/step - loss: 1.9629 - accuracy: 0.2324\n",
      "Epoch 16/20\n",
      "60000/60000 [==============================] - 1s 13us/step - loss: 1.8960 - accuracy: 0.2584\n",
      "Epoch 17/20\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 1.8609 - accuracy: 0.2731\n",
      "Epoch 18/20\n",
      "60000/60000 [==============================] - 1s 14us/step - loss: 1.8544 - accuracy: 0.2732\n",
      "Epoch 19/20\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 1.8420 - accuracy: 0.2760\n",
      "Epoch 20/20\n",
      "60000/60000 [==============================] - 1s 13us/step - loss: 1.8386 - accuracy: 0.2772\n",
      "10000/10000 [==============================] - 0s 10us/step\n"
     ]
    }
   ],
   "source": [
    "model.fit(x_train, y_train,\n",
    "          epochs=20,\n",
    "          batch_size=128)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s 9us/step\n"
     ]
    }
   ],
   "source": [
    "score = model.evaluate(x_test, y_test, batch_size=128)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1.8213214612960815, 0.2782999873161316]"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_5\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_layer (InputLayer)     (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "layer-1 (Dense)              (None, 30)                23550     \n",
      "_________________________________________________________________\n",
      "layer-2 (Dense)              (None, 10)                310       \n",
      "=================================================================\n",
      "Total params: 23,860\n",
      "Trainable params: 23,860\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "from keras.models import Model\n",
    "from keras.layers import Input, Dense, Dropout, Activation\n",
    "from keras.optimizers import SGD\n",
    "\n",
    "x_in = Input(shape=(28*28,),name='input_layer')\n",
    "x1 = Dense(30,\n",
    "           activation='relu',\n",
    "           input_dim=28*28,\n",
    "           name='layer-1')(x_in)\n",
    "\n",
    "x2 = Dense(10,\n",
    "           activation='softmax',\n",
    "           name='layer-2')(x1)\n",
    "\n",
    "model = Model(inputs=[x_in],outputs=[x2])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "sgd = SGD(lr=0.01)\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=sgd,\n",
    "              metrics=['accuracy'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 3.7316 - accuracy: 0.1867\n",
      "Epoch 2/20\n",
      "60000/60000 [==============================] - 1s 14us/step - loss: 2.0584 - accuracy: 0.2226\n",
      "Epoch 3/20\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 2.0196 - accuracy: 0.2345\n",
      "Epoch 4/20\n",
      "60000/60000 [==============================] - 1s 13us/step - loss: 1.9786 - accuracy: 0.2463\n",
      "Epoch 5/20\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 1.9436 - accuracy: 0.2570\n",
      "Epoch 6/20\n",
      "60000/60000 [==============================] - 1s 13us/step - loss: 1.9278 - accuracy: 0.2622\n",
      "Epoch 7/20\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 1.9486 - accuracy: 0.2578\n",
      "Epoch 8/20\n",
      "60000/60000 [==============================] - 1s 14us/step - loss: 1.9229 - accuracy: 0.2617\n",
      "Epoch 9/20\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 1.9182 - accuracy: 0.2633\n",
      "Epoch 10/20\n",
      "60000/60000 [==============================] - 1s 13us/step - loss: 1.9314 - accuracy: 0.2610\n",
      "Epoch 11/20\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 1.9088 - accuracy: 0.2654\n",
      "Epoch 12/20\n",
      "60000/60000 [==============================] - 1s 13us/step - loss: 1.9218 - accuracy: 0.2613\n",
      "Epoch 13/20\n",
      "60000/60000 [==============================] - 1s 13us/step - loss: 1.8975 - accuracy: 0.2678\n",
      "Epoch 14/20\n",
      "60000/60000 [==============================] - 1s 11us/step - loss: 1.9002 - accuracy: 0.2670\n",
      "Epoch 15/20\n",
      "60000/60000 [==============================] - 1s 14us/step - loss: 1.9346 - accuracy: 0.2586\n",
      "Epoch 16/20\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 1.9009 - accuracy: 0.2652\n",
      "Epoch 17/20\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 1.9968 - accuracy: 0.2415\n",
      "Epoch 18/20\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 1.8986 - accuracy: 0.2662\n",
      "Epoch 19/20\n",
      "60000/60000 [==============================] - 1s 12us/step - loss: 1.9105 - accuracy: 0.2652\n",
      "Epoch 20/20\n",
      "60000/60000 [==============================] - 1s 15us/step - loss: 1.9095 - accuracy: 0.2633\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(x_train,y_train,epochs=20,batch_size=128)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Variable 'layer-1_2/kernel:0' shape=(784, 30) dtype=float32, numpy=\n",
       " array([[-0.0541045 , -0.03272625,  0.07537361, ...,  0.03840913,\n",
       "          0.07351724,  0.02489997],\n",
       "        [-0.01743929,  0.02480133, -0.03109419, ..., -0.02269403,\n",
       "         -0.07807899, -0.03554087],\n",
       "        [ 0.03046632, -0.08105048, -0.05276664, ...,  0.0129194 ,\n",
       "          0.03348399, -0.00972317],\n",
       "        ...,\n",
       "        [-0.04795606,  0.03138547, -0.06077447, ...,  0.02830656,\n",
       "         -0.03357159,  0.06342835],\n",
       "        [ 0.03630394,  0.00091434,  0.0128255 , ..., -0.08245108,\n",
       "         -0.00340126,  0.07042164],\n",
       "        [ 0.06489775, -0.05820342, -0.00295708, ...,  0.04991683,\n",
       "          0.03642176,  0.00786474]], dtype=float32)>,\n",
       " <tf.Variable 'layer-1_2/bias:0' shape=(30,) dtype=float32, numpy=\n",
       " array([-0.00059162, -0.00031174, -0.0003843 , -0.00026466, -0.00022694,\n",
       "        -0.00040816, -0.00027154, -0.0003972 , -0.00074417, -0.00054189,\n",
       "        -0.00239426, -0.0007254 , -0.00097115, -0.00026469, -0.00278513,\n",
       "        -0.00014769, -0.00034628,  0.0015389 , -0.00126526, -0.00146342,\n",
       "        -0.00101497, -0.000866  , -0.00190585, -0.00074865, -0.00089866,\n",
       "        -0.00021481, -0.00078533, -0.00050606, -0.00187651, -0.00326058],\n",
       "       dtype=float32)>]"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.layers[1].weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f5f935714a8>"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADa9JREFUeJzt3X2MXPV1xvHnib1e4jW0OMTGNQYnhKA4NJBqYxK5rRxRp9AEmSiBYqmWK6UsakGCKmqLLEVBaptSFEJpk0ZyihsT8ZYGKFbipkFWW4pKHS+Id9NCqUtcb72AaW0C+AWf/rHX0QZ2fjvM2531+X4ka2buuXfu0fU+e2f2N3d+jggByOcddTcAoB6EH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUrN7ubM5HozjNNTLXQKpvK4f62AccDPrthV+2+dLuknSLEl/FRHXldY/TkM61+e1s0sABdtia9Prtvyy3/YsSV+TdIGkZZLW2F7W6vMB6K123vMvl/RsRDwXEQcl3SFpdWfaAtBt7YR/saQfTXq8q1r2U2yP2B61PXpIB9rYHYBOaif8U/1R4S3XB0fEhogYjojhAQ22sTsAndRO+HdJWjLp8SmSdrfXDoBeaSf82yWdYfs9tudIulTS5s60BaDbWh7qi4jDtq+U9PeaGOrbGBFPdqwzAF3V1jh/RGyRtKVDvQDoIT7eCyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFJtzdJre6ek/ZLekHQ4IoY70RSA7msr/JWPR8SLHXgeAD3Ey34gqXbDH5J+YPsh2yOdaAhAb7T7sn9FROy2vUDSfbafjoj7J69Q/VIYkaTjNLfN3QHolLbO/BGxu7odl3SPpOVTrLMhIoYjYnhAg+3sDkAHtRx+20O2jz96X9InJD3RqcYAdFc7L/sXSrrH9tHnuS0ivt+RrgB0Xcvhj4jnJJ3dwV4A9BBDfUBShB9IivADSRF+ICnCDyRF+IGkOnFVXwovXfaxhrVT1z5b3Pbp8YXF+sEDA8X64tvL9bm7XmlYO/LIU8VtkRdnfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IinH+Jv3+793WsPaZoZfLG5/e5s5Xlss7D7/asHbTCx9vc+cz1w/HT2tYG7rhZ4rbzt76UKfb6Tuc+YGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKUdEz3Z2gufHuT6vZ/vrpB9/9tyGtRc/VP4deuKO8jF++QMu1ud86H+L9evPurthbdU7Xytu+71X5xXrn5zb+LsC2vVaHCzWtx0YKtZXHneo5X2/73uXF+vvH9ne8nPXaVts1b7YW/6BqnDmB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkpr2e3/ZGSZ+SNB4RZ1XL5ku6U9JSSTslXRIR01zUPrMNfWdbodbec5/Q3ub6i5NXNqz90Yql5X3/U3nOgetXvq+Fjpoz+7UjxfrQY2PF+rvuv6tY//k5jec7mLuzPBdCBs2c+b8p6fw3LbtG0taIOEPS1uoxgBlk2vBHxP2S9r5p8WpJm6r7myRd1OG+AHRZq+/5F0bEmCRVtws61xKAXuj6d/jZHpE0IknHaW63dwegSa2e+ffYXiRJ1e14oxUjYkNEDEfE8IAGW9wdgE5rNfybJa2r7q+TdG9n2gHQK9OG3/btkh6UdKbtXbY/J+k6SatsPyNpVfUYwAwy7Xv+iFjToDQzL8w/Bh3+nz0Na0N3Na5J0hvTPPfQd15qoaPO2PNbHyvWPzin/OP75b1nNqwt/evnitseLlaPDXzCD0iK8ANJEX4gKcIPJEX4gaQIP5AUU3SjNrNPW1Ksf3X9V4v1Ac8q1v/mpl9pWHvX2IPFbTPgzA8kRfiBpAg/kBThB5Ii/EBShB9IivADSTHOj9o8/buLi/WPDJZnmn7yYHn68flPvfq2e8qEMz+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJMU4P7rqwCc/0rD28GdvnGbr8gxPv33VVcX6O//lh9M8f26c+YGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gqWnH+W1vlPQpSeMRcVa17FpJl0l6oVptfURs6VaTmLmev6Dx+WWey+P4a/5zVbE+9/uPFutRrKKZM/83JZ0/xfIbI+Kc6h/BB2aYacMfEfdL2tuDXgD0UDvv+a+0/ZjtjbZP7FhHAHqi1fB/XdLpks6RNCbphkYr2h6xPWp79JAOtLg7AJ3WUvgjYk9EvBERRyR9Q9LywrobImI4IoYHprlQA0DvtBR+24smPfy0pCc60w6AXmlmqO92SSslnWR7l6QvSlpp+xxNjKbslHR5F3sE0AXThj8i1kyx+OYu9IIZ6B3HH1+sr/2lBxrW9h15vbjt+JfeW6wPHtherKOMT/gBSRF+ICnCDyRF+IGkCD+QFOEHkuKru9GWZ679YLH+3ZP+smFt9TOfKW47uIWhvG7izA8kRfiBpAg/kBThB5Ii/EBShB9IivADSTHOj6L/+42PFuuP/fqfF+v/cfhQw9orf3pKcdtBjRXraA9nfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IinH+5GYv/rli/eov3FmsD7r8I3Tpo2sb1t79d1yvXyfO/EBShB9IivADSRF+ICnCDyRF+IGkCD+Q1LTj/LaXSLpF0smSjkjaEBE32Z4v6U5JSyXtlHRJRLzcvVbRCs8u/xef/d1dxfrF814q1m/dv6BYX/iFxueXI8Ut0W3NnPkPS/p8RHxA0kclXWF7maRrJG2NiDMkba0eA5ghpg1/RIxFxMPV/f2SdkhaLGm1pE3VapskXdStJgF03tt6z297qaQPS9omaWFEjEkTvyAklV//AegrTYff9jxJd0m6OiL2vY3tRmyP2h49pAOt9AigC5oKv+0BTQT/1oi4u1q8x/aiqr5I0vhU20bEhogYjojhAQ12omcAHTBt+G1b0s2SdkTEVyaVNktaV91fJ+nezrcHoFuauaR3haS1kh63/Ui1bL2k6yR92/bnJD0v6eLutIi2nH1msfyHC77V1tN/7Uvl//afffTBtp4f3TNt+CPiAUluUD6vs+0A6BU+4QckRfiBpAg/kBThB5Ii/EBShB9Iiq/uPgbMWvb+hrWRO9r77NWyjVcU60u/9a9tPT/qw5kfSIrwA0kRfiApwg8kRfiBpAg/kBThB5JinP8Y8PTvnNiwduHcpr9xbUqn/OPB8goRbT0/6sOZH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSYpx/Bnj9wuXF+tYLbyhU53a2GRwzOPMDSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFLTjvPbXiLpFkknSzoiaUNE3GT7WkmXSXqhWnV9RGzpVqOZ7V4xq1g/dXbrY/m37l9QrA/sK1/Pz9X8M1czH/I5LOnzEfGw7eMlPWT7vqp2Y0R8uXvtAeiWacMfEWOSxqr7+23vkLS4240B6K639Z7f9lJJH5a0rVp0pe3HbG+0PeV3SdkesT1qe/SQDrTVLIDOaTr8tudJukvS1RGxT9LXJZ0u6RxNvDKY8gPmEbEhIoYjYnhAgx1oGUAnNBV+2wOaCP6tEXG3JEXEnoh4IyKOSPqGpPLVJwD6yrTht21JN0vaERFfmbR80aTVPi3pic63B6Bbmvlr/wpJayU9bvuRatl6SWtsn6OJ0Z6dki7vSodoy5+8tKxYf/BXlxbrMfZ4B7tBP2nmr/0PSPIUJcb0gRmMT/gBSRF+ICnCDyRF+IGkCD+QFOEHknL0cIrlEzw/zvV5PdsfkM222Kp9sXeqofm34MwPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0n1dJzf9guS/mvSopMkvdizBt6efu2tX/uS6K1VnezttIh4dzMr9jT8b9m5PRoRw7U1UNCvvfVrXxK9taqu3njZDyRF+IGk6g7/hpr3X9KvvfVrXxK9taqW3mp9zw+gPnWf+QHUpJbw2z7f9r/Zftb2NXX00IjtnbYft/2I7dGae9loe9z2E5OWzbd9n+1nqtspp0mrqbdrbf93dewesf1rNfW2xPY/2N5h+0nbV1XLaz12hb5qOW49f9lve5akf5e0StIuSdslrYmIp3raSAO2d0oajojax4Rt/7KkVyTdEhFnVcuul7Q3Iq6rfnGeGBF/0Ce9XSvplbpnbq4mlFk0eWZpSRdJ+k3VeOwKfV2iGo5bHWf+5ZKejYjnIuKgpDskra6hj74XEfdL2vumxaslbarub9LED0/PNeitL0TEWEQ8XN3fL+nozNK1HrtCX7WoI/yLJf1o0uNd6q8pv0PSD2w/ZHuk7mamsLCaNv3o9OkLau7nzaadubmX3jSzdN8cu1ZmvO60OsI/1VcM9dOQw4qI+AVJF0i6onp5i+Y0NXNzr0wxs3RfaHXG606rI/y7JC2Z9PgUSbtr6GNKEbG7uh2XdI/6b/bhPUcnSa1ux2vu5yf6aebmqWaWVh8cu36a8bqO8G+XdIbt99ieI+lSSZtr6OMtbA9Vf4iR7SFJn1D/zT68WdK66v46SffW2MtP6ZeZmxvNLK2aj12/zXhdy4d8qqGMP5M0S9LGiPjjnjcxBdvv1cTZXpqYxPS2OnuzfbuklZq46muPpC9K+ltJ35Z0qqTnJV0cET3/w1uD3lZq4qXrT2ZuPvoeu8e9/aKkf5b0uKQj1eL1mnh/XduxK/S1RjUcNz7hByTFJ/yApAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyT1//RJwTziTb07AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(x_test[0].reshape(28,28))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 784)"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_test[0:1].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(model.predict(x_test[0:1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AND WE NEED A GOOD COMPUTER!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://miro.medium.com/max/776/1*Eb4YFMdn8LJhxjJCuykiLg.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import boston_housing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train, y_train), (X_test, y_test) = boston_housing.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset Description\n",
    "- Boston housing dataset has total 506 data instances (404 training & 102 test)\n",
    "- 13 attributes (features) to predict \"the median values of the houses at a location\"\n",
    "- Doc: https://keras.io/datasets/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(404, 13)\n",
      "(102, 13)\n",
      "(404,)\n",
      "(102,)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)\n",
    "print(X_test.shape)\n",
    "print(y_train.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Activation, Dense"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(10, input_shape = (13,)))    \n",
    "model.add(Activation('tanh'))\n",
    "model.add(Dense(10))                        \n",
    "model.add(Activation('tanh'))\n",
    "model.add(Dense(10))                       \n",
    "model.add(Activation('tanh'))\n",
    "model.add(Dense(1))  \n",
    "\n",
    "model.compile(optimizer = 'sgd', loss = 'mse')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is equivalent to the above code block\n",
    "model = Sequential()\n",
    "model.add(Dense(10, input_shape = (13,), activation = 'sigmoid'))\n",
    "model.add(Dense(10, activation = 'sigmoid'))\n",
    "model.add(Dense(10, activation = 'sigmoid'))\n",
    "model.add(Dense(1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import optimizers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "sgd = optimizers.SGD(lr = 0.01)    # stochastic gradient descent optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer = sgd, loss = 'mean_squared_error', metrics = ['mse'])    # for regression problems, mean squared error (MSE) is often employed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary of the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_9\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_30 (Dense)             (None, 10)                140       \n",
      "_________________________________________________________________\n",
      "activation_7 (Activation)    (None, 10)                0         \n",
      "_________________________________________________________________\n",
      "dense_31 (Dense)             (None, 10)                110       \n",
      "_________________________________________________________________\n",
      "activation_8 (Activation)    (None, 10)                0         \n",
      "_________________________________________________________________\n",
      "dense_32 (Dense)             (None, 10)                110       \n",
      "_________________________________________________________________\n",
      "activation_9 (Activation)    (None, 10)                0         \n",
      "_________________________________________________________________\n",
      "dense_33 (Dense)             (None, 1)                 11        \n",
      "=================================================================\n",
      "Total params: 371\n",
      "Trainable params: 371\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Training\n",
    "- Training the model with training data provided"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "404/404 [==============================] - 0s 193us/step - loss: 280.5018 - mse: 280.5018\n",
      "Epoch 2/100\n",
      "404/404 [==============================] - 0s 48us/step - loss: 88.6452 - mse: 88.6452\n",
      "Epoch 3/100\n",
      "404/404 [==============================] - 0s 46us/step - loss: 82.3068 - mse: 82.3068\n",
      "Epoch 4/100\n",
      "404/404 [==============================] - 0s 31us/step - loss: 77.5539 - mse: 77.5539\n",
      "Epoch 5/100\n",
      "404/404 [==============================] - 0s 37us/step - loss: 77.3340 - mse: 77.3340\n",
      "Epoch 6/100\n",
      "404/404 [==============================] - 0s 41us/step - loss: 76.1818 - mse: 76.1818\n",
      "Epoch 7/100\n",
      "404/404 [==============================] - 0s 41us/step - loss: 84.9779 - mse: 84.9779\n",
      "Epoch 8/100\n",
      "404/404 [==============================] - 0s 36us/step - loss: 86.1767 - mse: 86.1767\n",
      "Epoch 9/100\n",
      "404/404 [==============================] - 0s 28us/step - loss: 86.0187 - mse: 86.0187\n",
      "Epoch 10/100\n",
      "404/404 [==============================] - 0s 54us/step - loss: 85.6429 - mse: 85.6429\n",
      "Epoch 11/100\n",
      "404/404 [==============================] - 0s 51us/step - loss: 85.2739 - mse: 85.2738\n",
      "Epoch 12/100\n",
      "404/404 [==============================] - 0s 32us/step - loss: 85.1592 - mse: 85.1592\n",
      "Epoch 13/100\n",
      "404/404 [==============================] - 0s 41us/step - loss: 84.9405 - mse: 84.9405\n",
      "Epoch 14/100\n",
      "404/404 [==============================] - 0s 41us/step - loss: 86.2996 - mse: 86.2995\n",
      "Epoch 15/100\n",
      "404/404 [==============================] - 0s 34us/step - loss: 86.3779 - mse: 86.3779\n",
      "Epoch 16/100\n",
      "404/404 [==============================] - 0s 33us/step - loss: 85.3212 - mse: 85.3212\n",
      "Epoch 17/100\n",
      "404/404 [==============================] - 0s 48us/step - loss: 84.8672 - mse: 84.8672\n",
      "Epoch 18/100\n",
      "404/404 [==============================] - 0s 47us/step - loss: 84.8931 - mse: 84.8931\n",
      "Epoch 19/100\n",
      "404/404 [==============================] - 0s 38us/step - loss: 85.2429 - mse: 85.2429\n",
      "Epoch 20/100\n",
      "404/404 [==============================] - 0s 50us/step - loss: 85.8177 - mse: 85.8176\n",
      "Epoch 21/100\n",
      "404/404 [==============================] - 0s 38us/step - loss: 85.4106 - mse: 85.4106\n",
      "Epoch 22/100\n",
      "404/404 [==============================] - 0s 39us/step - loss: 84.8829 - mse: 84.8829\n",
      "Epoch 23/100\n",
      "404/404 [==============================] - 0s 39us/step - loss: 86.0949 - mse: 86.0949\n",
      "Epoch 24/100\n",
      "404/404 [==============================] - 0s 30us/step - loss: 85.2963 - mse: 85.2963\n",
      "Epoch 25/100\n",
      "404/404 [==============================] - 0s 32us/step - loss: 85.4459 - mse: 85.4458\n",
      "Epoch 26/100\n",
      "404/404 [==============================] - 0s 39us/step - loss: 85.2212 - mse: 85.2212\n",
      "Epoch 27/100\n",
      "404/404 [==============================] - 0s 40us/step - loss: 84.9078 - mse: 84.9078\n",
      "Epoch 28/100\n",
      "404/404 [==============================] - 0s 43us/step - loss: 85.1228 - mse: 85.1228\n",
      "Epoch 29/100\n",
      "404/404 [==============================] - 0s 46us/step - loss: 85.0764 - mse: 85.0764\n",
      "Epoch 30/100\n",
      "404/404 [==============================] - 0s 34us/step - loss: 84.8732 - mse: 84.8732\n",
      "Epoch 31/100\n",
      "404/404 [==============================] - 0s 27us/step - loss: 84.9909 - mse: 84.9909\n",
      "Epoch 32/100\n",
      "404/404 [==============================] - 0s 35us/step - loss: 87.6546 - mse: 87.6546\n",
      "Epoch 33/100\n",
      "404/404 [==============================] - 0s 35us/step - loss: 86.1705 - mse: 86.1705\n",
      "Epoch 34/100\n",
      "404/404 [==============================] - 0s 35us/step - loss: 84.7882 - mse: 84.7882\n",
      "Epoch 35/100\n",
      "404/404 [==============================] - 0s 39us/step - loss: 85.2738 - mse: 85.2738\n",
      "Epoch 36/100\n",
      "404/404 [==============================] - 0s 40us/step - loss: 84.7827 - mse: 84.7827\n",
      "Epoch 37/100\n",
      "404/404 [==============================] - 0s 36us/step - loss: 85.1586 - mse: 85.1586\n",
      "Epoch 38/100\n",
      "404/404 [==============================] - 0s 51us/step - loss: 84.9580 - mse: 84.9580\n",
      "Epoch 39/100\n",
      "404/404 [==============================] - 0s 39us/step - loss: 84.8355 - mse: 84.8355\n",
      "Epoch 40/100\n",
      "404/404 [==============================] - 0s 45us/step - loss: 84.9633 - mse: 84.9633\n",
      "Epoch 41/100\n",
      "404/404 [==============================] - 0s 29us/step - loss: 85.0074 - mse: 85.0074\n",
      "Epoch 42/100\n",
      "404/404 [==============================] - 0s 42us/step - loss: 85.0304 - mse: 85.0304\n",
      "Epoch 43/100\n",
      "404/404 [==============================] - 0s 48us/step - loss: 84.9443 - mse: 84.9443\n",
      "Epoch 44/100\n",
      "404/404 [==============================] - 0s 55us/step - loss: 85.0424 - mse: 85.0424\n",
      "Epoch 45/100\n",
      "404/404 [==============================] - 0s 36us/step - loss: 85.1705 - mse: 85.1705\n",
      "Epoch 46/100\n",
      "404/404 [==============================] - 0s 34us/step - loss: 85.1284 - mse: 85.1284\n",
      "Epoch 47/100\n",
      "404/404 [==============================] - 0s 41us/step - loss: 85.0883 - mse: 85.0883\n",
      "Epoch 48/100\n",
      "404/404 [==============================] - 0s 32us/step - loss: 84.9262 - mse: 84.9262\n",
      "Epoch 49/100\n",
      "404/404 [==============================] - 0s 46us/step - loss: 85.7650 - mse: 85.7650\n",
      "Epoch 50/100\n",
      "404/404 [==============================] - 0s 48us/step - loss: 86.3392 - mse: 86.3392\n",
      "Epoch 51/100\n",
      "404/404 [==============================] - 0s 50us/step - loss: 86.2141 - mse: 86.2141\n",
      "Epoch 52/100\n",
      "404/404 [==============================] - 0s 41us/step - loss: 85.2622 - mse: 85.2622\n",
      "Epoch 53/100\n",
      "404/404 [==============================] - 0s 59us/step - loss: 85.4004 - mse: 85.4004\n",
      "Epoch 54/100\n",
      "404/404 [==============================] - 0s 52us/step - loss: 84.9449 - mse: 84.9449\n",
      "Epoch 55/100\n",
      "404/404 [==============================] - 0s 41us/step - loss: 85.2436 - mse: 85.2436\n",
      "Epoch 56/100\n",
      "404/404 [==============================] - 0s 37us/step - loss: 86.5676 - mse: 86.5676\n",
      "Epoch 57/100\n",
      "404/404 [==============================] - 0s 50us/step - loss: 85.2498 - mse: 85.2498\n",
      "Epoch 58/100\n",
      "404/404 [==============================] - 0s 51us/step - loss: 85.2587 - mse: 85.2587\n",
      "Epoch 59/100\n",
      "404/404 [==============================] - 0s 36us/step - loss: 85.1914 - mse: 85.1914\n",
      "Epoch 60/100\n",
      "404/404 [==============================] - 0s 26us/step - loss: 85.1687 - mse: 85.1687\n",
      "Epoch 61/100\n",
      "404/404 [==============================] - 0s 50us/step - loss: 85.6311 - mse: 85.6311\n",
      "Epoch 62/100\n",
      "404/404 [==============================] - 0s 44us/step - loss: 85.5724 - mse: 85.5724\n",
      "Epoch 63/100\n",
      "404/404 [==============================] - 0s 44us/step - loss: 84.7762 - mse: 84.7762\n",
      "Epoch 64/100\n",
      "404/404 [==============================] - 0s 36us/step - loss: 85.1111 - mse: 85.1111\n",
      "Epoch 65/100\n",
      "404/404 [==============================] - 0s 32us/step - loss: 84.9617 - mse: 84.9617\n",
      "Epoch 66/100\n",
      "404/404 [==============================] - 0s 42us/step - loss: 84.9611 - mse: 84.9611\n",
      "Epoch 67/100\n",
      "404/404 [==============================] - 0s 31us/step - loss: 84.6818 - mse: 84.6818\n",
      "Epoch 68/100\n",
      "404/404 [==============================] - 0s 36us/step - loss: 84.7686 - mse: 84.7686\n",
      "Epoch 69/100\n",
      "404/404 [==============================] - 0s 37us/step - loss: 84.9616 - mse: 84.9616\n",
      "Epoch 70/100\n",
      "404/404 [==============================] - 0s 35us/step - loss: 85.5848 - mse: 85.5848\n",
      "Epoch 71/100\n",
      "404/404 [==============================] - 0s 29us/step - loss: 84.9279 - mse: 84.9279\n",
      "Epoch 72/100\n",
      "404/404 [==============================] - 0s 27us/step - loss: 85.7700 - mse: 85.7700\n",
      "Epoch 73/100\n",
      "404/404 [==============================] - 0s 23us/step - loss: 85.4047 - mse: 85.4047\n",
      "Epoch 74/100\n",
      "404/404 [==============================] - 0s 29us/step - loss: 85.3819 - mse: 85.3819\n",
      "Epoch 75/100\n",
      "404/404 [==============================] - 0s 23us/step - loss: 85.9064 - mse: 85.9064\n",
      "Epoch 76/100\n",
      "404/404 [==============================] - 0s 24us/step - loss: 85.0159 - mse: 85.0158\n",
      "Epoch 77/100\n",
      "404/404 [==============================] - 0s 25us/step - loss: 85.2425 - mse: 85.2424\n",
      "Epoch 78/100\n",
      "404/404 [==============================] - 0s 29us/step - loss: 84.9920 - mse: 84.9920\n",
      "Epoch 79/100\n",
      "404/404 [==============================] - 0s 26us/step - loss: 85.2100 - mse: 85.2100\n",
      "Epoch 80/100\n",
      "404/404 [==============================] - 0s 25us/step - loss: 85.2533 - mse: 85.2533\n",
      "Epoch 81/100\n",
      "404/404 [==============================] - 0s 30us/step - loss: 85.1789 - mse: 85.1789\n",
      "Epoch 82/100\n",
      "404/404 [==============================] - 0s 31us/step - loss: 84.8750 - mse: 84.8750\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 83/100\n",
      "404/404 [==============================] - 0s 28us/step - loss: 84.9674 - mse: 84.9674\n",
      "Epoch 84/100\n",
      "404/404 [==============================] - 0s 26us/step - loss: 85.0261 - mse: 85.0261\n",
      "Epoch 85/100\n",
      "404/404 [==============================] - 0s 32us/step - loss: 84.8147 - mse: 84.8147\n",
      "Epoch 86/100\n",
      "404/404 [==============================] - 0s 31us/step - loss: 85.0486 - mse: 85.0486\n",
      "Epoch 87/100\n",
      "404/404 [==============================] - 0s 36us/step - loss: 85.3336 - mse: 85.3336\n",
      "Epoch 88/100\n",
      "404/404 [==============================] - 0s 36us/step - loss: 85.2314 - mse: 85.2314\n",
      "Epoch 89/100\n",
      "404/404 [==============================] - 0s 32us/step - loss: 85.0297 - mse: 85.0297\n",
      "Epoch 90/100\n",
      "404/404 [==============================] - 0s 44us/step - loss: 85.1038 - mse: 85.1038\n",
      "Epoch 91/100\n",
      "404/404 [==============================] - 0s 27us/step - loss: 84.6853 - mse: 84.6853\n",
      "Epoch 92/100\n",
      "404/404 [==============================] - 0s 22us/step - loss: 85.0146 - mse: 85.0146\n",
      "Epoch 93/100\n",
      "404/404 [==============================] - 0s 40us/step - loss: 86.2467 - mse: 86.2467\n",
      "Epoch 94/100\n",
      "404/404 [==============================] - 0s 25us/step - loss: 84.9741 - mse: 84.9741\n",
      "Epoch 95/100\n",
      "404/404 [==============================] - 0s 41us/step - loss: 87.4263 - mse: 87.4263\n",
      "Epoch 96/100\n",
      "404/404 [==============================] - 0s 25us/step - loss: 85.8178 - mse: 85.8178\n",
      "Epoch 97/100\n",
      "404/404 [==============================] - 0s 42us/step - loss: 85.0414 - mse: 85.0414\n",
      "Epoch 98/100\n",
      "404/404 [==============================] - 0s 38us/step - loss: 85.0037 - mse: 85.0037\n",
      "Epoch 99/100\n",
      "404/404 [==============================] - 0s 34us/step - loss: 84.8184 - mse: 84.8184\n",
      "Epoch 100/100\n",
      "404/404 [==============================] - 0s 32us/step - loss: 84.9363 - mse: 84.9363\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.callbacks.History at 0x7f5f9386c160>"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train, batch_size = 50, epochs = 100, verbose = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Evaluation\n",
    "- Keras model can be evaluated with evaluate() function\n",
    "- Evaluation results are contained in a list\n",
    "    - Doc (metrics): https://keras.io/metrics/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "102/102 [==============================] - 0s 284us/step\n"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['loss', 'mse']\n",
      "[83.30722404928768, 83.3072280883789]\n"
     ]
    }
   ],
   "source": [
    "print(model.metrics_names)     # list of metric names the model is employing\n",
    "print(results)                 # actual figure of metrics computed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss:  83.30722404928768\n",
      "mse:  83.3072280883789\n"
     ]
    }
   ],
   "source": [
    "print('loss: ', results[0])\n",
    "print('mse: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MLP for classification tasks\n",
    "- When the target (**y**) is discrete (categorical)\n",
    "- For loss function, cross-entropy is used and for evaluation metric, accuracy is commonly used"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "whole_data = load_breast_cancer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_data = whole_data.data\n",
    "y_data = whole_data.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X_data, y_data, test_size = 0.3, random_state = 7) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset Description\n",
    "- Breast cancer dataset has total 569 data instances (212 malign, 357 benign instances)\n",
    "- 30 attributes (features) to predict the binary class (M/B)\n",
    "- Doc: http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html#sklearn.datasets.load_breast_cancer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(398, 30)\n",
      "(171, 30)\n",
      "(398,)\n",
      "(171,)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)\n",
    "print(X_test.shape)\n",
    "print(y_train.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Creating a model\n",
    "- Same with regression model at the outset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1-1. Adding layers\n",
    "- Keras layers can be **added** to the model\n",
    "- Adding layers are like stacking lego blocks one by one\n",
    "- It should be noted that as this is a classification problem, sigmoid layer (softmax for multi-class problems) should be added\n",
    "- Doc: https://keras.io/layers/core/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keras model with two hidden layer with 10 neurons each \n",
    "model = Sequential()\n",
    "model.add(Dense(10, input_shape = (30,)))    # Input layer => input_shape should be explicitly designated\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(10))                         # Hidden layer => only output dimension should be designated\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(10))                         # Hidden layer => only output dimension should be designated\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(1))                          # Output layer => output dimension = 1 since it is regression problem\n",
    "model.add(Activation('sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is equivalent to the above code block\n",
    "model = Sequential()\n",
    "model.add(Dense(10, input_shape = (30,), activation = 'sigmoid'))\n",
    "model.add(Dense(10, activation = 'sigmoid'))\n",
    "model.add(Dense(10, activation = 'sigmoid'))\n",
    "model.add(Dense(1, activation = 'sigmoid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1-2. Model compile\n",
    "- Keras model should be \"compiled\" prior to training\n",
    "- Types of loss (function) and optimizer should be designated\n",
    "    - Doc (optimizers): https://keras.io/optimizers/\n",
    "    - Doc (losses): https://keras.io/losses/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import optimizers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "sgd = optimizers.SGD(lr = 0.01)    # stochastic gradient descent optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer = sgd, loss = 'binary_crossentropy', metrics = ['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary of the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 10)                310       \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                110       \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 10)                110       \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 1)                 11        \n",
      "=================================================================\n",
      "Total params: 541\n",
      "Trainable params: 541\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Training\n",
    "- Training the model with training data provided"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "404/404 [==============================] - 0s 69us/step - loss: 77.0093 - mse: 77.0093\n",
      "Epoch 2/10\n",
      "404/404 [==============================] - 0s 66us/step - loss: 77.0731 - mse: 77.0731\n",
      "Epoch 3/10\n",
      "404/404 [==============================] - 0s 33us/step - loss: 77.2765 - mse: 77.2765\n",
      "Epoch 4/10\n",
      "404/404 [==============================] - 0s 21us/step - loss: 79.0802 - mse: 79.0802\n",
      "Epoch 5/10\n",
      "404/404 [==============================] - 0s 14us/step - loss: 77.5045 - mse: 77.5045\n",
      "Epoch 6/10\n",
      "404/404 [==============================] - 0s 14us/step - loss: 77.0580 - mse: 77.0580\n",
      "Epoch 7/10\n",
      "404/404 [==============================] - 0s 15us/step - loss: 78.2971 - mse: 78.2971\n",
      "Epoch 8/10\n",
      "404/404 [==============================] - 0s 28us/step - loss: 77.4874 - mse: 77.4874\n",
      "Epoch 9/10\n",
      "404/404 [==============================] - 0s 18us/step - loss: 79.7340 - mse: 79.7340\n",
      "Epoch 10/10\n",
      "404/404 [==============================] - 0s 24us/step - loss: 77.0675 - mse: 77.0675\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.callbacks.History at 0x7f5f94e439e8>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train, batch_size = 200, epochs = 10, verbose = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Evaluation\n",
    "- Keras model can be evaluated with evaluate() function\n",
    "- Evaluation results are contained in a list\n",
    "    - Doc (metrics): https://keras.io/metrics/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "102/102 [==============================] - 0s 160us/step\n"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['loss', 'mse']\n",
      "[77.59952141256893, 77.59952545166016]\n"
     ]
    }
   ],
   "source": [
    "print(model.metrics_names)     # list of metric names the model is employing\n",
    "print(results)                 # actual figure of metrics computed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss:  77.59952141256893\n",
      "accuracy:  77.59952545166016\n"
     ]
    }
   ],
   "source": [
    "print('loss: ', results[0])\n",
    "print('accuracy: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Famous Loss Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Squared Error Loss\n",
    "![](https://i1.wp.com/s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2019/08/mse.jpg.jpg?resize=289%2C96&ssl=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Absolute Error Loss\n",
    "![](https://i2.wp.com/s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2019/08/mae.jpg.jpg?resize=277%2C94&ssl=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Huber Loss\n",
    "![](https://i2.wp.com/s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2019/08/huber.jpg.jpg?resize=513%2C202&ssl=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Binary Cross Entropy Loss\n",
    "![](https://i0.wp.com/s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2019/06/entropy.jpg1.jpg?resize=418%2C132&ssl=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multi-Class Cross Entropy Loss\n",
    "![](https://i1.wp.com/s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2019/06/mce.jpg1.jpg?resize=679%2C210&ssl=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## KL-Divergence\n",
    "![](https://i1.wp.com/s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2019/06/kl.jpg1.jpg?resize=624%2C280&ssl=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Reference](https://www.analyticsvidhya.com/blog/2019/08/detailed-guide-7-loss-functions-machine-learning-python-code/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}