demo07_2_mnist_neural.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST Classification via a Neural Network\n",
    "\n",
    "In the the [MNIST SVM example](../svm/mnist_svm.ipynb), we introduced the classic MNIST digit classification problem and trained a simple SVM classifier for the model.  In this demo, we will try a simple neural network.  The network we will create will not perform quite as well -- we will obtain an accuracy of only around 97%, while the SVM classifier obtains an accuracy of over 98%.  However, once we understand these simple neural networks, we will be able to build more sophisticated networks that can obtain much better classification rate.  Also, in doing this demo, you will learn several important features of the `keras` package in addition to the concepts shown in the [simple neural network example](./synthetic.ipynb):\n",
    "\n",
    "* How to construct multi-class classifiers using categorical cross entropy.\n",
    "* How to optimize using mini-batches.\n",
    "* How to save and load the model after training.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the Keras package and the MNIST data\n",
    "\n",
    "We first load the `keras` package as in the [simple neural network example](./synthetic.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also load some other common packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we get the MNIST data as in the [MNIST SVM example](../svm/mnist_svm.ipynb).  We rescale the input `X` from values from 0 to 1 as this works better for the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Zafir\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:77: DeprecationWarning: Function fetch_mldata is deprecated; fetch_mldata was deprecated in version 0.20 and will be removed in version 0.22\n",
      "  warnings.warn(msg, category=DeprecationWarning)\n",
      "C:\\Users\\Zafir\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:77: DeprecationWarning: Function mldata_filename is deprecated; mldata_filename was deprecated in version 0.20 and will be removed in version 0.22\n",
      "  warnings.warn(msg, category=DeprecationWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(70000, 784)\n",
      "(70000,)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import fetch_mldata\n",
    "mnist = fetch_mldata(\"MNIST original\")\n",
    "\n",
    "X = (mnist.data/255)   # Change X from 0 to 1\n",
    "y = mnist.target\n",
    "\n",
    "print(X.shape)\n",
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also split the data into traing and test.  A standard split uses 50,000 examples for training and 10,000 for test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(50000, 784)\n",
      "(50000,)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "ntr = 50000\n",
    "nts = 10000\n",
    "\n",
    "Xtr, Xts, ytr, yts = train_test_split(X,y,train_size=ntr, test_size=nts,shuffle=True)\n",
    "\n",
    "print(Xtr.shape)\n",
    "print(ytr.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also use the `plt_digit` function from the [MNIST SVM example](../svm/mnist_svm.ipynb) to display digits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAABeCAYAAAAUjW5fAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAC0tJREFUeJzt3WuI1cUfx/H3ei+7KNpVyURNTUq0SIt8kJYWBtqFMlHKMEsxhAq6SHipoDT1iRZWICkUEumDtLJMQjG8ZolIajcNFW3DwjK7ef4PZL7n+/u77jlnz/ntb87u5/XEL7PnMkzb7PxmvjNTk8vlEBGR7LXIugIiInKaOmQRkUioQxYRiYQ6ZBGRSKhDFhGJhDpkEZFIqEMWEYmEOmQRkUioQxYRiYQ6ZBGRSLQq5cU1NTXaZ11ALperacj71LZFqc3lcheV+ia1bVEa1Lag9i1SUe2rEbJUk/1ZV6AJU9umq6j2VYcsIhIJdcgiIpFQhywiEgl1yCIikVCHLCISCXXIIiKRUIcsIhKJkjaGNJbOnTtbPHbs2DN+PmXKFACuuuqqej/nr7/+svjpp5+2+J133gGgtra2rHqKiFSSRsgiIpFQhywiEomaXK74behp7Fnv27cvkJ9GADjvvPMs7tGjR6W/ko0bNwIwfPhwK/vzzz8r8tk6yyJV23O53PWlvkltW5QGtS3E3b7XXHNN0a999NFHAWjbtq2VDR06FIDWrVtb2Zo1ayzeunUrAG+88Uahjy+qfTVCFhGJhDpkEZFINOqURbt27YBkxsO0adMA6NChQzkfzalTpyzeu3cvAH369Kn3PV9++aXFt956KwC//vprWfWoximLu+++2+L333//jJ/v2bPH4kJtmrKopyy6dOlicZs2bQAYPXq0lY0cORKAnj17WpmfKrvnnnsA2L17d6r1PIuqmrJo1ep0gphv33HjxgHJaYru3btX/Lt9X3P8+HEAOnbsWOhtmrIQEakmjZqH3KtXLwBmzJhR1uecOHECgB07dljZ7NmzLd61axcA1157rZX5UeDDDz8MwMCBA61s586dALzwwgtW9uabb5ZVz2oRng7Opnfv3ha/9tprQD4XvLn64IMPALj55putLIyKAWpqTj8ohadCyI+svv32WyvzbbthwwYABgwYYGUHDhyoZLWrmm/fxYsXA/Dggw9W7POPHDkCwKFDh6zshx9+AGDt2rVW5p+st2zZUrHvB42QRUSioQ5ZRCQSjTplsXTp0ga/1y+2hYn89evX1/uew4cPW+xzB//55x8g+djdtWtXAC677LIG17HahGmcyZMnF/2e8NrmOGWxcuVKi0eMGAHkF5eK8e+//wLJRdJLL73U4rAwdOedd1pZmCKSfJ4w1D1VEaYV/HELTzzxxBmvW758ucUHDx60+I8//gAqtyehITRCFhGJROppb7NmzbI4pLv5yflCQlqJ31W3efPmUquRcN111wH5XTbekiVLLJ40aZLF//33X1GfXU1pb2GEXFeqWyEZpcJlmvbmfwfCop23bt06i/v16wfAZ599ZmVXXHEFkFwIrIt/svMLz2HRKSVRpr35tLWvv/7a4rCb1y/AhXTCkydPplWdcijtTUSkmqhDFhGJROqLes8//3zJ7wnTFACjRo0Cyp+m8H766ScguSPq6quvBmDChAlW9tRTT1l87Nixin1/LFasWJF1FapKyEmF/OLv22+/bWU+p/iSSy4BktMMjzzyCJD8HVu4cOEZ3+MXln3+/Ouvv97gulerl156yWJ/6Fjw3XffWRzpVEVJNEIWEYmEOmQRkUhEeYXT9u3bLf78888r/vlHjx4F4IsvvrCyMGXhDR482OKPPvqo4vWIhX8ULpSTHLIrMj5kKBODBg2yOFwP5nNjfRxyWZ988skzPsdvyb/gggssDtv2W7ZsaWVpnAdeTXw2T13279/fSDVpHBohi4hEIvURss/XrC/n2R/Yce+996Zap2L5BYWmPEIuRXMcGQe//PKLxWFXnV+09r/r55xzDpB/GgPo1q0bkNwJ9vLLL1scdpX5S34nTpxo8YsvvgiUf0RsNfELpXXxO+2aAo2QRUQioQ5ZRCQSqU9ZFNqaHR6//AJaOIQla/3797c4HKajw14E4PfffweSB1z5Q4GCQotS3pw5cxL/QnLRb+rUqUB+6qI5KNR+Ibcb4Oeffwbgvffes7Kw56BaaIQsIhKJzNPewgg6llGx5xdpWrTQ3y7JC0e4fvzxx1bmR8g//vgjAEOGDCn6MwsdHvTAAw8AzWuEvGjRIov9At+YMWMA6NSpk5W9+uqrAEyfPt3K5s2bZ3EYOe/bty+dylaAehkRkUioQxYRiUTmUxYi1exsh06FG2huuOEGKyt0Iebtt99e788vv/xyIHnITlhcbKpqa2stHj9+vMXbtm0DkpcSh9zvkCMOyb0E4f3+tpFKX1JaLo2QRUQioQ5ZRCQSmrIQKcO7775r8SuvvGJxmLK47777rKyux+PWrVtbHKYsznZV1IUXXggkr4DyWR7VJEwvQPGXivp2mT9/PgAffvihlT3zzDNA/gx1yLcZ5LM0/LVafkrDb2PPikbIIiKRSH2E7C8m9DvfsnTxxRcDcNNNN2Vck+pQym6z5swfkBVGyHfddZeVPfvss0A+hxngueeeszgsRi1YsMDK/A7WG2+8scI1zs7atWstDrcB+dGqP8ipPt98843FDz30EAC9evWyMv+Zo0ePBqB9+/ZW5nf6rVmzBoAdO3YU9d1p0AhZRCQS6pBFRCKR+pTFbbfdZnG4oaNnz55WFg5POXDggJX5WytWr15dkXqEaQqATz75BKj7lhDv77//tvi3336rSD2k6fr0008tHjlyJADdu3e3srp+h9q0aWNxWLRavny5lfkpi8BPg1Trot6uXbssfvzxx4HkYlxoPz8lUSy/NdovqoapoGnTplmZ/+8TLv31ZY1NI2QRkUioQxYRiUTqUxZ+6+PixYsBmDt3rpWFCx3DqjQkczvDCVcNmbrwOYjr1q2zuNBURbB7926Lly1bVvL3NxWFrtGR0/zJZMHs2bMt9lt6A3/KYXhkL7Sdd+XKlQ2tYjT8hbAh6yGc4Ab5TAd/yfGmTZssDhfzHj9+3MratWsHJLezn3/++RaHLJgTJ05Y2bnnnmtxq1bZb8vQCFlEJBKN+ifhrbfeAuCxxx6zsjAybtu2rZX5w1PCGaZ+ct+fd1qX4cOHAzBhwgQr8zcv1Mf/db3//vuLeo/I/wujZf9kd8cddwDJPOTw/0QprrzyyvIqF5lx48YB8NVXX1nZpEmTABgxYoSV+XjGjBkAHD582Mo6dOgAJPPme/ToYbEfLcdKI2QRkUioQxYRiURNoUtIEy+uqSn+xUUKW0djuZZm5syZFvsFmWLlcrmawq86UxptWyx/cavPAa+LP+wmA9tzudz1pb4py7Yt18aNGy0OW/39gTohX7cCGtS2kE77hsV+P5Xp9zRUamGzrvOWwxbqCiuqfTVCFhGJROZ5HuHIu5MnT1qZX/Tzu/oqLVwbDnDLLbcA8P3336f2fdUqpBhJtkp5mq12YdeiT1FbtWqVxSEddtiwYVY2YMAAIHm40M6dOy0OiQF+dB12D0Mcu3E1QhYRiYQ6ZBGRSGQ+ZXHq1CkgfwMAwNKlSy0Ou3f8gSBhd1MpO8j8GadLliwBkjt//K48EYmPvzEkHMDkD2JqCjRCFhGJhDpkEZFIZD5lURefG7hw4cLEv9L4/HU70rj84TkhB3zo0KFZVUdSphGyiEgkMt+p19RU4049L+Rq+gXTjHfnec1up56/MWT9+vUAtGiRH0dV8MjIqHbqNUHaqSciUk3UIYuIRCLKRT3JTp8+fbKugjg+V95fiCpNk0bIIiKRUIcsIhIJdcgiIpFQhywiEolSF/Vqgf1pVKSJ6FbGe9W2hTW0fdW2hel3N11FtW9JG0NERCQ9mrIQEYmEOmQRkUioQxYRiYQ6ZBGRSKhDFhGJhDpkEZFIqEMWEYmEOmQRkUioQxYRicT/AKvqkLQfrcGUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plt_digit(x):\n",
    "    nrow = 28\n",
    "    ncol = 28\n",
    "    xsq = x.reshape((nrow,ncol))\n",
    "    plt.imshow(xsq,  cmap='Greys_r')\n",
    "    plt.xticks([])\n",
    "    plt.yticks([])\n",
    "    \n",
    "# Select random digits\n",
    "nplt = 4\n",
    "nsamp = X.shape[0]\n",
    "Iperm = np.random.permutation(nsamp)\n",
    "\n",
    "# Plot the images using the subplot command\n",
    "for i in range(nplt):\n",
    "    ind = Iperm[i]\n",
    "    plt.subplot(1,nplt,i+1)\n",
    "    plt_digit(X[ind,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Simple Neural Network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create the neural network, we first import the appropriate sub-packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Model, Sequential\n",
    "from keras.layers import Dense, Activation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we clear the session.  As in the [simple neural network example](./synthetic.ipynb), this step is not necessary, but it is good practice as it clears any model layers that you have built before.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras.backend as K\n",
    "K.clear_session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create a very simple network.  The features are:\n",
    "*  We have one hidden layer with `nh=100` units.  \n",
    "*  One output layer with `nout=10` units, one for each of the 10 possible classes\n",
    "*  The output activation is `softmax`, which is used for multi-class targets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "nin = X.shape[1]  # dimension of input data\n",
    "nh = 100     # number of hidden units\n",
    "nout = int(np.max(y)+1)    # number of outputs = 10 since there are 10 classes\n",
    "model = Sequential()\n",
    "model.add(Dense(nh, input_shape=(nin,), activation='sigmoid', name='hidden'))\n",
    "model.add(Dense(nout, activation='softmax', name='output'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We print the model summary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "hidden (Dense)               (None, 100)               78500     \n",
      "_________________________________________________________________\n",
      "output (Dense)               (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 79,510\n",
      "Trainable params: 79,510\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the Network\n",
    "\n",
    "As before, to train the network, we have to select an optimizer and a loss function.  Since this is a multi-class classification problem, we select the `sparse_categorial_crossentropy` loss.  We use the `adam` optimizer as before.  You may want to play with the learning rate `lr`.   We also set the `metrics` that we wish to track during the optimization.  In this case, we select `accuracy` on the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import optimizers\n",
    "\n",
    "opt = optimizers.Adam(lr=0.001) # x_1=0.9, x_2=0.999, epsilon=1e-08, decay=0.0)\n",
    "model.compile(optimizer=opt,\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keras provides a simple method `fit` to run the optimization.  You simply specify the number of epochs and the batch size, both discussed in class.  In addition, we specify the `validation_data` so that it can print the accuracy on the test data set as it performs the optimization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 50000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0122 - acc: 0.9990 - val_loss: 0.0762 - val_acc: 0.9774\n",
      "Epoch 2/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0110 - acc: 0.9992 - val_loss: 0.0748 - val_acc: 0.9775\n",
      "Epoch 3/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0098 - acc: 0.9994 - val_loss: 0.0757 - val_acc: 0.9785\n",
      "Epoch 4/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0089 - acc: 0.9994 - val_loss: 0.0754 - val_acc: 0.9781\n",
      "Epoch 5/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0081 - acc: 0.9995 - val_loss: 0.0750 - val_acc: 0.9784\n",
      "Epoch 6/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0072 - acc: 0.9997 - val_loss: 0.0761 - val_acc: 0.9771\n",
      "Epoch 7/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0065 - acc: 0.9998 - val_loss: 0.0748 - val_acc: 0.9793\n",
      "Epoch 8/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0058 - acc: 0.9997 - val_loss: 0.0759 - val_acc: 0.9779\n",
      "Epoch 9/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0052 - acc: 0.9998 - val_loss: 0.0769 - val_acc: 0.9784\n",
      "Epoch 10/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0047 - acc: 0.9999 - val_loss: 0.0767 - val_acc: 0.9789\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x27702279748>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(Xtr, ytr, epochs=10, batch_size=100, validation_data=(Xts,yts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running the 10 epochs, you should obtain a test accuracy of around 95.58%.  If we run it for another a few epochs, we can get slightly higher results.  We can just run the `model.fit` command again, and it will start where it left off."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 50000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0041 - acc: 0.9999 - val_loss: 0.0775 - val_acc: 0.9786\n",
      "Epoch 2/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0037 - acc: 0.9999 - val_loss: 0.0788 - val_acc: 0.9783\n",
      "Epoch 3/10\n",
      "50000/50000 [==============================] - 1s 23us/step - loss: 0.0033 - acc: 0.9999 - val_loss: 0.0785 - val_acc: 0.9782\n",
      "Epoch 4/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0030 - acc: 1.0000 - val_loss: 0.0785 - val_acc: 0.9787\n",
      "Epoch 5/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0026 - acc: 1.0000 - val_loss: 0.0794 - val_acc: 0.9790\n",
      "Epoch 6/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0024 - acc: 1.0000 - val_loss: 0.0815 - val_acc: 0.9775\n",
      "Epoch 7/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0021 - acc: 1.0000 - val_loss: 0.0809 - val_acc: 0.9789\n",
      "Epoch 8/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0019 - acc: 1.0000 - val_loss: 0.0814 - val_acc: 0.9789\n",
      "Epoch 9/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0016 - acc: 1.0000 - val_loss: 0.0813 - val_acc: 0.9789\n",
      "Epoch 10/10\n",
      "50000/50000 [==============================] - 1s 24us/step - loss: 0.0015 - acc: 1.0000 - val_loss: 0.0822 - val_acc: 0.9794\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x27702279a58>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(Xtr, ytr, epochs=10, batch_size=100, validation_data=(Xts,yts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should get a little more than 97.4% accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving and Loading the model\n",
    "\n",
    "Since the training takes a long time, it is useful to save the results.  See the [keras page](https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model) for many more useful saving and loading features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"mnist_mod.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now reload the model with the `load_model` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import load_model\n",
    "\n",
    "model = load_model(\"mnist_mod.h5\")\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can test the performance on the test data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "score, acc = model.evaluate(Xts, yts, verbose=0)\n",
    "print(\"accuracy = %f\" % acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}