CNNs_Part1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "CNNs-Part1.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PZJZhAeZfhu3",
        "colab_type": "text"
      },
      "source": [
        "# Convolutional Neural Networks (CNNs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_rkcndAwZSD4",
        "colab_type": "text"
      },
      "source": [
        "### Import TensorFlow"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jXMfpvmnZKhb",
        "colab_type": "code",
        "outputId": "b91c6f3a-4889-4eb3-8c7a-c044498a2ea5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 389
        }
      },
      "source": [
        "from __future__ import absolute_import, division, print_function, unicode_literals\n",
        "\n",
        "!pip install tensorflow-gpu==2.0.0-beta1\n",
        "import tensorflow as tf\n",
        "import datetime, os\n",
        "\n",
        "from tensorflow.keras import datasets, layers, models"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: tensorflow-gpu==2.0.0-beta1 in /usr/local/lib/python3.6/dist-packages (2.0.0b1)\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-beta1) (0.7.1)\n",
            "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-beta1) (3.7.1)\n",
            "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-beta1) (1.11.2)\n",
            "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-beta1) (0.1.7)\n",
            "Requirement already satisfied: tf-estimator-nightly<1.14.0.dev2019060502,>=1.14.0.dev2019060501 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-beta1) (1.14.0.dev2019060501)\n",
            "Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-beta1) (0.2.2)\n",
            "Requirement already satisfied: tb-nightly<1.14.0a20190604,>=1.14.0a20190603 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-beta1) (1.14.0a20190603)\n",
            "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-beta1) (0.8.0)\n",
            "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-beta1) (1.0.8)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-beta1) (0.33.4)\n",
            "Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-beta1) (1.16.4)\n",
            "Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-beta1) (1.12.0)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-beta1) (1.15.0)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-beta1) (1.1.0)\n",
            "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow-gpu==2.0.0-beta1) (1.1.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.6.1->tensorflow-gpu==2.0.0-beta1) (41.0.1)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tb-nightly<1.14.0a20190604,>=1.14.0a20190603->tensorflow-gpu==2.0.0-beta1) (0.15.5)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tb-nightly<1.14.0a20190604,>=1.14.0a20190603->tensorflow-gpu==2.0.0-beta1) (3.1.1)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.6->tensorflow-gpu==2.0.0-beta1) (2.8.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BIxrv51RgREg",
        "colab_type": "text"
      },
      "source": [
        "CNNs are used extensively in Computer Vision.  They are also used for modeling temporal data.\n",
        "\n",
        "\n",
        "The key idea in CNNs is to **capture local patterns in the data through a convolution operation and then to downsample the resulting output with the pooling layer**.   CNNs employ a series of convolution-pooling layers.\n",
        "\n",
        "\n",
        "Let's understand these operations in the context of image classification application.  In this case CNNs take an image as an input and predicts the corresponding digit.  The image is encoded as a 3d tensor with axes corresponding to height, width and depth of an image.  \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5gUtFxD6ZWyv",
        "colab_type": "text"
      },
      "source": [
        "### Download and prepare the MNIST dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "827UekN8ZpOk",
        "colab_type": "text"
      },
      "source": [
        "In the case of MNIST data, we are dealing with grey scale images and hence number of depth is 1 while height and width are both 28.  We will download and preprocess MNIST handwritten digit recognition dataset just as before except for a small change.   Here we reshape each image into a 3D tensor so that the input becomes a 4D tensor.  This is done to satisfy the input requirements of the CNNs.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AilreVGtZWdq",
        "colab_type": "code",
        "outputId": "e8e35ee4-e63c-4b01-cffd-71cdfebfd464",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 263
        }
      },
      "source": [
        "(train_images, train_labels), (test_images, test_labels) = datasets.mnist.load_data()\n",
        "\n",
        "print (\"Before reshaping:\")\n",
        "print (\"=================\")\n",
        "print (\"Number of axes in train_images: \", train_images.ndim)\n",
        "print (\"Number of axes in test_images: \", test_images.ndim)\n",
        "print (\"Shape of train_images: \", train_images.shape)\n",
        "print (\"Shape of test_images: \", test_images.shape)\n",
        "print ()\n",
        "\n",
        "train_images = train_images.reshape((60000, 28, 28, 1))\n",
        "test_images = test_images.reshape((10000, 28, 28, 1))\n",
        "\n",
        "print (\"After reshaping:\")\n",
        "print (\"=================\")\n",
        "print (\"Number of axes in train_images: \", train_images.ndim)\n",
        "print (\"Number of axes in test_images: \", test_images.ndim)\n",
        "print (\"Shape of train_images: \", train_images.shape)\n",
        "print (\"Shape of test_images: \", test_images.shape)\n",
        "print ()\n",
        "\n",
        "# Normalize pixel values to be between 0 and 1\n",
        "train_images, test_images = train_images / 255.0, test_images / 255.0"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Before reshaping:\n",
            "=================\n",
            "Number of axes in train_images:  3\n",
            "Number of axes in test_images:  3\n",
            "Shape of train_images:  (60000, 28, 28)\n",
            "Shape of test_images:  (10000, 28, 28)\n",
            "\n",
            "After reshaping:\n",
            "=================\n",
            "Number of axes in train_images:  4\n",
            "Number of axes in test_images:  4\n",
            "Shape of train_images:  (60000, 28, 28, 1)\n",
            "Shape of test_images:  (10000, 28, 28, 1)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mo6qGLv2glAk",
        "colab_type": "text"
      },
      "source": [
        "We print the number of axes and shape of train and test image tensors before and after the reshaping.  You can observe that the reshaping operation converts the input 3D tensor into the 4D tensor.   An additional axes is added in each image.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B9oxJoUFhiY-",
        "colab_type": "text"
      },
      "source": [
        "### Create the convolutional base"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1LnT4isci1DN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = models.Sequential()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dfrpnvVvh_Ov",
        "colab_type": "text"
      },
      "source": [
        "The 6 lines of code below define the convolutional base using a common pattern: a stack of Conv2D and MaxPooling2D layers.\n",
        "\n",
        "As input, a CNN takes tensors of shape (image_height, image_width, color_channels), ignoring the batch size. If you are new to color channels, MNIST has one (because the images are grayscale), whereas a color image has three (R,G,B). In this example, we will configure our CNN to process inputs of shape (28, 28, 1), which is the format of MNIST images. We do this by passing the argument input_shape to our first layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RFWcrrLNZ2z-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M-d2N3fXiyq3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.add(layers.MaxPooling2D((2, 2)))\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LXcPgH0si3R8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
        "model.add(layers.MaxPooling2D((2, 2)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H3y5gxTHi5J5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "model.add(layers.Conv2D(64, (3, 3), activation='relu'))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EZ_O0T_cih9y",
        "colab_type": "text"
      },
      "source": [
        "Let display the architecture of our model so far.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uR4JEw9TilHx",
        "colab_type": "code",
        "outputId": "d3f9cb5d-abf5-4028-f3ba-458c53338fb4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 334
        }
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_2\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv2d_2 (Conv2D)            (None, 26, 26, 32)        320       \n",
            "_________________________________________________________________\n",
            "max_pooling2d_2 (MaxPooling2 (None, 13, 13, 32)        0         \n",
            "_________________________________________________________________\n",
            "conv2d_3 (Conv2D)            (None, 11, 11, 64)        18496     \n",
            "_________________________________________________________________\n",
            "max_pooling2d_3 (MaxPooling2 (None, 5, 5, 64)          0         \n",
            "_________________________________________________________________\n",
            "conv2d_4 (Conv2D)            (None, 3, 3, 64)          36928     \n",
            "=================================================================\n",
            "Total params: 55,744\n",
            "Trainable params: 55,744\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VhFGyf-tlspP",
        "colab_type": "text"
      },
      "source": [
        "Above, you can see that the output of every Conv2D and MaxPooling2D layer is a 3D tensor of shape (height, width, channels). The width and height dimensions tend to shrink as we go deeper in the network. The number of output channels for each Conv2D layer is controlled by the first argument (e.g., 32 or 64). Typically, as the width and height shrink, we can afford (computationally) to add more output channels in each Conv2D layer."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cl5OBB2plw_S",
        "colab_type": "text"
      },
      "source": [
        "### Add Dense layers on top\n",
        "To complete our model, we will feed the last output tensor from the convolutional base (of shape (3, 3, 64)) into one or more Dense layers to perform classification. Dense layers take vectors as input (which are 1D), while the current output is a 3D tensor. First, we will flatten (or unroll) the 3D output to 1D,  then add one or more Dense layers on top. MNIST has 10 output classes, so we use a final Dense layer with 10 outputs and a softmax activation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l6z6_BHqloxG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.add(layers.Flatten())\n",
        "model.add(layers.Dense(64, activation='relu'))\n",
        "model.add(layers.Dense(10, activation='softmax'))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VbiedRiGl4pb",
        "colab_type": "text"
      },
      "source": [
        " Here's the complete architecture of our model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q6fsEkQRl5XD",
        "colab_type": "code",
        "outputId": "50871374-3651-4c60-a31f-1fdbc926dfcb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 439
        }
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_2\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv2d_2 (Conv2D)            (None, 26, 26, 32)        320       \n",
            "_________________________________________________________________\n",
            "max_pooling2d_2 (MaxPooling2 (None, 13, 13, 32)        0         \n",
            "_________________________________________________________________\n",
            "conv2d_3 (Conv2D)            (None, 11, 11, 64)        18496     \n",
            "_________________________________________________________________\n",
            "max_pooling2d_3 (MaxPooling2 (None, 5, 5, 64)          0         \n",
            "_________________________________________________________________\n",
            "conv2d_4 (Conv2D)            (None, 3, 3, 64)          36928     \n",
            "_________________________________________________________________\n",
            "flatten (Flatten)            (None, 576)               0         \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 64)                36928     \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 10)                650       \n",
            "=================================================================\n",
            "Total params: 93,322\n",
            "Trainable params: 93,322\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QvXYKIIZl-hK",
        "colab_type": "text"
      },
      "source": [
        "As you can see, our (3, 3, 64) outputs were flattened into vectors of shape (576) before going through two Dense layers."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z23ks72QmAMP",
        "colab_type": "text"
      },
      "source": [
        "### Compile and train the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HTH9XL__mF7D",
        "colab_type": "code",
        "outputId": "5264233e-a707-4bd0-a8cb-018973f20743",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 318
        }
      },
      "source": [
        "model.compile(optimizer='adam',\n",
        "              loss='sparse_categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "model.fit(train_images, train_labels, epochs=5)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING: Logging before flag parsing goes to stderr.\n",
            "W0725 11:30:09.988453 139963343931264 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 60000 samples\n",
            "Epoch 1/5\n",
            "60000/60000 [==============================] - 10s 171us/sample - loss: 0.1478 - accuracy: 0.9538\n",
            "Epoch 2/5\n",
            "60000/60000 [==============================] - 6s 107us/sample - loss: 0.0479 - accuracy: 0.9851\n",
            "Epoch 3/5\n",
            "60000/60000 [==============================] - 7s 109us/sample - loss: 0.0341 - accuracy: 0.9894\n",
            "Epoch 4/5\n",
            "60000/60000 [==============================] - 6s 105us/sample - loss: 0.0270 - accuracy: 0.9915\n",
            "Epoch 5/5\n",
            "60000/60000 [==============================] - 6s 106us/sample - loss: 0.0215 - accuracy: 0.9934\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f4b4015c630>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qfRjg3LWmJNp",
        "colab_type": "text"
      },
      "source": [
        "### Evaluate the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MgpufciamMvq",
        "colab_type": "code",
        "outputId": "2a0c8154-8fd3-4ba8-ecd1-bdeed190aeed",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "test_loss, test_acc = model.evaluate(test_images, test_labels)\n",
        "print(test_acc)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "10000/10000 [==============================] - 1s 68us/sample - loss: 0.0345 - accuracy: 0.9904\n",
            "0.9904\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ndnn0R1Onfg7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NjRSlgmVmSaq",
        "colab_type": "text"
      },
      "source": [
        "As you can see, our simple CNN has achieved a test accuracy of over 99%. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s_vEJnfhmVup",
        "colab_type": "text"
      },
      "source": [
        "### Classification with Feedforward Neural Networks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NsVlUu8Xma5A",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "mnist = tf.keras.datasets.mnist\n",
        "\n",
        "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
        "x_train, x_test = x_train / 255.0, x_test / 255.0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n23ZLUMsmn9f",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model_dnn = tf.keras.models.Sequential([\n",
        "  tf.keras.layers.Flatten(input_shape=(28, 28)),\n",
        "  tf.keras.layers.Dense(128, activation='relu'),\n",
        "  tf.keras.layers.Dropout(0.2),\n",
        "  tf.keras.layers.Dense(10, activation='softmax')\n",
        "])\n",
        "\n",
        "model_dnn.compile(optimizer='adam',\n",
        "              loss='sparse_categorical_crossentropy',\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DeGXUa6vmurm",
        "colab_type": "code",
        "outputId": "82040a3c-d1d9-417d-ec0b-15af65b67aaf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 298
        }
      },
      "source": [
        "model_dnn.summary()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_3\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "flatten_1 (Flatten)          (None, 784)               0         \n",
            "_________________________________________________________________\n",
            "dense_2 (Dense)              (None, 128)               100480    \n",
            "_________________________________________________________________\n",
            "dropout (Dropout)            (None, 128)               0         \n",
            "_________________________________________________________________\n",
            "dense_3 (Dense)              (None, 10)                1290      \n",
            "=================================================================\n",
            "Total params: 101,770\n",
            "Trainable params: 101,770\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VJYa4PvRnBC6",
        "colab_type": "code",
        "outputId": "09c671dd-3266-4850-a0bf-c32f0c2bdedb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 243
        }
      },
      "source": [
        "model_dnn.fit(x_train, y_train, epochs=5)\n",
        "model_dnn.evaluate(x_test, y_test)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 60000 samples\n",
            "Epoch 1/5\n",
            "60000/60000 [==============================] - 6s 93us/sample - loss: 0.3023 - accuracy: 0.9122\n",
            "Epoch 2/5\n",
            "60000/60000 [==============================] - 6s 93us/sample - loss: 0.1455 - accuracy: 0.9573\n",
            "Epoch 3/5\n",
            "60000/60000 [==============================] - 6s 93us/sample - loss: 0.1077 - accuracy: 0.9679\n",
            "Epoch 4/5\n",
            "60000/60000 [==============================] - 6s 92us/sample - loss: 0.0892 - accuracy: 0.9722\n",
            "Epoch 5/5\n",
            "60000/60000 [==============================] - 6s 94us/sample - loss: 0.0774 - accuracy: 0.9751\n",
            "10000/10000 [==============================] - 1s 57us/sample - loss: 0.0764 - accuracy: 0.9791\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.07640231210165657, 0.9791]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    }
  ]
}