DL_Cov2D.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "DL_Cov2D.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMr3DoLtuPXjdNEYDJAgKX1",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/wenxuan0923/My-notes/blob/master/DL_Cov2D.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bziWpHyyBLwY",
        "colab_type": "text"
      },
      "source": [
        "# The basics of convnets\n",
        "This note will mainly focus on **feature maps**, **convolution** and **max pooling**. The following questions will be answered along the way:\n",
        "\n",
        "* What does a Conv2D layer do\n",
        "* What does a MaxPooling2D layer do\n",
        "* How to interpret the output shape\n",
        "* How to implement a simple convolutional neural network\n",
        "\n",
        "We will continue using the well known MNIST dataset of handwritten digits for illustration. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UXe_5ATJC0v2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import keras\n",
        "from keras import models\n",
        "from keras import layers\n",
        "from keras.datasets import mnist\n",
        "from keras.utils import to_categorical"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rF06GSfGEzPs",
        "colab_type": "code",
        "outputId": "f6827d82-d576-4e13-c14f-b7db3f2bc084",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
        "print(x_train.shape)\n",
        "print(y_train.shape)"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz\n",
            "11493376/11490434 [==============================] - 2s 0us/step\n",
            "(60000, 28, 28)\n",
            "(60000,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DaImmzeiCdkp",
        "colab_type": "text"
      },
      "source": [
        "Before diving deep in to the CNN model, we need to process the data to make sure it is in the form that a convolutional neural network model would expect. To be more specific, we need to:\n",
        "\n",
        "**1. Convert x_train and x_test into 4D tensors** \n",
        "of shape (samples, height, width, color_depth), which can be processed by Conv2D layers \n",
        "\n",
        "**2. Scale the data so that all values are in the [0, 1] interval** to make it easier for our model to converge\n",
        "\n",
        "**3. Categorically encode the labels using one-hot encoder**, which can be easily done using the `keras.utils.to_categorical()` function, so that each target variable will be a 10 dimensional vector consists of 0s and 1"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NG9xjy6FG4a4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def preprocess_input(x):\n",
        "    desired_shape = (-1, 28, 28, 1)\n",
        "    return x.reshape(desired_shape) / 255.0\n",
        "\n",
        "def preprocess_output(y):\n",
        "    return keras.utils.to_categorical(y)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sQCQhqb0HTEu",
        "colab_type": "code",
        "outputId": "635e7521-e524-4b4e-97cf-56d2ee73c2ea",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "x_train, x_test = map(preprocess_input, [x_train, x_test])\n",
        "y_train, y_test = map(preprocess_output, [y_train, y_test])\n",
        "\n",
        "print(x_train.shape)\n",
        "print(y_train.shape)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(60000, 28, 28, 1)\n",
            "(60000, 10)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bg0MH1xdWGpJ",
        "colab_type": "text"
      },
      "source": [
        "## What does a Cov2D layer do?\n",
        "\n",
        "When you look at most images, you will notice there is a lot of wasted space in each image. While there are 28*28=784 pixels, it will be interesting to see if there is a way to condense the image down to the important features. That is where convolutions come in."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "suUQN5I50KGI",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "So how does Con2D work to extract informative information? Let's break down the concept little by little:\n",
        "* Convolutions operate over 3D tensors called **feature maps** \n",
        "> In our case it is (28, 28, 1), which represents (height, width, input_depth) respectively\n",
        "\n",
        "* The input_depth axis is also called **channels** axis\n",
        "> * For an RGB image, it has 3 channels: Red, Green and Blue\n",
        "> * For a black-and-white image, like the MNIST dights, the depth is 1: gray\n",
        "\n",
        "* The Conv2D layer extracts 3D patches of size `(patch_window_height, patch_window_width, input_depth)` from its input feature map\n",
        "> It slides a window of size `(patch_window_height, patch_window_width)` over the input feature map and stop at all possible locations. Each of these locations defines a patch\n",
        "\n",
        "* A transformation will be applied to each of these 3D patches to convert them into 1D vector of shape `(output_depth, )`\n",
        "> This means it generates 1 value for each output_depth \n",
        "\n",
        "* The output_depth can be any number, and it does not represent color channel anymore, they stand for **filters** instead\n",
        "\n",
        "* Each filter encode specific aspect of the input data by applying different transforamtion\n",
        "\n",
        "* All of these 1D vector are then *spatially* reassembled into a 3D **output feature map** of shape (height, width, output_depth)\n",
        "\n",
        "\n",
        "* The `patch_window_height`, `patch_window_width` and `output_depth` are defined in the layers as input arguments: <br>\n",
        " > `layers.Conv2D(output_depth, (patch_window_height, patch_window_width))`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ePe0RUND9SIF",
        "colab_type": "text"
      },
      "source": [
        "### Example 1: A naive example with `input_depth = 1` and `output_depth = 2`\n",
        "Consider a naive example, say we have a **input feature map of shape (3, 3, 1)**, a grayscale image, and we defined the Conv2D layer to be  `layers.Conv2D(2, (2, 2))`. By sliding a window of size (2, 2) over the input feature map at all possible locations, we will be able to extract **four** 3D patches and each of them has size (2, 2, 1). Extact 1 value will be generated for each patch at each output_depth and reassembled them spatially will get us a 3D output feature map of shape (2, 2, 2).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8hMpMq284mH3",
        "colab_type": "text"
      },
      "source": [
        "<p align=\"center\">\n",
        "<img src = 'https://drive.google.com/uc?id=1asIzBttuFQQ_XmGddtuw5l6DlqH5Lyav'\n",
        "height=\"440\" style=\"vertical-align:middle\"/>\n",
        "</p>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7FbJOI8cGq5f",
        "colab_type": "text"
      },
      "source": [
        "**Code it up using keras:**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rOxI0mc3Dezx",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "ee5703fd-d6d9-4b37-fe32-7b23762b3b24"
      },
      "source": [
        "model = models.Sequential()\n",
        "model.add(layers.Conv2D(2, (2, 2), input_shape=(3, 3, 1), activation='relu'))\n",
        "model.summary()"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_3\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv2d_5 (Conv2D)            (None, 2, 2, 2)           10        \n",
            "=================================================================\n",
            "Total params: 10\n",
            "Trainable params: 10\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xDU3jgXl4j5I",
        "colab_type": "text"
      },
      "source": [
        "Note the output from the Conv2D layer is (, 2, 2, 2), which is exactly what we expected."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x6x9NDLao9Cu",
        "colab_type": "text"
      },
      "source": [
        "### Example 2: A little complicated example with `input_depth = 2` and `output_depth = 3`\n",
        "\n",
        "Consider a little more complicated example, say we have a \n",
        "**input feature map of shape (5, 5, 2)**, and we defined the Conv2D layer to be  `layers.Conv2D(3, (3, 3))`. By sliding a window of size (3, 3) over the input feature map, we will be able to extract **nine** 3D patches. Each of them has size (3, 3, 2). Because of the output_depth = 3, each patch will output a 1D vector of shape (3, ). Reassemble them spatially will get us a 3D output feature map of shape (3, 3, 3).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "spPEGMSd1BLM",
        "colab_type": "text"
      },
      "source": [
        " <tr>\n",
        "    <td>    <img src = 'https://drive.google.com/uc?id=1X8H0LrLtZjRZ-vWICH0e4DjbDF9PFO5a'\n",
        "height=\"330\"/> </td>\n",
        "<td> <td> <td> <td> <td> <td> <td>\n",
        "    <td> <img src = 'https://drive.google.com/uc?id=1SVDdgtUrF-Hd9gjFKzsnyIlmD1u1nXsD'\n",
        "height=\"540\"/> </td>\n",
        "    </tr>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XHeQk5B1ttne",
        "colab_type": "text"
      },
      "source": [
        "**Code it up using keras:**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oCe6LaIJHcaK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "23fcf323-454c-47d0-bf8c-ddb5c3c3f6af"
      },
      "source": [
        "model = models.Sequential()\n",
        "model.add(layers.Conv2D(3, (3, 3), input_shape=(5, 5, 2), activation='relu'))\n",
        "model.summary()"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_6\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv2d_8 (Conv2D)            (None, 3, 3, 3)           57        \n",
            "=================================================================\n",
            "Total params: 57\n",
            "Trainable params: 57\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S72izST_FuMF",
        "colab_type": "text"
      },
      "source": [
        "### Example 3: Stacking multiple Conv2D layers\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6E8-ZIKvFckv",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 255
        },
        "outputId": "fb7b32ef-7106-4e45-87f3-a3044e23e9d9"
      },
      "source": [
        "model = models.Sequential()\n",
        "model.add(layers.Conv2D(32, (3, 3), input_shape=(28, 28, 1), activation='relu'))\n",
        "model.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
        "model.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
        "model.summary()"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_7\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv2d_9 (Conv2D)            (None, 26, 26, 32)        320       \n",
            "_________________________________________________________________\n",
            "conv2d_10 (Conv2D)           (None, 24, 24, 64)        18496     \n",
            "_________________________________________________________________\n",
            "conv2d_11 (Conv2D)           (None, 22, 22, 64)        36928     \n",
            "=================================================================\n",
            "Total params: 55,744\n",
            "Trainable params: 55,744\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z4EKc27dHapu",
        "colab_type": "text"
      },
      "source": [
        "From these examples above, we can see the output width and height may differ from the input width and height:\n",
        "* Example 1: (3, 3) $\\Rightarrow$ (2, 2)\n",
        "* Example 2: (5, 5) $\\Rightarrow$ (3, 3)\n",
        "* Example 3: (28, 28) $\\Rightarrow$ (26, 26) $\\Rightarrow$ (24, 24) $\\Rightarrow$ (22, 22) \n",
        "\n",
        "\n",
        "The input feature map shrinks a little after going through each Conv2D layer, because of the so called **border effect**. If you want to get an output feature map with the same spatial dimensions as the\n",
        "input, you can use **padding** argument in the Conv2D layers. We will take about it in another notes. For now, we will just use the default setting `padding = \"valid\"`, which means no padding. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ai3cIfWaHMo3",
        "colab_type": "text"
      },
      "source": [
        "## What does a MaxPooling2D layer do?\n",
        "\n",
        "When combing with some pooling method, the model we just built can become way more powerful.\n",
        "\n",
        "Pooling is a way of compressing an image. The MaxPooling2D layer aggressively downsample feature maps by extracting windows from the input feature maps and out putting the max value of each channel. \n",
        "> **Meaning of downsample**: to make a digital audio signal smaller by lowering its sampling rate or sample size\n",
        "\n",
        "A quick and easy way to do this is to go over the image of 4 pixels at a time, i.e. a 2*2 tensor, then it picks the biggest value and keep just that."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OmJzDzIjYdps",
        "colab_type": "text"
      },
      "source": [
        "<p align=\"center\">\n",
        "<img src = 'https://drive.google.com/uc?id=1ow2wcWhZSGNYgezsE1InLgaXS4nslM5G'\n",
        "height=\"420\" style=\"vertical-align:middle\"/>\n",
        "</p>\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CMxwUivwZQF7",
        "colab_type": "text"
      },
      "source": [
        "This method can be easily implemented by adding a `layers.MaxPooling2D((2, 2))` layer into our network. For every 4 pixels, only the biggest will survive."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fuVgaYdZZFgm",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "outputId": "a4f2d7ea-86f2-423d-9bba-2b384a0b9f34"
      },
      "source": [
        "model = models.Sequential()\n",
        "model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))\n",
        "model.add(layers.MaxPooling2D((2, 2)))\n",
        "model.summary()"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_9\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv2d_15 (Conv2D)           (None, 26, 26, 32)        320       \n",
            "_________________________________________________________________\n",
            "max_pooling2d_5 (MaxPooling2 (None, 13, 13, 32)        0         \n",
            "=================================================================\n",
            "Total params: 320\n",
            "Trainable params: 320\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ZtsEdWNcc-U",
        "colab_type": "text"
      },
      "source": [
        "Note: if you are wondering why the output shape change from (26, 26) to (13, 13):\n",
        "> $(26*26)/4 = 13*13 = 169$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YYXgRbG5ccIE",
        "colab_type": "text"
      },
      "source": [
        "We can then add another convolutional layer, and another max-pooling layer so that the network can further learn another set of convolutions on top of the existing one, and then again, pool to reduce the size. So, by the time the image gets to the flatten to go into the dense layers, it's already much smaller. It's being quartered, and then quartered again. So, its content has been greatly simplified, and thus mitigate overfitting.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vc72D2zRZFc2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 323
        },
        "outputId": "ae64e7d0-3b99-4d7e-a762-46e1f1cd1863"
      },
      "source": [
        "model = models.Sequential()\n",
        "model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))\n",
        "model.add(layers.MaxPooling2D((2, 2)))\n",
        "model.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
        "model.add(layers.MaxPooling2D((2,2)))\n",
        "model.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
        "model.summary()"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_10\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv2d_16 (Conv2D)           (None, 26, 26, 32)        320       \n",
            "_________________________________________________________________\n",
            "max_pooling2d_6 (MaxPooling2 (None, 13, 13, 32)        0         \n",
            "_________________________________________________________________\n",
            "conv2d_17 (Conv2D)           (None, 11, 11, 64)        18496     \n",
            "_________________________________________________________________\n",
            "max_pooling2d_7 (MaxPooling2 (None, 5, 5, 64)          0         \n",
            "_________________________________________________________________\n",
            "conv2d_18 (Conv2D)           (None, 3, 3, 64)          36928     \n",
            "=================================================================\n",
            "Total params: 55,744\n",
            "Trainable params: 55,744\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BtTb2yj8dDKG",
        "colab_type": "text"
      },
      "source": [
        "To make final prediction, we need to flatten out the 3D image and feed it into a dense layer, just like what we did in a regular deep neural net. For MNIST dataset, it is a classification problem with 10 classes, so we use softmax as activation function with 10 hidden units in the output layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lgt5PEJ0E0HP",
        "colab_type": "code",
        "outputId": "771fd024-5635-42d9-db07-734f5bc54e09",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 425
        }
      },
      "source": [
        "model = models.Sequential()\n",
        "model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))\n",
        "model.add(layers.MaxPooling2D((2, 2)))\n",
        "model.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
        "model.add(layers.MaxPooling2D((2,2)))\n",
        "model.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
        "model.add(layers.Flatten())\n",
        "model.add(layers.Dense(64, activation='relu'))\n",
        "model.add(layers.Dense(10, activation='softmax'))\n",
        "model.summary()"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_11\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv2d_19 (Conv2D)           (None, 26, 26, 32)        320       \n",
            "_________________________________________________________________\n",
            "max_pooling2d_8 (MaxPooling2 (None, 13, 13, 32)        0         \n",
            "_________________________________________________________________\n",
            "conv2d_20 (Conv2D)           (None, 11, 11, 64)        18496     \n",
            "_________________________________________________________________\n",
            "max_pooling2d_9 (MaxPooling2 (None, 5, 5, 64)          0         \n",
            "_________________________________________________________________\n",
            "conv2d_21 (Conv2D)           (None, 3, 3, 64)          36928     \n",
            "_________________________________________________________________\n",
            "flatten_2 (Flatten)          (None, 576)               0         \n",
            "_________________________________________________________________\n",
            "dense_3 (Dense)              (None, 64)                36928     \n",
            "_________________________________________________________________\n",
            "dense_4 (Dense)              (None, 10)                650       \n",
            "=================================================================\n",
            "Total params: 93,322\n",
            "Trainable params: 93,322\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QAblmO5aHL0A",
        "colab_type": "text"
      },
      "source": [
        "Note that `layers.Flatten()` function flatten the 3D outputs of shape (3, 3, 64) to 1D: a vector with 576 numbers (3 * 3 * 64 = 576).\n",
        "\n",
        "Now you just need to config the model using `model.compile` and fit it on the training data!\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dw4cpSamE0QD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(optimizer='rmsprop',\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4sG6AlZ1E0TE",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "54dfe512-c8e9-4f6e-efed-7b4388cf7ec0"
      },
      "source": [
        "history = model.fit(x_train, y_train, epochs=5, batch_size=64)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/5\n",
            "60000/60000 [==============================] - 45s 754us/step - loss: 0.4160 - accuracy: 0.8711\n",
            "Epoch 2/5\n",
            "60000/60000 [==============================] - 46s 759us/step - loss: 0.2398 - accuracy: 0.9263\n",
            "Epoch 3/5\n",
            "60000/60000 [==============================] - 46s 764us/step - loss: 0.1549 - accuracy: 0.9510\n",
            "Epoch 4/5\n",
            "60000/60000 [==============================] - 46s 760us/step - loss: 0.1119 - accuracy: 0.9644\n",
            "Epoch 5/5\n",
            "60000/60000 [==============================] - 46s 767us/step - loss: 0.0897 - accuracy: 0.9721\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1RJ_75NBE0OZ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "31eae875-18b0-481a-c49b-ba726134090c"
      },
      "source": [
        "history.history"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': [0.87105, 0.92635, 0.95105, 0.9643833, 0.97211665],\n",
              " 'loss': [0.4159620881001155,\n",
              "  0.23982162843942642,\n",
              "  0.15492084644238155,\n",
              "  0.11187756851116816,\n",
              "  0.08968276811937491]}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bUZgkPEeePwr",
        "colab_type": "text"
      },
      "source": [
        "Great! We have finished building our first convolutional neural net model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nLaDWyFVfFfw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}