Building_Autoencoders_in_Keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qMTDxDL7ZyNS"
   },
   "source": [
    "# Building Autoencoders in Keras\n",
    "\n",
    "## What are auto encoders?\n",
    "\n",
    "Auto encoders are used as compression and decompression algorithms which are learned from data instead of engineered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 237
    },
    "colab_type": "code",
    "id": "4j7hOeN5dKg2",
    "outputId": "ee7823c6-a7cf-41e4-b32f-b3e523a3ded5"
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "Image('https://blog.keras.io/img/ae/autoencoder_schema.jpg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZqEy3BYNZyNY"
   },
   "source": [
    "# Let's build the simplest autoencoder possible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "nHxETPXZZyNc",
    "outputId": "ce8e80f7-0050-4673-ef42-cdd35337092b"
   },
   "outputs": [],
   "source": [
    "from keras.layers import Input, Dense, Conv2D, MaxPooling2D, UpSampling2D\n",
    "from keras.models import Model\n",
    "\n",
    "# this is the size of our encoded representations\n",
    "encoding_dim = 32  # 32 floats -> compression of factor 24.5, assuming the input is 784 floats\n",
    "\n",
    "# this is our input placeholder\n",
    "input_img = Input(shape=(784,))\n",
    "# \"encoded\" is the encoded representation of the input\n",
    "encoded = Dense(encoding_dim, activation='relu')(input_img)\n",
    "# \"decoded\" is the lossy reconstruction of the input\n",
    "decoded = Dense(784, activation='sigmoid')(encoded)\n",
    "\n",
    "# this model maps an input to its reconstruction\n",
    "autoencoder = Model(input_img, decoded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jYMeiu4CZyNp"
   },
   "source": [
    "Let's create a separate encoder model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kRdYDCDFZyNr"
   },
   "outputs": [],
   "source": [
    "# this model maps an input to its encoded representation\n",
    "encoder = Model(input_img, encoded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BlR7zcrKZyNx"
   },
   "source": [
    "And a decoder model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "juHUlQuSZyNy"
   },
   "outputs": [],
   "source": [
    "# create a placeholder for an encoded (32-dimensional) input\n",
    "encoded_input = Input(shape=(encoding_dim,))\n",
    "# retrieve the last layer of the autoencoder model\n",
    "decoder_layer = autoencoder.layers[-1]\n",
    "# create the decoder model\n",
    "decoder = Model(encoded_input, decoder_layer(encoded_input))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pK-kvFmQZyN4"
   },
   "source": [
    "We will use a per-pixel binary crossentropy loss, which is generally used for binary classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "V0ww7ApAZyN5"
   },
   "outputs": [],
   "source": [
    "autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "G1v5yuhxZyN9"
   },
   "source": [
    "Load the MNIST data set, and discard the labels since we're only interested in encoding/decoding the images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "PShwswpWZyN-",
    "outputId": "baa26c59-6f76-4b11-a54e-80cdef9eae29"
   },
   "outputs": [],
   "source": [
    "from keras.datasets import mnist\n",
    "import numpy as np\n",
    "(x_train, _), (x_test, _) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2fz46oINZyOE"
   },
   "source": [
    "Show an example of an image in the data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 364
    },
    "colab_type": "code",
    "id": "JQDFsa2jZyOF",
    "outputId": "35cab28b-bd0b-4f1a-fc99-2bee5ec06d90"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.imshow(x_train[5, :, :])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lpqrUHswZyOJ"
   },
   "source": [
    "Normalize the training data to values between 0 and 1, and flatten the images into vectors of size 784."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "xOnphFBUZyOK",
    "outputId": "bae110dd-fe20-49c5-d3a2-a49991eb365c"
   },
   "outputs": [],
   "source": [
    "x_train = x_train.astype('float32') / 255.\n",
    "x_test = x_test.astype('float32') / 255.\n",
    "x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))\n",
    "x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))\n",
    "print(x_train.shape)\n",
    "print(x_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_0Yz9hIdZyOO"
   },
   "source": [
    "Lets train the encoder for 50 epochs (50 sweeps over the complete training set.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1734
    },
    "colab_type": "code",
    "id": "6xASuYSdZyOP",
    "outputId": "b4c4be0d-63dd-4396-fab7-c2a7e017c8f7",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "history = autoencoder.fit(x_train, x_train, epochs=50, batch_size=256, shuffle=True,\n",
    "                          validation_data=(x_test, x_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "H_k5F7zWZyOT"
   },
   "source": [
    "Make a plot of the convergence history of the training "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 514
    },
    "colab_type": "code",
    "id": "pLAbs6exZyOV",
    "outputId": "f4dd2cc4-ed45-4d01-b0be-82a265583363"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,8))\n",
    "plt.plot(history.history['loss'], label=\"Training loss\")\n",
    "plt.plot(history.history['val_loss'], label=\"Validation loss\")\n",
    "plt.grid(\"on\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XOyWY4-PZyOZ"
   },
   "source": [
    "The training loss and validation loss are starting to flatten out, indicating that the model is close to convergence. We can visualize the reconstructed inputs with the encoder and decoder network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ysjGKC8bZyOa"
   },
   "outputs": [],
   "source": [
    "# encode and decode some digits\n",
    "# note that we take them from the *test* set\n",
    "encoded_imgs = encoder.predict(x_test)\n",
    "decoded_imgs = decoder.predict(encoded_imgs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gFbKVdC2ZyOe"
   },
   "source": [
    "As shown in the figure, we are losing quite some detail with this basic approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 244
    },
    "colab_type": "code",
    "id": "OsgEj9CyZyOg",
    "outputId": "739e4c94-2a44-4786-8fb6-a53236bbe904"
   },
   "outputs": [],
   "source": [
    "def plot_digits(n, x_test, decoded_imgs):\n",
    "    plt.figure(figsize=(20, 4))\n",
    "    for i in range(n):\n",
    "        # display original\n",
    "        ax = plt.subplot(2, n, i + 1)\n",
    "        plt.imshow(x_test[i].reshape(28, 28))\n",
    "        plt.gray()\n",
    "        ax.get_xaxis().set_visible(False)\n",
    "        ax.get_yaxis().set_visible(False)\n",
    "\n",
    "        # display reconstruction\n",
    "        ax = plt.subplot(2, n, i + 1 + n)\n",
    "        plt.imshow(decoded_imgs[i].reshape(28, 28))\n",
    "        plt.gray()\n",
    "        ax.get_xaxis().set_visible(False)\n",
    "        ax.get_yaxis().set_visible(False)\n",
    "\n",
    "plot_digits(n=10, x_test=x_test, decoded_imgs=decoded_imgs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xjnzLDfQZyOl"
   },
   "source": [
    "# Deep autoencoder\n",
    "\n",
    "We do not have to limit ourselves to single layers as encoders and decoders. We can add more layers as follows.\n",
    "\n",
    "Try to change number of hidden units, and the number of layers to see how the architecture changes. Use `model.summary()` to print the shape of the input and outputs of each layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3_NPN8q2ZyOo"
   },
   "outputs": [],
   "source": [
    "input_img = Input(shape=(784,))\n",
    "encoded = Dense(128, activation='relu')(input_img)\n",
    "encoded = Dense(64, activation='relu')(encoded)\n",
    "encoded = Dense(32, activation='relu')(encoded)\n",
    "\n",
    "decoded = Dense(64, activation='relu')(encoded)\n",
    "decoded = Dense(128, activation='relu')(decoded)\n",
    "decoded = Dense(784, activation='sigmoid')(decoded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ieFfhE42ZyO1"
   },
   "source": [
    "Lets train the new model. We are now training for 100 epochs which results in a longer training time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 3434
    },
    "colab_type": "code",
    "id": "qgALHZNaZyO3",
    "outputId": "d252e9c9-0b33-4d09-d9f6-840d9802a5b5",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "autoencoder = Model(input_img, decoded)\n",
    "autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')\n",
    "\n",
    "history = autoencoder.fit(x_train, x_train, epochs=100, batch_size=256, shuffle=True,\n",
    "                          validation_data=(x_test, x_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LFIakUtJZyPA"
   },
   "source": [
    "Review the convergence history again to make sure that the model is converged and is not overfitting the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 514
    },
    "colab_type": "code",
    "id": "GvBvtNTwZyPC",
    "outputId": "0f84250b-eef4-4d2f-c437-d0cebe6b2dbf"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,8))\n",
    "plt.plot(history.history['loss'], label=\"Training loss\")\n",
    "plt.plot(history.history['val_loss'], label=\"Validation loss\")\n",
    "plt.grid(\"on\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "A7cyIFhNZyPF"
   },
   "source": [
    "The final loss is around 0.097, slightly better than our previous approach. Let's review the images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 244
    },
    "colab_type": "code",
    "id": "eI1MP200ZyPG",
    "outputId": "4e77adf8-6a1c-41a1-996a-02a6c255ec3f"
   },
   "outputs": [],
   "source": [
    "encoded_imgs = encoder.predict(x_test)\n",
    "decoded_imgs = decoder.predict(encoded_imgs)\n",
    "\n",
    "plot_digits(n=10, x_test=x_test, decoded_imgs=decoded_imgs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yY0z_-JJZyPJ"
   },
   "source": [
    "# Application to image denoising\n",
    "\n",
    "Let's train a new network which is able to remove noise from images. We simply add artificial noise to the training images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "06SGX3CRZyPK"
   },
   "outputs": [],
   "source": [
    "(x_train, _), (x_test, _) = mnist.load_data()\n",
    "\n",
    "x_train = x_train.astype('float32') / 255.\n",
    "x_test = x_test.astype('float32') / 255.\n",
    "x_train = np.reshape(x_train, (len(x_train), 28, 28, 1))  # adapt this if using `channels_first` image data format\n",
    "x_test = np.reshape(x_test, (len(x_test), 28, 28, 1))  # adapt this if using `channels_first` image data format\n",
    "\n",
    "noise_factor = 0.5\n",
    "x_train_noisy = x_train + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_train.shape) \n",
    "x_test_noisy = x_test + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_test.shape) \n",
    "\n",
    "x_train_noisy = np.clip(x_train_noisy, 0., 1.)\n",
    "x_test_noisy = np.clip(x_test_noisy, 0., 1.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hWamqrocZyPN"
   },
   "source": [
    "Let's review the noisy images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 125
    },
    "colab_type": "code",
    "id": "BvWGjVJCZyPP",
    "outputId": "37ee9c91-0b15-4c4a-cb86-9c7cabe90f83"
   },
   "outputs": [],
   "source": [
    "n = 10\n",
    "plt.figure(figsize=(20, 2))\n",
    "for i in range(n):\n",
    "    ax = plt.subplot(1, n, i + 1)\n",
    "    plt.imshow(x_test_noisy[i].reshape(28, 28))\n",
    "    plt.gray()\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kV_ohAf8ZyPT"
   },
   "source": [
    "Can our encoder remove this noise?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cajQgTKhZyPU"
   },
   "outputs": [],
   "source": [
    "input_img = Input(shape=(28, 28, 1))\n",
    "\n",
    "x = Conv2D(32, (3, 3), activation='relu', padding='same')(input_img)\n",
    "x = MaxPooling2D((2, 2), padding='same')(x)\n",
    "x = Conv2D(32, (3, 3), activation='relu', padding='same')(x)\n",
    "encoded = MaxPooling2D((2, 2), padding='same')(x)\n",
    "\n",
    "# at this point the representation is (7, 7, 32)\n",
    "\n",
    "x = Conv2D(32, (3, 3), activation='relu', padding='same')(encoded)\n",
    "x = UpSampling2D((2, 2))(x)\n",
    "x = Conv2D(32, (3, 3), activation='relu', padding='same')(x)\n",
    "x = UpSampling2D((2, 2))(x)\n",
    "decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)\n",
    "\n",
    "autoencoder = Model(input_img, decoded)\n",
    "autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5Gimc-5VZyPX"
   },
   "source": [
    "Training again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "id": "dccGYiipZyPX",
    "outputId": "6edb31b7-c29d-49e0-cb88-0ed4cfbbb833"
   },
   "outputs": [],
   "source": [
    "autoencoder.fit(x_train_noisy, x_train, epochs=2, batch_size=128, shuffle=True, validation_data=(x_test_noisy, x_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZT4Q0FGQZyPb"
   },
   "source": [
    "Finally review the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 244
    },
    "colab_type": "code",
    "id": "TC6-AaPGZyPf",
    "outputId": "e6fb3e22-0ea9-4148-f3db-97274155387d"
   },
   "outputs": [],
   "source": [
    "decoded_imgs = autoencoder.predict(x_test_noisy)\n",
    "\n",
    "plot_digits(n=10, x_test=x_test_noisy, decoded_imgs=decoded_imgs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7wxA0ktlZyPm"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Building Autoencoders in Keras",
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}