helloworld_keras_tuner.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "helloworld_keras-tuner.ipynb",
      "provenance": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyM7Iec48xr2I8hi2uhD7MoM",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mrm8488/shared_colab_notebooks/blob/master/helloworld_keras_tuner.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "veujC3mizbSj",
        "colab_type": "text"
      },
      "source": [
        "# Keras Tuner\n",
        "\n",
        "A hyperparameter tuner for [Keras](https://keras.io), specifically for `tf.keras` with TensorFlow 2.0.\n",
        "\n",
        "Full documentation and tutorials available on the [Keras Tuner website.](https://keras-team.github.io/keras-tuner/)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xqLMguwLj2K1",
        "colab_type": "text"
      },
      "source": [
        "### A simple helloworld example\n",
        "### Different workflows are shown here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mrKn4-bLj_KM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "56265a03-f17c-459a-93a3-b753a9ee45e7"
      },
      "source": [
        "try:\n",
        "  # %tensorflow_version only exists in Colab.\n",
        "  %tensorflow_version 2.x\n",
        "except Exception:\n",
        "  pass\n"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TensorFlow 2.x selected.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-3FDCgqtkkPy",
        "colab_type": "text"
      },
      "source": [
        "## 1. Install keras-tuner"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q_TecrqtkMq3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 310
        },
        "outputId": "0ddbfc99-6cc2-4e86-d39b-2e7d36d9e1ce"
      },
      "source": [
        "!pip install -U keras-tuner"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already up-to-date: keras-tuner in /usr/local/lib/python3.6/dist-packages (1.0.1)\n",
            "Requirement already satisfied, skipping upgrade: tabulate in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (0.8.6)\n",
            "Requirement already satisfied, skipping upgrade: terminaltables in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (3.1.0)\n",
            "Requirement already satisfied, skipping upgrade: requests in /tensorflow-2.1.0/python3.6 (from keras-tuner) (2.22.0)\n",
            "Requirement already satisfied, skipping upgrade: future in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (0.16.0)\n",
            "Requirement already satisfied, skipping upgrade: scipy in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (1.4.1)\n",
            "Requirement already satisfied, skipping upgrade: colorama in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (0.4.3)\n",
            "Requirement already satisfied, skipping upgrade: tqdm in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (4.28.1)\n",
            "Requirement already satisfied, skipping upgrade: numpy in /tensorflow-2.1.0/python3.6 (from keras-tuner) (1.18.1)\n",
            "Requirement already satisfied, skipping upgrade: scikit-learn in /usr/local/lib/python3.6/dist-packages (from keras-tuner) (0.22.1)\n",
            "Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /tensorflow-2.1.0/python3.6 (from requests->keras-tuner) (2019.11.28)\n",
            "Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /tensorflow-2.1.0/python3.6 (from requests->keras-tuner) (3.0.4)\n",
            "Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /tensorflow-2.1.0/python3.6 (from requests->keras-tuner) (2.8)\n",
            "Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /tensorflow-2.1.0/python3.6 (from requests->keras-tuner) (1.25.7)\n",
            "Requirement already satisfied, skipping upgrade: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->keras-tuner) (0.14.1)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZAl60URbkqNv",
        "colab_type": "text"
      },
      "source": [
        "## 2. Import required libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V8D63_sdkWQ_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers\n",
        "\n",
        "\n",
        "from kerastuner.tuners import RandomSearch\n",
        "from kerastuner.engine.hypermodel import HyperModel\n",
        "from kerastuner.engine.hyperparameters import HyperParameters"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ORfy6AgCkvx8",
        "colab_type": "text"
      },
      "source": [
        "## 3. Load dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rkxIyeB-kiKx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "(x, y), (val_x, val_y) = keras.datasets.mnist.load_data()\n",
        "x = x.astype('float32') / 255.\n",
        "val_x = val_x.astype('float32') / 255.\n",
        "\n",
        "x = x[:10000]\n",
        "y = y[:10000]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CunRembYk9rK",
        "colab_type": "text"
      },
      "source": [
        "## 4. Usage"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XhDKU4DplFUH",
        "colab_type": "text"
      },
      "source": [
        "### 4.1 Basic usage\n",
        "\n",
        "- We define a `build_model` function\n",
        "- It returns a compiled model\n",
        "- It uses hyperparameters defined on the fly"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fhQu4BU1lMNH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def build_model(hp):\n",
        "    model = keras.Sequential()\n",
        "    model.add(layers.Flatten(input_shape=(28, 28)))\n",
        "    for i in range(hp.Int('num_layers', 2, 20)):\n",
        "        model.add(layers.Dense(units=hp.Int('units_' + str(i), 32, 512, 32),\n",
        "                               activation='relu'))\n",
        "    model.add(layers.Dense(10, activation='softmax'))\n",
        "    model.compile(\n",
        "        optimizer=keras.optimizers.Adam(\n",
        "            hp.Choice('learning_rate', [1e-2, 1e-3, 1e-4])),\n",
        "        loss='sparse_categorical_crossentropy',\n",
        "        metrics=['accuracy'])\n",
        "    return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nccdvqozlMsc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tuner = RandomSearch(\n",
        "    build_model,\n",
        "    objective='val_accuracy',\n",
        "    max_trials=5,\n",
        "    executions_per_trial=3,\n",
        "    directory='test_dir',\n",
        "    project_name='helloworld')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Myhx5aWslqTq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 528
        },
        "outputId": "c300603a-ce37-47a6-f74b-ae2f8e2e1e98"
      },
      "source": [
        "tuner.search_space_summary()"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Search space summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Default search space size: 4</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">num_layers (Int)</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-default: None</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-max_value: 20</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-min_value: 2</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-sampling: None</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-step: 1</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">units_0 (Int)</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-default: None</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-max_value: 512</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-min_value: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-sampling: None</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-step: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">units_1 (Int)</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-default: None</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-max_value: 512</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-min_value: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-sampling: None</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-step: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">learning_rate (Choice)</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-default: 0.01</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-ordered: True</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-values: [0.01, 0.001, 0.0001]</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CqqitwawltwT",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "520b2f1c-bec7-4c56-b144-4d3a122f92c4"
      },
      "source": [
        "tuner.search(x=x, y=y, epochs=3, validation_data=(val_x, val_y))"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/3\n",
            "10000/10000 [==============================] - 3s 319us/sample - loss: 2.0455 - accuracy: 0.1926 - val_loss: 1.6428 - val_accuracy: 0.3011\n",
            "Epoch 2/3\n",
            "10000/10000 [==============================] - 2s 222us/sample - loss: 1.3102 - accuracy: 0.4784 - val_loss: 1.2893 - val_accuracy: 0.4917\n",
            "Epoch 3/3\n",
            "10000/10000 [==============================] - 2s 214us/sample - loss: 1.0460 - accuracy: 0.5852 - val_loss: 1.0564 - val_accuracy: 0.5867\n",
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/3\n",
            "10000/10000 [==============================] - 3s 291us/sample - loss: 2.1012 - accuracy: 0.2011 - val_loss: 1.9150 - val_accuracy: 0.2590\n",
            "Epoch 2/3\n",
            "10000/10000 [==============================] - 2s 209us/sample - loss: 1.6428 - accuracy: 0.3597 - val_loss: 1.4301 - val_accuracy: 0.4257\n",
            "Epoch 3/3\n",
            "10000/10000 [==============================] - 2s 212us/sample - loss: 1.2417 - accuracy: 0.5404 - val_loss: 1.2266 - val_accuracy: 0.5714\n",
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/3\n",
            "10000/10000 [==============================] - 3s 292us/sample - loss: 2.0575 - accuracy: 0.2086 - val_loss: 1.6333 - val_accuracy: 0.3113\n",
            "Epoch 2/3\n",
            "10000/10000 [==============================] - 2s 211us/sample - loss: 1.4054 - accuracy: 0.3423 - val_loss: 1.2619 - val_accuracy: 0.4066\n",
            "Epoch 3/3\n",
            "10000/10000 [==============================] - 2s 223us/sample - loss: 1.0236 - accuracy: 0.5969 - val_loss: 0.8858 - val_accuracy: 0.6715\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 4f5d730191eb8a5482bdc7702f711a79</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.6098666787147522</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.0001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 18</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 288</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_16: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_17: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_8: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_9: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/3\n",
            "10000/10000 [==============================] - 3s 279us/sample - loss: 2.3262 - accuracy: 0.1070 - val_loss: 2.3019 - val_accuracy: 0.1135\n",
            "Epoch 2/3\n",
            "10000/10000 [==============================] - 2s 208us/sample - loss: 2.3019 - accuracy: 0.1062 - val_loss: 2.3027 - val_accuracy: 0.1135\n",
            "Epoch 3/3\n",
            "10000/10000 [==============================] - 2s 207us/sample - loss: 2.3015 - accuracy: 0.1120 - val_loss: 2.3025 - val_accuracy: 0.1135\n",
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/3\n",
            "10000/10000 [==============================] - 3s 267us/sample - loss: 2.3191 - accuracy: 0.1104 - val_loss: 2.3021 - val_accuracy: 0.1135\n",
            "Epoch 2/3\n",
            "10000/10000 [==============================] - 2s 203us/sample - loss: 2.3020 - accuracy: 0.1087 - val_loss: 2.3018 - val_accuracy: 0.1135\n",
            "Epoch 3/3\n",
            "10000/10000 [==============================] - 2s 204us/sample - loss: 2.3022 - accuracy: 0.1106 - val_loss: 2.3018 - val_accuracy: 0.1135\n",
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/3\n",
            "10000/10000 [==============================] - 3s 275us/sample - loss: 2.3226 - accuracy: 0.1092 - val_loss: 2.3020 - val_accuracy: 0.1028\n",
            "Epoch 2/3\n",
            "10000/10000 [==============================] - 2s 199us/sample - loss: 2.3021 - accuracy: 0.1122 - val_loss: 2.3025 - val_accuracy: 0.1135\n",
            "Epoch 3/3\n",
            "10000/10000 [==============================] - 2s 204us/sample - loss: 2.3021 - accuracy: 0.1099 - val_loss: 2.3017 - val_accuracy: 0.1135\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: d36569062a498848f6f996cd8ac8e92a</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.11349999904632568</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.01</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 15</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_16: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_17: 512</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 448</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_8: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_9: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/3\n",
            "10000/10000 [==============================] - 3s 251us/sample - loss: 0.7355 - accuracy: 0.7515 - val_loss: 0.4377 - val_accuracy: 0.8887\n",
            "Epoch 2/3\n",
            "10000/10000 [==============================] - 2s 190us/sample - loss: 0.3326 - accuracy: 0.9169 - val_loss: 0.3285 - val_accuracy: 0.9216\n",
            "Epoch 3/3\n",
            "10000/10000 [==============================] - 2s 194us/sample - loss: 0.2451 - accuracy: 0.9404 - val_loss: 0.2454 - val_accuracy: 0.9386\n",
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/3\n",
            "10000/10000 [==============================] - 2s 243us/sample - loss: 0.8170 - accuracy: 0.7071 - val_loss: 0.7163 - val_accuracy: 0.7686\n",
            "Epoch 2/3\n",
            "10000/10000 [==============================] - 2s 188us/sample - loss: 0.3721 - accuracy: 0.9030 - val_loss: 0.3077 - val_accuracy: 0.9192\n",
            "Epoch 3/3\n",
            "10000/10000 [==============================] - 2s 197us/sample - loss: 0.2378 - accuracy: 0.9429 - val_loss: 0.2283 - val_accuracy: 0.9409\n",
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/3\n",
            "10000/10000 [==============================] - 2s 246us/sample - loss: 0.7097 - accuracy: 0.7712 - val_loss: 0.4007 - val_accuracy: 0.8980\n",
            "Epoch 2/3\n",
            "10000/10000 [==============================] - 2s 191us/sample - loss: 0.3283 - accuracy: 0.9200 - val_loss: 0.3311 - val_accuracy: 0.9168\n",
            "Epoch 3/3\n",
            "10000/10000 [==============================] - 2s 186us/sample - loss: 0.2569 - accuracy: 0.9384 - val_loss: 0.2924 - val_accuracy: 0.9293\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: b27cea37c56a13d55e29baa0ac15d913</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9362666606903076</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 11</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 512</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_16: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_17: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 128</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 288</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_8: 512</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_9: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/3\n",
            "10000/10000 [==============================] - 2s 205us/sample - loss: 1.1838 - accuracy: 0.6579 - val_loss: 0.4263 - val_accuracy: 0.8798\n",
            "Epoch 2/3\n",
            "10000/10000 [==============================] - 2s 165us/sample - loss: 0.3428 - accuracy: 0.9021 - val_loss: 0.3148 - val_accuracy: 0.9091\n",
            "Epoch 3/3\n",
            "10000/10000 [==============================] - 2s 167us/sample - loss: 0.2519 - accuracy: 0.9277 - val_loss: 0.2929 - val_accuracy: 0.9105\n",
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/3\n",
            "10000/10000 [==============================] - 2s 200us/sample - loss: 1.1339 - accuracy: 0.6449 - val_loss: 0.4265 - val_accuracy: 0.8815\n",
            "Epoch 2/3\n",
            "10000/10000 [==============================] - 2s 163us/sample - loss: 0.3452 - accuracy: 0.9031 - val_loss: 0.3151 - val_accuracy: 0.9094\n",
            "Epoch 3/3\n",
            "10000/10000 [==============================] - 2s 160us/sample - loss: 0.2543 - accuracy: 0.9287 - val_loss: 0.2620 - val_accuracy: 0.9240\n",
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/3\n",
            "10000/10000 [==============================] - 2s 210us/sample - loss: 1.1318 - accuracy: 0.6898 - val_loss: 0.4992 - val_accuracy: 0.8547\n",
            "Epoch 2/3\n",
            "10000/10000 [==============================] - 2s 164us/sample - loss: 0.3671 - accuracy: 0.8968 - val_loss: 0.3215 - val_accuracy: 0.9076\n",
            "Epoch 3/3\n",
            "10000/10000 [==============================] - 2s 166us/sample - loss: 0.2676 - accuracy: 0.9241 - val_loss: 0.2887 - val_accuracy: 0.9153\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 068e7666349c0bff13eca7cc10165da4</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9166000485420227</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.0001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 5</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 448</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 128</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_16: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_17: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 288</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 288</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_8: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_9: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/3\n",
            "10000/10000 [==============================] - 2s 208us/sample - loss: 0.5108 - accuracy: 0.8397 - val_loss: 0.3188 - val_accuracy: 0.9059\n",
            "Epoch 2/3\n",
            "10000/10000 [==============================] - 2s 166us/sample - loss: 0.2103 - accuracy: 0.9377 - val_loss: 0.2238 - val_accuracy: 0.9329\n",
            "Epoch 3/3\n",
            "10000/10000 [==============================] - 2s 168us/sample - loss: 0.1406 - accuracy: 0.9574 - val_loss: 0.2049 - val_accuracy: 0.9371\n",
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/3\n",
            "10000/10000 [==============================] - 2s 209us/sample - loss: 0.5185 - accuracy: 0.8372 - val_loss: 0.3065 - val_accuracy: 0.9040\n",
            "Epoch 2/3\n",
            "10000/10000 [==============================] - 2s 165us/sample - loss: 0.2101 - accuracy: 0.9363 - val_loss: 0.2254 - val_accuracy: 0.9291\n",
            "Epoch 3/3\n",
            "10000/10000 [==============================] - 2s 170us/sample - loss: 0.1427 - accuracy: 0.9573 - val_loss: 0.2016 - val_accuracy: 0.9387\n",
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/3\n",
            "10000/10000 [==============================] - 2s 206us/sample - loss: 0.5209 - accuracy: 0.8410 - val_loss: 0.2869 - val_accuracy: 0.9118\n",
            "Epoch 2/3\n",
            "10000/10000 [==============================] - 2s 161us/sample - loss: 0.2083 - accuracy: 0.9382 - val_loss: 0.2117 - val_accuracy: 0.9340\n",
            "Epoch 3/3\n",
            "10000/10000 [==============================] - 2s 168us/sample - loss: 0.1409 - accuracy: 0.9593 - val_loss: 0.2289 - val_accuracy: 0.9334\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 7d97518f88a2bbc1df2bb30e351af375</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9366000294685364</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 5</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 128</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_16: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_17: 448</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 288</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_8: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_9: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Oracle triggered exit\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RaJjtPJ9mLfP",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "773baf82-de15-4f76-f7e9-68bd9b77d704"
      },
      "source": [
        "tuner.results_summary()"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Results summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Results in test_dir/helloworld</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Showing 10 best trials</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Objective(name='val_accuracy', direction='max')</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 7d97518f88a2bbc1df2bb30e351af375</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9366000294685364</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 5</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 128</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_16: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_17: 448</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 288</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_8: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_9: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: b27cea37c56a13d55e29baa0ac15d913</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9362666606903076</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 11</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 512</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_16: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_17: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 128</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 288</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_8: 512</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_9: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 068e7666349c0bff13eca7cc10165da4</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9166000485420227</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.0001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 5</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 448</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 128</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_16: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_17: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 288</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 288</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_8: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_9: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 4f5d730191eb8a5482bdc7702f711a79</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.6098666787147522</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.0001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 18</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 288</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_16: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_17: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_8: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_9: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: d36569062a498848f6f996cd8ac8e92a</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.11349999904632568</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.01</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 15</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_16: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_17: 512</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 448</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_8: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_9: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PnzVhZEcmUzw",
        "colab_type": "text"
      },
      "source": [
        "### 4.2 We override the loss and metrics\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OK80C79ZmzIE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tuner = RandomSearch(\n",
        "    build_model,\n",
        "    objective='val_accuracy',\n",
        "    loss=keras.losses.SparseCategoricalCrossentropy(name='my_loss'),\n",
        "    metrics=['accuracy', 'mse'],\n",
        "    max_trials=5,\n",
        "    directory='test_dir',\n",
        "    project_name='helloworld2')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1K4nNc_xm3fj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "a842c503-aee0-4683-e96d-b0d6ce42beb6"
      },
      "source": [
        "tuner.search(x=x, y=y, epochs=5, validation_data=(val_x, val_y))"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 3s 278us/sample - loss: 2.0697 - accuracy: 0.2453 - mse: 27.1467 - val_loss: 1.6866 - val_accuracy: 0.3332 - val_mse: 27.2620\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 214us/sample - loss: 1.3720 - accuracy: 0.4770 - mse: 27.1656 - val_loss: 1.2037 - val_accuracy: 0.5582 - val_mse: 27.2799\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 214us/sample - loss: 0.9330 - accuracy: 0.6802 - mse: 27.1813 - val_loss: 0.7973 - val_accuracy: 0.7348 - val_mse: 27.2963\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 215us/sample - loss: 0.7074 - accuracy: 0.7675 - mse: 27.1936 - val_loss: 0.6685 - val_accuracy: 0.7836 - val_mse: 27.3030\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 214us/sample - loss: 0.5949 - accuracy: 0.8163 - mse: 27.2001 - val_loss: 0.5853 - val_accuracy: 0.8250 - val_mse: 27.3081\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 56b69c59faab8e7a8abdb2c86cbdcbe4</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.824999988079071</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.0001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 13</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_2: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_3: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_4: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_5: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_6: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_7: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_8: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_9: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 2s 210us/sample - loss: 0.5521 - accuracy: 0.8389 - mse: 27.2120 - val_loss: 0.4450 - val_accuracy: 0.8878 - val_mse: 27.3292\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 171us/sample - loss: 0.3669 - accuracy: 0.9083 - mse: 27.2206 - val_loss: 0.4560 - val_accuracy: 0.8875 - val_mse: 27.3261\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 168us/sample - loss: 0.2935 - accuracy: 0.9295 - mse: 27.2227 - val_loss: 0.3569 - val_accuracy: 0.9087 - val_mse: 27.3315\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 170us/sample - loss: 0.2466 - accuracy: 0.9354 - mse: 27.2241 - val_loss: 0.3850 - val_accuracy: 0.9016 - val_mse: 27.3308\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 178us/sample - loss: 0.2743 - accuracy: 0.9322 - mse: 27.2245 - val_loss: 0.3238 - val_accuracy: 0.9266 - val_mse: 27.3310\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 6e63568f8d382c5e06101f0dea8bd9b0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9265999794006348</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.01</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 3</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_2: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_3: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_4: 288</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_5: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_6: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_7: 448</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_8: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_9: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 3s 313us/sample - loss: 2.3041 - accuracy: 0.1069 - mse: 27.1430 - val_loss: 2.3020 - val_accuracy: 0.1135 - val_mse: 27.2504\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 233us/sample - loss: 2.3021 - accuracy: 0.1077 - mse: 27.1429 - val_loss: 2.3022 - val_accuracy: 0.1135 - val_mse: 27.2504\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 232us/sample - loss: 2.3024 - accuracy: 0.1049 - mse: 27.1429 - val_loss: 2.3020 - val_accuracy: 0.1135 - val_mse: 27.2504\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 236us/sample - loss: 2.3023 - accuracy: 0.1126 - mse: 27.1429 - val_loss: 2.3025 - val_accuracy: 0.1028 - val_mse: 27.2504\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 229us/sample - loss: 2.3016 - accuracy: 0.1057 - mse: 27.1429 - val_loss: 2.3017 - val_accuracy: 0.1135 - val_mse: 27.2504\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: f0e89b1c6126d88f7ba1344eae90d663</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.11349999904632568</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.01</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 18</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_16: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_17: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 448</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 448</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_8: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_9: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 2s 227us/sample - loss: 0.5322 - accuracy: 0.8320 - mse: 27.2074 - val_loss: 0.2436 - val_accuracy: 0.9257 - val_mse: 27.3276\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 193us/sample - loss: 0.2092 - accuracy: 0.9402 - mse: 27.2233 - val_loss: 0.2064 - val_accuracy: 0.9351 - val_mse: 27.3320\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 183us/sample - loss: 0.1430 - accuracy: 0.9575 - mse: 27.2263 - val_loss: 0.2061 - val_accuracy: 0.9400 - val_mse: 27.3350\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 182us/sample - loss: 0.1070 - accuracy: 0.9671 - mse: 27.2282 - val_loss: 0.2106 - val_accuracy: 0.9389 - val_mse: 27.3341\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 195us/sample - loss: 0.0839 - accuracy: 0.9734 - mse: 27.2289 - val_loss: 0.1965 - val_accuracy: 0.9465 - val_mse: 27.3358\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: eaaf3407d72ce0ac29e668f1ca11b44e</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9465000033378601</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 6</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 128</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_16: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_17: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 512</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_8: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_9: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 3s 313us/sample - loss: 2.3143 - accuracy: 0.1133 - mse: 27.1433 - val_loss: 2.3031 - val_accuracy: 0.1135 - val_mse: 27.2504\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 226us/sample - loss: 2.3022 - accuracy: 0.1091 - mse: 27.1429 - val_loss: 2.3021 - val_accuracy: 0.1135 - val_mse: 27.2504\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 240us/sample - loss: 2.3019 - accuracy: 0.1057 - mse: 27.1429 - val_loss: 2.3025 - val_accuracy: 0.1135 - val_mse: 27.2504\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 243us/sample - loss: 2.3024 - accuracy: 0.1090 - mse: 27.1429 - val_loss: 2.3022 - val_accuracy: 0.1135 - val_mse: 27.2504\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 244us/sample - loss: 2.3018 - accuracy: 0.1080 - mse: 27.1429 - val_loss: 2.3023 - val_accuracy: 0.1135 - val_mse: 27.2504\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: d109df290dd5cae189a5cc0425fdd32d</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.11349999904632568</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.01</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 16</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 448</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_16: 288</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_17: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 288</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_8: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_9: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Oracle triggered exit\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WfKmYJqUnXMZ",
        "colab_type": "text"
      },
      "source": [
        "### 4.3 We define a HyperModel subclass\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mYR7p4CXndYm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class MyHyperModel(HyperModel):\n",
        "\n",
        "    def __init__(self, img_size, num_classes):\n",
        "        self.img_size = img_size\n",
        "        self.num_classes = num_classes\n",
        "\n",
        "    def build(self, hp):\n",
        "        model = keras.Sequential()\n",
        "        model.add(layers.Flatten(input_shape=self.img_size))\n",
        "        for i in range(hp.Int('num_layers', 2, 20)):\n",
        "            model.add(layers.Dense(units=hp.Int('units_' + str(i), 32, 512, 32),\n",
        "                                   activation='relu'))\n",
        "        model.add(layers.Dense(self.num_classes, activation='softmax'))\n",
        "        model.compile(\n",
        "            optimizer=keras.optimizers.Adam(\n",
        "                hp.Choice('learning_rate', [1e-2, 1e-3, 1e-4])),\n",
        "            loss='sparse_categorical_crossentropy',\n",
        "            metrics=['accuracy'])\n",
        "        return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uNpwEF-8n3YS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tuner = RandomSearch(\n",
        "    MyHyperModel(img_size=(28, 28), num_classes=10),\n",
        "    objective='val_accuracy',\n",
        "    max_trials=5,\n",
        "    directory='test_dir',\n",
        "    project_name='helloworld3')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7fx7Fg9noDB5",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "a983f160-004a-462c-bbbf-72b9c984598d"
      },
      "source": [
        "tuner.search(x=x,y=y,epochs=5,validation_data=(val_x, val_y))"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 2s 228us/sample - loss: 0.8039 - accuracy: 0.7413 - val_loss: 0.3588 - val_accuracy: 0.8966\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 175us/sample - loss: 0.2703 - accuracy: 0.9230 - val_loss: 0.2409 - val_accuracy: 0.9296\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 172us/sample - loss: 0.1841 - accuracy: 0.9478 - val_loss: 0.2268 - val_accuracy: 0.9366\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 177us/sample - loss: 0.1320 - accuracy: 0.9617 - val_loss: 0.2368 - val_accuracy: 0.9323\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 179us/sample - loss: 0.1171 - accuracy: 0.9671 - val_loss: 0.1935 - val_accuracy: 0.9456\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 93484c0f362f82e241d9d8f09b3e6600</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9455999732017517</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 8</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 3s 304us/sample - loss: 1.7141 - accuracy: 0.2897 - val_loss: 1.3500 - val_accuracy: 0.4540\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 221us/sample - loss: 1.0279 - accuracy: 0.6263 - val_loss: 0.8371 - val_accuracy: 0.7294\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 220us/sample - loss: 0.7176 - accuracy: 0.7949 - val_loss: 0.7068 - val_accuracy: 0.8191\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 221us/sample - loss: 0.7854 - accuracy: 0.7572 - val_loss: 0.7486 - val_accuracy: 0.7282\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 214us/sample - loss: 0.7320 - accuracy: 0.7235 - val_loss: 0.8080 - val_accuracy: 0.7229\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 593188e1e4228ccd399af462098d5fa5</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.819100022315979</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 19</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_16: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_17: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_18: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_2: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_3: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_4: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_5: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_6: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_7: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_8: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_9: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 2s 194us/sample - loss: 1.2023 - accuracy: 0.6974 - val_loss: 0.4870 - val_accuracy: 0.8700\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 161us/sample - loss: 0.3874 - accuracy: 0.8940 - val_loss: 0.3358 - val_accuracy: 0.9047\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 160us/sample - loss: 0.2912 - accuracy: 0.9189 - val_loss: 0.2969 - val_accuracy: 0.9169\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 163us/sample - loss: 0.2434 - accuracy: 0.9333 - val_loss: 0.2673 - val_accuracy: 0.9220\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 162us/sample - loss: 0.2128 - accuracy: 0.9390 - val_loss: 0.2487 - val_accuracy: 0.9270\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: adee2cda932118f4cd7cd1531ba42a9d</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9269999861717224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.0001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 3</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_16: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_17: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_18: 288</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_2: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_3: 128</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_4: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_5: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_6: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_7: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_8: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_9: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 3s 275us/sample - loss: 2.3352 - accuracy: 0.1079 - val_loss: 2.3022 - val_accuracy: 0.1028\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 203us/sample - loss: 2.3021 - accuracy: 0.1091 - val_loss: 2.3028 - val_accuracy: 0.1135\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 202us/sample - loss: 2.3022 - accuracy: 0.1087 - val_loss: 2.3014 - val_accuracy: 0.1135\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 203us/sample - loss: 2.3015 - accuracy: 0.1110 - val_loss: 2.3028 - val_accuracy: 0.1028\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 199us/sample - loss: 2.3022 - accuracy: 0.1105 - val_loss: 2.3014 - val_accuracy: 0.1135\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 59137b3c78f70ca4a2b24218e01e8076</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.11349999904632568</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.01</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 14</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 448</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_16: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_17: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_18: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_2: 512</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_3: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_4: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_5: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_6: 448</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_7: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_8: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_9: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 2s 216us/sample - loss: 0.8835 - accuracy: 0.7440 - val_loss: 0.3397 - val_accuracy: 0.9023\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 172us/sample - loss: 0.2780 - accuracy: 0.9190 - val_loss: 0.2567 - val_accuracy: 0.9233\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 174us/sample - loss: 0.1987 - accuracy: 0.9421 - val_loss: 0.2517 - val_accuracy: 0.9284\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 176us/sample - loss: 0.1582 - accuracy: 0.9514 - val_loss: 0.2170 - val_accuracy: 0.9358\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 174us/sample - loss: 0.1224 - accuracy: 0.9651 - val_loss: 0.2127 - val_accuracy: 0.9372\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 2c95ad19130ca0f32485df3377a795e1</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9372000098228455</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.0001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 6</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 288</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_16: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_17: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_18: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_2: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_3: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_4: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_5: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_6: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_7: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_8: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_9: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Oracle triggered exit\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s0iQneevoOnc",
        "colab_type": "text"
      },
      "source": [
        "### 4.4 We restrict the search space\n",
        "- This means that default values are being used for params that are left out\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1gmVTL6Qod42",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "e0ed1f4b-a72e-4844-cb41-7300142bca10"
      },
      "source": [
        "hp = HyperParameters()\n",
        "hp.Choice('learning_rate', [1e-1, 1e-3])"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.1"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1tHL7yuBog3n",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tuner = RandomSearch(\n",
        "    build_model,\n",
        "    max_trials=5,\n",
        "    hyperparameters=hp,\n",
        "    tune_new_entries=False,\n",
        "    objective='val_accuracy',\n",
        "    directory='test_dir',\n",
        "    project_name='helloworld4')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BmS3ylv0okDG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 894
        },
        "outputId": "214e3635-3399-47cb-a5e5-f87f47c0959b"
      },
      "source": [
        "tuner.search(x=x, y=y, epochs=5, validation_data=(val_x, val_y))"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 2s 186us/sample - loss: 0.7432 - accuracy: 0.7824 - val_loss: 0.3444 - val_accuracy: 0.9004\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 154us/sample - loss: 0.2879 - accuracy: 0.9201 - val_loss: 0.2798 - val_accuracy: 0.9168\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 152us/sample - loss: 0.2253 - accuracy: 0.9345 - val_loss: 0.2641 - val_accuracy: 0.9221\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 153us/sample - loss: 0.1907 - accuracy: 0.9441 - val_loss: 0.2408 - val_accuracy: 0.9268\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 155us/sample - loss: 0.1661 - accuracy: 0.9506 - val_loss: 0.2321 - val_accuracy: 0.9297\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 7a2ffd3ef8023bc9a5ed61b929c52eb8</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9297000169754028</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 2</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 2s 185us/sample - loss: 1.3191 - accuracy: 0.5360 - val_loss: 1.0105 - val_accuracy: 0.6369\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 151us/sample - loss: 1.1033 - accuracy: 0.5981 - val_loss: 1.1338 - val_accuracy: 0.6073\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 152us/sample - loss: 1.4182 - accuracy: 0.4563 - val_loss: 1.8369 - val_accuracy: 0.3637\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 153us/sample - loss: 1.4824 - accuracy: 0.4359 - val_loss: 1.7240 - val_accuracy: 0.3577\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 154us/sample - loss: 1.6798 - accuracy: 0.3502 - val_loss: 1.6967 - val_accuracy: 0.3484\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 6fe721f5727d95f79112fc1880b0d65f</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.636900007724762</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.1</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 2</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Oracle triggered exit\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lrUAAhn2opJK",
        "colab_type": "text"
      },
      "source": [
        "### 4.5 We override specific parameters with fixed values that aren't the default"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Po9x5j2ZvYJW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tuner = RandomSearch(\n",
        "    build_model,\n",
        "    max_trials=5,\n",
        "    hyperparameters=hp,\n",
        "    tune_new_entries=True,\n",
        "    objective='val_accuracy',\n",
        "    directory='test_dir',\n",
        "    project_name='helloworld5')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EPcnd5Wsvl2J",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "5f2649f1-c393-4413-cb98-c2739d8ae27d"
      },
      "source": [
        "tuner.search(x=x, y=y, epochs=5, validation_data=(val_x, val_y))"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 3s 293us/sample - loss: 1.4778 - accuracy: 0.3929 - val_loss: 1.0899 - val_accuracy: 0.6163\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 215us/sample - loss: 0.7366 - accuracy: 0.7648 - val_loss: 0.5028 - val_accuracy: 0.8756\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 216us/sample - loss: 0.3756 - accuracy: 0.9086 - val_loss: 0.3552 - val_accuracy: 0.9120\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 214us/sample - loss: 0.2867 - accuracy: 0.9295 - val_loss: 0.2886 - val_accuracy: 0.9272\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 217us/sample - loss: 0.2472 - accuracy: 0.9416 - val_loss: 0.2710 - val_accuracy: 0.9350\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 04549200b2ee69fdbd53f1ad0c37bf57</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9350000023841858</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 16</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 448</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_8: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_9: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 3s 288us/sample - loss: 275609.7257 - accuracy: 0.1068 - val_loss: 2.3119 - val_accuracy: 0.0958\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 213us/sample - loss: 2.3101 - accuracy: 0.1049 - val_loss: 2.3149 - val_accuracy: 0.0980\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 215us/sample - loss: 2.3129 - accuracy: 0.1024 - val_loss: 2.3094 - val_accuracy: 0.1009\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 214us/sample - loss: 2.3098 - accuracy: 0.1033 - val_loss: 2.3075 - val_accuracy: 0.1135\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 213us/sample - loss: 2.3120 - accuracy: 0.1019 - val_loss: 2.3108 - val_accuracy: 0.0974\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 49ddb80c16828fde7b0fa748fb624239</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.11349999904632568</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.1</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 16</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 512</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 288</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 448</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_8: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_9: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 3s 291us/sample - loss: 19542144963.1396 - accuracy: 0.1077 - val_loss: 2.3100 - val_accuracy: 0.0958\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 215us/sample - loss: 2.3088 - accuracy: 0.1088 - val_loss: 2.3057 - val_accuracy: 0.1135\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 212us/sample - loss: 2.3107 - accuracy: 0.1042 - val_loss: 2.3100 - val_accuracy: 0.1135\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 216us/sample - loss: 2.3139 - accuracy: 0.1001 - val_loss: 2.3095 - val_accuracy: 0.1010\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 212us/sample - loss: 2.3129 - accuracy: 0.0963 - val_loss: 2.3126 - val_accuracy: 0.1135\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 24bd42b54234a2d70775bb4ec8dd351f</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.11349999904632568</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.1</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 16</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_8: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_9: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 2s 201us/sample - loss: 32.7340 - accuracy: 0.1024 - val_loss: 2.3115 - val_accuracy: 0.1028\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 161us/sample - loss: 2.3121 - accuracy: 0.1068 - val_loss: 2.3224 - val_accuracy: 0.1010\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 161us/sample - loss: 2.3099 - accuracy: 0.1098 - val_loss: 2.3109 - val_accuracy: 0.1135\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 163us/sample - loss: 2.3115 - accuracy: 0.1052 - val_loss: 2.3140 - val_accuracy: 0.0892\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 168us/sample - loss: 2.3138 - accuracy: 0.0976 - val_loss: 2.3076 - val_accuracy: 0.1135\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 954e65c5beeac316e2b8d31e70c47e11</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.11349999904632568</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.1</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 4</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 448</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 96</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 448</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 448</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_8: 416</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_9: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 2s 215us/sample - loss: 74.8611 - accuracy: 0.1014 - val_loss: 2.3065 - val_accuracy: 0.1032\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 169us/sample - loss: 2.3135 - accuracy: 0.1034 - val_loss: 2.3079 - val_accuracy: 0.0982\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 172us/sample - loss: 2.3111 - accuracy: 0.1028 - val_loss: 2.3096 - val_accuracy: 0.1009\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 171us/sample - loss: 2.3134 - accuracy: 0.1051 - val_loss: 2.3058 - val_accuracy: 0.1135\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 169us/sample - loss: 2.3125 - accuracy: 0.1026 - val_loss: 2.3428 - val_accuracy: 0.1028\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 1650a5ea7a0d544f0e32de21d31b594a</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.11349999904632568</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.1</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 6</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_10: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_11: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_12: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_13: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_14: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_15: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_6: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_7: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_8: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_9: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Oracle triggered exit\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-5NkEoxPvtm0",
        "colab_type": "text"
      },
      "source": [
        "### 4.6  We reparameterize the search space\n",
        "- This means that we override the distribution of specific hyperparameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZDfGA2Odv4Fk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "d3da4650-b72f-4592-f0dd-6341e71e9596"
      },
      "source": [
        "hp = HyperParameters()\n",
        "hp.Choice('learning_rate', [1e-1, 1e-3])"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.1"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ap3M3Vfbv-c9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tuner = RandomSearch(\n",
        "    build_model,\n",
        "    max_trials=5,\n",
        "    hyperparameters=hp,\n",
        "    tune_new_entries=True,\n",
        "    objective='val_accuracy',\n",
        "    directory='test_dir',\n",
        "    project_name='helloworld6')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "akzI_s_XwJo8",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "c2d09d90-1e0f-4ffa-bcbe-957c605f9128"
      },
      "source": [
        "tuner.search(x=x, y=y, epochs=5, validation_data=(val_x, val_y))"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 2s 213us/sample - loss: 0.6607 - accuracy: 0.7969 - val_loss: 0.2812 - val_accuracy: 0.9173\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 179us/sample - loss: 0.2166 - accuracy: 0.9376 - val_loss: 0.2485 - val_accuracy: 0.9278\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 178us/sample - loss: 0.1466 - accuracy: 0.9561 - val_loss: 0.1753 - val_accuracy: 0.9492\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 172us/sample - loss: 0.1013 - accuracy: 0.9707 - val_loss: 0.1829 - val_accuracy: 0.9483\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 171us/sample - loss: 0.0800 - accuracy: 0.9756 - val_loss: 0.2322 - val_accuracy: 0.9412\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: ce418686dc6fe80f588a25b7e0e9fb1e</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9491999745368958</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 6</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 32</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 2s 196us/sample - loss: 10.0208 - accuracy: 0.1605 - val_loss: 2.1226 - val_accuracy: 0.1710\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 163us/sample - loss: 2.1784 - accuracy: 0.1738 - val_loss: 2.2283 - val_accuracy: 0.1577\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 157us/sample - loss: 2.0870 - accuracy: 0.1944 - val_loss: 2.0981 - val_accuracy: 0.1875\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 161us/sample - loss: 2.0829 - accuracy: 0.1883 - val_loss: 2.1024 - val_accuracy: 0.1855\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 159us/sample - loss: 2.0905 - accuracy: 0.1890 - val_loss: 2.0924 - val_accuracy: 0.1884\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 2a6a046696373a1a020ee8735405de70</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.188400000333786</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.1</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 3</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 448</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 288</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 2s 220us/sample - loss: 0.4808 - accuracy: 0.8505 - val_loss: 0.2543 - val_accuracy: 0.9233\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 176us/sample - loss: 0.2043 - accuracy: 0.9415 - val_loss: 0.2206 - val_accuracy: 0.9391\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 172us/sample - loss: 0.1385 - accuracy: 0.9574 - val_loss: 0.2552 - val_accuracy: 0.9276\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 177us/sample - loss: 0.1156 - accuracy: 0.9676 - val_loss: 0.1740 - val_accuracy: 0.9513\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 172us/sample - loss: 0.0789 - accuracy: 0.9752 - val_loss: 0.2634 - val_accuracy: 0.9410\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: d21b030d3b3e9be00b3194780201bb4e</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9513000249862671</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 6</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 512</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 320</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 2s 212us/sample - loss: 316.5196 - accuracy: 0.1104 - val_loss: 2.3078 - val_accuracy: 0.1010\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 173us/sample - loss: 2.3072 - accuracy: 0.1046 - val_loss: 2.3085 - val_accuracy: 0.1135\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 168us/sample - loss: 2.3120 - accuracy: 0.1004 - val_loss: 2.3101 - val_accuracy: 0.1028\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 173us/sample - loss: 2.3115 - accuracy: 0.1036 - val_loss: 2.3056 - val_accuracy: 0.1028\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 170us/sample - loss: 2.3103 - accuracy: 0.1015 - val_loss: 2.3110 - val_accuracy: 0.1028\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 0b51e149bae7a6cc961ad70711871344</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.11349999904632568</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.1</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 6</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 256</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 224</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 2s 190us/sample - loss: 10.6260 - accuracy: 0.1069 - val_loss: 2.3082 - val_accuracy: 0.0980\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 157us/sample - loss: 2.3164 - accuracy: 0.0976 - val_loss: 2.3073 - val_accuracy: 0.0958\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 154us/sample - loss: 2.3111 - accuracy: 0.1088 - val_loss: 2.3071 - val_accuracy: 0.1010\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 166us/sample - loss: 2.3121 - accuracy: 0.1048 - val_loss: 2.3253 - val_accuracy: 0.1028\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 163us/sample - loss: 2.3145 - accuracy: 0.1001 - val_loss: 2.3408 - val_accuracy: 0.1032\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: bc118fa2efd63cc9e807e6e1d5055e97</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.10320000350475311</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.1</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 3</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_0: 384</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_1: 352</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_2: 480</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_3: 192</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-units_4: 64</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-units_5: 160</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Oracle triggered exit\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0s3X9KsEy0kX",
        "colab_type": "text"
      },
      "source": [
        "### 4.7 We predefine the search space\n",
        "- No unregistered parameters are allowed in `build`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aBxP6yrbzD4_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "d95a39a7-b1c0-4728-bb67-59ba5155b8e2"
      },
      "source": [
        "hp = HyperParameters()\n",
        "hp.Choice('learning_rate', [1e-1, 1e-3])\n",
        "hp.Int('num_layers', 2, 20)"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SrAH33M9zJiO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def build_model(hp):\n",
        "    model = keras.Sequential()\n",
        "    model.add(layers.Flatten(input_shape=(28, 28)))\n",
        "    for i in range(hp.get('num_layers')):\n",
        "        model.add(layers.Dense(32,\n",
        "                               activation='relu'))\n",
        "    model.add(layers.Dense(10, activation='softmax'))\n",
        "    model.compile(\n",
        "        optimizer=keras.optimizers.Adam(hp.get('learning_rate')),\n",
        "        loss='sparse_categorical_crossentropy',\n",
        "        metrics=['accuracy'])\n",
        "    return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uUIOPgYHzNd0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tuner = RandomSearch(\n",
        "    build_model,\n",
        "    max_trials=5,\n",
        "    hyperparameters=hp,\n",
        "    allow_new_entries=False,\n",
        "    objective='val_accuracy',\n",
        "    directory='test_dir',\n",
        "    project_name='helloworld7')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SnETHU4ozP1c",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "884a81f3-c9e1-44d5-e3fe-e5e5c1c989b7"
      },
      "source": [
        "tuner.search(x=x, y=y, epochs=5, validation_data=(val_x, val_y))"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 3s 293us/sample - loss: 2.9968 - accuracy: 0.1037 - val_loss: 2.3239 - val_accuracy: 0.0980\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 215us/sample - loss: 2.3146 - accuracy: 0.0995 - val_loss: 2.3064 - val_accuracy: 0.1010\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 233us/sample - loss: 2.3144 - accuracy: 0.1051 - val_loss: 2.3056 - val_accuracy: 0.1135\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 220us/sample - loss: 2.3142 - accuracy: 0.0996 - val_loss: 2.3086 - val_accuracy: 0.0980\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 212us/sample - loss: 2.3121 - accuracy: 0.1056 - val_loss: 2.3100 - val_accuracy: 0.1135\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 7b7d461ef8f82badf72903445cfe0dce</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.11349999904632568</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.1</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 17</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 3s 267us/sample - loss: 2.3565 - accuracy: 0.1091 - val_loss: 2.3138 - val_accuracy: 0.1135\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 203us/sample - loss: 2.3135 - accuracy: 0.1066 - val_loss: 2.3141 - val_accuracy: 0.0980\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 208us/sample - loss: 2.3112 - accuracy: 0.1045 - val_loss: 2.3117 - val_accuracy: 0.0982\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 215us/sample - loss: 2.3137 - accuracy: 0.0985 - val_loss: 2.3128 - val_accuracy: 0.1135\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 211us/sample - loss: 2.3153 - accuracy: 0.1047 - val_loss: 2.3123 - val_accuracy: 0.1010\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 92cf88ec07c4e6a89e939917a0492dc0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.11349999904632568</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.1</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 14</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 3s 289us/sample - loss: 1.4767 - accuracy: 0.4228 - val_loss: 0.7896 - val_accuracy: 0.6886\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 210us/sample - loss: 0.5939 - accuracy: 0.8071 - val_loss: 0.5428 - val_accuracy: 0.8425\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 207us/sample - loss: 0.3797 - accuracy: 0.8910 - val_loss: 0.4278 - val_accuracy: 0.8796\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 203us/sample - loss: 0.2852 - accuracy: 0.9211 - val_loss: 0.3488 - val_accuracy: 0.9087\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 205us/sample - loss: 0.2284 - accuracy: 0.9397 - val_loss: 0.3689 - val_accuracy: 0.9015\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 353fea6d5f062030a5dea110234c1e9d</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9086999893188477</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 14</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 3s 288us/sample - loss: 1.6500 - accuracy: 0.3384 - val_loss: 1.1636 - val_accuracy: 0.5977\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 213us/sample - loss: 0.8995 - accuracy: 0.6780 - val_loss: 0.8047 - val_accuracy: 0.7311\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 212us/sample - loss: 0.6493 - accuracy: 0.7551 - val_loss: 0.6171 - val_accuracy: 0.7788\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 214us/sample - loss: 0.5324 - accuracy: 0.8136 - val_loss: 0.5934 - val_accuracy: 0.7960\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 209us/sample - loss: 0.4328 - accuracy: 0.8489 - val_loss: 0.5006 - val_accuracy: 0.8457\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: f94560810d23539fa648faf9875f7a25</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.8457000255584717</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 16</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 10000 samples, validate on 10000 samples\n",
            "Epoch 1/5\n",
            "10000/10000 [==============================] - 2s 218us/sample - loss: 0.9403 - accuracy: 0.6780 - val_loss: 0.5038 - val_accuracy: 0.8482\n",
            "Epoch 2/5\n",
            "10000/10000 [==============================] - 2s 170us/sample - loss: 0.3494 - accuracy: 0.9012 - val_loss: 0.3439 - val_accuracy: 0.9023\n",
            "Epoch 3/5\n",
            "10000/10000 [==============================] - 2s 168us/sample - loss: 0.2446 - accuracy: 0.9307 - val_loss: 0.2994 - val_accuracy: 0.9132\n",
            "Epoch 4/5\n",
            "10000/10000 [==============================] - 2s 167us/sample - loss: 0.1973 - accuracy: 0.9451 - val_loss: 0.2707 - val_accuracy: 0.9197\n",
            "Epoch 5/5\n",
            "10000/10000 [==============================] - 2s 170us/sample - loss: 0.1597 - accuracy: 0.9542 - val_loss: 0.2443 - val_accuracy: 0.9298\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial complete</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#4527A0\"><h1 style=\"font-size:18px\">Trial summary</h1></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Trial ID: 9c3ba1464695c97974d99a670f296a2f</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Score: 0.9297999739646912</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-Best step: 0</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:#7E57C2\"><h2 style=\"font-size:16px\">Hyperparameters:</h2></span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:cyan\"> |-learning_rate: 0.001</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<span style=\"color:blue\"> |-num_layers: 6</span>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Oracle triggered exit\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}