Monitor-ML-runs-live.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Monitor ML runs live \n",
    "\n",
    "## Introduction\n",
    "\n",
    "This guide will show you how to:\n",
    "\n",
    "* Monitor training and evaluation metrics and losses live\n",
    "* Monitor hardware resources during training\n",
    "\n",
    "By the end of it, you will monitor your metrics, losses, and hardware live in Neptune!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Install dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: neptune-client in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (0.4.119)\n",
      "Requirement already satisfied: tensorflow in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (2.3.0)\n",
      "Requirement already satisfied: GitPython>=2.0.8 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from neptune-client) (3.1.3)\n",
      "Requirement already satisfied: requests>=2.20.0 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from neptune-client) (2.24.0)\n",
      "Requirement already satisfied: six>=1.12.0 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from neptune-client) (1.15.0)\n",
      "Requirement already satisfied: websocket-client>=0.35.0 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from neptune-client) (0.57.0)\n",
      "Requirement already satisfied: pandas in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from neptune-client) (1.0.5)\n",
      "Requirement already satisfied: py3nvml in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from neptune-client) (0.2.6)\n",
      "Requirement already satisfied: packaging in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from neptune-client) (20.4)\n",
      "Requirement already satisfied: Pillow>=1.1.6 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from neptune-client) (7.1.2)\n",
      "Requirement already satisfied: click>=7.0 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from neptune-client) (7.1.2)\n",
      "Requirement already satisfied: bravado in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from neptune-client) (10.6.2)\n",
      "Requirement already satisfied: future>=0.17.1 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from neptune-client) (0.18.2)\n",
      "Requirement already satisfied: oauthlib>=2.1.0 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from neptune-client) (3.1.0)\n",
      "Requirement already satisfied: requests-oauthlib>=1.0.0 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from neptune-client) (1.3.0)\n",
      "Requirement already satisfied: PyJWT in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from neptune-client) (1.7.1)\n",
      "Requirement already satisfied: numpy<1.19.0,>=1.16.0 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorflow) (1.18.5)\n",
      "Requirement already satisfied: tensorflow-estimator<2.4.0,>=2.3.0 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorflow) (2.3.0)\n",
      "Requirement already satisfied: keras-preprocessing<1.2,>=1.1.1 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorflow) (1.1.2)\n",
      "Requirement already satisfied: h5py<2.11.0,>=2.10.0 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorflow) (2.10.0)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorflow) (1.1.0)\n",
      "Requirement already satisfied: scipy==1.4.1 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorflow) (1.4.1)\n",
      "Requirement already satisfied: wheel>=0.26 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorflow) (0.34.2)\n",
      "Requirement already satisfied: google-pasta>=0.1.8 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorflow) (0.2.0)\n",
      "Requirement already satisfied: opt-einsum>=2.3.2 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorflow) (3.2.1)\n",
      "Requirement already satisfied: protobuf>=3.9.2 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorflow) (3.12.2)\n",
      "Requirement already satisfied: absl-py>=0.7.0 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorflow) (0.9.0)\n",
      "Requirement already satisfied: tensorboard<3,>=2.3.0 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorflow) (2.3.0)\n",
      "Requirement already satisfied: astunparse==1.6.3 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorflow) (1.6.3)\n",
      "Requirement already satisfied: grpcio>=1.8.6 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorflow) (1.30.0)\n",
      "Requirement already satisfied: gast==0.3.3 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorflow) (0.3.3)\n",
      "Requirement already satisfied: wrapt>=1.11.1 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorflow) (1.12.1)\n",
      "Requirement already satisfied: gitdb<5,>=4.0.1 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from GitPython>=2.0.8->neptune-client) (4.0.5)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from requests>=2.20.0->neptune-client) (2020.6.20)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from requests>=2.20.0->neptune-client) (1.25.9)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from requests>=2.20.0->neptune-client) (3.0.4)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from requests>=2.20.0->neptune-client) (2.10)\n",
      "Requirement already satisfied: python-dateutil>=2.6.1 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from pandas->neptune-client) (2.8.1)\n",
      "Requirement already satisfied: pytz>=2017.2 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from pandas->neptune-client) (2020.1)\n",
      "Requirement already satisfied: xmltodict in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from py3nvml->neptune-client) (0.12.0)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from packaging->neptune-client) (2.4.7)\n",
      "Requirement already satisfied: pyyaml in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from bravado->neptune-client) (5.3.1)\n",
      "Requirement already satisfied: typing-extensions in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from bravado->neptune-client) (3.7.4.2)\n",
      "Requirement already satisfied: simplejson in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from bravado->neptune-client) (3.17.0)\n",
      "Requirement already satisfied: msgpack-python in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from bravado->neptune-client) (0.5.6)\n",
      "Requirement already satisfied: monotonic in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from bravado->neptune-client) (1.5)\n",
      "Requirement already satisfied: bravado-core>=5.16.1 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from bravado->neptune-client) (5.17.0)\n",
      "Requirement already satisfied: setuptools in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from protobuf>=3.9.2->tensorflow) (47.3.1.post20200622)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorboard<3,>=2.3.0->tensorflow) (1.6.0.post3)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorboard<3,>=2.3.0->tensorflow) (0.4.1)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorboard<3,>=2.3.0->tensorflow) (1.0.1)\n",
      "Requirement already satisfied: google-auth<2,>=1.6.3 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorboard<3,>=2.3.0->tensorflow) (1.18.0)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from tensorboard<3,>=2.3.0->tensorflow) (3.2.2)\n",
      "Requirement already satisfied: smmap<4,>=3.0.1 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from gitdb<5,>=4.0.1->GitPython>=2.0.8->neptune-client) (3.0.4)\n",
      "Requirement already satisfied: msgpack>=0.5.2 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from bravado-core>=5.16.1->bravado->neptune-client) (1.0.0)\n",
      "Requirement already satisfied: jsonschema[format]>=2.5.1 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from bravado-core>=5.16.1->bravado->neptune-client) (3.2.0)\n",
      "Requirement already satisfied: swagger-spec-validator>=2.0.1 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from bravado-core>=5.16.1->bravado->neptune-client) (2.7.3)\n",
      "Requirement already satisfied: jsonref in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from bravado-core>=5.16.1->bravado->neptune-client) (0.2)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow) (4.1.0)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow) (0.2.8)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4; python_version >= \"3\" in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow) (4.6)\n",
      "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard<3,>=2.3.0->tensorflow) (1.7.0)\n",
      "Requirement already satisfied: attrs>=17.4.0 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from jsonschema[format]>=2.5.1->bravado-core>=5.16.1->bravado->neptune-client) (19.3.0)\n",
      "Requirement already satisfied: pyrsistent>=0.14.0 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from jsonschema[format]>=2.5.1->bravado-core>=5.16.1->bravado->neptune-client) (0.16.0)\n",
      "Requirement already satisfied: strict-rfc3339; extra == \"format\" in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from jsonschema[format]>=2.5.1->bravado-core>=5.16.1->bravado->neptune-client) (0.7)\n",
      "Requirement already satisfied: rfc3987; extra == \"format\" in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from jsonschema[format]>=2.5.1->bravado-core>=5.16.1->bravado->neptune-client) (1.3.8)\n",
      "Requirement already satisfied: webcolors; extra == \"format\" in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from jsonschema[format]>=2.5.1->bravado-core>=5.16.1->bravado->neptune-client) (1.11.1)\n",
      "Requirement already satisfied: jsonpointer>1.13; extra == \"format\" in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from jsonschema[format]>=2.5.1->bravado-core>=5.16.1->bravado->neptune-client) (2.0)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow) (0.4.8)\n",
      "Requirement already satisfied: zipp>=0.5 in /home/jakub/miniconda3/envs/py_37/lib/python3.7/site-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard<3,>=2.3.0->tensorflow) (3.1.0)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install neptune-client tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Create a basic training script\n",
    "\n",
    "As an example I'll use a script that trains a Keras model on mnist dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras\n",
    "\n",
    "# parameters\n",
    "PARAMS = {'epoch_nr': 100,\n",
    "          'batch_size': 256,\n",
    "          'lr': 0.005,\n",
    "          'momentum': 0.4,\n",
    "          'use_nesterov': True,\n",
    "          'unit_nr': 256,\n",
    "          'dropout': 0.05}\n",
    "\n",
    "mnist = keras.datasets.mnist\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "x_train, x_test = x_train / 255.0, x_test / 255.0\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(),\n",
    "    keras.layers.Dense(PARAMS['unit_nr'], activation=keras.activations.relu),\n",
    "    keras.layers.Dropout(PARAMS['dropout']),\n",
    "    keras.layers.Dense(10, activation=keras.activations.softmax)\n",
    "])\n",
    "\n",
    "optimizer = keras.optimizers.SGD(lr=PARAMS['lr'],\n",
    "                                 momentum=PARAMS['momentum'],\n",
    "                                 nesterov=PARAMS['use_nesterov'], )\n",
    "\n",
    "model.compile(optimizer=optimizer,\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Initialize Neptune\n",
    "\n",
    "Connects your script to Neptune application. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Project(shared/onboarding)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import neptune\n",
    "\n",
    "neptune.init(\n",
    "    api_token=\"ANONYMOUS\",\n",
    "    project_qualified_name=\"shared/onboarding\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You tell Neptune: \n",
    "\n",
    "* **who you are**: your Neptune API token `api_token` \n",
    "* **where you want to send your data**: your Neptune project `project_qualified_name`.\n",
    "\n",
    "---\n",
    "\n",
    "**Note:** \n",
    "\n",
    "\n",
    "Instead of logging data to the public project 'shared/onboarding' as an anonymous user 'neptuner' you can log it to your own project.\n",
    "\n",
    "To do that:\n",
    "\n",
    "1. Get your Neptune API token\n",
    "\n",
    "![image](https://neptune.ai/wp-content/uploads/get_token.gif)\n",
    "\n",
    "2. Pass the token to ``api_token`` argument of ``neptune.init()`` method: ``api_token=YOUR_API_TOKEN``\n",
    "3. Pass your username to the ``project_qualified_name`` argument of the ``neptune.init()`` method: ``project_qualified_name='YOUR_USERNAME/sandbox``. Keep `/sandbox` at the end, the `sandbox` project that was automatically created for you.\n",
    "\n",
    "For example:\n",
    "\n",
    "```python\n",
    "neptune.init(project_qualified_name='funky_steve/sandbox', \n",
    "             api_token='eyJhcGlfYW908fsdf23f940jiri0bn3085gh03riv03irn',\n",
    "            )\n",
    "```\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Create an experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "NVMLError: NVML Shared Library Not Found - GPU usage metrics may not be reported.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://ui.neptune.ai/shared/onboarding/e/ON-265\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Experiment(ON-265)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neptune.create_experiment(name='great-idea')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This opens a new \"experiment\" namespace in Neptune to which you can log various objects.\n",
    "\n",
    "Click on the link above to open this experiment in Neptune.\n",
    "\n",
    "For now it is empty but keep the tab with experiment open to see what happens next. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Add logging for metrics and losses\n",
    "\n",
    "Since we are using Keras we'll create a Callback that logs metrics and losses after every epoch. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NeptuneMonitor(keras.callbacks.Callback):\n",
    "    def on_epoch_end(self, epoch, logs=None):\n",
    "        for metric_name, metric_value in logs.items():\n",
    "            neptune.log_metric(metric_name, metric_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to pass it to the `callbacks` argument. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "235/235 [==============================] - 1s 4ms/step - loss: 1.5523 - accuracy: 0.6272\n",
      "Epoch 2/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.8450 - accuracy: 0.8181\n",
      "Epoch 3/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.6329 - accuracy: 0.8492\n",
      "Epoch 4/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.5385 - accuracy: 0.8652\n",
      "Epoch 5/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.4829 - accuracy: 0.8750\n",
      "Epoch 6/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.4481 - accuracy: 0.8816\n",
      "Epoch 7/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.4225 - accuracy: 0.8858\n",
      "Epoch 8/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.4028 - accuracy: 0.8912\n",
      "Epoch 9/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.3870 - accuracy: 0.8952\n",
      "Epoch 10/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.3744 - accuracy: 0.8973\n",
      "Epoch 11/100\n",
      "235/235 [==============================] - 1s 2ms/step - loss: 0.3630 - accuracy: 0.9002\n",
      "Epoch 12/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.3531 - accuracy: 0.9017\n",
      "Epoch 13/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.3446 - accuracy: 0.9043\n",
      "Epoch 14/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.3365 - accuracy: 0.9068\n",
      "Epoch 15/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.3293 - accuracy: 0.9078\n",
      "Epoch 16/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.3235 - accuracy: 0.9095\n",
      "Epoch 17/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.3177 - accuracy: 0.9110\n",
      "Epoch 18/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.3122 - accuracy: 0.9129\n",
      "Epoch 19/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.3072 - accuracy: 0.9146\n",
      "Epoch 20/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.3021 - accuracy: 0.9151\n",
      "Epoch 21/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2973 - accuracy: 0.9166\n",
      "Epoch 22/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2934 - accuracy: 0.9179\n",
      "Epoch 23/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2895 - accuracy: 0.9191\n",
      "Epoch 24/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2847 - accuracy: 0.9206\n",
      "Epoch 25/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2822 - accuracy: 0.9201\n",
      "Epoch 26/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2767 - accuracy: 0.9222\n",
      "Epoch 27/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2743 - accuracy: 0.9232\n",
      "Epoch 28/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2706 - accuracy: 0.9236\n",
      "Epoch 29/100\n",
      "235/235 [==============================] - 1s 4ms/step - loss: 0.2682 - accuracy: 0.9248\n",
      "Epoch 30/100\n",
      "235/235 [==============================] - 1s 4ms/step - loss: 0.2645 - accuracy: 0.9256\n",
      "Epoch 31/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2616 - accuracy: 0.9271\n",
      "Epoch 32/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2582 - accuracy: 0.9274\n",
      "Epoch 33/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2548 - accuracy: 0.9284\n",
      "Epoch 34/100\n",
      "235/235 [==============================] - 1s 2ms/step - loss: 0.2536 - accuracy: 0.9294\n",
      "Epoch 35/100\n",
      "235/235 [==============================] - 1s 2ms/step - loss: 0.2499 - accuracy: 0.9302\n",
      "Epoch 36/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2481 - accuracy: 0.9300\n",
      "Epoch 37/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2448 - accuracy: 0.9317\n",
      "Epoch 38/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2429 - accuracy: 0.9324\n",
      "Epoch 39/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2393 - accuracy: 0.9325\n",
      "Epoch 40/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2371 - accuracy: 0.9341\n",
      "Epoch 41/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2350 - accuracy: 0.9340\n",
      "Epoch 42/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2325 - accuracy: 0.9351\n",
      "Epoch 43/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2306 - accuracy: 0.9357\n",
      "Epoch 44/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2284 - accuracy: 0.9362\n",
      "Epoch 45/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2258 - accuracy: 0.9365\n",
      "Epoch 46/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2241 - accuracy: 0.9370\n",
      "Epoch 47/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2226 - accuracy: 0.9381\n",
      "Epoch 48/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2198 - accuracy: 0.9385\n",
      "Epoch 49/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2182 - accuracy: 0.9390\n",
      "Epoch 50/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2160 - accuracy: 0.9395\n",
      "Epoch 51/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2145 - accuracy: 0.9406\n",
      "Epoch 52/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2128 - accuracy: 0.9405\n",
      "Epoch 53/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2105 - accuracy: 0.9412\n",
      "Epoch 54/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2084 - accuracy: 0.9422\n",
      "Epoch 55/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2070 - accuracy: 0.9423\n",
      "Epoch 56/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2055 - accuracy: 0.9429\n",
      "Epoch 57/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2032 - accuracy: 0.9431\n",
      "Epoch 58/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2021 - accuracy: 0.9438\n",
      "Epoch 59/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.2006 - accuracy: 0.9443\n",
      "Epoch 60/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1985 - accuracy: 0.9443\n",
      "Epoch 61/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1978 - accuracy: 0.9453\n",
      "Epoch 62/100\n",
      "235/235 [==============================] - 1s 4ms/step - loss: 0.1950 - accuracy: 0.9459\n",
      "Epoch 63/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1938 - accuracy: 0.9455\n",
      "Epoch 64/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1928 - accuracy: 0.9463\n",
      "Epoch 65/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1917 - accuracy: 0.9468\n",
      "Epoch 66/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1892 - accuracy: 0.9481\n",
      "Epoch 67/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1890 - accuracy: 0.9471\n",
      "Epoch 68/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1877 - accuracy: 0.9479\n",
      "Epoch 69/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1857 - accuracy: 0.9480\n",
      "Epoch 70/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1843 - accuracy: 0.9478\n",
      "Epoch 71/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1830 - accuracy: 0.9489\n",
      "Epoch 72/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1815 - accuracy: 0.9497\n",
      "Epoch 73/100\n",
      "235/235 [==============================] - 1s 2ms/step - loss: 0.1806 - accuracy: 0.9498\n",
      "Epoch 74/100\n",
      "235/235 [==============================] - 1s 2ms/step - loss: 0.1798 - accuracy: 0.9492\n",
      "Epoch 75/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1776 - accuracy: 0.9507\n",
      "Epoch 76/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1765 - accuracy: 0.9510\n",
      "Epoch 77/100\n",
      "235/235 [==============================] - 1s 4ms/step - loss: 0.1755 - accuracy: 0.9510\n",
      "Epoch 78/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1734 - accuracy: 0.9518\n",
      "Epoch 79/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1729 - accuracy: 0.9522\n",
      "Epoch 80/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1722 - accuracy: 0.9520\n",
      "Epoch 81/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1707 - accuracy: 0.9525\n",
      "Epoch 82/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1698 - accuracy: 0.9523\n",
      "Epoch 83/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1684 - accuracy: 0.9531\n",
      "Epoch 84/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1673 - accuracy: 0.9538\n",
      "Epoch 85/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1659 - accuracy: 0.9540\n",
      "Epoch 86/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1651 - accuracy: 0.9540\n",
      "Epoch 87/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1641 - accuracy: 0.9543\n",
      "Epoch 88/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1632 - accuracy: 0.9549\n",
      "Epoch 89/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1626 - accuracy: 0.9549\n",
      "Epoch 90/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1609 - accuracy: 0.9551\n",
      "Epoch 91/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1603 - accuracy: 0.9549\n",
      "Epoch 92/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1587 - accuracy: 0.9559\n",
      "Epoch 93/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1579 - accuracy: 0.9558\n",
      "Epoch 94/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1577 - accuracy: 0.9559\n",
      "Epoch 95/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1565 - accuracy: 0.9564\n",
      "Epoch 96/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1557 - accuracy: 0.9564\n",
      "Epoch 97/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1542 - accuracy: 0.9571\n",
      "Epoch 98/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1531 - accuracy: 0.9572\n",
      "Epoch 99/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1524 - accuracy: 0.9578\n",
      "Epoch 100/100\n",
      "235/235 [==============================] - 1s 3ms/step - loss: 0.1517 - accuracy: 0.9581\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f2ccc10c050>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x_train, y_train,\n",
    "          epochs=PARAMS['epoch_nr'],\n",
    "          batch_size=PARAMS['batch_size'],\n",
    "          callbacks=[NeptuneMonitor()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: See results live in the UI\n",
    "\n",
    "Go to the `Logs` and `Charts` sections of the Neptune UI to see them. \n",
    "\n",
    "![image](https://neptune.ai/wp-content/uploads/logs_and_charts.gif)\n",
    "\n",
    "Neptune automatically logs the hardware consumption during the experiment. \n",
    "\n",
    "You can see it in the `Monitoring` section of the Neptune UI. \n",
    "\n",
    "![image](https://neptune.ai/wp-content/uploads/monitoring.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "**Note:**\n",
    "   \n",
    "When you track experiments with Neptune in Jupyter notebooks you need to explicitly stop the experiment by running `neptune.stop()`.\n",
    "\n",
    "If you are running Neptune in regular `.py` scripts it will stop automatically when your code stops running.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "neptune.stop()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py_37",
   "language": "python",
   "name": "py_37"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}