product-tours/how-it-works/showcase/Neptune-API-Tour.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dbL6wnySOyQ6",
    "tags": [
     "header"
    ]
   },
   "source": [
    "# Neptune API tour\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header",
     "comment"
    ]
   },
   "source": [
    "## Introduction\n",
    "\n",
    "This guide will show you how to:\n",
    "\n",
    "* Install neptune-client\n",
    "* Connect Neptune to your script and create the first experiment\n",
    "* Log simple metrics to Neptune and explore the in the UI\n",
    "* Log learning curves, images and model binaries from Keras training and see those in the Neptune UI\n",
    "* Fetch the data you logged to Neptune directly into your notebook and analyze them \n",
    "\n",
    "By the end of it, you will run your first experiment and see it in Neptune!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "installation",
     "header"
    ]
   },
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment",
     "installation"
    ]
   },
   "source": [
    "Install Neptune client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jnOHCjCZ4W4a",
    "tags": [
     "installation"
    ]
   },
   "outputs": [],
   "source": [
    "! pip install neptune-client==0.4.123"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7LFLtV5R5JKT",
    "tags": [
     "header"
    ]
   },
   "source": [
    "## Initialize Neptune"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "Connects your script to Neptune application. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "colab_type": "code",
    "id": "XMdpj-Se4t0U",
    "outputId": "d66a1df2-ed95-4790-c799-fe7f6a980b39",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "import neptune\n",
    "\n",
    "neptune.init(\n",
    "    api_token=\"ANONYMOUS\",\n",
    "    project_qualified_name=\"shared/colab-test-run\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "You tell Neptune: \n",
    "\n",
    "* **who you are**: your Neptune API token `api_token` \n",
    "* **where you want to send your data**: your Neptune project `project_qualified_name`.\n",
    "\n",
    "---\n",
    "\n",
    "**Note:** \n",
    "\n",
    "\n",
    "Instead of logging data to the public project 'shared/onboarding' as an anonymous user 'neptuner' you can log it to your own project.\n",
    "\n",
    "To do that:\n",
    "\n",
    "1. Get your Neptune API token\n",
    "\n",
    "![image](https://neptune.ai/wp-content/uploads/get_token.gif)\n",
    "\n",
    "2. Pass the token to ``api_token`` argument of ``neptune.init()`` method: ``api_token=YOUR_API_TOKEN``\n",
    "3. Pass your username to the ``project_qualified_name`` argument of the ``neptune.init()`` method: ``project_qualified_name='YOUR_USERNAME/sandbox``. Keep `/sandbox` at the end, the `sandbox` project that was automatically created for you.\n",
    "\n",
    "For example:\n",
    "\n",
    "```python\n",
    "neptune.init(project_qualified_name='funky_steve/sandbox', \n",
    "             api_token='eyJhcGlfYW908fsdf23f940jiri0bn3085gh03riv03irn',\n",
    "            )\n",
    "```\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Axi3bvZyERFs",
    "tags": [
     "header"
    ]
   },
   "source": [
    "## Basic Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "Lets start with something super simple.\n",
    "\n",
    "I will:\n",
    " create an experiment, add a tag, and send a metric value\n",
    "\n",
    "* create an experiment\n",
    "* log hyperparameters\n",
    "* log a metric\n",
    "* append a tag\n",
    "* stop experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "M-opxR0-DddI",
    "outputId": "9e2e8fbd-f006-4889-a9ad-2182f7c14761",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "neptune.create_experiment(\n",
    "    name='basic-colab-example',\n",
    "    params={'learning_rate':0.1}\n",
    ")\n",
    "\n",
    "neptune.log_metric('accuracy', 0.93)\n",
    "\n",
    "neptune.append_tags(['basic', 'finished_successfully'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "You can change the values and rerun to see your experiments appear in the dashboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "---\n",
    "\n",
    "**Note:**\n",
    "   \n",
    "When you track experiments with Neptune in Jupyter notebooks you need to explicitly stop the experiment by running `neptune.stop()`.\n",
    "\n",
    "If you are running Neptune in regular `.py` scripts it will stop automatically when your code stops running.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tests\n",
    "current_exp = neptune.get_experiment()\n",
    "\n",
    "if set(current_exp.get_logs().keys()) != set(['accuracy']):\n",
    "    raise ValueError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "neptune.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "G6hnHaoZDl9u",
    "tags": [
     "comment"
    ]
   },
   "source": [
    "All `basic-colab-example` experiments are grouped in [this dashboard view](https://ui.neptune.ai/o/shared/org/colab-test-run/experiments?viewId=8dbc02b5-c68c-4833-9b43-828678145442).\n",
    "\n",
    "![alt text](https://neptune.ai/wp-content/uploads/Screenshot-from-2020-03-18-11-58-14.png)\n",
    "\n",
    "There are many other things that you can log to neptune:\n",
    "\n",
    "* Images and charts\n",
    "* Artifacts like model weights or results\n",
    "* Text values\n",
    "* Hardware consumption\n",
    "* Code snapshots\n",
    "* and more\n",
    "\n",
    "You can go and see all that in the [documentation](https://docs.neptune.ai/python-api/introduction.html) but you can check out the next example to see some of those.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "## Keras classification example [Advanced]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "Install and import your machine learning libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "installation"
    ]
   },
   "outputs": [],
   "source": [
    "!pip install tensorflow==2.3.1 scikit-plot==0.3.7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bVzPPgOb_0nV",
    "tags": [
     "comment"
    ]
   },
   "source": [
    "Get the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mBWKuQx53i6i",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "mnist = tf.keras.datasets.mnist\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "x_train, x_test = x_train / 255.0, x_test / 255.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "j_vS4kUb3uu4",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "model = tf.keras.models.Sequential([\n",
    "    tf.keras.layers.Flatten(input_shape=(28, 28)),\n",
    "    tf.keras.layers.Dense(512, activation='relu'),\n",
    "    tf.keras.layers.Dropout(0.2),\n",
    "    tf.keras.layers.Dense(10, activation='softmax')\n",
    "])\n",
    "\n",
    "model.compile(optimizer='sgd',\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy']\n",
    "              )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pRMM3rQDADHK",
    "tags": [
     "comment"
    ]
   },
   "source": [
    "To log metrics after every batch and epoch let's create `NeptuneLogger` callback:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "e-qXSos03F61",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.callbacks import Callback\n",
    "\n",
    "class NeptuneLogger(Callback):\n",
    "\n",
    "    def on_batch_end(self, batch, logs={}):\n",
    "        for log_name, log_value in logs.items():\n",
    "            neptune.log_metric(f'batch_{log_name}', log_value)\n",
    "\n",
    "    def on_epoch_end(self, epoch, logs={}):\n",
    "        for log_name, log_value in logs.items():\n",
    "            neptune.log_metric(f'epoch_{log_name}', log_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eWHRrJa8AOU2",
    "tags": [
     "comment"
    ]
   },
   "source": [
    "Now we simply need to create an experiment. \n",
    "I will tag it with the name `advanced` and log hyperparameters `epoch_nr` and `batch_size`: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "yL1LRdOIW9SM",
    "outputId": "f7456bdf-a223-43e8-82f5-92ae45ab6f8b",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "EPOCH_NR = 6\n",
    "BATCH_SIZE = 32\n",
    "\n",
    "neptune.create_experiment(name='keras-metrics',\n",
    "                          params={'epoch_nr': EPOCH_NR,\n",
    "                                  'batch_size': BATCH_SIZE},\n",
    "                          tags=['advanced'],\n",
    "                          )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ta0dkCEv998B",
    "tags": [
     "comment"
    ]
   },
   "source": [
    "Now we pass our `NeptuneLogger` as keras callback and thats it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 292
    },
    "colab_type": "code",
    "id": "0Z6yLLMf3WZf",
    "outputId": "72f5bd1d-16e4-40ad-db52-4276e315577e",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "history = model.fit(x=x_train,\n",
    "                    y=y_train,\n",
    "                    epochs=EPOCH_NR,\n",
    "                    batch_size=BATCH_SIZE,\n",
    "                    validation_data=(x_test, y_test),\n",
    "                    callbacks=[NeptuneLogger()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9XWW1thT-BNj",
    "tags": [
     "comment"
    ]
   },
   "source": [
    "You can click on the experiment link above and monitor your learning curves as it is training!\n",
    "\n",
    "![alt text](https://neptune.ai/wp-content/uploads/monitor_training.png)\n",
    "\n",
    "Great thing is, you can log more things if you need to during or after the training is finished.\n",
    "\n",
    "For example, let's calculate some additional metrics on test data and log them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "B7s1Jpw-5D0p",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "y_test_pred = np.asarray(model.predict(x_test))\n",
    "y_test_pred_class = np.argmax(y_test_pred, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "L9yva_6V4uBh",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "\n",
    "f1 = f1_score(y_test, y_test_pred_class, average='micro')\n",
    "\n",
    "neptune.log_metric('test_f1', f1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AQ_9BX8o-L2K",
    "tags": [
     "comment"
    ]
   },
   "source": [
    "We can log diagnostic charts like confusion matrix or ROC AUC curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "H-l88eLG5N3b",
    "outputId": "68eb13f1-fae2-4439-f109-25f003cc8519",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from scikitplot.metrics import plot_confusion_matrix, plot_roc\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(16, 12))\n",
    "plot_confusion_matrix(y_test, y_test_pred_class, ax=ax)\n",
    "neptune.log_image('diagnostic_charts', fig)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(16, 12))\n",
    "plot_roc(y_test, y_test_pred, ax=ax)\n",
    "neptune.log_image('diagnostic_charts', fig)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "riS6P6eq-ULv",
    "tags": [
     "comment"
    ]
   },
   "source": [
    "![alt text](https://neptune.ai/wp-content/uploads/logging_charts.png)\n",
    "\n",
    "We can also log model weights to Neptune."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "x8usf_QZ44_b",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "model.save('my_model.h5')\n",
    "neptune.log_artifact('my_model.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EiFgEMDm4h0Y",
    "tags": [
     "comment"
    ]
   },
   "source": [
    "![alt text](https://neptune.ai/wp-content/uploads/logging_artifacts.png)\n",
    "\n",
    "With that you can share models with your teammates easily."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "---\n",
    "\n",
    "**Note:**\n",
    "   \n",
    "When you track experiments with Neptune in Jupyter notebooks you need to explicitly stop the experiment by running `neptune.stop()`.\n",
    "\n",
    "If you are running Neptune in regular `.py` scripts it will stop automatically when your code stops running.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "k1RQuAk5XE1x",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "neptune.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Xh9Vfi0nXUa_",
    "tags": [
     "comment"
    ]
   },
   "source": [
    "You can play around and run this experiment with different parameters and see results and compare them.\n",
    "\n",
    "Like I've done [here](https://ui.neptune.ai/o/shared/org/colab-test-run/compare?shortId=%5B%22COL-11%22%2C%22COL-10%22%2C%22COL-9%22%2C%22COL-6%22%5D&viewId=f93b0ebd-6c75-4862-96f3-df1a67c08ea9&chartFilter=epoch_val_acc&legendFields=%5B%22shortId%22%2C%22epoch_val_acc%22%2C%22epoch_val_loss%22%2C%22epoch_loss%22%2C%22epoch_acc%22%5D&legendFieldTypes=%5B%22native%22%2C%22numericChannels%22%2C%22numericChannels%22%2C%22numericChannels%22%2C%22numericChannels%22%5D):\n",
    "\n",
    "![alt text](https://neptune.ai/wp-content/uploads/exp_comparison-1.png)\n",
    "\n",
    "A cool thing is, once things are logged to Neptune you can access them from wherever you want.\n",
    "Let me show you. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "header"
    ]
   },
   "source": [
    "## Access data you logged programatically "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "Neptune lets you fetch whatever you logged to it directly to your notebooks and scripts.\n",
    "\n",
    "Just run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 54
    },
    "colab_type": "code",
    "id": "SCa2DlfDXmyh",
    "outputId": "61d4453c-8bd8-44bc-be60-9361e48511d9",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "my_project = neptune.init(api_token=\"ANONYMOUS\", project_qualified_name=\"shared/colab-test-run\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WA2OWYxmYPg3",
    "tags": [
     "comment"
    ]
   },
   "source": [
    "Now that your project is *fetched* you can download the experiment dashboard data.\n",
    "\n",
    "I will download only the experiment data with the `tag=\"advanced\"` :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 165
    },
    "colab_type": "code",
    "id": "1Puz1fCsYQcW",
    "outputId": "3eff2a4a-5c99-4a70-ce18-8f2ff5a93b31",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "my_project.get_leaderboard(tag=['advanced']).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fbItgfG-YtKB",
    "tags": [
     "comment"
    ]
   },
   "source": [
    "You can also access information from the individual experiment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "dv3SMobaYtX-",
    "outputId": "32428363-0ce5-48f6-f291-bbb57ef30fda",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "exp = my_project.get_experiments(id='COL-6')[0]\n",
    "exp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "UJ2ttIZ6YyRc",
    "outputId": "615ba5d2-88bc-415e-8a27-7461ea384d95",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "exp.get_numeric_channels_values(\"epoch_loss\", \"epoch_val_loss\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wyGzGc7CZFHc",
    "tags": [
     "comment"
    ]
   },
   "source": [
    "You can even download artifacts from that experiment if you want to:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "O8v9MZsC_asr",
    "tags": [
     "code",
     "exclude"
    ]
   },
   "outputs": [],
   "source": [
    "exp.download_artifact('my_model.h5','./')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "6WIlQbJv_kCo",
    "outputId": "272c2d10-926b-46b2-f07a-ca577b37c3e8",
    "tags": [
     "bash_code",
     "exclude"
    ]
   },
   "outputs": [],
   "source": [
    "! ls ./"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sPLP5i70U-6S",
    "tags": [
     "comment"
    ]
   },
   "source": [
    "## Learn more about Neptune\n",
    "\n",
    "Read about other Neptune features, create your free account and start logging!\n",
    "\n",
    "[**Go to Neptune**](https://neptune.ai/?utm_source=colab&utm_medium=notebook&utm_campaign=colab-examples&utm_content=api-tour)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Neptune-API-Tour.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}