sagemaker-debugger/mnist_tensor_analysis/mnist_tensor_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.073935,
     "end_time": "2021-05-27T00:12:05.311679",
     "exception": false,
     "start_time": "2021-05-27T00:12:05.237744",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Tensor Analysis Using Amazon SageMaker Debugger\n",
    "\n",
    "Looking at the distributions of activation inputs/outputs, gradients and weights per layer can give useful insights. For instance, it helps to understand whether the model runs into problems like neuron saturation, whether there are layers in your model that are not learning at all or whether the network consists of too many layers etc. \n",
    "\n",
    "The following animation shows the distribution of gradients of a convolutional layer from an example application  as the training progresses. We can see that it starts as Gaussian distribution but then becomes more and more narrow. We can also see that the range of gradients starts very small (order of $1e-5$) and becomes even tinier as training progresses. If tiny gradients are observed from the start of training, it is an indication that we should check the hyperparameters of our model. \n",
    "\n",
    "![](images/example.gif)\n",
    "\n",
    "In this notebook we will train a poorly configured neural network and use Amazon SageMaker Debugger with custom rules to aggregate and analyse specific tensors. Before we proceed let us install the smdebug binary which allows us to perform interactive analysis in this notebook. After installing it, please restart the kernel, and when you come back skip this cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.041743,
     "end_time": "2021-05-27T00:12:06.923550",
     "exception": false,
     "start_time": "2021-05-27T00:12:06.881807",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Configuring Parameters for SageMaker Training\n",
    "\n",
    "Now we'll call the Sagemaker `MXNet` Estimator to kick off a training job . The `entry_point_script` points to the MXNet training script. The users can create a custom *SessionHook* in their training script. If they chose not to create such hook in the training script (similar to the one we will be using in this example) Amazon SageMaker Debugger will create the appropriate *SessionHook* based on specified *DebugHookConfig* parameters.\n",
    "\n",
    "The `hyperparameters` are the parameters that will be passed to the training script. We choose `Uniform(1)` as initializer and learning rate of `0.001`. This leads to the model not training well because the model is poorly initialized.\n",
    "\n",
    "The goal of a good intialization is \n",
    "\n",
    "- to break the symmetry such that parameters do not receive same gradients and updates\n",
    "- to keep variance similar across layers\n",
    "\n",
    "A bad intialization may lead to vanishing or exploiding gradients and the model not training at all. Once the training is running we will look at the distirbutions of activation inputs/outputs, gradients and weights across the training to see how these hyperparameters influenced the training.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:12:07.013166Z",
     "iopub.status.busy": "2021-05-27T00:12:07.012375Z",
     "iopub.status.idle": "2021-05-27T00:12:07.015180Z",
     "shell.execute_reply": "2021-05-27T00:12:07.014722Z"
    },
    "papermill": {
     "duration": 0.049786,
     "end_time": "2021-05-27T00:12:07.015290",
     "exception": false,
     "start_time": "2021-05-27T00:12:06.965504",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "entry_point_script = \"mnist.py\"\n",
    "bad_hyperparameters = {\"initializer\": 2, \"lr\": 0.001}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:12:07.104143Z",
     "iopub.status.busy": "2021-05-27T00:12:07.103588Z",
     "iopub.status.idle": "2021-05-27T00:12:08.552062Z",
     "shell.execute_reply": "2021-05-27T00:12:08.552502Z"
    },
    "papermill": {
     "duration": 1.495518,
     "end_time": "2021-05-27T00:12:08.552648",
     "exception": false,
     "start_time": "2021-05-27T00:12:07.057130",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "from sagemaker.mxnet import MXNet\n",
    "from sagemaker.debugger import DebuggerHookConfig, CollectionConfig\n",
    "import boto3\n",
    "import os\n",
    "\n",
    "estimator = MXNet(\n",
    "    role=sagemaker.get_execution_role(),\n",
    "    base_job_name=\"mxnet\",\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.m5.xlarge\",\n",
    "    volume_size=400,\n",
    "    source_dir=\"src\",\n",
    "    entry_point=entry_point_script,\n",
    "    hyperparameters=bad_hyperparameters,\n",
    "    framework_version=\"1.6.0\",\n",
    "    py_version=\"py3\",\n",
    "    debugger_hook_config=DebuggerHookConfig(\n",
    "        collection_configs=[\n",
    "            CollectionConfig(name=\"all\", parameters={\"include_regex\": \".*\", \"save_interval\": \"100\"})\n",
    "        ]\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.045251,
     "end_time": "2021-05-27T00:12:08.641342",
     "exception": false,
     "start_time": "2021-05-27T00:12:08.596091",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Start the training job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:12:08.733285Z",
     "iopub.status.busy": "2021-05-27T00:12:08.732360Z",
     "iopub.status.idle": "2021-05-27T00:12:09.235343Z",
     "shell.execute_reply": "2021-05-27T00:12:09.235798Z"
    },
    "papermill": {
     "duration": 0.551612,
     "end_time": "2021-05-27T00:12:09.235949",
     "exception": false,
     "start_time": "2021-05-27T00:12:08.684337",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "estimator.fit(wait=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.042959,
     "end_time": "2021-05-27T00:12:09.322504",
     "exception": false,
     "start_time": "2021-05-27T00:12:09.279545",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Get the Amazon S3 Bucket Path of Tensors\n",
    "\n",
    "We can get information related to the training job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:12:09.415593Z",
     "iopub.status.busy": "2021-05-27T00:12:09.415070Z",
     "iopub.status.idle": "2021-05-27T00:12:09.444168Z",
     "shell.execute_reply": "2021-05-27T00:12:09.444588Z"
    },
    "papermill": {
     "duration": 0.079278,
     "end_time": "2021-05-27T00:12:09.444725",
     "exception": false,
     "start_time": "2021-05-27T00:12:09.365447",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "job_name = estimator.latest_training_job.name\n",
    "client = estimator.sagemaker_session.sagemaker_client\n",
    "description = client.describe_training_job(TrainingJobName=job_name)\n",
    "description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.044711,
     "end_time": "2021-05-27T00:12:09.535764",
     "exception": false,
     "start_time": "2021-05-27T00:12:09.491053",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can retrieve the S3 location of the tensors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:12:09.626692Z",
     "iopub.status.busy": "2021-05-27T00:12:09.626066Z",
     "iopub.status.idle": "2021-05-27T00:12:09.628721Z",
     "shell.execute_reply": "2021-05-27T00:12:09.629164Z"
    },
    "papermill": {
     "duration": 0.050101,
     "end_time": "2021-05-27T00:12:09.629294",
     "exception": false,
     "start_time": "2021-05-27T00:12:09.579193",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "path = estimator.latest_job_debugger_artifacts_path()\n",
    "print(\"Tensors are stored in: \", path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.043925,
     "end_time": "2021-05-27T00:12:09.717319",
     "exception": false,
     "start_time": "2021-05-27T00:12:09.673394",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can check the status of our training job, by executing `describe_training_job`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:12:09.811145Z",
     "iopub.status.busy": "2021-05-27T00:12:09.810596Z",
     "iopub.status.idle": "2021-05-27T00:12:09.830594Z",
     "shell.execute_reply": "2021-05-27T00:12:09.831041Z"
    },
    "papermill": {
     "duration": 0.06985,
     "end_time": "2021-05-27T00:12:09.831180",
     "exception": false,
     "start_time": "2021-05-27T00:12:09.761330",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "job_name = estimator.latest_training_job.name\n",
    "print(\"Training job name: {}\".format(job_name))\n",
    "\n",
    "client = estimator.sagemaker_session.sagemaker_client\n",
    "\n",
    "description = client.describe_training_job(TrainingJobName=job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.044384,
     "end_time": "2021-05-27T00:12:09.919930",
     "exception": false,
     "start_time": "2021-05-27T00:12:09.875546",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can access the tensors from S3 once the training job is in status `Training` or `Completed`. In the following code cell we check the job status:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:12:10.015624Z",
     "iopub.status.busy": "2021-05-27T00:12:10.015030Z",
     "iopub.status.idle": "2021-05-27T00:15:10.709649Z",
     "shell.execute_reply": "2021-05-27T00:15:10.710150Z"
    },
    "papermill": {
     "duration": 180.745999,
     "end_time": "2021-05-27T00:15:10.710319",
     "exception": false,
     "start_time": "2021-05-27T00:12:09.964320",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "if description[\"TrainingJobStatus\"] != \"Completed\":\n",
    "    while description[\"SecondaryStatus\"] not in {\"Training\", \"Completed\"}:\n",
    "        description = client.describe_training_job(TrainingJobName=job_name)\n",
    "        primary_status = description[\"TrainingJobStatus\"]\n",
    "        secondary_status = description[\"SecondaryStatus\"]\n",
    "        print(\n",
    "            \"Current job status: [PrimaryStatus: {}, SecondaryStatus: {}]\".format(\n",
    "                primary_status, secondary_status\n",
    "            )\n",
    "        )\n",
    "        time.sleep(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.047948,
     "end_time": "2021-05-27T00:15:10.806687",
     "exception": false,
     "start_time": "2021-05-27T00:15:10.758739",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Once the job is in status `Training` or `Completed`, we can create the trial that allows us to access the tensors in Amazon S3. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install the `smdebug` Client Library\n",
    "\n",
    "**NOTE:** This notebook works with the `smdebug` library v1.0.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:12:05.398626Z",
     "iopub.status.busy": "2021-05-27T00:12:05.397728Z",
     "iopub.status.idle": "2021-05-27T00:12:06.839114Z",
     "shell.execute_reply": "2021-05-27T00:12:06.839594Z"
    },
    "papermill": {
     "duration": 1.487112,
     "end_time": "2021-05-27T00:12:06.839750",
     "exception": false,
     "start_time": "2021-05-27T00:12:05.352638",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "!  python -m pip install -U smdebug==1.0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create an `smdebug` Trial Object to Retrieve the Saved Tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:15:10.905584Z",
     "iopub.status.busy": "2021-05-27T00:15:10.905034Z",
     "iopub.status.idle": "2021-05-27T00:15:23.479830Z",
     "shell.execute_reply": "2021-05-27T00:15:23.480311Z"
    },
    "papermill": {
     "duration": 12.626168,
     "end_time": "2021-05-27T00:15:23.480466",
     "exception": false,
     "start_time": "2021-05-27T00:15:10.854298",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from smdebug.trials import create_trial\n",
    "\n",
    "trial1 = create_trial(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.047968,
     "end_time": "2021-05-27T00:15:23.576874",
     "exception": false,
     "start_time": "2021-05-27T00:15:23.528906",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can check the available steps. A step presents one forward and backward pass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:15:23.678565Z",
     "iopub.status.busy": "2021-05-27T00:15:23.677999Z",
     "iopub.status.idle": "2021-05-27T00:15:23.717042Z",
     "shell.execute_reply": "2021-05-27T00:15:23.717490Z"
    },
    "papermill": {
     "duration": 0.092804,
     "end_time": "2021-05-27T00:15:23.717637",
     "exception": false,
     "start_time": "2021-05-27T00:15:23.624833",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "trial1.steps()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.049087,
     "end_time": "2021-05-27T00:15:23.815810",
     "exception": false,
     "start_time": "2021-05-27T00:15:23.766723",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "As training progresses more steps will become available. \n",
    "\n",
    "Next we will access specific tensors like weights, gradients and activation outputs and plot their distributions.  We will use Amazon SageMaker Debugger and define custom rules to retrieve certain tensors. Rules are supposed to return True or False. However in this notebook we will use custom rules to store dictionaries of aggregated tensors per layer and step, which we then plot afterwards.\n",
    "\n",
    "A custom rule inherits from the smdebug Rule class and implements the function `invoke_at_step`. This function is called everytime tensors of a new step become available:\n",
    "\n",
    "```\n",
    "\n",
    "from smdebug.rules.rule import Rule\n",
    "\n",
    "class MyCustomRule(Rule):\n",
    "    def __init__(self, base_trial):\n",
    "        super().__init__(base_trial)\n",
    "        \n",
    "    def invoke_at_step(self, step):  \n",
    "        if np.max(self.base_trial.tensor('conv0_relu_output_0').value(step) < 0.001:\n",
    "            return True\n",
    "    return False\n",
    "\n",
    "```   \n",
    "\n",
    "Above example rule checks if the first convolutional layer outputs only small values. If so the rule returns `True` which corresponds to an `Issue found`, otherwise False `No Issue found`.\n",
    "\n",
    "\n",
    "## Visualize Activation Outputs\n",
    "This rule will use Amazon SageMaker Debugger to retrieve tensors from the ReLU output layers. It sums the activations across batch and steps. If there is a large fraction of ReLUs outputing 0 across many steps it means that the neuron is dying."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:15:23.918397Z",
     "iopub.status.busy": "2021-05-27T00:15:23.917419Z",
     "iopub.status.idle": "2021-05-27T00:15:24.010864Z",
     "shell.execute_reply": "2021-05-27T00:15:24.011382Z"
    },
    "papermill": {
     "duration": 0.146788,
     "end_time": "2021-05-27T00:15:24.011539",
     "exception": false,
     "start_time": "2021-05-27T00:15:23.864751",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from smdebug.trials import create_trial\n",
    "from smdebug.rules.rule_invoker import invoke_rule\n",
    "from smdebug.exceptions import NoMoreData\n",
    "from smdebug.rules.rule import Rule\n",
    "import numpy as np\n",
    "import utils\n",
    "import collections\n",
    "import os\n",
    "from IPython.display import Image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:15:24.118969Z",
     "iopub.status.busy": "2021-05-27T00:15:24.118154Z",
     "iopub.status.idle": "2021-05-27T00:16:06.991911Z",
     "shell.execute_reply": "2021-05-27T00:16:06.992342Z"
    },
    "papermill": {
     "duration": 42.931699,
     "end_time": "2021-05-27T00:16:06.992493",
     "exception": false,
     "start_time": "2021-05-27T00:15:24.060794",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class ActivationOutputs(Rule):\n",
    "    def __init__(self, base_trial):\n",
    "        super().__init__(base_trial)\n",
    "        self.tensors = collections.OrderedDict()\n",
    "\n",
    "    def invoke_at_step(self, step):\n",
    "        for tname in self.base_trial.tensor_names(regex=\".*relu_output\"):\n",
    "            if \"gradients\" not in tname:\n",
    "                try:\n",
    "                    tensor = self.base_trial.tensor(tname).value(step)\n",
    "                    if tname not in self.tensors:\n",
    "                        self.tensors[tname] = collections.OrderedDict()\n",
    "                    if step not in self.tensors[tname]:\n",
    "                        self.tensors[tname][step] = 0\n",
    "                    neg_values = np.where(tensor <= 0)[0]\n",
    "                    if len(neg_values) > 0:\n",
    "                        self.logger.info(\n",
    "                            f\" Step {step} tensor  {tname}  has {len(neg_values)/tensor.size*100}% activation outputs which are smaller than 0 \"\n",
    "                        )\n",
    "                    batch_over_sum = np.sum(tensor, axis=0) / tensor.shape[0]\n",
    "                    self.tensors[tname][step] += batch_over_sum\n",
    "                except:\n",
    "                    self.logger.warning(f\"Can not fetch tensor {tname}\")\n",
    "        return False\n",
    "\n",
    "\n",
    "rule = ActivationOutputs(trial1)\n",
    "try:\n",
    "    invoke_rule(rule)\n",
    "except NoMoreData:\n",
    "    print(\n",
    "        \"The training has ended and there is no more data to be analyzed. This is expected behavior.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.065578,
     "end_time": "2021-05-27T00:16:07.124234",
     "exception": false,
     "start_time": "2021-05-27T00:16:07.058656",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Plot the histograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:16:07.274401Z",
     "iopub.status.busy": "2021-05-27T00:16:07.272303Z",
     "iopub.status.idle": "2021-05-27T00:16:21.359875Z",
     "shell.execute_reply": "2021-05-27T00:16:21.360339Z"
    },
    "papermill": {
     "duration": 14.170716,
     "end_time": "2021-05-27T00:16:21.360493",
     "exception": false,
     "start_time": "2021-05-27T00:16:07.189777",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "utils.create_interactive_matplotlib_histogram(\n",
    "    rule.tensors, filename=\"images/activation_outputs.gif\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:16:21.497126Z",
     "iopub.status.busy": "2021-05-27T00:16:21.496306Z",
     "iopub.status.idle": "2021-05-27T00:16:21.499194Z",
     "shell.execute_reply": "2021-05-27T00:16:21.499614Z"
    },
    "papermill": {
     "duration": 0.073776,
     "end_time": "2021-05-27T00:16:21.499751",
     "exception": false,
     "start_time": "2021-05-27T00:16:21.425975",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "Image(url=\"images/activation_outputs.gif\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.065772,
     "end_time": "2021-05-27T00:16:21.631493",
     "exception": false,
     "start_time": "2021-05-27T00:16:21.565721",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Visualize Activation Inputs\n",
    "In this rule we look at the inputs into activation function, rather than the output. This can be helpful to understand if there are extreme negative or positive values that saturate the activation functions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:16:21.771028Z",
     "iopub.status.busy": "2021-05-27T00:16:21.770208Z",
     "iopub.status.idle": "2021-05-27T00:16:25.629482Z",
     "shell.execute_reply": "2021-05-27T00:16:25.628927Z"
    },
    "papermill": {
     "duration": 3.932772,
     "end_time": "2021-05-27T00:16:25.629603",
     "exception": false,
     "start_time": "2021-05-27T00:16:21.696831",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class ActivationInputs(Rule):\n",
    "    def __init__(self, base_trial):\n",
    "        super().__init__(base_trial)\n",
    "        self.tensors = collections.OrderedDict()\n",
    "\n",
    "    def invoke_at_step(self, step):\n",
    "        for tname in self.base_trial.tensor_names(regex=\".*relu_input\"):\n",
    "            if \"gradients\" not in tname:\n",
    "                try:\n",
    "                    tensor = self.base_trial.tensor(tname).value(step)\n",
    "                    if tname not in self.tensors:\n",
    "                        self.tensors[tname] = {}\n",
    "                    if step not in self.tensors[tname]:\n",
    "                        self.tensors[tname][step] = 0\n",
    "                    neg_values = np.where(tensor <= 0)[0]\n",
    "                    if len(neg_values) > 0:\n",
    "                        self.logger.info(\n",
    "                            f\" Tensor  {tname}  has {len(neg_values)/tensor.size*100}% activation inputs which are smaller than 0 \"\n",
    "                        )\n",
    "                    batch_over_sum = np.sum(tensor, axis=0) / tensor.shape[0]\n",
    "                    self.tensors[tname][step] += batch_over_sum\n",
    "                except:\n",
    "                    self.logger.warning(f\"Can not fetch tensor {tname}\")\n",
    "        return False\n",
    "\n",
    "\n",
    "rule = ActivationInputs(trial1)\n",
    "try:\n",
    "    invoke_rule(rule)\n",
    "except NoMoreData:\n",
    "    print(\n",
    "        \"The training has ended and there is no more data to be analyzed. This is expected behavior.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.082352,
     "end_time": "2021-05-27T00:16:25.793861",
     "exception": false,
     "start_time": "2021-05-27T00:16:25.711509",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Plot the histograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:16:25.976683Z",
     "iopub.status.busy": "2021-05-27T00:16:25.974280Z",
     "iopub.status.idle": "2021-05-27T00:16:39.650021Z",
     "shell.execute_reply": "2021-05-27T00:16:39.650530Z"
    },
    "papermill": {
     "duration": 13.774921,
     "end_time": "2021-05-27T00:16:39.650684",
     "exception": false,
     "start_time": "2021-05-27T00:16:25.875763",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "utils.create_interactive_matplotlib_histogram(rule.tensors, filename=\"images/activation_inputs.gif\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.08168,
     "end_time": "2021-05-27T00:16:39.814333",
     "exception": false,
     "start_time": "2021-05-27T00:16:39.732653",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can see that second convolutional layer `conv1_relu_input_0` receives only negative input values, which means that all ReLUs in this layer output 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:16:39.981555Z",
     "iopub.status.busy": "2021-05-27T00:16:39.980897Z",
     "iopub.status.idle": "2021-05-27T00:16:39.984125Z",
     "shell.execute_reply": "2021-05-27T00:16:39.983651Z"
    },
    "papermill": {
     "duration": 0.088537,
     "end_time": "2021-05-27T00:16:39.984235",
     "exception": false,
     "start_time": "2021-05-27T00:16:39.895698",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "Image(url=\"images/activation_inputs.gif\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.081851,
     "end_time": "2021-05-27T00:16:40.148583",
     "exception": false,
     "start_time": "2021-05-27T00:16:40.066732",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Visualize Gradients\n",
    "The following code retrieves the gradients and plots their distribution. If variance is tiny, that means that the model parameters do not get updated effectively with each training step or that the training has converged to a minimum."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:16:40.319621Z",
     "iopub.status.busy": "2021-05-27T00:16:40.318853Z",
     "iopub.status.idle": "2021-05-27T00:16:48.409507Z",
     "shell.execute_reply": "2021-05-27T00:16:48.409942Z"
    },
    "papermill": {
     "duration": 8.179636,
     "end_time": "2021-05-27T00:16:48.410115",
     "exception": false,
     "start_time": "2021-05-27T00:16:40.230479",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class GradientsLayer(Rule):\n",
    "    def __init__(self, base_trial):\n",
    "        super().__init__(base_trial)\n",
    "        self.tensors = collections.OrderedDict()\n",
    "\n",
    "    def invoke_at_step(self, step):\n",
    "        for tname in self.base_trial.tensor_names(regex=\".*gradient\"):\n",
    "            try:\n",
    "                tensor = self.base_trial.tensor(tname).value(step)\n",
    "                if tname not in self.tensors:\n",
    "                    self.tensors[tname] = {}\n",
    "\n",
    "                self.logger.info(\n",
    "                    f\" Tensor  {tname}  has gradients range: {np.min(tensor)} {np.max(tensor)} \"\n",
    "                )\n",
    "                self.tensors[tname][step] = tensor\n",
    "            except:\n",
    "                self.logger.warning(f\"Can not fetch tensor {tname}\")\n",
    "        return False\n",
    "\n",
    "\n",
    "rule = GradientsLayer(trial1)\n",
    "try:\n",
    "    invoke_rule(rule)\n",
    "except NoMoreData:\n",
    "    print(\n",
    "        \"The training has ended and there is no more data to be analyzed. This is expected behavior.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.120593,
     "end_time": "2021-05-27T00:16:48.651876",
     "exception": false,
     "start_time": "2021-05-27T00:16:48.531283",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Plot the histograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:16:48.909278Z",
     "iopub.status.busy": "2021-05-27T00:16:48.908705Z",
     "iopub.status.idle": "2021-05-27T00:17:25.740865Z",
     "shell.execute_reply": "2021-05-27T00:17:25.741320Z"
    },
    "papermill": {
     "duration": 36.969208,
     "end_time": "2021-05-27T00:17:25.741474",
     "exception": false,
     "start_time": "2021-05-27T00:16:48.772266",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "utils.create_interactive_matplotlib_histogram(rule.tensors, filename=\"images/gradients.gif\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:17:25.987371Z",
     "iopub.status.busy": "2021-05-27T00:17:25.986716Z",
     "iopub.status.idle": "2021-05-27T00:17:25.989987Z",
     "shell.execute_reply": "2021-05-27T00:17:25.989451Z"
    },
    "papermill": {
     "duration": 0.127803,
     "end_time": "2021-05-27T00:17:25.990122",
     "exception": false,
     "start_time": "2021-05-27T00:17:25.862319",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "Image(url=\"images/gradients.gif\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.121691,
     "end_time": "2021-05-27T00:17:26.233383",
     "exception": false,
     "start_time": "2021-05-27T00:17:26.111692",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Check Variance across Layers\n",
    "The rule retrieves gradients, but this time we compare variance of gradient distribution across layers. We want to identify if there is a large difference between the min and max variance per training step. For instance, very deep neural networks may suffer from vanishing gradients the deeper we go. By checking this ratio we can determine if we run into such a situation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:17:26.485396Z",
     "iopub.status.busy": "2021-05-27T00:17:26.484358Z",
     "iopub.status.idle": "2021-05-27T00:17:34.277221Z",
     "shell.execute_reply": "2021-05-27T00:17:34.277652Z"
    },
    "papermill": {
     "duration": 7.922793,
     "end_time": "2021-05-27T00:17:34.277797",
     "exception": false,
     "start_time": "2021-05-27T00:17:26.355004",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class GradientsAcrossLayers(Rule):\n",
    "    def __init__(\n",
    "        self,\n",
    "        base_trial,\n",
    "    ):\n",
    "        super().__init__(base_trial)\n",
    "        self.tensors = collections.OrderedDict()\n",
    "\n",
    "    def invoke_at_step(self, step):\n",
    "        for tname in self.base_trial.tensor_names(regex=\".*gradient\"):\n",
    "            try:\n",
    "                tensor = self.base_trial.tensor(tname).value(step)\n",
    "                if step not in self.tensors:\n",
    "                    self.tensors[step] = [np.inf, 0]\n",
    "                variance = np.var(tensor.flatten())\n",
    "                if variance < self.tensors[step][0]:\n",
    "                    self.tensors[step][0] = variance\n",
    "                elif variance > self.tensors[step][1]:\n",
    "                    self.tensors[step][1] = variance\n",
    "                self.logger.info(\n",
    "                    f\" Step {step} current ratio: {self.tensors[step][0]} {self.tensors[step][1]} Ratio: {self.tensors[step][1] / self.tensors[step][0]}\"\n",
    "                )\n",
    "            except:\n",
    "                self.logger.warning(f\"Can not fetch tensor {tname}\")\n",
    "        return False\n",
    "\n",
    "\n",
    "rule = GradientsAcrossLayers(trial1)\n",
    "try:\n",
    "    invoke_rule(rule)\n",
    "except NoMoreData:\n",
    "    print(\n",
    "        \"The training has ended and there is no more data to be analyzed. This is expected behavior.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.160295,
     "end_time": "2021-05-27T00:17:34.597710",
     "exception": false,
     "start_time": "2021-05-27T00:17:34.437415",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Let's check min and max values of the gradients across layers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:17:34.923168Z",
     "iopub.status.busy": "2021-05-27T00:17:34.922601Z",
     "iopub.status.idle": "2021-05-27T00:17:34.933298Z",
     "shell.execute_reply": "2021-05-27T00:17:34.932843Z"
    },
    "papermill": {
     "duration": 0.175803,
     "end_time": "2021-05-27T00:17:34.933411",
     "exception": false,
     "start_time": "2021-05-27T00:17:34.757608",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "for step in rule.tensors:\n",
    "    print(\n",
    "        \"Step\",\n",
    "        step,\n",
    "        \"variance of gradients: \",\n",
    "        rule.tensors[step][0],\n",
    "        \" to \",\n",
    "        rule.tensors[step][1],\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.161047,
     "end_time": "2021-05-27T00:17:35.254797",
     "exception": false,
     "start_time": "2021-05-27T00:17:35.093750",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Distribution of Weights\n",
    "This rule retrieves the weight tensors and checks the variance. If the distribution does not change much across steps it may indicate that the learning rate is too low, that gradients are too small or that the training has converged to a minimum."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:17:35.583312Z",
     "iopub.status.busy": "2021-05-27T00:17:35.582502Z",
     "iopub.status.idle": "2021-05-27T00:17:40.404474Z",
     "shell.execute_reply": "2021-05-27T00:17:40.404014Z"
    },
    "papermill": {
     "duration": 4.989775,
     "end_time": "2021-05-27T00:17:40.404594",
     "exception": false,
     "start_time": "2021-05-27T00:17:35.414819",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class WeightRatio(Rule):\n",
    "    def __init__(\n",
    "        self,\n",
    "        base_trial,\n",
    "    ):\n",
    "        super().__init__(base_trial)\n",
    "        self.tensors = collections.OrderedDict()\n",
    "\n",
    "    def invoke_at_step(self, step):\n",
    "        for tname in self.base_trial.tensor_names(regex=\".*weight\"):\n",
    "            if \"gradient\" not in tname:\n",
    "                try:\n",
    "                    tensor = self.base_trial.tensor(tname).value(step)\n",
    "                    if tname not in self.tensors:\n",
    "                        self.tensors[tname] = {}\n",
    "\n",
    "                    self.logger.info(\n",
    "                        f\" Tensor  {tname}  has weights with variance: {np.var(tensor.flatten())} \"\n",
    "                    )\n",
    "                    self.tensors[tname][step] = tensor\n",
    "                except:\n",
    "                    self.logger.warning(f\"Can not fetch tensor {tname}\")\n",
    "        return False\n",
    "\n",
    "\n",
    "rule = WeightRatio(trial1)\n",
    "try:\n",
    "    invoke_rule(rule)\n",
    "except NoMoreData:\n",
    "    print(\n",
    "        \"The training has ended and there is no more data to be analyzed. This is expected behavior.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.179398,
     "end_time": "2021-05-27T00:17:40.764533",
     "exception": false,
     "start_time": "2021-05-27T00:17:40.585135",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Plot the histograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:17:41.157744Z",
     "iopub.status.busy": "2021-05-27T00:17:41.135975Z",
     "iopub.status.idle": "2021-05-27T00:17:59.991093Z",
     "shell.execute_reply": "2021-05-27T00:17:59.991567Z"
    },
    "papermill": {
     "duration": 19.047552,
     "end_time": "2021-05-27T00:17:59.991734",
     "exception": false,
     "start_time": "2021-05-27T00:17:40.944182",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "utils.create_interactive_matplotlib_histogram(rule.tensors, filename=\"images/weights.gif\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:18:00.357969Z",
     "iopub.status.busy": "2021-05-27T00:18:00.357410Z",
     "iopub.status.idle": "2021-05-27T00:18:00.360342Z",
     "shell.execute_reply": "2021-05-27T00:18:00.360775Z"
    },
    "papermill": {
     "duration": 0.18754,
     "end_time": "2021-05-27T00:18:00.360912",
     "exception": false,
     "start_time": "2021-05-27T00:18:00.173372",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "Image(url=\"images/weights.gif\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.180098,
     "end_time": "2021-05-27T00:18:00.722436",
     "exception": false,
     "start_time": "2021-05-27T00:18:00.542338",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Inputs\n",
    "\n",
    "This rule retrieves layer inputs excluding activation inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:18:01.089095Z",
     "iopub.status.busy": "2021-05-27T00:18:01.088298Z",
     "iopub.status.idle": "2021-05-27T00:18:14.064489Z",
     "shell.execute_reply": "2021-05-27T00:18:14.064922Z"
    },
    "papermill": {
     "duration": 13.163445,
     "end_time": "2021-05-27T00:18:14.065067",
     "exception": false,
     "start_time": "2021-05-27T00:18:00.901622",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class Inputs(Rule):\n",
    "    def __init__(\n",
    "        self,\n",
    "        base_trial,\n",
    "    ):\n",
    "        super().__init__(base_trial)\n",
    "        self.tensors = collections.OrderedDict()\n",
    "\n",
    "    def invoke_at_step(self, step):\n",
    "        for tname in self.base_trial.tensor_names(regex=\".*input\"):\n",
    "            if \"relu\" not in tname:\n",
    "                try:\n",
    "                    tensor = self.base_trial.tensor(tname).value(step)\n",
    "                    if tname not in self.tensors:\n",
    "                        self.tensors[tname] = {}\n",
    "\n",
    "                    self.logger.info(\n",
    "                        f\" Tensor  {tname}  has inputs with variance: {np.var(tensor.flatten())} \"\n",
    "                    )\n",
    "                    self.tensors[tname][step] = tensor\n",
    "                except:\n",
    "                    self.logger.warning(f\"Can not fetch tensor {tname}\")\n",
    "        return False\n",
    "\n",
    "\n",
    "rule = Inputs(trial1)\n",
    "try:\n",
    "    invoke_rule(rule)\n",
    "except NoMoreData:\n",
    "    print(\n",
    "        \"The training has ended and there is no more data to be analyzed. This is expected behavior.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.223424,
     "end_time": "2021-05-27T00:18:14.515619",
     "exception": false,
     "start_time": "2021-05-27T00:18:14.292195",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Plot the histograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:18:14.997203Z",
     "iopub.status.busy": "2021-05-27T00:18:14.980533Z",
     "iopub.status.idle": "2021-05-27T00:18:55.753886Z",
     "shell.execute_reply": "2021-05-27T00:18:55.753378Z"
    },
    "papermill": {
     "duration": 41.014457,
     "end_time": "2021-05-27T00:18:55.754010",
     "exception": false,
     "start_time": "2021-05-27T00:18:14.739553",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "utils.create_interactive_matplotlib_histogram(rule.tensors, filename=\"images/layer_inputs.gif\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:18:56.205796Z",
     "iopub.status.busy": "2021-05-27T00:18:56.205090Z",
     "iopub.status.idle": "2021-05-27T00:18:56.207955Z",
     "shell.execute_reply": "2021-05-27T00:18:56.208385Z"
    },
    "papermill": {
     "duration": 0.231317,
     "end_time": "2021-05-27T00:18:56.208523",
     "exception": false,
     "start_time": "2021-05-27T00:18:55.977206",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "Image(url=\"images/layer_inputs.gif\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.225022,
     "end_time": "2021-05-27T00:18:56.656862",
     "exception": false,
     "start_time": "2021-05-27T00:18:56.431840",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Layer Outputs\n",
    "This rule retrieves outputs of layers excluding activation outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:18:57.111203Z",
     "iopub.status.busy": "2021-05-27T00:18:57.110504Z",
     "iopub.status.idle": "2021-05-27T00:19:08.859101Z",
     "shell.execute_reply": "2021-05-27T00:19:08.858636Z"
    },
    "papermill": {
     "duration": 11.978508,
     "end_time": "2021-05-27T00:19:08.859218",
     "exception": false,
     "start_time": "2021-05-27T00:18:56.880710",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class Outputs(Rule):\n",
    "    def __init__(\n",
    "        self,\n",
    "        base_trial,\n",
    "    ):\n",
    "        super().__init__(base_trial)\n",
    "        self.tensors = collections.OrderedDict()\n",
    "\n",
    "    def invoke_at_step(self, step):\n",
    "        for tname in self.base_trial.tensor_names(regex=\".*output\"):\n",
    "            if \"relu\" not in tname:\n",
    "                try:\n",
    "                    tensor = self.base_trial.tensor(tname).value(step)\n",
    "                    if tname not in self.tensors:\n",
    "                        self.tensors[tname] = {}\n",
    "\n",
    "                    self.logger.info(\n",
    "                        f\" Tensor  {tname}  has inputs with variance: {np.var(tensor.flatten())} \"\n",
    "                    )\n",
    "                    self.tensors[tname][step] = tensor\n",
    "                except:\n",
    "                    self.logger.warning(f\"Can not fetch tensor {tname}\")\n",
    "        return False\n",
    "\n",
    "\n",
    "rule = Outputs(trial1)\n",
    "try:\n",
    "    invoke_rule(rule)\n",
    "except NoMoreData:\n",
    "    print(\n",
    "        \"The training has ended and there is no more data to be analyzed. This is expected behavior.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.264154,
     "end_time": "2021-05-27T00:19:09.390388",
     "exception": false,
     "start_time": "2021-05-27T00:19:09.126234",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Plot the histograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:19:09.939979Z",
     "iopub.status.busy": "2021-05-27T00:19:09.932035Z",
     "iopub.status.idle": "2021-05-27T00:19:46.675464Z",
     "shell.execute_reply": "2021-05-27T00:19:46.675948Z"
    },
    "papermill": {
     "duration": 37.02184,
     "end_time": "2021-05-27T00:19:46.676101",
     "exception": false,
     "start_time": "2021-05-27T00:19:09.654261",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "utils.create_interactive_matplotlib_histogram(rule.tensors, filename=\"images/layer_outputs.gif\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:19:47.210212Z",
     "iopub.status.busy": "2021-05-27T00:19:47.209481Z",
     "iopub.status.idle": "2021-05-27T00:19:47.212396Z",
     "shell.execute_reply": "2021-05-27T00:19:47.212817Z"
    },
    "papermill": {
     "duration": 0.273187,
     "end_time": "2021-05-27T00:19:47.212951",
     "exception": false,
     "start_time": "2021-05-27T00:19:46.939764",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "Image(url=\"images/layer_outputs.gif\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.263939,
     "end_time": "2021-05-27T00:19:47.739424",
     "exception": false,
     "start_time": "2021-05-27T00:19:47.475485",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Comparison \n",
    "\n",
    "In the previous section we have looked at the distribution of gradients, activation outputs and weights of a model that has not trained well due to poor initialization. Now we will compare some of these distributions with a model that is well intialized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:19:48.268629Z",
     "iopub.status.busy": "2021-05-27T00:19:48.268051Z",
     "iopub.status.idle": "2021-05-27T00:19:48.270532Z",
     "shell.execute_reply": "2021-05-27T00:19:48.270051Z"
    },
    "papermill": {
     "duration": 0.268814,
     "end_time": "2021-05-27T00:19:48.270646",
     "exception": false,
     "start_time": "2021-05-27T00:19:48.001832",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "entry_point_script = \"mnist.py\"\n",
    "hyperparameters = {\"lr\": 0.01}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:19:48.834236Z",
     "iopub.status.busy": "2021-05-27T00:19:48.833610Z",
     "iopub.status.idle": "2021-05-27T00:19:49.222866Z",
     "shell.execute_reply": "2021-05-27T00:19:49.223300Z"
    },
    "papermill": {
     "duration": 0.661481,
     "end_time": "2021-05-27T00:19:49.223448",
     "exception": false,
     "start_time": "2021-05-27T00:19:48.561967",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "estimator = MXNet(\n",
    "    role=sagemaker.get_execution_role(),\n",
    "    base_job_name=\"mxnet\",\n",
    "    train_instance_count=1,\n",
    "    train_instance_type=\"ml.m5.xlarge\",\n",
    "    train_volume_size=400,\n",
    "    source_dir=\"src\",\n",
    "    entry_point=entry_point_script,\n",
    "    hyperparameters=hyperparameters,\n",
    "    framework_version=\"1.6.0\",\n",
    "    py_version=\"py3\",\n",
    "    debugger_hook_config=DebuggerHookConfig(\n",
    "        collection_configs=[\n",
    "            CollectionConfig(name=\"all\", parameters={\"include_regex\": \".*\", \"save_interval\": \"100\"})\n",
    "        ]\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.264163,
     "end_time": "2021-05-27T00:19:49.751886",
     "exception": false,
     "start_time": "2021-05-27T00:19:49.487723",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Start the training job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:19:50.286391Z",
     "iopub.status.busy": "2021-05-27T00:19:50.285773Z",
     "iopub.status.idle": "2021-05-27T00:19:50.786891Z",
     "shell.execute_reply": "2021-05-27T00:19:50.787312Z"
    },
    "papermill": {
     "duration": 0.769986,
     "end_time": "2021-05-27T00:19:50.787470",
     "exception": false,
     "start_time": "2021-05-27T00:19:50.017484",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "estimator.fit(wait=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.266435,
     "end_time": "2021-05-27T00:19:51.320938",
     "exception": false,
     "start_time": "2021-05-27T00:19:51.054503",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Get S3 path where tensors have been stored"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:19:51.859619Z",
     "iopub.status.busy": "2021-05-27T00:19:51.859003Z",
     "iopub.status.idle": "2021-05-27T00:19:51.861473Z",
     "shell.execute_reply": "2021-05-27T00:19:51.861929Z"
    },
    "papermill": {
     "duration": 0.274291,
     "end_time": "2021-05-27T00:19:51.862066",
     "exception": false,
     "start_time": "2021-05-27T00:19:51.587775",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "path = estimator.latest_job_debugger_artifacts_path()\n",
    "print(\"Tensors are stored in: \", path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.266104,
     "end_time": "2021-05-27T00:19:52.393964",
     "exception": false,
     "start_time": "2021-05-27T00:19:52.127860",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Check the status of the training job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T00:19:52.935787Z",
     "iopub.status.busy": "2021-05-27T00:19:52.932297Z",
     "iopub.status.idle": "2021-05-27T00:21:37.965685Z",
     "shell.execute_reply": "2021-05-27T00:21:37.966227Z"
    },
    "papermill": {
     "duration": 105.306138,
     "end_time": "2021-05-27T00:21:37.966490",
     "exception": true,
     "start_time": "2021-05-27T00:19:52.660352",
     "status": "failed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "job_name = estimator.latest_training_job.name\n",
    "print(\"Training job name: {}\".format(job_name))\n",
    "\n",
    "client = estimator.sagemaker_session.sagemaker_client\n",
    "\n",
    "description = client.describe_training_job(TrainingJobName=job_name)\n",
    "\n",
    "if description[\"TrainingJobStatus\"] != \"Completed\":\n",
    "    while description[\"SecondaryStatus\"] not in {\"Training\", \"Completed\"}:\n",
    "        description = client.describe_training_job(TrainingJobName=job_name)\n",
    "        primary_status = description[\"TrainingJobStatus\"]\n",
    "        secondary_status = description[\"SecondaryStatus\"]\n",
    "        print(\n",
    "            \"Current job status: [PrimaryStatus: {}, SecondaryStatus: {}]\".format(\n",
    "                primary_status, secondary_status\n",
    "            )\n",
    "        )\n",
    "        time.sleep(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "Now we create a new trial object, `trial2`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from smdebug.trials import create_trial\n",
    "\n",
    "trial2 = create_trial(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "### Gradients\n",
    "\n",
    "Lets compare distribution of gradients of the convolutional layers of both trials. `trial` is the trial object of the first training job, `trial2` is the trial object of second training job. We can now easily compare tensors from both training jobs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The first trial (`trial1`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "rule = GradientsLayer(trial1)\n",
    "try:\n",
    "    invoke_rule(rule)\n",
    "except NoMoreData:\n",
    "    print(\n",
    "        \"The training has ended and there is no more data to be analyzed. This is expected behavior.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "dict_gradients = {}\n",
    "dict_gradients[\"gradient/conv0_weight_bad_hyperparameters\"] = rule.tensors[\"gradient/conv0_weight\"]\n",
    "dict_gradients[\"gradient/conv1_weight_bad_hyperparameters\"] = rule.tensors[\"gradient/conv1_weight\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "#### The second trial (`trial2`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "rule = GradientsLayer(trial2)\n",
    "try:\n",
    "    invoke_rule(rule)\n",
    "except NoMoreData:\n",
    "    print(\n",
    "        \"The training has ended and there is no more data to be analyzed. This is expected behavior.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "dict_gradients[\"gradient/conv0_weight_good_hyperparameters\"] = rule.tensors[\"gradient/conv0_weight\"]\n",
    "dict_gradients[\"gradient/conv1_weight_good_hyperparameters\"] = rule.tensors[\"gradient/conv1_weight\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "#### Plot the histograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "utils.create_interactive_matplotlib_histogram(\n",
    "    dict_gradients, filename=\"images/gradients_comparison.gif\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "In the case of the poorly initalized model, gradients are fluctuating a lot leading to very high variance. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "Image(url=\"images/gradients_comparison.gif\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "### Activation inputs\n",
    "\n",
    "Lets compare distribution of activation inputs of both trials."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The first trial (`trial1`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "rule = ActivationInputs(trial1)\n",
    "try:\n",
    "    invoke_rule(rule)\n",
    "except NoMoreData:\n",
    "    print(\n",
    "        \"The training has ended and there is no more data to be analyzed. This is expected behavior.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "dict_activation_inputs = {}\n",
    "dict_activation_inputs[\"conv0_relu_input_0_bad_hyperparameters\"] = rule.tensors[\n",
    "    \"conv0_relu_input_0\"\n",
    "]\n",
    "dict_activation_inputs[\"conv1_relu_input_0_bad_hyperparameters\"] = rule.tensors[\n",
    "    \"conv1_relu_input_0\"\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "#### The second trial (`trial2`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "rule = ActivationInputs(trial2)\n",
    "try:\n",
    "    invoke_rule(rule)\n",
    "except NoMoreData:\n",
    "    print(\n",
    "        \"The training has ended and there is no more data to be analyzed. This is expected behavior.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "dict_activation_inputs[\"conv0_relu_input_0_good_hyperparameters\"] = rule.tensors[\n",
    "    \"conv0_relu_input_0\"\n",
    "]\n",
    "dict_activation_inputs[\"conv1_relu_input_0_good_hyperparameters\"] = rule.tensors[\n",
    "    \"conv1_relu_input_0\"\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "#### Plot the histograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "utils.create_interactive_matplotlib_histogram(\n",
    "    dict_activation_inputs, filename=\"images/activation_inputs_comparison.gif\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "source": [
    "The distribution of activation inputs into first activation layer `conv0_relu_input_0` look quite similar in both trials. However in the case of the second layer they drastically differ. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "papermill": {
     "duration": null,
     "end_time": null,
     "exception": null,
     "start_time": null,
     "status": "pending"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "Image(url=\"images/activation_inputs_comparison.gif\")"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (MXNet 1.8 Python 3.7 CPU Optimized)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/mxnet-1.8-cpu-py37-ubuntu16.04-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 575.345136,
   "end_time": "2021-05-27T00:21:39.654897",
   "environment_variables": {},
   "exception": true,
   "input_path": "mnist_tensor_analysis.ipynb",
   "output_path": "/opt/ml/processing/output/mnist_tensor_analysis-2021-05-27-00-08-02.ipynb",
   "parameters": {
    "kms_key": "arn:aws:kms:us-west-2:521695447989:key/6e9984db-50cf-4c7e-926c-877ec47a8b25"
   },
   "start_time": "2021-05-27T00:12:04.309761",
   "version": "2.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}