API_Umbrella_Elasticsearch_workflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring API Umbrella Elasticsearch analytics\n",
    "This notebook shows some basic techniques to explore analytics data from an API Umbrella Elasticsearch instance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports\n",
    "This notebook relies on several libraries. They are imported, and configured where possible, below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from pprint import pprint\n",
    "\n",
    "# Data import\n",
    "from elasticsearch import Elasticsearch\n",
    "from elasticsearch_dsl import Search\n",
    "\n",
    "# Data analysis/exploration\n",
    "import pandas as pd\n",
    "from pandas.io.json import json_normalize\n",
    "import numpy as np\n",
    "\n",
    "# Visualization\n",
    "import matplotlib.pyplot as plt\n",
    "plt.figure(figsize=(17,7))\n",
    "\n",
    "import seaborn as sns\n",
    "sns.set(style=\"white\", color_codes=True)\n",
    "\n",
    "# Classification\n",
    "from sklearn import preprocessing\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.model_selection import train_test_split\n",
    "from tpot import TPOTClassifier\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Elasticsearch initialization and query\n",
    "The data for this notebook originates from an Elasticsearch server. The following code initializes the Elasticsearch client and requests the analytics data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get Elasticsearch URL from environment variable\n",
    "elasticsearch_url = os.environ['ELASTICSEARCH_URL']\n",
    "\n",
    "# Create Elasticsearch client\n",
    "client = Elasticsearch([elasticsearch_url])\n",
    "\n",
    "# Make sure client can connect to Elasticsearch instance\n",
    "client.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create Elasticsearch search instance\n",
    "search = Search(using=client)\n",
    "\n",
    "# Get the total number of results\n",
    "total = search.count()\n",
    "\n",
    "# Update the search instance to contain all results\n",
    "search = search[0:total]\n",
    "\n",
    "# Execute the search\n",
    "results = search.execute()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas DataFrame\n",
    "In order to explore the data, we want to load it into a Pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the search results to a Pandas DataFrame\n",
    "results_df = json_normalize(results.hits.hits)\n",
    "\n",
    "# List dataframe columns\n",
    "for column in results_df.columns:\n",
    "    print(column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count number of results containing value for each column\n",
    "results_count_sorted = results_df.count().sort_values(ascending=False).iteritems()\n",
    "\n",
    "# Print each metric and value\n",
    "for key, value in results_count_sorted:\n",
    "    print(key, value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add request_date column by converting request_at column to datetime\n",
    "results_df['request_date'] = pd.to_datetime(results_df['_source.request_at'], unit='ms')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use request_date for dataframe index\n",
    "results_df.set_index('request_date', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Derived column(s)\n",
    "We may want to compute some values, based on existing columns. For example, determining whether requests are successful or failure, based on status code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Is success?\n",
    "Check whether the request was successful by looking at status code. Divide the data into two groups 'success' and 'failure', where success is any response with a 2xx status code and failure is any other response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def is_success_function(row):\n",
    "    \"\"\"\n",
    "    Check if row (request) was successful\n",
    "    Successful means a response with 2xx status\n",
    "    \n",
    "    return True if successful False otherwise\n",
    "    \"\"\"\n",
    "    success = (row['_source.response_status'] >= 200 and row['_source.response_status'] < 300)\n",
    "    \n",
    "    if success:\n",
    "        return 'Success'\n",
    "    else:\n",
    "        return 'Failure'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add 'Success' column to API Logs\n",
    "# Success is determined by status code, where any status in the 200s is considered successful\n",
    "results_df['Outcome'] = results_df.apply(is_success_function, axis='columns')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Response status class\n",
    "Divide the data into four groups, based on response status:\n",
    "\n",
    "- 2xx: 'success'\n",
    "- 3xx: 'warning'\n",
    "- 4xx: 'client_error'\n",
    "- 5xx: 'server_error'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def determine_response_class(row):\n",
    "    \"\"\"\n",
    "    Check the status code of a given row, and return a response class.\n",
    "    \n",
    "    - 2xx: 'success'\n",
    "    - 3xx: 'warning'\n",
    "    - 4xx: 'client_error'\n",
    "    - 5xx: 'server_error'\n",
    "    - NaN or 0: 'unknown'\n",
    "    \"\"\"\n",
    "    \n",
    "    response_status = row['_source.response_status']\n",
    "    \n",
    "    if response_status >= 200 and response_status < 300:\n",
    "        return 'success'\n",
    "    elif response_status >= 300 and response_status < 400:\n",
    "        return 'warning'\n",
    "    elif response_status >= 400 and response_status < 500:\n",
    "        return 'client_error'\n",
    "    elif response_status >= 500 and response_status < 600:\n",
    "        return 'server_error'\n",
    "    else:\n",
    "        return 'unknown'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add 'Response class' column to API Logs, where response class is determined by HTTP status code\n",
    "results_df['Response class'] = results_df.apply(determine_response_class, axis='columns')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Status codes\n",
    "Status code indicates whether request was, more or less, successful or failure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count response status codes\n",
    "status_code_value_counts = results_df['_source.response_status'].value_counts()\n",
    "\n",
    "status_code_value_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Color map for response classes\n",
    "responseClassPalette = {\n",
    "        'success': 'green',\n",
    "        'warning': 'yellow',\n",
    "        'client_error': 'orange',\n",
    "        'server_error': 'red',\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sorted distribution of statuses in the data\n",
    "api_response_code_distributions_plot = status_code_value_counts.plot('bar', title='Response code counts', figsize=(17, 7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make the plot wider and taller\n",
    "plt.figure(figsize=(17,7))\n",
    "\n",
    "# Use seaborn to produce a countplot of response status codes\n",
    "status_code_counts_plot = sns.countplot(\n",
    "    x='_source.response_status',\n",
    "    data=results_df,\n",
    "    color='teal',\n",
    "    saturation=0.7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Group responses by HTTP response code and count number of responses for each code\n",
    "http_status_code_counts = results_df.groupby([\"_source.response_status\",\"Response class\"]).size().reset_index(name=\"count\")\n",
    "\n",
    "# Sort the grouped/counted HTTP response codes in ascending order\n",
    "http_status_code_counts_sorted = http_status_code_counts.sort_values(by=\"count\", ascending=0)\n",
    "\n",
    "# Initialize plot\n",
    "fig, axes = plt.subplots(ncols=4, sharey=True, figsize=(17,5))\n",
    "\n",
    "# Group http status code counts by response class, and plot each class in a row\n",
    "for ax,(n, group) in zip(axes, http_status_code_counts_sorted.groupby(\"Response class\")):\n",
    "    sns.barplot(x=\"_source.response_status\",y=\"count\", data=group, ax=ax, color=\"C0\", order=group[\"_source.response_status\"])\n",
    "    ax.set_title(n)\n",
    "\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Success or failure\n",
    "Show the proportion of calls that are successful (2xx) or failure (3xx, 4xx, 5xx)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "success_failure_pairplot = sns.pairplot(\n",
    "    results_df,\n",
    "    vars=[\n",
    "        '_source.response_time',\n",
    "        '_source.response_size',\n",
    "        '_source.proxy_overhead',\n",
    "        '_source.internal_gatekeeper_time',\n",
    "    ],\n",
    "    hue='Response class',\n",
    "    palette=responseClassPalette\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Proxy overhead\n",
    "Proxy overhead is a measurement of how much latency the proxy adds to a request."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame.hist(results_df, column='_source.proxy_overhead', bins=15, figsize=(17, 7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the daily median proxy overhead\n",
    "daily_median_proxy_overhead = results_df['_source.proxy_overhead'].resample('D').median()\n",
    "\n",
    "# Fill empty values with zero\n",
    "daily_median_proxy_overhead_filled = daily_median_proxy_overhead.fillna(0)\n",
    "\n",
    "# Plot chart and save as variable\n",
    "daily_median_proxy_overhead_chart = daily_median_proxy_overhead_filled.plot(\n",
    "    figsize=(17, 7),\n",
    "    title='Daily median proxy overhead')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysis\n",
    "Look for patterns and correlations in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find correlation between columns in the analytics DataFrame\n",
    "correlation_matrix = results_df.corr()\n",
    "\n",
    "# Create mask for upper right half of heatmap\n",
    "mask = np.zeros_like(correlation_matrix)\n",
    "mask[np.triu_indices_from(mask)] = True\n",
    "\n",
    "# Make the plot wider and taller\n",
    "plt.figure(figsize=(17,7))\n",
    "\n",
    "# Show correlations as a heatmap with white background\n",
    "correlation_heatmap = sns.heatmap(correlation_matrix, annot=True, fmt='.2f', mask=mask, square=True, vmax=.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show relationship proxy overhead and response content length\n",
    "proxy_overhead_response_content_length_plot = sns.jointplot(\n",
    "    data=results_df,\n",
    "    x='_source.proxy_overhead',\n",
    "    y='_source.response_content_length',\n",
    "    kind=\"reg\",\n",
    "    size=10,\n",
    ").set_axis_labels(\"Proxy overhead\", \"Response content length\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show relationship between proxy overhead and response time\n",
    "proxy_overhead_response_time_plot = sns.jointplot(\n",
    "    data=results_df,\n",
    "    x='_source.proxy_overhead',\n",
    "    y='_source.response_time',\n",
    "    kind=\"reg\",\n",
    "    size=10,\n",
    ").set_axis_labels(\"Proxy overhead\", \"Response time\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification\n",
    "Create classifiers to automatically detect patterns in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select columns for classification, including label column (Outcome)\n",
    "classification_df = results_df[[\n",
    "    '_source.response_time',\n",
    "    '_source.response_size',\n",
    "    '_source.proxy_overhead',\n",
    "    '_source.internal_gatekeeper_time',\n",
    "    '_source.request_size',\n",
    "    'Outcome'\n",
    "]].dropna() # Remove rows with NA values (e.g. NaN)\n",
    "\n",
    "# Select classification columns\n",
    "X = classification_df[['_source.response_time', '_source.response_size', '_source.proxy_overhead', '_source.internal_gatekeeper_time', '_source.request_size']]\n",
    "\n",
    "# Create scaler for classification values\n",
    "min_max_scaler = preprocessing.MinMaxScaler()\n",
    "\n",
    "# Scale classification values to values from 0 to 1\n",
    "X_scaled = min_max_scaler.fit_transform(X)\n",
    "\n",
    "# Create a label (category) encoder object\n",
    "le = preprocessing.LabelEncoder()\n",
    "\n",
    "# Determine integer mapping for label column\n",
    "category_integers = le.fit(classification_df['Outcome'])\n",
    "\n",
    "# Map labels to integers\n",
    "y = category_integers.transform(classification_df['Outcome'])\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, train_size=0.75, test_size=0.25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create TPOT pipeline optimizer\n",
    "pipeline_optimizer = TPOTClassifier(generations=5, population_size=20, cv=5,\n",
    "                                    random_state=42, verbosity=2)\n",
    "\n",
    "# Find optimal classifier and parameters\n",
    "pipeline_optimizer.fit(X_train, y_train)\n",
    "\n",
    "# Print details about optimal classifier\n",
    "print(pipeline_optimizer.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_optimizer.export('tpot_exported_pipeline.py')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}