Creating_and_Evaluating_Related_Time_Predictors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating and Evaluating Predictors: Part 2 - Related Time Series\n",
    "\n",
    "This notebook will build off of all the ealrier work and requires that at least the importing of target time series and related time series data be complete. If you have not performed those steps yet, go back, do so, then continue.\n",
    "\n",
    "At this point you now have a target-time-series dataset and a related-time-series dataset loaded into a singular Dataset Group, this is what is required to leverage the models that support related data in Amazon Forecast. If your data supports item level metadata it could be added to the dataset group as well and would benefit only DeepAR+. \n",
    "\n",
    "To continue the work, start with the imports, determine your region, establish your API connections, and load all previously stored values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "from time import sleep\n",
    "import subprocess\n",
    "import pandas as pd\n",
    "import json\n",
    "import time\n",
    "import pprint\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.dates import DateFormatter\n",
    "import matplotlib.dates as mdates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('/opt/ml/metadata/resource-metadata.json') as notebook_info:\n",
    "    data = json.load(notebook_info)\n",
    "    resource_arn = data['ResourceArn']\n",
    "    region = resource_arn.split(':')[3]\n",
    "print(region)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session = boto3.Session(region_name=region)\n",
    "forecast = session.client(service_name='forecast')\n",
    "forecast_query = session.client(service_name='forecastquery')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating and Training Predictors\n",
    " \n",
    "Given that that our data is hourly and we want to generate a forecast on the hour, Forecast limits us to a horizon of 500 of whatever the slice is. This means we will be able to predict about 20 days into the future.\n",
    "\n",
    "The cells below will define a few variables to be used with all of our models. Then there will be an API call to create each `Predictor` where they are based on Prophet and DeepAR+ respectfully.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecastHorizon = 480\n",
    "NumberOfBacktestWindows = 4\n",
    "BackTestWindowOffset = 480\n",
    "ForecastFrequency = \"H\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arima_algorithmArn = 'arn:aws:forecast:::algorithm/ARIMA'\n",
    "prophet_algorithmArn = 'arn:aws:forecast:::algorithm/Prophet'\n",
    "deepAR_Plus_algorithmArn = 'arn:aws:forecast:::algorithm/Deep_AR_Plus'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prophet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prophet Specifics\n",
    "# Note the REL to indicate related time series data\n",
    "prophet_predictorName= project+'_prophet_rel_algo_1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build Prophet:\n",
    "prophet_create_predictor_response=forecast.create_predictor(PredictorName=prophet_predictorName, \n",
    "                                                  AlgorithmArn=prophet_algorithmArn,\n",
    "                                                  ForecastHorizon=forecastHorizon,\n",
    "                                                  PerformAutoML= False,\n",
    "                                                  PerformHPO=False,\n",
    "                                                  EvaluationParameters= {\"NumberOfBacktestWindows\": NumberOfBacktestWindows, \n",
    "                                                                         \"BackTestWindowOffset\": BackTestWindowOffset}, \n",
    "                                                  InputDataConfig= {\"DatasetGroupArn\": datasetGroupArn, \"SupplementaryFeatures\": [ \n",
    "                                                                     { \n",
    "                                                                        \"Name\": \"holiday\",\n",
    "                                                                        \"Value\": \"US\"\n",
    "                                                                     }\n",
    "                                                                  ]},\n",
    "                                                  FeaturizationConfig= {\"ForecastFrequency\": ForecastFrequency, \n",
    "                                                                        \"Featurizations\": \n",
    "                                                                        [\n",
    "                                                                          {\"AttributeName\": \"target_value\", \n",
    "                                                                           \"FeaturizationPipeline\": \n",
    "                                                                            [\n",
    "                                                                              {\"FeaturizationMethodName\": \"filling\", \n",
    "                                                                               \"FeaturizationMethodParameters\": \n",
    "                                                                                {\"frontfill\": \"none\", \n",
    "                                                                                 \"middlefill\": \"zero\", \n",
    "                                                                                 \"backfill\": \"zero\"}\n",
    "                                                                              }\n",
    "                                                                            ]\n",
    "                                                                          }\n",
    "                                                                        ]\n",
    "                                                                       }\n",
    "                                                 )\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DeepAR+"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DeepAR+ Specifics\n",
    "prophet_predictorName= project+'_deeparp_rel_algo_1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build DeepAR+:\n",
    "deeparp_create_predictor_response=forecast.create_predictor(PredictorName=prophet_predictorName, \n",
    "                                                  AlgorithmArn=deepAR_Plus_algorithmArn,\n",
    "                                                  ForecastHorizon=forecastHorizon,\n",
    "                                                  PerformAutoML= False,\n",
    "                                                  PerformHPO=False,\n",
    "                                                  EvaluationParameters= {\"NumberOfBacktestWindows\": NumberOfBacktestWindows, \n",
    "                                                                         \"BackTestWindowOffset\": BackTestWindowOffset}, \n",
    "                                                  InputDataConfig= {\"DatasetGroupArn\": datasetGroupArn, \"SupplementaryFeatures\": [ \n",
    "                                                                     { \n",
    "                                                                        \"Name\": \"holiday\",\n",
    "                                                                        \"Value\": \"US\"\n",
    "                                                                     }\n",
    "                                                                  ]},\n",
    "                                                  FeaturizationConfig= {\"ForecastFrequency\": ForecastFrequency, \n",
    "                                                                        \"Featurizations\": \n",
    "                                                                        [\n",
    "                                                                          {\"AttributeName\": \"target_value\", \n",
    "                                                                           \"FeaturizationPipeline\": \n",
    "                                                                            [\n",
    "                                                                              {\"FeaturizationMethodName\": \"filling\", \n",
    "                                                                               \"FeaturizationMethodParameters\": \n",
    "                                                                                {\"frontfill\": \"none\", \n",
    "                                                                                 \"middlefill\": \"zero\", \n",
    "                                                                                 \"backfill\": \"zero\"}\n",
    "                                                                              }\n",
    "                                                                            ]\n",
    "                                                                          }\n",
    "                                                                        ]\n",
    "                                                                       }\n",
    "                                                 )\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally in our notebooks we would have a while loop that polls for each of these to determine the status of the models in training. For simplicity sake here we are going to rely on you opening a new browser tab and following along in the console until a predictor has been created for each algorithm. \n",
    "\n",
    "Your previous tab from opening this session of Jupyter Lab should still be open, from there navigate to the Amazon Forecast service page, then select your dataset group. Lastly click `Predictors` and you should see the creation in progress. Once they are active you are ready to continue."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examining the Predictors\n",
    "\n",
    "Once each of the Predictors is in an `Active` state you can get metrics about it to better understand its accuracy and behavior. These are computed based on the hold out periods we defined when building the Predictor. The metrics are meant to guide our decisions when we use a particular Predictor to generate a forecast"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prophet\n",
    "\n",
    "Here we are going to look to see the metrics from this Predictor like the earlier sessions, we will now add the related data metrics to the table from the previous notebook as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prophet Metrics\n",
    "prophet_arn = prophet_create_predictor_response['PredictorArn']\n",
    "prophet_metrics = forecast.get_accuracy_metrics(PredictorArn=prophet_arn)\n",
    "pp = pprint.PrettyPrinter()\n",
    "pp.pprint(prophet_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see an RMSE of 2337.240584382631 which is unfortunately a little worse than the original RMSE indicating that we may not be best served using related data for this algorithm.\n",
    "\n",
    "\n",
    "| Predictor              | RMSE               | 10%                 | 50%                 | 90%                 |\n",
    "|------------------------|--------------------|---------------------|---------------------|---------------------|\n",
    "| ARIMA                  | 2315.5733958045103 | 0.18638100224895027 | 0.35619575117523433 | 0.2906928828133737  |\n",
    "| Prophet                | 2326.907237203276  | 0.15434241067213655 | 0.3880618345549856  | 0.28403268455977804 |\n",
    "| Prophet + Related Data | 2337.240584382631  | 0.15668701263879603 | 0.38782245560595247 | 0.28403480713820367 |\n",
    "| DeepAR+                | 2343.76942963439   | 0.09470210806720818 | 0.27579575099681214 | 0.32289195793540876 |\n",
    "\n",
    "When digging into the metrics we did not see a single improvement to Prophet, next let us see how DeepAR+ performed.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DeepAR+\n",
    "\n",
    "Same as Prophet, now you should look at the metrics from it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DeepAR+ Metrics\n",
    "deeparp_arn = deeparp_create_predictor_response['PredictorArn']\n",
    "deeparp_metrics = forecast.get_accuracy_metrics(PredictorArn=deeparp_arn)\n",
    "pp = pprint.PrettyPrinter()\n",
    "pp.pprint(deeparp_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now after training with DeepAR+ we can seen an RMSE of 2339.5169670021673 which is still not ideal but the full break down is:\n",
    "\n",
    "\n",
    "| Predictor              | RMSE               | 10%                 | 50%                 | 90%                 |\n",
    "|------------------------|--------------------|---------------------|---------------------|---------------------|\n",
    "| ARIMA                  | 2315.5733958045103 | 0.18638100224895027 | 0.35619575117523433 | 0.2906928828133737  |\n",
    "| Prophet                | 2326.907237203276  | 0.15434241067213655 | 0.3880618345549856  | 0.28403268455977804 |\n",
    "| Prophet & Related Data | 2337.240584382631  | 0.15668701263879603 | 0.38782245560595247 | 0.28403480713820367 |\n",
    "| DeepAR+                | 2343.76942963439   | 0.09470210806720818 | 0.27579575099681214 | 0.32289195793540876 |\n",
    "| DeepAR+ & Related Data | 2339.5169670021673 | 0.09773726043488253 | 0.2809238443812311  | 0.30924327596172524 |\n",
    "\n",
    "\n",
    "From this table we can see that DeepAR+ with the related data is th eleader for the 10% and 50% quantiles. If you are predicting in this range then it is a clear leader for usage. However if 90% is the target then Prophet without related data is the leader for now.\n",
    "\n",
    "Additional work would need to be kicked off from here to determine the specific impact of these figures and how they compare to the existing Forecasting approaches performed by your customer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}