Validating_and_Importing_Related_Time_Series_Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Validating and Importing Related Time Series Data\n",
    "\n",
    "## Obtaining Your Data\n",
    "\n",
    "This will take off where you stopped regarding your target time series data. In this particular exmaple, one master file contained both the target and the related time series information. That may or may not be the case for your problem. The goal here is to produce a file that contains the following 2 required attributes:\n",
    "\n",
    "1. Timestamp - Must be of the same format and total range as the target-time series data, as well as slices of values into the dates for your forecast.\n",
    "1. Item_ID - Must exist for all the time stamps for each item in your time series dataset\n",
    "\n",
    "In addition to those attributes we are looking for variables that shift over time that are impactful in some way towards our desired goal of predicting traffic volumes.\n",
    "\n",
    "Again the data was already bundled together for us in this sample so we will skip obtaining it a second time but that is where you would start otherwise.\n",
    "\n",
    "With the data ready to go, skip the blank cell ( feel free to add to it if you need to manipulate your own data) and execute the cells to handle our imports and retrieving our stored values from the previous notebook.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "from time import sleep\n",
    "import subprocess\n",
    "import pandas as pd\n",
    "import json\n",
    "import time\n",
    "import pprint\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.dates import DateFormatter\n",
    "import matplotlib.dates as mdates\n",
    "%store -r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building The Related Time Series File\n",
    "\n",
    "The challenge here is to make sure that we leave absolutely 0 entries with NaN values or the service will throw an error when building a Predictor. This is because the values must be present in order for us to make assumptions about their impact overall."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "related_time_series_df = targtet_df.copy()\n",
    "related_time_series_df.dropna()\n",
    "related_time_series_df = full_df.join(related_time_series_df, how='outer')\n",
    "cols = related_time_series_df.columns.tolist()\n",
    "related_time_series_df[cols] = related_time_series_df[cols].replace('', np.nan).ffill()\n",
    "related_time_series_df = related_time_series_df.loc['2017-01-01':]\n",
    "print (related_time_series_df.index.min())\n",
    "print (related_time_series_df.index.max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see now that the data covers the range of our target time series of 2017's entire year to the end of our known data about 2018. We have not yet defined a forecast horizon yet but it is important to note here that the related data needs to cover that time span. To spoil later work, the horizon for us is 480 hours or 20 days, plenty of time with 9 months of validation data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly on prepping the base set of the data we validate there are no blanks or NaNs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "related_time_series_df[related_time_series_df.isnull().any(axis=1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Look at the columns and decide what we should keep:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "related_time_series_df.sample(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few things to note here:\n",
    "\n",
    "* Holidays are not needed given this date is in the US, we can just use the Holidays feature within Forecast: https://docs.aws.amazon.com/forecast/latest/dg/API_SupplementaryFeature.html\n",
    "* Weather description seems to have more variety\n",
    "* Traffic volume will be removed here. \n",
    "* We still need to add back the item_id field.\n",
    "\n",
    "This leaves us with the following schema:\n",
    "\n",
    "* `timestamp` - The Index\n",
    "* `temp` - float\n",
    "* `rain_1h` - float\n",
    "* `snow_1h` - float\n",
    "* `clouds_all` - float\n",
    "* `weather_description` - string\n",
    "* `item_ID` - string\n",
    "\n",
    "The cell below will build that file for us.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Restrict the columns to timestamp, traffic_volume\n",
    "related_time_series_df = related_time_series_df[['temp', 'rain_1h', 'snow_1h', 'clouds_all', 'weather_description']]\n",
    "# Add in item_id\n",
    "related_time_series_df['item_ID'] = \"1\"\n",
    "# Validate the structure\n",
    "related_time_series_df.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save it off as a file:\n",
    "related_time_series_filename = \"related_time_series.csv\"\n",
    "related_time_series_path = data_dir + \"/\" + related_time_series_filename\n",
    "related_time_series_df.to_csv(related_time_series_path, header=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding Related Data to the DatasetGroup\n",
    "\n",
    "Next we are going to create a related-time-series dataset, then add it to our dataset group and finally import our information and validate that it looks good. We will also delete this dataset import after we are done so that the first models do not yet receive any extra info from the related data.\n",
    "\n",
    "You can of course to not delete and get started right away with related data informed models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session = boto3.Session(region_name=region)\n",
    "forecast = session.client(service_name='forecast')\n",
    "forecast_query = session.client(service_name='forecastquery')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upload Related File\n",
    "boto3.Session().resource('s3').Bucket(bucket_name).Object(related_time_series_filename).upload_file(related_time_series_path)\n",
    "related_s3DataPath = \"s3://\"+bucket_name+\"/\"+related_time_series_filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify the schema of your dataset here. Make sure the order of columns matches the raw data files.\n",
    "related_schema ={\n",
    "   \"Attributes\":[\n",
    "      {\n",
    "         \"AttributeName\":\"timestamp\",\n",
    "         \"AttributeType\":\"timestamp\"\n",
    "      },\n",
    "      {\n",
    "         \"AttributeName\":\"temperature\",\n",
    "         \"AttributeType\":\"float\"\n",
    "      },\n",
    "       {\n",
    "         \"AttributeName\":\"rain_1h\",\n",
    "         \"AttributeType\":\"float\"\n",
    "      },\n",
    "       {\n",
    "         \"AttributeName\":\"snow_1h\",\n",
    "         \"AttributeType\":\"float\"\n",
    "      },\n",
    "       {\n",
    "         \"AttributeName\":\"clouds_all\",\n",
    "         \"AttributeType\":\"float\"\n",
    "      },\n",
    "       {\n",
    "         \"AttributeName\":\"weather\",\n",
    "         \"AttributeType\":\"string\"\n",
    "      },\n",
    "      {\n",
    "         \"AttributeName\":\"item_id\",\n",
    "         \"AttributeType\":\"string\"\n",
    "      }\n",
    "   ]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "related_DSN = datasetName + \"_related\"\n",
    "response=forecast.create_dataset(\n",
    "                    Domain=\"CUSTOM\",\n",
    "                    DatasetType='RELATED_TIME_SERIES',\n",
    "                    DatasetName=related_DSN,\n",
    "                    DataFrequency=DATASET_FREQUENCY, \n",
    "                    Schema = related_schema\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "related_datasetArn = response['DatasetArn']\n",
    "forecast.describe_dataset(DatasetArn=related_datasetArn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datasetImportJobName = 'DSIMPORT_JOB_RELATEDPOC'\n",
    "related_ds_import_job_response=forecast.create_dataset_import_job(DatasetImportJobName=datasetImportJobName,\n",
    "                                                          DatasetArn=related_datasetArn,\n",
    "                                                          DataSource= {\n",
    "                                                              \"S3Config\" : {\n",
    "                                                                 \"Path\":related_s3DataPath,\n",
    "                                                                 \"RoleArn\": role_arn\n",
    "                                                              } \n",
    "                                                          },\n",
    "                                                          TimestampFormat=TIMESTAMP_FORMAT\n",
    "                                                         )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rel_ds_import_job_arn=related_ds_import_job_response['DatasetImportJobArn']\n",
    "print(rel_ds_import_job_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cell below will poll until the import process has completed, once that has been accomplished we can review the metrics and decide to delete the data or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "while True:\n",
    "    dataImportStatus = forecast.describe_dataset_import_job(DatasetImportJobArn=rel_ds_import_job_arn)['Status']\n",
    "    print(dataImportStatus)\n",
    "    if dataImportStatus != 'ACTIVE' and dataImportStatus != 'CREATE_FAILED':\n",
    "        sleep(30)\n",
    "    else:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating the Related Time Series Data\n",
    "\n",
    "First let us examine the dataframe that we provided to Forecast:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "related_time_series_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "related_time_series_df.sample(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above we see 18,609 entries and not one is a NaN value! This is perfect. Now to double check what we imported:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast.describe_dataset_import_job(DatasetImportJobArn=rel_ds_import_job_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fantastict! No NaNs or nulls and the entire dataset is ready to go. If you'd like to delete this information so you can build your models without related data simply uncomment the cell below. Once that is done you are ready to move forward building your models with Amazon Forecast."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#forecast.delete_dataset_import_job(DatasetImportJobArn=rel_ds_import_job_arn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}