python-sdk/tutorials/automl-with-azureml/forecasting-backtest-many-models/auto-ml-forecasting-backtest-many-models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "\n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/forecasting-hierarchical-timeseries/auto-ml-forecasting-hierarchical-timeseries.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Many Models with Backtesting - Automated ML\n",
    "**_Backtest many models time series forecasts with Automated Machine Learning_**\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this notebook we are using a synthetic dataset to demonstrate the back testing in many model scenario. This allows us to check historical performance of AutoML on a historical data. To do that we step back on the backtesting period by the data set several times and split the data to train and test sets. Then these data sets are used for training and evaluation of model.<br>\n",
    "\n",
    "Thus, it is a quick way of evaluating AutoML as if it was in production. Here, we do not test historical performance of a particular model, for this see the [notebook](../forecasting-backtest-single-model/auto-ml-forecasting-backtest-single-model.ipynb). Instead, the best model for every backtest iteration can be different since AutoML chooses the best model for a given training set.\n",
    "![Backtesting](Backtesting.png)\n",
    "\n",
    "**NOTE: There are limits on how many runs we can do in parallel per workspace, and we currently recommend to set the parallelism to maximum of 320 runs per experiment per workspace. If users want to have more parallelism and increase this limit they might encounter Too Many Requests errors (HTTP 429).**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prerequisites\n",
    "You'll need to create a compute Instance by following the instructions in the [EnvironmentSetup.md](../Setup_Resources/EnvironmentSetup.md)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.0 Set up workspace, datastore, experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1613003526897
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import azureml.core\n",
    "from azureml.core import Workspace, Datastore\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from pandas.tseries.frequencies import to_offset\n",
    "\n",
    "# Set up your workspace\n",
    "ws = Workspace.from_config()\n",
    "ws.get_details()\n",
    "\n",
    "# Set up your datastores\n",
    "dstore = ws.get_default_datastore()\n",
    "\n",
    "output = {}\n",
    "output[\"SDK version\"] = azureml.core.VERSION\n",
    "output[\"Subscription ID\"] = ws.subscription_id\n",
    "output[\"Workspace\"] = ws.name\n",
    "output[\"Resource Group\"] = ws.resource_group\n",
    "output[\"Location\"] = ws.location\n",
    "output[\"Default datastore name\"] = dstore.name\n",
    "pd.set_option(\"display.max_colwidth\", -1)\n",
    "outputDf = pd.DataFrame(data=output, index=[\"\"])\n",
    "outputDf.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is compatible with Azure ML SDK version 1.35.1 or later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choose an experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1613003540729
    }
   },
   "outputs": [],
   "source": [
    "from azureml.core import Experiment\n",
    "\n",
    "experiment = Experiment(ws, \"automl-many-models-backtest\")\n",
    "\n",
    "print(\"Experiment name: \" + experiment.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.0 Data\n",
    "\n",
    "#### 2.1 Data generation\n",
    "For this notebook we will generate the artificial data set with two [time series IDs](https://docs.microsoft.com/en-us/python/api/azureml-automl-core/azureml.automl.core.forecasting_parameters.forecastingparameters?view=azure-ml-py). Then we will generate backtest folds and will upload it to the default BLOB storage and create a [TabularDataset](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.tabular_dataset.tabulardataset?view=azure-ml-py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# simulate data: 2 grains - 700\n",
    "TIME_COLNAME = \"date\"\n",
    "TARGET_COLNAME = \"value\"\n",
    "TIME_SERIES_ID_COLNAME = \"ts_id\"\n",
    "\n",
    "sample_size = 700\n",
    "# Set the random seed for reproducibility of results.\n",
    "np.random.seed(20)\n",
    "X1 = pd.DataFrame(\n",
    "    {\n",
    "        TIME_COLNAME: pd.date_range(start=\"2018-01-01\", periods=sample_size),\n",
    "        TARGET_COLNAME: np.random.normal(loc=100, scale=20, size=sample_size),\n",
    "        TIME_SERIES_ID_COLNAME: \"ts_A\",\n",
    "    }\n",
    ")\n",
    "X2 = pd.DataFrame(\n",
    "    {\n",
    "        TIME_COLNAME: pd.date_range(start=\"2018-01-01\", periods=sample_size),\n",
    "        TARGET_COLNAME: np.random.normal(loc=100, scale=20, size=sample_size),\n",
    "        TIME_SERIES_ID_COLNAME: \"ts_B\",\n",
    "    }\n",
    ")\n",
    "\n",
    "X = pd.concat([X1, X2], ignore_index=True, sort=False)\n",
    "print(\"Simulated dataset contains {} rows \\n\".format(X.shape[0]))\n",
    "X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will generate 8 backtesting folds with backtesting period of 7 days and with the same forecasting horizon. We will add the column \"backtest_iteration\", which will identify the backtesting period by the last training date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "offset_type = \"7D\"\n",
    "NUMBER_OF_BACKTESTS = 8  # number of train/test sets to generate\n",
    "\n",
    "dfs_train = []\n",
    "dfs_test = []\n",
    "for ts_id, df_one in X.groupby(TIME_SERIES_ID_COLNAME):\n",
    "\n",
    "    data_end = df_one[TIME_COLNAME].max()\n",
    "\n",
    "    for i in range(NUMBER_OF_BACKTESTS):\n",
    "        train_cutoff_date = data_end - to_offset(offset_type)\n",
    "        df_one = df_one.copy()\n",
    "        df_one[\"backtest_iteration\"] = \"iteration_\" + str(train_cutoff_date)\n",
    "        train = df_one[df_one[TIME_COLNAME] <= train_cutoff_date]\n",
    "        test = df_one[\n",
    "            (df_one[TIME_COLNAME] > train_cutoff_date)\n",
    "            & (df_one[TIME_COLNAME] <= data_end)\n",
    "        ]\n",
    "        data_end = train[TIME_COLNAME].max()\n",
    "        dfs_train.append(train)\n",
    "        dfs_test.append(test)\n",
    "\n",
    "X_train = pd.concat(dfs_train, sort=False, ignore_index=True)\n",
    "X_test = pd.concat(dfs_test, sort=False, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2 Create the Tabular Data Set.\n",
    "\n",
    "A Datastore is a place where data can be stored that is then made accessible to a compute either by means of mounting or copying the data to the compute target.\n",
    "\n",
    "Please refer to [Datastore](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.datastore(class)?view=azure-ml-py) documentation on how to access data from Datastore.\n",
    "\n",
    "In this next step, we will upload the data and create a TabularDataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.data.dataset_factory import TabularDatasetFactory\n",
    "\n",
    "ds = ws.get_default_datastore()\n",
    "# Upload saved data to the default data store.\n",
    "train_data = TabularDatasetFactory.register_pandas_dataframe(\n",
    "    X_train, target=(ds, \"data_mm\"), name=\"data_train\"\n",
    ")\n",
    "test_data = TabularDatasetFactory.register_pandas_dataframe(\n",
    "    X_test, target=(ds, \"data_mm\"), name=\"data_test\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.0 Build the training pipeline\n",
    "Now that the dataset, WorkSpace, and datastore are set up, we can put together a pipeline for training.\n",
    "\n",
    "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choose a compute target\n",
    "\n",
    "You will need to create a [compute target](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-set-up-training-targets#amlcompute) for your AutoML run. In this tutorial, you create AmlCompute as your training compute resource.\n",
    "\n",
    "\\*\\*Creation of AmlCompute takes approximately 5 minutes.**\n",
    "\n",
    "If the AmlCompute with that name is already in your workspace this code will skip the creation process. As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read this [article](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-manage-quotas) on the default limits and how to request more quota."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1613007037308
    }
   },
   "outputs": [],
   "source": [
    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
    "\n",
    "# Name your cluster\n",
    "compute_name = \"backtest-mm\"\n",
    "\n",
    "\n",
    "if compute_name in ws.compute_targets:\n",
    "    compute_target = ws.compute_targets[compute_name]\n",
    "    if compute_target and type(compute_target) is AmlCompute:\n",
    "        print(\"Found compute target: \" + compute_name)\n",
    "else:\n",
    "    print(\"Creating a new compute target...\")\n",
    "    provisioning_config = AmlCompute.provisioning_configuration(\n",
    "        vm_size=\"STANDARD_DS12_V2\", max_nodes=6\n",
    "    )\n",
    "    # Create the compute target\n",
    "    compute_target = ComputeTarget.create(ws, compute_name, provisioning_config)\n",
    "\n",
    "    # Can poll for a minimum number of nodes and for a specific timeout.\n",
    "    # If no min node count is provided it will use the scale settings for the cluster\n",
    "    compute_target.wait_for_completion(\n",
    "        show_output=True, min_node_count=None, timeout_in_minutes=20\n",
    "    )\n",
    "\n",
    "    # For a more detailed view of current cluster status, use the 'status' property\n",
    "    print(compute_target.status.serialize())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up training parameters\n",
    "\n",
    "This dictionary defines the AutoML and many models settings. For this forecasting task we need to define several settings including the name of the time column, the maximum forecast horizon, and the partition column name definition. Please note, that in this case we are setting grain_column_names to be the time series ID column plus iteration, because we want to train a separate model for each time series and iteration.\n",
    "\n",
    "| Property                           | Description|\n",
    "| :---------------                   | :------------------- |\n",
    "| **task**                           | forecasting |\n",
    "| **primary_metric**                 | This is the metric that you want to optimize.<br> Forecasting supports the following primary metrics <br><i>normalized_root_mean_squared_error</i><br><i>normalized_mean_absolute_error</i> |\n",
    "| **iteration_timeout_minutes**      | Maximum amount of time in minutes that the model can train. This is optional but provides customers with greater control on exit criteria. |\n",
    "| **iterations**                     | Number of models to train. This is optional but provides customers with greater control on exit criteria. |\n",
    "| **experiment_timeout_hours**       | Maximum amount of time in hours that the experiment can take before it terminates. This is optional but provides customers with greater control on exit criteria. |\n",
    "| **label_column_name**              | The name of the label column. |\n",
    "| **max_horizon**               | The forecast horizon is how many periods forward you would like to forecast. This integer horizon is in units of the timeseries frequency (e.g. daily, weekly). Periods are inferred from your data. |\n",
    "| **n_cross_validations**            | Number of cross validation splits. Rolling Origin Validation is used to split time-series in a temporally consistent way. |\n",
    "| **time_column_name**               | The name of your time column. |\n",
    "| **grain_column_names**     | The column names used to uniquely identify timeseries in data that has multiple rows with the same timestamp. |\n",
    "| **track_child_runs**               | Flag to disable tracking of child runs. Only best run is tracked if the flag is set to False (this includes the model and metrics of the run). |\n",
    "| **partition_column_names**         | The names of columns used to group your models. For timeseries, the groups must not split up individual time-series. That is, each group must contain one or more whole time-series. |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1613007061544
    }
   },
   "outputs": [],
   "source": [
    "from azureml.train.automl.runtime._many_models.many_models_parameters import (\n",
    "    ManyModelsTrainParameters,\n",
    ")\n",
    "\n",
    "partition_column_names = [TIME_SERIES_ID_COLNAME, \"backtest_iteration\"]\n",
    "automl_settings = {\n",
    "    \"task\": \"forecasting\",\n",
    "    \"primary_metric\": \"normalized_root_mean_squared_error\",\n",
    "    \"iteration_timeout_minutes\": 10,  # This needs to be changed based on the dataset. We ask customer to explore how long training is taking before settings this value\n",
    "    \"iterations\": 15,\n",
    "    \"experiment_timeout_hours\": 0.25,  # This also needs to be changed based on the dataset. For larger data set this number needs to be bigger.\n",
    "    \"label_column_name\": TARGET_COLNAME,\n",
    "    \"n_cross_validations\": 3,\n",
    "    \"time_column_name\": TIME_COLNAME,\n",
    "    \"max_horizon\": 6,\n",
    "    \"grain_column_names\": partition_column_names,\n",
    "    \"track_child_runs\": False,\n",
    "}\n",
    "\n",
    "mm_paramters = ManyModelsTrainParameters(\n",
    "    automl_settings=automl_settings, partition_column_names=partition_column_names\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up many models pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Parallel run step is leveraged to train multiple models at once. To configure the ParallelRunConfig you will need to determine the appropriate number of workers and nodes for your use case. The process_count_per_node is based off the number of cores of the compute VM. The node_count will determine the number of master nodes to use, increasing the node count will speed up the training process.\n",
    "\n",
    "| Property                           | Description|\n",
    "| :---------------                   | :------------------- |\n",
    "| **experiment**                     | The experiment used for training. |\n",
    "| **train_data**                     | The file dataset to be used as input to the training run. |\n",
    "| **node_count**                     | The number of compute nodes to be used for running the user script. We recommend to start with 3 and increase the node_count if the training time is taking too long. |\n",
    "| **process_count_per_node**         | Process count per node, we recommend 2:1 ratio for number of cores: number of processes per node. eg. If node has 16 cores then configure 8 or less process count per node or optimal performance. |\n",
    "| **train_pipeline_parameters**      | The set of configuration parameters defined in the previous section. |\n",
    "\n",
    "Calling this method will create a new aggregated dataset which is generated dynamically on pipeline execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.contrib.automl.pipeline.steps import AutoMLPipelineBuilder\n",
    "\n",
    "\n",
    "training_pipeline_steps = AutoMLPipelineBuilder.get_many_models_train_steps(\n",
    "    experiment=experiment,\n",
    "    train_data=train_data,\n",
    "    compute_target=compute_target,\n",
    "    node_count=2,\n",
    "    process_count_per_node=2,\n",
    "    run_invocation_timeout=920,\n",
    "    train_pipeline_parameters=mm_paramters,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.pipeline.core import Pipeline\n",
    "\n",
    "training_pipeline = Pipeline(ws, steps=training_pipeline_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Submit the pipeline to run\n",
    "Next we submit our pipeline to run. The whole training pipeline takes about 20 minutes using a STANDARD_DS12_V2 VM with our current ParallelRunConfig setting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_run = experiment.submit(training_pipeline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_run.wait_for_completion(show_output=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the run status, if training_run is in completed state, continue to next section. Otherwise, check the portal for failures."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.0 Backtesting\n",
    "Now that we selected the best AutoML model for each backtest fold, we will use these models to generate the forecasts and compare with the actuals."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up output dataset for inference data\n",
    "Output of inference can be represented as [OutputFileDatasetConfig](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.output_dataset_config.outputdatasetconfig?view=azure-ml-py) object and OutputFileDatasetConfig can be registered as a dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.data import OutputFileDatasetConfig\n",
    "\n",
    "output_inference_data_ds = OutputFileDatasetConfig(\n",
    "    name=\"many_models_inference_output\",\n",
    "    destination=(dstore, \"backtesting/inference_data/\"),\n",
    ").register_on_complete(name=\"backtesting_data_ds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For many models we need to provide the ManyModelsInferenceParameters object.\n",
    "\n",
    "#### ManyModelsInferenceParameters arguments\n",
    "| Property                           | Description|\n",
    "| :---------------                   | :------------------- |\n",
    "| **partition_column_names**         | List of column names that identifies groups.                                           |\n",
    "| **target_column_name**             | \\[Optional\\] Column name only if the inference dataset has the target. |\n",
    "| **time_column_name**               | Column name only if it is timeseries. |\n",
    "| **many_models_run_id**             | \\[Optional\\] Many models pipeline run id where models were trained. |\n",
    "\n",
    "#### get_many_models_batch_inference_steps arguments\n",
    "| Property                           | Description|\n",
    "| :---------------                   | :------------------- |\n",
    "| **experiment**                     | The experiment used for inference run. |\n",
    "| **inference_data**                 | The data to use for inferencing. It should be the same schema as used for training.\n",
    "| **compute_target**                 | The compute target that runs the inference pipeline.|\n",
    "| **node_count**                     | The number of compute nodes to be used for running the user script. We recommend to start with the number of cores per node (varies by compute sku). |\n",
    "| **process_count_per_node**         | The number of processes per node.\n",
    "| **train_run_id**                   | \\[Optional\\] The run id of the hierarchy training, by default it is the latest successful training many model run in the experiment. |\n",
    "| **train_experiment_name**          | \\[Optional\\] The train experiment that contains the train pipeline. This one is only needed when the train pipeline is not in the same experiement as the inference pipeline. |\n",
    "| **process_count_per_node**         | \\[Optional\\] The number of processes per node, by default it's 4. |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.contrib.automl.pipeline.steps import AutoMLPipelineBuilder\n",
    "from azureml.train.automl.runtime._many_models.many_models_parameters import (\n",
    "    ManyModelsInferenceParameters,\n",
    ")\n",
    "\n",
    "mm_parameters = ManyModelsInferenceParameters(\n",
    "    partition_column_names=partition_column_names,\n",
    "    time_column_name=TIME_COLNAME,\n",
    "    target_column_name=TARGET_COLNAME,\n",
    ")\n",
    "\n",
    "inference_steps = AutoMLPipelineBuilder.get_many_models_batch_inference_steps(\n",
    "    experiment=experiment,\n",
    "    inference_data=test_data,\n",
    "    node_count=2,\n",
    "    process_count_per_node=2,\n",
    "    compute_target=compute_target,\n",
    "    run_invocation_timeout=300,\n",
    "    output_datastore=output_inference_data_ds,\n",
    "    train_run_id=training_run.id,\n",
    "    train_experiment_name=training_run.experiment.name,\n",
    "    inference_pipeline_parameters=mm_parameters,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.pipeline.core import Pipeline\n",
    "\n",
    "inference_pipeline = Pipeline(ws, steps=inference_steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inference_run = experiment.submit(inference_pipeline)\n",
    "inference_run.wait_for_completion(show_output=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.0 Retrieve results and calculate metrics\n",
    "\n",
    "The pipeline returns one file with the predictions for each times series ID and outputs the result to the forecasting_output Blob container. The details of the blob container is listed in 'forecasting_output.txt' under Outputs+logs. \n",
    "\n",
    "The next code snippet does the following:\n",
    "1. Downloads the contents of the output folder that is passed in the parallel run step \n",
    "2. Reads the parallel_run_step.txt file that has the predictions as pandas dataframe \n",
    "3. Saves the table in csv format and \n",
    "4. Displays the top 10 rows of the predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.contrib.automl.pipeline.steps.utilities import get_output_from_mm_pipeline\n",
    "\n",
    "forecasting_results_name = \"forecasting_results\"\n",
    "forecasting_output_name = \"many_models_inference_output\"\n",
    "forecast_file = get_output_from_mm_pipeline(\n",
    "    inference_run, forecasting_results_name, forecasting_output_name\n",
    ")\n",
    "df = pd.read_csv(forecast_file, delimiter=\" \", header=None, parse_dates=[0])\n",
    "df.columns = list(X_train.columns) + [\"predicted_level\"]\n",
    "print(\n",
    "    \"Prediction has \", df.shape[0], \" rows. Here the first 10 rows are being displayed.\"\n",
    ")\n",
    "# Save the scv file with header to read it in the next step.\n",
    "df.rename(columns={TARGET_COLNAME: \"actual_level\"}, inplace=True)\n",
    "df.to_csv(os.path.join(forecasting_results_name, \"forecast.csv\"), index=False)\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View metrics\n",
    "We will read in the obtained results and run the helper script, which will generate metrics and create the plots of predicted versus actual values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from assets.score import calculate_scores_and_build_plots\n",
    "\n",
    "backtesting_results = \"backtesting_mm_results\"\n",
    "os.makedirs(backtesting_results, exist_ok=True)\n",
    "calculate_scores_and_build_plots(\n",
    "    forecasting_results_name, backtesting_results, automl_settings\n",
    ")\n",
    "pd.DataFrame({\"File\": os.listdir(backtesting_results)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The directory contains a set of files with results:\n",
    "- forecast.csv contains forecasts for all backtest iterations. The backtest_iteration column contains iteration identifier with the last training date as a suffix\n",
    "- scores.csv contains all metrics. If data set contains several time series, the metrics are given for all combinations of time series id and iterations, as well as scores for all iterations and time series ids, which are marked as \"all_sets\"\n",
    "- plots_fcst_vs_actual.pdf contains the predictions vs forecast plots for each iteration and, eash time series is saved as separate plot.\n",
    "\n",
    "For demonstration purposes we will display the table of metrics for one of the time series with ID \"ts0\". We will create the utility function, which will build the table with metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_metrics_for_ts(all_metrics, ts):\n",
    "    \"\"\"\n",
    "    Get the metrics for the time series with ID ts and return it as pandas data frame.\n",
    "\n",
    "    :param all_metrics: The table with all the metrics.\n",
    "    :param ts: The ID of a time series of interest.\n",
    "    :return: The pandas DataFrame with metrics for one time series.\n",
    "    \"\"\"\n",
    "    results_df = None\n",
    "    for ts_id, one_series in all_metrics.groupby(\"time_series_id\"):\n",
    "        if not ts_id.startswith(ts):\n",
    "            continue\n",
    "        iteration = ts_id.split(\"|\")[-1]\n",
    "        df = one_series[[\"metric_name\", \"metric\"]]\n",
    "        df.rename({\"metric\": iteration}, axis=1, inplace=True)\n",
    "        df.set_index(\"metric_name\", inplace=True)\n",
    "        if results_df is None:\n",
    "            results_df = df\n",
    "        else:\n",
    "            results_df = results_df.merge(\n",
    "                df, how=\"inner\", left_index=True, right_index=True\n",
    "            )\n",
    "    results_df.sort_index(axis=1, inplace=True)\n",
    "    return results_df\n",
    "\n",
    "\n",
    "metrics_df = pd.read_csv(os.path.join(backtesting_results, \"scores.csv\"))\n",
    "ts = \"ts_A\"\n",
    "get_metrics_for_ts(metrics_df, ts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Forecast vs actuals plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import IFrame\n",
    "\n",
    "IFrame(\"./backtesting_mm_results/plots_fcst_vs_actual.pdf\", width=800, height=300)"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "jialiu"
   }
  ],
  "categories": [
   "how-to-use-azureml",
   "automated-machine-learning"
  ],
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}