03-Working_with_Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with Data\n",
    "\n",
    "Data is the foundation on which machine learning models are built. Managing data centrally in the cloud, and making it accessible to teams of data scientists who are running experiments and training models on multiple workstations and compute targets is an important part of any professional data science solution.\n",
    "\n",
    "In this lab, you'll explore two Azure Machine Learning objects for working with data: *Datastores*, and *Datasets*.\n",
    "\n",
    "## Before You Start\n",
    "\n",
    "Before you start this lab, ensure that you have completed the *Create an Azure Machine Learning Workspace* and *Create a Compute Instance* tasks in [Lab 1: Getting Started with Azure Machine Learning](./labdocs/Lab01.md). Then open this notebook in Jupyter on your Compute Instance.\n",
    "\n",
    "## Connect to Your Workspace\n",
    "\n",
    "The first thing you need to do is to connect to your workspace using the Azure ML SDK.\n",
    "\n",
    "> **Note**: If you do not have a current authenticated session with your Azure subscription, you'll be prompted to authenticate. Follow the instructions to authenticate using the code provided."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import azureml.core\n",
    "from azureml.core import Workspace\n",
    "\n",
    "# Load the workspace from the saved config file\n",
    "ws = Workspace.from_config()\n",
    "print('Ready to use Azure ML {} to work with {}'.format(azureml.core.VERSION, ws.name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Work with a Datastore\n",
    "\n",
    "In Azure ML, *datastores* are references to storage locations, such as Azure Storage blob containers. Every workspace has a default datastore - usually the Azure storage blob container that was created with the workspace. If you need to work with data that is stored in different locations, you can add custom datastores to your workspace and set any of them to be the default.\n",
    "\n",
    "### View Datastores\n",
    "\n",
    "Run the following code to determine the datastores in your workspace:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the default datastore\n",
    "default_ds = ws.get_default_datastore()\n",
    "\n",
    "# Enumerate all datastores, indicating which is the default\n",
    "for ds_name in ws.datastores:\n",
    "    print(ds_name, \"- Default =\", ds_name == default_ds.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also view and manage datastores in your workspace on the Datastores page for your workspace in [Azure ML Studio](https://ml.azure.com).\n",
    "\n",
    "### Upload Data to a Datastore\n",
    "\n",
    "Now that you have determined the available datastores, you can upload files from your local file system to a datastore so that it will be accessible to experiments running in the workspace, regardless of where the experiment script is actually being run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "default_ds.upload_files(files=['./data/diabetes.csv', './data/diabetes2.csv'], # Upload the diabetes csv files in /data\n",
    "                       target_path='diabetes-data/', # Put it in a folder path in the datastore\n",
    "                       overwrite=True, # Replace existing files of the same name\n",
    "                       show_progress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train a Model from a Datastore\n",
    "\n",
    "When you uploaded the files in the code cell above, note that the code returned a *data reference*. A data reference provides a way to pass the path to a folder in a datastore to a script, regardless of where the script is being run, so that the script can access data in the datastore location.\n",
    "\n",
    "The following code gets a reference to the **diabetes-data** folder where you uploaded the diabetes CSV files, and specifically configures the data reference for *download* - in other words, it can be used to download the contents of the folder to the compute context where the data reference is being used. Downloading data works well for small volumes of data that will be processed on local compute. When working with remote compute, you can also configure a data reference to *mount* the datastore location and read data directly from the data source.\n",
    "\n",
    "> **More Information**: For more details about using datastores, see the [Azure ML documentation](https://docs.microsoft.com/azure/machine-learning/how-to-access-data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_ref = default_ds.path('diabetes-data').as_download(path_on_compute='diabetes_data')\n",
    "print(data_ref)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use the data reference in a training script, you must define a parameter for it. Run the following two code cells to create:\n",
    "\n",
    "1. A folder named **diabetes_training_from_datastore**\n",
    "2. A script that trains a classification model by using the training data in all of the CSV files in the folder referenced by the data reference parameter passed to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# Create a folder for the experiment files\n",
    "experiment_folder = 'diabetes_training_from_datastore'\n",
    "os.makedirs(experiment_folder, exist_ok=True)\n",
    "print(experiment_folder, 'folder created.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $experiment_folder/diabetes_training.py\n",
    "# Import libraries\n",
    "import os\n",
    "import argparse\n",
    "from azureml.core import Run\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import joblib\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import roc_curve\n",
    "\n",
    "# Get parameters\n",
    "parser = argparse.ArgumentParser()\n",
    "parser.add_argument('--regularization', type=float, dest='reg_rate', default=0.01, help='regularization rate')\n",
    "parser.add_argument('--data-folder', type=str, dest='data_folder', help='data folder reference')\n",
    "args = parser.parse_args()\n",
    "reg = args.reg_rate\n",
    "\n",
    "# Get the experiment run context\n",
    "run = Run.get_context()\n",
    "\n",
    "# load the diabetes data from the data reference\n",
    "data_folder = args.data_folder\n",
    "print(\"Loading data from\", data_folder)\n",
    "# Load all files and concatenate their contents as a single dataframe\n",
    "all_files = os.listdir(data_folder)\n",
    "diabetes = pd.concat((pd.read_csv(os.path.join(data_folder,csv_file)) for csv_file in all_files))\n",
    "\n",
    "# Separate features and labels\n",
    "X, y = diabetes[['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']].values, diabetes['Diabetic'].values\n",
    "\n",
    "# Split data into training set and test set\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)\n",
    "\n",
    "# Train a logistic regression model\n",
    "print('Training a logistic regression model with regularization rate of', reg)\n",
    "run.log('Regularization Rate',  np.float(reg))\n",
    "model = LogisticRegression(C=1/reg, solver=\"liblinear\").fit(X_train, y_train)\n",
    "\n",
    "# calculate accuracy\n",
    "y_hat = model.predict(X_test)\n",
    "acc = np.average(y_hat == y_test)\n",
    "print('Accuracy:', acc)\n",
    "run.log('Accuracy', np.float(acc))\n",
    "\n",
    "# calculate AUC\n",
    "y_scores = model.predict_proba(X_test)\n",
    "auc = roc_auc_score(y_test,y_scores[:,1])\n",
    "print('AUC: ' + str(auc))\n",
    "run.log('AUC', np.float(auc))\n",
    "\n",
    "os.makedirs('outputs', exist_ok=True)\n",
    "# note file saved in the outputs folder is automatically uploaded into experiment record\n",
    "joblib.dump(value=model, filename='outputs/diabetes_model.pkl')\n",
    "\n",
    "run.complete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The script will load the training data from the data reference passed to it as a parameter, so now you just need to set up the script parameters to pass the file reference when we run the experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.train.sklearn import SKLearn\n",
    "from azureml.core import Experiment\n",
    "from azureml.widgets import RunDetails\n",
    "\n",
    "# Set up the parameters\n",
    "script_params = {\n",
    "    '--regularization': 0.1, # regularization rate\n",
    "    '--data-folder': data_ref # data reference to download files from datastore\n",
    "}\n",
    "\n",
    "\n",
    "# Create an estimator\n",
    "estimator = SKLearn(source_directory=experiment_folder,\n",
    "                    entry_script='diabetes_training.py',\n",
    "                    script_params=script_params,\n",
    "                    compute_target = 'local'\n",
    "                   )\n",
    "\n",
    "# Create an experiment\n",
    "experiment_name = 'diabetes-training'\n",
    "experiment = Experiment(workspace = ws, name = experiment_name)\n",
    "\n",
    "# Run the experiment\n",
    "run = experiment.submit(config=estimator)\n",
    "# Show the run details while running\n",
    "RunDetails(run).show()\n",
    "run.wait_for_completion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first time the experiment is run, it may take some time to set up the Python environment - subsequent runs will be quicker.\n",
    "\n",
    "When the experiment has completed, in the widget, view the **azureml-logs/70_driver_log.txt** output log to verify that the data files were downloaded before the experiment script was run."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Work with Datasets\n",
    "\n",
    "While you can read data directly from datastores, Azure Machine Learning provides a further abstraction for data in the form of *datasets*. A dataset is a versioned reference to a specific set of data that you may want to use in an experiment. Datasets can be *tabular* or *file*-based.\n",
    "\n",
    "### Create a Tabular Dataset\n",
    "\n",
    "Let's create a dataset from the diabetes data you uploaded to the datastore, and view the first 20 records. In this case, the data is in a structured format in a CSV file, so we'll use a *tabular* dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Dataset\n",
    "\n",
    "# Get the default datastore\n",
    "default_ds = ws.get_default_datastore()\n",
    "\n",
    "#Create a tabular dataset from the path on the datastore (this may take a short while)\n",
    "tab_data_set = Dataset.Tabular.from_delimited_files(path=(default_ds, 'diabetes-data/*.csv'))\n",
    "\n",
    "# Display the first 20 rows as a Pandas dataframe\n",
    "tab_data_set.take(20).to_pandas_dataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see in the code above, it's easy to convert a tabular dataset to a Pandas dataframe, enabling you to work with the data using common python techniques.\n",
    "\n",
    "### Create a File Dataset\n",
    "\n",
    "The dataset you created is a *tabular* dataset that can be read as a dataframe containing all of the data in the structured files that are included in the dataset definition. This works well for tabular data, but in some machine learning scenarios you might need to work with data that is unstructured; or you may simply want to handle reading the data from files in your own code. To accomplish this, you can use a *file* dataset, which creates a list of file paths in a virtual mount point, which you can use to read the data in the files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create a file dataset from the path on the datastore (this may take a short while)\n",
    "file_data_set = Dataset.File.from_files(path=(default_ds, 'diabetes-data/*.csv'))\n",
    "\n",
    "# Get the files in the dataset\n",
    "for file_path in file_data_set.to_path():\n",
    "    print(file_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Register Datasets\n",
    "\n",
    "Now that you have created datasets that reference the diabetes data, you can register them to make them easily accessible to any experiment being run in the workspace.\n",
    "\n",
    "We'll register the tabular dataset as **diabetes dataset**, and the file dataset as **diabetes files**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Register the tabular dataset\n",
    "try:\n",
    "    tab_data_set = tab_data_set.register(workspace=ws, \n",
    "                                        name='diabetes dataset',\n",
    "                                        description='diabetes data',\n",
    "                                        tags = {'format':'CSV'},\n",
    "                                        create_new_version=True)\n",
    "except Exception as ex:\n",
    "    print(ex)\n",
    "\n",
    "# Register the file dataset\n",
    "try:\n",
    "    file_data_set = file_data_set.register(workspace=ws,\n",
    "                                            name='diabetes file dataset',\n",
    "                                            description='diabetes files',\n",
    "                                            tags = {'format':'CSV'},\n",
    "                                            create_new_version=True)\n",
    "except Exception as ex:\n",
    "    print(ex)\n",
    "\n",
    "print('Datasets registered')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can view and manage datasets on the **Datasets** page for your workspace in [Azure ML Studio](https://ml.azure.com). You cal also get a list of datasets from the workspace object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Datasets:\")\n",
    "for dataset_name in list(ws.datasets.keys()):\n",
    "    dataset = Dataset.get_by_name(ws, dataset_name)\n",
    "    print(\"\\t\", dataset.name, 'version', dataset.version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ability to version datasets enables you to redefine datasets without breaking existing experiments or pipelines that rely on previous definitions. By default, the latest version of a named dataset is returned, but you can retrieve a specific version of a dataset by specifying the version number, like this:\n",
    "\n",
    "```python\n",
    "dataset_v1 = Dataset.get_by_name(ws, 'diabetes dataset', version = 1)\n",
    "```\n",
    "\n",
    "\n",
    "### Train a Model from a Tabular Dataset\n",
    "\n",
    "Now that you have datasets, you're ready to start training models from them. You can pass datasets to scripts as *inputs* in the estimator being used to run the script.\n",
    "\n",
    "Run the following two code cells to create:\n",
    "\n",
    "1. A folder named **diabetes_training_from_tab_dataset**\n",
    "2. A script that trains a classification model by using a tabular dataset that is passed to is as an *input*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# Create a folder for the experiment files\n",
    "experiment_folder = 'diabetes_training_from_tab_dataset'\n",
    "os.makedirs(experiment_folder, exist_ok=True)\n",
    "print(experiment_folder, 'folder created')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $experiment_folder/diabetes_training.py\n",
    "# Import libraries\n",
    "import os\n",
    "import argparse\n",
    "from azureml.core import Run\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import joblib\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import roc_curve\n",
    "\n",
    "# Set regularization hyperparameter (passed as an argument to the script)\n",
    "parser = argparse.ArgumentParser()\n",
    "parser.add_argument('--regularization', type=float, dest='reg_rate', default=0.01, help='regularization rate')\n",
    "args = parser.parse_args()\n",
    "reg = args.reg_rate\n",
    "\n",
    "# Get the experiment run context\n",
    "run = Run.get_context()\n",
    "\n",
    "# load the diabetes data (passed as an input dataset)\n",
    "print(\"Loading Data...\")\n",
    "diabetes = run.input_datasets['diabetes'].to_pandas_dataframe()\n",
    "\n",
    "# Separate features and labels\n",
    "X, y = diabetes[['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']].values, diabetes['Diabetic'].values\n",
    "\n",
    "# Split data into training set and test set\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)\n",
    "\n",
    "# Train a logistic regression model\n",
    "print('Training a logistic regression model with regularization rate of', reg)\n",
    "run.log('Regularization Rate',  np.float(reg))\n",
    "model = LogisticRegression(C=1/reg, solver=\"liblinear\").fit(X_train, y_train)\n",
    "\n",
    "# calculate accuracy\n",
    "y_hat = model.predict(X_test)\n",
    "acc = np.average(y_hat == y_test)\n",
    "print('Accuracy:', acc)\n",
    "run.log('Accuracy', np.float(acc))\n",
    "\n",
    "# calculate AUC\n",
    "y_scores = model.predict_proba(X_test)\n",
    "auc = roc_auc_score(y_test,y_scores[:,1])\n",
    "print('AUC: ' + str(auc))\n",
    "run.log('AUC', np.float(auc))\n",
    "\n",
    "os.makedirs('outputs', exist_ok=True)\n",
    "# note file saved in the outputs folder is automatically uploaded into experiment record\n",
    "joblib.dump(value=model, filename='outputs/diabetes_model.pkl')\n",
    "\n",
    "run.complete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can create an estimator to run the script, and define a named *input* for the training dataset, which is read by the script.\n",
    "\n",
    "> **Note**: The **Dataset** class is defined in the **azureml-dataprep** package (which is installed with the SDK), and this package includes optional support for **pandas** (which is used by the **to_pandas_dataframe()** method, so you need to include this package in the environment where the training experiment will be run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.train.sklearn import SKLearn\n",
    "from azureml.core import Experiment\n",
    "from azureml.widgets import RunDetails\n",
    "\n",
    "# Set the script parameters\n",
    "script_params = {\n",
    "    '--regularization': 0.1\n",
    "}\n",
    "\n",
    "# Get the training dataset\n",
    "diabetes_ds = ws.datasets.get(\"diabetes dataset\")\n",
    "\n",
    "# Create an estimator\n",
    "estimator = SKLearn(source_directory=experiment_folder,\n",
    "                    entry_script='diabetes_training.py',\n",
    "                    script_params=script_params,\n",
    "                    compute_target = 'local',\n",
    "                    inputs=[diabetes_ds.as_named_input('diabetes')], # Pass the Dataset object as an input...\n",
    "                    pip_packages=['azureml-dataprep[pandas]'] # ...so you need the dataprep package\n",
    "                   )\n",
    "\n",
    "# Create an experiment\n",
    "experiment_name = 'diabetes-training'\n",
    "experiment = Experiment(workspace = ws, name = experiment_name)\n",
    "\n",
    "# Run the experiment\n",
    "run = experiment.submit(config=estimator)\n",
    "# Show the run details while running\n",
    "RunDetails(run).show()\n",
    "run.wait_for_completion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first time the experiment is run, it may take some time to set up the Python environment - subsequent runs will be quicker.\n",
    "\n",
    "When the experiment has completed, in the widget, view the **azureml-logs/70_driver_log.txt** output log and the metrics generated by the run."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train a Model from a File Dataset\n",
    "\n",
    "You've seen how to train a model using training data in a *tabular* dataset; but what about a *file* dataset?\n",
    "\n",
    "When you're using a file dataset, the dataset input passed to the script represents a mount point containing file paths. How you read the data from these files depends on the kind of data in the files and what you want to do with it. In the case of the diabetes CSV files, you can use the Python **glob** module to create a list of files in the virtual mount point defined by the dataset, and read them all into Pandas dataframes that are concatenated into a single dataframe.\n",
    "\n",
    "Run the following two code cells to create:\n",
    "\n",
    "1. A folder named **diabetes_training_from_file_dataset**\n",
    "2. A script that trains a classification model by using a file dataset that is passed to is as an *input*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# Create a folder for the experiment files\n",
    "experiment_folder = 'diabetes_training_from_file_dataset'\n",
    "os.makedirs(experiment_folder, exist_ok=True)\n",
    "print(experiment_folder, 'folder created')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile $experiment_folder/diabetes_training.py\n",
    "# Import libraries\n",
    "import os\n",
    "import argparse\n",
    "from azureml.core import Workspace, Dataset, Experiment, Run\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import joblib\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import roc_curve\n",
    "import glob\n",
    "\n",
    "# Set regularization hyperparameter (passed as an argument to the script)\n",
    "parser = argparse.ArgumentParser()\n",
    "parser.add_argument('--regularization', type=float, dest='reg_rate', default=0.01, help='regularization rate')\n",
    "args = parser.parse_args()\n",
    "reg = args.reg_rate\n",
    "\n",
    "# Get the experiment run context\n",
    "run = Run.get_context()\n",
    "\n",
    "# load the diabetes dataset\n",
    "print(\"Loading Data...\")\n",
    "data_path = run.input_datasets['diabetes'] # Get the training data from the estimator input\n",
    "all_files = glob.glob(data_path + \"/*.csv\")\n",
    "diabetes = pd.concat((pd.read_csv(f) for f in all_files))\n",
    "\n",
    "# Separate features and labels\n",
    "X, y = diabetes[['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']].values, diabetes['Diabetic'].values\n",
    "\n",
    "# Split data into training set and test set\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)\n",
    "\n",
    "# Train a logistic regression model\n",
    "print('Training a logistic regression model with regularization rate of', reg)\n",
    "run.log('Regularization Rate',  np.float(reg))\n",
    "model = LogisticRegression(C=1/reg, solver=\"liblinear\").fit(X_train, y_train)\n",
    "\n",
    "# calculate accuracy\n",
    "y_hat = model.predict(X_test)\n",
    "acc = np.average(y_hat == y_test)\n",
    "print('Accuracy:', acc)\n",
    "run.log('Accuracy', np.float(acc))\n",
    "\n",
    "# calculate AUC\n",
    "y_scores = model.predict_proba(X_test)\n",
    "auc = roc_auc_score(y_test,y_scores[:,1])\n",
    "print('AUC: ' + str(auc))\n",
    "run.log('AUC', np.float(auc))\n",
    "\n",
    "os.makedirs('outputs', exist_ok=True)\n",
    "# note file saved in the outputs folder is automatically uploaded into experiment record\n",
    "joblib.dump(value=model, filename='outputs/diabetes_model.pkl')\n",
    "\n",
    "run.complete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we need to change the way we pass the dataset to the estimator - it needs to define a mount point from which the script can read the files. For large volumes of data, you'd generally use the **as_mount** method to stream the files directly from the dataset source; but when running on local compute (as we are in this example), you need to use the **as_download** option to download the dataset files to a local folder.\n",
    "\n",
    "Also, since the **Dataset** class is defined in the **azureml-dataprep** package, we need to include that in the experiment environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.train.sklearn import SKLearn\n",
    "from azureml.core import Experiment\n",
    "from azureml.widgets import RunDetails\n",
    "\n",
    "# Set the script parameters\n",
    "script_params = {\n",
    "    '--regularization': 0.1\n",
    "}\n",
    "\n",
    "# Get the training dataset\n",
    "diabetes_ds = ws.datasets.get(\"diabetes file dataset\")\n",
    "\n",
    "# Create an estimator\n",
    "estimator = SKLearn(source_directory=experiment_folder,\n",
    "                    entry_script='diabetes_training.py',\n",
    "                    script_params=script_params,\n",
    "                    compute_target = 'local',\n",
    "                    inputs=[diabetes_ds.as_named_input('diabetes').as_download(path_on_compute='diabetes_data')], # Pass the Dataset object as an input\n",
    "                    pip_packages=['azureml-dataprep[pandas]'] # so we need the dataprep package\n",
    "                   )\n",
    "\n",
    "# Create an experiment\n",
    "experiment_name = 'diabetes-training'\n",
    "experiment = Experiment(workspace = ws, name = experiment_name)\n",
    "\n",
    "# Run the experiment\n",
    "run = experiment.submit(config=estimator)\n",
    "# Show the run details while running\n",
    "RunDetails(run).show()\n",
    "run.wait_for_completion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the experiment has completed, in the widget, view the **azureml-logs/70_driver_log.txt** output log to verify that the file dataset was processed and the data files downloaded."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}