Train_Reviews_AutoPilot.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a Model with SageMaker AutoPilot\n",
    "We will use AutoPilot to predict sentiment of customer reviews.\n",
    "\n",
    "### Introduction\n",
    "Amazon SageMaker Autopilot is a service to perform automated machine learning (AutoML) on your datasets.  AutoPilot is available through the UI or AWS SDK.  In this notebook, we will use the AWS SDK to create and deploy a text processing and sentiment classification machine learning pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup\n",
    "\n",
    "Let's start by specifying:\n",
    "\n",
    "* The S3 bucket and prefix to use to train our model.  _Note:  This should be in the same region as this notebook._\n",
    "* The IAM role of this notebook needs access to your data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "import pandas as pd\n",
    "\n",
    "sess   = sagemaker.Session()\n",
    "bucket = sess.default_bucket()\n",
    "role = sagemaker.get_execution_role()\n",
    "region = boto3.Session().region_name\n",
    "\n",
    "sm = boto3.Session().client(service_name='sagemaker', region_name=region)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-east-2-533787958253/data/amazon_reviews_us_Digital_Software_v1_00_header.csv\n"
     ]
    }
   ],
   "source": [
    "%store -r header_train_s3_uri\n",
    "\n",
    "print(header_train_s3_uri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-04-29 14:18:44   15164605 amazon_reviews_us_Digital_Software_v1_00_header.csv\r\n"
     ]
    }
   ],
   "source": [
    "!aws s3 ls $header_train_s3_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup the S3 Location for the AutoPilot-Generated Assets \n",
    "This include Jupyter Notebooks (Analysis), Python Scripts (Feature Engineering), and Trained Models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-east-2-533787958253/models/autopilot\n"
     ]
    }
   ],
   "source": [
    "prefix_model_output = 'models/autopilot'\n",
    "\n",
    "model_output_s3_uri = 's3://{}/{}'.format(bucket, prefix_model_output)\n",
    "\n",
    "print(model_output_s3_uri)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_candidates = 3\n",
    "\n",
    "job_config = {\n",
    "    'CompletionCriteria': {\n",
    "      'MaxRuntimePerTrainingJobInSeconds': 600,\n",
    "      'MaxCandidates': max_candidates,\n",
    "      'MaxAutoMLJobRuntimeInSeconds': 3600\n",
    "    },\n",
    "}\n",
    "\n",
    "input_data_config = [{\n",
    "      'DataSource': {\n",
    "        'S3DataSource': {\n",
    "          'S3DataType': 'S3Prefix',\n",
    "          'S3Uri': '{}'.format(header_train_s3_uri)\n",
    "        }\n",
    "      },\n",
    "      'TargetAttributeName': 'star_rating'\n",
    "    }\n",
    "]\n",
    "\n",
    "output_data_config = {\n",
    "    'S3OutputPath': '{}'.format(model_output_s3_uri)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Launch the SageMaker AutoPilot job\n",
    "\n",
    "We can now launch the job by calling the `create_auto_ml_job` API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AutoMLJobName: automl-dm-27-20-20-56\n"
     ]
    }
   ],
   "source": [
    "from time import gmtime, strftime, sleep\n",
    "timestamp_suffix = strftime('%d-%H-%M-%S', gmtime())\n",
    "\n",
    "auto_ml_job_name = 'automl-dm-' + timestamp_suffix\n",
    "print('AutoMLJobName: ' + auto_ml_job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Note that we are not specifying the `ProblemType`.  AutoPilot will automatically detect if we're using regression or classification (binary or multi-class)._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'AutoMLJobArn': 'arn:aws:sagemaker:us-east-2:533787958253:automl-job/automl-dm-27-20-20-56',\n",
       " 'ResponseMetadata': {'RequestId': '9e4b42e5-13eb-463e-a80f-6e081152e2bf',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '9e4b42e5-13eb-463e-a80f-6e081152e2bf',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '92',\n",
       "   'date': 'Mon, 27 Apr 2020 20:21:24 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sm.create_auto_ml_job(AutoMLJobName=auto_ml_job_name,\n",
    "                      InputDataConfig=input_data_config,\n",
    "                      OutputDataConfig=output_data_config,\n",
    "                      AutoMLJobConfig=job_config,\n",
    "#                      ProblemType=\"Classification\",\n",
    "                      RoleArn=role)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tracking the progress of the AutoPilot job\n",
    "SageMaker AutoPilot job consists of the following high-level steps: \n",
    "* _Data Analysis_ where the data is summarized and analyzed to determine which feature engineering techniques, hyper-parameters, and models to explore.\n",
    "* _Feature Engineering_ where the data is scrubbed, balanced, combined, and split into train and validation\n",
    "* _Model Training and Tuning_ where the top performing features, hyper-parameters, and models are selected and trained."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzing Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress FeatureEngineering\n",
      "Data analysis complete\n",
      "{'AutoMLJobName': 'automl-dm-27-20-20-56', 'AutoMLJobArn': 'arn:aws:sagemaker:us-east-2:533787958253:automl-job/automl-dm-27-20-20-56', 'InputDataConfig': [{'DataSource': {'S3DataSource': {'S3DataType': 'S3Prefix', 'S3Uri': 's3://sagemaker-us-east-2-533787958253/data/amazon_reviews_us_Digital_Software_v1_00_header.csv'}}, 'TargetAttributeName': 'star_rating'}], 'OutputDataConfig': {'S3OutputPath': 's3://sagemaker-us-east-2-533787958253/models/autopilot'}, 'RoleArn': 'arn:aws:iam::533787958253:role/CoE_AI_SageMaker_Notebook', 'AutoMLJobConfig': {'CompletionCriteria': {'MaxCandidates': 3, 'MaxRuntimePerTrainingJobInSeconds': 600, 'MaxAutoMLJobRuntimeInSeconds': 3600}}, 'CreationTime': datetime.datetime(2020, 4, 27, 20, 21, 24, 711000, tzinfo=tzlocal()), 'LastModifiedTime': datetime.datetime(2020, 4, 27, 20, 28, 44, 865000, tzinfo=tzlocal()), 'AutoMLJobStatus': 'InProgress', 'AutoMLJobSecondaryStatus': 'FeatureEngineering', 'GenerateCandidateDefinitionsOnly': False, 'AutoMLJobArtifacts': {'CandidateDefinitionNotebookLocation': 's3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/sagemaker-automl-candidates/pr-1-abdf3cc0395847a59aae3eb1548fe43ad9ba1a26a4d444ef80c480bf1a/notebooks/SageMakerAutopilotCandidateDefinitionNotebook.ipynb', 'DataExplorationNotebookLocation': 's3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/sagemaker-automl-candidates/pr-1-abdf3cc0395847a59aae3eb1548fe43ad9ba1a26a4d444ef80c480bf1a/notebooks/SageMakerAutopilotDataExplorationNotebook.ipynb'}, 'ResolvedAttributes': {'AutoMLJobObjective': {'MetricName': 'Accuracy'}, 'ProblemType': 'MulticlassClassification', 'CompletionCriteria': {'MaxCandidates': 3, 'MaxRuntimePerTrainingJobInSeconds': 600, 'MaxAutoMLJobRuntimeInSeconds': 3600}}, 'ResponseMetadata': {'RequestId': '7918980b-1831-4c46-b946-e746b303f4b8', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '7918980b-1831-4c46-b946-e746b303f4b8', 'content-type': 'application/x-amz-json-1.1', 'content-length': '1623', 'date': 'Mon, 27 Apr 2020 20:28:45 GMT'}, 'RetryAttempts': 0}}\n"
     ]
    }
   ],
   "source": [
    "# Sleep for a bit to ensure the AutoML job above has time to start\n",
    "import time\n",
    "time.sleep(3)\n",
    "\n",
    "job = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "job_status = job['AutoMLJobStatus']\n",
    "job_sec_status = job['AutoMLJobSecondaryStatus']\n",
    "\n",
    "if job_status not in ('Stopped', 'Failed'):\n",
    "    while job_status in ('InProgress') and job_sec_status in ('AnalyzingData'):\n",
    "        job = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "        job_status = job['AutoMLJobStatus']\n",
    "        job_sec_status = job['AutoMLJobSecondaryStatus']\n",
    "        print(job_status, job_sec_status)\n",
    "        sleep(30)\n",
    "    print(\"Data analysis complete\")\n",
    "    \n",
    "print(job)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'AutoMLJobName': 'automl-dm-27-20-20-56', 'AutoMLJobArn': 'arn:aws:sagemaker:us-east-2:533787958253:automl-job/automl-dm-27-20-20-56', 'InputDataConfig': [{'DataSource': {'S3DataSource': {'S3DataType': 'S3Prefix', 'S3Uri': 's3://sagemaker-us-east-2-533787958253/data/amazon_reviews_us_Digital_Software_v1_00_header.csv'}}, 'TargetAttributeName': 'star_rating'}], 'OutputDataConfig': {'S3OutputPath': 's3://sagemaker-us-east-2-533787958253/models/autopilot'}, 'RoleArn': 'arn:aws:iam::533787958253:role/CoE_AI_SageMaker_Notebook', 'AutoMLJobConfig': {'CompletionCriteria': {'MaxCandidates': 3, 'MaxRuntimePerTrainingJobInSeconds': 600, 'MaxAutoMLJobRuntimeInSeconds': 3600}}, 'CreationTime': datetime.datetime(2020, 4, 27, 20, 21, 24, 711000, tzinfo=tzlocal()), 'LastModifiedTime': datetime.datetime(2020, 4, 27, 20, 39, 4, 752000, tzinfo=tzlocal()), 'AutoMLJobStatus': 'InProgress', 'AutoMLJobSecondaryStatus': 'ModelTuning', 'GenerateCandidateDefinitionsOnly': False, 'AutoMLJobArtifacts': {'CandidateDefinitionNotebookLocation': 's3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/sagemaker-automl-candidates/pr-1-abdf3cc0395847a59aae3eb1548fe43ad9ba1a26a4d444ef80c480bf1a/notebooks/SageMakerAutopilotCandidateDefinitionNotebook.ipynb', 'DataExplorationNotebookLocation': 's3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/sagemaker-automl-candidates/pr-1-abdf3cc0395847a59aae3eb1548fe43ad9ba1a26a4d444ef80c480bf1a/notebooks/SageMakerAutopilotDataExplorationNotebook.ipynb'}, 'ResolvedAttributes': {'AutoMLJobObjective': {'MetricName': 'Accuracy'}, 'ProblemType': 'MulticlassClassification', 'CompletionCriteria': {'MaxCandidates': 3, 'MaxRuntimePerTrainingJobInSeconds': 600, 'MaxAutoMLJobRuntimeInSeconds': 3600}}, 'ResponseMetadata': {'RequestId': '38a988fa-c1fb-4018-8718-4c65c5340617', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '38a988fa-c1fb-4018-8718-4c65c5340617', 'content-type': 'application/x-amz-json-1.1', 'content-length': '1616', 'date': 'Mon, 27 Apr 2020 20:39:13 GMT'}, 'RetryAttempts': 0}}\n"
     ]
    }
   ],
   "source": [
    "job = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "print(job)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed\n",
      "MaxCandidatesReached\n",
      "Feature engineering complete\n",
      "{'AutoMLJobName': 'automl-dm-27-20-20-56', 'AutoMLJobArn': 'arn:aws:sagemaker:us-east-2:533787958253:automl-job/automl-dm-27-20-20-56', 'InputDataConfig': [{'DataSource': {'S3DataSource': {'S3DataType': 'S3Prefix', 'S3Uri': 's3://sagemaker-us-east-2-533787958253/data/amazon_reviews_us_Digital_Software_v1_00_header.csv'}}, 'TargetAttributeName': 'star_rating'}], 'OutputDataConfig': {'S3OutputPath': 's3://sagemaker-us-east-2-533787958253/models/autopilot'}, 'RoleArn': 'arn:aws:iam::533787958253:role/CoE_AI_SageMaker_Notebook', 'AutoMLJobConfig': {'CompletionCriteria': {'MaxCandidates': 3, 'MaxRuntimePerTrainingJobInSeconds': 600, 'MaxAutoMLJobRuntimeInSeconds': 3600}}, 'CreationTime': datetime.datetime(2020, 4, 27, 20, 21, 24, 711000, tzinfo=tzlocal()), 'EndTime': datetime.datetime(2020, 4, 27, 20, 39, 40, 922000, tzinfo=tzlocal()), 'LastModifiedTime': datetime.datetime(2020, 4, 27, 20, 39, 40, 947000, tzinfo=tzlocal()), 'BestCandidate': {'CandidateName': 'tuning-job-1-5ebf290f7b714803b3-002-a0c1c68c', 'FinalAutoMLJobObjectiveMetric': {'MetricName': 'validation:accuracy', 'Value': 0.3787199854850769}, 'ObjectiveStatus': 'Succeeded', 'CandidateSteps': [{'CandidateStepType': 'AWS::SageMaker::ProcessingJob', 'CandidateStepArn': 'arn:aws:sagemaker:us-east-2:533787958253:processing-job/db-1-534bccbc2999467f9253dc442fe05c712814095946424fb8b9c0987dfc', 'CandidateStepName': 'db-1-534bccbc2999467f9253dc442fe05c712814095946424fb8b9c0987dfc'}, {'CandidateStepType': 'AWS::SageMaker::TrainingJob', 'CandidateStepArn': 'arn:aws:sagemaker:us-east-2:533787958253:training-job/automl-dm--dpp2-1-f6fa3c0be79a4063b97b6571ff3321c366be99ac91d64', 'CandidateStepName': 'automl-dm--dpp2-1-f6fa3c0be79a4063b97b6571ff3321c366be99ac91d64'}, {'CandidateStepType': 'AWS::SageMaker::TransformJob', 'CandidateStepArn': 'arn:aws:sagemaker:us-east-2:533787958253:transform-job/automl-dm--dpp2-rpb-1-6ff77104e6944bb28dc4fcb89de37d80ec85d33ef', 'CandidateStepName': 'automl-dm--dpp2-rpb-1-6ff77104e6944bb28dc4fcb89de37d80ec85d33ef'}, {'CandidateStepType': 'AWS::SageMaker::TrainingJob', 'CandidateStepArn': 'arn:aws:sagemaker:us-east-2:533787958253:training-job/tuning-job-1-5ebf290f7b714803b3-002-a0c1c68c', 'CandidateStepName': 'tuning-job-1-5ebf290f7b714803b3-002-a0c1c68c'}], 'CandidateStatus': 'Completed', 'InferenceContainers': [{'Image': '257758044811.dkr.ecr.us-east-2.amazonaws.com/sagemaker-sklearn-automl:0.1.0-cpu-py3', 'ModelDataUrl': 's3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/data-processor-models/automl-dm--dpp2-1-f6fa3c0be79a4063b97b6571ff3321c366be99ac91d64/output/model.tar.gz', 'Environment': {'AUTOML_SPARSE_ENCODE_RECORDIO_PROTOBUF': '1', 'AUTOML_TRANSFORM_MODE': 'feature-transform', 'SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT': 'application/x-recordio-protobuf', 'SAGEMAKER_PROGRAM': 'sagemaker_serve', 'SAGEMAKER_SUBMIT_DIRECTORY': '/opt/ml/model/sagemaker_serve.py'}}, {'Image': '257758044811.dkr.ecr.us-east-2.amazonaws.com/sagemaker-xgboost:0.90-1-cpu-py3', 'ModelDataUrl': 's3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/tuning/automl-dm--dpp2-xgb/tuning-job-1-5ebf290f7b714803b3-002-a0c1c68c/output/model.tar.gz', 'Environment': {'MAX_CONTENT_LENGTH': '20971520', 'SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT': 'text/csv'}}, {'Image': '257758044811.dkr.ecr.us-east-2.amazonaws.com/sagemaker-sklearn-automl:0.1.0-cpu-py3', 'ModelDataUrl': 's3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/data-processor-models/automl-dm--dpp2-1-f6fa3c0be79a4063b97b6571ff3321c366be99ac91d64/output/model.tar.gz', 'Environment': {'AUTOML_TRANSFORM_MODE': 'inverse-label-transform', 'SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT': 'text/csv', 'SAGEMAKER_PROGRAM': 'sagemaker_serve', 'SAGEMAKER_SUBMIT_DIRECTORY': '/opt/ml/model/sagemaker_serve.py'}}], 'CreationTime': datetime.datetime(2020, 4, 27, 20, 38, 21, tzinfo=tzlocal()), 'EndTime': datetime.datetime(2020, 4, 27, 20, 39, 5, tzinfo=tzlocal()), 'LastModifiedTime': datetime.datetime(2020, 4, 27, 20, 39, 40, 167000, tzinfo=tzlocal())}, 'AutoMLJobStatus': 'Completed', 'AutoMLJobSecondaryStatus': 'MaxCandidatesReached', 'GenerateCandidateDefinitionsOnly': False, 'AutoMLJobArtifacts': {'CandidateDefinitionNotebookLocation': 's3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/sagemaker-automl-candidates/pr-1-abdf3cc0395847a59aae3eb1548fe43ad9ba1a26a4d444ef80c480bf1a/notebooks/SageMakerAutopilotCandidateDefinitionNotebook.ipynb', 'DataExplorationNotebookLocation': 's3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/sagemaker-automl-candidates/pr-1-abdf3cc0395847a59aae3eb1548fe43ad9ba1a26a4d444ef80c480bf1a/notebooks/SageMakerAutopilotDataExplorationNotebook.ipynb'}, 'ResolvedAttributes': {'AutoMLJobObjective': {'MetricName': 'Accuracy'}, 'ProblemType': 'MulticlassClassification', 'CompletionCriteria': {'MaxCandidates': 3, 'MaxRuntimePerTrainingJobInSeconds': 600, 'MaxAutoMLJobRuntimeInSeconds': 3600}}, 'ResponseMetadata': {'RequestId': '9719e6a5-dcd4-4057-9964-72de180fee59', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '9719e6a5-dcd4-4057-9964-72de180fee59', 'content-type': 'application/x-amz-json-1.1', 'content-length': '4560', 'date': 'Mon, 27 Apr 2020 20:39:45 GMT'}, 'RetryAttempts': 0}}\n"
     ]
    }
   ],
   "source": [
    "job = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "job_status = job['AutoMLJobStatus']\n",
    "job_sec_status = job['AutoMLJobSecondaryStatus']\n",
    "print(job_status)\n",
    "print(job_sec_status)\n",
    "if job_status not in ('Stopped', 'Failed'):\n",
    "    while job_status in ('InProgress') and job_sec_status in ('FeatureEngineering'):\n",
    "        job = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "        job_status = job['AutoMLJobStatus']\n",
    "        job_sec_status = job['AutoMLJobSecondaryStatus']\n",
    "        print(job_status, job_sec_status)\n",
    "        sleep(30)\n",
    "    print(\"Feature engineering complete\")\n",
    "    \n",
    "print(job)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Training and Tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed\n",
      "MaxCandidatesReached\n",
      "Model tuning complete\n",
      "{'AutoMLJobName': 'automl-dm-27-20-20-56', 'AutoMLJobArn': 'arn:aws:sagemaker:us-east-2:533787958253:automl-job/automl-dm-27-20-20-56', 'InputDataConfig': [{'DataSource': {'S3DataSource': {'S3DataType': 'S3Prefix', 'S3Uri': 's3://sagemaker-us-east-2-533787958253/data/amazon_reviews_us_Digital_Software_v1_00_header.csv'}}, 'TargetAttributeName': 'star_rating'}], 'OutputDataConfig': {'S3OutputPath': 's3://sagemaker-us-east-2-533787958253/models/autopilot'}, 'RoleArn': 'arn:aws:iam::533787958253:role/CoE_AI_SageMaker_Notebook', 'AutoMLJobConfig': {'CompletionCriteria': {'MaxCandidates': 3, 'MaxRuntimePerTrainingJobInSeconds': 600, 'MaxAutoMLJobRuntimeInSeconds': 3600}}, 'CreationTime': datetime.datetime(2020, 4, 27, 20, 21, 24, 711000, tzinfo=tzlocal()), 'EndTime': datetime.datetime(2020, 4, 27, 20, 39, 40, 922000, tzinfo=tzlocal()), 'LastModifiedTime': datetime.datetime(2020, 4, 27, 20, 39, 40, 947000, tzinfo=tzlocal()), 'BestCandidate': {'CandidateName': 'tuning-job-1-5ebf290f7b714803b3-002-a0c1c68c', 'FinalAutoMLJobObjectiveMetric': {'MetricName': 'validation:accuracy', 'Value': 0.3787199854850769}, 'ObjectiveStatus': 'Succeeded', 'CandidateSteps': [{'CandidateStepType': 'AWS::SageMaker::ProcessingJob', 'CandidateStepArn': 'arn:aws:sagemaker:us-east-2:533787958253:processing-job/db-1-534bccbc2999467f9253dc442fe05c712814095946424fb8b9c0987dfc', 'CandidateStepName': 'db-1-534bccbc2999467f9253dc442fe05c712814095946424fb8b9c0987dfc'}, {'CandidateStepType': 'AWS::SageMaker::TrainingJob', 'CandidateStepArn': 'arn:aws:sagemaker:us-east-2:533787958253:training-job/automl-dm--dpp2-1-f6fa3c0be79a4063b97b6571ff3321c366be99ac91d64', 'CandidateStepName': 'automl-dm--dpp2-1-f6fa3c0be79a4063b97b6571ff3321c366be99ac91d64'}, {'CandidateStepType': 'AWS::SageMaker::TransformJob', 'CandidateStepArn': 'arn:aws:sagemaker:us-east-2:533787958253:transform-job/automl-dm--dpp2-rpb-1-6ff77104e6944bb28dc4fcb89de37d80ec85d33ef', 'CandidateStepName': 'automl-dm--dpp2-rpb-1-6ff77104e6944bb28dc4fcb89de37d80ec85d33ef'}, {'CandidateStepType': 'AWS::SageMaker::TrainingJob', 'CandidateStepArn': 'arn:aws:sagemaker:us-east-2:533787958253:training-job/tuning-job-1-5ebf290f7b714803b3-002-a0c1c68c', 'CandidateStepName': 'tuning-job-1-5ebf290f7b714803b3-002-a0c1c68c'}], 'CandidateStatus': 'Completed', 'InferenceContainers': [{'Image': '257758044811.dkr.ecr.us-east-2.amazonaws.com/sagemaker-sklearn-automl:0.1.0-cpu-py3', 'ModelDataUrl': 's3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/data-processor-models/automl-dm--dpp2-1-f6fa3c0be79a4063b97b6571ff3321c366be99ac91d64/output/model.tar.gz', 'Environment': {'AUTOML_SPARSE_ENCODE_RECORDIO_PROTOBUF': '1', 'AUTOML_TRANSFORM_MODE': 'feature-transform', 'SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT': 'application/x-recordio-protobuf', 'SAGEMAKER_PROGRAM': 'sagemaker_serve', 'SAGEMAKER_SUBMIT_DIRECTORY': '/opt/ml/model/sagemaker_serve.py'}}, {'Image': '257758044811.dkr.ecr.us-east-2.amazonaws.com/sagemaker-xgboost:0.90-1-cpu-py3', 'ModelDataUrl': 's3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/tuning/automl-dm--dpp2-xgb/tuning-job-1-5ebf290f7b714803b3-002-a0c1c68c/output/model.tar.gz', 'Environment': {'MAX_CONTENT_LENGTH': '20971520', 'SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT': 'text/csv'}}, {'Image': '257758044811.dkr.ecr.us-east-2.amazonaws.com/sagemaker-sklearn-automl:0.1.0-cpu-py3', 'ModelDataUrl': 's3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/data-processor-models/automl-dm--dpp2-1-f6fa3c0be79a4063b97b6571ff3321c366be99ac91d64/output/model.tar.gz', 'Environment': {'AUTOML_TRANSFORM_MODE': 'inverse-label-transform', 'SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT': 'text/csv', 'SAGEMAKER_PROGRAM': 'sagemaker_serve', 'SAGEMAKER_SUBMIT_DIRECTORY': '/opt/ml/model/sagemaker_serve.py'}}], 'CreationTime': datetime.datetime(2020, 4, 27, 20, 38, 21, tzinfo=tzlocal()), 'EndTime': datetime.datetime(2020, 4, 27, 20, 39, 5, tzinfo=tzlocal()), 'LastModifiedTime': datetime.datetime(2020, 4, 27, 20, 39, 40, 167000, tzinfo=tzlocal())}, 'AutoMLJobStatus': 'Completed', 'AutoMLJobSecondaryStatus': 'MaxCandidatesReached', 'GenerateCandidateDefinitionsOnly': False, 'AutoMLJobArtifacts': {'CandidateDefinitionNotebookLocation': 's3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/sagemaker-automl-candidates/pr-1-abdf3cc0395847a59aae3eb1548fe43ad9ba1a26a4d444ef80c480bf1a/notebooks/SageMakerAutopilotCandidateDefinitionNotebook.ipynb', 'DataExplorationNotebookLocation': 's3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/sagemaker-automl-candidates/pr-1-abdf3cc0395847a59aae3eb1548fe43ad9ba1a26a4d444ef80c480bf1a/notebooks/SageMakerAutopilotDataExplorationNotebook.ipynb'}, 'ResolvedAttributes': {'AutoMLJobObjective': {'MetricName': 'Accuracy'}, 'ProblemType': 'MulticlassClassification', 'CompletionCriteria': {'MaxCandidates': 3, 'MaxRuntimePerTrainingJobInSeconds': 600, 'MaxAutoMLJobRuntimeInSeconds': 3600}}, 'ResponseMetadata': {'RequestId': '150cebab-3fc4-44f9-83ce-ac46b7d67485', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '150cebab-3fc4-44f9-83ce-ac46b7d67485', 'content-type': 'application/x-amz-json-1.1', 'content-length': '4560', 'date': 'Mon, 27 Apr 2020 20:40:17 GMT'}, 'RetryAttempts': 0}}\n"
     ]
    }
   ],
   "source": [
    "job = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "job_status = job['AutoMLJobStatus']\n",
    "job_sec_status = job['AutoMLJobSecondaryStatus']\n",
    "print(job_status)\n",
    "print(job_sec_status)\n",
    "if job_status not in ('Stopped', 'Failed'):\n",
    "    while job_status in ('InProgress') and job_sec_status in ('ModelTuning'):\n",
    "        job = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "        job_status = job['AutoMLJobStatus']\n",
    "        job_sec_status = job['AutoMLJobSecondaryStatus']\n",
    "        print(job_status, job_sec_status)\n",
    "        sleep(30)\n",
    "    print(\"Model tuning complete\")\n",
    "    \n",
    "print(job)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wait Until All Jobs are Done Above Before Proceeding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# View Generated Notebooks\n",
    "Once data analysis is complete, SageMaker AutoPilot generates two notebooks: \n",
    "* Data exploration,\n",
    "* Candidate definition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Copy the Generated Notebooks Locally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'s3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/sagemaker-automl-candidates/pr-1-abdf3cc0395847a59aae3eb1548fe43ad9ba1a26a4d444ef80c480bf1'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generated_resources = job['AutoMLJobArtifacts']['DataExplorationNotebookLocation'].rstrip('notebooks/SageMakerAutopilotDataExplorationNotebook.ipynb')\n",
    "generated_resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 cp --recursive $generated_resources ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In the file view, open the following folders:\n",
    "```\n",
    "notebooks/\n",
    "generated_module/\n",
    "```\n",
    "\n",
    "Lots of useful information ^^ in these folders ^^"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Viewing All Candidates\n",
    "Once model tuning is complete, you can view all the candidates (pipeline evaluations with different hyperparameter combinations) that were explored by AutoML and sort them by their final performance metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0  tuning-job-1-5ebf290f7b714803b3-002-a0c1c68c  0.3787199854850769\n",
      "1  tuning-job-1-5ebf290f7b714803b3-001-25c033cf  0.3745099902153015\n",
      "2  tuning-job-1-5ebf290f7b714803b3-003-fe0a1316  0.25881800055503845\n"
     ]
    }
   ],
   "source": [
    "candidates = sm.list_candidates_for_auto_ml_job(AutoMLJobName=auto_ml_job_name, \n",
    "                                                SortBy='FinalObjectiveMetricValue')['Candidates']\n",
    "for index, candidate in enumerate(candidates):\n",
    "    print(str(index) + \"  \" \n",
    "        + candidate['CandidateName'] + \"  \" \n",
    "        + str(candidate['FinalAutoMLJobObjectiveMetric']['Value']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect Trials using Experiments API\n",
    "SageMaker AutoPilot automatically creates a new experiment, and pushes information for each trial. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                  TrialComponentName  \\\n",
      "0  tuning-job-1-5ebf290f7b714803b3-001-25c033cf-a...   \n",
      "1  tuning-job-1-5ebf290f7b714803b3-002-a0c1c68c-a...   \n",
      "2  tuning-job-1-5ebf290f7b714803b3-003-fe0a1316-a...   \n",
      "3  automl-dm--dpp2-rpb-1-6ff77104e6944bb28dc4fcb8...   \n",
      "4  automl-dm--dpp1-csv-1-c09fa794a16f4b27aef1f73f...   \n",
      "5  automl-dm--dpp0-rpb-1-35a30e7d22ad447687fbd336...   \n",
      "6  automl-dm--dpp1-1-600729a9ec964286861e092a3c26...   \n",
      "7  automl-dm--dpp0-1-23b98bdeb3d846c79017cb9f359a...   \n",
      "8  automl-dm--dpp2-1-f6fa3c0be79a4063b97b6571ff33...   \n",
      "9  db-1-534bccbc2999467f9253dc442fe05c71281409594...   \n",
      "\n",
      "                                         DisplayName  \\\n",
      "0  tuning-job-1-5ebf290f7b714803b3-001-25c033cf-a...   \n",
      "1  tuning-job-1-5ebf290f7b714803b3-002-a0c1c68c-a...   \n",
      "2  tuning-job-1-5ebf290f7b714803b3-003-fe0a1316-a...   \n",
      "3  automl-dm--dpp2-rpb-1-6ff77104e6944bb28dc4fcb8...   \n",
      "4  automl-dm--dpp1-csv-1-c09fa794a16f4b27aef1f73f...   \n",
      "5  automl-dm--dpp0-rpb-1-35a30e7d22ad447687fbd336...   \n",
      "6  automl-dm--dpp1-1-600729a9ec964286861e092a3c26...   \n",
      "7  automl-dm--dpp0-1-23b98bdeb3d846c79017cb9f359a...   \n",
      "8  automl-dm--dpp2-1-f6fa3c0be79a4063b97b6571ff33...   \n",
      "9  db-1-534bccbc2999467f9253dc442fe05c71281409594...   \n",
      "\n",
      "                                           SourceArn  \\\n",
      "0  arn:aws:sagemaker:us-east-2:533787958253:train...   \n",
      "1  arn:aws:sagemaker:us-east-2:533787958253:train...   \n",
      "2  arn:aws:sagemaker:us-east-2:533787958253:train...   \n",
      "3  arn:aws:sagemaker:us-east-2:533787958253:trans...   \n",
      "4  arn:aws:sagemaker:us-east-2:533787958253:trans...   \n",
      "5  arn:aws:sagemaker:us-east-2:533787958253:trans...   \n",
      "6  arn:aws:sagemaker:us-east-2:533787958253:train...   \n",
      "7  arn:aws:sagemaker:us-east-2:533787958253:train...   \n",
      "8  arn:aws:sagemaker:us-east-2:533787958253:train...   \n",
      "9  arn:aws:sagemaker:us-east-2:533787958253:proce...   \n",
      "\n",
      "                                  SageMaker.ImageUri  SageMaker.InstanceCount  \\\n",
      "0  257758044811.dkr.ecr.us-east-2.amazonaws.com/s...                      1.0   \n",
      "1  257758044811.dkr.ecr.us-east-2.amazonaws.com/s...                      1.0   \n",
      "2  257758044811.dkr.ecr.us-east-2.amazonaws.com/s...                      1.0   \n",
      "3                                                NaN                      1.0   \n",
      "4                                                NaN                      1.0   \n",
      "5                                                NaN                      1.0   \n",
      "6  257758044811.dkr.ecr.us-east-2.amazonaws.com/s...                      1.0   \n",
      "7  257758044811.dkr.ecr.us-east-2.amazonaws.com/s...                      1.0   \n",
      "8  257758044811.dkr.ecr.us-east-2.amazonaws.com/s...                      1.0   \n",
      "9                                                NaN                      1.0   \n",
      "\n",
      "  SageMaker.InstanceType  SageMaker.VolumeSizeInGB _tuning_objective_metric  \\\n",
      "0          ml.m5.4xlarge                      50.0      validation:accuracy   \n",
      "1          ml.m5.4xlarge                      50.0      validation:accuracy   \n",
      "2          ml.m5.4xlarge                      50.0      validation:accuracy   \n",
      "3          ml.m5.4xlarge                       NaN                      NaN   \n",
      "4          ml.m5.4xlarge                       NaN                      NaN   \n",
      "5          ml.m5.4xlarge                       NaN                      NaN   \n",
      "6          ml.m5.4xlarge                      50.0                      NaN   \n",
      "7          ml.m5.4xlarge                      50.0                      NaN   \n",
      "8          ml.m5.4xlarge                      50.0                      NaN   \n",
      "9          ml.m5.2xlarge                     250.0                      NaN   \n",
      "\n",
      "      alpha  colsample_bytree  ...  train:accuracy - Count  \\\n",
      "0  0.025187          0.817766  ...                     3.0   \n",
      "1  0.025187          0.817766  ...                     3.0   \n",
      "2  0.025187          0.817766  ...                     3.0   \n",
      "3       NaN               NaN  ...                     NaN   \n",
      "4       NaN               NaN  ...                     NaN   \n",
      "5       NaN               NaN  ...                     NaN   \n",
      "6       NaN               NaN  ...                     NaN   \n",
      "7       NaN               NaN  ...                     NaN   \n",
      "8       NaN               NaN  ...                     NaN   \n",
      "9       NaN               NaN  ...                     NaN   \n",
      "\n",
      "                                 SageMaker.ModelName  \\\n",
      "0                                                NaN   \n",
      "1                                                NaN   \n",
      "2                                                NaN   \n",
      "3  automl-dm-27-20-20-56-automl-dm--dpp2-model-ad...   \n",
      "4  automl-dm-27-20-20-56-automl-dm--dpp1-model-db...   \n",
      "5  automl-dm-27-20-20-56-automl-dm--dpp0-model-a3...   \n",
      "6                                                NaN   \n",
      "7                                                NaN   \n",
      "8                                                NaN   \n",
      "9                                                NaN   \n",
      "\n",
      "                      SageMaker.ModelPrimary.DataUrl  \\\n",
      "0                                                NaN   \n",
      "1                                                NaN   \n",
      "2                                                NaN   \n",
      "3  s3://sagemaker-us-east-2-533787958253/models/a...   \n",
      "4  s3://sagemaker-us-east-2-533787958253/models/a...   \n",
      "5  s3://sagemaker-us-east-2-533787958253/models/a...   \n",
      "6                                                NaN   \n",
      "7                                                NaN   \n",
      "8                                                NaN   \n",
      "9                                                NaN   \n",
      "\n",
      "                        SageMaker.ModelPrimary.Image  \\\n",
      "0                                                NaN   \n",
      "1                                                NaN   \n",
      "2                                                NaN   \n",
      "3  257758044811.dkr.ecr.us-east-2.amazonaws.com/s...   \n",
      "4  257758044811.dkr.ecr.us-east-2.amazonaws.com/s...   \n",
      "5  257758044811.dkr.ecr.us-east-2.amazonaws.com/s...   \n",
      "6                                                NaN   \n",
      "7                                                NaN   \n",
      "8                                                NaN   \n",
      "9                                                NaN   \n",
      "\n",
      "                 processor_module                  sagemaker_program  \\\n",
      "0                             NaN                                NaN   \n",
      "1                             NaN                                NaN   \n",
      "2                             NaN                                NaN   \n",
      "3                             NaN                                NaN   \n",
      "4                             NaN                                NaN   \n",
      "5                             NaN                                NaN   \n",
      "6  candidate_data_processors.dpp1  candidate_data_processors.trainer   \n",
      "7  candidate_data_processors.dpp0  candidate_data_processors.trainer   \n",
      "8  candidate_data_processors.dpp2  candidate_data_processors.trainer   \n",
      "9                             NaN                                NaN   \n",
      "\n",
      "   sagemaker_submit_directory input_channel_mode               job_name  \\\n",
      "0                         NaN                NaN                    NaN   \n",
      "1                         NaN                NaN                    NaN   \n",
      "2                         NaN                NaN                    NaN   \n",
      "3                         NaN                NaN                    NaN   \n",
      "4                         NaN                NaN                    NaN   \n",
      "5                         NaN                NaN                    NaN   \n",
      "6     /opt/ml/input/data/code                NaN                    NaN   \n",
      "7     /opt/ml/input/data/code                NaN                    NaN   \n",
      "8     /opt/ml/input/data/code                NaN                    NaN   \n",
      "9                         NaN               Pipe  automl-dm-27-20-20-56   \n",
      "\n",
      "     label_col  \n",
      "0          NaN  \n",
      "1          NaN  \n",
      "2          NaN  \n",
      "3          NaN  \n",
      "4          NaN  \n",
      "5          NaN  \n",
      "6          NaN  \n",
      "7          NaN  \n",
      "8          NaN  \n",
      "9  star_rating  \n",
      "\n",
      "[10 rows x 58 columns]\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.analytics import ExperimentAnalytics, TrainingJobAnalytics\n",
    "\n",
    "exp = ExperimentAnalytics(\n",
    "    sagemaker_session=sess, \n",
    "    experiment_name=auto_ml_job_name + '-aws-auto-ml-job',\n",
    ")\n",
    "\n",
    "df = exp.dataframe()\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the Best Candidate\n",
    "Now that we have successfully completed the AutoML job on our dataset and visualized the trials, we can create a model from any of the trials with a single API call and then deploy that model for online or batch prediction using [Inference Pipelines](https://docs.aws.amazon.com/sagemaker/latest/dg/inference-pipelines.html). For this notebook, we deploy only the best performing trial for inference.\n",
    "\n",
    "The best candidate is the one we're really interested in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Candidate name: tuning-job-1-5ebf290f7b714803b3-002-a0c1c68c\n",
      "Metric name: validation:accuracy\n",
      "Metric value: 0.3787199854850769\n"
     ]
    }
   ],
   "source": [
    "best_candidate = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)['BestCandidate']\n",
    "best_candidate_identifier = best_candidate['CandidateName']\n",
    "\n",
    "print(\"Candidate name: \" + best_candidate_identifier)\n",
    "print(\"Metric name: \" + best_candidate['FinalAutoMLJobObjectiveMetric']['MetricName'])\n",
    "print(\"Metric value: \" + str(best_candidate['FinalAutoMLJobObjectiveMetric']['Value']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'CandidateName': 'tuning-job-1-5ebf290f7b714803b3-002-a0c1c68c',\n",
       " 'FinalAutoMLJobObjectiveMetric': {'MetricName': 'validation:accuracy',\n",
       "  'Value': 0.3787199854850769},\n",
       " 'ObjectiveStatus': 'Succeeded',\n",
       " 'CandidateSteps': [{'CandidateStepType': 'AWS::SageMaker::ProcessingJob',\n",
       "   'CandidateStepArn': 'arn:aws:sagemaker:us-east-2:533787958253:processing-job/db-1-534bccbc2999467f9253dc442fe05c712814095946424fb8b9c0987dfc',\n",
       "   'CandidateStepName': 'db-1-534bccbc2999467f9253dc442fe05c712814095946424fb8b9c0987dfc'},\n",
       "  {'CandidateStepType': 'AWS::SageMaker::TrainingJob',\n",
       "   'CandidateStepArn': 'arn:aws:sagemaker:us-east-2:533787958253:training-job/automl-dm--dpp2-1-f6fa3c0be79a4063b97b6571ff3321c366be99ac91d64',\n",
       "   'CandidateStepName': 'automl-dm--dpp2-1-f6fa3c0be79a4063b97b6571ff3321c366be99ac91d64'},\n",
       "  {'CandidateStepType': 'AWS::SageMaker::TransformJob',\n",
       "   'CandidateStepArn': 'arn:aws:sagemaker:us-east-2:533787958253:transform-job/automl-dm--dpp2-rpb-1-6ff77104e6944bb28dc4fcb89de37d80ec85d33ef',\n",
       "   'CandidateStepName': 'automl-dm--dpp2-rpb-1-6ff77104e6944bb28dc4fcb89de37d80ec85d33ef'},\n",
       "  {'CandidateStepType': 'AWS::SageMaker::TrainingJob',\n",
       "   'CandidateStepArn': 'arn:aws:sagemaker:us-east-2:533787958253:training-job/tuning-job-1-5ebf290f7b714803b3-002-a0c1c68c',\n",
       "   'CandidateStepName': 'tuning-job-1-5ebf290f7b714803b3-002-a0c1c68c'}],\n",
       " 'CandidateStatus': 'Completed',\n",
       " 'InferenceContainers': [{'Image': '257758044811.dkr.ecr.us-east-2.amazonaws.com/sagemaker-sklearn-automl:0.1.0-cpu-py3',\n",
       "   'ModelDataUrl': 's3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/data-processor-models/automl-dm--dpp2-1-f6fa3c0be79a4063b97b6571ff3321c366be99ac91d64/output/model.tar.gz',\n",
       "   'Environment': {'AUTOML_SPARSE_ENCODE_RECORDIO_PROTOBUF': '1',\n",
       "    'AUTOML_TRANSFORM_MODE': 'feature-transform',\n",
       "    'SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT': 'application/x-recordio-protobuf',\n",
       "    'SAGEMAKER_PROGRAM': 'sagemaker_serve',\n",
       "    'SAGEMAKER_SUBMIT_DIRECTORY': '/opt/ml/model/sagemaker_serve.py'}},\n",
       "  {'Image': '257758044811.dkr.ecr.us-east-2.amazonaws.com/sagemaker-xgboost:0.90-1-cpu-py3',\n",
       "   'ModelDataUrl': 's3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/tuning/automl-dm--dpp2-xgb/tuning-job-1-5ebf290f7b714803b3-002-a0c1c68c/output/model.tar.gz',\n",
       "   'Environment': {'MAX_CONTENT_LENGTH': '20971520',\n",
       "    'SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT': 'text/csv'}},\n",
       "  {'Image': '257758044811.dkr.ecr.us-east-2.amazonaws.com/sagemaker-sklearn-automl:0.1.0-cpu-py3',\n",
       "   'ModelDataUrl': 's3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/data-processor-models/automl-dm--dpp2-1-f6fa3c0be79a4063b97b6571ff3321c366be99ac91d64/output/model.tar.gz',\n",
       "   'Environment': {'AUTOML_TRANSFORM_MODE': 'inverse-label-transform',\n",
       "    'SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT': 'text/csv',\n",
       "    'SAGEMAKER_PROGRAM': 'sagemaker_serve',\n",
       "    'SAGEMAKER_SUBMIT_DIRECTORY': '/opt/ml/model/sagemaker_serve.py'}}],\n",
       " 'CreationTime': datetime.datetime(2020, 4, 27, 20, 38, 21, tzinfo=tzlocal()),\n",
       " 'EndTime': datetime.datetime(2020, 4, 27, 20, 39, 5, tzinfo=tzlocal()),\n",
       " 'LastModifiedTime': datetime.datetime(2020, 4, 27, 20, 39, 40, 167000, tzinfo=tzlocal())}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_candidate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the containers and models composing the Inference Pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "257758044811.dkr.ecr.us-east-2.amazonaws.com/sagemaker-sklearn-automl:0.1.0-cpu-py3\n",
      "s3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/data-processor-models/automl-dm--dpp2-1-f6fa3c0be79a4063b97b6571ff3321c366be99ac91d64/output/model.tar.gz\n",
      "======================\n",
      "257758044811.dkr.ecr.us-east-2.amazonaws.com/sagemaker-xgboost:0.90-1-cpu-py3\n",
      "s3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/tuning/automl-dm--dpp2-xgb/tuning-job-1-5ebf290f7b714803b3-002-a0c1c68c/output/model.tar.gz\n",
      "======================\n",
      "257758044811.dkr.ecr.us-east-2.amazonaws.com/sagemaker-sklearn-automl:0.1.0-cpu-py3\n",
      "s3://sagemaker-us-east-2-533787958253/models/autopilot/automl-dm-27-20-20-56/data-processor-models/automl-dm--dpp2-1-f6fa3c0be79a4063b97b6571ff3321c366be99ac91d64/output/model.tar.gz\n",
      "======================\n"
     ]
    }
   ],
   "source": [
    "for container in best_candidate['InferenceContainers']:\n",
    "    print(container['Image'])\n",
    "    print(container['ModelDataUrl'])\n",
    "    print('======================')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Best Candidate!\n",
    "Note that AutoPilot chose different hyper-parameters and feature transformations to achieve the best score for accuracy. We don't have any mention of predictor speed though."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy the Model as a REST Endpoint\n",
    "Batch transformations are also supported, but for now, we will use a REST Endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best candidate model ARN:  arn:aws:sagemaker:us-east-2:533787958253:model/automl-dm-model-27-20-20-56\n"
     ]
    }
   ],
   "source": [
    "model_name = 'automl-dm-model-' + timestamp_suffix\n",
    "\n",
    "model_arn = sm.create_model(Containers=best_candidate['InferenceContainers'],\n",
    "                            ModelName=model_name,\n",
    "                            ExecutionRoleArn=role)\n",
    "\n",
    "print('Best candidate model ARN: ', model_arn['ModelArn'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Amazon Resource Name (ARN)\n",
    "\n",
    "As a refresher: Amazon Resource Name (ARN) is a file naming convention used to identify a particular resource in the Amazon Web Services (AWS) public cloud. ARNs, which are specific to AWS, help an administrator track and use AWS items and policies across AWS products and API calls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "automl-dm-ep-28-11-57-43\n",
      "automl-dm-variant-28-11-57-43\n"
     ]
    }
   ],
   "source": [
    "# EndpointConfig name\n",
    "timestamp_suffix = strftime('%d-%H-%M-%S', gmtime())\n",
    "epc_name = 'automl-dm-epc-' + timestamp_suffix\n",
    "\n",
    "# Endpoint name\n",
    "xgb_endpoint_name = 'automl-dm-ep-' + timestamp_suffix\n",
    "variant_name = 'automl-dm-variant-' + timestamp_suffix\n",
    "\n",
    "print(xgb_endpoint_name)\n",
    "print(variant_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "ep_config = sm.create_endpoint_config(EndpointConfigName = epc_name,\n",
    "                                      ProductionVariants=[{'InstanceType':'ml.m4.xlarge',\n",
    "                                                           'InitialInstanceCount': 1,\n",
    "                                                           'ModelName': model_name,\n",
    "                                                           'VariantName': variant_name}])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arn:aws:sagemaker:us-east-2:533787958253:endpoint/automl-dm-ep-28-11-57-43\n"
     ]
    }
   ],
   "source": [
    "create_endpoint_response = sm.create_endpoint(EndpointName=xgb_endpoint_name,\n",
    "                                              EndpointConfigName=epc_name)\n",
    "print(create_endpoint_response['EndpointArn'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wait for the Model to Deploy\n",
    "This may take 5-10 mins.  Please be patient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm.get_waiter('endpoint_in_service').wait(EndpointName=xgb_endpoint_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Arn: arn:aws:sagemaker:us-east-2:533787958253:endpoint/automl-dm-ep-28-11-57-43\n",
      "Status: InService\n"
     ]
    }
   ],
   "source": [
    "resp = sm.describe_endpoint(EndpointName=xgb_endpoint_name)\n",
    "status = resp['EndpointStatus']\n",
    "\n",
    "print(\"Arn: \" + resp['EndpointArn'])\n",
    "print(\"Status: \" + status)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test Our Model with Some Example Reviews\n",
    "Let's do some ad-hoc predictions on our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm_runtime = boto3.client('sagemaker-runtime')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'4'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "csv_line_predict_positive = \"\"\"I loved it!\"\"\"\n",
    "\n",
    "response = sm_runtime.invoke_endpoint(EndpointName=xgb_endpoint_name, ContentType='text/csv', Accept='text/csv', Body=csv_line_predict_positive)\n",
    "\n",
    "response_body = response['Body'].read().decode('utf-8').strip()\n",
    "response_body"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "csv_line_predict_meh = \"\"\"It's OK.\"\"\"\n",
    "\n",
    "response = sm_runtime.invoke_endpoint(EndpointName=xgb_endpoint_name, ContentType='text/csv', Accept='text/csv', Body=csv_line_predict_meh)\n",
    "\n",
    "response_body = response['Body'].read().decode('utf-8').strip()\n",
    "response_body"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1'"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "csv_line_predict_negative = \"\"\"The worst product ever.\"\"\"\n",
    "\n",
    "response = sm_runtime.invoke_endpoint(EndpointName=xgb_endpoint_name, ContentType='text/csv', Accept='text/csv', Body=csv_line_predict_negative)\n",
    "\n",
    "response_body = response['Body'].read().decode('utf-8').strip()\n",
    "response_body"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "We used AutoPilot to automatically find the best model, hyper-parameters, and feature-engineering scripts for our dataset.  \n",
    "\n",
    "AutoPilot uses a white-box approach to generate re-usable exploration Jupyter Notebooks and transformation Python scripts to continue to train and deploy our model on new data - well after this initial interaction with the AutoPilot service."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}