AUTOGLUON-WALKTHROUGH.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"padding: 10px; border: 1px solid black;\">\n",
    "<img src=\"./../../images/MLU-NEW-logo.png\" alt=\"drawing\" width=\"400\"/> <br/>\n",
    "\n",
    "\n",
    "# <a name=\"0\">Code Walkthrough & Advanced AutoGluon Features</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook shows how to use AutoGluon `TabularPredictor` to solve two machine learning tasks: a __regression task__ (book price prediction) and a __multiclass classification task__ (occupation prediction). \n",
    "\n",
    "<a href=\"#01\">Part I - Solution Walkthrough & Discussions</a>, covers a basic solution for the Book Price regression problem from the *MLU-DAY-ONE-ML-Hands-On.ipynb* notebook.\n",
    "\n",
    "<a href=\"#02\">Part II - Advanced AutoGluon Features</a>, dives deeper into more advanced AutoGluon features, solving a multiclass classification task of predicting the occupation of individuals using US census data.\n",
    "\n",
    "1. <a href=\"#1\">ML Problem Description</a>\n",
    "2. <a href=\"#2\">Loading the Data</a>\n",
    "3. <a href=\"#5\">Model Training with AutoGluon</a>\n",
    "    * Specifying performance metric\n",
    "    * Specifying settings for TabularPredictor\n",
    "    * Specifying hyperparameters and tuning them\n",
    "    \n",
    "4. <a href=\"#7\">Model ensembling with stacking/bagging</a>\n",
    "5. <a href=\"#8\">Prediction options (inference)</a>\n",
    "6. <a href=\"#10\">Selecting individual models for predictions</a>\n",
    "7. <a href=\"#11\">Interpretability: Feature importance</a>\n",
    "8. <a href=\"#12\">Inference Speed: Model distillation</a>\n",
    "    * Training student models\n",
    "    * Excluding models\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Jupiter notebooks environment__:\n",
    "\n",
    "* Jupiter notebooks allow creating and sharing documents that contain both code and rich text cells. If you are not familiar with Jupiter notebooks, read more [here](https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/what_is_jupyter.html). \n",
    "* This is a quick-start demo to bring you up to speed on coding and experimenting with machine learning. Move through the notebook __from top to bottom__. \n",
    "* Run each code cell to see its output. To run a cell, click within the cell and press __Shift+Enter__, or click __Run__ from the top of the page menu. \n",
    "* A `[*]` symbol next to the cell indicates the code is still running. A `[#]` symbol, where # is an integer, indicates it is finished.\n",
    "* Beware, __some code cells might take longer to run__, sometimes 5-10 minutes (depending on the task, installing packages and libraries, training models, etc.)\n",
    "\n",
    "Let's start by loading some libraries and packages!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install -q autogluon==0.6.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load in libraries\n",
    "import pandas as pd\n",
    "# Importing the libraries needed to work with our Tabular dataset.\n",
    "from autogluon.tabular import TabularPredictor, TabularDataset\n",
    "# Additional library for tuning\n",
    "import autogluon.core as ag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# <a name=\"01\">Part I - Walkthrough & Discussions</a>\n",
    "(<a href=\"#0\">Go to top</a>)\n",
    "\n",
    "Now that you have finished your hands-on activity, let's walk through the code you have used and discuss it. <br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No path specified. Models will be saved in: \"AutogluonModels/ag-20230207_190023/\"\n",
      "Beginning AutoGluon training ... Time limit = 60s\n",
      "AutoGluon will save models to \"AutogluonModels/ag-20230207_190023/\"\n",
      "AutoGluon Version:  0.6.2\n",
      "Python Version:     3.9.13\n",
      "Operating System:   Linux\n",
      "Platform Machine:   x86_64\n",
      "Platform Version:   #1 SMP Fri Oct 14 01:16:24 UTC 2022\n",
      "Train Data Rows:    5051\n",
      "Train Data Columns: 9\n",
      "Label Column: Price\n",
      "Preprocessing data ...\n",
      "AutoGluon infers your prediction problem is: 'regression' (because dtype of label-column == float and many unique label-values observed).\n",
      "\tLabel info (max, min, mean, stddev): (4.149249912590282, 1.414973347970818, 2.60147, 0.33003)\n",
      "\tIf 'regression' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n",
      "Using Feature Generators to preprocess the data ...\n",
      "Fitting AutoMLPipelineFeatureGenerator...\n",
      "\tAvailable Memory:                    14882.21 MB\n",
      "\tTrain Data (Original)  Memory Usage: 10.91 MB (0.1% of available memory)\n",
      "\tInferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.\n",
      "\tStage 1 Generators:\n",
      "\t\tFitting AsTypeFeatureGenerator...\n",
      "\tStage 2 Generators:\n",
      "\t\tFitting FillNaFeatureGenerator...\n",
      "\tStage 3 Generators:\n",
      "\t\tFitting IdentityFeatureGenerator...\n",
      "\t\tFitting CategoryFeatureGenerator...\n",
      "\t\t\tFitting CategoryMemoryMinimizeFeatureGenerator...\n",
      "\t\tFitting TextSpecialFeatureGenerator...\n",
      "\t\t\tFitting BinnedFeatureGenerator...\n",
      "\t\t\tFitting DropDuplicatesFeatureGenerator...\n",
      "\t\tFitting TextNgramFeatureGenerator...\n",
      "\t\t\tFitting CountVectorizer for text features: ['Title', 'Edition', 'Ratings', 'Synopsis']\n",
      "\t\t\tCountVectorizer fit with vocabulary size = 4920\n",
      "\tStage 4 Generators:\n",
      "\t\tFitting DropUniqueFeatureGenerator...\n",
      "\tTypes of features in original data (raw dtype, special dtypes):\n",
      "\t\t('int', [])          : 1 | ['ID']\n",
      "\t\t('object', [])       : 4 | ['Author', 'Reviews', 'Genre', 'BookCategory']\n",
      "\t\t('object', ['text']) : 4 | ['Title', 'Edition', 'Ratings', 'Synopsis']\n",
      "\tTypes of features in processed data (raw dtype, special dtypes):\n",
      "\t\t('category', [])                    :    4 | ['Author', 'Reviews', 'Genre', 'BookCategory']\n",
      "\t\t('category', ['text_as_category'])  :    4 | ['Title', 'Edition', 'Ratings', 'Synopsis']\n",
      "\t\t('int', [])                         :    1 | ['ID']\n",
      "\t\t('int', ['binned', 'text_special']) :   56 | ['Title.char_count', 'Title.word_count', 'Title.capital_ratio', 'Title.lower_ratio', 'Title.digit_ratio', ...]\n",
      "\t\t('int', ['text_ngram'])             : 4921 | ['__nlp__.000', '__nlp__.10', '__nlp__.10 customer', '__nlp__.10 customer reviews', '__nlp__.100', ...]\n",
      "\t13.6s = Fit runtime\n",
      "\t9 features in original data used to generate 4986 features in processed data.\n",
      "\tTrain Data (Processed) Memory Usage: 50.11 MB (0.3% of available memory)\n",
      "Data preprocessing and feature engineering runtime = 14.3s ...\n",
      "AutoGluon will gauge predictive performance using evaluation metric: 'root_mean_squared_error'\n",
      "\tThis metric's sign has been flipped to adhere to being higher_is_better. The metric score can be multiplied by -1 to get the metric value.\n",
      "\tTo change this, specify the eval_metric parameter of Predictor()\n",
      "Automatically generating train/validation split with holdout_frac=0.1, Train Rows: 4545, Val Rows: 506\n",
      "Fitting 11 L1 models ...\n",
      "Fitting model: KNeighborsUnif ... Training model for up to 45.7s of the 45.57s of remaining time.\n",
      "\t-0.3518\t = Validation score   (-root_mean_squared_error)\n",
      "\t1.41s\t = Training   runtime\n",
      "\t1.12s\t = Validation runtime\n",
      "Fitting model: KNeighborsDist ... Training model for up to 42.86s of the 42.72s of remaining time.\n",
      "\t-0.3505\t = Validation score   (-root_mean_squared_error)\n",
      "\t1.35s\t = Training   runtime\n",
      "\t0.74s\t = Validation runtime\n",
      "Fitting model: LightGBMXT ... Training model for up to 40.48s of the 40.34s of remaining time.\n",
      "\t-0.2142\t = Validation score   (-root_mean_squared_error)\n",
      "\t12.96s\t = Training   runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "Fitting model: LightGBM ... Training model for up to 27.23s of the 27.02s of remaining time.\n",
      "\t-0.2129\t = Validation score   (-root_mean_squared_error)\n",
      "\t10.25s\t = Training   runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "Fitting model: RandomForestMSE ... Training model for up to 16.64s of the 16.42s of remaining time.\n",
      "\t-0.236\t = Validation score   (-root_mean_squared_error)\n",
      "\t439.35s\t = Training   runtime\n",
      "\t0.08s\t = Validation runtime\n",
      "Fitting model: WeightedEnsemble_L2 ... Training model for up to 45.7s of the -424.42s of remaining time.\n",
      "\t-0.2103\t = Validation score   (-root_mean_squared_error)\n",
      "\t0.14s\t = Training   runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "AutoGluon training complete, total runtime = 484.88s ... Best model: \"WeightedEnsemble_L2\"\n",
      "TabularPredictor saved. To load, use: predictor = TabularPredictor.load(\"AutogluonModels/ag-20230207_190023/\")\n"
     ]
    }
   ],
   "source": [
    "# Loading the train and test datasets\n",
    "df_train = TabularDataset(\"../../data/training.csv\")\n",
    "df_test = TabularDataset(\"../../data/mlu-leaderboard-test.csv\")\n",
    "\n",
    "# Train a model with AutoGluon on the train dataset\n",
    "# Set the training time to a minute here (60 seconds), for fast experimentation\n",
    "predictor = TabularPredictor(label=\"Price\", eval_metric=\"mean_squared_error\").fit(\n",
    "    train_data=df_train, time_limit=60\n",
    ")\n",
    "\n",
    "# Make predictions on the test dataset with the AutoGluon model\n",
    "predictions = predictor.predict(df_test)\n",
    "\n",
    "# Creating a new dataframe for the MLU Leaderboard submission\n",
    "submission = df_test[[\"ID\"]].copy(deep=True)\n",
    "\n",
    "# Creating label column from price prediction list\n",
    "submission[\"Price\"] = predictions\n",
    "\n",
    "# Saving the dataframe as a csv file for MLU Leaderboard submission\n",
    "# index=False prevents printing the row IDs as separate values\n",
    "submission.to_csv(\n",
    "    \"../../data/predictions/Solution-Demo.csv\",\n",
    "    index=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# <a name=\"02\">Part II - Advanced AutoGluon Features</a>\n",
    "(<a href=\"#0\">Go to top</a>)\n",
    "\n",
    "---\n",
    "## <a name=\"1\">ML Problem Description</a>\n",
    "\n",
    "Predict the occupation of individuals using census data. \n",
    "> This is a __multiclass classification__ task (15 distinct classes). <br>\n",
    "\n",
    "For the advanced feature demonstration we use a new dataset: Census data. In this particular dataset, each row corresponds to an individual person, and the columns contain various demographic characteristics collected for the census.\n",
    "\n",
    "We predict the occupation of an individual - this is a multiclass classification problem. Start by importing AutoGluon’s `TabularPredictor` and `TabularDataset`, and load the data from a S3 bucket."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## <a name=\"2\">Loading the data</a>\n",
    "(<a href=\"#0\">Go to top</a>)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loaded data from: https://autogluon.s3.amazonaws.com/datasets/Inc/train.csv | Columns = 15 / 15 | Rows = 39073 -> 39073\n",
      "Loaded data from: https://autogluon.s3.amazonaws.com/datasets/Inc/test.csv | Columns = 15 / 15 | Rows = 9769 -> 9769\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education</th>\n",
       "      <th>education-num</th>\n",
       "      <th>marital-status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>capital-gain</th>\n",
       "      <th>capital-loss</th>\n",
       "      <th>hours-per-week</th>\n",
       "      <th>native-country</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6118</th>\n",
       "      <td>51</td>\n",
       "      <td>Private</td>\n",
       "      <td>39264</td>\n",
       "      <td>Some-college</td>\n",
       "      <td>10</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23204</th>\n",
       "      <td>58</td>\n",
       "      <td>Private</td>\n",
       "      <td>51662</td>\n",
       "      <td>10th</td>\n",
       "      <td>6</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Other-service</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29590</th>\n",
       "      <td>40</td>\n",
       "      <td>Private</td>\n",
       "      <td>326310</td>\n",
       "      <td>Some-college</td>\n",
       "      <td>10</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Craft-repair</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>44</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18116</th>\n",
       "      <td>37</td>\n",
       "      <td>Private</td>\n",
       "      <td>222450</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Sales</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>2339</td>\n",
       "      <td>40</td>\n",
       "      <td>El-Salvador</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33964</th>\n",
       "      <td>62</td>\n",
       "      <td>Private</td>\n",
       "      <td>109190</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>15024</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       age workclass  fnlwgt      education  education-num  \\\n",
       "6118    51   Private   39264   Some-college             10   \n",
       "23204   58   Private   51662           10th              6   \n",
       "29590   40   Private  326310   Some-college             10   \n",
       "18116   37   Private  222450        HS-grad              9   \n",
       "33964   62   Private  109190      Bachelors             13   \n",
       "\n",
       "            marital-status        occupation    relationship    race      sex  \\\n",
       "6118    Married-civ-spouse   Exec-managerial            Wife   White   Female   \n",
       "23204   Married-civ-spouse     Other-service            Wife   White   Female   \n",
       "29590   Married-civ-spouse      Craft-repair         Husband   White     Male   \n",
       "18116        Never-married             Sales   Not-in-family   White     Male   \n",
       "33964   Married-civ-spouse   Exec-managerial         Husband   White     Male   \n",
       "\n",
       "       capital-gain  capital-loss  hours-per-week  native-country   class  \n",
       "6118              0             0              40   United-States    >50K  \n",
       "23204             0             0               8   United-States   <=50K  \n",
       "29590             0             0              44   United-States   <=50K  \n",
       "18116             0          2339              40     El-Salvador   <=50K  \n",
       "33964         15024             0              40   United-States    >50K  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load in the dataset\n",
    "train_data = TabularDataset(\"https://autogluon.s3.amazonaws.com/datasets/Inc/train.csv\")\n",
    "\n",
    "# Let's load the test data\n",
    "test_data = TabularDataset(\"https://autogluon.s3.amazonaws.com/datasets/Inc/test.csv\")\n",
    "\n",
    "# Subsample a subset of data for faster demo, try setting this to much larger values\n",
    "subsample_size = 1000\n",
    "\n",
    "train_data = train_data.sample(n=subsample_size, random_state=0)\n",
    "train_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## <a name=\"5\">Model Training with AutoGluon</a>\n",
    "(<a href=\"#0\">Go to top</a>)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specifying performance metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We specify eval-metric just for demo (unnecessary as it's the default)\n",
    "metric = \"accuracy\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The full list of AutoGluon classification metrics can be found here:\n",
    "\n",
    "`'accuracy', 'balanced_accuracy', 'f1', 'f1_macro', 'f1_micro', 'f1_weighted', 'roc_auc', 'average_precision', 'precision', 'precision_macro', 'precision_micro', 'precision_weighted', 'recall', 'recall_macro', 'recall_micro', 'recall_weighted', 'log_loss', 'pac_score'`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specifying settings for TabularPredictor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train various models for ~2 min\n",
    "time_limit = 2 * 60"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specifying hyperparameters and tuning them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set Neural Net options\n",
    "# Specifies non-default hyperparameter values for neural network models\n",
    "nn_options = {\n",
    "    # number of training epochs (controls training time of NN models)\n",
    "    \"num_epochs\": 10,\n",
    "    # learning rate used in training (real-valued hyperparameter searched on log-scale)\n",
    "    \"learning_rate\": ag.space.Real(1e-4, 1e-2, default=5e-4, log=True),\n",
    "    # activation function used in NN (categorical hyperparameter, default = first entry)\n",
    "    \"activation\": ag.space.Categorical(\"relu\", \"softrelu\", \"tanh\"),\n",
    "    # dropout probability (real-valued hyperparameter)\n",
    "    \"dropout_prob\": ag.space.Real(0.0, 0.5, default=0.1),\n",
    "}\n",
    "\n",
    "# Set GBM options\n",
    "# Specifies non-default hyperparameter values for lightGBM gradient boosted trees\n",
    "gbm_options = {\n",
    "    # number of boosting rounds (controls training time of GBM models)\n",
    "    \"num_boost_round\": 100,\n",
    "    # number of leaves in trees (integer hyperparameter)\n",
    "    \"num_leaves\": ag.space.Int(lower=26, upper=66, default=36),\n",
    "}\n",
    "\n",
    "# Add both NN and GBM options into a hyperparameter dictionary\n",
    "# hyperparameters of each model type\n",
    "# When these keys are missing from the hyperparameters dict, no models of that type are trained\n",
    "hyperparameters = {\n",
    "    \"GBM\": gbm_options,\n",
    "    \"NN_TORCH\": nn_options,\n",
    "}\n",
    "\n",
    "# To tune hyperparameters using Bayesian optimization to find best combination of params\n",
    "search_strategy = \"auto\"\n",
    "\n",
    "# Number of trials for hyperparameters\n",
    "num_trials = 5\n",
    "\n",
    "# HPO is not performed unless hyperparameter_tune_kwargs is specified\n",
    "hyperparameter_tune_kwargs = {\n",
    "    \"num_trials\": num_trials,\n",
    "    \"scheduler\": \"local\",\n",
    "    \"searcher\": search_strategy,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train & Tune Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fitted model: NeuralNetTorch/d3eb103e ...\n",
      "\t0.355\t = Validation score   (accuracy)\n",
      "\t2.39s\t = Training   runtime\n",
      "\t0.02s\t = Validation runtime\n",
      "Fitted model: NeuralNetTorch/d526198a ...\n",
      "\t0.32\t = Validation score   (accuracy)\n",
      "\t2.96s\t = Training   runtime\n",
      "\t0.02s\t = Validation runtime\n",
      "Fitted model: NeuralNetTorch/d528a1fa ...\n",
      "\t0.36\t = Validation score   (accuracy)\n",
      "\t2.97s\t = Training   runtime\n",
      "\t0.02s\t = Validation runtime\n",
      "Fitted model: NeuralNetTorch/d52be89c ...\n",
      "\t0.35\t = Validation score   (accuracy)\n",
      "\t2.73s\t = Training   runtime\n",
      "\t0.02s\t = Validation runtime\n",
      "Fitted model: NeuralNetTorch/d53001ac ...\n",
      "\t0.35\t = Validation score   (accuracy)\n",
      "\t1.54s\t = Training   runtime\n",
      "\t0.02s\t = Validation runtime\n",
      "Fitting model: WeightedEnsemble_L2 ... Training model for up to 119.9s of the 100.99s of remaining time.\n",
      "\t0.405\t = Validation score   (accuracy)\n",
      "\t0.28s\t = Training   runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "AutoGluon training complete, total runtime = 19.31s ... Best model: \"WeightedEnsemble_L2\"\n",
      "TabularPredictor saved. To load, use: predictor = TabularPredictor.load(\"AutogluonModels/ag-20230207_190830/\")\n"
     ]
    }
   ],
   "source": [
    "predictor = TabularPredictor(label=\"occupation\", eval_metric=metric).fit(\n",
    "    train_data,\n",
    "    time_limit=time_limit,\n",
    "    hyperparameters=hyperparameters,\n",
    "    hyperparameter_tune_kwargs=hyperparameter_tune_kwargs,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the following to view a summary of what happened during the fit. Now this command will show details of the hyperparameter-tuning process for each type of model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** Summary of fit() ***\n",
      "Estimated performance of each model:\n",
      "                      model  score_val  pred_time_val  fit_time  pred_time_val_marginal  fit_time_marginal  stack_level  can_infer  fit_order\n",
      "0       WeightedEnsemble_L2      0.405       0.052138  6.178108                0.000443           0.280164            2       True         11\n",
      "1               LightGBM/T3      0.375       0.004916  0.403158                0.004916           0.403158            1       True          3\n",
      "2               LightGBM/T5      0.375       0.006534  0.592385                0.006534           0.592385            1       True          5\n",
      "3               LightGBM/T1      0.370       0.004292  0.633591                0.004292           0.633591            1       True          1\n",
      "4               LightGBM/T4      0.360       0.009535  0.657517                0.009535           0.657517            1       True          4\n",
      "5   NeuralNetTorch/d528a1fa      0.360       0.019321  2.972190                0.019321           2.972190            1       True          8\n",
      "6               LightGBM/T2      0.355       0.005768  0.666110                0.005768           0.666110            1       True          2\n",
      "7   NeuralNetTorch/d3eb103e      0.355       0.020069  2.388131                0.020069           2.388131            1       True          6\n",
      "8   NeuralNetTorch/d53001ac      0.350       0.017180  1.537102                0.017180           1.537102            1       True         10\n",
      "9   NeuralNetTorch/d52be89c      0.350       0.018773  2.731708                0.018773           2.731708            1       True          9\n",
      "10  NeuralNetTorch/d526198a      0.320       0.016947  2.960038                0.016947           2.960038            1       True          7\n",
      "Number of models trained: 11\n",
      "Types of models trained:\n",
      "{'LGBModel', 'WeightedEnsembleModel', 'TabularNeuralNetTorchModel'}\n",
      "Bagging used: False \n",
      "Multi-layer stack-ensembling used: False \n",
      "Feature Metadata (Processed):\n",
      "(raw dtype, special dtypes):\n",
      "('category', [])  : 6 | ['workclass', 'education', 'marital-status', 'relationship', 'race', ...]\n",
      "('int', [])       : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]\n",
      "('int', ['bool']) : 2 | ['sex', 'class']\n",
      "Plot summary of models saved to file: AutogluonModels/ag-20230207_190830/SummaryOfModels.html\n",
      "*** End of fit() summary ***\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'model_types': {'LightGBM/T1': 'LGBModel',\n",
       "  'LightGBM/T2': 'LGBModel',\n",
       "  'LightGBM/T3': 'LGBModel',\n",
       "  'LightGBM/T4': 'LGBModel',\n",
       "  'LightGBM/T5': 'LGBModel',\n",
       "  'NeuralNetTorch/d3eb103e': 'TabularNeuralNetTorchModel',\n",
       "  'NeuralNetTorch/d526198a': 'TabularNeuralNetTorchModel',\n",
       "  'NeuralNetTorch/d528a1fa': 'TabularNeuralNetTorchModel',\n",
       "  'NeuralNetTorch/d52be89c': 'TabularNeuralNetTorchModel',\n",
       "  'NeuralNetTorch/d53001ac': 'TabularNeuralNetTorchModel',\n",
       "  'WeightedEnsemble_L2': 'WeightedEnsembleModel'},\n",
       " 'model_performance': {'LightGBM/T1': 0.37,\n",
       "  'LightGBM/T2': 0.355,\n",
       "  'LightGBM/T3': 0.375,\n",
       "  'LightGBM/T4': 0.36,\n",
       "  'LightGBM/T5': 0.375,\n",
       "  'NeuralNetTorch/d3eb103e': 0.355,\n",
       "  'NeuralNetTorch/d526198a': 0.32,\n",
       "  'NeuralNetTorch/d528a1fa': 0.36,\n",
       "  'NeuralNetTorch/d52be89c': 0.35,\n",
       "  'NeuralNetTorch/d53001ac': 0.35,\n",
       "  'WeightedEnsemble_L2': 0.405},\n",
       " 'model_best': 'WeightedEnsemble_L2',\n",
       " 'model_paths': {'LightGBM/T1': '/home/ec2-user/SageMaker/MLA-DAY1-Course/notebooks/demo/AutogluonModels/ag-20230207_190830/models/LightGBM/T1/',\n",
       "  'LightGBM/T2': '/home/ec2-user/SageMaker/MLA-DAY1-Course/notebooks/demo/AutogluonModels/ag-20230207_190830/models/LightGBM/T2/',\n",
       "  'LightGBM/T3': '/home/ec2-user/SageMaker/MLA-DAY1-Course/notebooks/demo/AutogluonModels/ag-20230207_190830/models/LightGBM/T3/',\n",
       "  'LightGBM/T4': '/home/ec2-user/SageMaker/MLA-DAY1-Course/notebooks/demo/AutogluonModels/ag-20230207_190830/models/LightGBM/T4/',\n",
       "  'LightGBM/T5': '/home/ec2-user/SageMaker/MLA-DAY1-Course/notebooks/demo/AutogluonModels/ag-20230207_190830/models/LightGBM/T5/',\n",
       "  'NeuralNetTorch/d3eb103e': '/home/ec2-user/SageMaker/MLA-DAY1-Course/notebooks/demo/AutogluonModels/ag-20230207_190830/models/NeuralNetTorch/d3eb103e/',\n",
       "  'NeuralNetTorch/d526198a': '/home/ec2-user/SageMaker/MLA-DAY1-Course/notebooks/demo/AutogluonModels/ag-20230207_190830/models/NeuralNetTorch/d526198a/',\n",
       "  'NeuralNetTorch/d528a1fa': '/home/ec2-user/SageMaker/MLA-DAY1-Course/notebooks/demo/AutogluonModels/ag-20230207_190830/models/NeuralNetTorch/d528a1fa/',\n",
       "  'NeuralNetTorch/d52be89c': '/home/ec2-user/SageMaker/MLA-DAY1-Course/notebooks/demo/AutogluonModels/ag-20230207_190830/models/NeuralNetTorch/d52be89c/',\n",
       "  'NeuralNetTorch/d53001ac': '/home/ec2-user/SageMaker/MLA-DAY1-Course/notebooks/demo/AutogluonModels/ag-20230207_190830/models/NeuralNetTorch/d53001ac/',\n",
       "  'WeightedEnsemble_L2': 'AutogluonModels/ag-20230207_190830/models/WeightedEnsemble_L2/'},\n",
       " 'model_fit_times': {'LightGBM/T1': 0.6335914134979248,\n",
       "  'LightGBM/T2': 0.6661102771759033,\n",
       "  'LightGBM/T3': 0.40315771102905273,\n",
       "  'LightGBM/T4': 0.6575171947479248,\n",
       "  'LightGBM/T5': 0.5923850536346436,\n",
       "  'NeuralNetTorch/d3eb103e': 2.3881313800811768,\n",
       "  'NeuralNetTorch/d526198a': 2.960038423538208,\n",
       "  'NeuralNetTorch/d528a1fa': 2.9721896648406982,\n",
       "  'NeuralNetTorch/d52be89c': 2.731707811355591,\n",
       "  'NeuralNetTorch/d53001ac': 1.5371019840240479,\n",
       "  'WeightedEnsemble_L2': 0.2801644802093506},\n",
       " 'model_pred_times': {'LightGBM/T1': 0.004291534423828125,\n",
       "  'LightGBM/T2': 0.005768299102783203,\n",
       "  'LightGBM/T3': 0.004916191101074219,\n",
       "  'LightGBM/T4': 0.009534835815429688,\n",
       "  'LightGBM/T5': 0.00653386116027832,\n",
       "  'NeuralNetTorch/d3eb103e': 0.020069360733032227,\n",
       "  'NeuralNetTorch/d526198a': 0.01694655418395996,\n",
       "  'NeuralNetTorch/d528a1fa': 0.019321441650390625,\n",
       "  'NeuralNetTorch/d52be89c': 0.01877307891845703,\n",
       "  'NeuralNetTorch/d53001ac': 0.017180204391479492,\n",
       "  'WeightedEnsemble_L2': 0.00044345855712890625},\n",
       " 'num_bag_folds': 0,\n",
       " 'max_stack_level': 2,\n",
       " 'num_classes': 13,\n",
       " 'model_hyperparams': {'LightGBM/T1': {'learning_rate': 0.05,\n",
       "   'num_boost_round': 100,\n",
       "   'num_leaves': 36,\n",
       "   'feature_fraction': 1.0,\n",
       "   'min_data_in_leaf': 20},\n",
       "  'LightGBM/T2': {'learning_rate': 0.06994332504138305,\n",
       "   'num_boost_round': 100,\n",
       "   'num_leaves': 29,\n",
       "   'feature_fraction': 0.8872033759818312,\n",
       "   'min_data_in_leaf': 5},\n",
       "  'LightGBM/T3': {'learning_rate': 0.049883446878335284,\n",
       "   'num_boost_round': 100,\n",
       "   'num_leaves': 62,\n",
       "   'feature_fraction': 0.9618129346960314,\n",
       "   'min_data_in_leaf': 52},\n",
       "  'LightGBM/T4': {'learning_rate': 0.006163502781172814,\n",
       "   'num_boost_round': 100,\n",
       "   'num_leaves': 27,\n",
       "   'feature_fraction': 0.824383651636118,\n",
       "   'min_data_in_leaf': 14},\n",
       "  'LightGBM/T5': {'learning_rate': 0.035179640321040824,\n",
       "   'num_boost_round': 100,\n",
       "   'num_leaves': 43,\n",
       "   'feature_fraction': 0.9479312595206661,\n",
       "   'min_data_in_leaf': 26},\n",
       "  'NeuralNetTorch/d3eb103e': {'num_epochs': 10,\n",
       "   'epochs_wo_improve': 20,\n",
       "   'activation': 'relu',\n",
       "   'embedding_size_factor': 1.0,\n",
       "   'embed_exponent': 0.56,\n",
       "   'max_embedding_dim': 100,\n",
       "   'y_range': None,\n",
       "   'y_range_extend': 0.05,\n",
       "   'dropout_prob': 0.1,\n",
       "   'optimizer': 'adam',\n",
       "   'learning_rate': 0.0005,\n",
       "   'weight_decay': 1e-06,\n",
       "   'proc.embed_min_categories': 4,\n",
       "   'proc.impute_strategy': 'median',\n",
       "   'proc.max_category_levels': 100,\n",
       "   'proc.skew_threshold': 0.99,\n",
       "   'use_ngram_features': False,\n",
       "   'num_layers': 2,\n",
       "   'hidden_size': 128,\n",
       "   'max_batch_size': 512,\n",
       "   'use_batchnorm': False,\n",
       "   'loss_function': 'auto'},\n",
       "  'NeuralNetTorch/d526198a': {'num_epochs': 10,\n",
       "   'epochs_wo_improve': 20,\n",
       "   'activation': 'relu',\n",
       "   'embedding_size_factor': 1.2,\n",
       "   'embed_exponent': 0.56,\n",
       "   'max_embedding_dim': 100,\n",
       "   'y_range': None,\n",
       "   'y_range_extend': 0.05,\n",
       "   'dropout_prob': 0.31980971957609977,\n",
       "   'optimizer': 'adam',\n",
       "   'learning_rate': 0.0016701807999313742,\n",
       "   'weight_decay': 0.0002974302508639784,\n",
       "   'proc.embed_min_categories': 4,\n",
       "   'proc.impute_strategy': 'most_frequent',\n",
       "   'proc.max_category_levels': 100,\n",
       "   'proc.skew_threshold': 10.0,\n",
       "   'use_ngram_features': False,\n",
       "   'num_layers': 4,\n",
       "   'hidden_size': 128,\n",
       "   'max_batch_size': 512,\n",
       "   'use_batchnorm': False,\n",
       "   'loss_function': 'auto'},\n",
       "  'NeuralNetTorch/d528a1fa': {'num_epochs': 10,\n",
       "   'epochs_wo_improve': 20,\n",
       "   'activation': 'softrelu',\n",
       "   'embedding_size_factor': 0.5,\n",
       "   'embed_exponent': 0.56,\n",
       "   'max_embedding_dim': 100,\n",
       "   'y_range': None,\n",
       "   'y_range_extend': 0.05,\n",
       "   'dropout_prob': 0.33356705595003966,\n",
       "   'optimizer': 'adam',\n",
       "   'learning_rate': 0.00040199441211342187,\n",
       "   'weight_decay': 3.3805894971643656e-07,\n",
       "   'proc.embed_min_categories': 1000,\n",
       "   'proc.impute_strategy': 'mean',\n",
       "   'proc.max_category_levels': 20,\n",
       "   'proc.skew_threshold': 0.9,\n",
       "   'use_ngram_features': False,\n",
       "   'num_layers': 2,\n",
       "   'hidden_size': 512,\n",
       "   'max_batch_size': 512,\n",
       "   'use_batchnorm': False,\n",
       "   'loss_function': 'auto'},\n",
       "  'NeuralNetTorch/d52be89c': {'num_epochs': 10,\n",
       "   'epochs_wo_improve': 20,\n",
       "   'activation': 'relu',\n",
       "   'embedding_size_factor': 1.3,\n",
       "   'embed_exponent': 0.56,\n",
       "   'max_embedding_dim': 100,\n",
       "   'y_range': None,\n",
       "   'y_range_extend': 0.05,\n",
       "   'dropout_prob': 0.3110941443313673,\n",
       "   'optimizer': 'adam',\n",
       "   'learning_rate': 0.0002419459078909391,\n",
       "   'weight_decay': 2.7690674481100185e-06,\n",
       "   'proc.embed_min_categories': 100,\n",
       "   'proc.impute_strategy': 'most_frequent',\n",
       "   'proc.max_category_levels': 500,\n",
       "   'proc.skew_threshold': 10.0,\n",
       "   'use_ngram_features': False,\n",
       "   'num_layers': 3,\n",
       "   'hidden_size': 128,\n",
       "   'max_batch_size': 512,\n",
       "   'use_batchnorm': True,\n",
       "   'loss_function': 'auto'},\n",
       "  'NeuralNetTorch/d53001ac': {'num_epochs': 10,\n",
       "   'epochs_wo_improve': 20,\n",
       "   'activation': 'softrelu',\n",
       "   'embedding_size_factor': 1.2,\n",
       "   'embed_exponent': 0.56,\n",
       "   'max_embedding_dim': 100,\n",
       "   'y_range': None,\n",
       "   'y_range_extend': 0.05,\n",
       "   'dropout_prob': 0.17794680143342906,\n",
       "   'optimizer': 'adam',\n",
       "   'learning_rate': 0.0015781484866279445,\n",
       "   'weight_decay': 1.5770895023387964e-07,\n",
       "   'proc.embed_min_categories': 4,\n",
       "   'proc.impute_strategy': 'median',\n",
       "   'proc.max_category_levels': 500,\n",
       "   'proc.skew_threshold': 0.999,\n",
       "   'use_ngram_features': False,\n",
       "   'num_layers': 2,\n",
       "   'hidden_size': 512,\n",
       "   'max_batch_size': 512,\n",
       "   'use_batchnorm': False,\n",
       "   'loss_function': 'auto'},\n",
       "  'WeightedEnsemble_L2': {'use_orig_features': False,\n",
       "   'max_base_models': 25,\n",
       "   'max_base_models_per_type': 5,\n",
       "   'save_bag_folds': True}},\n",
       " 'leaderboard':                       model  score_val  pred_time_val  fit_time  \\\n",
       " 0       WeightedEnsemble_L2      0.405       0.052138  6.178108   \n",
       " 1               LightGBM/T3      0.375       0.004916  0.403158   \n",
       " 2               LightGBM/T5      0.375       0.006534  0.592385   \n",
       " 3               LightGBM/T1      0.370       0.004292  0.633591   \n",
       " 4               LightGBM/T4      0.360       0.009535  0.657517   \n",
       " 5   NeuralNetTorch/d528a1fa      0.360       0.019321  2.972190   \n",
       " 6               LightGBM/T2      0.355       0.005768  0.666110   \n",
       " 7   NeuralNetTorch/d3eb103e      0.355       0.020069  2.388131   \n",
       " 8   NeuralNetTorch/d53001ac      0.350       0.017180  1.537102   \n",
       " 9   NeuralNetTorch/d52be89c      0.350       0.018773  2.731708   \n",
       " 10  NeuralNetTorch/d526198a      0.320       0.016947  2.960038   \n",
       " \n",
       "     pred_time_val_marginal  fit_time_marginal  stack_level  can_infer  \\\n",
       " 0                 0.000443           0.280164            2       True   \n",
       " 1                 0.004916           0.403158            1       True   \n",
       " 2                 0.006534           0.592385            1       True   \n",
       " 3                 0.004292           0.633591            1       True   \n",
       " 4                 0.009535           0.657517            1       True   \n",
       " 5                 0.019321           2.972190            1       True   \n",
       " 6                 0.005768           0.666110            1       True   \n",
       " 7                 0.020069           2.388131            1       True   \n",
       " 8                 0.017180           1.537102            1       True   \n",
       " 9                 0.018773           2.731708            1       True   \n",
       " 10                0.016947           2.960038            1       True   \n",
       " \n",
       "     fit_order  \n",
       " 0          11  \n",
       " 1           3  \n",
       " 2           5  \n",
       " 3           1  \n",
       " 4           4  \n",
       " 5           8  \n",
       " 6           2  \n",
       " 7           6  \n",
       " 8          10  \n",
       " 9           9  \n",
       " 10          7  }"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.fit_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above example, the predictive performance may be poor because we are using few training data points and small ranges for hyperparameters to ensure quick run times. You can call `fit()` multiple times while modifying these settings to better understand how these choices affect performance outcomes. For example: you can increase `subsample_size` to train using a larger dataset, increase the `num_epochs` and `num_boost_round` hyperparameters, and increase the `time_limit` (which you should do for all code in these tutorials). To see more detailed output during the execution of `fit()`, you can also pass in the argument: `verbosity = 3`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## <a name=\"7\">Model ensembling with stacking/bagging</a>\n",
    "(<a href=\"#0\">Go to top</a>)\n",
    "\n",
    "Beyond hyperparameter-tuning with a correctly-specified evaluation metric, there are two other methods to boost predictive performance:\n",
    "- bagging and \n",
    "- stack-ensembling\n",
    "\n",
    "You’ll often see performance improve if you specify `num_bag_folds = 5-10`, `num_stack_levels = 1-3` in the call to `fit()`. Beware that doing this will increase training times and memory/disk usage.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No path specified. Models will be saved in: \"AutogluonModels/ag-20230207_190851/\"\n",
      "Beginning AutoGluon training ...\n",
      "AutoGluon will save models to \"AutogluonModels/ag-20230207_190851/\"\n",
      "AutoGluon Version:  0.6.2\n",
      "Python Version:     3.9.13\n",
      "Operating System:   Linux\n",
      "Platform Machine:   x86_64\n",
      "Platform Version:   #1 SMP Fri Oct 14 01:16:24 UTC 2022\n",
      "Train Data Rows:    1000\n",
      "Train Data Columns: 14\n",
      "Label Column: occupation\n",
      "Preprocessing data ...\n",
      "AutoGluon infers your prediction problem is: 'multiclass' (because dtype of label-column == object).\n",
      "\tFirst 10 (of 15) unique label values:  [' Exec-managerial', ' Other-service', ' Craft-repair', ' Sales', ' Prof-specialty', ' Protective-serv', ' ?', ' Adm-clerical', ' Machine-op-inspct', ' Tech-support']\n",
      "\tIf 'multiclass' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n",
      "Warning: Some classes in the training set have fewer than 10 examples. AutoGluon will only keep 13 out of 15 classes for training and will not try to predict the rare classes. To keep more classes, increase the number of datapoints from these rare classes in the training data or reduce label_count_threshold.\n",
      "Fraction of data from classes with at least 10 examples that will be kept for training models: 0.996\n",
      "Train Data Class Count: 13\n",
      "Using Feature Generators to preprocess the data ...\n",
      "Fitting AutoMLPipelineFeatureGenerator...\n",
      "\tAvailable Memory:                    13681.93 MB\n",
      "\tTrain Data (Original)  Memory Usage: 0.58 MB (0.0% of available memory)\n",
      "\tInferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.\n",
      "\tStage 1 Generators:\n",
      "\t\tFitting AsTypeFeatureGenerator...\n",
      "\t\t\tNote: Converting 2 features to boolean dtype as they only contain 2 unique values.\n",
      "\tStage 2 Generators:\n",
      "\t\tFitting FillNaFeatureGenerator...\n",
      "\tStage 3 Generators:\n",
      "\t\tFitting IdentityFeatureGenerator...\n",
      "\t\tFitting CategoryFeatureGenerator...\n",
      "\t\t\tFitting CategoryMemoryMinimizeFeatureGenerator...\n",
      "\tStage 4 Generators:\n",
      "\t\tFitting DropUniqueFeatureGenerator...\n",
      "\tTypes of features in original data (raw dtype, special dtypes):\n",
      "\t\t('int', [])    : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]\n",
      "\t\t('object', []) : 8 | ['workclass', 'education', 'marital-status', 'relationship', 'race', ...]\n",
      "\tTypes of features in processed data (raw dtype, special dtypes):\n",
      "\t\t('category', [])  : 6 | ['workclass', 'education', 'marital-status', 'relationship', 'race', ...]\n",
      "\t\t('int', [])       : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]\n",
      "\t\t('int', ['bool']) : 2 | ['sex', 'class']\n",
      "\t0.1s = Fit runtime\n",
      "\t14 features in original data used to generate 14 features in processed data.\n",
      "\tTrain Data (Processed) Memory Usage: 0.06 MB (0.0% of available memory)\n",
      "Data preprocessing and feature engineering runtime = 0.1s ...\n",
      "AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'\n",
      "\tTo change this, specify the eval_metric parameter of Predictor()\n",
      "AutoGluon will fit 2 stack levels (L1 to L2) ...\n",
      "Fitting 13 L1 models ...\n",
      "Fitting model: KNeighborsUnif_BAG_L1 ...\n",
      "\t0.1155\t = Validation score   (accuracy)\n",
      "\t0.01s\t = Training   runtime\n",
      "\t0.14s\t = Validation runtime\n",
      "Fitting model: KNeighborsDist_BAG_L1 ...\n",
      "\t0.0994\t = Validation score   (accuracy)\n",
      "\t0.0s\t = Training   runtime\n",
      "\t0.01s\t = Validation runtime\n",
      "Fitting model: NeuralNetFastAI_BAG_L1 ...\n",
      "\tFitting 5 child models (S1F1 - S1F5) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.3394\t = Validation score   (accuracy)\n",
      "\t9.22s\t = Training   runtime\n",
      "\t0.09s\t = Validation runtime\n",
      "Fitting model: LightGBMXT_BAG_L1 ...\n",
      "\tFitting 5 child models (S1F1 - S1F5) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.3805\t = Validation score   (accuracy)\n",
      "\t7.91s\t = Training   runtime\n",
      "\t0.15s\t = Validation runtime\n",
      "Fitting model: LightGBM_BAG_L1 ...\n",
      "\tFitting 5 child models (S1F1 - S1F5) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.3635\t = Validation score   (accuracy)\n",
      "\t8.09s\t = Training   runtime\n",
      "\t0.06s\t = Validation runtime\n",
      "Fitting model: RandomForestGini_BAG_L1 ...\n",
      "\t0.3183\t = Validation score   (accuracy)\n",
      "\t0.94s\t = Training   runtime\n",
      "\t0.16s\t = Validation runtime\n",
      "Fitting model: RandomForestEntr_BAG_L1 ...\n",
      "\t0.3183\t = Validation score   (accuracy)\n",
      "\t0.83s\t = Training   runtime\n",
      "\t0.14s\t = Validation runtime\n",
      "Fitting model: CatBoost_BAG_L1 ...\n",
      "\tFitting 5 child models (S1F1 - S1F5) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.3715\t = Validation score   (accuracy)\n",
      "\t195.67s\t = Training   runtime\n",
      "\t0.46s\t = Validation runtime\n",
      "Fitting model: ExtraTreesGini_BAG_L1 ...\n",
      "\t0.3032\t = Validation score   (accuracy)\n",
      "\t0.88s\t = Training   runtime\n",
      "\t0.14s\t = Validation runtime\n",
      "Fitting model: ExtraTreesEntr_BAG_L1 ...\n",
      "\t0.3062\t = Validation score   (accuracy)\n",
      "\t0.67s\t = Training   runtime\n",
      "\t0.14s\t = Validation runtime\n",
      "Fitting model: XGBoost_BAG_L1 ...\n",
      "\tFitting 5 child models (S1F1 - S1F5) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.3645\t = Validation score   (accuracy)\n",
      "\t6.63s\t = Training   runtime\n",
      "\t0.05s\t = Validation runtime\n",
      "Fitting model: NeuralNetTorch_BAG_L1 ...\n",
      "\tFitting 5 child models (S1F1 - S1F5) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.3795\t = Validation score   (accuracy)\n",
      "\t13.83s\t = Training   runtime\n",
      "\t0.12s\t = Validation runtime\n",
      "Fitting model: LightGBMLarge_BAG_L1 ...\n",
      "\tFitting 5 child models (S1F1 - S1F5) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.3042\t = Validation score   (accuracy)\n",
      "\t13.54s\t = Training   runtime\n",
      "\t0.08s\t = Validation runtime\n",
      "Fitting model: WeightedEnsemble_L2 ...\n",
      "\t0.3805\t = Validation score   (accuracy)\n",
      "\t0.61s\t = Training   runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "Fitting 11 L2 models ...\n",
      "Fitting model: NeuralNetFastAI_BAG_L2 ...\n",
      "\tFitting 5 child models (S1F1 - S1F5) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.3524\t = Validation score   (accuracy)\n",
      "\t9.07s\t = Training   runtime\n",
      "\t0.55s\t = Validation runtime\n",
      "Fitting model: LightGBMXT_BAG_L2 ...\n",
      "\tFitting 5 child models (S1F1 - S1F5) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.3675\t = Validation score   (accuracy)\n",
      "\t45.2s\t = Training   runtime\n",
      "\t0.27s\t = Validation runtime\n",
      "Fitting model: LightGBM_BAG_L2 ...\n",
      "\tFitting 5 child models (S1F1 - S1F5) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.3775\t = Validation score   (accuracy)\n",
      "\t80.56s\t = Training   runtime\n",
      "\t0.09s\t = Validation runtime\n",
      "Fitting model: RandomForestGini_BAG_L2 ...\n",
      "\t0.3584\t = Validation score   (accuracy)\n",
      "\t1.4s\t = Training   runtime\n",
      "\t0.16s\t = Validation runtime\n",
      "Fitting model: RandomForestEntr_BAG_L2 ...\n",
      "\t0.3635\t = Validation score   (accuracy)\n",
      "\t3.3s\t = Training   runtime\n",
      "\t0.16s\t = Validation runtime\n",
      "Fitting model: CatBoost_BAG_L2 ...\n",
      "\tFitting 5 child models (S1F1 - S1F5) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.3805\t = Validation score   (accuracy)\n",
      "\t498.02s\t = Training   runtime\n",
      "\t0.09s\t = Validation runtime\n",
      "Fitting model: ExtraTreesGini_BAG_L2 ...\n",
      "\t0.3544\t = Validation score   (accuracy)\n",
      "\t0.92s\t = Training   runtime\n",
      "\t0.17s\t = Validation runtime\n",
      "Fitting model: ExtraTreesEntr_BAG_L2 ...\n",
      "\t0.3594\t = Validation score   (accuracy)\n",
      "\t0.9s\t = Training   runtime\n",
      "\t0.17s\t = Validation runtime\n",
      "Fitting model: XGBoost_BAG_L2 ...\n",
      "\tFitting 5 child models (S1F1 - S1F5) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.3544\t = Validation score   (accuracy)\n",
      "\t83.73s\t = Training   runtime\n",
      "\t0.13s\t = Validation runtime\n",
      "Fitting model: NeuralNetTorch_BAG_L2 ...\n",
      "\tFitting 5 child models (S1F1 - S1F5) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.3705\t = Validation score   (accuracy)\n",
      "\t10.92s\t = Training   runtime\n",
      "\t0.92s\t = Validation runtime\n",
      "Fitting model: LightGBMLarge_BAG_L2 ...\n",
      "\tFitting 5 child models (S1F1 - S1F5) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.3695\t = Validation score   (accuracy)\n",
      "\t238.12s\t = Training   runtime\n",
      "\t0.45s\t = Validation runtime\n",
      "Fitting model: WeightedEnsemble_L3 ...\n",
      "\t0.3886\t = Validation score   (accuracy)\n",
      "\t0.53s\t = Training   runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "AutoGluon training complete, total runtime = 1282.26s ... Best model: \"WeightedEnsemble_L3\"\n",
      "TabularPredictor saved. To load, use: predictor = TabularPredictor.load(\"AutogluonModels/ag-20230207_190851/\")\n"
     ]
    }
   ],
   "source": [
    "predictor = TabularPredictor(label=\"occupation\", eval_metric=metric).fit(\n",
    "    train_data,\n",
    "    num_bag_folds=5,\n",
    "    num_bag_sets=1,\n",
    "    num_stack_levels=1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should not provide `tuning_data` when stacking/bagging, and instead provide all your available data as train_data (which AutoGluon will split in more intelligent ways). Parameter `num_bag_sets` controls how many times the K-fold bagging process is repeated to further reduce variance (increasing this may further boost accuracy but will substantially increase training times, inference latency, and memory/disk usage). Rather than manually searching for good bagging/stacking values yourself, AutoGluon will automatically select good values for you if you specify `auto_stack` instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Stack configuration (auto_stack=True): num_stack_levels=1, num_bag_folds=8, num_bag_sets=20\n",
      "Beginning AutoGluon training ... Time limit = 30s\n",
      "AutoGluon will save models to \"agModels-predictOccupation/\"\n",
      "AutoGluon Version:  0.6.2\n",
      "Python Version:     3.9.13\n",
      "Operating System:   Linux\n",
      "Platform Machine:   x86_64\n",
      "Platform Version:   #1 SMP Fri Oct 14 01:16:24 UTC 2022\n",
      "Train Data Rows:    1000\n",
      "Train Data Columns: 14\n",
      "Label Column: occupation\n",
      "Preprocessing data ...\n",
      "AutoGluon infers your prediction problem is: 'multiclass' (because dtype of label-column == object).\n",
      "\tFirst 10 (of 15) unique label values:  [' Exec-managerial', ' Other-service', ' Craft-repair', ' Sales', ' Prof-specialty', ' Protective-serv', ' ?', ' Adm-clerical', ' Machine-op-inspct', ' Tech-support']\n",
      "\tIf 'multiclass' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n",
      "Warning: Some classes in the training set have fewer than 10 examples. AutoGluon will only keep 13 out of 15 classes for training and will not try to predict the rare classes. To keep more classes, increase the number of datapoints from these rare classes in the training data or reduce label_count_threshold.\n",
      "Fraction of data from classes with at least 10 examples that will be kept for training models: 0.996\n",
      "Train Data Class Count: 13\n",
      "Using Feature Generators to preprocess the data ...\n",
      "Fitting AutoMLPipelineFeatureGenerator...\n",
      "\tAvailable Memory:                    12692.73 MB\n",
      "\tTrain Data (Original)  Memory Usage: 0.58 MB (0.0% of available memory)\n",
      "\tInferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.\n",
      "\tStage 1 Generators:\n",
      "\t\tFitting AsTypeFeatureGenerator...\n",
      "\t\t\tNote: Converting 2 features to boolean dtype as they only contain 2 unique values.\n",
      "\tStage 2 Generators:\n",
      "\t\tFitting FillNaFeatureGenerator...\n",
      "\tStage 3 Generators:\n",
      "\t\tFitting IdentityFeatureGenerator...\n",
      "\t\tFitting CategoryFeatureGenerator...\n",
      "\t\t\tFitting CategoryMemoryMinimizeFeatureGenerator...\n",
      "\tStage 4 Generators:\n",
      "\t\tFitting DropUniqueFeatureGenerator...\n",
      "\tTypes of features in original data (raw dtype, special dtypes):\n",
      "\t\t('int', [])    : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]\n",
      "\t\t('object', []) : 8 | ['workclass', 'education', 'marital-status', 'relationship', 'race', ...]\n",
      "\tTypes of features in processed data (raw dtype, special dtypes):\n",
      "\t\t('category', [])  : 6 | ['workclass', 'education', 'marital-status', 'relationship', 'race', ...]\n",
      "\t\t('int', [])       : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]\n",
      "\t\t('int', ['bool']) : 2 | ['sex', 'class']\n",
      "\t0.1s = Fit runtime\n",
      "\t14 features in original data used to generate 14 features in processed data.\n",
      "\tTrain Data (Processed) Memory Usage: 0.06 MB (0.0% of available memory)\n",
      "Data preprocessing and feature engineering runtime = 0.13s ...\n",
      "AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'\n",
      "\tTo change this, specify the eval_metric parameter of Predictor()\n",
      "AutoGluon will fit 2 stack levels (L1 to L2) ...\n",
      "Fitting 13 L1 models ...\n",
      "Fitting model: KNeighborsUnif_BAG_L1 ... Training model for up to 19.91s of the 29.87s of remaining time.\n",
      "\t0.1155\t = Validation score   (accuracy)\n",
      "\t0.01s\t = Training   runtime\n",
      "\t0.01s\t = Validation runtime\n",
      "Fitting model: KNeighborsDist_BAG_L1 ... Training model for up to 19.88s of the 29.84s of remaining time.\n",
      "\t0.0994\t = Validation score   (accuracy)\n",
      "\t0.01s\t = Training   runtime\n",
      "\t0.01s\t = Validation runtime\n",
      "Fitting model: NeuralNetFastAI_BAG_L1 ... Training model for up to 19.86s of the 29.81s of remaining time.\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.3544\t = Validation score   (accuracy)\n",
      "\t11.52s\t = Training   runtime\n",
      "\t0.14s\t = Validation runtime\n",
      "Fitting model: LightGBMXT_BAG_L1 ... Training model for up to 5.21s of the 15.17s of remaining time.\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.3876\t = Validation score   (accuracy)\n",
      "\t9.12s\t = Training   runtime\n",
      "\t0.09s\t = Validation runtime\n",
      "Completed 1/20 k-fold bagging repeats ...\n",
      "Fitting model: WeightedEnsemble_L2 ... Training model for up to 29.87s of the 2.17s of remaining time.\n",
      "\t0.3876\t = Validation score   (accuracy)\n",
      "\t0.39s\t = Training   runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "Fitting 11 L2 models ...\n",
      "Fitting model: NeuralNetFastAI_BAG_L2 ... Training model for up to 1.76s of the 1.74s of remaining time.\n",
      "\tFitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy\n",
      "\t0.3394\t = Validation score   (accuracy)\n",
      "\t9.53s\t = Training   runtime\n",
      "\t0.39s\t = Validation runtime\n",
      "Completed 1/20 k-fold bagging repeats ...\n",
      "Fitting model: WeightedEnsemble_L3 ... Training model for up to 29.87s of the -11.07s of remaining time.\n",
      "\t0.3394\t = Validation score   (accuracy)\n",
      "\t0.0s\t = Training   runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "AutoGluon training complete, total runtime = 41.09s ... Best model: \"WeightedEnsemble_L2\"\n",
      "TabularPredictor saved. To load, use: predictor = TabularPredictor.load(\"agModels-predictOccupation/\")\n"
     ]
    }
   ],
   "source": [
    "# Folder where to store trained models\n",
    "save_path = \"agModels-predictOccupation\"\n",
    "\n",
    "predictor = TabularPredictor(label=\"occupation\", eval_metric=metric, path=save_path).fit(\n",
    "    train_data,\n",
    "    auto_stack=True,\n",
    "    time_limit=30\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often stacking/bagging will produce superior accuracy than hyperparameter-tuning, but you may try combining both techniques (note: specifying `presets='best_quality'` in `fit()` simply sets `auto_stack=True`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## <a name=\"8\">Prediction options (inference)</a>\n",
    "(<a href=\"#0\">Go to top</a>)\n",
    "\n",
    "Even if you’ve started a new Python session since last calling `fit()`, you can still load a previously trained predictor from disk:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# `predictor.path` is another way to get the relative path needed to later load predictor.\n",
    "predictor = TabularPredictor.load(save_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above `save_path` is the same folder previously passed to `TabularPredictor`, in which all the trained models have been saved. You can train easily models on one machine and deploy them on another. Simply copy the `save_path` folder to the new machine and specify its new path in `TabularPredictor.load()`.\n",
    "\n",
    "We can make a prediction on an individual example rather than on a full dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     Other-service\n",
       "Name: occupation, dtype: object"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Select one datapoint to make a prediction\n",
    "datapoint = test_data.iloc[[0]] # Note: .iloc[0] won't work because it returns pandas Series instead of DataFrame\n",
    "\n",
    "predictor.predict(datapoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To output predicted class probabilities instead of predicted classes, you can use:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>?</th>\n",
       "      <th>Adm-clerical</th>\n",
       "      <th>Armed-Forces</th>\n",
       "      <th>Craft-repair</th>\n",
       "      <th>Exec-managerial</th>\n",
       "      <th>Farming-fishing</th>\n",
       "      <th>Handlers-cleaners</th>\n",
       "      <th>Machine-op-inspct</th>\n",
       "      <th>Other-service</th>\n",
       "      <th>Priv-house-serv</th>\n",
       "      <th>Prof-specialty</th>\n",
       "      <th>Protective-serv</th>\n",
       "      <th>Sales</th>\n",
       "      <th>Tech-support</th>\n",
       "      <th>Transport-moving</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.038548</td>\n",
       "      <td>0.234544</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.050518</td>\n",
       "      <td>0.06077</td>\n",
       "      <td>0.008894</td>\n",
       "      <td>0.073112</td>\n",
       "      <td>0.048278</td>\n",
       "      <td>0.298018</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.049102</td>\n",
       "      <td>0.004876</td>\n",
       "      <td>0.085449</td>\n",
       "      <td>0.021308</td>\n",
       "      <td>0.026581</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          ?   Adm-clerical   Armed-Forces   Craft-repair   Exec-managerial  \\\n",
       "0  0.038548       0.234544            0.0       0.050518           0.06077   \n",
       "\n",
       "    Farming-fishing   Handlers-cleaners   Machine-op-inspct   Other-service  \\\n",
       "0          0.008894            0.073112            0.048278        0.298018   \n",
       "\n",
       "    Priv-house-serv   Prof-specialty   Protective-serv     Sales  \\\n",
       "0               0.0         0.049102          0.004876  0.085449   \n",
       "\n",
       "    Tech-support   Transport-moving  \n",
       "0       0.021308           0.026581  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Returns a DataFrame that shows which probability corresponds to which class\n",
    "predictor.predict_proba(datapoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, `predict()` and `predict_proba()` will utilize the model that AutoGluon thinks is most accurate, which is usually an ensemble of many individual models. Here’s how to see which model this corresponds to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'WeightedEnsemble_L2'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.get_model_best()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## <a name=\"10\">Selecting individual models for predictions</a>\n",
    "(<a href=\"#0\">Go to top</a>)\n",
    "\n",
    "We can specify a particular model to use for predictions (e.g. to reduce inference latency). Note that a ‘model’ in AutoGluon may refer to for example a single Neural Network, a bagged ensemble of many Neural Network copies trained on different training/validation splits, a weighted ensemble that aggregates the predictions of many other models, or a stacked model that operates on predictions output by other models. This is akin to viewing a RandomForest as one ‘model’ when it is in fact an ensemble of many decision trees.\n",
    "\n",
    "\n",
    "Here’s how to specify a particular model to use for prediction instead of AutoGluon’s default model-choice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction from KNeighborsUnif_BAG_L1 model:  Adm-clerical\n"
     ]
    }
   ],
   "source": [
    "# index of model to use\n",
    "i = 0\n",
    "model_to_use = predictor.get_model_names()[i]\n",
    "model_pred = predictor.predict(datapoint, model=model_to_use)\n",
    "print(f\"Prediction from {model_to_use} model: {model_pred.iloc[0]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can easily access information about the trained predictor or a particular model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_models = predictor.get_model_names()\n",
    "model_to_use = all_models[i]\n",
    "specific_model = predictor._trainer.load_model(model_to_use)\n",
    "\n",
    "# Objects defined below are dicts with information (not printed here as they are quite large):\n",
    "model_info = specific_model.get_info()\n",
    "predictor_information = predictor.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the label columns remains in the `test_data` DataFrame, we can instead use the shorthand:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Evaluation: accuracy on test data: 0.35438632408639575\n",
      "Evaluations on test data:\n",
      "{\n",
      "    \"accuracy\": 0.35438632408639575,\n",
      "    \"balanced_accuracy\": 0.2380254766456843,\n",
      "    \"mcc\": 0.27835195719142447\n",
      "}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.35438632408639575,\n",
       " 'balanced_accuracy': 0.2380254766456843,\n",
       " 'mcc': 0.27835195719142447}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.evaluate(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## <a name=\"11\">Interpretability: Feature importance</a>\n",
    "(<a href=\"#0\">Go to top</a>)\n",
    "\n",
    "To better understand our trained predictor, we can estimate the overall importance of each feature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Computing feature importance via permutation shuffling for 14 features using 5000 rows with 5 shuffle sets...\n",
      "\t58.83s\t= Expected runtime (11.77s per shuffle set)\n",
      "\t49.9s\t= Actual runtime (Completed 5 of 5 shuffle sets)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>importance</th>\n",
       "      <th>stddev</th>\n",
       "      <th>p_value</th>\n",
       "      <th>n</th>\n",
       "      <th>p99_high</th>\n",
       "      <th>p99_low</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>workclass</th>\n",
       "      <td>0.07032</td>\n",
       "      <td>0.002331</td>\n",
       "      <td>1.445937e-07</td>\n",
       "      <td>5</td>\n",
       "      <td>0.075119</td>\n",
       "      <td>0.065521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sex</th>\n",
       "      <td>0.06248</td>\n",
       "      <td>0.005137</td>\n",
       "      <td>5.435720e-06</td>\n",
       "      <td>5</td>\n",
       "      <td>0.073058</td>\n",
       "      <td>0.051902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>education-num</th>\n",
       "      <td>0.05004</td>\n",
       "      <td>0.006349</td>\n",
       "      <td>3.043899e-05</td>\n",
       "      <td>5</td>\n",
       "      <td>0.063112</td>\n",
       "      <td>0.036968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hours-per-week</th>\n",
       "      <td>0.02208</td>\n",
       "      <td>0.006774</td>\n",
       "      <td>9.419456e-04</td>\n",
       "      <td>5</td>\n",
       "      <td>0.036029</td>\n",
       "      <td>0.008131</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>class</th>\n",
       "      <td>0.02004</td>\n",
       "      <td>0.003122</td>\n",
       "      <td>6.846919e-05</td>\n",
       "      <td>5</td>\n",
       "      <td>0.026469</td>\n",
       "      <td>0.013611</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>education</th>\n",
       "      <td>0.01792</td>\n",
       "      <td>0.003087</td>\n",
       "      <td>1.016721e-04</td>\n",
       "      <td>5</td>\n",
       "      <td>0.024277</td>\n",
       "      <td>0.011563</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>0.00752</td>\n",
       "      <td>0.003466</td>\n",
       "      <td>4.164539e-03</td>\n",
       "      <td>5</td>\n",
       "      <td>0.014656</td>\n",
       "      <td>0.000384</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>relationship</th>\n",
       "      <td>0.00144</td>\n",
       "      <td>0.002609</td>\n",
       "      <td>1.423639e-01</td>\n",
       "      <td>5</td>\n",
       "      <td>0.006812</td>\n",
       "      <td>-0.003932</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fnlwgt</th>\n",
       "      <td>0.00116</td>\n",
       "      <td>0.001519</td>\n",
       "      <td>8.147191e-02</td>\n",
       "      <td>5</td>\n",
       "      <td>0.004288</td>\n",
       "      <td>-0.001968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>race</th>\n",
       "      <td>0.00048</td>\n",
       "      <td>0.001361</td>\n",
       "      <td>2.372046e-01</td>\n",
       "      <td>5</td>\n",
       "      <td>0.003282</td>\n",
       "      <td>-0.002322</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>native-country</th>\n",
       "      <td>0.00024</td>\n",
       "      <td>0.000942</td>\n",
       "      <td>2.997389e-01</td>\n",
       "      <td>5</td>\n",
       "      <td>0.002180</td>\n",
       "      <td>-0.001700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>marital-status</th>\n",
       "      <td>-0.00044</td>\n",
       "      <td>0.000740</td>\n",
       "      <td>8.727166e-01</td>\n",
       "      <td>5</td>\n",
       "      <td>0.001084</td>\n",
       "      <td>-0.001964</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>capital-gain</th>\n",
       "      <td>-0.00084</td>\n",
       "      <td>0.001187</td>\n",
       "      <td>9.056980e-01</td>\n",
       "      <td>5</td>\n",
       "      <td>0.001603</td>\n",
       "      <td>-0.003283</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>capital-loss</th>\n",
       "      <td>-0.00104</td>\n",
       "      <td>0.000607</td>\n",
       "      <td>9.907187e-01</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000209</td>\n",
       "      <td>-0.002289</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                importance    stddev       p_value  n  p99_high   p99_low\n",
       "workclass          0.07032  0.002331  1.445937e-07  5  0.075119  0.065521\n",
       "sex                0.06248  0.005137  5.435720e-06  5  0.073058  0.051902\n",
       "education-num      0.05004  0.006349  3.043899e-05  5  0.063112  0.036968\n",
       "hours-per-week     0.02208  0.006774  9.419456e-04  5  0.036029  0.008131\n",
       "class              0.02004  0.003122  6.846919e-05  5  0.026469  0.013611\n",
       "education          0.01792  0.003087  1.016721e-04  5  0.024277  0.011563\n",
       "age                0.00752  0.003466  4.164539e-03  5  0.014656  0.000384\n",
       "relationship       0.00144  0.002609  1.423639e-01  5  0.006812 -0.003932\n",
       "fnlwgt             0.00116  0.001519  8.147191e-02  5  0.004288 -0.001968\n",
       "race               0.00048  0.001361  2.372046e-01  5  0.003282 -0.002322\n",
       "native-country     0.00024  0.000942  2.997389e-01  5  0.002180 -0.001700\n",
       "marital-status    -0.00044  0.000740  8.727166e-01  5  0.001084 -0.001964\n",
       "capital-gain      -0.00084  0.001187  9.056980e-01  5  0.001603 -0.003283\n",
       "capital-loss      -0.00104  0.000607  9.907187e-01  5  0.000209 -0.002289"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.feature_importance(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Computed via permutation-shuffling, these feature importance scores quantify the drop in predictive performance (of the already trained predictor) when one columns values are randomly shuffled across rows. The top features in this list contribute most to AutoGluon’s accuracy. Features with non-positive importance score hardly contribute to the predictors accuracy, or may even be actively harmful to include in the data (consider removing these features from your data and calling `fit` again). These scores facilitate interpretability of the predictors global behavior (which features it relies on for all predictions) rather than local explanations that only rationalize one particular prediction.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## <a name=\"12\"> Inference Speed: Model distillation</a>\n",
    "(<a href=\"#0\">Go to top</a>)\n",
    "\n",
    "While computationally-favorable, single individual models will usually have lower accuracy than weighted/stacked/bagged ensembles. Model Distillation offers one way to retain the computational benefits of a single model, while enjoying some of the accuracy-boost that comes with ensembling. The idea is to train the individual model (which we can call the student) to mimic the predictions of the full stack ensemble (the teacher). Like `refit_full()`, the `distill()` function will produce additional models we can opt to use for prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training student models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Distilling with teacher='WeightedEnsemble_L2', teacher_preds=soft, augment_method=spunge ...\n",
      "SPUNGE: Augmenting training data with 3980 synthetic samples for distillation...\n",
      "Distilling with each of these student models: ['LightGBM_DSTL', 'NeuralNetMXNet_DSTL', 'RandomForestMSE_DSTL', 'CatBoost_DSTL', 'NeuralNetTorch_DSTL']\n",
      "Fitting 5 L1 models ...\n",
      "Fitting model: LightGBM_DSTL ... Training model for up to 30.0s of the 30.0s of remaining time.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1000]\tvalid_set's soft_log_loss: -1.67598\n",
      "[2000]\tvalid_set's soft_log_loss: -1.67213\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tRan out of time, early stopping on iteration 2666. Best iteration is:\n",
      "\t[2112]\tvalid_set's soft_log_loss: -1.67157\n",
      "\tNote: model has different eval_metric than default.\n",
      "\t-1.6716\t = Validation score   (-soft_log_loss)\n",
      "\t33.73s\t = Training   runtime\n",
      "\t0.4s\t = Validation runtime\n",
      "Completed 1/20 k-fold bagging repeats ...\n",
      "Distilling with each of these student models: ['WeightedEnsemble_L2_DSTL']\n",
      "Fitting model: WeightedEnsemble_L2_DSTL ... Training model for up to 30.0s of the -7.34s of remaining time.\n",
      "\tNote: model has different eval_metric than default.\n",
      "\t-1.6716\t = Validation score   (-soft_log_loss)\n",
      "\t0.0s\t = Training   runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "Distilled model leaderboard:\n",
      "                      model  score_val  pred_time_val   fit_time  pred_time_val_marginal  fit_time_marginal  stack_level  can_infer  fit_order\n",
      "0             LightGBM_DSTL      0.425       0.397912  33.728100                0.397912          33.728100            1       True          8\n",
      "1  WeightedEnsemble_L2_DSTL      0.425       0.398333  33.732611                0.000421           0.004511            2       True          9\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['LightGBM_DSTL', 'WeightedEnsemble_L2_DSTL']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Specify much longer time limit in real applications\n",
    "student_models = predictor.distill(time_limit=30)\n",
    "student_models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predictions from LightGBM_DSTL: [' Adm-clerical', ' Farming-fishing', ' Sales', ' Sales', ' Handlers-cleaners']\n"
     ]
    }
   ],
   "source": [
    "preds_student = predictor.predict(test_data, model=student_models[0])\n",
    "print(f\"predictions from {student_models[0]}: {list(preds_student)[:5]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Excluding models\n",
    "\n",
    "Finally, you may also exclude specific unwieldy models from being trained at all. Below we exclude models that tend to be slower (K Nearest Neighbors, Neural Network, models with custom larger-than-default hyperparameters):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No path specified. Models will be saved in: \"AutogluonModels/ag-20230207_193310/\"\n",
      "Beginning AutoGluon training ... Time limit = 30s\n",
      "AutoGluon will save models to \"AutogluonModels/ag-20230207_193310/\"\n",
      "AutoGluon Version:  0.6.2\n",
      "Python Version:     3.9.13\n",
      "Operating System:   Linux\n",
      "Platform Machine:   x86_64\n",
      "Platform Version:   #1 SMP Fri Oct 14 01:16:24 UTC 2022\n",
      "Train Data Rows:    1000\n",
      "Train Data Columns: 14\n",
      "Label Column: occupation\n",
      "Preprocessing data ...\n",
      "AutoGluon infers your prediction problem is: 'multiclass' (because dtype of label-column == object).\n",
      "\tFirst 10 (of 15) unique label values:  [' Exec-managerial', ' Other-service', ' Craft-repair', ' Sales', ' Prof-specialty', ' Protective-serv', ' ?', ' Adm-clerical', ' Machine-op-inspct', ' Tech-support']\n",
      "\tIf 'multiclass' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])\n",
      "Warning: Some classes in the training set have fewer than 10 examples. AutoGluon will only keep 13 out of 15 classes for training and will not try to predict the rare classes. To keep more classes, increase the number of datapoints from these rare classes in the training data or reduce label_count_threshold.\n",
      "Fraction of data from classes with at least 10 examples that will be kept for training models: 0.996\n",
      "Train Data Class Count: 13\n",
      "Using Feature Generators to preprocess the data ...\n",
      "Fitting AutoMLPipelineFeatureGenerator...\n",
      "\tAvailable Memory:                    12879.65 MB\n",
      "\tTrain Data (Original)  Memory Usage: 0.58 MB (0.0% of available memory)\n",
      "\tInferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.\n",
      "\tStage 1 Generators:\n",
      "\t\tFitting AsTypeFeatureGenerator...\n",
      "\t\t\tNote: Converting 2 features to boolean dtype as they only contain 2 unique values.\n",
      "\tStage 2 Generators:\n",
      "\t\tFitting FillNaFeatureGenerator...\n",
      "\tStage 3 Generators:\n",
      "\t\tFitting IdentityFeatureGenerator...\n",
      "\t\tFitting CategoryFeatureGenerator...\n",
      "\t\t\tFitting CategoryMemoryMinimizeFeatureGenerator...\n",
      "\tStage 4 Generators:\n",
      "\t\tFitting DropUniqueFeatureGenerator...\n",
      "\tTypes of features in original data (raw dtype, special dtypes):\n",
      "\t\t('int', [])    : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]\n",
      "\t\t('object', []) : 8 | ['workclass', 'education', 'marital-status', 'relationship', 'race', ...]\n",
      "\tTypes of features in processed data (raw dtype, special dtypes):\n",
      "\t\t('category', [])  : 6 | ['workclass', 'education', 'marital-status', 'relationship', 'race', ...]\n",
      "\t\t('int', [])       : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]\n",
      "\t\t('int', ['bool']) : 2 | ['sex', 'class']\n",
      "\t0.1s = Fit runtime\n",
      "\t14 features in original data used to generate 14 features in processed data.\n",
      "\tTrain Data (Processed) Memory Usage: 0.06 MB (0.0% of available memory)\n",
      "Data preprocessing and feature engineering runtime = 0.1s ...\n",
      "AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'\n",
      "\tTo change this, specify the eval_metric parameter of Predictor()\n",
      "Automatically generating train/validation split with holdout_frac=0.2, Train Rows: 796, Val Rows: 200\n",
      "Excluded Model Types: ['KNN', 'NN', 'custom']\n",
      "\tFound 'KNN' model in hyperparameters, but 'KNN' is present in `excluded_model_types` and will be removed.\n",
      "\tFound 'KNN' model in hyperparameters, but 'KNN' is present in `excluded_model_types` and will be removed.\n",
      "Fitting 11 L1 models ...\n",
      "Fitting model: NeuralNetFastAI ... Training model for up to 29.9s of the 29.9s of remaining time.\n",
      "\t0.37\t = Validation score   (accuracy)\n",
      "\t1.15s\t = Training   runtime\n",
      "\t0.01s\t = Validation runtime\n",
      "Fitting model: LightGBMXT ... Training model for up to 28.72s of the 28.72s of remaining time.\n",
      "\t0.385\t = Validation score   (accuracy)\n",
      "\t1.33s\t = Training   runtime\n",
      "\t0.01s\t = Validation runtime\n",
      "Fitting model: LightGBM ... Training model for up to 27.29s of the 27.29s of remaining time.\n",
      "\t0.37\t = Validation score   (accuracy)\n",
      "\t1.74s\t = Training   runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "Fitting model: RandomForestGini ... Training model for up to 25.54s of the 25.53s of remaining time.\n",
      "\t0.36\t = Validation score   (accuracy)\n",
      "\t0.75s\t = Training   runtime\n",
      "\t0.08s\t = Validation runtime\n",
      "Fitting model: RandomForestEntr ... Training model for up to 24.67s of the 24.67s of remaining time.\n",
      "\t0.35\t = Validation score   (accuracy)\n",
      "\t0.79s\t = Training   runtime\n",
      "\t0.08s\t = Validation runtime\n",
      "Fitting model: CatBoost ... Training model for up to 23.74s of the 23.73s of remaining time.\n",
      "\tRan out of time, early stopping on iteration 286.\n",
      "\t0.39\t = Validation score   (accuracy)\n",
      "\t23.6s\t = Training   runtime\n",
      "\t0.01s\t = Validation runtime\n",
      "Fitting model: ExtraTreesGini ... Training model for up to 0.13s of the 0.13s of remaining time.\n",
      "\tWarning: Reducing model 'n_estimators' from 300 -> 41 due to low time. Expected time usage reduced from 0.9s -> 0.1s...\n",
      "\t0.3\t = Validation score   (accuracy)\n",
      "\t0.13s\t = Training   runtime\n",
      "\t0.02s\t = Validation runtime\n",
      "Fitting model: WeightedEnsemble_L2 ... Training model for up to 29.9s of the -0.05s of remaining time.\n",
      "\t0.41\t = Validation score   (accuracy)\n",
      "\t0.21s\t = Training   runtime\n",
      "\t0.0s\t = Validation runtime\n",
      "AutoGluon training complete, total runtime = 30.28s ... Best model: \"WeightedEnsemble_L2\"\n",
      "TabularPredictor saved. To load, use: predictor = TabularPredictor.load(\"AutogluonModels/ag-20230207_193310/\")\n"
     ]
    }
   ],
   "source": [
    "excluded_model_types = [\"KNN\", \"NN\", \"custom\"]\n",
    "predictor_light = TabularPredictor(label=\"occupation\", eval_metric=metric).fit(\n",
    "    train_data, excluded_model_types=excluded_model_types, time_limit=30\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## <a name=\"13\">Before You Go</a>\n",
    "(<a href=\"#0\">Go to top</a>)\n",
    "\n",
    "After you are done with this Demo, clean model artifacts by uncommenting and executing the cell below.\n",
    "\n",
    "__It is always good practice to clean everything when you are done, preventing the disk from getting full.__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -r AutogluonModels\n",
    "!rm -r agModels-predictOccupation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"padding: 10px; border: 1px solid black;\">\n",
    "<img src=\"./../../images/MLU-NEW-logo.png\" alt=\"drawing\" width=\"400\"/> <br/>\n",
    "\n",
    "# Thank you!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}