exercise-xgboost.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5f6e94fa",
   "metadata": {
    "papermill": {
     "duration": 0.0154,
     "end_time": "2021-11-14T14:09:07.918967",
     "exception": false,
     "start_time": "2021-11-14T14:09:07.903567",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**This notebook is an exercise in the [Intermediate Machine Learning](https://www.kaggle.com/learn/intermediate-machine-learning) course.  You can reference the tutorial at [this link](https://www.kaggle.com/alexisbcook/xgboost).**\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae1548c8",
   "metadata": {
    "papermill": {
     "duration": 0.01525,
     "end_time": "2021-11-14T14:09:07.949692",
     "exception": false,
     "start_time": "2021-11-14T14:09:07.934442",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In this exercise, you will use your new knowledge to train a model with **gradient boosting**.\n",
    "\n",
    "# Setup\n",
    "\n",
    "The questions below will give you feedback on your work. Run the following cell to set up the feedback system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "65a24066",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-14T14:09:07.985178Z",
     "iopub.status.busy": "2021-11-14T14:09:07.983975Z",
     "iopub.status.idle": "2021-11-14T14:09:10.623893Z",
     "shell.execute_reply": "2021-11-14T14:09:10.622721Z",
     "shell.execute_reply.started": "2021-11-14T13:39:04.417653Z"
    },
    "papermill": {
     "duration": 2.657932,
     "end_time": "2021-11-14T14:09:10.624142",
     "exception": false,
     "start_time": "2021-11-14T14:09:07.966210",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setup Complete\n"
     ]
    }
   ],
   "source": [
    "# Set up code checking\n",
    "import os\n",
    "if not os.path.exists(\"../input/train.csv\"):\n",
    "    os.symlink(\"../input/home-data-for-ml-course/train.csv\", \"../input/train.csv\")  \n",
    "    os.symlink(\"../input/home-data-for-ml-course/test.csv\", \"../input/test.csv\") \n",
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.ml_intermediate.ex6 import *\n",
    "print(\"Setup Complete\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ffeb13c",
   "metadata": {
    "papermill": {
     "duration": 0.016259,
     "end_time": "2021-11-14T14:09:10.656856",
     "exception": false,
     "start_time": "2021-11-14T14:09:10.640597",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You will work with the [Housing Prices Competition for Kaggle Learn Users](https://www.kaggle.com/c/home-data-for-ml-course) dataset from the previous exercise. \n",
    "\n",
    "![Ames Housing dataset image](https://i.imgur.com/lTJVG4e.png)\n",
    "\n",
    "Run the next code cell without changes to load the training and validation sets in `X_train`, `X_valid`, `y_train`, and `y_valid`.  The test set is loaded in `X_test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f7f27181",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-14T14:09:10.696103Z",
     "iopub.status.busy": "2021-11-14T14:09:10.695120Z",
     "iopub.status.idle": "2021-11-14T14:09:10.698326Z",
     "shell.execute_reply": "2021-11-14T14:09:10.697764Z",
     "shell.execute_reply.started": "2021-11-14T13:39:09.498016Z"
    },
    "papermill": {
     "duration": 0.025769,
     "end_time": "2021-11-14T14:09:10.698499",
     "exception": false,
     "start_time": "2021-11-14T14:09:10.672730",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# # Read the data\n",
    "# X = pd.read_csv('../input/train.csv', index_col='Id')\n",
    "# X_test_full = pd.read_csv('../input/test.csv', index_col='Id')\n",
    "\n",
    "# # Remove rows with missing target, separate target from predictors\n",
    "# X.dropna(axis=0, subset=['SalePrice'], inplace=True)\n",
    "# y = X.SalePrice              \n",
    "# X.drop(['SalePrice'], axis=1, inplace=True)\n",
    "\n",
    "# # Break off validation set from training data\n",
    "# X_train_full, X_valid_full, y_train, y_valid = train_test_split(X, y, train_size=0.8, test_size=0.2,\n",
    "#                                                                 random_state=0)\n",
    "\n",
    "# # \"Cardinality\" means the number of unique values in a column\n",
    "# # Select categorical columns with relatively low cardinality (convenient but arbitrary)\n",
    "# low_cardinality_cols = [cname for cname in X_train_full.columns if X_train_full[cname].nunique() < 10 and \n",
    "#                         X_train_full[cname].dtype == \"object\"]\n",
    "\n",
    "# # Select numeric columns\n",
    "# numeric_cols = [cname for cname in X_train_full.columns if X_train_full[cname].dtype in ['int64', 'float64']]\n",
    "\n",
    "# # Keep selected columns only\n",
    "# my_cols = low_cardinality_cols + numeric_cols\n",
    "# X_train = X_train_full[my_cols].copy()\n",
    "# X_valid = X_valid_full[my_cols].copy()\n",
    "# X_test = X_test_full[my_cols].copy()\n",
    "\n",
    "# # One-hot encode the data (to shorten the code, we use pandas)\n",
    "# X_train = pd.get_dummies(X_train)\n",
    "# X_valid = pd.get_dummies(X_valid)\n",
    "# X_test = pd.get_dummies(X_test)\n",
    "# X_train, X_valid = X_train.align(X_valid, join='left', axis=1)\n",
    "# X_train, X_test = X_train.align(X_test, join='left', axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3d2cfb6",
   "metadata": {
    "papermill": {
     "duration": 0.016405,
     "end_time": "2021-11-14T14:09:10.731190",
     "exception": false,
     "start_time": "2021-11-14T14:09:10.714785",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Step 1: Build model\n",
    "\n",
    "### Part A\n",
    "\n",
    "In this step, you'll build and train your first model with gradient boosting.\n",
    "\n",
    "- Begin by setting `my_model_1` to an XGBoost model.  Use the [XGBRegressor](https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBRegressor) class, and set the random seed to 0 (`random_state=0`).  **Leave all other parameters as default.**\n",
    "- Then, fit the model to the training data in `X_train` and `y_train`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e39a8b1e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-14T14:09:10.771879Z",
     "iopub.status.busy": "2021-11-14T14:09:10.770167Z",
     "iopub.status.idle": "2021-11-14T14:09:10.774083Z",
     "shell.execute_reply": "2021-11-14T14:09:10.774834Z",
     "shell.execute_reply.started": "2021-11-14T13:41:11.271138Z"
    },
    "papermill": {
     "duration": 0.026256,
     "end_time": "2021-11-14T14:09:10.775043",
     "exception": false,
     "start_time": "2021-11-14T14:09:10.748787",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from xgboost import XGBRegressor\n",
    "\n",
    "# # Define the model\n",
    "# my_model_1 = XGBRegressor(random_state=0)\n",
    "\n",
    "# # Fit the model\n",
    "# my_model_1.fit(X_tr)\n",
    "\n",
    "# # Check your answer\n",
    "# step_1.a.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f9b6773b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-14T14:09:10.813209Z",
     "iopub.status.busy": "2021-11-14T14:09:10.812150Z",
     "iopub.status.idle": "2021-11-14T14:09:10.815369Z",
     "shell.execute_reply": "2021-11-14T14:09:10.815903Z"
    },
    "papermill": {
     "duration": 0.025107,
     "end_time": "2021-11-14T14:09:10.816126",
     "exception": false,
     "start_time": "2021-11-14T14:09:10.791019",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_1.a.hint()\n",
    "#step_1.a.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcc67a11",
   "metadata": {
    "papermill": {
     "duration": 0.015024,
     "end_time": "2021-11-14T14:09:10.848596",
     "exception": false,
     "start_time": "2021-11-14T14:09:10.833572",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Part B\n",
    "\n",
    "Set `predictions_1` to the model's predictions for the validation data.  Recall that the validation features are stored in `X_valid`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5f62776a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-14T14:09:10.886187Z",
     "iopub.status.busy": "2021-11-14T14:09:10.885257Z",
     "iopub.status.idle": "2021-11-14T14:09:10.888654Z",
     "shell.execute_reply": "2021-11-14T14:09:10.887934Z",
     "shell.execute_reply.started": "2021-11-14T13:41:39.173686Z"
    },
    "papermill": {
     "duration": 0.023264,
     "end_time": "2021-11-14T14:09:10.888825",
     "exception": false,
     "start_time": "2021-11-14T14:09:10.865561",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_absolute_error\n",
    "\n",
    "# # Get predictions\n",
    "# predictions_1 = my_model_1.predict(X_valid) # Your code here\n",
    "\n",
    "# # Check your answer\n",
    "# step_1.b.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4cc37eea",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-14T14:09:10.927335Z",
     "iopub.status.busy": "2021-11-14T14:09:10.926528Z",
     "iopub.status.idle": "2021-11-14T14:09:10.930192Z",
     "shell.execute_reply": "2021-11-14T14:09:10.929630Z"
    },
    "papermill": {
     "duration": 0.023419,
     "end_time": "2021-11-14T14:09:10.930340",
     "exception": false,
     "start_time": "2021-11-14T14:09:10.906921",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_1.b.hint()\n",
    "#step_1.b.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "753cc313",
   "metadata": {
    "papermill": {
     "duration": 0.015413,
     "end_time": "2021-11-14T14:09:10.963235",
     "exception": false,
     "start_time": "2021-11-14T14:09:10.947822",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Part C\n",
    "\n",
    "Finally, use the `mean_absolute_error()` function to calculate the mean absolute error (MAE) corresponding to the predictions for the validation set.  Recall that the labels for the validation data are stored in `y_valid`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8bc25586",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-14T14:09:10.999656Z",
     "iopub.status.busy": "2021-11-14T14:09:10.998692Z",
     "iopub.status.idle": "2021-11-14T14:09:11.002076Z",
     "shell.execute_reply": "2021-11-14T14:09:11.001011Z",
     "shell.execute_reply.started": "2021-11-14T13:42:18.186703Z"
    },
    "papermill": {
     "duration": 0.023329,
     "end_time": "2021-11-14T14:09:11.002224",
     "exception": false,
     "start_time": "2021-11-14T14:09:10.978895",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# # Calculate MAE\n",
    "# mae_1 = mean_absolute_error(y_valid, predictions_1)\n",
    "\n",
    "# # Uncomment to print MAE\n",
    "# print(\"Mean Absolute Error:\" , mae_1)\n",
    "\n",
    "# # Check your answer\n",
    "# step_1.c.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "1089b828",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-14T14:09:11.039517Z",
     "iopub.status.busy": "2021-11-14T14:09:11.037988Z",
     "iopub.status.idle": "2021-11-14T14:09:11.043418Z",
     "shell.execute_reply": "2021-11-14T14:09:11.042769Z"
    },
    "papermill": {
     "duration": 0.025568,
     "end_time": "2021-11-14T14:09:11.043611",
     "exception": false,
     "start_time": "2021-11-14T14:09:11.018043",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_1.c.hint()\n",
    "#step_1.c.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10a51ff1",
   "metadata": {
    "papermill": {
     "duration": 0.015624,
     "end_time": "2021-11-14T14:09:11.075554",
     "exception": false,
     "start_time": "2021-11-14T14:09:11.059930",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Step 2: Improve the model\n",
    "\n",
    "Now that you've trained a default model as baseline, it's time to tinker with the parameters, to see if you can get better performance!\n",
    "- Begin by setting `my_model_2` to an XGBoost model, using the [XGBRegressor](https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBRegressor) class.  Use what you learned in the previous tutorial to figure out how to change the default parameters (like `n_estimators` and `learning_rate`) to get better results.\n",
    "- Then, fit the model to the training data in `X_train` and `y_train`.\n",
    "- Set `predictions_2` to the model's predictions for the validation data.  Recall that the validation features are stored in `X_valid`.\n",
    "- Finally, use the `mean_absolute_error()` function to calculate the mean absolute error (MAE) corresponding to the predictions on the validation set.  Recall that the labels for the validation data are stored in `y_valid`.\n",
    "\n",
    "In order for this step to be marked correct, your model in `my_model_2` must attain lower MAE than the model in `my_model_1`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d7972901",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-14T14:09:11.110655Z",
     "iopub.status.busy": "2021-11-14T14:09:11.109963Z",
     "iopub.status.idle": "2021-11-14T14:09:11.115061Z",
     "shell.execute_reply": "2021-11-14T14:09:11.114068Z",
     "shell.execute_reply.started": "2021-11-14T13:45:50.749911Z"
    },
    "papermill": {
     "duration": 0.023285,
     "end_time": "2021-11-14T14:09:11.115278",
     "exception": false,
     "start_time": "2021-11-14T14:09:11.091993",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# # Define the model\n",
    "# my_model_2 = XGBRegressor(n_estimators=1000, learning_rate=0.05)\n",
    "\n",
    "# # Fit the model\n",
    "# my_model_2.fit(X_train, y_train,\n",
    "#                early_stopping_rounds=5,\n",
    "#                eval_set=[(X_valid, y_valid)]\n",
    "# )\n",
    "\n",
    "# # Get predictions\n",
    "# predictions_2 = my_model_2.predict(X_valid)\n",
    "\n",
    "# # Calculate MAE\n",
    "# mae_2 = mean_absolute_error(y_valid, predictions_2)\n",
    "\n",
    "# # Uncomment to print MAE\n",
    "# print(\"Mean Absolute Error:\" , mae_2)\n",
    "\n",
    "# # Check your answer\n",
    "# step_2.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "3c2c735c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-14T14:09:11.151345Z",
     "iopub.status.busy": "2021-11-14T14:09:11.150571Z",
     "iopub.status.idle": "2021-11-14T14:09:11.154629Z",
     "shell.execute_reply": "2021-11-14T14:09:11.154034Z"
    },
    "papermill": {
     "duration": 0.024152,
     "end_time": "2021-11-14T14:09:11.154784",
     "exception": false,
     "start_time": "2021-11-14T14:09:11.130632",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_2.hint()\n",
    "#step_2.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "664b5ec2",
   "metadata": {
    "papermill": {
     "duration": 0.016358,
     "end_time": "2021-11-14T14:09:11.186951",
     "exception": false,
     "start_time": "2021-11-14T14:09:11.170593",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Step 3: Break the model\n",
    "\n",
    "In this step, you will create a model that performs worse than the original model in Step 1.  This will help you to develop your intuition for how to set parameters.  You might even find that you accidentally get better performance, which is ultimately a nice problem to have and a valuable learning experience!\n",
    "- Begin by setting `my_model_3` to an XGBoost model, using the [XGBRegressor](https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBRegressor) class.  Use what you learned in the previous tutorial to figure out how to change the default parameters (like `n_estimators` and `learning_rate`) to design a model to get high MAE.\n",
    "- Then, fit the model to the training data in `X_train` and `y_train`.\n",
    "- Set `predictions_3` to the model's predictions for the validation data.  Recall that the validation features are stored in `X_valid`.\n",
    "- Finally, use the `mean_absolute_error()` function to calculate the mean absolute error (MAE) corresponding to the predictions on the validation set.  Recall that the labels for the validation data are stored in `y_valid`.\n",
    "\n",
    "In order for this step to be marked correct, your model in `my_model_3` must attain higher MAE than the model in `my_model_1`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8b8b3a61",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-14T14:09:11.222597Z",
     "iopub.status.busy": "2021-11-14T14:09:11.221881Z",
     "iopub.status.idle": "2021-11-14T14:09:11.223217Z",
     "shell.execute_reply": "2021-11-14T14:09:11.223822Z",
     "shell.execute_reply.started": "2021-11-14T13:50:28.555898Z"
    },
    "papermill": {
     "duration": 0.021935,
     "end_time": "2021-11-14T14:09:11.224029",
     "exception": false,
     "start_time": "2021-11-14T14:09:11.202094",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# # Define the model\n",
    "# my_model_3 = XGBRegressor(n_estimators=10)\n",
    "\n",
    "# # Fit the model\n",
    "# my_model_3.fit(X_train, y_train)\n",
    "\n",
    "# # Get predictions\n",
    "# predictions_3 = my_model_3.predict(X_valid)\n",
    "\n",
    "# # Calculate MAE\n",
    "# mae_3 = mean_absolute_error(y_valid, predictions_3)\n",
    "\n",
    "# # Uncomment to print MAE\n",
    "# print(\"Mean Absolute Error:\" , mae_3)\n",
    "\n",
    "# # Check your answer\n",
    "# step_3.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d42715a7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-14T14:09:11.259331Z",
     "iopub.status.busy": "2021-11-14T14:09:11.258705Z",
     "iopub.status.idle": "2021-11-14T14:09:11.262034Z",
     "shell.execute_reply": "2021-11-14T14:09:11.262609Z"
    },
    "papermill": {
     "duration": 0.023,
     "end_time": "2021-11-14T14:09:11.262828",
     "exception": false,
     "start_time": "2021-11-14T14:09:11.239828",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_3.hint()\n",
    "#step_3.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29d8cdac",
   "metadata": {
    "papermill": {
     "duration": 0.016312,
     "end_time": "2021-11-14T14:09:11.295999",
     "exception": false,
     "start_time": "2021-11-14T14:09:11.279687",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Keep going\n",
    "\n",
    "Continue to learn about **[data leakage](https://www.kaggle.com/alexisbcook/data-leakage)**.  This is an important issue for a data scientist to understand, and it has the potential to ruin your models in subtle and dangerous ways!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3342b2eb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-14T14:09:11.329484Z",
     "iopub.status.busy": "2021-11-14T14:09:11.328829Z",
     "iopub.status.idle": "2021-11-14T14:09:11.634370Z",
     "shell.execute_reply": "2021-11-14T14:09:11.634891Z",
     "shell.execute_reply.started": "2021-11-14T13:55:27.279836Z"
    },
    "papermill": {
     "duration": 0.323909,
     "end_time": "2021-11-14T14:09:11.635076",
     "exception": false,
     "start_time": "2021-11-14T14:09:11.311167",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.preprocessing import OneHotEncoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c50d524c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-14T14:09:11.670148Z",
     "iopub.status.busy": "2021-11-14T14:09:11.669445Z",
     "iopub.status.idle": "2021-11-14T14:09:11.743300Z",
     "shell.execute_reply": "2021-11-14T14:09:11.742673Z",
     "shell.execute_reply.started": "2021-11-14T13:55:27.684219Z"
    },
    "papermill": {
     "duration": 0.093097,
     "end_time": "2021-11-14T14:09:11.743488",
     "exception": false,
     "start_time": "2021-11-14T14:09:11.650391",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Read the data\n",
    "X_full = pd.read_csv('../input/train.csv', index_col='Id')\n",
    "X_test_full = pd.read_csv('../input/test.csv', index_col='Id')\n",
    "\n",
    "# Remove rows with missing target, separate target from predictors\n",
    "X_full.dropna(axis=0, subset=['SalePrice'], inplace=True)\n",
    "y = X_full.SalePrice\n",
    "X_full.drop(['SalePrice'], axis=1, inplace=True)\n",
    "\n",
    "# Break off validation set from training data\n",
    "X_train_full, X_valid_full, y_train, y_valid = train_test_split(X_full, y, \n",
    "                                                                train_size=0.8, test_size=0.2,\n",
    "                                                                random_state=0)\n",
    "\n",
    "# \"Cardinality\" means the number of unique values in a column\n",
    "# Select categorical columns with relatively low cardinality (convenient but arbitrary)\n",
    "categorical_cols = [cname for cname in X_train_full.columns if\n",
    "                    X_train_full[cname].nunique() < 10 and \n",
    "                    X_train_full[cname].dtype == \"object\"]\n",
    "\n",
    "# Select numerical columns\n",
    "numerical_cols = [cname for cname in X_train_full.columns if \n",
    "                X_train_full[cname].dtype in ['int64', 'float64']]\n",
    "\n",
    "# Keep selected columns only\n",
    "my_cols = categorical_cols + numerical_cols\n",
    "X_train = X_train_full[my_cols].copy()\n",
    "X_valid = X_valid_full[my_cols].copy()\n",
    "X_test = X_test_full[my_cols].copy()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "59520e36",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-14T14:09:11.784115Z",
     "iopub.status.busy": "2021-11-14T14:09:11.783359Z",
     "iopub.status.idle": "2021-11-14T14:09:11.786497Z",
     "shell.execute_reply": "2021-11-14T14:09:11.785976Z",
     "shell.execute_reply.started": "2021-11-14T13:59:13.465650Z"
    },
    "papermill": {
     "duration": 0.026948,
     "end_time": "2021-11-14T14:09:11.786659",
     "exception": false,
     "start_time": "2021-11-14T14:09:11.759711",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Preprocessing for numerical data\n",
    "numerical_transformer = SimpleImputer(strategy='median')\n",
    "\n",
    "# Preprocessing for categorical data\n",
    "categorical_transformer = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='constant')),\n",
    "    ('onehot', OneHotEncoder(handle_unknown='ignore'))\n",
    "])\n",
    "# Bundle preprocessing for numerical and categorical data\n",
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        ('num', numerical_transformer, numerical_cols),\n",
    "        ('cat', categorical_transformer, categorical_cols)\n",
    "    ])\n",
    "\n",
    "# Define model\n",
    "model = XGBRegressor(n_estimators=1000, learning_rate=0.05)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "61f69863",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-14T14:09:11.831926Z",
     "iopub.status.busy": "2021-11-14T14:09:11.830946Z",
     "iopub.status.idle": "2021-11-14T14:09:23.854868Z",
     "shell.execute_reply": "2021-11-14T14:09:23.855442Z",
     "shell.execute_reply.started": "2021-11-14T13:59:13.805181Z"
    },
    "papermill": {
     "duration": 12.053538,
     "end_time": "2021-11-14T14:09:23.855656",
     "exception": false,
     "start_time": "2021-11-14T14:09:11.802118",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE: 16625.319670376713\n"
     ]
    }
   ],
   "source": [
    "# Bundle preprocessing and modeling code in a pipeline\n",
    "my_pipeline = Pipeline(steps=[('preprocessor', preprocessor),\n",
    "                              ('model', model)\n",
    "                             ])\n",
    "\n",
    "# Preprocessing of training data, fit model \n",
    "my_pipeline.fit(X_train, y_train,\n",
    "#          model__early_stopping_rounds=5,\n",
    "#          model__eval_set=[(X_valid, y_valid)]\n",
    ")\n",
    "\n",
    "\n",
    "# Preprocessing of validation data, get predictions\n",
    "preds = my_pipeline.predict(X_valid)\n",
    "\n",
    "# Evaluate the model\n",
    "score = mean_absolute_error(y_valid, preds)\n",
    "print('MAE:', score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "fbe732a4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-11-14T14:09:23.897314Z",
     "iopub.status.busy": "2021-11-14T14:09:23.889740Z",
     "iopub.status.idle": "2021-11-14T14:09:23.957000Z",
     "shell.execute_reply": "2021-11-14T14:09:23.957571Z",
     "shell.execute_reply.started": "2021-11-14T13:59:56.840525Z"
    },
    "papermill": {
     "duration": 0.086669,
     "end_time": "2021-11-14T14:09:23.957768",
     "exception": false,
     "start_time": "2021-11-14T14:09:23.871099",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "preds_test = my_pipeline.predict(X_test)\n",
    "# Save test predictions to file\n",
    "output = pd.DataFrame({'Id': X_test.index,\n",
    "                       'SalePrice': preds_test})\n",
    "output.to_csv('submission.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a217568",
   "metadata": {
    "papermill": {
     "duration": 0.017234,
     "end_time": "2021-11-14T14:09:23.990151",
     "exception": false,
     "start_time": "2021-11-14T14:09:23.972917",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "---\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/intermediate-machine-learning/discussion) to chat with other learners.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 26.561133,
   "end_time": "2021-11-14T14:09:24.817444",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2021-11-14T14:08:58.256311",
   "version": "2.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}