3 Classification/Regression_Trees to predict house prices.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Regression Trees**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **20** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab you will learn how to implement regression trees using ScikitLearn. We will show what parameters are important, how to train a regression tree, and finally how to determine our regression trees accuracy.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After completing this lab you will be able to:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   Train a Regression Tree\n",
    "*   Evaluate a Regression Trees Performance\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this lab, we are going to be using Python and several Python libraries. Some of these libraries might be installed in your lab environment or in SN Labs. Others may need to be installed by you. The cells below will install these libraries when executed.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install libraries not already in the environment using pip\n",
    "#!pip install pandas==1.3.4\n",
    "#!pip install sklearn==0.20.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pandas will allow us to create a dataframe of the data so it can be used and manipulated\n",
    "import pandas as pd\n",
    "# Regression Tree Algorithm\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "# Split our data into a training and testing data\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About the Dataset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Imagine you are a data scientist working for a real estate company that is planning to invest in Boston real estate. You have collected information about various areas of Boston and are tasked with created a model that can predict the median price of houses for that area so it can be used to make offers.\n",
    "\n",
    "The dataset had information on areas/towns not individual houses, the features are\n",
    "\n",
    "CRIM: Crime per capita\n",
    "\n",
    "ZN: Proportion of residential land zoned for lots over 25,000 sq.ft.\n",
    "\n",
    "INDUS: Proportion of non-retail business acres per town\n",
    "\n",
    "CHAS: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)\n",
    "\n",
    "NOX: Nitric oxides concentration (parts per 10 million)\n",
    "\n",
    "RM: Average number of rooms per dwelling\n",
    "\n",
    "AGE: Proportion of owner-occupied units built prior to 1940\n",
    "\n",
    "DIS: Weighted distances to ﬁve Boston employment centers\n",
    "\n",
    "RAD: Index of accessibility to radial highways\n",
    "\n",
    "TAX: Full-value property-tax rate per $10,000\n",
    "\n",
    "PTRAIO: Pupil-teacher ratio by town\n",
    "\n",
    "LSTAT: Percent lower status of the population\n",
    "\n",
    "MEDV: Median value of owner-occupied homes in $1000s\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Read the Data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets read in the data we have downloaded\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-SkillsNetwork/labs/Module%203/data/real_estate_data.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>LSTAT</th>\n",
       "      <th>MEDV</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00632</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>6.575</td>\n",
       "      <td>65.2</td>\n",
       "      <td>4.0900</td>\n",
       "      <td>1</td>\n",
       "      <td>296</td>\n",
       "      <td>15.3</td>\n",
       "      <td>4.98</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.02731</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>6.421</td>\n",
       "      <td>78.9</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2</td>\n",
       "      <td>242</td>\n",
       "      <td>17.8</td>\n",
       "      <td>9.14</td>\n",
       "      <td>21.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.02729</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>7.185</td>\n",
       "      <td>61.1</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2</td>\n",
       "      <td>242</td>\n",
       "      <td>17.8</td>\n",
       "      <td>4.03</td>\n",
       "      <td>34.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.03237</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>6.998</td>\n",
       "      <td>45.8</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3</td>\n",
       "      <td>222</td>\n",
       "      <td>18.7</td>\n",
       "      <td>2.94</td>\n",
       "      <td>33.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.06905</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>7.147</td>\n",
       "      <td>54.2</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3</td>\n",
       "      <td>222</td>\n",
       "      <td>18.7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>36.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      CRIM    ZN  INDUS  CHAS    NOX     RM   AGE     DIS  RAD  TAX  PTRATIO  \\\n",
       "0  0.00632  18.0   2.31   0.0  0.538  6.575  65.2  4.0900    1  296     15.3   \n",
       "1  0.02731   0.0   7.07   0.0  0.469  6.421  78.9  4.9671    2  242     17.8   \n",
       "2  0.02729   0.0   7.07   0.0  0.469  7.185  61.1  4.9671    2  242     17.8   \n",
       "3  0.03237   0.0   2.18   0.0  0.458  6.998  45.8  6.0622    3  222     18.7   \n",
       "4  0.06905   0.0   2.18   0.0  0.458  7.147  54.2  6.0622    3  222     18.7   \n",
       "\n",
       "   LSTAT  MEDV  \n",
       "0   4.98  24.0  \n",
       "1   9.14  21.6  \n",
       "2   4.03  34.7  \n",
       "3   2.94  33.4  \n",
       "4    NaN  36.2  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets learn about the size of our data, there are 506 rows and 13 columns\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(506, 13)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CRIM       20\n",
       "ZN         20\n",
       "INDUS      20\n",
       "CHAS       20\n",
       "NOX         0\n",
       "RM          0\n",
       "AGE        20\n",
       "DIS         0\n",
       "RAD         0\n",
       "TAX         0\n",
       "PTRATIO     0\n",
       "LSTAT      20\n",
       "MEDV        0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 506 entries, 0 to 505\n",
      "Data columns (total 13 columns):\n",
      " #   Column   Non-Null Count  Dtype  \n",
      "---  ------   --------------  -----  \n",
      " 0   CRIM     486 non-null    float64\n",
      " 1   ZN       486 non-null    float64\n",
      " 2   INDUS    486 non-null    float64\n",
      " 3   CHAS     486 non-null    float64\n",
      " 4   NOX      506 non-null    float64\n",
      " 5   RM       506 non-null    float64\n",
      " 6   AGE      486 non-null    float64\n",
      " 7   DIS      506 non-null    float64\n",
      " 8   RAD      506 non-null    int64  \n",
      " 9   TAX      506 non-null    int64  \n",
      " 10  PTRATIO  506 non-null    float64\n",
      " 11  LSTAT    486 non-null    float64\n",
      " 12  MEDV     506 non-null    float64\n",
      "dtypes: float64(11), int64(2)\n",
      "memory usage: 51.5 KB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the data is valid, there are rows with missing values which we will deal with in pre-processing\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CRIM       20\n",
       "ZN         20\n",
       "INDUS      20\n",
       "CHAS       20\n",
       "NOX         0\n",
       "RM          0\n",
       "AGE        20\n",
       "DIS         0\n",
       "RAD         0\n",
       "TAX         0\n",
       "PTRATIO     0\n",
       "LSTAT      20\n",
       "MEDV        0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Pre-Processing\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First lets drop the rows with missing values because we have enough data in our dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can see our dataset has no missing values\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CRIM       0\n",
       "ZN         0\n",
       "INDUS      0\n",
       "CHAS       0\n",
       "NOX        0\n",
       "RM         0\n",
       "AGE        0\n",
       "DIS        0\n",
       "RAD        0\n",
       "TAX        0\n",
       "PTRATIO    0\n",
       "LSTAT      0\n",
       "MEDV       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets split the dataset into our features and what we are predicting (target)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = data.drop(columns=[\"MEDV\"])\n",
    "Y = data[\"MEDV\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>LSTAT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00632</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>6.575</td>\n",
       "      <td>65.2</td>\n",
       "      <td>4.0900</td>\n",
       "      <td>1</td>\n",
       "      <td>296</td>\n",
       "      <td>15.3</td>\n",
       "      <td>4.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.02731</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>6.421</td>\n",
       "      <td>78.9</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2</td>\n",
       "      <td>242</td>\n",
       "      <td>17.8</td>\n",
       "      <td>9.14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.02729</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>7.185</td>\n",
       "      <td>61.1</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2</td>\n",
       "      <td>242</td>\n",
       "      <td>17.8</td>\n",
       "      <td>4.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.03237</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>6.998</td>\n",
       "      <td>45.8</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3</td>\n",
       "      <td>222</td>\n",
       "      <td>18.7</td>\n",
       "      <td>2.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.02985</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>6.430</td>\n",
       "      <td>58.7</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3</td>\n",
       "      <td>222</td>\n",
       "      <td>18.7</td>\n",
       "      <td>5.21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.14455</td>\n",
       "      <td>12.5</td>\n",
       "      <td>7.87</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.524</td>\n",
       "      <td>6.172</td>\n",
       "      <td>96.1</td>\n",
       "      <td>5.9505</td>\n",
       "      <td>5</td>\n",
       "      <td>311</td>\n",
       "      <td>15.2</td>\n",
       "      <td>19.15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.21124</td>\n",
       "      <td>12.5</td>\n",
       "      <td>7.87</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.524</td>\n",
       "      <td>5.631</td>\n",
       "      <td>100.0</td>\n",
       "      <td>6.0821</td>\n",
       "      <td>5</td>\n",
       "      <td>311</td>\n",
       "      <td>15.2</td>\n",
       "      <td>29.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.22489</td>\n",
       "      <td>12.5</td>\n",
       "      <td>7.87</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.524</td>\n",
       "      <td>6.377</td>\n",
       "      <td>94.3</td>\n",
       "      <td>6.3467</td>\n",
       "      <td>5</td>\n",
       "      <td>311</td>\n",
       "      <td>15.2</td>\n",
       "      <td>20.45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.11747</td>\n",
       "      <td>12.5</td>\n",
       "      <td>7.87</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.524</td>\n",
       "      <td>6.009</td>\n",
       "      <td>82.9</td>\n",
       "      <td>6.2267</td>\n",
       "      <td>5</td>\n",
       "      <td>311</td>\n",
       "      <td>15.2</td>\n",
       "      <td>13.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.09378</td>\n",
       "      <td>12.5</td>\n",
       "      <td>7.87</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.524</td>\n",
       "      <td>5.889</td>\n",
       "      <td>39.0</td>\n",
       "      <td>5.4509</td>\n",
       "      <td>5</td>\n",
       "      <td>311</td>\n",
       "      <td>15.2</td>\n",
       "      <td>15.71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.62976</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.14</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>5.949</td>\n",
       "      <td>61.8</td>\n",
       "      <td>4.7075</td>\n",
       "      <td>4</td>\n",
       "      <td>307</td>\n",
       "      <td>21.0</td>\n",
       "      <td>8.26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0.62739</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.14</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>5.834</td>\n",
       "      <td>56.5</td>\n",
       "      <td>4.4986</td>\n",
       "      <td>4</td>\n",
       "      <td>307</td>\n",
       "      <td>21.0</td>\n",
       "      <td>8.47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1.05393</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.14</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>5.935</td>\n",
       "      <td>29.3</td>\n",
       "      <td>4.4986</td>\n",
       "      <td>4</td>\n",
       "      <td>307</td>\n",
       "      <td>21.0</td>\n",
       "      <td>6.58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0.78420</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.14</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>5.990</td>\n",
       "      <td>81.7</td>\n",
       "      <td>4.2579</td>\n",
       "      <td>4</td>\n",
       "      <td>307</td>\n",
       "      <td>21.0</td>\n",
       "      <td>14.67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0.80271</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.14</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>5.456</td>\n",
       "      <td>36.6</td>\n",
       "      <td>3.7965</td>\n",
       "      <td>4</td>\n",
       "      <td>307</td>\n",
       "      <td>21.0</td>\n",
       "      <td>11.69</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       CRIM    ZN  INDUS  CHAS    NOX     RM    AGE     DIS  RAD  TAX  \\\n",
       "0   0.00632  18.0   2.31   0.0  0.538  6.575   65.2  4.0900    1  296   \n",
       "1   0.02731   0.0   7.07   0.0  0.469  6.421   78.9  4.9671    2  242   \n",
       "2   0.02729   0.0   7.07   0.0  0.469  7.185   61.1  4.9671    2  242   \n",
       "3   0.03237   0.0   2.18   0.0  0.458  6.998   45.8  6.0622    3  222   \n",
       "5   0.02985   0.0   2.18   0.0  0.458  6.430   58.7  6.0622    3  222   \n",
       "7   0.14455  12.5   7.87   0.0  0.524  6.172   96.1  5.9505    5  311   \n",
       "8   0.21124  12.5   7.87   0.0  0.524  5.631  100.0  6.0821    5  311   \n",
       "10  0.22489  12.5   7.87   0.0  0.524  6.377   94.3  6.3467    5  311   \n",
       "11  0.11747  12.5   7.87   0.0  0.524  6.009   82.9  6.2267    5  311   \n",
       "12  0.09378  12.5   7.87   0.0  0.524  5.889   39.0  5.4509    5  311   \n",
       "13  0.62976   0.0   8.14   0.0  0.538  5.949   61.8  4.7075    4  307   \n",
       "15  0.62739   0.0   8.14   0.0  0.538  5.834   56.5  4.4986    4  307   \n",
       "16  1.05393   0.0   8.14   0.0  0.538  5.935   29.3  4.4986    4  307   \n",
       "17  0.78420   0.0   8.14   0.0  0.538  5.990   81.7  4.2579    4  307   \n",
       "18  0.80271   0.0   8.14   0.0  0.538  5.456   36.6  3.7965    4  307   \n",
       "\n",
       "    PTRATIO  LSTAT  \n",
       "0      15.3   4.98  \n",
       "1      17.8   9.14  \n",
       "2      17.8   4.03  \n",
       "3      18.7   2.94  \n",
       "5      18.7   5.21  \n",
       "7      15.2  19.15  \n",
       "8      15.2  29.93  \n",
       "10     15.2  20.45  \n",
       "11     15.2  13.27  \n",
       "12     15.2  15.71  \n",
       "13     21.0   8.26  \n",
       "15     21.0   8.47  \n",
       "16     21.0   6.58  \n",
       "17     21.0  14.67  \n",
       "18     21.0  11.69  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head(15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    24.0\n",
       "1    21.6\n",
       "2    34.7\n",
       "3    33.4\n",
       "5    28.7\n",
       "Name: MEDV, dtype: float64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally lets split our data into a training and testing dataset using `train_test_split` from `sklearn.model_selection`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=.2, random_state=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Regression Tree\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regression Trees are implemented using `DecisionTreeRegressor` from `sklearn.tree`\n",
    "\n",
    "The important parameters of `DecisionTreeRegressor` are\n",
    "\n",
    "`criterion`: {\"mse\", \"friedman_mse\", \"mae\", \"poisson\"} - The function used to measure error\n",
    "\n",
    "`max_depth` - The max depth the tree can be\n",
    "\n",
    "`min_samples_split` - The minimum number of samples required to split a node\n",
    "\n",
    "`min_samples_leaf` - The minimum number of samples that a leaf can contain\n",
    "\n",
    "`max_features`: {\"auto\", \"sqrt\", \"log2\"} - The number of feature we examine looking for the best one, used to speed up training\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First lets start by creating a `DecisionTreeRegressor` object, setting the `criterion` parameter to `mse` for Mean Squared Error\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "regression_tree = DecisionTreeRegressor(criterion = \"mse\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets train our model using the `fit` method on the `DecisionTreeRegressor` object providing our training data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor()"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regression_tree.fit(X_train, Y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To evaluate our dataset we will use the `score` method of the `DecisionTreeRegressor` object providing our testing data, this number is the $R^2$ value which indicates the coefficient of determination\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8286767180715902"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regression_tree.score(X_test, Y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also find the average error in our testing set which is the average error in median home value prediction\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "$ 2983.544303797468\n"
     ]
    }
   ],
   "source": [
    "prediction = regression_tree.predict(X_test)\n",
    "\n",
    "print(\"$\",(prediction - Y_test).abs().mean()*1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Excercise\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train a regression tree using the `criterion` `mae` then report its $R^2$ value and average error\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7748648815869272\n",
      "$ 3013.9240506329106\n"
     ]
    }
   ],
   "source": [
    "#creating a DecisionTreeRegressor with criterion 'mae'\n",
    "regression_tree = DecisionTreeRegressor(criterion = \"mae\")\n",
    "\n",
    "#train our model using the fit method on the DecisionTreeRegressor\n",
    "regression_tree.fit(X_train, Y_train)\n",
    "\n",
    "# Evaluate the model and dataset\n",
    "print(regression_tree.score(X_test, Y_test))\n",
    "\n",
    "\n",
    "#Find and report its  𝑅2  value and average error\n",
    "prediction = regression_tree.predict(X_test)\n",
    "print(\"$\",(prediction - Y_test).abs().mean()*1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>Click here for the solution</summary>\n",
    "\n",
    "```python\n",
    "regression_tree = DecisionTreeRegressor(criterion = \"mae\")\n",
    "\n",
    "regression_tree.fit(X_train, Y_train)\n",
    "\n",
    "print(regression_tree.score(X_test, Y_test))\n",
    "\n",
    "prediction = regression_tree.predict(X_test)\n",
    "\n",
    "print(\"$\",(prediction - Y_test).abs().mean()*1000)\n",
    "\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}