Exercise-08.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# California housing dataset regression with decision trees \n",
    "\n",
    "In this notebook, we'll use [decision trees](http://scikit-learn.org/stable/modules/tree.html) and [ensembles of trees](http://scikit-learn.org/stable/modules/ensemble.html) to estimate median house values on Californian housing districts using scikit-learn and [XGBoost](https://xgboost.readthedocs.io/en/latest/).\n",
    "\n",
    "First, the needed imports. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import numpy as np\n",
    "from sklearn import datasets, __version__\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from xgboost import XGBRegressor\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "\n",
    "Then we load the California housing data. First time we need to download the data, which can take a while."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "chd = datasets.fetch_california_housing()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll split the data into a training and a test set.\n",
    "\n",
    "Let's also select a single attribute to start the analysis with, say *MedInc*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "test_size = 5000\n",
    "single_attribute = 'MedInc'\n",
    "\n",
    "X_train_all, X_test_all, y_train, y_test = train_test_split(\n",
    "    chd.data, chd.target, test_size=test_size, shuffle=True)\n",
    "\n",
    "attribute_index = chd.feature_names.index(single_attribute)\n",
    "X_train_single = X_train_all[:, attribute_index].reshape(-1, 1)\n",
    "X_test_single = X_test_all[:, attribute_index].reshape(-1, 1)\n",
    "     \n",
    "print()\n",
    "print('California housing data: train:',len(X_train_all),'test:',len(X_test_all))\n",
    "print()\n",
    "print('X_train_all:', X_train_all.shape)\n",
    "print('X_train_single:', X_train_single.shape)\n",
    "print('y_train:', y_train.shape)\n",
    "print()\n",
    "print('X_test_all', X_test_all.shape)\n",
    "print('X_test_single', X_test_single.shape)\n",
    "print('y_test', y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training data matrix `X_train_all` is a matrix of size (`n_train`, 8), and `X_train_single` contains only the first attribute (*MedInc* by default). The vector `y_train` contains the target value (median house value) for each housing district in the training set.\n",
    "\n",
    "Let's start our analysis with the single attribute. Later, you can set `only_single_attribute = False` to use all eight attributes in the regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "only_single_attribute = True\n",
    "\n",
    "if only_single_attribute:\n",
    "    X_train = X_train_single\n",
    "    X_test = X_test_single\n",
    "else:\n",
    "    X_train = X_train_all\n",
    "    X_test = X_test_all\n",
    "\n",
    "print('X_train:', X_train.shape)\n",
    "print('X_test:', X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision tree\n",
    "\n",
    "Decision tree is a model that predicts the value of a target variable by learning simple *if-then-else* decision rules inferred from the data features.\n",
    "\n",
    "### Learning\n",
    "\n",
    "The parameter `max_depth` specifies the maximum depth of the tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "max_depth = 3\n",
    "dt_reg = DecisionTreeRegressor(max_depth=max_depth)\n",
    "dt_reg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualize the results if we are using only a single attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if X_train.shape[1] == 1:\n",
    "    plt.figure(figsize=(10, 10))\n",
    "    plt.scatter(X_train, y_train, s=5)\n",
    "    reg_x = np.arange(np.min(X_train), np.max(X_train), 0.01).reshape(-1, 1)\n",
    "    plt.plot(reg_x, dt_reg.predict(reg_x), lw=4, c=sns.color_palette()[1],\n",
    "             label='decision tree')\n",
    "    plt.legend(loc='best');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference\n",
    "\n",
    "We use *mean squared error* as the performance measure for our regression algorihm: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "predictions = dt_reg.predict(X_test)\n",
    "print(\"Mean squared error: %.3f\"\n",
    "      % mean_squared_error(y_test, predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random forest\n",
    "\n",
    "Random forest is an ensemble (or a group; hence the name *forest*) of decision trees, obtained by introducing randomness into the tree generation. The prediction of the random forest is obtained by *averaging* the predictions of the individual trees.\n",
    "\n",
    "Random forest is a solid workhorse that almost always produces serviceable results without much tuning.\n",
    "\n",
    "### Learning\n",
    "\n",
    "Random forest classifiers are quick to train, quite robust to hyperparameter values, and often work relatively well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "n_estimators = 10\n",
    "max_depth = 3\n",
    "rf_reg = RandomForestRegressor(n_estimators=n_estimators,\n",
    "                               max_depth=max_depth)\n",
    "rf_reg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if X_train.shape[1] == 1:\n",
    "    plt.figure(figsize=(10, 10))\n",
    "    plt.scatter(X_train, y_train, s=5)\n",
    "    reg_x = np.arange(np.min(X_train), np.max(X_train), 0.01).reshape(-1, 1)\n",
    "    plt.plot(reg_x, dt_reg.predict(reg_x), lw=4, c=sns.color_palette()[1],\n",
    "             label='decision tree')\n",
    "    plt.plot(reg_x, rf_reg.predict(reg_x), lw=4, c=sns.color_palette()[2],\n",
    "             label='random forest')\n",
    "    plt.legend(loc='best');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "predictions = rf_reg.predict(X_test)\n",
    "print(\"Mean squared error: %.3f\"\n",
    "      % mean_squared_error(y_test, predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient boosted trees (XGBoost)\n",
    "\n",
    "Gradient boosted trees (or extreme gradient boosted trees) is another way of constructing ensembles of decision trees, using the *boosting* framework.  Let's use a popular separate package, [XGBoost](http://xgboost.readthedocs.io/en/latest/), to train gradient boosted trees for regression.  \n",
    "\n",
    "XGBoost has been recently used to obtain record-breaking results on many machine learning competitions, but have quite a lot of hyperparameters that need to be carefully tuned to get the best performance.\n",
    "\n",
    "### Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "xgb_reg = XGBRegressor()\n",
    "xgb_reg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if X_train.shape[1] == 1:\n",
    "    plt.figure(figsize=(10, 10))\n",
    "    plt.scatter(X_train, y_train, s=5)\n",
    "    reg_x = np.arange(np.min(X_train), np.max(X_train), 0.01).reshape(-1, 1)\n",
    "    plt.plot(reg_x, dt_reg.predict(reg_x), lw=4, c=sns.color_palette()[1],\n",
    "             label='decision tree')\n",
    "    plt.plot(reg_x, rf_reg.predict(reg_x), lw=4, c=sns.color_palette()[2],\n",
    "             label='random forest')\n",
    "    plt.plot(reg_x, xgb_reg.predict(reg_x), lw=4, c=sns.color_palette()[3],\n",
    "             label='XGBoost')\n",
    "    plt.legend(loc='best');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "predictions = xgb_reg.predict(X_test)\n",
    "print(\"Mean squared error: %.3f\"\n",
    "      % mean_squared_error(y_test, predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Study the documentation of the different decision tree models used in this notebook ([decision trees](http://scikit-learn.org/stable/modules/tree.html), [tree ensembles](http://scikit-learn.org/stable/modules/ensemble.html), [XGBoost](https://xgboost.readthedocs.io/en/latest/)), and experiment with different hyperparameter values.  \n",
    "\n",
    "Report the lowest mean squared error you manage to obtain for each model type.  Also mark down the parameters you used, so others can try to reproduce your results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}