Chapter_8_sec_8.3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8.3 Lab: Decision Trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.tree import DecisionTreeClassifier, export_graphviz, DecisionTreeRegressor, plot_tree\n",
    "from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score, mean_squared_error\n",
    "from sklearn import tree\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8.3.1 Fitting Classification Trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\" \n",
    "The sklearn library has a lot of useful tools for tress. We first use classification trees to analyze the Carseats data set.\n",
    "In these data, Sales is a continuous variable, and so we begin by recoding it as a binary variable (by thresholding). \n",
    "We use the map() function to create a variable, called High, which takes on a value of 'Y' if the Sales variable exceeds 8, \n",
    "and takes on a value of 'N' otherwise. In Python, we need to code catergorical variable into dummy variable.\n",
    "\"\"\"\n",
    "\n",
    "carseats = pd.read_csv('./data/Carseats.csv')\n",
    "carseats['High'] = carseats.Sales.map(lambda x: 'Y' if x>8 else 'N')\n",
    "carseats.ShelveLoc = pd.factorize(carseats.ShelveLoc)[0]\n",
    "carseats.Urban = carseats.Urban.map({'No':0, 'Yes':1})\n",
    "carseats.US = carseats.US.map({'No':0, 'Yes':1})\n",
    "print(carseats.describe())\n",
    "print(carseats.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# we first split the dataset into training (200 samples) and test sets.\n",
    "X = carseats.drop(['Sales', 'High'], axis=1)\n",
    "y = carseats.High\n",
    "train_size = 200\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=train_size, \n",
    "                                    test_size=X.shape[0]-train_size, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to build a tree, we could use 'gini' or 'entropy' as split criterion at each node. \n",
    "# here we use an example use 'gini' and with a few other hyperparameters.\n",
    "criteria = \"gini\" \n",
    "max_depth = 6 \n",
    "min_sample_leaf = 4\n",
    "clf_gini = DecisionTreeClassifier(criterion=criteria, random_state=100,\n",
    "                                max_depth=max_depth, min_samples_leaf=min_sample_leaf)\n",
    "clf_gini.fit(X_train, y_train)\n",
    "print(clf_gini.score(X_train, y_train))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# one attractive feature of a tree is visulization. \n",
    "plt.figure(figsize=(40,20))  # customize according to the size of your tree\n",
    "plot_tree(clf_gini, feature_names = X_train.columns)\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let us build the confusion matrix to evaluate the model in accuracy for both training and test datasets.\n",
    "# we could also compute more metrics such as precision, recall, f1-score, etc.\n",
    "y_pred_train = clf_gini.predict(X_train)\n",
    "cm = pd.DataFrame(confusion_matrix(y_train, y_pred_train).T, index=['No', 'Yes'], columns=['No', 'Yes'])\n",
    "print(cm)\n",
    "print(\"Train Accuracy is \", accuracy_score(y_train,y_pred_train)*100)\n",
    "\n",
    "\n",
    "y_pred = clf_gini.predict(X_test)\n",
    "cm = pd.DataFrame(confusion_matrix(y_test, y_pred).T, index=['No', 'Yes'], columns=['No', 'Yes'])\n",
    "print(cm)\n",
    "print(\"Test Accuracy is \", accuracy_score(y_test,y_pred)*100)\n",
    "\n",
    "\"\"\"\n",
    "The test accuracy of our model is significant lower than our training result, this may indicate overfitting. \n",
    "we can go back and change the hyperparameters in the training process to reduce the dimension of the parameter space.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8.3.2 Fitting Regression Trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\" \n",
    "Another attractive feature of a tree is the ability to solve both classification and regression problems.\n",
    "Here we fit a regression tree to the Boston data set. First, we create a training set, and fit the tree to the training data. \n",
    "Since Python does not support back-prune, let us use the max_depth at 2.\n",
    "\"\"\"\n",
    "\n",
    "# as we move forward, it is good to keep the hyperparameters together for future iterations.\n",
    "boston = pd.read_csv('./data/Boston.csv')\n",
    "X = boston.drop('medv', axis=1)\n",
    "y = boston.medv\n",
    "train_size = 0.5 # we used specific train size before, we can also use a percentage. \n",
    "random_state = 0 \n",
    "max_depth = 2\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=train_size, random_state=random_state)\n",
    "regr_tree = DecisionTreeRegressor(max_depth=max_depth)\n",
    "regr_tree.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(40,20))  # customize according to the size of your tree\n",
    "plot_tree(regr_tree, feature_names = X_train.columns)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = regr_tree.predict(X_test)\n",
    "print(mean_squared_error(y_test, y_pred))\n",
    "\"\"\" \n",
    "We could look more into train and test accuracy to see whether the current model is overfitting or underfitting.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8.3.3 Bagging and Random Forests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Here we apply bagging and random forests (RF) to the Boston data.\n",
    "RF is an ensemble method, which means it combines the results of multiple decision trees. \n",
    "As a result, RF could help to reduce the variance of the model. \n",
    "Similar to decision trees, RF can be used to solve both classification and regression problems.\n",
    "\n",
    "\n",
    "In this excercise, we will use the randomForest package in Python. \n",
    "The exact results obtained in this section may depend on the version of Python and the version of the randomForest package installed on your computer. \n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we reused the previous train and test sets.\n",
    "all_features = X_train.shape[1]\n",
    "regr_bagging = RandomForestRegressor(max_features=all_features, random_state=1)\n",
    "regr_bagging.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = regr_bagging.predict(X_test)\n",
    "print(mean_squared_error(y_test, y_pred))\n",
    "\n",
    "\"\"\"\n",
    "we can compare the test accuracy of the bagging model with the test accuracy of the singl regression tree above.\n",
    "Normally, the bagging model is better than the single tree model.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "We can grow a random forest in exactly the same way, except that we'll use a smaller value of the max_features argument. \n",
    "Theoretically, radomly selecting a subset of features could reduce the correlation of the tress and can reduce the variance of the model.\n",
    "\"\"\"\n",
    "# here we'll use max_features = 3 (close to square root of all features as a rule of thumb)\n",
    "regr_rf = RandomForestRegressor(max_features=3, random_state=1)\n",
    "regr_rf.fit(X_train, y_train)\n",
    "\n",
    "y_pred = regr_rf.predict(X_test)\n",
    "print(mean_squared_error(y_test, y_pred))\n",
    "\n",
    "\"\"\" \n",
    "The test set MSE is even lower; this indicates that random forests yielded an improvement over bagging in this case.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# RF provides multiple ways to interpret the model. One way is to see the importance of each feature.\n",
    "Importance = pd.DataFrame({'Importance':regr_rf.feature_importances_*100}, index=X_train.columns)\n",
    "Importance.sort_values(by='Importance', axis=0, ascending=True).plot(kind='barh', color='r', )\n",
    "plt.xlabel('Variable Importance')\n",
    "plt.gca().legend_ = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8.3.4 Boosting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we use the GradientBoostingRegressor package. The argument n_estimators=500 indicates that we want 500 trees, and the option interaction.depth=4 limits the depth of each tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Boosting is another ensemble method. Gradient Boosting is a popular method, \n",
    "and other well-known methods include AdaBoost, XGBoost, and LightGBM are buiit on top of it.\n",
    "\n",
    "Here we use the GradientBoostingRegressor package. The argument n_estimators=500 indicates that we want 500 trees, \n",
    "and the option interaction.depth=4 limits the depth of each tree. See the manuel for more details.\n",
    "\"\"\"\n",
    "\n",
    "regr_boost = GradientBoostingRegressor(n_estimators=500, learning_rate=0.02, max_depth=4, random_state=1)\n",
    "regr_boost.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let us check the feature importance and MSE.\n",
    "feature_importance = regr_boost.feature_importances_*100\n",
    "rel_imp = pd.Series(feature_importance, index=X_train.columns).sort_values(inplace=False)\n",
    "rel_imp.T.plot(kind='barh', color='r', )\n",
    "plt.xlabel('Variable Importance')\n",
    "plt.gca().legend_ = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = regr_boost.predict(X_test)\n",
    "print(mean_squared_error(y_test,y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8.3.5 Bayesian Additive Regression Trees\n",
    "As of now (2021), I was not able to find a good package for BART in Python. Please reach out if you have a package that works.\n",
    "** [To do: find a package for BART in Python.] **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Here we discuss Bayesian additive regression trees (BART), another ensemble method that uses decision trees as its building blocks.\n",
    "BART is related to both Random forest and boosting: each tree is constructed in a random manner as in bagging and random forests, \n",
    "and each tree tries to capture signal not yet accounted for by the current model, as in boosting. \n",
    "The main novelty in BART is the way in which new trees are generated.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# End of Chapter 8"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "4548a0e672c5b3a287feee7b2962606840aa548749d1830ef724408652b0c250"
  },
  "kernelspec": {
   "display_name": "Python 2.7.16 64-bit ('base': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}