Bagging and boosting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bagging and Boosting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This work aims to show the advantage that bagging and boosting bring to a machine learning problem. Both techniques involve a form of aggregation of several estimators to achieve better results.\n",
    "\n",
    "Here, we consider a classification problem that is predicting whether a household earns more or less than 50K. We will evaluate it using a linear classifier. Then, we will leverage it through bagging and boosting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data cleaning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import timeit\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.ensemble import RandomForestClassifier,AdaBoostClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_train = pd.read_csv('adult-new.data',header=None)\n",
    "df_test= pd.read_csv('adult-new.test',header=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "columns=[\"age\",\"workclass\",\"fnlwgt\",\"education\",\"education_num\",\"marital_status\",\"occupation\",\"relationship\",\"race\",\n",
    "         \"sex\",\"capital_gain\",\"capital_loss\",\"hours_per_week\",\"native_country\",\"target\"]\n",
    "df_train=df_train.rename({i:columns[i] for i in range(len(columns))},axis=1).drop(\"education_num\",axis=1)\n",
    "df_test=df_test.rename({i:columns[i] for i in range(len(columns))},axis=1).drop(\"education_num\",axis=1)\n",
    "df_train.target=np.array(df_train.target==' >50K').astype('uint8')\n",
    "df_test.target=np.array(df_test.target==' >50K').astype('uint8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The categorical variables are : workclass, education, marital_status, occupation, relationship, race, sex and native country. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "columns_categorical=[\"workclass\",\"education\",\"marital_status\",\"occupation\",\"relationship\",\"race\",\n",
    "                     \"sex\",\"native_country\"]\n",
    "columns_numeric=[\"age\",\"capital_gain\",\"capital_loss\",\"hours_per_week\"]\n",
    "df_train_dummies=pd.get_dummies(df_train[columns_categorical]).join(df_train[[i for i in df_train.columns\n",
    "                                                                              if i not in columns_categorical]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the test set, after creating the dummies, we must ensure that it has the same columns as the train set. We also don't want to leak information in the train set. Therefore, we proceed like following on the test set:\n",
    "* Drop all the dummy features that are on the test set but not on the train set\n",
    "* Create zero columns for features that are on the train set but not on the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_test_dummies=pd.get_dummies(df_test[columns_categorical]).join(df_test[[i for i in df_test.columns\n",
    "                                                                              if i not in columns_categorical]])\n",
    "df_test_dummies=df_test_dummies[[i for i in df_train_dummies.columns if i in df_test_dummies.columns]]\n",
    "for i in set(df_train_dummies)-set(df_test_dummies):\n",
    "    df_test_dummies[i]=np.zeros(len(df_test_dummies))\n",
    "df_test_dummies=df_test_dummies[df_train_dummies.columns]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have 108 features for both data set, and the features are the same for both of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "(32561, 108) (16281, 108)\n"
     ]
    }
   ],
   "source": [
    "print(list(df_train_dummies.columns)==list(df_test_dummies.columns))\n",
    "print(df_train_dummies.shape,df_test_dummies.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also scale the variables that are not categorical. We use the Minmax scaler to preserve the zero entries in the sparse columns capital_gain and capital_loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "scaler=MinMaxScaler()\n",
    "df_train_dummies[columns_numeric]=scaler.fit_transform(df_train_dummies[columns_numeric])\n",
    "df_test_dummies[columns_numeric]=scaler.transform(df_test_dummies[columns_numeric])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are using a  Linear Support Vector Classification with the hinge loss and l2 penalty. Three paramaters to fit: the loss parameter C, the maximum number of iterations, and wether to weight the classes.\n",
    "\n",
    "To fit those parameters, we use a grid search on the train dataset. We test each combinations of parameters using crossvalidation with 3 folds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=None, error_score='raise',\n",
       "       estimator=LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,\n",
       "     intercept_scaling=1, loss='hinge', max_iter=1000, multi_class='ovr',\n",
       "     penalty='l2', random_state=None, tol=0.0001, verbose=0),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'C': array([1, 2, 3, 4, 5, 6, 7, 8, 9]), 'max_iter': [1000, 10000, 100000], 'class_weight': ['balanced', None]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring=None, verbose=0)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier_1=LinearSVC(loss=\"hinge\",penalty='l2')\n",
    "param_grid={\"C\":np.arange(1,10),\"max_iter\":[1000,10000,100000],\"class_weight\":[\"balanced\",None]}\n",
    "grid=GridSearchCV(classifier_1,param_grid)\n",
    "grid.fit(df_train_dummies.drop([\"target\",\"fnlwgt\"],axis=1),df_train_dummies.target,sample_weight=df_train_dummies.fnlwgt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'C': 4, 'class_weight': None, 'max_iter': 100000}\n",
      "0/1 train error :  0.14799914007555048\n",
      "0/1 test error :  0.14870093974571585\n"
     ]
    }
   ],
   "source": [
    "print(grid.best_params_)\n",
    "model=grid.best_estimator_\n",
    "print(\"0/1 train error : \",1-model.score(df_train_dummies.drop([\"target\",\"fnlwgt\"],axis=1),df_train_dummies.target))\n",
    "print(\"0/1 test error : \",1-model.score(df_test_dummies.drop([\"target\",\"fnlwgt\"],axis=1),df_test_dummies.target))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We reach a 0/1 error of **0.149** with the affine classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's look at the false positive and negative rates for females and males."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rates for the first classifier:\n",
      "FP rate Female: 0.018204385601985933\n",
      "FN rate Female: 0.5201401050788091\n",
      "FP rate Male: 0.09252434851276652\n",
      "FN rate Male: 0.4066503965832825\n"
     ]
    }
   ],
   "source": [
    "y_test_predict=model.predict(df_test_dummies.drop([\"target\",\"fnlwgt\"],axis=1))\n",
    "y_test_predict_f=y_test_predict[df_test_dummies[\"sex_ Female\"]==1]\n",
    "y_test_predict_m=y_test_predict[df_test_dummies[\"sex_ Male\"]==1]\n",
    "y_test_f=df_test_dummies.target[df_test_dummies[\"sex_ Female\"]==1]\n",
    "y_test_m=df_test_dummies.target[df_test_dummies[\"sex_ Male\"]==1]\n",
    "print(\"Rates for the first classifier:\")\n",
    "print(\"FP rate Female:\", sum((y_test_predict_f==1)&(y_test_f==0))/sum(y_test_f==0))\n",
    "print(\"FN rate Female:\", sum((y_test_predict_f==0)&(y_test_f==1))/sum(y_test_f==1))\n",
    "print(\"FP rate Male:\", sum((y_test_predict_m==1)&(y_test_m==0))/sum(y_test_m==0))\n",
    "print(\"FN rate Male:\", sum((y_test_predict_m==0)&(y_test_m==1))/sum(y_test_m==1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bagging with Random Forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our second classifier, we use a random forest classifier. Random Forest uses bagging because it trains multiple decision trees on subsamples using bootstrap. It also trains each decision tree on a subsample of the features, making it an improved version of bagging. Eventually, all these decision trees vote to make a prediction.\n",
    "\n",
    "\n",
    "We tune the following parameters to control the size of our random forest and the numbers of trees: max_depth and min_samples_leaf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=None, error_score='raise',\n",
       "       estimator=RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "            oob_score=False, random_state=None, verbose=0,\n",
       "            warm_start=False),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'max_depth': array([ 5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,\n",
       "       22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38,\n",
       "       39]), 'min_samples_leaf': array([1, 2, 3, 4, 5])},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring=None, verbose=0)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier_2=RandomForestClassifier()\n",
    "param_grid={\"max_depth\":np.arange(5,40),\"min_samples_leaf\":np.arange(1,6)}\n",
    "grid=GridSearchCV(classifier_2,param_grid)\n",
    "grid.fit(df_train_dummies.drop([\"target\",\"fnlwgt\"],axis=1),df_train_dummies.target,sample_weight=df_train_dummies.fnlwgt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'max_depth': 38, 'min_samples_leaf': 3}\n",
      "0/1 train error :  0.12235496452811645\n",
      "0/1 test error :  0.13795221423745474\n"
     ]
    }
   ],
   "source": [
    "print(grid.best_params_)\n",
    "model=grid.best_estimator_\n",
    "print(\"0/1 train error : \",1-model.score(df_train_dummies.drop([\"target\",\"fnlwgt\"],axis=1),df_train_dummies.target))\n",
    "print(\"0/1 test error : \",1-model.score(df_test_dummies.drop([\"target\",\"fnlwgt\"],axis=1),df_test_dummies.target))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rates for the first classifier:\n",
      "FP rate Female: 0.018204385601985933\n",
      "FN rate Female: 0.46935201401050786\n",
      "FP rate Male: 0.09226112134772309\n",
      "FN rate Male: 0.3627211714460037\n"
     ]
    }
   ],
   "source": [
    "y_test_predict=model.predict(df_test_dummies.drop([\"target\",\"fnlwgt\"],axis=1))\n",
    "y_test_predict_f=y_test_predict[df_test_dummies[\"sex_ Female\"]==1]\n",
    "y_test_predict_m=y_test_predict[df_test_dummies[\"sex_ Male\"]==1]\n",
    "y_test_f=df_test_dummies.target[df_test_dummies[\"sex_ Female\"]==1]\n",
    "y_test_m=df_test_dummies.target[df_test_dummies[\"sex_ Male\"]==1]\n",
    "print(\"Rates for the second classifier:\")\n",
    "print(\"FP rate Female:\", sum((y_test_predict_f==1)&(y_test_f==0))/sum(y_test_f==0))\n",
    "print(\"FN rate Female:\", sum((y_test_predict_f==0)&(y_test_f==1))/sum(y_test_f==1))\n",
    "print(\"FP rate Male:\", sum((y_test_predict_m==1)&(y_test_m==0))/sum(y_test_m==0))\n",
    "print(\"FN rate Male:\", sum((y_test_predict_m==0)&(y_test_m==1))/sum(y_test_m==1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The 0/1 risk is lower using Random Forest - Bagging helped!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Boosting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To illustrate boosting, we will use the AdaBoost algorithm. It trains several Decision Trees where at each step the tree has a larger focus on misclassified data observations from the previous steps. Eventually, all the predictions of the trees are combined in a vote."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=None, error_score='raise',\n",
       "       estimator=AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None,\n",
       "          learning_rate=1.0, n_estimators=50, random_state=None),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'n_estimators': [20, 50, 100, 150], 'learning_rate': array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. , 1.1, 1.2, 1.3,\n",
       "       1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2. ])},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring=None, verbose=0)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier_3=AdaBoostClassifier()\n",
    "param_grid={\"n_estimators\":[20,50,100,150],\"learning_rate\":np.arange(1,21)/10}\n",
    "grid=GridSearchCV(classifier_3,param_grid)\n",
    "grid.fit(df_train_dummies.drop([\"target\",\"fnlwgt\"],axis=1),df_train_dummies.target,sample_weight=df_train_dummies.fnlwgt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'learning_rate': 1.6, 'n_estimators': 150}\n",
      "0/1 train error :  0.13150701759773964\n",
      "0/1 test error :  0.13359130274553155\n"
     ]
    }
   ],
   "source": [
    "print(grid.best_params_)\n",
    "model=grid.best_estimator_\n",
    "print(\"0/1 train error : \",1-model.score(df_train_dummies.drop([\"target\",\"fnlwgt\"],axis=1),df_train_dummies.target))\n",
    "print(\"0/1 test error : \",1-model.score(df_test_dummies.drop([\"target\",\"fnlwgt\"],axis=1),df_test_dummies.target))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rates for the first classifier:\n",
      "FP rate Female: 0.01985932974762102\n",
      "FN rate Female: 0.46059544658493873\n",
      "FP rate Male: 0.10028954988154777\n",
      "FN rate Male: 0.32153752287980475\n"
     ]
    }
   ],
   "source": [
    "y_test_predict=model.predict(df_test_dummies.drop([\"target\",\"fnlwgt\"],axis=1))\n",
    "y_test_predict_f=y_test_predict[df_test_dummies[\"sex_ Female\"]==1]\n",
    "y_test_predict_m=y_test_predict[df_test_dummies[\"sex_ Male\"]==1]\n",
    "y_test_f=df_test_dummies.target[df_test_dummies[\"sex_ Female\"]==1]\n",
    "y_test_m=df_test_dummies.target[df_test_dummies[\"sex_ Male\"]==1]\n",
    "print(\"Rates for the third classifier:\")\n",
    "print(\"FP rate Female:\", sum((y_test_predict_f==1)&(y_test_f==0))/sum(y_test_f==0))\n",
    "print(\"FN rate Female:\", sum((y_test_predict_f==0)&(y_test_f==1))/sum(y_test_f==1))\n",
    "print(\"FP rate Male:\", sum((y_test_predict_m==1)&(y_test_m==0))/sum(y_test_m==0))\n",
    "print(\"FN rate Male:\", sum((y_test_predict_m==0)&(y_test_m==1))/sum(y_test_m==1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our example, it seems like AdaBoost yields the best results. Interestingly, it also leads to less overfitting as the train and test errors are very close."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}