RCBoost.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Rule Cover Boosting - RCBoost\n",
    "\n",
    "In this note, we compare the solutions obtained with Rule Cover Boosting (RCBoost) algorithm against several well-known ensemble methods. \n",
    "\n",
    "We start with adding the required packages. Please note that except RCBoost (provided here) all other packages are bundled with the standard installation of [Anaconda Distribution](https://www.anaconda.com/products/individual) (Pyhton 3.7). For solving the linear programming problems, RCBoost also uses the `gurobipy` Python package that can be separately installed by the Anaconda package manager. Note that along with the Python package, you also need to install [Gurobi Optimizer](https://www.gurobi.com/academia/academic-program-and-licenses/), which is free for research and educational work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from RCBoost import RCBoost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we list the set of problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import RuleCoverDatasets as RCDS\n",
    "problems = [RCDS.banknote, RCDS.ILPD, RCDS.ionosphere,\n",
    "            RCDS.transfusion, RCDS.liver, RCDS.tictactoe,\n",
    "            RCDS.wdbc, RCDS.mammography, RCDS.diabetes, \n",
    "            RCDS.oilspill, RCDS.phoneme, RCDS.seeds, RCDS.wine,\n",
    "            RCDS.glass, RCDS.ecoli]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To give an example, let's solve problem `ionosphere` with different methods. We first import the dataset and apply a standard train-test split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = np.array(RCDS.ionosphere('datasets/'))                                                \n",
    "X = df[:, 0:-1]                                                                    \n",
    "y = df[:, -1]                                                                                      \n",
    "\n",
    "randomstate = 29 # Random seed is fixed\n",
    "\n",
    "# Train (70%) - Test (30%) split\n",
    "X_train, X_test, y_train, y_test = \\\n",
    "    train_test_split(X, y, test_size=0.3, random_state=randomstate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The contender methods are Random Forest (RF), ADABoost (ADA), Gradient Boosting (GDB) and RCBoost (RCB). We set different hyperparameters for the methods. Note that the base estimator for ADA is a decision tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "maxdepth = 10 # Used by all methods\n",
    "crit = \"gini\" # Used by all methods - only other option is \"entropy\"\n",
    "nestimators = 50 # Used by RF, ADA and GDB\n",
    "maxrmpcalls = 50 # Maximum number of RMP calls - Used only by RCB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to train all contender methods and obtain their predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using license file /opt/gurobi/gurobi.lic\n",
      "Academic license - for non-commercial use only\n"
     ]
    }
   ],
   "source": [
    "# Random Forest\n",
    "RF = RandomForestClassifier(random_state=randomstate, criterion=crit)\n",
    "RF_fit = RF.fit(X_train, y_train)\n",
    "RF_pred = RF_fit.predict(X_test)\n",
    "\n",
    "# ADABoost\n",
    "ADA = AdaBoostClassifier(base_estimator=\n",
    "                         DecisionTreeClassifier(max_depth=maxdepth,criterion=crit),\n",
    "                         random_state=randomstate, n_estimators=nestimators)\n",
    "ADA_fit = ADA.fit(X_train, y_train)\n",
    "ADA_pred = ADA_fit.predict(X_test)\n",
    "\n",
    "# Gradient Boosting\n",
    "GDB = GradientBoostingClassifier(max_depth=maxdepth, n_estimators=nestimators,\n",
    "                                random_state=randomstate)\n",
    "GDB_fit = GDB.fit(X_train, y_train)\n",
    "GDB_pred = GDB_fit.predict(X_test)\n",
    "\n",
    "# RCBoost\n",
    "RCB = RCBoost(max_depth=maxdepth, maxNumOfRMPCalls=maxrmpcalls,\n",
    "              random_state=randomstate)                                                       \n",
    "RCB_fit = RCB.fit(X_train, y_train)\n",
    "RCB_pred = RCB_fit.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the initial set of rules in RCBoost comes from a decision tree, we also report the results with this base tree to see the improvement obtained with RCBoost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Starting Decision Tree of RCBoost\n",
    "initDT_fit = RCB_fit.initialEstimator.fit(X_train, y_train)\n",
    "initDT_pred = initDT_fit.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We next check the accuracies of different methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "## ACCURACIES ##\n",
      "Random Forest:  0.9339622641509434\n",
      "ADABoost:  0.9339622641509434\n",
      "Gradient Boosting:  0.8584905660377359\n",
      "RCBoost:  0.9433962264150944\n",
      "Initial Decision Tree:  0.8490566037735849\n"
     ]
    }
   ],
   "source": [
    "print('## ACCURACIES ##')\n",
    "print('Random Forest: ', accuracy_score(RF_pred, y_test)) \n",
    "print('ADABoost: ', accuracy_score(ADA_pred, y_test)) \n",
    "print('Gradient Boosting: ', accuracy_score(GDB_pred, y_test)) \n",
    "print('RCBoost: ', accuracy_score(RCB_pred, y_test)) \n",
    "print('Initial Decision Tree: ', accuracy_score(initDT_pred, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RCBoost obtains a better accuracy than all methods with this particular random seed. We can also see how many RMP calls are made with RCBoost to obtain this result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Number of RMP calls by RCBoost:  11\n"
     ]
    }
   ],
   "source": [
    "print('\\nNumber of RMP calls by RCBoost: ', RCB_fit.nofRMPcalls)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}