Task2_ModelSelection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Task 2: Model selection\n",
    "#### Data Analytics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Introduction\n",
    "The dataset named *task2_data* (*task2_data.csv*) has 515 samples and 8 features.\n",
    "\n",
    "The main objective of the task is to **preprocess** the train data in the way it is indicated, perform a guided **train/validation model selection** step, and **test the final winner model on test data**.\n",
    "\n",
    "If you try anytime several options it is important to show the results of those discarded trials, because what is not visible cannot be evaluated. If any cells are not meant to be executed but you leave them to show your trials, then comment the code so a full run is possible.\n",
    "\n",
    "The deliverable of this task is this Jupyter Notebook containing the code, plus some short answers in markdown cells if required. All the cells in the notebook should be run. You should also upload the downloaded html and pdf formats (*File > Export > Files or selection to HTML... (.html)*)\n",
    "\n",
    "NOTE: Keep in mind that some functions accept both Pandas dataframes and Numpy arrays, but some others only one of them. Nevertheless, we should know how to pass from one to the other and viceversa.\n",
    "\n",
    "NOTE: If you work in pairs, please add the name of your partner in brackets besides yours in the *Name & Surnames* field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "seed = 26"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Problem:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Consider the dataset *task2_data.csv*, which is a regression dataset that we will transform into a binary classification problem, just binarizing the target.\n",
    "\n",
    "\n",
    "* (a) We split the target into *high values* and *low values*. Binarize the target so that it is possible to determine whether a sample is high (*target* $>$ 4.0) or not (*target* $\\leq$ 4.0), overwritting your dataframe. [5%]\n",
    "\n",
    "\n",
    "* (b)  Split the data into train, validation and test sets, so that the test and validation sets have the same size, being one third of the train set size. Make sure that the proportion af the classes is the same in all parts. [5%]\n",
    "\n",
    "\n",
    "**Preprocessing**\n",
    "We will perform three preprocessing steps:\n",
    "* (c.1) Scale the features to a [0, 1] range. [5%]\n",
    "* (c.2) Perform a *recursive feature elimination* (RFE) in order to reduce the dimensionality of our data in at least 20%. You are free to choose the *estimator* you consider as the right one (argue your choice), and you **do not need** to use cross validation (CV). [10%]\n",
    "* (c.3) Check the imbalance degree of your data. If the imbalance is higher than 4 to 1, then reduce it to 2 to 1 using ADASYN algorithm. [5%]\n",
    "\n",
    "\n",
    "**Model selection**\n",
    "We will perform a 2-step model selection strategy. First, we will select a promising family of models just comparing their default ones. Then, once we have a preferred family, we will find the best parameters we can in order to get a winner final model.\n",
    "* (d.1) Consider the default *support vector machines* (SVM), *random forests* (RF), and *multilinear perceptron* (MLP) algorithms. Using *f1_macro* as score and 4-fold CV, determine which is the most promising family of algorithms. [15%]\n",
    "\n",
    "In this task, the most relevant parameters for each family are:\n",
    "- **SVM**:  *C* (unlimited options), *kernel* (5 options, but we ignore *linear* and *precomputed*), and *gamma* (2 options).\n",
    "- **RF**: *n_estimators* (unlimited options), *min_samples_split* (unlimited options if using float), *max_features* (3 options if we ignore integers and floats, considering that *auto* and *sqrt* are the same).\n",
    "- **MLP**: *hidden_layer_sizes* (unlimited in number of layers and neurons per layer), *solver* (use *'sgd'*), *learning_rate* (3 options), and *learning_rate_init* (unlimited options).\n",
    "\n",
    "\n",
    "* (d.2.1) For the most promising family found in (d.1), taking into account the info above, we will consider all tunable parameters (remember that in *MLP*, *solver* parameter must be *'sgd'* so it is not tunable). For each of them, consider at least two options in such a way that the total number of possible models is at least 20. Once you made your choices, exactly how many possible models could you have? [10%]\n",
    "\n",
    "* (d.2.2) Use a train/validation strategy to check all models. The best of all will be the final winning model. Which are the best parameters? And the best score? [30%]\n",
    "\n",
    "\n",
    "**Model validation**\n",
    "We will obtain the test score for the winning model and comment about the results achieved.\n",
    "* (e.1) Taking into account the final best parameters obtained in (d), train the final model with the right data. [5%]\n",
    "\n",
    "* (e.2) Calculate the final test score. [5%]\n",
    "\n",
    "* (e.3) Comparing this test score with the train/validation score obtained in (d.2.2) for that model, would you say that the winner model overfits? [5%]\n",
    "\n",
    "\n",
    "__Solution:__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "data = pd.read_csv('task2_data.csv')\n",
    "\n",
    "# Change V6 column data type from int64 to float64\n",
    "data['V6'] = data['V6'].astype('float64')\n",
    "\n",
    "# (a) We split the target into high values and low values. Binarize the target so that it is possible to determine whether a sample is high (target>4.0) or not\n",
    "# (target <= 4.0), overwritting your dataframe.\n",
    "\n",
    "x = data.iloc[:,:-1]\n",
    "y = data.iloc[:,-1]\n",
    "\n",
    "# Binarize y\n",
    "for i in range(len(y)):\n",
    "    if y[i] > 4.0:\n",
    "        y[i] = 1\n",
    "    else:\n",
    "        y[i] = 0\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "V1    float64\n",
       "V2    float64\n",
       "V3    float64\n",
       "V4    float64\n",
       "V5    float64\n",
       "V6    float64\n",
       "V7    float64\n",
       "V8    float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Imbalance rate:  4.466019417475728 %\n"
     ]
    }
   ],
   "source": [
    "# We check for imbalance rate (only for information purposes)\n",
    "imb_rate = sum(data.iloc[:, -1])/len(data.iloc[:, -1])\n",
    "print('Imbalance rate: ',imb_rate*100,'%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "343\n",
      "86\n",
      "86\n",
      "Suma xval xtest:  172\n",
      "(xval + xtest) * 2 =  344\n"
     ]
    }
   ],
   "source": [
    "# (b) Split the data into train, validation and test sets, so that the test and validation sets have the same size, being one third of the train set size. Make sure that\n",
    "# the proportion af the classes is the same in all parts.\n",
    "\n",
    "xtr, xtv, ytr, ytv = train_test_split(x, y, test_size=1/3, random_state=seed, stratify=data.iloc[:,-1])\n",
    "\n",
    "xval, xtest, yval, ytest = train_test_split(xtv, ytv, test_size=1/2, random_state=seed, stratify=ytv)\n",
    "\n",
    "print(len(xtr))\n",
    "print(len(xval))\n",
    "print(len(xtest))\n",
    "\n",
    "print('Suma xval xtest: ', len(xval)+len(xtest))\n",
    "print('(xval + xtest) * 2 = ', len(xtv)*2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>V1</th>\n",
       "      <th>V2</th>\n",
       "      <th>V3</th>\n",
       "      <th>V4</th>\n",
       "      <th>V5</th>\n",
       "      <th>V6</th>\n",
       "      <th>V7</th>\n",
       "      <th>V8</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>86.000000</td>\n",
       "      <td>86.000000</td>\n",
       "      <td>86.000000</td>\n",
       "      <td>86.000000</td>\n",
       "      <td>86.000000</td>\n",
       "      <td>86.000000</td>\n",
       "      <td>86.00000</td>\n",
       "      <td>86.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>90.037209</td>\n",
       "      <td>110.068605</td>\n",
       "      <td>526.576744</td>\n",
       "      <td>8.563953</td>\n",
       "      <td>18.298837</td>\n",
       "      <td>45.116279</td>\n",
       "      <td>4.09186</td>\n",
       "      <td>0.020930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>5.668241</td>\n",
       "      <td>70.369304</td>\n",
       "      <td>268.976847</td>\n",
       "      <td>4.546179</td>\n",
       "      <td>5.627485</td>\n",
       "      <td>18.708550</td>\n",
       "      <td>1.95630</td>\n",
       "      <td>0.156518</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>50.400000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>7.900000</td>\n",
       "      <td>0.400000</td>\n",
       "      <td>4.200000</td>\n",
       "      <td>18.000000</td>\n",
       "      <td>0.90000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>89.325000</td>\n",
       "      <td>47.700000</td>\n",
       "      <td>353.775000</td>\n",
       "      <td>6.300000</td>\n",
       "      <td>14.825000</td>\n",
       "      <td>29.000000</td>\n",
       "      <td>2.70000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>91.000000</td>\n",
       "      <td>112.950000</td>\n",
       "      <td>668.000000</td>\n",
       "      <td>7.500000</td>\n",
       "      <td>18.900000</td>\n",
       "      <td>40.500000</td>\n",
       "      <td>3.60000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>92.575000</td>\n",
       "      <td>139.075000</td>\n",
       "      <td>709.900000</td>\n",
       "      <td>10.275000</td>\n",
       "      <td>21.525000</td>\n",
       "      <td>58.750000</td>\n",
       "      <td>5.27500</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>96.100000</td>\n",
       "      <td>290.000000</td>\n",
       "      <td>855.300000</td>\n",
       "      <td>22.700000</td>\n",
       "      <td>33.100000</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>9.40000</td>\n",
       "      <td>1.400000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              V1          V2          V3         V4         V5         V6  \\\n",
       "count  86.000000   86.000000   86.000000  86.000000  86.000000  86.000000   \n",
       "mean   90.037209  110.068605  526.576744   8.563953  18.298837  45.116279   \n",
       "std     5.668241   70.369304  268.976847   4.546179   5.627485  18.708550   \n",
       "min    50.400000    3.000000    7.900000   0.400000   4.200000  18.000000   \n",
       "25%    89.325000   47.700000  353.775000   6.300000  14.825000  29.000000   \n",
       "50%    91.000000  112.950000  668.000000   7.500000  18.900000  40.500000   \n",
       "75%    92.575000  139.075000  709.900000  10.275000  21.525000  58.750000   \n",
       "max    96.100000  290.000000  855.300000  22.700000  33.100000  99.000000   \n",
       "\n",
       "             V7         V8  \n",
       "count  86.00000  86.000000  \n",
       "mean    4.09186   0.020930  \n",
       "std     1.95630   0.156518  \n",
       "min     0.90000   0.000000  \n",
       "25%     2.70000   0.000000  \n",
       "50%     3.60000   0.000000  \n",
       "75%     5.27500   0.000000  \n",
       "max     9.40000   1.400000  "
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xtest.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn import preprocessing\n",
    "#(c.1) Scale the features to a [0, 1] range.\n",
    "\n",
    "# Fit the min max scaler with the train dataset and transform it, and then only transform the test and validation datasets\n",
    "\n",
    "xtr_values = xtr.values # numpy array\n",
    "min_max_scaler = preprocessing.MinMaxScaler()\n",
    "x_scaled = min_max_scaler.fit_transform(xtr_values)\n",
    "xtr = pd.DataFrame(x_scaled, columns=xtr.columns)\n",
    "\n",
    "xval_values = xval.values # numpy array\n",
    "x_scaled = min_max_scaler.transform(xval_values)\n",
    "xval = pd.DataFrame(x_scaled, columns=xval.columns)\n",
    "\n",
    "xtest_values = xtest.values # numpy array\n",
    "x_scaled = min_max_scaler.transform(xtest_values)\n",
    "xtest = pd.DataFrame(x_scaled, columns=xtest.columns)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>V1</th>\n",
       "      <th>V2</th>\n",
       "      <th>V3</th>\n",
       "      <th>V4</th>\n",
       "      <th>V5</th>\n",
       "      <th>V6</th>\n",
       "      <th>V7</th>\n",
       "      <th>V8</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.911215</td>\n",
       "      <td>0.399792</td>\n",
       "      <td>0.772098</td>\n",
       "      <td>0.531532</td>\n",
       "      <td>0.552632</td>\n",
       "      <td>0.231707</td>\n",
       "      <td>0.600000</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.857477</td>\n",
       "      <td>0.129110</td>\n",
       "      <td>0.084132</td>\n",
       "      <td>0.261261</td>\n",
       "      <td>0.427632</td>\n",
       "      <td>0.146341</td>\n",
       "      <td>0.300000</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.904206</td>\n",
       "      <td>0.295258</td>\n",
       "      <td>0.834931</td>\n",
       "      <td>0.409910</td>\n",
       "      <td>0.523026</td>\n",
       "      <td>0.475610</td>\n",
       "      <td>0.300000</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.890187</td>\n",
       "      <td>0.442021</td>\n",
       "      <td>0.936694</td>\n",
       "      <td>0.319820</td>\n",
       "      <td>0.605263</td>\n",
       "      <td>0.268293</td>\n",
       "      <td>0.155556</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.892523</td>\n",
       "      <td>0.619245</td>\n",
       "      <td>0.707017</td>\n",
       "      <td>0.324324</td>\n",
       "      <td>0.562500</td>\n",
       "      <td>0.560976</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>0.871495</td>\n",
       "      <td>0.465559</td>\n",
       "      <td>0.955272</td>\n",
       "      <td>0.288288</td>\n",
       "      <td>0.351974</td>\n",
       "      <td>0.292683</td>\n",
       "      <td>0.255556</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>0.878505</td>\n",
       "      <td>0.569401</td>\n",
       "      <td>0.872204</td>\n",
       "      <td>0.301802</td>\n",
       "      <td>0.621711</td>\n",
       "      <td>0.682927</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.21875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>0.922897</td>\n",
       "      <td>0.453098</td>\n",
       "      <td>0.809490</td>\n",
       "      <td>0.396396</td>\n",
       "      <td>0.565789</td>\n",
       "      <td>0.048780</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>0.913551</td>\n",
       "      <td>0.296296</td>\n",
       "      <td>0.808307</td>\n",
       "      <td>0.301802</td>\n",
       "      <td>0.595395</td>\n",
       "      <td>0.365854</td>\n",
       "      <td>0.300000</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>0.810748</td>\n",
       "      <td>0.080651</td>\n",
       "      <td>0.061650</td>\n",
       "      <td>0.153153</td>\n",
       "      <td>0.447368</td>\n",
       "      <td>0.146341</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>86 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          V1        V2        V3        V4        V5        V6        V7  \\\n",
       "0   0.911215  0.399792  0.772098  0.531532  0.552632  0.231707  0.600000   \n",
       "1   0.857477  0.129110  0.084132  0.261261  0.427632  0.146341  0.300000   \n",
       "2   0.904206  0.295258  0.834931  0.409910  0.523026  0.475610  0.300000   \n",
       "3   0.890187  0.442021  0.936694  0.319820  0.605263  0.268293  0.155556   \n",
       "4   0.892523  0.619245  0.707017  0.324324  0.562500  0.560976  0.500000   \n",
       "..       ...       ...       ...       ...       ...       ...       ...   \n",
       "81  0.871495  0.465559  0.955272  0.288288  0.351974  0.292683  0.255556   \n",
       "82  0.878505  0.569401  0.872204  0.301802  0.621711  0.682927  0.800000   \n",
       "83  0.922897  0.453098  0.809490  0.396396  0.565789  0.048780  0.100000   \n",
       "84  0.913551  0.296296  0.808307  0.301802  0.595395  0.365854  0.300000   \n",
       "85  0.810748  0.080651  0.061650  0.153153  0.447368  0.146341  0.800000   \n",
       "\n",
       "         V8  \n",
       "0   0.00000  \n",
       "1   0.00000  \n",
       "2   0.00000  \n",
       "3   0.00000  \n",
       "4   0.00000  \n",
       "..      ...  \n",
       "81  0.00000  \n",
       "82  0.21875  \n",
       "83  0.00000  \n",
       "84  0.00000  \n",
       "85  0.00000  \n",
       "\n",
       "[86 rows x 8 columns]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xtest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Outliers: 35\n",
      "From 0 (majority) class: 34\n",
      "From 1 (minority) class: 1\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Outliers in xtrain\n",
    "from sklearn.covariance import EllipticEnvelope\n",
    "elip_env = EllipticEnvelope(random_state=seed).fit(xtr)\n",
    "\n",
    "detection = elip_env.predict(xtr)\n",
    "outlier_positions_mah = [x for x in range(xtr.shape[0]) if detection[x] == -1]\n",
    "print(\"Outliers: \" + str(len(outlier_positions_mah)))\n",
    "print(\"From 0 (majority) class: \" + str(sum(ytr.iloc[outlier_positions_mah] == 0)))\n",
    "print(\"From 1 (minority) class: \" + str(sum(ytr.iloc[outlier_positions_mah] == 1)))\n",
    "outlier_positions_mah_major = [x for x in range(xtr.shape[0]) if (detection[x] == -1 and ytr.iloc[x] == 0)]\n",
    "print(len(outlier_positions_mah_major) == sum(ytr.iloc[outlier_positions_mah] == 0))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Outliers: 13\n",
      "From 0 (majority) class: 13\n",
      "From 1 (minority) class: 0\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Outliers in xtest\n",
    "detection_test = elip_env.predict(xtest)\n",
    "outlier_positions_mah_test = [x for x in range(xtest.shape[0]) if detection_test[x] == -1]\n",
    "# Total amount of outliers in train\n",
    "print(\"Outliers: \" + str(len(outlier_positions_mah_test)))\n",
    "# Those from majority class (0.0)\n",
    "print(\"From 0 (majority) class: \" + str(sum(ytest.iloc[outlier_positions_mah_test] == 0)))\n",
    "# and minority class (1.0)\n",
    "print(\"From 1 (minority) class: \" + str(sum(ytest.iloc[outlier_positions_mah_test] == 1)))\n",
    "# Positions from majority class train outliers\n",
    "outlier_positions_mah_major_test = [x for x in range(xtest.shape[0]) if (detection_test[x] == -1 and ytest.iloc[x] == 0)]\n",
    "# Check\n",
    "print(len(outlier_positions_mah_major_test) == sum(ytest.iloc[outlier_positions_mah_test] == 0))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Outliers: 11\n",
      "From 0 (majority) class: 10\n",
      "From 1 (minority) class: 1\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Outliers in xval\n",
    "detection_val = elip_env.predict(xval)\n",
    "outlier_positions_mah_val = [x for x in range(xval.shape[0]) if detection_val[x] == -1]\n",
    "# Total amount of outliers in train\n",
    "print(\"Outliers: \" + str(len(outlier_positions_mah_val)))\n",
    "# Those from majority class (0.0)\n",
    "print(\"From 0 (majority) class: \" + str(sum(yval.iloc[outlier_positions_mah_val] == 0)))\n",
    "# and minority class (1.0)\n",
    "print(\"From 1 (minority) class: \" + str(sum(yval.iloc[outlier_positions_mah_val] == 1)))\n",
    "# Positions from majority class train outliers\n",
    "outlier_positions_mah_major_val = [x for x in range(xval.shape[0]) if (detection_val[x] == -1 and yval.iloc[x] == 0)]\n",
    "# Check\n",
    "print(len(outlier_positions_mah_major_val) == sum(yval.iloc[outlier_positions_mah_val] == 0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(309, 8), (309,), (73, 8), (73,), (76, 8), (76,)]\n"
     ]
    }
   ],
   "source": [
    "# Majority class outliers deletion in train\n",
    "xtr.drop(xtr.index[outlier_positions_mah_major], inplace=True)\n",
    "ytr.drop(ytr.index[outlier_positions_mah_major], inplace=True)\n",
    "# Majority class outliers deletion in validation\n",
    "xval.drop(xval.index[outlier_positions_mah_major_val], inplace=True)\n",
    "yval.drop(yval.index[outlier_positions_mah_major_val], inplace=True)\n",
    "# Majority class outliers deletion test (only for informative purposes at the end)\n",
    "# Note*: we are performing the preprocessing tasks on test at the same time for simplicity.\n",
    "# Nevertheless, we will not use the test set for prediction until last section.\n",
    "xtest.drop(xtest.index[outlier_positions_mah_major_test], inplace=True)\n",
    "ytest.drop(ytest.index[outlier_positions_mah_major_test], inplace=True)\n",
    "# Check after deleting the majority outliers\n",
    "print([xtr.shape, ytr.shape, xtest.shape, ytest.shape, xval.shape, yval.shape])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "#(c.2) Perform a recursive feature elimination (RFE) in order to reduce the dimensionality of our data in at least 20%.\n",
    "# You are free to choose the estimator you consider as the right one (argue your choice), and you do not need to use cross validation (CV)\n",
    "\n",
    "from sklearn.feature_selection import RFE\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "# I am going to use SVR estimator, because the size of the data is smaller than 10.000, for larger datasets it is used LinearSVR or SGDRegressor\n",
    "\n",
    "selectorTrain = RFE(estimator=DecisionTreeClassifier(), n_features_to_select=6, step=1)\n",
    "\n",
    "xtr_np = xtr.to_numpy()\n",
    "ytr_np = ytr.to_numpy()\n",
    "xtest_np = xtest.to_numpy()\n",
    "xval_np = xval.to_numpy()\n",
    "\n",
    "selectorTrain.fit(xtr_np, ytr_np)\n",
    "\n",
    "xtr_sel_features = selectorTrain.transform(xtr_np)\n",
    "xtest_sel_features = selectorTrain.transform(xtest_np)\n",
    "xval_sel_features = selectorTrain.transform(xval_np)\n",
    "\n",
    "xtr = pd.DataFrame(xtr_sel_features, columns=['V1','V2','V3','V4','V5','V6'])\n",
    "xtest = pd.DataFrame(xtest_sel_features, columns=['V1','V2','V3','V4','V5','V6'])\n",
    "xval = pd.DataFrame(xval_sel_features, columns=['V1','V2','V3','V4','V5','V6'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Imbalance rate train:  4.854368932038835 %\n",
      "Resampled dataset shape Counter({0.0: 294, 1.0: 15})\n",
      "\n",
      "Imbalance rate train:  33.63431151241535 %\n",
      "Resampled dataset shape Counter({0.0: 294, 1.0: 149})\n"
     ]
    }
   ],
   "source": [
    "#(c.3) Check the imbalance degree of your data. If the imbalance is higher than 4 to 1, then reduce it to 2 to 1 using ADASYN algorithm. [5%]\n",
    "\n",
    "from collections import Counter\n",
    "from imblearn.over_sampling import ADASYN\n",
    "\n",
    "# Imbalance rate before resample\n",
    "imb_rate = sum(ytr)/len(ytr)\n",
    "print('Imbalance rate train: ',imb_rate*100,'%')\n",
    "\n",
    "print('Resampled dataset shape %s' % Counter(ytr))\n",
    "\n",
    "ada_train = ADASYN(random_state=seed, sampling_strategy=1/2, n_neighbors=3)\n",
    "xtr, ytr = ada_train.fit_resample(xtr, ytr)\n",
    "print()\n",
    "\n",
    "# Imbalance rate after resample\n",
    "imb_rate = sum(ytr)/len(ytr)\n",
    "print('Imbalance rate train: ',imb_rate*100,'%')\n",
    "\n",
    "print('Resampled dataset shape %s' % Counter(ytr))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8516516952005198\n"
     ]
    }
   ],
   "source": [
    "# (d.1) Consider the default support vector machines (SVM), random forests (RF), and multilinear perceptron (MLP) algorithms. Using f1_macro as score and 4-fold CV,\n",
    "# determine which is the most promising family of algorithms. [15%]\n",
    "from sklearn.metrics import f1_score\n",
    "\n",
    "\n",
    "# Random Forest\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "average_score = cross_val_score(estimator=RandomForestClassifier(), X=xtr, y=ytr, cv=4, scoring='f1_macro').mean()\n",
    "print(average_score)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7733366160759356\n"
     ]
    }
   ],
   "source": [
    "# SVM\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn import svm\n",
    "\n",
    "average_score = cross_val_score(estimator=svm.SVC(), X=xtr, y=ytr, cv=4, scoring='f1_macro').mean()\n",
    "print(average_score)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# MLP\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "\n",
    "average_score = cross_val_score(estimator=MLPClassifier(), X=xtr, y=ytr, cv=4, scoring='f1_macro').mean()\n",
    "print(average_score)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration  0\n",
      "0.4722222222222222\n",
      "\n",
      "Iteration  1\n",
      "0.5866355866355867\n",
      "\n",
      "Iteration  2\n",
      "0.4722222222222222\n",
      "\n",
      "Iteration  3\n",
      "0.5866355866355867\n",
      "\n",
      "Iteration  4\n",
      "0.5866355866355867\n",
      "\n",
      "Iteration  5\n",
      "0.571830985915493\n",
      "\n",
      "Iteration  6\n",
      "0.5866355866355867\n",
      "\n",
      "Iteration  7\n",
      "0.571830985915493\n",
      "\n",
      "Iteration  8\n",
      "0.47586206896551725\n",
      "\n",
      "Iteration  9\n",
      "0.4722222222222222\n",
      "\n",
      "Iteration  10\n",
      "0.4722222222222222\n",
      "\n",
      "Iteration  11\n",
      "0.47586206896551725\n",
      "\n",
      "Iteration  12\n",
      "0.5866355866355867\n",
      "\n",
      "Iteration  13\n",
      "0.5866355866355867\n",
      "\n",
      "Iteration  14\n",
      "0.7047397047397047\n",
      "\n",
      "Iteration  15\n",
      "0.5866355866355867\n",
      "\n",
      "Iteration  16\n",
      "0.6041666666666667\n",
      "\n",
      "Iteration  17\n",
      "0.7047397047397047\n",
      "\n",
      "Iteration  18\n",
      "0.5866355866355867\n",
      "\n",
      "Iteration  19\n",
      "0.4722222222222222\n",
      "\n",
      "Iteration  20\n",
      "0.6041666666666667\n",
      "\n",
      "Iteration  21\n",
      "0.5866355866355867\n",
      "\n",
      "Iteration  22\n",
      "0.5866355866355867\n",
      "\n",
      "Iteration  23\n",
      "0.6041666666666667\n",
      "\n",
      "Iteration  24\n",
      "0.5866355866355867\n",
      "\n",
      "Iteration  25\n",
      "0.5866355866355867\n",
      "\n",
      "Iteration  26\n",
      "0.5866355866355867\n",
      "\n",
      "The estimator with highest cv is in the index  14 , with score of  0.7047397047397047\n",
      "The best params are: \n",
      "\tn_estimators =  60 \n",
      "\tmin_samples_split =  3 \n",
      "\tmax_features =  6\n"
     ]
    }
   ],
   "source": [
    "# In this task, the most relevant parameters for each family are:\n",
    "# -SVM: C (unlimited options), kernel (5 options, but we ignore linear and precomputed), and gamma (2 options).\n",
    "# -RF: n_estimators (unlimited options), min_samples_split (unlimited options if using float), max_features (3 options if we ignore integers and floats, considering that auto and sqrt are the same).\n",
    "# -MLP: hidden_layer_sizes (unlimited in number of layers and neurons per layer), solver (use ‘sgd’), learning_rate (3 options), and learning_rate_init (unlimited options).\n",
    "\n",
    "# (d.2.1) For the most promising family found in (d.1), taking into account the info above, we will consider all tunable parameters (remember that in MLP, solver parameter\n",
    "# must be ‘sgd’ so it is not tunable). For each of them, consider at least two options in such a way that the total number of possible models is at least 20. Once you made\n",
    "# your choices, exactly how many possible models could you have? [10%]\n",
    "\n",
    "# (d.2.2) Use a train/validation strategy to check all models. The best of all will be the final winning model. Which are the best parameters? And the best score? [30%]\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# The best estimator is Random Forest\n",
    "\n",
    "n_estimators_try = [20, 60, 100]\n",
    "min_samples_split_try = [2, 3, 4]\n",
    "max_features_try = [2, 4, 6]\n",
    "\n",
    "fitted_estimators = []\n",
    "f1_scores = []\n",
    "\n",
    "iter = 0\n",
    "params = []\n",
    "\n",
    "# First we fit the estimators with all the combination of parameters\n",
    "\n",
    "for i in n_estimators_try:\n",
    "    for j in min_samples_split_try:\n",
    "        for k in max_features_try:\n",
    "            rnd_clf = RandomForestClassifier(n_estimators=i, min_samples_split=j, max_features=k)\n",
    "            fitted_estimators.append(rnd_clf.fit(xtr, ytr))\n",
    "            params.append((i,j,k))\n",
    "\n",
    "# Now we get the f1 scores for all the fitted estimators with the validation dataset\n",
    "\n",
    "for i in fitted_estimators:\n",
    "    print('Iteration ', iter)\n",
    "\n",
    "    predicted = i.predict(xval)\n",
    "    expected = yval\n",
    "\n",
    "    f1_partial = f1_score(expected, predicted, average='macro')\n",
    "    print(f1_partial)\n",
    "    f1_scores.append(f1_partial)\n",
    "\n",
    "    iter += 1\n",
    "    print()\n",
    "\n",
    "# Lastly, we search the best f1 score obtained from the various fitted estimators with each combination of parameters\n",
    "\n",
    "max = 0\n",
    "index_max = 0\n",
    "for i in range(len(f1_scores)):\n",
    "    if f1_scores[i] > max:\n",
    "        index_max = i\n",
    "        max = f1_scores[i]\n",
    "\n",
    "print('The estimator with highest cv is in the index ', index_max, ', with score of ', f1_scores[index_max])\n",
    "print('The best params are: \\n\\tn_estimators = ',params[index_max][0], '\\n\\tmin_samples_split = ',params[index_max][1],'\\n\\tmax_features = ',params[index_max][2])\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "f1 macro score: 0.47857142857142854\n",
      "The area under the ROC curve obtained is: 0.4855072463768116\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "# (e.1) Taking into account the final best parameters obtained in (d), train the final model with the right data. [5%]\n",
    "\n",
    "# First we concatenate the train and validation datasets\n",
    "sumax = [xtr, xval]\n",
    "xtrain_val = pd.concat(sumax)\n",
    "\n",
    "sumay = [ytr, yval]\n",
    "ytrain_val = pd.concat(sumay)\n",
    "\n",
    "# Then we create the estimator with the best parameters obtained, and then fit\n",
    "rnd_clf = RandomForestClassifier(n_estimators=params[index_max][0], min_samples_split=params[index_max][1], max_features=params[index_max][2])\n",
    "rnd_clf.fit(xtrain_val, ytrain_val)\n",
    "\n",
    "# (e.2) Calculate the final test score. [5%]\n",
    "\n",
    "predicted = rnd_clf.predict(xtest)\n",
    "expected = ytest\n",
    "\n",
    "f1_partial = f1_score(expected, predicted, average='macro')\n",
    "print('f1 macro score: ' + str(f1_partial))\n",
    "\n",
    "\n",
    "#(e.3) Comparing this test score with the train/validation score obtained in (d.2.2) for that model, would you say that the winner model overfits? [5%]\n",
    "\n",
    "# We can anticipate overfitting due to the concatenation of train and validation datasets, because although we fixed the train imbalace,\n",
    "# we can't fix the validation dataset, so the concatenation of both will imbalance the result\n",
    "\n",
    "# The results obtained, indicate overfitting because the roc curve is 0.48, under 0.8, which would be a good result.\n",
    "auc = roc_auc_score(expected, predicted)\n",
    "print('The area under the ROC curve obtained is: ' + str(auc))\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}