2. Linear Regression/5. Advanced Regression techniques with Scikit-Learn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Linear Regression**: Human Brain Weight 🧠"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset was compiled using a medical study conducted on a group of people. This dataset shows a few variations of head sizes and masses of brains, it also consists additional gender and age group columns. The goal of the study is to predict the masses of the brains using the data, and later compare it with the actual masses of brains mentioned in the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Advanced Regression techniques with Scikit-Learn**\n",
    "\n",
    "$\\qquad$ <span style=\"color:gray\"><b>0.</b> Settings </span><br>\n",
    "$\\qquad$ <span style=\"color:gray\"><b>1.</b> Dataset </span><br>\n",
    "$\\qquad$ <span style=\"color:gray\"><b>2.</b> Data Preprocessing </span><br>\n",
    "$\\qquad$ <span style=\"color:gray\"><b>3.</b> Data Preparation </span><br>\n",
    "$\\qquad\\qquad$ <span style=\"color:gray\"><b>3.1.</b> One-Hot-Encoding </span><br>\n",
    "$\\qquad\\qquad$ <span style=\"color:gray\"><b>3.2.</b> Ordinal-Encoding </span><br>\n",
    "$\\qquad\\qquad$ <span style=\"color:gray\"><b>3.3.</b> Update </span><br>\n",
    "$\\qquad\\qquad$ <span style=\"color:gray\"><b>3.4.</b> Split in train and validation </span><br>\n",
    "$\\qquad$ <span style=\"color:gray\"><b>4.</b> Regression with Scikit-Learn </span><br>\n",
    "$\\qquad\\qquad$ <span style=\"color:gray\"><b>4.1.</b> Training </span><br>\n",
    "$\\qquad\\qquad$ <span style=\"color:gray\"><b>4.2.</b> Model choice </span><br>\n",
    "$\\qquad\\qquad$ <span style=\"color:gray\"><b>4.3.</b> Model evaluation </span><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **0.** Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Libraries\n",
    "from sklearn.preprocessing import OrdinalEncoder, OneHotEncoder\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import OrdinalEncoder\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "import matplotlib.pyplot as plt \n",
    "import seaborn as sns \n",
    "import pandas as pd  \n",
    "import numpy as np\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "# Metrics\n",
    "from sklearn.metrics import mean_squared_error, r2_score\n",
    "\n",
    "# Models\n",
    "from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet\n",
    "from sklearn.neighbors    import KNeighborsRegressor\n",
    "from sklearn.ensemble     import RandomForestRegressor, GradientBoostingRegressor, AdaBoostRegressor\n",
    "from sklearn.tree         import DecisionTreeRegressor\n",
    "from sklearn.svm          import SVR, LinearSVR\n",
    "from catboost             import CatBoostRegressor\n",
    "from xgboost              import XGBRegressor, XGBRFRegressor\n",
    "\n",
    "# Warnings off\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **1.** Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "    DATASET INFORMATIONS\n",
    "\n",
    "    |--------------|------------|-------|--------------------------------------------------------------|\n",
    "    | Name         | Data Type  | Meas. | Description                                                  |\n",
    "    |--------------|------------|-------|--------------------------------------------------------------|\n",
    "    | Gender       | nominal    |       | 1 represents Male , 2 represents Female                      |\n",
    "    | Age Range    | nominal    |       | 1 represents >18 years of age, 2 represents <18 years of age |\n",
    "    | Head Size    | continuous | cm^3  | Head volume                                                  |\n",
    "    | Brain Weight | continuous | grams | Mass of brains                                               |\n",
    "    |--------------|------------|-------|--------------------------------------------------------------|  \n",
    "\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Gender</th>\n",
       "      <th>Age Range</th>\n",
       "      <th>Head Size(cm^3)</th>\n",
       "      <th>Brain Weight(grams)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4512</td>\n",
       "      <td>1530</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3738</td>\n",
       "      <td>1297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4261</td>\n",
       "      <td>1335</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3777</td>\n",
       "      <td>1282</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4177</td>\n",
       "      <td>1590</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Gender  Age Range  Head Size(cm^3)  Brain Weight(grams)\n",
       "0       1          1             4512                 1530\n",
       "1       1          1             3738                 1297\n",
       "2       1          1             4261                 1335\n",
       "3       1          1             3777                 1282\n",
       "4       1          1             4177                 1590"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import the dataset\n",
    "data = pd.read_csv('./dataset.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Gender', 'Age Range', 'Head Size(cm^3)', 'Brain Weight(grams)'], dtype='object')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Gender', 'Age Range', 'Head Size', 'Brain Weight'], dtype='object')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns = ['Gender', 'Age Range', 'Head Size', 'Brain Weight']\n",
    "data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 237 entries, 0 to 236\n",
      "Data columns (total 4 columns):\n",
      " #   Column        Non-Null Count  Dtype\n",
      "---  ------        --------------  -----\n",
      " 0   Gender        237 non-null    int64\n",
      " 1   Age Range     237 non-null    int64\n",
      " 2   Head Size     237 non-null    int64\n",
      " 3   Brain Weight  237 non-null    int64\n",
      "dtypes: int64(4)\n",
      "memory usage: 7.5 KB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **2.** Data Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Gender          0\n",
       "Age Range       0\n",
       "Head Size       0\n",
       "Brain Weight    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Null elements\n",
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are no missing values in this dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **3.** Data Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = data[['Gender', 'Age Range', 'Head Size']]\n",
    "Y = data['Brain Weight']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Gender       int64\n",
       "Age Range    int64\n",
       "Head Size    int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is not good, the 'Age Range' and the 'Gender' are not numbers that should be considered as numbers, they are categories and should therefore be dummies-converted. Actually to 'Age Range' we can apply an Ordinal-Encoding (since it makes sense to give growth meaning), while to 'Gender' we apply a One-Hot-Encoder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **3.1.** One-Hot-Encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0  1\n",
       "0  1  0\n",
       "1  1  0\n",
       "2  1  0\n",
       "3  1  0\n",
       "4  1  0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## ONE-HOT-ENCODING: 'Gender'\n",
    "\n",
    "# Encoding\n",
    "gender_encoded = OneHotEncoder().fit_transform(X[['Gender']]);\n",
    "\n",
    "# Transform in a DataFrame\n",
    "gender_encoded = pd.DataFrame(gender_encoded.toarray()).astype(int)\n",
    "\n",
    "# Show results\n",
    "gender_encoded.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    1\n",
       "2    1\n",
       "3    1\n",
       "4    1\n",
       "Name: Gender, dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What is the association?\n",
    "X['Gender'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Male</th>\n",
       "      <th>Female</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>232</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>233</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>234</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>235</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>236</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>237 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Male  Female\n",
       "0       1       0\n",
       "1       1       0\n",
       "2       1       0\n",
       "3       1       0\n",
       "4       1       0\n",
       "..    ...     ...\n",
       "232     0       1\n",
       "233     0       1\n",
       "234     0       1\n",
       "235     0       1\n",
       "236     0       1\n",
       "\n",
       "[237 rows x 2 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# It is possible to conclude:\n",
    "#   1 -> Male   -> [1, 0]\n",
    "#   2 -> Female -> [0, 1]\n",
    "\n",
    "gender_encoded.columns = ['Male', 'Female']\n",
    "gender_encoded"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **3.2.** Ordinal-Encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0\n",
       "0  0\n",
       "1  0\n",
       "2  0\n",
       "3  0\n",
       "4  0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## ORDINAL-ENCODING: 'Age Range' \n",
    "\n",
    "# Encoding\n",
    "age_encoded = OrdinalEncoder().fit_transform(X[['Age Range']]);\n",
    "\n",
    "# Transform in a DataFrame\n",
    "age_encoded = pd.DataFrame(age_encoded).astype(int)\n",
    "\n",
    "# Show results\n",
    "age_encoded.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0\n",
       "0  1\n",
       "1  1\n",
       "2  1\n",
       "3  1\n",
       "4  1"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Perform a replacement to have: 0 == <18, 1 == >18\n",
    "age_encoded = age_encoded.replace({0:1, 1:0})\n",
    "age_encoded.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    1\n",
       "2    1\n",
       "3    1\n",
       "4    1\n",
       "Name: Age Range, dtype: int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What is the association?\n",
    "X['Age Range'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# It is possible to conclude:\n",
    "#   1 -> >18 -> 1\n",
    "#   2 -> <18 -> 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **3.3.** Update"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age Range</th>\n",
       "      <th>Head Size</th>\n",
       "      <th>Male</th>\n",
       "      <th>Female</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>4512</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>3738</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>4261</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3777</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>4177</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>232</th>\n",
       "      <td>0</td>\n",
       "      <td>3214</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>233</th>\n",
       "      <td>0</td>\n",
       "      <td>3394</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>234</th>\n",
       "      <td>0</td>\n",
       "      <td>3233</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>235</th>\n",
       "      <td>0</td>\n",
       "      <td>3352</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>236</th>\n",
       "      <td>0</td>\n",
       "      <td>3391</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>237 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Age Range  Head Size  Male  Female\n",
       "0            1       4512     1       0\n",
       "1            1       3738     1       0\n",
       "2            1       4261     1       0\n",
       "3            1       3777     1       0\n",
       "4            1       4177     1       0\n",
       "..         ...        ...   ...     ...\n",
       "232          0       3214     0       1\n",
       "233          0       3394     0       1\n",
       "234          0       3233     0       1\n",
       "235          0       3352     0       1\n",
       "236          0       3391     0       1\n",
       "\n",
       "[237 rows x 4 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add 'Gender' encoded\n",
    "X = pd.concat([X, gender_encoded], axis=1)\n",
    "\n",
    "# Drop the old 'Gender'  \n",
    "X = X.drop('Gender', axis=1) \n",
    "\n",
    "# Substitute 'Age Range' with 'age_encoded'\n",
    "X['Age Range'] = age_encoded\n",
    "\n",
    "# Check\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **3.4.** Split in train and validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(189, 4)\n",
      "(48, 4)\n",
      "(189,)\n",
      "(48,)\n"
     ]
    }
   ],
   "source": [
    "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2, random_state=5)\n",
    "print(X_train.shape)\n",
    "print(X_test.shape)\n",
    "print(Y_train.shape)\n",
    "print(Y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **4.** Regression with **Scikit-Learn**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **4.1.** Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Advanced model definition\n",
    "models = {\n",
    "    'Linear'                    : LinearRegression(),\n",
    "    'Ridge'                     : Ridge(),\n",
    "    'Lasso'                     : Lasso(),\n",
    "    'ElasticNet'                : ElasticNet(),\n",
    "    'KNeighborsRegressor'       : KNeighborsRegressor(),\n",
    "    'SVR'                       : SVR(),\n",
    "    'LinearSVR'                 : LinearSVR(),\n",
    "    'DecisionTreeRegressor'     : DecisionTreeRegressor(),\n",
    "    'RandomForestRegressor'     : RandomForestRegressor(),\n",
    "    'GradientBoostingRegressor' : GradientBoostingRegressor(),\n",
    "    'AdaBoostRegressor'         : AdaBoostRegressor(),\n",
    "    'XGBRegressor'              : XGBRegressor(objective='reg:squarederror'),\n",
    "    'XGBRFRegressor'            : XGBRFRegressor(objective='reg:squarederror'),\n",
    "    'CatBoostRegressor'         : CatBoostRegressor(verbose=0)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_models(models, X_train, X_test, Y_train, Y_test):\n",
    "\n",
    "    np.random.seed(42)\n",
    "\n",
    "    count_models = 0 \n",
    "\n",
    "    for name, model in models.items():\n",
    "\n",
    "        # Training/Fitting\n",
    "        model.fit(X_train, Y_train)\n",
    "\n",
    "        # Predictions (on the training set)\n",
    "        Y_pred_train = model.predict(X_train)\n",
    "       \n",
    "        # Metrics\n",
    "        model_train_score = model.score(X_train, Y_train)\n",
    "        model_test_score  = model.score(X_test, Y_test)\n",
    "        model_mse         = mean_squared_error(Y_train, Y_pred_train)\n",
    "        model_rmse        = np.sqrt(mean_squared_error(Y_train, Y_pred_train))\n",
    "        model_mae         = mean_absolute_error(Y_train, Y_pred_train)\n",
    "        model_r2          = r2_score(Y_train, Y_pred_train)\n",
    "\n",
    "        # If this is the first model trained we create the DataFrame 'results'\n",
    "        if count_models == 0:\n",
    "            results = pd.DataFrame([\n",
    "                model_train_score, \n",
    "                model_test_score, \n",
    "                model_mse, \n",
    "                model_rmse, \n",
    "                model_mae, \n",
    "                model_r2],\n",
    "                index = ['Train Accuracy', 'Test Accuracy', 'MSE', 'RMSE', 'MAE', 'R2'],\n",
    "                columns = [name])\n",
    "\n",
    "        # otherwise we just append the results to the already-existsing DataFrame 'results'\n",
    "        else:\n",
    "            results[name] = [\n",
    "                model_train_score, \n",
    "                model_test_score, \n",
    "                model_mse, \n",
    "                model_rmse, \n",
    "                model_mae, \n",
    "                model_r2]\n",
    "\n",
    "        count_models += 1\n",
    "        \n",
    "    print(f'Trained {count_models} models')\n",
    "\n",
    "    return results.T\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trained 14 models\n"
     ]
    }
   ],
   "source": [
    "results = train_models(models, X_train, X_test, Y_train, Y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **4.2.** Model choice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Train Accuracy</th>\n",
       "      <th>Test Accuracy</th>\n",
       "      <th>MSE</th>\n",
       "      <th>RMSE</th>\n",
       "      <th>MAE</th>\n",
       "      <th>R2</th>\n",
       "      <th>Train-Test Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Linear</th>\n",
       "      <td>0.614110</td>\n",
       "      <td>0.776742</td>\n",
       "      <td>5403.786805</td>\n",
       "      <td>73.510454</td>\n",
       "      <td>57.113235</td>\n",
       "      <td>0.614110</td>\n",
       "      <td>-0.162632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ridge</th>\n",
       "      <td>0.614105</td>\n",
       "      <td>0.776455</td>\n",
       "      <td>5403.858223</td>\n",
       "      <td>73.510939</td>\n",
       "      <td>57.101169</td>\n",
       "      <td>0.614105</td>\n",
       "      <td>-0.162350</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lasso</th>\n",
       "      <td>0.613298</td>\n",
       "      <td>0.770211</td>\n",
       "      <td>5415.164016</td>\n",
       "      <td>73.587798</td>\n",
       "      <td>56.962264</td>\n",
       "      <td>0.613298</td>\n",
       "      <td>-0.156913</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ElasticNet</th>\n",
       "      <td>0.608830</td>\n",
       "      <td>0.762208</td>\n",
       "      <td>5477.733661</td>\n",
       "      <td>74.011713</td>\n",
       "      <td>57.154729</td>\n",
       "      <td>0.608830</td>\n",
       "      <td>-0.153378</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>KNeighborsRegressor</th>\n",
       "      <td>0.684380</td>\n",
       "      <td>0.660024</td>\n",
       "      <td>4419.768677</td>\n",
       "      <td>66.481341</td>\n",
       "      <td>51.960847</td>\n",
       "      <td>0.684380</td>\n",
       "      <td>0.024356</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SVR</th>\n",
       "      <td>0.034419</td>\n",
       "      <td>0.040464</td>\n",
       "      <td>13521.465522</td>\n",
       "      <td>116.281837</td>\n",
       "      <td>91.570459</td>\n",
       "      <td>0.034419</td>\n",
       "      <td>-0.006045</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LinearSVR</th>\n",
       "      <td>-0.633724</td>\n",
       "      <td>-0.588115</td>\n",
       "      <td>22877.776085</td>\n",
       "      <td>151.254012</td>\n",
       "      <td>128.258052</td>\n",
       "      <td>-0.633724</td>\n",
       "      <td>-0.045610</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DecisionTreeRegressor</th>\n",
       "      <td>0.980595</td>\n",
       "      <td>0.498853</td>\n",
       "      <td>271.740741</td>\n",
       "      <td>16.484561</td>\n",
       "      <td>2.571429</td>\n",
       "      <td>0.980595</td>\n",
       "      <td>0.481742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RandomForestRegressor</th>\n",
       "      <td>0.913179</td>\n",
       "      <td>0.625193</td>\n",
       "      <td>1215.787015</td>\n",
       "      <td>34.868138</td>\n",
       "      <td>25.537453</td>\n",
       "      <td>0.913179</td>\n",
       "      <td>0.287986</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GradientBoostingRegressor</th>\n",
       "      <td>0.855369</td>\n",
       "      <td>0.665128</td>\n",
       "      <td>2025.335071</td>\n",
       "      <td>45.003723</td>\n",
       "      <td>33.676275</td>\n",
       "      <td>0.855369</td>\n",
       "      <td>0.190241</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AdaBoostRegressor</th>\n",
       "      <td>0.647325</td>\n",
       "      <td>0.702923</td>\n",
       "      <td>4938.666389</td>\n",
       "      <td>70.275646</td>\n",
       "      <td>54.551859</td>\n",
       "      <td>0.647325</td>\n",
       "      <td>-0.055598</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>XGBRegressor</th>\n",
       "      <td>0.977133</td>\n",
       "      <td>0.594989</td>\n",
       "      <td>320.219044</td>\n",
       "      <td>17.894665</td>\n",
       "      <td>7.773017</td>\n",
       "      <td>0.977133</td>\n",
       "      <td>0.382144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>XGBRFRegressor</th>\n",
       "      <td>0.820068</td>\n",
       "      <td>0.686390</td>\n",
       "      <td>2519.670062</td>\n",
       "      <td>50.196315</td>\n",
       "      <td>38.646878</td>\n",
       "      <td>0.820068</td>\n",
       "      <td>0.133678</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CatBoostRegressor</th>\n",
       "      <td>0.882014</td>\n",
       "      <td>0.643639</td>\n",
       "      <td>1652.215261</td>\n",
       "      <td>40.647451</td>\n",
       "      <td>31.404458</td>\n",
       "      <td>0.882014</td>\n",
       "      <td>0.238375</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           Train Accuracy  Test Accuracy           MSE  \\\n",
       "Linear                           0.614110       0.776742   5403.786805   \n",
       "Ridge                            0.614105       0.776455   5403.858223   \n",
       "Lasso                            0.613298       0.770211   5415.164016   \n",
       "ElasticNet                       0.608830       0.762208   5477.733661   \n",
       "KNeighborsRegressor              0.684380       0.660024   4419.768677   \n",
       "SVR                              0.034419       0.040464  13521.465522   \n",
       "LinearSVR                       -0.633724      -0.588115  22877.776085   \n",
       "DecisionTreeRegressor            0.980595       0.498853    271.740741   \n",
       "RandomForestRegressor            0.913179       0.625193   1215.787015   \n",
       "GradientBoostingRegressor        0.855369       0.665128   2025.335071   \n",
       "AdaBoostRegressor                0.647325       0.702923   4938.666389   \n",
       "XGBRegressor                     0.977133       0.594989    320.219044   \n",
       "XGBRFRegressor                   0.820068       0.686390   2519.670062   \n",
       "CatBoostRegressor                0.882014       0.643639   1652.215261   \n",
       "\n",
       "                                 RMSE         MAE        R2  \\\n",
       "Linear                      73.510454   57.113235  0.614110   \n",
       "Ridge                       73.510939   57.101169  0.614105   \n",
       "Lasso                       73.587798   56.962264  0.613298   \n",
       "ElasticNet                  74.011713   57.154729  0.608830   \n",
       "KNeighborsRegressor         66.481341   51.960847  0.684380   \n",
       "SVR                        116.281837   91.570459  0.034419   \n",
       "LinearSVR                  151.254012  128.258052 -0.633724   \n",
       "DecisionTreeRegressor       16.484561    2.571429  0.980595   \n",
       "RandomForestRegressor       34.868138   25.537453  0.913179   \n",
       "GradientBoostingRegressor   45.003723   33.676275  0.855369   \n",
       "AdaBoostRegressor           70.275646   54.551859  0.647325   \n",
       "XGBRegressor                17.894665    7.773017  0.977133   \n",
       "XGBRFRegressor              50.196315   38.646878  0.820068   \n",
       "CatBoostRegressor           40.647451   31.404458  0.882014   \n",
       "\n",
       "                           Train-Test Accuracy  \n",
       "Linear                               -0.162632  \n",
       "Ridge                                -0.162350  \n",
       "Lasso                                -0.156913  \n",
       "ElasticNet                           -0.153378  \n",
       "KNeighborsRegressor                   0.024356  \n",
       "SVR                                  -0.006045  \n",
       "LinearSVR                            -0.045610  \n",
       "DecisionTreeRegressor                 0.481742  \n",
       "RandomForestRegressor                 0.287986  \n",
       "GradientBoostingRegressor             0.190241  \n",
       "AdaBoostRegressor                    -0.055598  \n",
       "XGBRegressor                          0.382144  \n",
       "XGBRFRegressor                        0.133678  \n",
       "CatBoostRegressor                     0.238375  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Overall results\n",
    "results['Train-Test Accuracy'] = results['Train Accuracy'] - results['Test Accuracy']\n",
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Metrics to maximize: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_2956e_row0_col1, #T_2956e_row7_col0, #T_2956e_row7_col2 {\n",
       "  background-color: lightgreen;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_2956e\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_2956e_level0_col0\" class=\"col_heading level0 col0\" >Train Accuracy</th>\n",
       "      <th id=\"T_2956e_level0_col1\" class=\"col_heading level0 col1\" >Test Accuracy</th>\n",
       "      <th id=\"T_2956e_level0_col2\" class=\"col_heading level0 col2\" >R2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_2956e_level0_row0\" class=\"row_heading level0 row0\" >Linear</th>\n",
       "      <td id=\"T_2956e_row0_col0\" class=\"data row0 col0\" >0.614110</td>\n",
       "      <td id=\"T_2956e_row0_col1\" class=\"data row0 col1\" >0.776742</td>\n",
       "      <td id=\"T_2956e_row0_col2\" class=\"data row0 col2\" >0.614110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_2956e_level0_row1\" class=\"row_heading level0 row1\" >Ridge</th>\n",
       "      <td id=\"T_2956e_row1_col0\" class=\"data row1 col0\" >0.614105</td>\n",
       "      <td id=\"T_2956e_row1_col1\" class=\"data row1 col1\" >0.776455</td>\n",
       "      <td id=\"T_2956e_row1_col2\" class=\"data row1 col2\" >0.614105</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_2956e_level0_row2\" class=\"row_heading level0 row2\" >Lasso</th>\n",
       "      <td id=\"T_2956e_row2_col0\" class=\"data row2 col0\" >0.613298</td>\n",
       "      <td id=\"T_2956e_row2_col1\" class=\"data row2 col1\" >0.770211</td>\n",
       "      <td id=\"T_2956e_row2_col2\" class=\"data row2 col2\" >0.613298</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_2956e_level0_row3\" class=\"row_heading level0 row3\" >ElasticNet</th>\n",
       "      <td id=\"T_2956e_row3_col0\" class=\"data row3 col0\" >0.608830</td>\n",
       "      <td id=\"T_2956e_row3_col1\" class=\"data row3 col1\" >0.762208</td>\n",
       "      <td id=\"T_2956e_row3_col2\" class=\"data row3 col2\" >0.608830</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_2956e_level0_row4\" class=\"row_heading level0 row4\" >KNeighborsRegressor</th>\n",
       "      <td id=\"T_2956e_row4_col0\" class=\"data row4 col0\" >0.684380</td>\n",
       "      <td id=\"T_2956e_row4_col1\" class=\"data row4 col1\" >0.660024</td>\n",
       "      <td id=\"T_2956e_row4_col2\" class=\"data row4 col2\" >0.684380</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_2956e_level0_row5\" class=\"row_heading level0 row5\" >SVR</th>\n",
       "      <td id=\"T_2956e_row5_col0\" class=\"data row5 col0\" >0.034419</td>\n",
       "      <td id=\"T_2956e_row5_col1\" class=\"data row5 col1\" >0.040464</td>\n",
       "      <td id=\"T_2956e_row5_col2\" class=\"data row5 col2\" >0.034419</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_2956e_level0_row6\" class=\"row_heading level0 row6\" >LinearSVR</th>\n",
       "      <td id=\"T_2956e_row6_col0\" class=\"data row6 col0\" >-0.633724</td>\n",
       "      <td id=\"T_2956e_row6_col1\" class=\"data row6 col1\" >-0.588115</td>\n",
       "      <td id=\"T_2956e_row6_col2\" class=\"data row6 col2\" >-0.633724</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_2956e_level0_row7\" class=\"row_heading level0 row7\" >DecisionTreeRegressor</th>\n",
       "      <td id=\"T_2956e_row7_col0\" class=\"data row7 col0\" >0.980595</td>\n",
       "      <td id=\"T_2956e_row7_col1\" class=\"data row7 col1\" >0.498853</td>\n",
       "      <td id=\"T_2956e_row7_col2\" class=\"data row7 col2\" >0.980595</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_2956e_level0_row8\" class=\"row_heading level0 row8\" >RandomForestRegressor</th>\n",
       "      <td id=\"T_2956e_row8_col0\" class=\"data row8 col0\" >0.913179</td>\n",
       "      <td id=\"T_2956e_row8_col1\" class=\"data row8 col1\" >0.625193</td>\n",
       "      <td id=\"T_2956e_row8_col2\" class=\"data row8 col2\" >0.913179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_2956e_level0_row9\" class=\"row_heading level0 row9\" >GradientBoostingRegressor</th>\n",
       "      <td id=\"T_2956e_row9_col0\" class=\"data row9 col0\" >0.855369</td>\n",
       "      <td id=\"T_2956e_row9_col1\" class=\"data row9 col1\" >0.665128</td>\n",
       "      <td id=\"T_2956e_row9_col2\" class=\"data row9 col2\" >0.855369</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_2956e_level0_row10\" class=\"row_heading level0 row10\" >AdaBoostRegressor</th>\n",
       "      <td id=\"T_2956e_row10_col0\" class=\"data row10 col0\" >0.647325</td>\n",
       "      <td id=\"T_2956e_row10_col1\" class=\"data row10 col1\" >0.702923</td>\n",
       "      <td id=\"T_2956e_row10_col2\" class=\"data row10 col2\" >0.647325</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_2956e_level0_row11\" class=\"row_heading level0 row11\" >XGBRegressor</th>\n",
       "      <td id=\"T_2956e_row11_col0\" class=\"data row11 col0\" >0.977133</td>\n",
       "      <td id=\"T_2956e_row11_col1\" class=\"data row11 col1\" >0.594989</td>\n",
       "      <td id=\"T_2956e_row11_col2\" class=\"data row11 col2\" >0.977133</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_2956e_level0_row12\" class=\"row_heading level0 row12\" >XGBRFRegressor</th>\n",
       "      <td id=\"T_2956e_row12_col0\" class=\"data row12 col0\" >0.820068</td>\n",
       "      <td id=\"T_2956e_row12_col1\" class=\"data row12 col1\" >0.686390</td>\n",
       "      <td id=\"T_2956e_row12_col2\" class=\"data row12 col2\" >0.820068</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_2956e_level0_row13\" class=\"row_heading level0 row13\" >CatBoostRegressor</th>\n",
       "      <td id=\"T_2956e_row13_col0\" class=\"data row13 col0\" >0.882014</td>\n",
       "      <td id=\"T_2956e_row13_col1\" class=\"data row13 col1\" >0.643639</td>\n",
       "      <td id=\"T_2956e_row13_col2\" class=\"data row13 col2\" >0.882014</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x1f7b5b29970>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics_to_maximize = ['Train Accuracy', 'Test Accuracy', 'R2']\n",
    "metrics_to_minimize = ['MSE', 'RMSE', 'MAE', 'Train-Test Accuracy']\n",
    "\n",
    "print('Metrics to maximize: ')\n",
    "results[metrics_to_maximize].style.highlight_max(color= 'lightgreen', axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Metrics to minimize: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_0abf9_row0_col3, #T_0abf9_row7_col0, #T_0abf9_row7_col1, #T_0abf9_row7_col2 {\n",
       "  background-color: lightgreen;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_0abf9\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_0abf9_level0_col0\" class=\"col_heading level0 col0\" >MSE</th>\n",
       "      <th id=\"T_0abf9_level0_col1\" class=\"col_heading level0 col1\" >RMSE</th>\n",
       "      <th id=\"T_0abf9_level0_col2\" class=\"col_heading level0 col2\" >MAE</th>\n",
       "      <th id=\"T_0abf9_level0_col3\" class=\"col_heading level0 col3\" >Train-Test Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_0abf9_level0_row0\" class=\"row_heading level0 row0\" >Linear</th>\n",
       "      <td id=\"T_0abf9_row0_col0\" class=\"data row0 col0\" >5403.786805</td>\n",
       "      <td id=\"T_0abf9_row0_col1\" class=\"data row0 col1\" >73.510454</td>\n",
       "      <td id=\"T_0abf9_row0_col2\" class=\"data row0 col2\" >57.113235</td>\n",
       "      <td id=\"T_0abf9_row0_col3\" class=\"data row0 col3\" >-0.162632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_0abf9_level0_row1\" class=\"row_heading level0 row1\" >Ridge</th>\n",
       "      <td id=\"T_0abf9_row1_col0\" class=\"data row1 col0\" >5403.858223</td>\n",
       "      <td id=\"T_0abf9_row1_col1\" class=\"data row1 col1\" >73.510939</td>\n",
       "      <td id=\"T_0abf9_row1_col2\" class=\"data row1 col2\" >57.101169</td>\n",
       "      <td id=\"T_0abf9_row1_col3\" class=\"data row1 col3\" >-0.162350</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_0abf9_level0_row2\" class=\"row_heading level0 row2\" >Lasso</th>\n",
       "      <td id=\"T_0abf9_row2_col0\" class=\"data row2 col0\" >5415.164016</td>\n",
       "      <td id=\"T_0abf9_row2_col1\" class=\"data row2 col1\" >73.587798</td>\n",
       "      <td id=\"T_0abf9_row2_col2\" class=\"data row2 col2\" >56.962264</td>\n",
       "      <td id=\"T_0abf9_row2_col3\" class=\"data row2 col3\" >-0.156913</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_0abf9_level0_row3\" class=\"row_heading level0 row3\" >ElasticNet</th>\n",
       "      <td id=\"T_0abf9_row3_col0\" class=\"data row3 col0\" >5477.733661</td>\n",
       "      <td id=\"T_0abf9_row3_col1\" class=\"data row3 col1\" >74.011713</td>\n",
       "      <td id=\"T_0abf9_row3_col2\" class=\"data row3 col2\" >57.154729</td>\n",
       "      <td id=\"T_0abf9_row3_col3\" class=\"data row3 col3\" >-0.153378</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_0abf9_level0_row4\" class=\"row_heading level0 row4\" >KNeighborsRegressor</th>\n",
       "      <td id=\"T_0abf9_row4_col0\" class=\"data row4 col0\" >4419.768677</td>\n",
       "      <td id=\"T_0abf9_row4_col1\" class=\"data row4 col1\" >66.481341</td>\n",
       "      <td id=\"T_0abf9_row4_col2\" class=\"data row4 col2\" >51.960847</td>\n",
       "      <td id=\"T_0abf9_row4_col3\" class=\"data row4 col3\" >0.024356</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_0abf9_level0_row5\" class=\"row_heading level0 row5\" >SVR</th>\n",
       "      <td id=\"T_0abf9_row5_col0\" class=\"data row5 col0\" >13521.465522</td>\n",
       "      <td id=\"T_0abf9_row5_col1\" class=\"data row5 col1\" >116.281837</td>\n",
       "      <td id=\"T_0abf9_row5_col2\" class=\"data row5 col2\" >91.570459</td>\n",
       "      <td id=\"T_0abf9_row5_col3\" class=\"data row5 col3\" >-0.006045</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_0abf9_level0_row6\" class=\"row_heading level0 row6\" >LinearSVR</th>\n",
       "      <td id=\"T_0abf9_row6_col0\" class=\"data row6 col0\" >22877.776085</td>\n",
       "      <td id=\"T_0abf9_row6_col1\" class=\"data row6 col1\" >151.254012</td>\n",
       "      <td id=\"T_0abf9_row6_col2\" class=\"data row6 col2\" >128.258052</td>\n",
       "      <td id=\"T_0abf9_row6_col3\" class=\"data row6 col3\" >-0.045610</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_0abf9_level0_row7\" class=\"row_heading level0 row7\" >DecisionTreeRegressor</th>\n",
       "      <td id=\"T_0abf9_row7_col0\" class=\"data row7 col0\" >271.740741</td>\n",
       "      <td id=\"T_0abf9_row7_col1\" class=\"data row7 col1\" >16.484561</td>\n",
       "      <td id=\"T_0abf9_row7_col2\" class=\"data row7 col2\" >2.571429</td>\n",
       "      <td id=\"T_0abf9_row7_col3\" class=\"data row7 col3\" >0.481742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_0abf9_level0_row8\" class=\"row_heading level0 row8\" >RandomForestRegressor</th>\n",
       "      <td id=\"T_0abf9_row8_col0\" class=\"data row8 col0\" >1215.787015</td>\n",
       "      <td id=\"T_0abf9_row8_col1\" class=\"data row8 col1\" >34.868138</td>\n",
       "      <td id=\"T_0abf9_row8_col2\" class=\"data row8 col2\" >25.537453</td>\n",
       "      <td id=\"T_0abf9_row8_col3\" class=\"data row8 col3\" >0.287986</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_0abf9_level0_row9\" class=\"row_heading level0 row9\" >GradientBoostingRegressor</th>\n",
       "      <td id=\"T_0abf9_row9_col0\" class=\"data row9 col0\" >2025.335071</td>\n",
       "      <td id=\"T_0abf9_row9_col1\" class=\"data row9 col1\" >45.003723</td>\n",
       "      <td id=\"T_0abf9_row9_col2\" class=\"data row9 col2\" >33.676275</td>\n",
       "      <td id=\"T_0abf9_row9_col3\" class=\"data row9 col3\" >0.190241</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_0abf9_level0_row10\" class=\"row_heading level0 row10\" >AdaBoostRegressor</th>\n",
       "      <td id=\"T_0abf9_row10_col0\" class=\"data row10 col0\" >4938.666389</td>\n",
       "      <td id=\"T_0abf9_row10_col1\" class=\"data row10 col1\" >70.275646</td>\n",
       "      <td id=\"T_0abf9_row10_col2\" class=\"data row10 col2\" >54.551859</td>\n",
       "      <td id=\"T_0abf9_row10_col3\" class=\"data row10 col3\" >-0.055598</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_0abf9_level0_row11\" class=\"row_heading level0 row11\" >XGBRegressor</th>\n",
       "      <td id=\"T_0abf9_row11_col0\" class=\"data row11 col0\" >320.219044</td>\n",
       "      <td id=\"T_0abf9_row11_col1\" class=\"data row11 col1\" >17.894665</td>\n",
       "      <td id=\"T_0abf9_row11_col2\" class=\"data row11 col2\" >7.773017</td>\n",
       "      <td id=\"T_0abf9_row11_col3\" class=\"data row11 col3\" >0.382144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_0abf9_level0_row12\" class=\"row_heading level0 row12\" >XGBRFRegressor</th>\n",
       "      <td id=\"T_0abf9_row12_col0\" class=\"data row12 col0\" >2519.670062</td>\n",
       "      <td id=\"T_0abf9_row12_col1\" class=\"data row12 col1\" >50.196315</td>\n",
       "      <td id=\"T_0abf9_row12_col2\" class=\"data row12 col2\" >38.646878</td>\n",
       "      <td id=\"T_0abf9_row12_col3\" class=\"data row12 col3\" >0.133678</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_0abf9_level0_row13\" class=\"row_heading level0 row13\" >CatBoostRegressor</th>\n",
       "      <td id=\"T_0abf9_row13_col0\" class=\"data row13 col0\" >1652.215261</td>\n",
       "      <td id=\"T_0abf9_row13_col1\" class=\"data row13 col1\" >40.647451</td>\n",
       "      <td id=\"T_0abf9_row13_col2\" class=\"data row13 col2\" >31.404458</td>\n",
       "      <td id=\"T_0abf9_row13_col3\" class=\"data row13 col3\" >0.238375</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x1f7b5b49640>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Metrics to minimize: ')\n",
    "results[metrics_to_minimize].style.highlight_min(color='lightgreen', axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **4.3.** Model evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training of the model\n",
    "model = LinearRegression()\n",
    "model.fit(X_train, Y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Coefficients analysis\n",
    "\n",
    "coefficients = {}\n",
    "\n",
    "# Save the coefficients\n",
    "for i in range(len(model.coef_)):\n",
    "    coefficients[X.columns[i]] = model.coef_[i]\n",
    "\n",
    "# Add the intercept\n",
    "coefficients['Intercept'] = model.intercept_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age Range</th>\n",
       "      <th>Head Size</th>\n",
       "      <th>Male</th>\n",
       "      <th>Female</th>\n",
       "      <th>Intercept</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22.562647</td>\n",
       "      <td>0.242566</td>\n",
       "      <td>7.104745</td>\n",
       "      <td>-7.104745</td>\n",
       "      <td>393.024595</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Age Range  Head Size      Male    Female   Intercept\n",
       "0  22.562647   0.242566  7.104745 -7.104745  393.024595"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coefficients_dataframe = pd.DataFrame(coefficients.values()).T\n",
    "coefficients_dataframe.columns = coefficients.keys()\n",
    "coefficients_dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA20AAAE/CAYAAADVKysfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAl/ElEQVR4nO3de7xldV3/8dcbBpHSROREIwyOKVloOeqEKKYkakgp+EsNfipo1Ngv8PLzrpVBaWlKlKn0IyHGSyJ5CTJSibgIeZtB7mhMCjLjAEdFRE2K4fP7Y39Pbo7nzDlnzjl7r33O6/l47Mf+ru/6rrU+e52993d/1vqudVJVSJIkSZK6aadhByBJkiRJmp5JmyRJkiR1mEmbJEmSJHWYSZskSZIkdZhJmyRJkiR1mEmbJEmSJHWYSZuWvSS/lOTL25m/OkklWTHIuCRJWgz2e9LoMWnTkpPk9Un+eVLd9dPUHVlVn66qh/XV35DkKfPY/hlJ3jTLtickef+Obmux2GFL0ugYdr83x1hn7CNb//PQWa5vYLHPRVf7d40ukzYtRRcDj0+yM0CSlcAuwKMm1T20tR1ZJlWSJJZRv7eQ0uNvYY0E36hair5Ar7Na06Z/CbgA+PKkuv+oqq8nOTjJZoAk7wP2Bf4xyXeTvKZvvc9L8rUk30jye7MJpO+M1TGTl01yKPAG4Dfatq5o9fdLclqSrUm2JHlTX6f7wiSXJjk5yTeBE5LsluSkJDcmuT3JJUl2a+0PTPJvSb6d5IokB/fFdmGSP03y+STfSXJ2kj3a7IlO/dsttsfN5vVKkoZioP1ekl2T/EWSr7fHXyTZtc17YZJL+oObOHOWZB3wPOA1bVv/ONMLa2eszkry3iR3JLkmydrtxT6Lvu/NSS4Fvg/8dJKHJzkvybeS3JLkDa3tTklel+Q/knyzxbFHmzfRv69r+2Brkle1eVP279J8mLRpyamq/wI+BzyxVT0R+DRwyaS6HznaWFUvAL4GPKOq7lNVf9Y3+wnAw4BDgDcm+bk5hPUjy1bVJ4A/AT7UtvXI1vYM4C56R0QfBTwN+K2+dT0W+AqwF/Bm4O3AY4DHA3sArwHuTrI38E/Am1r9q4CPJBnrW9fRwG8CK9s239G3fwB2b7F9Zg6vVZI0QEPo934POJBeQvhI4ADg92cR56nAB4A/a9t6xixf4jOBM4HdgXOAd04X+yz7vhcA64D7ArcA/wJ8Anggvb73/NbuJcARwJPavNuAd02K7ZeB/ej11a9N8pTt9O/SDjNp01J1ET/sqH6JXuf16Ul1F81xnSdW1X9W1RXAFfQ6qgVdNslewGHAy6vqe1V1K3AycGRfs69X1V9V1V3AnfSSrpdV1Zaq2lZV/1ZVdwLPB86tqnOr6u6qOg/Y0NY/4X1VdXVVfQ/4A+C5E2f1JEkjZZD93vOAP6qqW6tqHDiRXiK0WC5pfdk24H1sv/+dTd93RlVd0/rRXwNurqqTquoHVXVHVX2utfsd4PeqanPrV08Anp17XppwYuuvrwL+FjhqQV6xNInXw2ipuhg4rg1jGKuq65PcAqxvdY9g7uP6b+4rfx+4zyIs+yB6Q1y2Jpmo2wm4qa9Nf3lP4N7Af0yzruck6T+SuQu9ITNTrevGNn/PaWKTJHXXIPu9B9LrMybc2OoWy+Q47p1kRUu6Jptr37eKqfvQiXV9LMndfXXb6I10mWpdNwI/P826pHkxadNS9RngfsBvA5cCVNV3kny91X29qr46zbI1mBCn3NZN9M6e7TlNZzR5mW8APwAeQu8o6OR1va+qfns721/VV94X+O+2zn1miFuS1C2D7Pe+Ti+huaZN79vqAL4H/NhEwyQ/Nc9tzWSqfnSmvq9/mZu452iWyev6zaq6dPKMJKtbcRXwpVbu3w+D/C2hZcDhkVqSquo/6Q2HeAW94SETLml12zvaeAvw04sX3Y9sa3Xa3auqaivwKeCkJD/RLoJ+SJInTbVwVd0NnA78eZIHJtk5yePaBeHvB56R5Fda/b3bxef9Cdnzk+yf5MeAPwI+3IafjAN3M7j9IEmahwH3ex8Efj/JWJI9gTfS63OgdwDx4UnWJLk3vSGF89nWTCavbzZ9X7+PAyuTvLzdYOW+SR7b5v018OYkDwJor/fwScv/QZIfS/Jw4EXAh/ri+p/+XZov30hayi4CfpJehzXh061ue53Xn9LrjL49cSeoRfT37fmbSS5r5aOBewHX0rvo+cP0bhQynVcBV9G7e9i3gLcCO1XVTcDh9O5gNU7viOGruefn/n30bnxyM71hli8FqKrv07vJyaVtPxw4r1cpSRqEQfV7b6KXIF5Jr/+5rNVRVf9O7yDgvwDXT4oF4DRg/7atf5jFtmZyj9hn2ff9j6q6A3gq8Ax6feH19G4uAvCX9G588qkkdwCfpXczsH4XAZvo3bzk7VX1qVY/Vf8u7bBUefZWWo6SXAi8v6reM+xYJEkaJW145FeBXbZzOYO0YDzTJkmSJEkdZtImSZIkSR3m8EhJkiRJ6jDPtEmSJElSh5m0SZIkSVKHdeKfa++55561evXqYYchSRqAjRs3fqOqxoYdx6iwj5Sk5WF7/WMnkrbVq1ezYcOGYYchSRqAJDcOO4ZRYh8pScvD9vpHh0dKkiRJUoeZtEmSJElSh806aUuyc5IvJvl4m35wks8l2ZTkQ0nu1ep3bdOb2vzVixS7JEmSJC15cznT9jLgur7ptwInV9VDgduAY1v9scBtrf7k1k6SJEmStANmlbQl2Qf4VeA9bTrAk4EPtybrgSNa+fA2TZt/SGsvSZIkSZqj2Z5p+wvgNcDdbfoBwLer6q42vRnYu5X3Bm4CaPNvb+0lSZIkSXM0Y9KW5NeAW6tq40JuOMm6JBuSbBgfH1/IVUuSJEnSkjGbM20HAc9McgNwJr1hkX8J7J5k4v+87QNsaeUtwCqANv9+wDcnr7SqTq2qtVW1dmzM/7EqSZIkSVOZMWmrqtdX1T5VtRo4EvjXqnoecAHw7NbsGODsVj6nTdPm/2tV1YJGLUmSJEnLxHz+T9trgVck2UTvmrXTWv1pwANa/SuA180vREmSJElavlbM3OSHqupC4MJW/gpwwBRtfgA8ZwFik6SR985X/uOwQxi44096xrBDWHYe8+r3DjuEkbLxbUcPOwRJmpP5nGmTJEmSJC0ykzZJkiRJ6jCTNkmSJEnqMJM2SZIkSeowkzZJkiRJ6jCTNkmSJEnqMJM2SZIkSeowkzZJkiRJ6jCTNkmSJEnqMJM2SZIkSeowkzZJkiRJ6jCTNkmSJEnqMJM2SZIkSeowkzZJkiRJ6jCTNkmSJEnqMJM2SZIkSeowkzZJkiRJ6jCTNkmSFliSVUkuSHJtkmuSvKzV75HkvCTXt+f7DztWSVL3mbRJkrTw7gJeWVX7AwcCxyXZH3gdcH5V7Qec36YlSdoukzZJkhZYVW2tqsta+Q7gOmBv4HBgfWu2HjhiKAFKkkaKSZskSYsoyWrgUcDngL2qamubdTOw17DikiSNDpM2SZIWSZL7AB8BXl5V3+mfV1UF1DTLrUuyIcmG8fHxAUQqSeqyGZO2JPdO8vkkV7SLqU9s9Wck+WqSy9tjTatPknck2ZTkyiSPXuTXIElS5yTZhV7C9oGq+mirviXJyjZ/JXDrVMtW1alVtbaq1o6NjQ0mYElSZ62YRZs7gSdX1XdbB3RJkn9u815dVR+e1P7pwH7t8VjglPYsSdKykCTAacB1VfXnfbPOAY4B3tKezx5CeJKkETPjmbbq+W6b3KU9phzO0RwOvLct91lg94mjipIkLRMHAS8Antw3IuUwesnaU5NcDzylTUuStF2zOdNGkp2BjcBDgXdV1eeS/B/gzUneSLttcVXdSe/uWDf1Lb651W1FkqRloKouATLN7EMGGYskafTN6kYkVbWtqtYA+wAHJHkE8HrgZ4FfBPYAXjuXDXuRtSRJkiTNbE53j6yqbwMXAIe2/0FT7eza3wIHtGZbgFV9i+3T6iavy4usJUmSJGkGs7l75FiS3Vt5N+CpwJf67n4Vev8c9Oq2yDnA0e0ukgcCt/f9TxpJkiRJ0hzM5pq2lcD6dl3bTsBZVfXxJP+aZIzemP3Lgd9p7c8FDgM2Ad8HXrTgUUuSJEnSMjFj0lZVVwKPmqL+ydO0L+C4+YcmSZIkSZrTNW2SJEmSpMEyaZMkSZKkDjNpkyRJkqQOM2mTJEmSpA4zaZMkSZKkDjNpkyRJkqQOM2mTJEmSpA4zaZMkSZKkDjNpkyRJkqQOM2mTJEmSpA4zaZMkSZKkDjNpkyRJkqQOM2mTJEmSpA4zaZMkSZKkDjNpkyRJkqQOM2mTJEmSpA4zaZMkSZKkDjNpkyRJkqQOM2mTJEmSpA4zaZMkSZKkDjNpkyRJkqQOM2mTJEmSpA6bMWlLcu8kn09yRZJrkpzY6h+c5HNJNiX5UJJ7tfpd2/SmNn/1Ir8GSZIkSVqyZnOm7U7gyVX1SGANcGiSA4G3AidX1UOB24BjW/tjgdta/cmtnSRJkiRpB8yYtFXPd9vkLu1RwJOBD7f69cARrXx4m6bNPyRJFipgSZIkSVpOZnVNW5Kdk1wO3AqcB/wH8O2quqs12Qzs3cp7AzcBtPm3Aw9YwJglSZIkadmYVdJWVduqag2wD3AA8LPz3XCSdUk2JNkwPj4+39VJkiRJ0pI0p7tHVtW3gQuAxwG7J1nRZu0DbGnlLcAqgDb/fsA3p1jXqVW1tqrWjo2N7Vj0kiRJkrTEzebukWNJdm/l3YCnAtfRS96e3ZodA5zdyue0adr8f62qWsCYJUnqtCSnJ7k1ydV9dSck2ZLk8vY4bJgxSpJGx4qZm7ASWJ9kZ3pJ3llV9fEk1wJnJnkT8EXgtNb+NOB9STYB3wKOXIS4JUnqsjOAdwLvnVR/clW9ffDhSJJG2YxJW1VdCTxqivqv0Lu+bXL9D4DnLEh0kiSNoKq62P9TKklaKHO6pk2SJM3L8UmubMMn7z9dI2/WJUnqZ9ImSdJgnAI8BFgDbAVOmq6hN+uSJPUzaZMkaQCq6pb2L3TuBv6GKS4xkCRpKiZtkiQNQJKVfZPPAq6erq0kSf1mc/dISZI0B0k+CBwM7JlkM/CHwMFJ1gAF3AC8eFjxSZJGi0mbJEkLrKqOmqL6tCnqJEmakcMjJUmSJKnDTNokSZIkqcNM2iRJkiSpw0zaJEmSJKnDTNokSZIkqcNM2iRJkiSpw0zaJEmSJKnDTNokSZIkqcNM2iRJkiSpw0zaJEmSJKnDTNokSZIkqcNM2iRJkiSpw0zaJEmSJKnDTNokSZIkqcNM2iRJkiSpw0zaJEmSJKnDZkzakqxKckGSa5Nck+Rlrf6EJFuSXN4eh/Ut8/okm5J8OcmvLOYLkCRJkqSlbMUs2twFvLKqLktyX2BjkvPavJOr6u39jZPsDxwJPBx4IPAvSX6mqrYtZOCSJEmStBzMeKatqrZW1WWtfAdwHbD3dhY5HDizqu6sqq8Cm4ADFiJYSZIkSVpu5nRNW5LVwKOAz7Wq45NcmeT0JPdvdXsDN/UttpntJ3mSJEmSpGnMOmlLch/gI8DLq+o7wCnAQ4A1wFbgpLlsOMm6JBuSbBgfH5/LopIkSZK0bMwqaUuyC72E7QNV9VGAqrqlqrZV1d3A3/DDIZBbgFV9i+/T6u6hqk6tqrVVtXZsbGw+r0GSJEmSlqzZ3D0ywGnAdVX15331K/uaPQu4upXPAY5MsmuSBwP7AZ9fuJAlSZIkafmYzd0jDwJeAFyV5PJW9wbgqCRrgAJuAF4MUFXXJDkLuJbenSeP886RkiRJkrRjZkzaquoSIFPMOnc7y7wZePM84pIkSZIkMce7R0qSJEmSBsukTZIkSZI6zKRNkiRJkjrMpE2SJEmSOsykTZIkSZI6zKRNkiRJkjrMpE2SJEmSOsykTZKkBZbk9CS3Jrm6r26PJOclub4933+YMUqSRodJmyRJC+8M4NBJda8Dzq+q/YDz27QkSTMyaZMkaYFV1cXAtyZVHw6sb+X1wBGDjEmSNLpM2iRJGoy9qmprK98M7DXMYCRJo8OkTZKkAauqAmq6+UnWJdmQZMP4+PgAI5MkdZFJmyRJg3FLkpUA7fnW6RpW1alVtbaq1o6NjQ0sQElSN5m0SZI0GOcAx7TyMcDZQ4xFkjRCTNokSVpgST4IfAZ4WJLNSY4F3gI8Ncn1wFPatCRJM1ox7AAkSVpqquqoaWYdMtBAJElLgmfaJEmSJKnDTNokSZIkqcNM2iRJkiSpw0zaJEmSJKnDTNokSZIkqcNM2iRJkiSpw2ZM2pKsSnJBkmuTXJPkZa1+jyTnJbm+Pd+/1SfJO5JsSnJlkkcv9ouQJEmSpKVqNmfa7gJeWVX7AwcCxyXZH3gdcH5V7Qec36YBng7s1x7rgFMWPGpJkiRJWiZmTNqqamtVXdbKdwDXAXsDhwPrW7P1wBGtfDjw3ur5LLB7kpULHbgkSZIkLQdzuqYtyWrgUcDngL2qamubdTOwVyvvDdzUt9jmVidJkiRJmqNZJ21J7gN8BHh5VX2nf15VFVBz2XCSdUk2JNkwPj4+l0UlSZIkadmYVdKWZBd6CdsHquqjrfqWiWGP7fnWVr8FWNW3+D6t7h6q6tSqWltVa8fGxnY0fkmSJEla0mZz98gApwHXVdWf9806BzimlY8Bzu6rP7rdRfJA4Pa+YZSSJEmSpDlYMYs2BwEvAK5KcnmrewPwFuCsJMcCNwLPbfPOBQ4DNgHfB160kAFLkiRJ0nIyY9JWVZcAmWb2IVO0L+C4ecYlSZIkSWKOd4+UJEmSJA2WSZskSZIkdZhJmyRJkiR1mEmbJEmSJHXYbO4eKUmSJHXKQX910LBDGDmXvuTSYYegHeSZNkmSJEnqMJM2SZIkSeowkzZJkiRJ6jCTNkmSJEnqMJM2SZIkSeowkzZJkiRJ6jCTNkmSJEnqMJM2SZIkSeowkzZJkiRJ6jCTNkmSJEnqMJM2SZIkSeowkzZJkiRJ6jCTNkmSJEnqsBXDDkCSpOUkyQ3AHcA24K6qWjvciCRJXWfSJknS4P1yVX1j2EFIkkaDwyMlSZIkqcNM2iRJGqwCPpVkY5J1ww5GktR9MyZtSU5PcmuSq/vqTkiyJcnl7XFY37zXJ9mU5MtJfmWxApckaUQ9oaoeDTwdOC7JEyc3SLIuyYYkG8bHxwcfoSSpU2Zzpu0M4NAp6k+uqjXtcS5Akv2BI4GHt2XenWTnhQpWkqRRV1Vb2vOtwMeAA6Zoc2pVra2qtWNjY4MOUZLUMTMmbVV1MfCtWa7vcODMqrqzqr4KbGKKzkiSpOUoyY8nue9EGXgacPX2l5IkLXfzuabt+CRXtuGT9291ewM39bXZ3Op+hEM/JEnL0F7AJUmuAD4P/FNVfWLIMUmSOm5Hk7ZTgIcAa4CtwElzXYFDPyRJy01VfaWqHtkeD6+qNw87JklS9+1Q0lZVt1TVtqq6G/gbfjgEcguwqq/pPq1OkiRJkrQDdihpS7Kyb/JZ/HA8/jnAkUl2TfJgYD96wz8kSZIkSTtgxUwNknwQOBjYM8lm4A+Bg5Osofe/Zm4AXgxQVdckOQu4FrgLOK6qti1K5JIkSZK0DMyYtFXVUVNUn7ad9m8GHKMvSZIkSQtgPnePlCRJkiQtMpM2SZIkSeowkzZJkiRJ6jCTNkmSJEnqMJM2SZIkSeowkzZJkiRJ6jCTNkmSJEnqMJM2SZIkSeowkzZJkiRJ6jCTNkmSJEnqMJM2SZIkSeowkzZJkiRJ6jCTNkmSJEnqMJM2SZIkSeowkzZJkiRJ6jCTNkmSJEnqMJM2SZIkSeowkzZJkiRJ6jCTNkmSJEnqMJM2SZIkSeowkzZJkiRJ6jCTNkmSJEnqsBmTtiSnJ7k1ydV9dXskOS/J9e35/q0+Sd6RZFOSK5M8ejGDlyRJkqSlbjZn2s4ADp1U9zrg/KraDzi/TQM8HdivPdYBpyxMmJIkSZK0PM2YtFXVxcC3JlUfDqxv5fXAEX31762ezwK7J1m5QLFKkiRJ0rKzo9e07VVVW1v5ZmCvVt4buKmv3eZW9yOSrEuyIcmG8fHxHQxDkiRJkpa2ed+IpKoKqB1Y7tSqWltVa8fGxuYbhiRJkiQtSTuatN0yMeyxPd/a6rcAq/ra7dPqJEmSJEk7YEeTtnOAY1r5GODsvvqj210kDwRu7xtGKUnSspfk0CRfbndaft3MS0iSlrsVMzVI8kHgYGDPJJuBPwTeApyV5FjgRuC5rfm5wGHAJuD7wIsWIWZJkkZSkp2BdwFPpXfd9xeSnFNV1w43MklSl82YtFXVUdPMOmSKtgUcN9+gJElaog4ANlXVVwCSnEnvzssmbSPqa3/088MOYaTs+8arhh2CNJLmfSMSSZI0a7O+y7IkSRNmPNMmSZIGK8k6YB3AvvvuO2P7jW87erFD0jQ8czQ8l77k0mGHsKxd9MQnDTuEkfKkiy+a1/KeaZMkaXBmdZdl/y2OJKmfSZskSYPzBWC/JA9Oci/gSHp3XpYkaVoOj5QkaUCq6q4kxwOfBHYGTq+qa4YcliSp40zaJEkaoKo6l96/yJEkaVYcHilJkiRJHWbSJkmSJEkdZtImSZIkSR1m0iZJkiRJHWbSJkmSJEkdZtImSZIkSR1m0iZJkiRJHWbSJkmSJEkdZtImSZIkSR1m0iZJkiRJHWbSJkmSJEkdZtImSZIkSR1m0iZJkiRJHWbSJkmSJEkdZtImSZIkSR22Yj4LJ7kBuAPYBtxVVWuT7AF8CFgN3AA8t6pum1+YkiRJkrQ8LcSZtl+uqjVVtbZNvw44v6r2A85v05IkSZKkHbAYwyMPB9a38nrgiEXYhiRJkiQtC/NN2gr4VJKNSda1ur2qamsr3wzsNc9tSJIkSdKyNa9r2oAnVNWWJD8JnJfkS/0zq6qS1FQLtiRvHcC+++47zzAkSZIkaWma15m2qtrSnm8FPgYcANySZCVAe751mmVPraq1VbV2bGxsPmFIkiRJ0pK1w0lbkh9Pct+JMvA04GrgHOCY1uwY4Oz5BilJkiRJy9V8hkfuBXwsycR6/q6qPpHkC8BZSY4FbgSeO/8wJUmSJGl52uGkraq+AjxyivpvAofMJyhJkiRJUs9i3PJfkiRJkrRATNokSZIkqcNM2iRJkiSpw0zaJEmSJKnDTNokSZIkqcNM2iRJkiSpw0zaJEmSJKnDTNokSZIkqcNM2iRJkiSpw0zaJEkagCQnJNmS5PL2OGzYMUmSRsOKYQcgSdIycnJVvX3YQUiSRotn2iRJkiSpw0biTNtjXv3eYYcwcBvfdvSwQ5AkLbzjkxwNbABeWVW3DTsgSVL3eaZNkqQFkuRfklw9xeNw4BTgIcAaYCtw0nbWsy7JhiQbxsfHBxO8JKmzRuJMmyRJo6CqnjKbdkn+Bvj4dtZzKnAqwNq1a2thopMkjSrPtEmSNABJVvZNPgu4elixSJJGi2faJEkajD9LsgYo4AbgxUONRpI0MkzaJEkagKp6wbBjkCSNJodHSpIkSVKHmbRJkiRJUoeZtEmSJElSh5m0SZIkSVKHLdqNSJIcCvwlsDPwnqp6y2JtS5qPg/7qoGGHMHCXvuTSYYcgSZKkWVqUM21JdgbeBTwd2B84Ksn+i7EtSZIkSVrKFmt45AHApqr6SlX9F3AmcPgibUuSJEmSlqzFStr2Bm7qm97c6iRJkiRJczC0f66dZB2wDmDffffdbtuNbzt6ECFpmfL6rrm56IlPGnYIA/ekiy/a4WWPP+kZCxiJJElajhbrTNsWYFXf9D6t7n9U1alVtbaq1o6NjS1SGJIkSZI02hYrafsCsF+SBye5F3AkcM4ibUuSJEmSlqxFGR5ZVXclOR74JL1b/p9eVdcsxrYkSZIkaSlbtGvaqupc4NzFWr+k4ZjP9V2SJGlp8PfAYC3W8EhJkiRJ0gIwaZMkSZKkDjNpkyRJkqQOM2mTJEmSpA4zaZMkSZKkDjNpkyRJkqQOM2mTJEmSpA4zaZMkSZKkDjNpkyRJkqQOM2mTJEmSpA4zaZMkSZKkDjNpkyRJkqQOS1UNOwaSjAM3DjuOKewJfGPYQYwQ99fcuL/mzn02N13dXw+qqrFhBzEqOtxHzkZX34PLgft+uNz/wzPK+37a/rETSVtXJdlQVWuHHceocH/Njftr7txnc+P+0rD5Hhwe9/1wuf+HZ6nue4dHSpIkSVKHmbRJkiRJUoeZtG3fqcMOYMS4v+bG/TV37rO5cX9p2HwPDo/7frjc/8OzJPe917RJkiRJUod5pk2SJEmSOmxkk7YkRySpJD+7gOs8IcmWJJcnuTbJUQu17q5I8t1J0y9M8s4FWveFSX7kbj1Jfi3JF5Nc0fbri1v97yQ5eiG23SXtffn+vukVScaTfHyG5Q6eqc2oSrKtfa4mHqsXcVs3JNlzsda/GCZ/Lqdp8/IkPzaIeKbY9u5JfncY29bisR8dPvvk0WL/vvj8vTC9FcMOYB6OAi5pz3+4gOs9uarenmQ/YGOSD1fVfy/g+peVJLvQG1t8QFVtTrIrsBqgqv56mLEtou8Bj0iyW1X9J/BUYMuQYxq2/6yqNcMOYsS9HHg/8P3ZLpBk56ratgDb3h34XeDdC7AudYf96DKzTPvkhWT/vvj8vTCNkTzTluQ+wBOAY4Ej++p3SvLuJF9Kcl6Sc5M8u817TJKLkmxM8skkK7e3jaq6nt6Po/u35U9JsiHJNUlO7NvmDUlOTHJZkqsmjlgmGWsxXJPkPUlunMjmkzw/yefbEYT/l2TnBd5FO6TF/JEkX2iPg1r9AUk+047M/VuSh7X63ZKcmeS6JB8Ddptitfeld3DgmwBVdWdVfbktf0KSVyV54KSjKtuSPGi6eEbEucCvtvJRwAcnZky3P/sl+fEkp7f3yReTHD6guAdmus9kOzp8cvu8XZfkF5N8NMn1Sd7Ut/w/tGWvSbJumm108rM2nXY09sIkH27fYx9Iz0uBBwIXJLmgtX1aex9dluTv2/fixHfSW5NcBjwnyaGtzRVJzm9tpnx/pXeU/+wWw/VJJn7IvwV4SNuPbxv4jtGCi/1o503XB07Xh8Q+eVDs3wdsuu+eLLffC1U1cg/gecBprfxvwGNa+dn0Pkw7AT8F3Nbqdmntxlq73wBOn2K9JwCvauVHA5/um7dHe94ZuBD4hTZ9A/CSVv5d4D2t/E7g9a18KFD0/kP7zwH/COzS5r0bOHqA+24bcHnf42vAO9u8vwOe0Mr7Ate18k8AK1r5KcBHWvkVE/sR+AXgLmDtFNt8D3ArvS+25wE7Td7ffW2PA87aXjxdfwDfbfvjw8C9234+GPj4DPuzv82fAM9v5d2Bfwd+fNivbYHedx/b3meyfb7e2sovA74OrAR2BTYDD2jzJj6TuwFX99Xf0IXP2lzfM33vgduBfeh9j32m7zNwA7BnK+8JXDzxngBeC7yxr91rWnkMuAl48KR9NuX7C3ghsBV4QN9+XUvvSPzVw95PPhb0PWc/2oEH9skj9cD+fRD72N8L0zxGdXjkUcBftvKZbXojvaOGf19VdwM3TxyRBh4GPAI4Lwn0Ooyt06z7/yZ5EfAzwDP66p/bsvMV9N4Q+wNXtnkfbc8bgf/Vyk8AngVQVZ9IclurPwR4DPCFFstu9L48B+Uep52TvJDejzLofcHs3+IC+Il2NPZ+wPr0hroUvQ8QwBOBdwBU1ZVJJvbHPVTVbyX5+bb+V9EbTvDCye3aUbvfprfvpo2nqma8/mfY2v5YTe+9ee6k2dPtz35PA56Z5FVt+t60TnJxIl50k993j2D7n8lz2vNVwDVVtbUt9xVgFb2jxC9N8qzWbhWwX6ufMOzP2o76fFVtBkhyOb2E6ZJJbQ6k9x10aXtt96KX4E34UF+7i6vqqwBV9a1WP937C+C8qvpm2/5H6X0e/2EBXpe6xX60G+yTR4z9+6Lz98I0Ri5pS7IH8GTg55MUvT9eJXn19haj94d83Cw2cXL1xuI/EzgtyUPodS6vAn6xqm5Lcga9D9mEO9vzNmbepwHWV9XrZxHLoO0EHFhVP+ivTO+i6Auq6lnti+rCua64qq4CrkryPuCrTOog2qnu04Bn9nUAU8YzQs4B3k7vCNsD+ur/mJn3Z4BfrzZsZQma6TM58Zm6u688Mb0iycH0fkA8rqq+n+RC7vmZnNhGVz9r29P/eqf7Tgm95Gq6mzx8b4ZtTPn+SvJYej80+vl/YZYY+9GRYZ/cXfbvg+PvhWYUr2l7NvC+qnpQVa2uqlX0vnB+CbgU+PX0xuTvRe/DBPBlYCzJ46B3IW6Sh29vI1V1DrABOIbe6e7vAbe39T59FnFeCjy3be9ptDH9wPnAs5P8ZJu3R5IHze6lL7pPAS+ZmEiyphXvxw8vtH1hX/uLgf/d2j6C3pCBe0hyn/aBmbAGuHFSm12AvwdeW1X/Pot4RsXpwImtc+w33f7s90ngJWmHfJI8alEiHJ45fyYnuR9wW/sC/ll6Z5Qm6/JnbUfcQe96FIDPAgcleSj8zzUSPzPFMp8Fnpjkwa3dHq1+e++vp7Z9tRtwBL3vsv5ta/TZj44G++Tusn8fHH8vNKOYtB1Fb4xrv4+0+o/QG8N6Lb27rF0G3F5V/0Wvk3prkivojZN9/Cy29Uf0xohfBXwR+BK9Md2XzmLZE4GnJbkaeA5wM3BHVV0L/D7wqTZ04Tx6RyC74KXA2iRXJrkW+J1W/2fAnyb5Ivc8AnoKcJ8k19HbVxunWGeA1yT5cnpDvU7kR7/IHk9vOMiJ+eGFzw/cTjwjoao2V9U7ppg13f7s98f0hlVcmeSaNr1kzOMzOeET9I6gXUfvJhmfnWIbXf6s7YhTgU8kuaCqxul9jj7YXttngB+5bXtrtw74aNvPE8Mmt/f++jy979Ir6V2PsaENl7w0ydXxRiRLgf3oaLBP7ij798Hx98IPpWppjXxJG1+d5AH0fnwcVFU3DyGOXYFtVXVXOzpwSnkLU0kdlnY9TVUdP+xYNDz2o5LUPSN3TdssfDzJ7vQuzP/jYXQ0zb7AWUl2Av6L3sW8kiR1nf2oJHXMkjvTJkmSJElLyShe0yZJkiRJy4ZJmyRJkiR1mEmbJEmSJHWYSZskSZIkdZhJmyRJkiR1mEmbJEmSJHXY/wc4Pov8CCTyOQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1080x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(15, 5))\n",
    "\n",
    "plt.subplot(1,2,1)\n",
    "sns.barplot(data = coefficients_dataframe);\n",
    "plt.title('With Intercept');\n",
    "\n",
    "plt.subplot(1,2,2)\n",
    "sns.barplot(data = coefficients_dataframe[['Age Range', 'Head Size', 'Male', 'Female']]);\n",
    "plt.title('Without Intercept');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "R2 Score: 0.6141102935912988\n",
      "Training Accuracy: 0.6141102935912988\n",
      "Test Accuracy: 0.7767424819420188\n",
      "Mean Absolute Error: 57.11323529480088\n",
      "Mean Square Error: 5403.78680464104\n",
      "Root Mean Square Error: 73.5104537099387\n"
     ]
    }
   ],
   "source": [
    "# Evaluation\n",
    "Y_pred_train = model.predict(X_train)\n",
    "\n",
    "# Metrics\n",
    "r2          = r2_score(Y_train, Y_pred_train)\n",
    "mae         = mean_absolute_error(Y_train, Y_pred_train)\n",
    "mse         = mean_squared_error(Y_train, Y_pred_train)\n",
    "rmse        = np.sqrt(mse)\n",
    "score_train = model.score(X_train, Y_train)\n",
    "score_test  = model.score(X_test, Y_test)\n",
    "\n",
    "print(f'R2 Score: {r2}')\n",
    "print(f'Training Accuracy: {score_train}')\n",
    "print(f'Test Accuracy: {score_test}')\n",
    "print(f'Mean Absolute Error: {mae}')\n",
    "print(f'Mean Square Error: {mse}')\n",
    "print(f'Root Mean Square Error: {rmse}')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.10 64-bit (system)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "9805bccb659ffcb6952b8b29440036fac091f8170d71a5a402ad9a8f2988209f"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}