additional_topics/ML_ZOOMCAMP_CROSS_VALIDATION_METHODS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d590e167",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Cross Validation in Practice: main types and examples\n",
    "One of the topics that it called my attention from weeks 1 and 2 of #mlzoomcamp was cross validation. After some search in Google, I can se that there are various types of Cross Valitation used in machine learning. Since the list of available types is extensive, I only put my attention in the types with most mentions in my search.\n",
    "\n",
    "The list with methods with most mentions is:\n",
    "- [Holdout cross-validation](#method-1)\n",
    "- [Leave p out cross-validation (LpOCV)](#method-2)\n",
    "- [Leave one out cross-validation (LoOCV)](#method-3)\n",
    "- [K-fold cross-validation](#method-4)\n",
    "- [Stratified k-fold cross-validation](#method-5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eccb5d6d",
   "metadata": {},
   "source": [
    "### Import modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "456b58ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.model_selection import train_test_split # Holdout cross-validation\n",
    "from sklearn.model_selection import LeavePOut # Leave p out cross-validation\n",
    "from sklearn.model_selection import LeaveOneOut # Leave one out cross-validation\n",
    "from sklearn.model_selection import KFold # K-fold cross-validation\n",
    "from sklearn.model_selection import StratifiedKFold # Stratified k-fold cross-validation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cadb7f12",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5bc1876a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Model</th>\n",
       "      <th>Year</th>\n",
       "      <th>Engine Fuel Type</th>\n",
       "      <th>Engine HP</th>\n",
       "      <th>Engine Cylinders</th>\n",
       "      <th>Transmission Type</th>\n",
       "      <th>Driven_Wheels</th>\n",
       "      <th>Number of Doors</th>\n",
       "      <th>Market Category</th>\n",
       "      <th>Vehicle Size</th>\n",
       "      <th>Vehicle Style</th>\n",
       "      <th>highway MPG</th>\n",
       "      <th>city mpg</th>\n",
       "      <th>Popularity</th>\n",
       "      <th>MSRP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series M</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>335.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Factory Tuner,Luxury,High-Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>26</td>\n",
       "      <td>19</td>\n",
       "      <td>3916</td>\n",
       "      <td>46135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>28</td>\n",
       "      <td>19</td>\n",
       "      <td>3916</td>\n",
       "      <td>40650</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,High-Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>20</td>\n",
       "      <td>3916</td>\n",
       "      <td>36350</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "      <td>29450</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "      <td>34500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "      <td>31200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>26</td>\n",
       "      <td>17</td>\n",
       "      <td>3916</td>\n",
       "      <td>44100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,High-Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>20</td>\n",
       "      <td>3916</td>\n",
       "      <td>39300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "      <td>36900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2013</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>27</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "      <td>37200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Make       Model  Year             Engine Fuel Type  Engine HP  \\\n",
       "0  BMW  1 Series M  2011  premium unleaded (required)      335.0   \n",
       "1  BMW    1 Series  2011  premium unleaded (required)      300.0   \n",
       "2  BMW    1 Series  2011  premium unleaded (required)      300.0   \n",
       "3  BMW    1 Series  2011  premium unleaded (required)      230.0   \n",
       "4  BMW    1 Series  2011  premium unleaded (required)      230.0   \n",
       "5  BMW    1 Series  2012  premium unleaded (required)      230.0   \n",
       "6  BMW    1 Series  2012  premium unleaded (required)      300.0   \n",
       "7  BMW    1 Series  2012  premium unleaded (required)      300.0   \n",
       "8  BMW    1 Series  2012  premium unleaded (required)      230.0   \n",
       "9  BMW    1 Series  2013  premium unleaded (required)      230.0   \n",
       "\n",
       "   Engine Cylinders Transmission Type     Driven_Wheels  Number of Doors  \\\n",
       "0               6.0            MANUAL  rear wheel drive              2.0   \n",
       "1               6.0            MANUAL  rear wheel drive              2.0   \n",
       "2               6.0            MANUAL  rear wheel drive              2.0   \n",
       "3               6.0            MANUAL  rear wheel drive              2.0   \n",
       "4               6.0            MANUAL  rear wheel drive              2.0   \n",
       "5               6.0            MANUAL  rear wheel drive              2.0   \n",
       "6               6.0            MANUAL  rear wheel drive              2.0   \n",
       "7               6.0            MANUAL  rear wheel drive              2.0   \n",
       "8               6.0            MANUAL  rear wheel drive              2.0   \n",
       "9               6.0            MANUAL  rear wheel drive              2.0   \n",
       "\n",
       "                         Market Category Vehicle Size Vehicle Style  \\\n",
       "0  Factory Tuner,Luxury,High-Performance      Compact         Coupe   \n",
       "1                     Luxury,Performance      Compact   Convertible   \n",
       "2                Luxury,High-Performance      Compact         Coupe   \n",
       "3                     Luxury,Performance      Compact         Coupe   \n",
       "4                                 Luxury      Compact   Convertible   \n",
       "5                     Luxury,Performance      Compact         Coupe   \n",
       "6                     Luxury,Performance      Compact   Convertible   \n",
       "7                Luxury,High-Performance      Compact         Coupe   \n",
       "8                                 Luxury      Compact   Convertible   \n",
       "9                                 Luxury      Compact   Convertible   \n",
       "\n",
       "   highway MPG  city mpg  Popularity   MSRP  \n",
       "0           26        19        3916  46135  \n",
       "1           28        19        3916  40650  \n",
       "2           28        20        3916  36350  \n",
       "3           28        18        3916  29450  \n",
       "4           28        18        3916  34500  \n",
       "5           28        18        3916  31200  \n",
       "6           26        17        3916  44100  \n",
       "7           28        20        3916  39300  \n",
       "8           28        18        3916  36900  \n",
       "9           27        18        3916  37200  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read some data to test \n",
    "if not os.path.isfile(\"data.csv\"):\n",
    "    !wget https://raw.githubusercontent.com/alexeygrigorev/mlbookcamp-code/master/chapter-02-car-price/data.csv\n",
    "        \n",
    "df = pd.read_csv(\"data.csv\")\n",
    "X = df.iloc[np.arange(10)]\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "37803c95",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    46135\n",
       "1    40650\n",
       "2    36350\n",
       "3    29450\n",
       "4    34500\n",
       "5    31200\n",
       "6    44100\n",
       "7    39300\n",
       "8    36900\n",
       "9    37200\n",
       "Name: MSRP, dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Extract target column\n",
    "y = X.MSRP\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d9f96ce6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Model</th>\n",
       "      <th>Year</th>\n",
       "      <th>Engine Fuel Type</th>\n",
       "      <th>Engine HP</th>\n",
       "      <th>Engine Cylinders</th>\n",
       "      <th>Transmission Type</th>\n",
       "      <th>Driven_Wheels</th>\n",
       "      <th>Number of Doors</th>\n",
       "      <th>Market Category</th>\n",
       "      <th>Vehicle Size</th>\n",
       "      <th>Vehicle Style</th>\n",
       "      <th>highway MPG</th>\n",
       "      <th>city mpg</th>\n",
       "      <th>Popularity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series M</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>335.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Factory Tuner,Luxury,High-Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>26</td>\n",
       "      <td>19</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>28</td>\n",
       "      <td>19</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,High-Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>20</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>26</td>\n",
       "      <td>17</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,High-Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>20</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2013</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>27</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Make       Model  Year             Engine Fuel Type  Engine HP  \\\n",
       "0  BMW  1 Series M  2011  premium unleaded (required)      335.0   \n",
       "1  BMW    1 Series  2011  premium unleaded (required)      300.0   \n",
       "2  BMW    1 Series  2011  premium unleaded (required)      300.0   \n",
       "3  BMW    1 Series  2011  premium unleaded (required)      230.0   \n",
       "4  BMW    1 Series  2011  premium unleaded (required)      230.0   \n",
       "5  BMW    1 Series  2012  premium unleaded (required)      230.0   \n",
       "6  BMW    1 Series  2012  premium unleaded (required)      300.0   \n",
       "7  BMW    1 Series  2012  premium unleaded (required)      300.0   \n",
       "8  BMW    1 Series  2012  premium unleaded (required)      230.0   \n",
       "9  BMW    1 Series  2013  premium unleaded (required)      230.0   \n",
       "\n",
       "   Engine Cylinders Transmission Type     Driven_Wheels  Number of Doors  \\\n",
       "0               6.0            MANUAL  rear wheel drive              2.0   \n",
       "1               6.0            MANUAL  rear wheel drive              2.0   \n",
       "2               6.0            MANUAL  rear wheel drive              2.0   \n",
       "3               6.0            MANUAL  rear wheel drive              2.0   \n",
       "4               6.0            MANUAL  rear wheel drive              2.0   \n",
       "5               6.0            MANUAL  rear wheel drive              2.0   \n",
       "6               6.0            MANUAL  rear wheel drive              2.0   \n",
       "7               6.0            MANUAL  rear wheel drive              2.0   \n",
       "8               6.0            MANUAL  rear wheel drive              2.0   \n",
       "9               6.0            MANUAL  rear wheel drive              2.0   \n",
       "\n",
       "                         Market Category Vehicle Size Vehicle Style  \\\n",
       "0  Factory Tuner,Luxury,High-Performance      Compact         Coupe   \n",
       "1                     Luxury,Performance      Compact   Convertible   \n",
       "2                Luxury,High-Performance      Compact         Coupe   \n",
       "3                     Luxury,Performance      Compact         Coupe   \n",
       "4                                 Luxury      Compact   Convertible   \n",
       "5                     Luxury,Performance      Compact         Coupe   \n",
       "6                     Luxury,Performance      Compact   Convertible   \n",
       "7                Luxury,High-Performance      Compact         Coupe   \n",
       "8                                 Luxury      Compact   Convertible   \n",
       "9                                 Luxury      Compact   Convertible   \n",
       "\n",
       "   highway MPG  city mpg  Popularity  \n",
       "0           26        19        3916  \n",
       "1           28        19        3916  \n",
       "2           28        20        3916  \n",
       "3           28        18        3916  \n",
       "4           28        18        3916  \n",
       "5           28        18        3916  \n",
       "6           26        17        3916  \n",
       "7           28        20        3916  \n",
       "8           28        18        3916  \n",
       "9           27        18        3916  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Delete target column from features data\n",
    "del X['MSRP']\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34e07dd0",
   "metadata": {},
   "source": [
    "### Holdout cross-validation <a class=\"anchor\" id=\"method-1\"></a>\n",
    "Is the most basic used approach. It consist in split the dataset in two subsets: train and validation (test). Generally the split ratio is 80:20 or 70:30.\n",
    "\n",
    "<img src=\"images/hold_out.jpg\" width=400 height=400 />\n",
    "<a href=\"https://www.mygreatlearning.com/blog/cross-validation/#sh211\">Image Source</a>\n",
    "\n",
    "This approach is recommended for large datasets. Otherwise, there is a high possibility that the testing data may contain some important information that we lose at training time. **Note:** The accuracy of the model is very sensible to the performed split so may vary for different splits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "881b9dc7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set size: 7\n",
      "Test set size: 3\n"
     ]
    }
   ],
   "source": [
    "# Perform Hold Out method\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1)\n",
    "print(f\"Train set size: {len(X_train)}\")\n",
    "print(f\"Test set size: {len(X_test)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "695af682",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Model</th>\n",
       "      <th>Year</th>\n",
       "      <th>Engine Fuel Type</th>\n",
       "      <th>Engine HP</th>\n",
       "      <th>Engine Cylinders</th>\n",
       "      <th>Transmission Type</th>\n",
       "      <th>Driven_Wheels</th>\n",
       "      <th>Number of Doors</th>\n",
       "      <th>Market Category</th>\n",
       "      <th>Vehicle Size</th>\n",
       "      <th>Vehicle Style</th>\n",
       "      <th>highway MPG</th>\n",
       "      <th>city mpg</th>\n",
       "      <th>Popularity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series M</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>335.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Factory Tuner,Luxury,High-Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>26</td>\n",
       "      <td>19</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>28</td>\n",
       "      <td>19</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,High-Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>20</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Make       Model  Year             Engine Fuel Type  Engine HP  \\\n",
       "4  BMW    1 Series  2011  premium unleaded (required)      230.0   \n",
       "0  BMW  1 Series M  2011  premium unleaded (required)      335.0   \n",
       "3  BMW    1 Series  2011  premium unleaded (required)      230.0   \n",
       "1  BMW    1 Series  2011  premium unleaded (required)      300.0   \n",
       "7  BMW    1 Series  2012  premium unleaded (required)      300.0   \n",
       "8  BMW    1 Series  2012  premium unleaded (required)      230.0   \n",
       "5  BMW    1 Series  2012  premium unleaded (required)      230.0   \n",
       "\n",
       "   Engine Cylinders Transmission Type     Driven_Wheels  Number of Doors  \\\n",
       "4               6.0            MANUAL  rear wheel drive              2.0   \n",
       "0               6.0            MANUAL  rear wheel drive              2.0   \n",
       "3               6.0            MANUAL  rear wheel drive              2.0   \n",
       "1               6.0            MANUAL  rear wheel drive              2.0   \n",
       "7               6.0            MANUAL  rear wheel drive              2.0   \n",
       "8               6.0            MANUAL  rear wheel drive              2.0   \n",
       "5               6.0            MANUAL  rear wheel drive              2.0   \n",
       "\n",
       "                         Market Category Vehicle Size Vehicle Style  \\\n",
       "4                                 Luxury      Compact   Convertible   \n",
       "0  Factory Tuner,Luxury,High-Performance      Compact         Coupe   \n",
       "3                     Luxury,Performance      Compact         Coupe   \n",
       "1                     Luxury,Performance      Compact   Convertible   \n",
       "7                Luxury,High-Performance      Compact         Coupe   \n",
       "8                                 Luxury      Compact   Convertible   \n",
       "5                     Luxury,Performance      Compact         Coupe   \n",
       "\n",
       "   highway MPG  city mpg  Popularity  \n",
       "4           28        18        3916  \n",
       "0           26        19        3916  \n",
       "3           28        18        3916  \n",
       "1           28        19        3916  \n",
       "7           28        20        3916  \n",
       "8           28        18        3916  \n",
       "5           28        18        3916  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "34318cb2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Model</th>\n",
       "      <th>Year</th>\n",
       "      <th>Engine Fuel Type</th>\n",
       "      <th>Engine HP</th>\n",
       "      <th>Engine Cylinders</th>\n",
       "      <th>Transmission Type</th>\n",
       "      <th>Driven_Wheels</th>\n",
       "      <th>Number of Doors</th>\n",
       "      <th>Market Category</th>\n",
       "      <th>Vehicle Size</th>\n",
       "      <th>Vehicle Style</th>\n",
       "      <th>highway MPG</th>\n",
       "      <th>city mpg</th>\n",
       "      <th>Popularity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,High-Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>20</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2013</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>27</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>26</td>\n",
       "      <td>17</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Make     Model  Year             Engine Fuel Type  Engine HP  \\\n",
       "2  BMW  1 Series  2011  premium unleaded (required)      300.0   \n",
       "9  BMW  1 Series  2013  premium unleaded (required)      230.0   \n",
       "6  BMW  1 Series  2012  premium unleaded (required)      300.0   \n",
       "\n",
       "   Engine Cylinders Transmission Type     Driven_Wheels  Number of Doors  \\\n",
       "2               6.0            MANUAL  rear wheel drive              2.0   \n",
       "9               6.0            MANUAL  rear wheel drive              2.0   \n",
       "6               6.0            MANUAL  rear wheel drive              2.0   \n",
       "\n",
       "           Market Category Vehicle Size Vehicle Style  highway MPG  city mpg  \\\n",
       "2  Luxury,High-Performance      Compact         Coupe           28        20   \n",
       "9                   Luxury      Compact   Convertible           27        18   \n",
       "6       Luxury,Performance      Compact   Convertible           26        17   \n",
       "\n",
       "   Popularity  \n",
       "2        3916  \n",
       "9        3916  \n",
       "6        3916  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6042f2c2",
   "metadata": {},
   "source": [
    "### Leave p out cross-validation (LpOCV) <a class=\"anchor\" id=\"method-2\"></a>\n",
    "It is an exhaustive cross-validation method where all possible combinations are tested by dividing the data (n observations), into a validation set of “p” observations and a training set of (n-p) observations.\n",
    "\n",
    "To get the final accuracy, we must average the accuracies of all iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7a07ec66",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "120"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Configure LeavePOut instance\n",
    "lpo = LeavePOut(3) # 3 observations\n",
    "lpo.get_n_splits(X) #Combinations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "11465fb4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train: [3 4 5 6 7 8 9] Test: [0 1 2]\n",
      "Train: [2 4 5 6 7 8 9] Test: [0 1 3]\n",
      "Train: [2 3 5 6 7 8 9] Test: [0 1 4]\n",
      "Train: [2 3 4 6 7 8 9] Test: [0 1 5]\n",
      "Train: [2 3 4 5 7 8 9] Test: [0 1 6]\n",
      "Train: [2 3 4 5 6 8 9] Test: [0 1 7]\n",
      "Train: [2 3 4 5 6 7 9] Test: [0 1 8]\n",
      "Train: [2 3 4 5 6 7 8] Test: [0 1 9]\n",
      "Train: [1 4 5 6 7 8 9] Test: [0 2 3]\n",
      "Train: [1 3 5 6 7 8 9] Test: [0 2 4]\n",
      "Train: [1 3 4 6 7 8 9] Test: [0 2 5]\n",
      "Train: [1 3 4 5 7 8 9] Test: [0 2 6]\n",
      "Train: [1 3 4 5 6 8 9] Test: [0 2 7]\n",
      "Train: [1 3 4 5 6 7 9] Test: [0 2 8]\n",
      "Train: [1 3 4 5 6 7 8] Test: [0 2 9]\n",
      "Train: [1 2 5 6 7 8 9] Test: [0 3 4]\n",
      "Train: [1 2 4 6 7 8 9] Test: [0 3 5]\n",
      "Train: [1 2 4 5 7 8 9] Test: [0 3 6]\n",
      "Train: [1 2 4 5 6 8 9] Test: [0 3 7]\n",
      "Train: [1 2 4 5 6 7 9] Test: [0 3 8]\n",
      "Train: [1 2 4 5 6 7 8] Test: [0 3 9]\n",
      "Train: [1 2 3 6 7 8 9] Test: [0 4 5]\n",
      "Train: [1 2 3 5 7 8 9] Test: [0 4 6]\n",
      "Train: [1 2 3 5 6 8 9] Test: [0 4 7]\n",
      "Train: [1 2 3 5 6 7 9] Test: [0 4 8]\n",
      "Train: [1 2 3 5 6 7 8] Test: [0 4 9]\n",
      "Train: [1 2 3 4 7 8 9] Test: [0 5 6]\n",
      "Train: [1 2 3 4 6 8 9] Test: [0 5 7]\n",
      "Train: [1 2 3 4 6 7 9] Test: [0 5 8]\n",
      "Train: [1 2 3 4 6 7 8] Test: [0 5 9]\n",
      "Train: [1 2 3 4 5 8 9] Test: [0 6 7]\n",
      "Train: [1 2 3 4 5 7 9] Test: [0 6 8]\n",
      "Train: [1 2 3 4 5 7 8] Test: [0 6 9]\n",
      "Train: [1 2 3 4 5 6 9] Test: [0 7 8]\n",
      "Train: [1 2 3 4 5 6 8] Test: [0 7 9]\n",
      "Train: [1 2 3 4 5 6 7] Test: [0 8 9]\n",
      "Train: [0 4 5 6 7 8 9] Test: [1 2 3]\n",
      "Train: [0 3 5 6 7 8 9] Test: [1 2 4]\n",
      "Train: [0 3 4 6 7 8 9] Test: [1 2 5]\n",
      "Train: [0 3 4 5 7 8 9] Test: [1 2 6]\n",
      "Train: [0 3 4 5 6 8 9] Test: [1 2 7]\n",
      "Train: [0 3 4 5 6 7 9] Test: [1 2 8]\n",
      "Train: [0 3 4 5 6 7 8] Test: [1 2 9]\n",
      "Train: [0 2 5 6 7 8 9] Test: [1 3 4]\n",
      "Train: [0 2 4 6 7 8 9] Test: [1 3 5]\n",
      "Train: [0 2 4 5 7 8 9] Test: [1 3 6]\n",
      "Train: [0 2 4 5 6 8 9] Test: [1 3 7]\n",
      "Train: [0 2 4 5 6 7 9] Test: [1 3 8]\n",
      "Train: [0 2 4 5 6 7 8] Test: [1 3 9]\n",
      "Train: [0 2 3 6 7 8 9] Test: [1 4 5]\n",
      "Train: [0 2 3 5 7 8 9] Test: [1 4 6]\n",
      "Train: [0 2 3 5 6 8 9] Test: [1 4 7]\n",
      "Train: [0 2 3 5 6 7 9] Test: [1 4 8]\n",
      "Train: [0 2 3 5 6 7 8] Test: [1 4 9]\n",
      "Train: [0 2 3 4 7 8 9] Test: [1 5 6]\n",
      "Train: [0 2 3 4 6 8 9] Test: [1 5 7]\n",
      "Train: [0 2 3 4 6 7 9] Test: [1 5 8]\n",
      "Train: [0 2 3 4 6 7 8] Test: [1 5 9]\n",
      "Train: [0 2 3 4 5 8 9] Test: [1 6 7]\n",
      "Train: [0 2 3 4 5 7 9] Test: [1 6 8]\n",
      "Train: [0 2 3 4 5 7 8] Test: [1 6 9]\n",
      "Train: [0 2 3 4 5 6 9] Test: [1 7 8]\n",
      "Train: [0 2 3 4 5 6 8] Test: [1 7 9]\n",
      "Train: [0 2 3 4 5 6 7] Test: [1 8 9]\n",
      "Train: [0 1 5 6 7 8 9] Test: [2 3 4]\n",
      "Train: [0 1 4 6 7 8 9] Test: [2 3 5]\n",
      "Train: [0 1 4 5 7 8 9] Test: [2 3 6]\n",
      "Train: [0 1 4 5 6 8 9] Test: [2 3 7]\n",
      "Train: [0 1 4 5 6 7 9] Test: [2 3 8]\n",
      "Train: [0 1 4 5 6 7 8] Test: [2 3 9]\n",
      "Train: [0 1 3 6 7 8 9] Test: [2 4 5]\n",
      "Train: [0 1 3 5 7 8 9] Test: [2 4 6]\n",
      "Train: [0 1 3 5 6 8 9] Test: [2 4 7]\n",
      "Train: [0 1 3 5 6 7 9] Test: [2 4 8]\n",
      "Train: [0 1 3 5 6 7 8] Test: [2 4 9]\n",
      "Train: [0 1 3 4 7 8 9] Test: [2 5 6]\n",
      "Train: [0 1 3 4 6 8 9] Test: [2 5 7]\n",
      "Train: [0 1 3 4 6 7 9] Test: [2 5 8]\n",
      "Train: [0 1 3 4 6 7 8] Test: [2 5 9]\n",
      "Train: [0 1 3 4 5 8 9] Test: [2 6 7]\n",
      "Train: [0 1 3 4 5 7 9] Test: [2 6 8]\n",
      "Train: [0 1 3 4 5 7 8] Test: [2 6 9]\n",
      "Train: [0 1 3 4 5 6 9] Test: [2 7 8]\n",
      "Train: [0 1 3 4 5 6 8] Test: [2 7 9]\n",
      "Train: [0 1 3 4 5 6 7] Test: [2 8 9]\n",
      "Train: [0 1 2 6 7 8 9] Test: [3 4 5]\n",
      "Train: [0 1 2 5 7 8 9] Test: [3 4 6]\n",
      "Train: [0 1 2 5 6 8 9] Test: [3 4 7]\n",
      "Train: [0 1 2 5 6 7 9] Test: [3 4 8]\n",
      "Train: [0 1 2 5 6 7 8] Test: [3 4 9]\n",
      "Train: [0 1 2 4 7 8 9] Test: [3 5 6]\n",
      "Train: [0 1 2 4 6 8 9] Test: [3 5 7]\n",
      "Train: [0 1 2 4 6 7 9] Test: [3 5 8]\n",
      "Train: [0 1 2 4 6 7 8] Test: [3 5 9]\n",
      "Train: [0 1 2 4 5 8 9] Test: [3 6 7]\n",
      "Train: [0 1 2 4 5 7 9] Test: [3 6 8]\n",
      "Train: [0 1 2 4 5 7 8] Test: [3 6 9]\n",
      "Train: [0 1 2 4 5 6 9] Test: [3 7 8]\n",
      "Train: [0 1 2 4 5 6 8] Test: [3 7 9]\n",
      "Train: [0 1 2 4 5 6 7] Test: [3 8 9]\n",
      "Train: [0 1 2 3 7 8 9] Test: [4 5 6]\n",
      "Train: [0 1 2 3 6 8 9] Test: [4 5 7]\n",
      "Train: [0 1 2 3 6 7 9] Test: [4 5 8]\n",
      "Train: [0 1 2 3 6 7 8] Test: [4 5 9]\n",
      "Train: [0 1 2 3 5 8 9] Test: [4 6 7]\n",
      "Train: [0 1 2 3 5 7 9] Test: [4 6 8]\n",
      "Train: [0 1 2 3 5 7 8] Test: [4 6 9]\n",
      "Train: [0 1 2 3 5 6 9] Test: [4 7 8]\n",
      "Train: [0 1 2 3 5 6 8] Test: [4 7 9]\n",
      "Train: [0 1 2 3 5 6 7] Test: [4 8 9]\n",
      "Train: [0 1 2 3 4 8 9] Test: [5 6 7]\n",
      "Train: [0 1 2 3 4 7 9] Test: [5 6 8]\n",
      "Train: [0 1 2 3 4 7 8] Test: [5 6 9]\n",
      "Train: [0 1 2 3 4 6 9] Test: [5 7 8]\n",
      "Train: [0 1 2 3 4 6 8] Test: [5 7 9]\n",
      "Train: [0 1 2 3 4 6 7] Test: [5 8 9]\n",
      "Train: [0 1 2 3 4 5 9] Test: [6 7 8]\n",
      "Train: [0 1 2 3 4 5 8] Test: [6 7 9]\n",
      "Train: [0 1 2 3 4 5 7] Test: [6 8 9]\n",
      "Train: [0 1 2 3 4 5 6] Test: [7 8 9]\n"
     ]
    }
   ],
   "source": [
    "for train_index, test_index in lpo.split(X.values):\n",
    "    print(\"Train:\", train_index, \"Test:\", test_index)\n",
    "    X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n",
    "    y_train, y_test = y.iloc[train_index], y.iloc[test_index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7d4b88b5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Model</th>\n",
       "      <th>Year</th>\n",
       "      <th>Engine Fuel Type</th>\n",
       "      <th>Engine HP</th>\n",
       "      <th>Engine Cylinders</th>\n",
       "      <th>Transmission Type</th>\n",
       "      <th>Driven_Wheels</th>\n",
       "      <th>Number of Doors</th>\n",
       "      <th>Market Category</th>\n",
       "      <th>Vehicle Size</th>\n",
       "      <th>Vehicle Style</th>\n",
       "      <th>highway MPG</th>\n",
       "      <th>city mpg</th>\n",
       "      <th>Popularity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series M</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>335.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Factory Tuner,Luxury,High-Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>26</td>\n",
       "      <td>19</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>28</td>\n",
       "      <td>19</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,High-Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>20</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>26</td>\n",
       "      <td>17</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Make       Model  Year             Engine Fuel Type  Engine HP  \\\n",
       "0  BMW  1 Series M  2011  premium unleaded (required)      335.0   \n",
       "1  BMW    1 Series  2011  premium unleaded (required)      300.0   \n",
       "2  BMW    1 Series  2011  premium unleaded (required)      300.0   \n",
       "3  BMW    1 Series  2011  premium unleaded (required)      230.0   \n",
       "4  BMW    1 Series  2011  premium unleaded (required)      230.0   \n",
       "5  BMW    1 Series  2012  premium unleaded (required)      230.0   \n",
       "6  BMW    1 Series  2012  premium unleaded (required)      300.0   \n",
       "\n",
       "   Engine Cylinders Transmission Type     Driven_Wheels  Number of Doors  \\\n",
       "0               6.0            MANUAL  rear wheel drive              2.0   \n",
       "1               6.0            MANUAL  rear wheel drive              2.0   \n",
       "2               6.0            MANUAL  rear wheel drive              2.0   \n",
       "3               6.0            MANUAL  rear wheel drive              2.0   \n",
       "4               6.0            MANUAL  rear wheel drive              2.0   \n",
       "5               6.0            MANUAL  rear wheel drive              2.0   \n",
       "6               6.0            MANUAL  rear wheel drive              2.0   \n",
       "\n",
       "                         Market Category Vehicle Size Vehicle Style  \\\n",
       "0  Factory Tuner,Luxury,High-Performance      Compact         Coupe   \n",
       "1                     Luxury,Performance      Compact   Convertible   \n",
       "2                Luxury,High-Performance      Compact         Coupe   \n",
       "3                     Luxury,Performance      Compact         Coupe   \n",
       "4                                 Luxury      Compact   Convertible   \n",
       "5                     Luxury,Performance      Compact         Coupe   \n",
       "6                     Luxury,Performance      Compact   Convertible   \n",
       "\n",
       "   highway MPG  city mpg  Popularity  \n",
       "0           26        19        3916  \n",
       "1           28        19        3916  \n",
       "2           28        20        3916  \n",
       "3           28        18        3916  \n",
       "4           28        18        3916  \n",
       "5           28        18        3916  \n",
       "6           26        17        3916  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show the last split\n",
    "X_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c8c40782",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Model</th>\n",
       "      <th>Year</th>\n",
       "      <th>Engine Fuel Type</th>\n",
       "      <th>Engine HP</th>\n",
       "      <th>Engine Cylinders</th>\n",
       "      <th>Transmission Type</th>\n",
       "      <th>Driven_Wheels</th>\n",
       "      <th>Number of Doors</th>\n",
       "      <th>Market Category</th>\n",
       "      <th>Vehicle Size</th>\n",
       "      <th>Vehicle Style</th>\n",
       "      <th>highway MPG</th>\n",
       "      <th>city mpg</th>\n",
       "      <th>Popularity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,High-Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>20</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2013</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>27</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Make     Model  Year             Engine Fuel Type  Engine HP  \\\n",
       "7  BMW  1 Series  2012  premium unleaded (required)      300.0   \n",
       "8  BMW  1 Series  2012  premium unleaded (required)      230.0   \n",
       "9  BMW  1 Series  2013  premium unleaded (required)      230.0   \n",
       "\n",
       "   Engine Cylinders Transmission Type     Driven_Wheels  Number of Doors  \\\n",
       "7               6.0            MANUAL  rear wheel drive              2.0   \n",
       "8               6.0            MANUAL  rear wheel drive              2.0   \n",
       "9               6.0            MANUAL  rear wheel drive              2.0   \n",
       "\n",
       "           Market Category Vehicle Size Vehicle Style  highway MPG  city mpg  \\\n",
       "7  Luxury,High-Performance      Compact         Coupe           28        20   \n",
       "8                   Luxury      Compact   Convertible           28        18   \n",
       "9                   Luxury      Compact   Convertible           27        18   \n",
       "\n",
       "   Popularity  \n",
       "7        3916  \n",
       "8        3916  \n",
       "9        3916  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce94942d",
   "metadata": {},
   "source": [
    "### Leave one out cross-validation (LoOCV) <a class=\"anchor\" id=\"method-3\"></a>\n",
    "\n",
    "It is an exhaustive cross validation method, which is a special case of leave p out cross validation (LpOCV), where p = 1. To get the final accuracy, we must average the accuracies of all iterations.\n",
    "\n",
    "<img src=\"images/loocv.gif\" width=400 height=400 />\n",
    "<a href=\"https://en.wikipedia.org/wiki/File:LOOCV.gif\">Image Source</a>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a1abb0c9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train: [1 2 3 4 5 6 7 8 9] Test: [0]\n",
      "Train: [0 2 3 4 5 6 7 8 9] Test: [1]\n",
      "Train: [0 1 3 4 5 6 7 8 9] Test: [2]\n",
      "Train: [0 1 2 4 5 6 7 8 9] Test: [3]\n",
      "Train: [0 1 2 3 5 6 7 8 9] Test: [4]\n",
      "Train: [0 1 2 3 4 6 7 8 9] Test: [5]\n",
      "Train: [0 1 2 3 4 5 7 8 9] Test: [6]\n",
      "Train: [0 1 2 3 4 5 6 8 9] Test: [7]\n",
      "Train: [0 1 2 3 4 5 6 7 9] Test: [8]\n",
      "Train: [0 1 2 3 4 5 6 7 8] Test: [9]\n"
     ]
    }
   ],
   "source": [
    "loo = LeaveOneOut()\n",
    "for train_index, test_index in loo.split(X):\n",
    "    print(\"Train:\", train_index, \"Test:\", test_index)\n",
    "    X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n",
    "    y_train, y_test = y.iloc[train_index], y.iloc[test_index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6bad888f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Model</th>\n",
       "      <th>Year</th>\n",
       "      <th>Engine Fuel Type</th>\n",
       "      <th>Engine HP</th>\n",
       "      <th>Engine Cylinders</th>\n",
       "      <th>Transmission Type</th>\n",
       "      <th>Driven_Wheels</th>\n",
       "      <th>Number of Doors</th>\n",
       "      <th>Market Category</th>\n",
       "      <th>Vehicle Size</th>\n",
       "      <th>Vehicle Style</th>\n",
       "      <th>highway MPG</th>\n",
       "      <th>city mpg</th>\n",
       "      <th>Popularity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series M</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>335.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Factory Tuner,Luxury,High-Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>26</td>\n",
       "      <td>19</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>28</td>\n",
       "      <td>19</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,High-Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>20</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2011</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>26</td>\n",
       "      <td>17</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>300.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury,High-Performance</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Coupe</td>\n",
       "      <td>28</td>\n",
       "      <td>20</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2012</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Make       Model  Year             Engine Fuel Type  Engine HP  \\\n",
       "0  BMW  1 Series M  2011  premium unleaded (required)      335.0   \n",
       "1  BMW    1 Series  2011  premium unleaded (required)      300.0   \n",
       "2  BMW    1 Series  2011  premium unleaded (required)      300.0   \n",
       "3  BMW    1 Series  2011  premium unleaded (required)      230.0   \n",
       "4  BMW    1 Series  2011  premium unleaded (required)      230.0   \n",
       "5  BMW    1 Series  2012  premium unleaded (required)      230.0   \n",
       "6  BMW    1 Series  2012  premium unleaded (required)      300.0   \n",
       "7  BMW    1 Series  2012  premium unleaded (required)      300.0   \n",
       "8  BMW    1 Series  2012  premium unleaded (required)      230.0   \n",
       "\n",
       "   Engine Cylinders Transmission Type     Driven_Wheels  Number of Doors  \\\n",
       "0               6.0            MANUAL  rear wheel drive              2.0   \n",
       "1               6.0            MANUAL  rear wheel drive              2.0   \n",
       "2               6.0            MANUAL  rear wheel drive              2.0   \n",
       "3               6.0            MANUAL  rear wheel drive              2.0   \n",
       "4               6.0            MANUAL  rear wheel drive              2.0   \n",
       "5               6.0            MANUAL  rear wheel drive              2.0   \n",
       "6               6.0            MANUAL  rear wheel drive              2.0   \n",
       "7               6.0            MANUAL  rear wheel drive              2.0   \n",
       "8               6.0            MANUAL  rear wheel drive              2.0   \n",
       "\n",
       "                         Market Category Vehicle Size Vehicle Style  \\\n",
       "0  Factory Tuner,Luxury,High-Performance      Compact         Coupe   \n",
       "1                     Luxury,Performance      Compact   Convertible   \n",
       "2                Luxury,High-Performance      Compact         Coupe   \n",
       "3                     Luxury,Performance      Compact         Coupe   \n",
       "4                                 Luxury      Compact   Convertible   \n",
       "5                     Luxury,Performance      Compact         Coupe   \n",
       "6                     Luxury,Performance      Compact   Convertible   \n",
       "7                Luxury,High-Performance      Compact         Coupe   \n",
       "8                                 Luxury      Compact   Convertible   \n",
       "\n",
       "   highway MPG  city mpg  Popularity  \n",
       "0           26        19        3916  \n",
       "1           28        19        3916  \n",
       "2           28        20        3916  \n",
       "3           28        18        3916  \n",
       "4           28        18        3916  \n",
       "5           28        18        3916  \n",
       "6           26        17        3916  \n",
       "7           28        20        3916  \n",
       "8           28        18        3916  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show the last split\n",
    "X_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8576f42d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Make</th>\n",
       "      <th>Model</th>\n",
       "      <th>Year</th>\n",
       "      <th>Engine Fuel Type</th>\n",
       "      <th>Engine HP</th>\n",
       "      <th>Engine Cylinders</th>\n",
       "      <th>Transmission Type</th>\n",
       "      <th>Driven_Wheels</th>\n",
       "      <th>Number of Doors</th>\n",
       "      <th>Market Category</th>\n",
       "      <th>Vehicle Size</th>\n",
       "      <th>Vehicle Style</th>\n",
       "      <th>highway MPG</th>\n",
       "      <th>city mpg</th>\n",
       "      <th>Popularity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>BMW</td>\n",
       "      <td>1 Series</td>\n",
       "      <td>2013</td>\n",
       "      <td>premium unleaded (required)</td>\n",
       "      <td>230.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>MANUAL</td>\n",
       "      <td>rear wheel drive</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Luxury</td>\n",
       "      <td>Compact</td>\n",
       "      <td>Convertible</td>\n",
       "      <td>27</td>\n",
       "      <td>18</td>\n",
       "      <td>3916</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Make     Model  Year             Engine Fuel Type  Engine HP  \\\n",
       "9  BMW  1 Series  2013  premium unleaded (required)      230.0   \n",
       "\n",
       "   Engine Cylinders Transmission Type     Driven_Wheels  Number of Doors  \\\n",
       "9               6.0            MANUAL  rear wheel drive              2.0   \n",
       "\n",
       "  Market Category Vehicle Size Vehicle Style  highway MPG  city mpg  \\\n",
       "9          Luxury      Compact   Convertible           27        18   \n",
       "\n",
       "   Popularity  \n",
       "9        3916  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c00adcea",
   "metadata": {},
   "source": [
    "### K-fold cross-validation <a class=\"anchor\" id=\"method-4\"></a>\n",
    "\n",
    "In this method, the dataset is divided into K equal sections (folds). Then at the time of training, one of these sections is selected to be used as a validation set and the remaining sections (K-1) are used as a training set. This process is repeated until each section (fold) has been used as a validation set.\n",
    "\n",
    "<img src=\"images/KfoldCV.gif\" width=400 height=400 />\n",
    "<a href=\"https://en.wikipedia.org/wiki/File:KfoldCV.gif\">Image Source</a>\n",
    "\n",
    "**Note**: In some experiments, the training data may be unbalanced. If this happens, the trained model may get biased. K-fold cross validation is not careful in this aspect at the moment of the split. One solution to this problem is to use the approach Stratified k-fold cross-validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "f523cd50",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.],\n",
       "       [1.]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create some data for a classification problem with unbalanced data\n",
    "X_unb, y_unb = np.ones((16, 1)), np.hstack(([0] * 12, [1] * 4))\n",
    "X_unb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6e89037e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_unb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "8724e351",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train: [ 4  5  6  7  8  9 10 11 12 13 14 15] Test: [0 1 2 3]\n",
      "Train: [ 0  1  2  3  8  9 10 11 12 13 14 15] Test: [4 5 6 7]\n",
      "Train: [ 0  1  2  3  4  5  6  7 12 13 14 15] Test: [ 8  9 10 11]\n",
      "Train: [ 0  1  2  3  4  5  6  7  8  9 10 11] Test: [12 13 14 15]\n"
     ]
    }
   ],
   "source": [
    "kf = KFold(n_splits=4)\n",
    "for train_index, test_index in kf.split(X_unb, y_unb):\n",
    "    print(\"Train:\", train_index, \"Test:\", test_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c85201ba",
   "metadata": {},
   "source": [
    "However, as we can see below, the training and test sets can be made up of a single class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "459b4460",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classes Distribution ([0, 1]):  Train -  [8 4]   |   Test -  [4]\n",
      "Classes Distribution ([0, 1]):  Train -  [8 4]   |   Test -  [4]\n",
      "Classes Distribution ([0, 1]):  Train -  [8 4]   |   Test -  [4]\n",
      "Classes Distribution ([0, 1]):  Train -  [12]   |   Test -  [0 4]\n"
     ]
    }
   ],
   "source": [
    "for train_index, test_index in kf.split(X_unb, y_unb):\n",
    "    print('Classes Distribution ([0, 1]):  Train -  {}   |   Test -  {}'.format(\n",
    "    np.bincount(y_unb[train_index]), np.bincount(y_unb[test_index])))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ad0b4b6",
   "metadata": {},
   "source": [
    "### Stratified k-fold cross-validation <a class=\"anchor\" id=\"method-5\"></a>\n",
    "\n",
    "This cross validation approach is similar to k-folds cross validation in that the data is divided into k-folds. However, by rearranging the data, it ensures that the distribution / proportion of the data in each fold is representative of the entire dataset. Thus it is possible to use this approach on unbalanced datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "613e2906",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train: [ 3  4  5  6  7  8  9 10 11 13 14 15] Test: [ 0  1  2 12]\n",
      "Train: [ 0  1  2  6  7  8  9 10 11 12 14 15] Test: [ 3  4  5 13]\n",
      "Train: [ 0  1  2  3  4  5  9 10 11 12 13 15] Test: [ 6  7  8 14]\n",
      "Train: [ 0  1  2  3  4  5  6  7  8 12 13 14] Test: [ 9 10 11 15]\n"
     ]
    }
   ],
   "source": [
    "skf = StratifiedKFold(n_splits=4)\n",
    "for train_index, test_index in skf.split(X_unb, y_unb):\n",
    "    print(\"Train:\", train_index, \"Test:\", test_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "09a181eb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classes Distribution ([0, 1]):  Train -  [9 3]   |   Test -  [3 1]\n",
      "Classes Distribution ([0, 1]):  Train -  [9 3]   |   Test -  [3 1]\n",
      "Classes Distribution ([0, 1]):  Train -  [9 3]   |   Test -  [3 1]\n",
      "Classes Distribution ([0, 1]):  Train -  [9 3]   |   Test -  [3 1]\n"
     ]
    }
   ],
   "source": [
    "for train_index, test_index in skf.split(X_unb, y_unb):\n",
    "    print('Classes Distribution ([0, 1]):  Train -  {}   |   Test -  {}'.format(\n",
    "    np.bincount(y_unb[train_index]), np.bincount(y_unb[test_index])))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "720b3244",
   "metadata": {},
   "source": [
    "**Note**: As can be seen, unlike k-fold, stratified k-fold cross-validation takes into account the distribution of the classes when doing the split, and can be used in unbalanced datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ff51706",
   "metadata": {},
   "source": [
    "## References\n",
    "<ul>\n",
    "    <li><a href=\"https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation-and-model-selection\">scikit-learn- Cross-validation: evaluating estimator performance</a></li>\n",
    "    <li><a href=\"https://www.analyticsvidhya.com/blog/2021/05/4-ways-to-evaluate-your-machine-learning-model-cross-validation-techniques-with-python-code/\">4 Ways to Evaluate your Machine Learning Model: Cross-Validation Techniques (with Python code)</a> </li>\n",
    "    <li><a href=\"https://towardsdatascience.com/understanding-8-types-of-cross-validation-80c935a4976d\">Understanding 8 types of Cross-Validation</a> </li>\n",
    "    <li><a href=\"https://www.analyticssteps.com/blogs/7-types-cross-validation\">7 Types of Cross-validation</a> </li>\n",
    "    <li><a href=\"https://www.kaggle.com/general/204878\">Cross Validation and its types!</a> </li>\n",
    "    <li><a href=\"https://medium.com/nerd-for-tech/cross-validation-and-types-a7498a68f413\">Cross Validation and types</a> </li>\n",
    "    <li><a href=\"https://www.mygreatlearning.com/blog/cross-validation/\">What is Cross Validation in Machine learning? Types of Cross Validation</a> </li>\n",
    "\n",
    "</ul>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}