tutorials/data_preparation/missing_data/missing_data_introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0228b350",
   "metadata": {},
   "source": [
    "# Introduction to Missing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba659ddd",
   "metadata": {},
   "source": [
    "Real-world domains often have [missing data](https://ydata.ai/resources/what-is-missing-data-in-machine-learning). Data can have missing values for a [number of reasons](https://ydata.ai/resources/understanding-missing-data-mechanisms) such as observations that were not recorded or data corruption. \n",
    "\n",
    "Handling missing data is important as many machine learning algorithms do not support data with missing values. And event when they do, their predictions can be biased due to the presence of missing information.\n",
    "\n",
    "In this tutorial, you will discover how to handle missing data for machine learning with Python. \n",
    "\n",
    "This is what we will cover:\n",
    "\n",
    "1. How to mark invalid or corrupt values as missing in your dataset\n",
    "2. How to confirm that the presence of marked missing values causes problems for learning algorithms\n",
    "3. How to remove rows with missing data from your dataset and evaluate a learning algorithm on the transformed dataset.\n",
    "\n",
    "To follow the tutorial, you can download the data from Kaggle: [Pima Indians Dataset](https://www.kaggle.com/datasets/uciml/pima-indians-diabetes-database)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c52959f",
   "metadata": {},
   "source": [
    "## 1. How to identify and mark Missing Values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4eb66b1f",
   "metadata": {},
   "source": [
    "As the basis of this tutorial, we will use the[Pima Indians dataset](https://www.kaggle.com/datasets/uciml/pima-indians-diabetes-database) (also called \"diabetes\" dataset) that has been widely studied as a machine learning dataset since the 1990s. \n",
    "\n",
    "The dataset classifies patient data as either an onset of diabetes within five years or not. \n",
    "\n",
    "It is a binary classification problem and there are 768 examples and 8 input variables.\n",
    "\n",
    "You can learn more about this dataset by following our previous tutorials on [descriptive statistics](https://github.com/Data-Centric-AI-Community/awesome-python-for-data-science/blob/main/tutorials/data_descriptive_statistics.ipynb) and [data visualization](https://github.com/Data-Centric-AI-Community/awesome-python-for-data-science/blob/main/tutorials/data_basic_visualization.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "139c5075",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "17a541fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"../../data/diabetes.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "00c7ca16",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "0            6      148             72             35        0  33.6   \n",
       "1            1       85             66             29        0  26.6   \n",
       "2            8      183             64              0        0  23.3   \n",
       "3            1       89             66             23       94  28.1   \n",
       "4            0      137             40             35      168  43.1   \n",
       "\n",
       "   DiabetesPedigreeFunction  Age  Outcome  \n",
       "0                     0.627   50        1  \n",
       "1                     0.351   31        0  \n",
       "2                     0.672   32        1  \n",
       "3                     0.167   21        0  \n",
       "4                     2.288   33        1  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "83c14bea",
   "metadata": {},
   "outputs": [],
   "source": [
    "rows, cols = data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "148b1ff5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "768"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "f02943ae",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols # Outcome is the target, so there are 9 feature, but 8 predictors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2d78364",
   "metadata": {},
   "source": [
    "Looking at the data, we can see that all nine input variables are numerical."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e78b2765",
   "metadata": {},
   "source": [
    "**This dataset is known to have missing values.**\n",
    "\n",
    "Specifically, there are missing observations for some columns that are marked as a zero value. \n",
    "\n",
    "We can corroborate this by the definition of those columns and the domain knowledge that a zero value is invalid for those measures, e.g. a zero for body mass index or blood pressure is invalid.\n",
    "\n",
    "Most data has missing values, and the likelihood of having missing values increases with the size of the dataset.\n",
    "\n",
    "**Let's identify and mark values as missing. We can use plots and summary statistics to help identify missing or corrupt data.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a01fe3c7",
   "metadata": {},
   "source": [
    "We can load the dataset as a Pandas DataFrame and print summary statistics on each attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "7bee52e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3.845052</td>\n",
       "      <td>120.894531</td>\n",
       "      <td>69.105469</td>\n",
       "      <td>20.536458</td>\n",
       "      <td>79.799479</td>\n",
       "      <td>31.992578</td>\n",
       "      <td>0.471876</td>\n",
       "      <td>33.240885</td>\n",
       "      <td>0.348958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>3.369578</td>\n",
       "      <td>31.972618</td>\n",
       "      <td>19.355807</td>\n",
       "      <td>15.952218</td>\n",
       "      <td>115.244002</td>\n",
       "      <td>7.884160</td>\n",
       "      <td>0.331329</td>\n",
       "      <td>11.760232</td>\n",
       "      <td>0.476951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.078000</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>27.300000</td>\n",
       "      <td>0.243750</td>\n",
       "      <td>24.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>117.000000</td>\n",
       "      <td>72.000000</td>\n",
       "      <td>23.000000</td>\n",
       "      <td>30.500000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>0.372500</td>\n",
       "      <td>29.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6.000000</td>\n",
       "      <td>140.250000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>127.250000</td>\n",
       "      <td>36.600000</td>\n",
       "      <td>0.626250</td>\n",
       "      <td>41.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>17.000000</td>\n",
       "      <td>199.000000</td>\n",
       "      <td>122.000000</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>846.000000</td>\n",
       "      <td>67.100000</td>\n",
       "      <td>2.420000</td>\n",
       "      <td>81.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Pregnancies     Glucose  BloodPressure  SkinThickness     Insulin  \\\n",
       "count   768.000000  768.000000     768.000000     768.000000  768.000000   \n",
       "mean      3.845052  120.894531      69.105469      20.536458   79.799479   \n",
       "std       3.369578   31.972618      19.355807      15.952218  115.244002   \n",
       "min       0.000000    0.000000       0.000000       0.000000    0.000000   \n",
       "25%       1.000000   99.000000      62.000000       0.000000    0.000000   \n",
       "50%       3.000000  117.000000      72.000000      23.000000   30.500000   \n",
       "75%       6.000000  140.250000      80.000000      32.000000  127.250000   \n",
       "max      17.000000  199.000000     122.000000      99.000000  846.000000   \n",
       "\n",
       "              BMI  DiabetesPedigreeFunction         Age     Outcome  \n",
       "count  768.000000                768.000000  768.000000  768.000000  \n",
       "mean    31.992578                  0.471876   33.240885    0.348958  \n",
       "std      7.884160                  0.331329   11.760232    0.476951  \n",
       "min      0.000000                  0.078000   21.000000    0.000000  \n",
       "25%     27.300000                  0.243750   24.000000    0.000000  \n",
       "50%     32.000000                  0.372500   29.000000    0.000000  \n",
       "75%     36.600000                  0.626250   41.000000    1.000000  \n",
       "max     67.100000                  2.420000   81.000000    1.000000  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02975230",
   "metadata": {},
   "source": [
    "This is useful. We can see that there are columns that have a minimum value of zero (0). On some columns, a value of zero does not make sense and indicates an invalid or missing value.\n",
    "\n",
    "**Missing values are frequently indicated by out-of-range entries; perhaps a negative number (e.g., -1) in a numeric field that is normally only positive, or a 0 in a numeric field that can never normally be 0.**\n",
    "\n",
    "Specifically, the following columns have an invalid zero minimum value:\n",
    "\n",
    "- Plasma glucose concentration (`Glucose`)\n",
    "- Diastolic Blood pressure (`Blood Pressure`)\n",
    "- Triceps skinfold thickness (`SkinThickness`)\n",
    "- 2-Hour serum insulin (`Insulin`)\n",
    "- Body Mass Index (`BMI`)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "868b940d",
   "metadata": {},
   "source": [
    "**We can get a count of the number of missing values on each of these columns.**\n",
    "\n",
    "We can do this by marking all of the values in the subset of the DataFrame we are interested in that have zero values as `Treu`. We can then count the number of true values in each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "10325faf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Summarizing the number of missing values for each feature\n",
    "data_missing = data[[\"Glucose\",\"BloodPressure\", \"SkinThickness\",\"Insulin\", \"BMI\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "096cedcb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>89</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>137</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>763</th>\n",
       "      <td>101</td>\n",
       "      <td>76</td>\n",
       "      <td>48</td>\n",
       "      <td>180</td>\n",
       "      <td>32.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>764</th>\n",
       "      <td>122</td>\n",
       "      <td>70</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "      <td>36.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>765</th>\n",
       "      <td>121</td>\n",
       "      <td>72</td>\n",
       "      <td>23</td>\n",
       "      <td>112</td>\n",
       "      <td>26.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>766</th>\n",
       "      <td>126</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>767</th>\n",
       "      <td>93</td>\n",
       "      <td>70</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "      <td>30.4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>768 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Glucose  BloodPressure  SkinThickness  Insulin   BMI\n",
       "0        148             72             35        0  33.6\n",
       "1         85             66             29        0  26.6\n",
       "2        183             64              0        0  23.3\n",
       "3         89             66             23       94  28.1\n",
       "4        137             40             35      168  43.1\n",
       "..       ...            ...            ...      ...   ...\n",
       "763      101             76             48      180  32.9\n",
       "764      122             70             27        0  36.8\n",
       "765      121             72             23      112  26.2\n",
       "766      126             60              0        0  30.1\n",
       "767       93             70             31        0  30.4\n",
       "\n",
       "[768 rows x 5 columns]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_missing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "6cd99e99",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_missing = (data_missing == 0).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "44aeaffb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Glucose            5\n",
       "BloodPressure     35\n",
       "SkinThickness    227\n",
       "Insulin          374\n",
       "BMI               11\n",
       "dtype: int64"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_missing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8208fe1",
   "metadata": {},
   "source": [
    "We can see that columns Glucose, BloodPressure and BMI have just a few zero values, whereas columns SkinThickness and Insulin show a lot more, nearly half of the rows. \n",
    "\n",
    "**This highlights that different missing value strategies may be needed for different columns**, e.g. to ensure that there are still a sufficient number of records left to train a predictive model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d79eb49",
   "metadata": {},
   "source": [
    "**In Python, specifically Pandas, NumPy and Scikit-Learn, we mark missing values as NaN.** \n",
    "\n",
    "Values with a NaN value are ignored from operations like sum, count, etc. \n",
    "\n",
    "We can mark values as NaN easily with the Pandas DataFrame by using the `replace()` function on a subset of the columns we are interested in. \n",
    "\n",
    "After we have marked the missing values, we can use the `isna()` or `isnull()` functions to mark all of the NaN values in the dataset as `True` and get a count of the missing values for each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "d77914e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Marking Missing Values with NaN values\n",
    "import numpy as np\n",
    "\n",
    "data.iloc[:, [1, 2, 3, 4, 5]] = data.iloc[:, [1, 2, 3, 4, 5]].replace(0, np.nan)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbd91edf",
   "metadata": {},
   "source": [
    "In the code above, we're using `iloc` to make the code easider to read. Columns 1 to 5 simply correspond to Glucose, BloodPressure\", SkinThickness, Insulin, and BMI, respectively. To access NaN values, we also needed to import the `numpy` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "87973f5a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pregnancies                   0\n",
       "Glucose                       5\n",
       "BloodPressure                35\n",
       "SkinThickness               227\n",
       "Insulin                     374\n",
       "BMI                          11\n",
       "DiabetesPedigreeFunction      0\n",
       "Age                           0\n",
       "Outcome                       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08447df1",
   "metadata": {},
   "source": [
    "The code above prints the number of missing values in each column. We can see that columns 1 to 5 have the same number of missing values as zero values identified above. This is a sign that we have marked the identified missing values correctly.\n",
    "\n",
    "If we want to confirm that we have not fooled ourselves somehow, we can print the first 20 rows of data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "a5f0518f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>168.0</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>116.0</td>\n",
       "      <td>74.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>25.6</td>\n",
       "      <td>0.201</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3</td>\n",
       "      <td>78.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>88.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>0.248</td>\n",
       "      <td>26</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>10</td>\n",
       "      <td>115.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>35.3</td>\n",
       "      <td>0.134</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2</td>\n",
       "      <td>197.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>543.0</td>\n",
       "      <td>30.5</td>\n",
       "      <td>0.158</td>\n",
       "      <td>53</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>8</td>\n",
       "      <td>125.0</td>\n",
       "      <td>96.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.232</td>\n",
       "      <td>54</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4</td>\n",
       "      <td>110.0</td>\n",
       "      <td>92.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>37.6</td>\n",
       "      <td>0.191</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>10</td>\n",
       "      <td>168.0</td>\n",
       "      <td>74.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>38.0</td>\n",
       "      <td>0.537</td>\n",
       "      <td>34</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>10</td>\n",
       "      <td>139.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>27.1</td>\n",
       "      <td>1.441</td>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1</td>\n",
       "      <td>189.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>846.0</td>\n",
       "      <td>30.1</td>\n",
       "      <td>0.398</td>\n",
       "      <td>59</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>5</td>\n",
       "      <td>166.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>175.0</td>\n",
       "      <td>25.8</td>\n",
       "      <td>0.587</td>\n",
       "      <td>51</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>7</td>\n",
       "      <td>100.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>30.0</td>\n",
       "      <td>0.484</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0</td>\n",
       "      <td>118.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>47.0</td>\n",
       "      <td>230.0</td>\n",
       "      <td>45.8</td>\n",
       "      <td>0.551</td>\n",
       "      <td>31</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>7</td>\n",
       "      <td>107.0</td>\n",
       "      <td>74.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29.6</td>\n",
       "      <td>0.254</td>\n",
       "      <td>31</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1</td>\n",
       "      <td>103.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>83.0</td>\n",
       "      <td>43.3</td>\n",
       "      <td>0.183</td>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>1</td>\n",
       "      <td>115.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>96.0</td>\n",
       "      <td>34.6</td>\n",
       "      <td>0.529</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "0             6    148.0           72.0           35.0      NaN  33.6   \n",
       "1             1     85.0           66.0           29.0      NaN  26.6   \n",
       "2             8    183.0           64.0            NaN      NaN  23.3   \n",
       "3             1     89.0           66.0           23.0     94.0  28.1   \n",
       "4             0    137.0           40.0           35.0    168.0  43.1   \n",
       "5             5    116.0           74.0            NaN      NaN  25.6   \n",
       "6             3     78.0           50.0           32.0     88.0  31.0   \n",
       "7            10    115.0            NaN            NaN      NaN  35.3   \n",
       "8             2    197.0           70.0           45.0    543.0  30.5   \n",
       "9             8    125.0           96.0            NaN      NaN   NaN   \n",
       "10            4    110.0           92.0            NaN      NaN  37.6   \n",
       "11           10    168.0           74.0            NaN      NaN  38.0   \n",
       "12           10    139.0           80.0            NaN      NaN  27.1   \n",
       "13            1    189.0           60.0           23.0    846.0  30.1   \n",
       "14            5    166.0           72.0           19.0    175.0  25.8   \n",
       "15            7    100.0            NaN            NaN      NaN  30.0   \n",
       "16            0    118.0           84.0           47.0    230.0  45.8   \n",
       "17            7    107.0           74.0            NaN      NaN  29.6   \n",
       "18            1    103.0           30.0           38.0     83.0  43.3   \n",
       "19            1    115.0           70.0           30.0     96.0  34.6   \n",
       "\n",
       "    DiabetesPedigreeFunction  Age  Outcome  \n",
       "0                      0.627   50        1  \n",
       "1                      0.351   31        0  \n",
       "2                      0.672   32        1  \n",
       "3                      0.167   21        0  \n",
       "4                      2.288   33        1  \n",
       "5                      0.201   30        0  \n",
       "6                      0.248   26        1  \n",
       "7                      0.134   29        0  \n",
       "8                      0.158   53        1  \n",
       "9                      0.232   54        1  \n",
       "10                     0.191   30        0  \n",
       "11                     0.537   34        1  \n",
       "12                     1.441   57        0  \n",
       "13                     0.398   59        1  \n",
       "14                     0.587   51        1  \n",
       "15                     0.484   32        1  \n",
       "16                     0.551   31        1  \n",
       "17                     0.254   31        1  \n",
       "18                     0.183   33        0  \n",
       "19                     0.529   32        1  "
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a204a335",
   "metadata": {},
   "source": [
    "Running the example, we can clearly see NaN values in the columns 2, 3, 4 and 5. There are only 5 missing values in column 1, so it is not surprising we did not see an example in the first 20 rows. It is clear from the raw data that marking the missing values had the intended effect."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57c3f006",
   "metadata": {},
   "source": [
    "## 2. Confirm that missing values cause problems for machine learning algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9449acc",
   "metadata": {},
   "source": [
    "Before we look at handling missing values, let’s first demonstrate that having missing values in a dataset can cause problems.\n",
    "\n",
    "Although missing values are common occurrences in data, **most predictive modeling techniques cannot handle any missing values**. Therefore, this problem must be addressed prior to modeling.\n",
    "\n",
    "We'll try to evaluate the Linear Discriminant Analysis (LDA) algorithm on the dataset with missing values. This is an algorithm that does not work when there are missing values in the dataset. The example below marks the missing values in the dataset, as we did in the previous section, then attempts to evaluate LDA using 3-fold cross-validation and print the mean accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "70a92ed3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary libraries\n",
    "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "76fb4856",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split data into inputs and outputs\n",
    "X = data.drop('Outcome', axis=1)\n",
    "y = data['Outcome']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "26a9517e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>168.0</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>763</th>\n",
       "      <td>10</td>\n",
       "      <td>101.0</td>\n",
       "      <td>76.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>180.0</td>\n",
       "      <td>32.9</td>\n",
       "      <td>0.171</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>764</th>\n",
       "      <td>2</td>\n",
       "      <td>122.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>36.8</td>\n",
       "      <td>0.340</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>765</th>\n",
       "      <td>5</td>\n",
       "      <td>121.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>112.0</td>\n",
       "      <td>26.2</td>\n",
       "      <td>0.245</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>766</th>\n",
       "      <td>1</td>\n",
       "      <td>126.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>30.1</td>\n",
       "      <td>0.349</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>767</th>\n",
       "      <td>1</td>\n",
       "      <td>93.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>30.4</td>\n",
       "      <td>0.315</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>768 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "0              6    148.0           72.0           35.0      NaN  33.6   \n",
       "1              1     85.0           66.0           29.0      NaN  26.6   \n",
       "2              8    183.0           64.0            NaN      NaN  23.3   \n",
       "3              1     89.0           66.0           23.0     94.0  28.1   \n",
       "4              0    137.0           40.0           35.0    168.0  43.1   \n",
       "..           ...      ...            ...            ...      ...   ...   \n",
       "763           10    101.0           76.0           48.0    180.0  32.9   \n",
       "764            2    122.0           70.0           27.0      NaN  36.8   \n",
       "765            5    121.0           72.0           23.0    112.0  26.2   \n",
       "766            1    126.0           60.0            NaN      NaN  30.1   \n",
       "767            1     93.0           70.0           31.0      NaN  30.4   \n",
       "\n",
       "     DiabetesPedigreeFunction  Age  \n",
       "0                       0.627   50  \n",
       "1                       0.351   31  \n",
       "2                       0.672   32  \n",
       "3                       0.167   21  \n",
       "4                       2.288   33  \n",
       "..                        ...  ...  \n",
       "763                     0.171   63  \n",
       "764                     0.340   27  \n",
       "765                     0.245   30  \n",
       "766                     0.349   47  \n",
       "767                     0.315   23  \n",
       "\n",
       "[768 rows x 8 columns]"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "265dcfcf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      1\n",
       "1      0\n",
       "2      1\n",
       "3      0\n",
       "4      1\n",
       "      ..\n",
       "763    0\n",
       "764    0\n",
       "765    0\n",
       "766    1\n",
       "767    0\n",
       "Name: Outcome, Length: 768, dtype: int64"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "dbcbd0f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split your data into training and testing sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "c926cc05",
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Input X contains NaN.\nLinearDiscriminantAnalysis does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[73], line 5\u001b[0m\n\u001b[1;32m      2\u001b[0m lda \u001b[38;5;241m=\u001b[39m LinearDiscriminantAnalysis()\n\u001b[1;32m      4\u001b[0m \u001b[38;5;66;03m# Train the LDA model using the training data\u001b[39;00m\n\u001b[0;32m----> 5\u001b[0m lda\u001b[38;5;241m.\u001b[39mfit(X_train, y_train)\n",
      "File \u001b[0;32m~/opt/anaconda3/envs/ds-env/lib/python3.11/site-packages/sklearn/discriminant_analysis.py:575\u001b[0m, in \u001b[0;36mLinearDiscriminantAnalysis.fit\u001b[0;34m(self, X, y)\u001b[0m\n\u001b[1;32m    571\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_validate_params()\n\u001b[1;32m    573\u001b[0m xp, _ \u001b[38;5;241m=\u001b[39m get_namespace(X)\n\u001b[0;32m--> 575\u001b[0m X, y \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_validate_data(\n\u001b[1;32m    576\u001b[0m     X, y, ensure_min_samples\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m2\u001b[39m, dtype\u001b[38;5;241m=\u001b[39m[xp\u001b[38;5;241m.\u001b[39mfloat64, xp\u001b[38;5;241m.\u001b[39mfloat32]\n\u001b[1;32m    577\u001b[0m )\n\u001b[1;32m    578\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mclasses_ \u001b[38;5;241m=\u001b[39m unique_labels(y)\n\u001b[1;32m    579\u001b[0m n_samples, _ \u001b[38;5;241m=\u001b[39m X\u001b[38;5;241m.\u001b[39mshape\n",
      "File \u001b[0;32m~/opt/anaconda3/envs/ds-env/lib/python3.11/site-packages/sklearn/base.py:584\u001b[0m, in \u001b[0;36mBaseEstimator._validate_data\u001b[0;34m(self, X, y, reset, validate_separately, **check_params)\u001b[0m\n\u001b[1;32m    582\u001b[0m         y \u001b[38;5;241m=\u001b[39m check_array(y, input_name\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124my\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mcheck_y_params)\n\u001b[1;32m    583\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m--> 584\u001b[0m         X, y \u001b[38;5;241m=\u001b[39m check_X_y(X, y, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mcheck_params)\n\u001b[1;32m    585\u001b[0m     out \u001b[38;5;241m=\u001b[39m X, y\n\u001b[1;32m    587\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m no_val_X \u001b[38;5;129;01mand\u001b[39;00m check_params\u001b[38;5;241m.\u001b[39mget(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mensure_2d\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;28;01mTrue\u001b[39;00m):\n",
      "File \u001b[0;32m~/opt/anaconda3/envs/ds-env/lib/python3.11/site-packages/sklearn/utils/validation.py:1106\u001b[0m, in \u001b[0;36mcheck_X_y\u001b[0;34m(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)\u001b[0m\n\u001b[1;32m   1101\u001b[0m         estimator_name \u001b[38;5;241m=\u001b[39m _check_estimator_name(estimator)\n\u001b[1;32m   1102\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m   1103\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mestimator_name\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m requires y to be passed, but the target y is None\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m   1104\u001b[0m     )\n\u001b[0;32m-> 1106\u001b[0m X \u001b[38;5;241m=\u001b[39m check_array(\n\u001b[1;32m   1107\u001b[0m     X,\n\u001b[1;32m   1108\u001b[0m     accept_sparse\u001b[38;5;241m=\u001b[39maccept_sparse,\n\u001b[1;32m   1109\u001b[0m     accept_large_sparse\u001b[38;5;241m=\u001b[39maccept_large_sparse,\n\u001b[1;32m   1110\u001b[0m     dtype\u001b[38;5;241m=\u001b[39mdtype,\n\u001b[1;32m   1111\u001b[0m     order\u001b[38;5;241m=\u001b[39morder,\n\u001b[1;32m   1112\u001b[0m     copy\u001b[38;5;241m=\u001b[39mcopy,\n\u001b[1;32m   1113\u001b[0m     force_all_finite\u001b[38;5;241m=\u001b[39mforce_all_finite,\n\u001b[1;32m   1114\u001b[0m     ensure_2d\u001b[38;5;241m=\u001b[39mensure_2d,\n\u001b[1;32m   1115\u001b[0m     allow_nd\u001b[38;5;241m=\u001b[39mallow_nd,\n\u001b[1;32m   1116\u001b[0m     ensure_min_samples\u001b[38;5;241m=\u001b[39mensure_min_samples,\n\u001b[1;32m   1117\u001b[0m     ensure_min_features\u001b[38;5;241m=\u001b[39mensure_min_features,\n\u001b[1;32m   1118\u001b[0m     estimator\u001b[38;5;241m=\u001b[39mestimator,\n\u001b[1;32m   1119\u001b[0m     input_name\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mX\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[1;32m   1120\u001b[0m )\n\u001b[1;32m   1122\u001b[0m y \u001b[38;5;241m=\u001b[39m _check_y(y, multi_output\u001b[38;5;241m=\u001b[39mmulti_output, y_numeric\u001b[38;5;241m=\u001b[39my_numeric, estimator\u001b[38;5;241m=\u001b[39mestimator)\n\u001b[1;32m   1124\u001b[0m check_consistent_length(X, y)\n",
      "File \u001b[0;32m~/opt/anaconda3/envs/ds-env/lib/python3.11/site-packages/sklearn/utils/validation.py:921\u001b[0m, in \u001b[0;36mcheck_array\u001b[0;34m(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)\u001b[0m\n\u001b[1;32m    915\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m    916\u001b[0m             \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mFound array with dim \u001b[39m\u001b[38;5;132;01m%d\u001b[39;00m\u001b[38;5;124m. \u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m expected <= 2.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    917\u001b[0m             \u001b[38;5;241m%\u001b[39m (array\u001b[38;5;241m.\u001b[39mndim, estimator_name)\n\u001b[1;32m    918\u001b[0m         )\n\u001b[1;32m    920\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m force_all_finite:\n\u001b[0;32m--> 921\u001b[0m         _assert_all_finite(\n\u001b[1;32m    922\u001b[0m             array,\n\u001b[1;32m    923\u001b[0m             input_name\u001b[38;5;241m=\u001b[39minput_name,\n\u001b[1;32m    924\u001b[0m             estimator_name\u001b[38;5;241m=\u001b[39mestimator_name,\n\u001b[1;32m    925\u001b[0m             allow_nan\u001b[38;5;241m=\u001b[39mforce_all_finite \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mallow-nan\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[1;32m    926\u001b[0m         )\n\u001b[1;32m    928\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m ensure_min_samples \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m0\u001b[39m:\n\u001b[1;32m    929\u001b[0m     n_samples \u001b[38;5;241m=\u001b[39m _num_samples(array)\n",
      "File \u001b[0;32m~/opt/anaconda3/envs/ds-env/lib/python3.11/site-packages/sklearn/utils/validation.py:161\u001b[0m, in \u001b[0;36m_assert_all_finite\u001b[0;34m(X, allow_nan, msg_dtype, estimator_name, input_name)\u001b[0m\n\u001b[1;32m    144\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m estimator_name \u001b[38;5;129;01mand\u001b[39;00m input_name \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mX\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01mand\u001b[39;00m has_nan_error:\n\u001b[1;32m    145\u001b[0m     \u001b[38;5;66;03m# Improve the error message on how to handle missing values in\u001b[39;00m\n\u001b[1;32m    146\u001b[0m     \u001b[38;5;66;03m# scikit-learn.\u001b[39;00m\n\u001b[1;32m    147\u001b[0m     msg_err \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m (\n\u001b[1;32m    148\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;132;01m{\u001b[39;00mestimator_name\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m does not accept missing values\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    149\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m encoded as NaN natively. For supervised learning, you might want\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    159\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m#estimators-that-handle-nan-values\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    160\u001b[0m     )\n\u001b[0;32m--> 161\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(msg_err)\n",
      "\u001b[0;31mValueError\u001b[0m: Input X contains NaN.\nLinearDiscriminantAnalysis does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values"
     ]
    }
   ],
   "source": [
    "# Initialize the LDA model\n",
    "lda = LinearDiscriminantAnalysis()\n",
    "\n",
    "# Train the LDA model using the training data\n",
    "lda.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "155dfb61",
   "metadata": {},
   "source": [
    "**As you can see from the above error message, LDA does not accept missing values!**\n",
    "\n",
    "ValueError: Input X contains NaN.\n",
    "\n",
    "LinearDiscriminantAnalysis does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba0c35bc",
   "metadata": {},
   "source": [
    "This is as we expect. We are prevented from evaluating an LDA algorithm (and other algorithms) on the dataset with missing values.\n",
    "\n",
    "**Many popular predictive models such as support vector machines and neural networks, cannot tolerate any amount of missing values.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f982b96",
   "metadata": {},
   "source": [
    "## 3. Remove rows with Missing Values and try again!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3730cd64",
   "metadata": {},
   "source": [
    "The simplest strategy for handling missing data is to remove records that contain a missing value.\n",
    "\n",
    "This is **not always the best approach**, but for an introdutory tutorial to the field, let's do it!\n",
    "\n",
    "We can do this by creating a new Pandas DataFrame with the rows containing missing values removed. Pandas provides the `dropna()` function that can be used to drop either columns or rows with missing data. \n",
    "\n",
    "We can use `dropna()` to remove all rows with missing data, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "d9d1c7f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Start from the beginning, removing rows with missing values! \n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Load the data\n",
    "data = pd.read_csv(\"../../data/diabetes.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "095e984b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "0            6      148             72             35        0  33.6   \n",
       "1            1       85             66             29        0  26.6   \n",
       "2            8      183             64              0        0  23.3   \n",
       "3            1       89             66             23       94  28.1   \n",
       "4            0      137             40             35      168  43.1   \n",
       "\n",
       "   DiabetesPedigreeFunction  Age  Outcome  \n",
       "0                     0.627   50        1  \n",
       "1                     0.351   31        0  \n",
       "2                     0.672   32        1  \n",
       "3                     0.167   21        0  \n",
       "4                     2.288   33        1  "
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "3590c00d",
   "metadata": {},
   "outputs": [],
   "source": [
    "rows, cols = data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "21aa24f3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "768"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "a71a9253",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "5563c67d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace '0' values with 'nan' in columns 1-5\n",
    "data.iloc[:, [1,2,3,4,5]] = data.iloc[:, [1,2,3,4,5]].replace(0, np.nan)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "58997c09",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148.0</td>\n",
       "      <td>72.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>168.0</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "0            6    148.0           72.0           35.0      NaN  33.6   \n",
       "1            1     85.0           66.0           29.0      NaN  26.6   \n",
       "2            8    183.0           64.0            NaN      NaN  23.3   \n",
       "3            1     89.0           66.0           23.0     94.0  28.1   \n",
       "4            0    137.0           40.0           35.0    168.0  43.1   \n",
       "\n",
       "   DiabetesPedigreeFunction  Age  Outcome  \n",
       "0                     0.627   50        1  \n",
       "1                     0.351   31        0  \n",
       "2                     0.672   32        1  \n",
       "3                     0.167   21        0  \n",
       "4                     2.288   33        1  "
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "574a0620",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop missing values\n",
    "data.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1588092b",
   "metadata": {},
   "source": [
    "In the context of the `data.dropna(inplace=True)` method, the `inplace=True` argument is used to modify the DataFrame directly, without creating a new DataFrame. When `inplace=True`, the method will remove rows containing missing values (NaN) from the DataFrame, and the changes will be applied to the original DataFrame object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "446a9ad8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>168.0</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3</td>\n",
       "      <td>78.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>88.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>0.248</td>\n",
       "      <td>26</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2</td>\n",
       "      <td>197.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>543.0</td>\n",
       "      <td>30.5</td>\n",
       "      <td>0.158</td>\n",
       "      <td>53</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1</td>\n",
       "      <td>189.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>846.0</td>\n",
       "      <td>30.1</td>\n",
       "      <td>0.398</td>\n",
       "      <td>59</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "3             1     89.0           66.0           23.0     94.0  28.1   \n",
       "4             0    137.0           40.0           35.0    168.0  43.1   \n",
       "6             3     78.0           50.0           32.0     88.0  31.0   \n",
       "8             2    197.0           70.0           45.0    543.0  30.5   \n",
       "13            1    189.0           60.0           23.0    846.0  30.1   \n",
       "\n",
       "    DiabetesPedigreeFunction  Age  Outcome  \n",
       "3                      0.167   21        0  \n",
       "4                      2.288   33        1  \n",
       "6                      0.248   26        1  \n",
       "8                      0.158   53        1  \n",
       "13                     0.398   59        1  "
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "77870ff3",
   "metadata": {},
   "outputs": [],
   "source": [
    "rows, cols = data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "3d64d1fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "392"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "30462017",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae60a698",
   "metadata": {},
   "source": [
    "We can see that the number of rows has been aggressively cut from 768 in the original dataset to 392 with all rows containing a NaN removed.\n",
    "\n",
    "**We now have a dataset that we could use to evaluate an algorithm sensitive to missing values like LDA.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "1e9bdbf6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LinearDiscriminantAnalysis()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LinearDiscriminantAnalysis</label><div class=\"sk-toggleable__content\"><pre>LinearDiscriminantAnalysis()</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LinearDiscriminantAnalysis()"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import necessary libraries\n",
    "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split data into inputs and outputs\n",
    "X = data.drop('Outcome', axis=1)\n",
    "y = data['Outcome']\n",
    "\n",
    "# Split your data into training and testing sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "# Initialize the LDA model\n",
    "lda = LinearDiscriminantAnalysis()\n",
    "\n",
    "# Train the LDA model using the training data\n",
    "lda.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "5ba0cf7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict on the test set\n",
    "y_pred = lda.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "9895f2cb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.7468354430379747\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.80      0.83      0.81        52\n",
      "           1       0.64      0.59      0.62        27\n",
      "\n",
      "    accuracy                           0.75        79\n",
      "   macro avg       0.72      0.71      0.71        79\n",
      "weighted avg       0.74      0.75      0.74        79\n",
      "\n",
      "Confusion Matrix:\n",
      "[[43  9]\n",
      " [11 16]]\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the model's performance\n",
    "from sklearn.metrics import accuracy_score, classification_report, confusion_matrix\n",
    "\n",
    "# Calculate accuracy\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "print(\"Accuracy:\", accuracy)\n",
    "\n",
    "# Print classification report\n",
    "print(classification_report(y_test, y_pred))\n",
    "\n",
    "# Print confusion matrix\n",
    "print(\"Confusion Matrix:\")\n",
    "print(confusion_matrix(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a194f490",
   "metadata": {},
   "source": [
    "**LDA is not able to operate! However, note that removing rows with missing values can be too limiting on some predictive modeling problems, an alternative is to impute missing values.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f5bd5e5",
   "metadata": {},
   "source": [
    "# What is Next?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33113d67",
   "metadata": {},
   "source": [
    "**Data Imputation!** In the next tutorial, we will explore how we can impute missing data values using statistic methods and machine learning methods!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "986a63a5",
   "metadata": {},
   "source": [
    "### 👾 Join our [Discord community](https://tiny.ydata.ai/dcai-community-github) and follow our Code-with-Me sessions to learn more about data science!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}