Lab 9 - Detect heart disease in anonymous patients.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 9 - Detect heart disease in anonymous patients\n",
    "*© 2021 Colin Conrad*\n",
    "\n",
    "We have made so much progress in the past nine weeks. This lab marks a milestone; it is the final lab in our unit on \"scrappy data science for information managers\", designed to bring you from basic programming to basic data science in Python. In this final lab, we will dig a bit more deeply into data mining by drawing from our visualization and statistics skills, but most importantly, but exploring _classification_. This technique is one of the common tasks of machine learning, a whole world onto itself. The objective of classification is to train a computer algorithm that can detect a target data feature (in this case, whether or not someone has heart disease) based on the other data features. This task will introduce you to some of the key machine learning concepts as they pertain to information managers. We will draw from a prominent Python library called Scikit-Learn to do this (Pedregosa et al., 2011).\n",
    "\n",
    "This lab is perhaps the most technically complex of all of the labs (from a theoretical perspective). To help make this simpler, we will explore a relatively simple data set. In fact, it is one of the most commonly explored data sets for this purpose. In many ways, it is also a perfect example of how information managers could help improve researchers' data governance and knowledge discovery processes. If you are interested in exploring this subject further, I am always happy to take on students for directed studies on data mining or machine learning projects.\n",
    "\n",
    "**This week, we will achieve the following objectives:**\n",
    "- Explore the factors that indicate heart disease\n",
    "- Apply a classifier\n",
    "- Compare and assess the performance of multiple classifiers\n",
    "\n",
    "\n",
    "# Case: The Heart Disease Dataset\n",
    "This week's lab is based on the [UCI Heart Disease Dataset](https://archive.ics.uci.edu/ml/datasets/Heart+Disease) provided by David Aha. The dataset consists of 303 records of patient data collected in the 1980s, as described by Detrano et al. (1989), Aha & Kibler (n.d.) and Gennari et al. (1989). Though the original records collected comprehensive data from patients, the original fields have been corrupted, and only a limited selection of data remain in the repository (_this_ is why we should teach kids to back up their data... you just never know). The 303 patient records that remain consist of the following data features:\n",
    "\n",
    "- age: The person's age in years\n",
    "- sex: The person's sex (1 = male, 0 = female)\n",
    "- cp: The chest pain experienced (Value 1: typical angina, Value 2: atypical angina, Value 3: non-anginal pain, Value 4: asymptomatic)\n",
    "- trestbps: The person's resting blood pressure (mm Hg on admission to the hospital)\n",
    "- chol: The person's cholesterol measurement in mg/dl\n",
    "- fbs: The person's fasting blood sugar (> 120 mg/dl, 1 = true; 0 = false)\n",
    "- restecg: Resting electrocardiographic measurement (0 = normal, 1 = having ST-T wave abnormality, 2 = showing probable or definite left ventricular hypertrophy by Estes' criteria)\n",
    "- thalach: Maximum heart rate achieved\n",
    "- exang: Exercise induced angina (1 = yes; 0 = no)\n",
    "- oldpeak: ST depression induced by exercise relative to rest\n",
    "- slope: the slope of the peak exercise ST segment (Value 1: upsloping, Value 2: flat, Value 3: downsloping)\n",
    "- ca: The number of major vessels (0-3)\n",
    "- thal: A blood disorder called thalassemia (0 = fixed defect; 1 = normal defect; 2 = reversable defect __Note:__ changed for our class for simplicity)\n",
    "- target: Heart disease (0 = no, 1 = yes)\n",
    "\n",
    "This data set has been the subject of hundreds of publications, and all of them (to the best of my knowledge) concern the summarized data similar to that contained in this week's class file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Objective 1: Explore the factors that indicate heart disease\n",
    "This data set is real and interesting. As a data scientist, one of the first things that you would normally do is explore the data set to understand the nuances of the data. Visualization is best suited for this task, hence why we spent so much time on it. Let's start by importing seaborn and exploring the data head."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd # these are the same as Lab 6\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "sns.set()\n",
    "\n",
    "hd = pd.read_csv('data/w9_heart.csv')\n",
    "hd.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This data is well and good, though it is a little vague, especially to non-medical professionals. We should consider first changing the names of the columns to something more descriptive. The following code renames the columns sequentially. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hd.columns = [\n",
    "    'age', \n",
    "    'sex', \n",
    "    'chest_pain_experienced', \n",
    "    'resting_blood_pressure', \n",
    "    'cholesterol', \n",
    "    'fasting_blood_sugar', \n",
    "    'electrocardiograph',\n",
    "    'maximum_heart_rate',\n",
    "    'exercise_induced_angina',\n",
    "    'st_depression',\n",
    "    'st_slope',\n",
    "    'major_vessels',\n",
    "    'thalassemia',\n",
    "    'target'\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "men = hd[hd[\"sex\"] == 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "men"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "means_men = men.groupby(by=\"chest_pain_experienced\").mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "means_men"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "women = hd[hd[\"sex\"] == 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "means_women = women.groupby(by=\"chest_pain_experienced\").mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "means_women"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "means = hd.groupby(by=\"chest_pain_experienced\").mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize the data and understand the indicators\n",
    "One of the first things we should do is understand our task. As mentioned in the introduction, our goal is to build an algorithm that can predict whether or not a patient has heart disease, based on their health data. As such, the first thing we should do is observe the number of target variables, which are the indicators of whether someone actually has heart disease. We will find that this data consists of 165 records of patients who were diagnosed with heart disease (`target == 1`) and 138 records of patients who did not have heart disease (`target == 0`). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.countplot(x=\"target\", data=hd) # countplots are ways of visualizing counts in seaborn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Scatterplots\n",
    "Now that we understand the data, we can explore it. One thing that stood out to me was that we had age and maximum_heart_rate records. It is well known that your heart rate maximum decreases as you age, though people with heart disease also have higher heart rates. A simple way to explore these relationships could be to create a scatter plot that visualizes these three factors. The plot below uses seaborn to achieve this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.scatterplot(data=hd, x=\"maximum_heart_rate\", y=\"age\", hue=\"target\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using stats to assess factors\n",
    "It looks like there may be a relationship between these factors, perhaps most notably, `maximum_heart_rate`. This data is structured in a way that we could easily compare the distribution of our samples, similarly to Lab 7. For example, we could observe the Pearson correlation of `maximum_heart_rate` and `age` to determine whether these are correlated. The code below executes this correlation; they are moderately negatively correlated (as expected) with a very low p-value, suggesting that the correlation is significant. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import stats\n",
    "stats.pearsonr(hd[\"maximum_heart_rate\"], hd[\"age\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we could visualize the `maximum_heart_rate` in light of the target and non-target (diseased and non-disease) records. This will reveal very interesting and suggestive distributions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.displot(hd, x=\"maximum_heart_rate\", hue=\"target\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can similarly run a Pearson correlation on this data, which again suggests a positive correlation between `maximum_heart_rate` and `target`. __Note:__ We could have also run a t-test here to observe whether people with heart disease are likely to have a higher maximum heart rate (which would have indicated ...yes). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stats.pearsonr(hd[\"maximum_heart_rate\"], hd[\"target\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Visualizing nominal variables\n",
    "One of the differences between this data set and what we have seen before is that there are many nominal variables (e.g. variables that are categories). In this case, `thalassemia` indicates one of three states: 0: it has been fixed, 1: the patient has it and 2: that it is \"reversable\". I am not qualified to speak more about this beyond speculation, though the [literature](https://www.sciencedirect.com/science/article/abs/pii/0002914989905249) suggests that this is a relevant predictor. \n",
    "\n",
    "Seaborn's catplot can be used to visualize the counts of these three states. The code below describes the frequencies for each of the target conditions. This brings us to the end of our initial exploration... it will be up to you to continue this!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.catplot(x=\"target\", col=\"thalassemia\", col_wrap=3,\n",
    "                data=hd,\n",
    "                kind=\"count\", height=4, aspect=.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### _Challenge Question 1 (1 point)_\n",
    "Modify and apply one of the plots above to visualize the possible impact of `major_vessels` in predicting heart disease. Do you think this might be a relevant factor for predicting hearth disease? If so (or not) write it in a comment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# insert your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### _Challenge Question 2 (1 point)_:\n",
    "For years dietians and food scientists have stated that cholesterol was a really important factor in heart disease, though other studies have pointed to the contrary. Though we are not qualified to tell science otherwise, we can check to see whether the data, at face value, supports this finding. Drawing from the examples above and your expereince from prior labs, do the following:\n",
    "\n",
    "- Visualize the distribtuions of cholesterol levels for the patient records with heart disease and those who do not have it\n",
    "- Complete a Pearson correlation of the relationship between cholesterol and heart disease\n",
    "- Leave a comment somewhere in your answer about whether it is a significant relationship"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# insert your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import stats\n",
    "stats.pearsonr(hd[\"cholesterol\"], hd[\"target\"]) # not significant"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objective 2: Apply a classifier\n",
    "We're now ready to start exploring classification. The concept of classification is far more difficult than the code that supports it, so please consider watching the _Introduction to Machine Learning_ video linked on Brightspace. This is the \"Colin's Notes\" (*chough* not Coles Notes) version.\n",
    "\n",
    "Algorithms are a series of logical steps used to complete a task. The computer programs that you write in Python are algorithms; your computer interprets these and executes them. However, when it comes to larger or non-obvious data sets, it is impossible to explicitly tell your computer what to do. Last week, for example, we had 500 000 records of e-commerce transactions. Imagine having to sort through 500 000 records to find out which ones were related! _Not going to happen_.\n",
    "\n",
    "Fortunately, we have machine learning. The goal of machine learning is to create algorithms that accomplish tasks without explicitly programed (Grimson, 2017). To do this, machine learning uses a special set of \"machine learning algorithms\" to _fit_ to a set of data. The created algorithm can then (in the case of classification) classify new instances or data that was not part of the original set.\n",
    "\n",
    "To demonstrate classification, we will complete a very simple task. We will use an algorithm called _RandomForest_ to _fit_ to part of our data set (around two thirds). The result will be a algorithm that can tell whether a patient has heart disease. We will then test its performance on the remaining third of the data.\n",
    "\n",
    "To accomplish this task, we will draw from Scikit-Learn, one of Python's most prominent libraries.\n",
    "\n",
    "#### About Random Forest\n",
    "The [Random Forest classifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) is part of an algorithm family called decision trees. This algorithm might be fairly familiar to you; the goal of Random Forest is to create a series of decisions similarly to a flow chart. The algorithm essetnailly observes the data and determines a series of rules that are best suited to classifying the data based on the information gained by the decision tree. When we specify a Random Forest classifier, we can specify the number of rules that it looks for by adjusting the `n_estimators` variable, among others.\n",
    "\n",
    "#### Load the classifier\n",
    "The first thing that we will need to do is load the classifier. We can do this by loading Scikit-Learn's `RandomForestClassifier` object from its `ensamble` library. We will save the classifier as `clf`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier # import sklearn\n",
    "\n",
    "clf = RandomForestClassifier(n_estimators=10, random_state=0) # save the random forest classifier as clf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Resample the data\n",
    "Currently, our data set is structured such that the records that were positive for heart disease consist of the first 165 rows,  while those that are negative occupied the remaining. This is unfriendly to machine learning. If we are going to train our algorithm on a portion of the data, it is important that this is a random sample. We should thus randomize our data. The line below saves a new data frame called `hdr` (heart disease random) that is a randomized version of our old frame. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hdr = hd.sample(frac=1) # save a randomized data frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now divide the data. We will take the first 150 records from `hdr` for training the algorithm and the remaining 153 for testing. __Note__ that this is a simple way of selecting test and training data and is used for teaching purposes. There are other ways of resampling data which might be better suited for this task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = hdr[:150] # take the first 200 records as train\n",
    "test = hdr[150:] # take the ramining 103 records as test\n",
    "\n",
    "print(\"Train: \" + str(len(train)) + \" Test: \" + str(len(test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train the classifier\n",
    "We are now ready to train the classifier by _fitting_ it to the data. Scikit-Learn's `fit()` method takes two inputs: the data that you wish to classify and labels of the data being classified. This part is a little confusing, but is done this was for a very good reason. It is important to separate the labels from the observed training data because the labels are what we want the algorithm to predict. If we included the labels with the training data, the Random Forest would detect the label as the best feature to ... predict the labels.\n",
    "\n",
    "The code below creates a dataframe called `training_observed` consisting of all of the training data _other_ thank the labels and a list consisting of the `target` column data called `training_labels`. We then call RandomForest's `fit()` to the data. We now have a trained algorithm! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_observed = train.drop('target', 1) # training data without target labels\n",
    "training_labels = train['target'] # the target column as labels\n",
    "\n",
    "clf.fit(training_observed, training_labels) # fit the training data and the labels to create a classifier called clf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create predictions\n",
    "RandomForest is now ready for action. We can test the algorithm's performance on the remaining 1/3 of the data that we saved earlier. The code below similarly separates the labels out of the `test` data. Using this, we can create predictions. The code below separates the data as before, but also uses the data to create predictions, which is saved in `preds`. You can see the predictions that the algorithm made below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_observed = test.drop('target', 1) # test data without labels\n",
    "\n",
    "preds = clf.predict(test_observed) # ask the classifier (clf) to predict\n",
    "preds # show the predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test the algorithm's performance\n",
    "Finally, we are now able to test the performance of the algorithm. There are a few different ways that this can be done. One common way to do this is to measure the algorithm's accuracy (the rate of correct negatives and correct positives vs all observations). \n",
    "\n",
    "The code uses Scikit-Learn's `accuracy_score` method to calculate this for us. It takes our predictions and true labels as inputs and then calculates the result. Though I don't know for sure (because the data is randomize) I imagine that your algorithm has an accuracy of somewhere between 70% and 85%. Not too shabby! Congratulations, you have just trained your first machine learning algorithm! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score # import the accuracy score calculator\n",
    "\n",
    "test_labels = test['target'] # the test data target labels\n",
    "accuracy_score(preds, test_labels) # compare the accuracy of the predicted labels and the actual labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## _Challenge Question 3 (1 point)_\n",
    "In the steps mentioned above, we divided the data into train and test datasets; the train data was 1/2 of the overall data and the test data was 1/2 of the data. What happens if we instead take around 80% of the data (the first 240 instances) as training data and 20% of the data (the remaining 63) as test? \n",
    "\n",
    "Implement code in the cell below that replicates the results above (Random Forest classifier), but with modified train/test data. Was it more accurate? It might not be the case; it's always good to double-check though!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# insert your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Objective 3: Apply and compare multiple classifiers\n",
    "Though we have one working classifier, it's also important to test other classifiers. Though the decision tree works well in this context, other classifiers might also work well or better. There is no perfect classifer for all context; some classifiers will be a better fit for different data in different contexts. We should try a few techniques and compare their results before calling it a day. Before we get started, lets start by returning to the original data that we had (50-50 split) so that we can assess the algorithms consistently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = hdr[:150] # train is the first 150\n",
    "test = hdr[150:] # test is the last 150\n",
    "\n",
    "training_observed = train.drop('target', 1) # prepare training data by removing labels\n",
    "training_labels = train['target']\n",
    "\n",
    "test_observed = test.drop('target', 1) # prepare test data by removing labels\n",
    "test_labels = test['target']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Support Vector Machines (SVM)\n",
    "One popular technique is called _support vector machines_. Unlike decision trees, it's not that straight forward how these work \"under the hood\". This classifier reduces the data to a simpler form, fits a simple algorithm to it, and then expands that algorithm to the multidimensional data that we have. Don't worry too much about the details; if you are interested, you can learn more in [scikit-learn's SVM documentation](https://scikit-learn.org/stable/modules/svm.html). \n",
    "\n",
    "This doesn't stop us from trying it though. The code below implements the SVM classifier. Try running it and see what happens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import svm # import SVM\n",
    "clf = svm.SVC(gamma='scale') # save clf as the new SVM classifier\n",
    "\n",
    "clf.fit(training_observed, training_labels) # fit the SVM classifier\n",
    "preds = clf.predict(test_observed) # predict the results\n",
    "accuracy_score(preds, test_labels) # display the accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chances are that your accuracy will not be very good, around 60-65%. This is because we did not select a good kernel for this task. By default, SVM uses a kernel called `radial basis function` which is good for some types of normalized data. When using SVM, it's usually a good idea to also try the `linear` kernel, which is implemented in the code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import svm # as before\n",
    "clf = svm.SVC(gamma='scale', kernel=\"linear\") # note that we have the option to change the kernel to 'linear' here\n",
    "\n",
    "clf.fit(training_observed, training_labels) # as before\n",
    "preds = clf.predict(test_observed)\n",
    "accuracy_score(preds, test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Naive Bayes\n",
    "A third technique we can try is called Naive Bayes. This is a probabilistic classifier based on [Bayes Theorem](https://en.wikipedia.org/wiki/Bayes%27_theorem). It is primarily used in text analysis, but can be used in our context as well. Let's train this classifier using the same code as before. You will probably end up with good results using Naive Bayes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import GaussianNB # import GaussianNB\n",
    "clf = GaussianNB() # specify the NB classifier\n",
    "\n",
    "clf.fit(training_observed, training_labels)\n",
    "preds = clf.predict(test_observed)\n",
    "accuracy_score(preds, test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, it is worth mentioning that we can also dig more deeply into an algorithm's performance. For example, we could observe the _Confusion Matrix_ which is a way of visualizing exactly how the results were classified. In a [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix) each row represents an instance of the predicted class, while each column represents the actual class. By visualizing the results this way, we can get a sense of exactly how the classification performed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "confusion_matrix(preds, test_labels) #shows the confusion matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how to interpret the confusion matrix:\n",
    "- The values in the top left of the matrix were classified as \"no heart disease\" and indeed had no heart disease\n",
    "- The values in the top right of the matrix were classified as \"no heart disease\" but had heart disease\n",
    "- The values in the bottom right of the matrix were classified as \"has heart disease\" and indeed had heart disease\n",
    "- The values in the bottom left of the matrix were classified as \"no heart disease\" but had heart disease\n",
    "\n",
    "This confusion matrix suggests that our Naive Bayes algorithm had a bit of trouble classifying people as \"no heart disease\" when they actually did. For more information about how to interpret these results from the perspective of statistics, read through the Wikipedia page on the concepts of `accuracy, precision and recall` as discssed above, or in Grimson's video. We will not expand on these any further here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### _Challenge Question 4 (2 points)_\n",
    "This brings us to our final challenge question. In addition to Random Forest, Support Vector Machines or Naive Bayes, there are many other algorithms which can be tried. For example, you could try [linear discriminant analysis (LDA)](https://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html#sklearn.discriminant_analysis.LinearDiscriminantAnalysis).\n",
    "\n",
    "Read through the documentation on Scikit-Learn's LDA classifier, which can be [found here](https://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html#sklearn.discriminant_analysis.LinearDiscriminantAnalysis). Based on what you have read, implement the LDA classifier with this data. Write a comment somewhere below that states whether this is higher or lower than your previously observed classification tasks. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# insert your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "Aha, David. Heart disease data set. https://archive.ics.uci.edu/ml/datasets/Heart+Disease\n",
    "\n",
    "Detrano, R., Janosi, A., Steinbrunn, W., Pfisterer, M., Schmid, J., Sandhu, S., Guppy, K., Lee, S., & Froelicher, V. (1989). International application of a new probability algorithm for the diagnosis of coronary artery disease. American Journal of Cardiology, 64,304--310.\n",
    "\n",
    "David W. Aha & Dennis Kibler. \"Instance-based prediction of heart-disease presence with the Cleveland database.\"\n",
    "\n",
    "Gennari, J.H., Langley, P, & Fisher, D. (1989). Models of incremental concept formation. Artificial Intelligence, 40, 11--61.\n",
    "\n",
    "Grimson, E. (2017). Introduction to machine learning. MIT & YouTube. https://www.youtube.com/watch?v=h0e2HAPTGF4 \n",
    "\n",
    "Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., ... & Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. the Journal of machine Learning research, 12, 2825-2830."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}