Assignment_9.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "429ff5ed-76f4-40ee-a5dd-7d8771a3392c",
   "metadata": {},
   "source": [
    "# Assignment 9: Multivariate and Machine Learning Analysis for Intracranial EEG Data\n",
    "Please submit this assignment to Canvas as a jupyter notebook (.ipynb).  The assignment will have you utilize machine learning techniques to classify memory states."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "df1e9298-f81c-4f78-b88c-3b63897c2a2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import cmlreaders as cml\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import roc_auc_score, roc_curve\n",
    "from sklearn.model_selection import KFold\n",
    "from scipy.stats import ttest_ind"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c5f7c47-bf8b-4a6d-9b38-6fc08058edb0",
   "metadata": {},
   "source": [
    "## Question 1, Optimizing hyperparameters using nested cross-validation\n",
    "\n",
    "In the previous assignment you used a fixed penalty parameter (C = 1) for all subjects. Oftentimes we want to tune the hyperparameters of a model to optimize its performance. It is crucial to make it clear that the aim of cross-validation is not to get one or multiple trained models for inference, but to estimate an unbiased generalization performance. We can use a grid search approach in which we search over a grid of 10 values of C logarithmically spaced between $10^{−6}$ and $10^2$ (np.logspace(-6,2,10)). One naive approach is to repeat Part I for every C value and select the optimal C that maximizes the average AUC across folds. The problem is, if we use the test set multiple times for different trained models, during our selection of the optimal model, the test set actually “leaks” information, and is thus impure or biased. To rigorously select the optimal parameter C and correctly estimate the prediction error of the optimal model, we utilize a nested cross-validation procedure. As the name suggests, you will perform two rounds of cross-validation with the inner CV nested in the outer CV. The outer CV is responsible for obtaining the prediction error for the model and the inner CV is responsible for selecting the optimal hyperparameter for each outer CV fold. Apply the nested cross-validation procedure from the notes to the dataset.\n",
    "\n",
    "Does optimizing the regularization hyperparameter help? Use barplots, scatterplots, and appropriate statistical tests to support your conclusions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3c3ccfae-31c0-4619-a1ef-d1386f8ed7cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Question 1\n",
    "### YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eac2c9bb-30de-4254-b42a-b66bd19174a0",
   "metadata": {},
   "source": [
    "## Question 2, Comparison between different penalization schemes and z-scoring features\n",
    "\n",
    "1. In this section you will investigate how different penalization schemes and how z-scoring features can produce different behaviors in the classifier. Recall that the objective function for penalized logistic regression is: \n",
    "\n",
    "$l(\\beta) = \\Sigma_{i=1}^N{y_i log p_i + (1 − y_i) log(1 − p_i)} + \\frac{\\alpha}{2} r||\\beta||_2^2 + \\alpha(1 − r)||\\beta||_1$\n",
    "\n",
    "where, $\\alpha = 1/C$ is the penalty parameter, r is the contribution of L2 penalty, and 1 − r is the contribution of L1 penalty. When r = 1, we have a strictly L2 penalized logistic regression. When r = 0, we have a strictly L1 penalized regression (a.k.a. Lasso). When 0 < r < 1, we have a mixture of both L1 and L2, which is called elastic net. In this part, you will compare the performances of different penalization schemes: strictly L2, strictly L1, and elastic net with r = 0.5. Repeat the nested cross-validation procedure in Part 2\n",
    "using z-scored features for L2, L1, and elastic net. You should again use sklearn’s linear_model.LogisticRegression class for your classifier, appropriately selecting the classifier hyperparameters to obtain the L1 and elastic net regularization schemes.\n",
    "* Compare the performances (AUCs) across these three schemes using a barplot or whatever you see fit, including some visualization of variability in the outcomes for these methods. Does one scheme do better than the others?\n",
    "* Keep in mind that the sklearn LogisticRegression parameter `C` is an inverse regularization strength, meaning the regularization strength is equal to $\\alpha$ = 1/C. So higher `C` means lower regularization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a24966f4-4a4c-426e-a7d3-3b0ce38f0a4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Question 2\n",
    "### YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b943511d-6121-4a87-b5fa-8cbeddd28abe",
   "metadata": {},
   "source": [
    "## Question 3\n",
    "\n",
    "Use the *model.coef_* attribute (*model* is your classifier object) to investigate the learned coefficients of the classifier for each subject. You can pool classifier weights across all outer cross-validation folds. Generate a plot for each of the first three subjects containing three histograms (one for each penalization scheme) of the model coefficients (use the *alpha* parameter of the plt.hist function to ensure the histograms do not cover each other up). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "05e1c70e-f927-4afc-adcd-0cd4aec79067",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Question 3\n",
    "### YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7774bf1b-6cac-4a81-848b-dfbd5059fc40",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Question 4, Does z-scoring improve performance?\n",
    "* In this part, repeat the analysis from Part 2, Question 1 but without z-scored features. Which is better, raw features or z-scored features? Give an intuitive explanation as to why one is better than the other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "68f48a0a-82b9-4717-8d1a-ff8bd2f2c9c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Question 4\n",
    "### YOUR CODE HERE"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "workshop_311",
   "language": "python",
   "name": "workshop_311"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}