[Phase 0] General Overview and Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('fivethirtyeight')\n",
    "import warnings\n",
    "warnings.simplefilter('ignore', FutureWarning)\n",
    "from matplotlib import patches\n",
    "from ipywidgets import interact, interactive, fixed #pip installed ipywdigets in Terminal\n",
    "import ipywidgets as widgets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MRI and Alzheimers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hello Team,\n",
    "\n",
    "\n",
    "Let's do this project real quick. It is going to be a fun and promising project since all of us have some background in CS. However, it is worth noting that CS and Data Science are different aspect as well. Therefore, make sure to understand the statistics and methodology of them. Do not feel hesitant in contacting other team members. I am sure that we all will be like 'WTF' at some point.\n",
    "\n",
    "Anyway, the project is broken down in 6 phases:\n",
    "\n",
    "* **Phase 0** : Schedule, Plan and Preliminary Look at Dataset\n",
    "* **Phase 1** : Cleaning the Data, Computing General Trends and Splitting of Test and Training Set 🧽\n",
    "* **Phase 2** : Looking for correlations (coefficient, data analysis) ⌗\n",
    "* **Phase 3** : Visualizations 📊\n",
    "* **Phase 4** : Classification 📌\n",
    "* **Phase 5** : Evaluation and Improvement\n",
    "* **Phase 6** : Conclusions\n",
    "\n",
    "#### Notes: \n",
    "If you are `pip instal`ling something, make sure to make a `#markdown` next to it or to refer to it in a markdown under it to explain what you did. This way we will not have any instances similar to the BigQuery issue.\n",
    "\n",
    "Feel free to work on whatever you want. Make sure to let each other know what we are working and what we are doing. This means by telling each other that we are working on a particular phase (and therefore should not push/commit something at the same time) but also to make sure that we are all on the same page with regards to the execution of certain things. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we get started and jump right into it, here a list of useful resources:\n",
    "\n",
    "- **GitHub:** https://github.com/Kevin-Miao/MRIAlzheimers\n",
    "- **Jupyter Notebook Markdown Cheat Sheet:** https://medium.com/ibm-data-science-experience/markdown-for-jupyter-notebooks-cheatsheet-386c05aeebed\n",
    "- **Kaggle** (For learning Pandas/Python, but also for datasets)\n",
    "- **Pandas Reference Sheet:** https://pandas.pydata.org/Pandas_Cheat_Sheet.pdf\n",
    "- **DS100** Useful textbook: http://www.ds100.org/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variables and Context\n",
    "\n",
    "One of the datasets is a cross-section (so horizontal cut of the brain) and the other data set (longitudinal) is a slice through the middle of the head."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-Sectional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>M/F</th>\n",
       "      <th>Hand</th>\n",
       "      <th>Age</th>\n",
       "      <th>Educ</th>\n",
       "      <th>SES</th>\n",
       "      <th>MMSE</th>\n",
       "      <th>CDR</th>\n",
       "      <th>eTIV</th>\n",
       "      <th>nWBV</th>\n",
       "      <th>ASF</th>\n",
       "      <th>Delay</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>OAS1_0001_MR1</td>\n",
       "      <td>F</td>\n",
       "      <td>R</td>\n",
       "      <td>74</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1344</td>\n",
       "      <td>0.743</td>\n",
       "      <td>1.306</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>OAS1_0002_MR1</td>\n",
       "      <td>F</td>\n",
       "      <td>R</td>\n",
       "      <td>55</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1147</td>\n",
       "      <td>0.810</td>\n",
       "      <td>1.531</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>OAS1_0003_MR1</td>\n",
       "      <td>F</td>\n",
       "      <td>R</td>\n",
       "      <td>73</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1454</td>\n",
       "      <td>0.708</td>\n",
       "      <td>1.207</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>OAS1_0004_MR1</td>\n",
       "      <td>M</td>\n",
       "      <td>R</td>\n",
       "      <td>28</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1588</td>\n",
       "      <td>0.803</td>\n",
       "      <td>1.105</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>OAS1_0005_MR1</td>\n",
       "      <td>M</td>\n",
       "      <td>R</td>\n",
       "      <td>18</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1737</td>\n",
       "      <td>0.848</td>\n",
       "      <td>1.010</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              ID M/F Hand  Age  Educ  SES  MMSE  CDR  eTIV   nWBV    ASF  \\\n",
       "0  OAS1_0001_MR1   F    R   74   2.0  3.0  29.0  0.0  1344  0.743  1.306   \n",
       "1  OAS1_0002_MR1   F    R   55   4.0  1.0  29.0  0.0  1147  0.810  1.531   \n",
       "2  OAS1_0003_MR1   F    R   73   4.0  3.0  27.0  0.5  1454  0.708  1.207   \n",
       "3  OAS1_0004_MR1   M    R   28   NaN  NaN   NaN  NaN  1588  0.803  1.105   \n",
       "4  OAS1_0005_MR1   M    R   18   NaN  NaN   NaN  NaN  1737  0.848  1.010   \n",
       "\n",
       "   Delay  \n",
       "0    NaN  \n",
       "1    NaN  \n",
       "2    NaN  \n",
       "3    NaN  \n",
       "4    NaN  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('oasis_cross-sectional.csv').head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Variables**\n",
    "\n",
    "*Demographics*\n",
    "* **ID**: ID number (1-\n",
    "* **M/F**: Gender (M/F)\n",
    "* **Hand**: Handedness (L/R) (All people were right handed)\n",
    "* **Age**\n",
    "* **Education**: Years of Education\n",
    "* **SES**: Social Economic Status by Hollingshead Index of Social Position (1: Highest Status, 5: Lowest Status)\n",
    "\n",
    "*Clinical Data* (Actually has to do with Alzheimers)\n",
    "* **MMSE:** Mini-Mental State Examination Score (0 = Worst, 30=Best)\n",
    "Cutoffs for this test. 0-9: severe, 10-18: mildly impacted, 19-23: cognitive impairment, 23-30: healthy.\n",
    "* **CDR:** Clinical Dementia Rating (Classified by a doctor) \n",
    "- 0 = Normal\n",
    "- 0.5 = Very Mild Dementia\n",
    "- 1 = Mild Dementia\n",
    "- 2 = Moderate Dementia\n",
    "- 3 = Severe Dementia\n",
    "\n",
    "*MRI Data* (From imaging)\n",
    "* **eTIv:** Estimated Intracranial Volume (in mm^3)\n",
    "* **nWBV:** Normalized Whole-Brain Volume (as a percentage of all matter that is labeled as grey or white by software)\n",
    "* **ASF:** Atlas Scaling Factor (Unitless) \n",
    "https://www.ncbi.nlm.nih.gov/pubmed/15488422"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Longitudinal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Subject ID</th>\n",
       "      <th>MRI ID</th>\n",
       "      <th>Group</th>\n",
       "      <th>Visit</th>\n",
       "      <th>MR Delay</th>\n",
       "      <th>M/F</th>\n",
       "      <th>Hand</th>\n",
       "      <th>Age</th>\n",
       "      <th>EDUC</th>\n",
       "      <th>SES</th>\n",
       "      <th>MMSE</th>\n",
       "      <th>CDR</th>\n",
       "      <th>eTIV</th>\n",
       "      <th>nWBV</th>\n",
       "      <th>ASF</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>OAS2_0001</td>\n",
       "      <td>OAS2_0001_MR1</td>\n",
       "      <td>Nondemented</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>R</td>\n",
       "      <td>87</td>\n",
       "      <td>14</td>\n",
       "      <td>2.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1987</td>\n",
       "      <td>0.696</td>\n",
       "      <td>0.883</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>OAS2_0001</td>\n",
       "      <td>OAS2_0001_MR2</td>\n",
       "      <td>Nondemented</td>\n",
       "      <td>2</td>\n",
       "      <td>457</td>\n",
       "      <td>M</td>\n",
       "      <td>R</td>\n",
       "      <td>88</td>\n",
       "      <td>14</td>\n",
       "      <td>2.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2004</td>\n",
       "      <td>0.681</td>\n",
       "      <td>0.876</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>OAS2_0002</td>\n",
       "      <td>OAS2_0002_MR1</td>\n",
       "      <td>Demented</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>R</td>\n",
       "      <td>75</td>\n",
       "      <td>12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1678</td>\n",
       "      <td>0.736</td>\n",
       "      <td>1.046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>OAS2_0002</td>\n",
       "      <td>OAS2_0002_MR2</td>\n",
       "      <td>Demented</td>\n",
       "      <td>2</td>\n",
       "      <td>560</td>\n",
       "      <td>M</td>\n",
       "      <td>R</td>\n",
       "      <td>76</td>\n",
       "      <td>12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1738</td>\n",
       "      <td>0.713</td>\n",
       "      <td>1.010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>OAS2_0002</td>\n",
       "      <td>OAS2_0002_MR3</td>\n",
       "      <td>Demented</td>\n",
       "      <td>3</td>\n",
       "      <td>1895</td>\n",
       "      <td>M</td>\n",
       "      <td>R</td>\n",
       "      <td>80</td>\n",
       "      <td>12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1698</td>\n",
       "      <td>0.701</td>\n",
       "      <td>1.034</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Subject ID         MRI ID        Group  Visit  MR Delay M/F Hand  Age  EDUC  \\\n",
       "0  OAS2_0001  OAS2_0001_MR1  Nondemented      1         0   M    R   87    14   \n",
       "1  OAS2_0001  OAS2_0001_MR2  Nondemented      2       457   M    R   88    14   \n",
       "2  OAS2_0002  OAS2_0002_MR1     Demented      1         0   M    R   75    12   \n",
       "3  OAS2_0002  OAS2_0002_MR2     Demented      2       560   M    R   76    12   \n",
       "4  OAS2_0002  OAS2_0002_MR3     Demented      3      1895   M    R   80    12   \n",
       "\n",
       "   SES  MMSE  CDR  eTIV   nWBV    ASF  \n",
       "0  2.0  27.0  0.0  1987  0.696  0.883  \n",
       "1  2.0  30.0  0.0  2004  0.681  0.876  \n",
       "2  NaN  23.0  0.5  1678  0.736  1.046  \n",
       "3  NaN  28.0  0.5  1738  0.713  1.010  \n",
       "4  NaN  22.0  0.5  1698  0.701  1.034  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('oasis_longitudinal.csv').head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Group**: Demented and Nondemented\n",
    "* **Visit**: Number of visit\n",
    "* **MR Delay**: Not useful (Just technical data collection tool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}