notes_cross_validation_bootstrap.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:c6b34f2db7ba5a818e7c5644379f2364d04c0096fa7fdb159df2723a43f4b017"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#<span style=\"background-color:#66FF99\">Cross Validation<span/>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "* Resampling is very important! Draw samples from a training set.\n",
      "* Most common: CV and Bootstrap\n",
      "* Model assessment: To evaluate a model's performance\n",
      "* Model selection: Selecting the model flexibility\n",
      "* Bootstrap used to provide a measure of accuracy of a parameter estimate or statistical learning method\n",
      "\n",
      "###Validation Set Approach\n",
      "* Divide the available set into 2 parts (training and validation)\n",
      "* Fit on training set and predict on validation set\n",
      "* Use MSE of training set and validation set to measure the accuracy and choose which polynomial\n",
      "\n",
      "###Leave-One-Out CV\n",
      "* Instead of splitting into 2 parts, LOOCV use 1 single observation as validation set and the rest for training\n",
      "* Test error estimated by averagingthe MSE\n",
      "* Adv: less bias, since we run through the whole set\n",
      "* Disadv: Expensive to implement and very time consuming\n",
      "\n",
      "###k-Fold CV\n",
      "* Randomly dividing the observationsinto k-groups (folds) of equal size\n",
      "* 1st fold use as validation set and fit on the rest folds\n",
      "* Adv: Cheaper to implement\n",
      "\n",
      "###Bias-Variance Trade-off for k-Fold CV\n",
      "* Gives more accurate estimates of the test error rate! Due to bias-variance trade off.\n",
      "* Why? LOOCV has higher variance since it's using every data point (overfitting?)\n",
      "* Typically use k=5 or k=10, so the test error rate does not suffer from high bias or high variance (just nice!)\n",
      "\n",
      "###CV on classification problems\n",
      "* Instead of using MSE for test error, we use misclassified observations\n",
      "* How to decide which degree of polynomial of Logistic Regression? CV!\n",
      "* Plot order of polynomials against error rate (Test error, Training error, and 10-fold CV error)\n",
      "* But which K to use? Plot 1/K against Error rate!\n",
      "* Training error rate cannot be used to select optimal value for K"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#<span style=\"background-color:#66FF99\">Bootstrap<span/>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "* Used to estimate the standard errors of coefficients\n",
      "* Bootstrap approach can emulate process of obtaining new sample sets\n",
      "* Sampling method is performed using *replacement* - same observation can occur more than once"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}