model_selection_and_regularization.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:a3fae68f42544a102e0bf10b43c6f7737d192246ee2772273bf3f2e9562ff8a1"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#<span style=\"background-color:#66FF99)\">Model Selection and Regularization</span>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "* Other than least squares (OLS) fitting, we can use other fitting methods\n",
      "    1. Subset selection - identify a subset of predictors which are important and use OLS\n",
      "    * Shrinkage (aka regularization) - fit all models but estimated coefficients are shrunk towards zero to reduce variance\n",
      "    * Dimension reduction - projecting p predictors into M-dimensional subspace, by computing M different linear combinations/projections of the variables. These M projects are used as predictors to fit by OLS"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###<span style=\"background-color:#FF99FF)\">Subset Selection</span>\n",
      "1. Best subset model selection\n",
      "    * Fit a OLS for each possible combination of the predictors ($2^p$ possibilities)!\n",
      "    * Very computational expensive\n",
      "    * Pick the best combination for each number of predictors by looking at smallest RSS and largest $R^2$\n",
      "    * Select single best model by using Cross validated prediction error, AIC, BIC or adjusted R-squared\n",
      "* Stepwise model selection\n",
      "    * Forward: Consider all models which improves the accuracy by adding 1 additional predictor\n",
      "    * Backward: Consider all models that contain all but 1 predictors\n",
      "    * Hybrid: Combination of both above\n",
      "    * To select the best model, RSS and R-squared might not be always suitable, therefore use $C_p$, AIC, BIC and adjusted R-squared\n",
      "    * The best way is still using Cross-validation!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#<span style=\"background-color:#FF99FF)\">Shrinkage Methods</span>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "* Technique that regularizes the coefficient estimates towards zero, so reducing number of coefficients\n",
      "* Best known: Ridge regression and Lasso\n",
      "\n",
      "##1. OLS estimates coefficients using values that minimize:\n",
      "\n",
      "##$RSS = \\sum_{i=1}^n(y_i - \\beta_0 - \\sum_{j=1}^p \\beta_j x_{ij})^2$\n",
      "\n",
      "##2. Ridge Regression estimates coefficients using values that minimize:\n",
      "##$RSS + \\lambda\\sum_{j=1}^p \\beta_j^2$\n",
      "##$\\sum_{i=1}^n(y_i - \\beta_0 - \\sum_{j=1}^p \\beta_j x_{ij})^2 + \\lambda\\sum_{j=1}^p \\beta_j^2$\n",
      "##$\\lambda \\geq 0$ : *tuning parameter* \n",
      "##$\\lambda\\sum_{j=1}^p \\beta_j^2$ :  *Shrinkage penalty* is small when coefficients are close to zero, therefore has effects of shrinking estimates to zero\n",
      "\n",
      "* When $\\lambda$ is zero, the penalty term has no effect, therefore will return same as OLS\n",
      "* When $\\lambda$ is infinity large, impact of penalty is huge, the coefficient will approach Zero (shrink!)\n",
      "\n",
      "##3. Lasso estimates coefficients using values that minimize:\n",
      "##$RSS + \\lambda\\sum_{j=1}^p \\mid\\beta_j\\mid$\n",
      "##$\\sum_{i=1}^n(y_i - \\beta_0 - \\sum_{j=1}^p \\beta_j x_{ij})^2 + \\lambda\\sum_{j=1}^p \\mid\\beta_j\\mid$\n",
      "\n",
      "* Ridge will shrink all close to zero but not Zero. Lasso fixes that and set them to zero\n",
      "* Lasso uses $l_1$ penalty: effect of forcing some coefficients to be zero when tuning parameter ($\\lambda$) is large \n",
      "* Ridge uses $l_2$ penalty\n",
      "* Lasso makes the model more interpretable (lesser coefficients)\n",
      "* How to select tuning parameter $\\lambda$? Use Cross-validation!\n",
      "* Lasso or Ridge leads to better results? Lasso perform better when a small number of coefficients have substantial coefficients. Always use CV to compare.\n",
      "\n",
      "##Bayesian Interpretation for Ridge and Lasso\n",
      "* Assumes (1) coefficients have some prior distribution and (2) a likelihood of the data\n",
      "* Posterior distribution: Prior distribution X Likelihood\n",
      "* If g (density function) is Gaussian...\n",
      "* If g is double-exponential (Laplace)...\n",
      "\n",
      "##Selecting the Tuning Parameter\n",
      "* Using CV! Choose a grid of $\\lambda$ and compute the CV error for each"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#<span style=\"background-color:#FF99FF)\">Dimension Reduction Methods</span>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "* All reduction methods:\n",
      "    1. Transform the predictors to $Z_1, Z_2, Z_3...Z_M$\n",
      "    2. Model is fit using these M predictors\n",
      "* The way to do it is PCA and Partial Least Squares\n",
      "\n",
      "#Principal Components Analysis (PCA)\n",
      "* Popular dimension reduction technique\n",
      "* First Principal Component: direction of the data where observations vary the most\n",
      "* Key idea: small number of principal components are enough to explain most of the variability and the relationship with the response\n",
      "* In 2-D data, at most 2 principal components\n",
      "* PCR: construct first M principal components, and use them as predictors in a linear regression that is fit using OLS\n",
      "* PCR is NOT a feature selection method\n",
      "* As more components are used, bias decreases but variance increases\n",
      "* No. of principal components, chosen by cross-validation\n",
      "* When performing PCR, standardizing each predictor is recommended to ensure all variables on same scale\n",
      "\n",
      "#Partial Least Squares (PLS)\n",
      "* PCR is unsupervised in identifiying the principal component driections\n",
      "* Drawback: no guarantee the directions best explain the predictors will be best for predicting the response\n",
      "* Partial Least Squares - supervised alternative\n",
      "* PLS also identify new set of features which are linear combintions, then fit a linear model using least squares\n",
      "* It is supervised : uses response Y to identify new features\n",
      "* Does not perform any better than Ridge or PCR (not much overall benefit of PLS)\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#<span style=\"background-color:#FF99FF)\">Considerations in High Dimensions</span>\n",
      "1. High Dimensional Data\n",
      "    * Regression and Classification techniques mostly for low-dimensional setting (n >> p)\n",
      "    * Recent technology causes p >> n (high-dimensional)\n",
      "    * OLS will not be appropriate in high-dimensional setting\n",
      "* What goes wrong in High Dimensions?\n",
      "    * Why OLS shouuld not be performed in high dimensional setting (p > n)? OLS is too flexible and overfits the data\n",
      "    * It might perfectly fit the high-dimensional setting but perform poorly on test set\n",
      "    * Other alternative approaches are better-suited\n",
      "* Regression in High Dimensions\n",
      "    * 3 important points: Regularization (shrinkage) plays a key role in HD problems, appropriate tuning parameter is crucial, test error increase as dimensions increase\n",
      "    * Curse of dimensionality: adding additional features might not reduce the training test error\n",
      "* Interpreting Results in High Dimensions\n",
      "    * One should never use sum of squared errors, p-values, R-squared as evidence of good model fit\n",
      "    * Report results on independent test test, or cross validation errors!"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}