notes_linear_regression.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:eea0ea54545312d3c1658b6f03af1e6c33ffd82b4e1bb27b65bbe52010b0c339"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "<span style=\"background-color:#66FF99\">Simple Linear Regression<span/>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "* Important to have good understanding of linear regression before studying more complex models\n",
      "* Predict a quantitative response Y based on a single predictor X"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "$Y = \\beta_0 + \\beta_1X$\n",
      "\n",
      "$\\hat Y = \\hat \\beta_0 + \\hat \\beta_1 x_1$\n",
      "\n",
      "* $\\beta_0$ and $\\beta_1$ are know as coefficients or parameters\n",
      "* $\\beta_0$ = intercept, $\\beta_1$ = slope of line\n",
      "* $\\hat y$ is the estimated predicted value\n",
      "\n",
      "###How to estimate the coefficients?\n",
      "* Least Squares method -  minimizing the sum of the squared errors to get the coefficients\n",
      "\n",
      "$e_i = y_i - \\hat y_i$\n",
      "* residual error = real y minus predicted y\n",
      "\n",
      "$RSS = e_1^2 + e_2^2 + ... + e_n^2$\n",
      "* Residual sum of squares (RSS)\n",
      "\n",
      "* Least squares method choose $\\beta_0$ and $\\beta_1$ to minimize RSS\n",
      "* Can use calculus to find the minimizers\n",
      "\n",
      "###How to assess the accuracy of the coefficient estimates?\n",
      "* Summary\n",
      "    1. SE small, accuracy of coefficients high\n",
      "    * Confidence Interval to see 95% chance the interval contain true value\n",
      "    * t-stats big, the stronger the relationship b/w X and Y\n",
      "    * p-value small, the stronger the relationship b/w X and Y\n",
      "* **Standard Error** of the coefficients\n",
      "* Standard Error to compute **confidence intervals**\n",
      "    * 95% chance that the interval will contain true value of parameters\n",
      "* Standard Error to perform hypothesis tests\n",
      "    * $H_0$ = no relationship between X and Y\n",
      "    * $H_1$ = some relationship between X and Y\n",
      "    * $\\hat\\beta_1$ needs to be far from zero to be confident that $\\beta_1$ is non-zero\n",
      "    * If SE($\\hat \\beta_1$) is very small, shows that $\\hat \\beta_1$ is non-zero, therefore strong relationship between X and Y\n",
      "    * **t-statistics** measures the number of SD that $\\hat \\beta_1$ is away from zero \n",
      "    * The larger the absolute value of t, bigger the SD, the more $\\beta_1$ is away from zero, therefore strong relationship between X & Y\n",
      "    * **p-value** - probability of observing large t, assuming $\\beta_1$ is zero. So small p-value, low probability $\\beta_1$ is zero, so strong relationship between X & Y\n",
      "* The true relationship is given by $Y = \\beta_0 + \\beta_1 X + \\epsilon$\n",
      "* $\\epsilon$ captures the other error not captured by model\n",
      "* The true relationship formula gives the progression regression line (best linear approximation)\n",
      "* Population mean ($\\hat U$) can be estimated by sample mean ($\\bar{y}$)\n",
      "\n",
      "###How to assess the accuracy of Model?\n",
      "* **Residual Standard Error (RSE)**\n",
      "    * How much the response deviate from the true regression line\n",
      "    * Absolute measure of the lack of fit of the model\n",
      "    * e.g RSE = 3.26, any prediction of Y is still 3.26 units off on average\n",
      "* **R-squared statistic**\n",
      "    * Proportion measure of the lack of fit\n",
      "    * Takes the value between 0 and 1 (close to 1 = large proportion of variability has been explained)\n",
      "    * Proportion of variability in Y that can be explained using X"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "<span style=\"background-color:#66FF99\">Multiple Linear Regression</span>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "* Often has more than 1 predictor\n",
      "\n",
      "$Y = \\beta_0 + \\beta_1X_1 + \\beta_2X_2 + .... + \\beta_pX_p + \\epsilon$\n",
      "\n",
      "###Estimating the Regression Coefficients\n",
      "* Uses least squares approach : minimize sum of squared residuals\n",
      "\n",
      "$RSS = \\sum_{i=1}^n(y_i - \\hat y_i)^2$\n",
      "\n",
      "* The multiple regression coefficients takes complicated form, best represented using matrix algebra\n",
      "* Use the correlation matrix to see how each feature correlate to each other\n",
      "\n",
      "###Important questions when using multiple regression\n",
      "1. Is at least 1 of the predictors useful in predicting the response?\n",
      "    * Use F-statistics to check whether ALL the coefficients are Zero\n",
      "    * Examine the -value for each predictors\n",
      "    * If F-statistics close to 1: there is no relationship between response and predicots\n",
      "    * If F-statistics far larger than 1, suggests at least 1 of the predictor must be related to the response\n",
      "    * It works best when p < n\n",
      "    * When p > n (high-dimensional setting), we use other approaches\n",
      "* Do ALL predictors help to explain Y or only some?\n",
      "    * To determine which are the important predictors is called variable selection\n",
      "    * Perform variable selection by trying our a lot different models\n",
      "    * How to determine which model is best?\n",
      "        1. Forward selection\n",
      "        * Backward selection\n",
      "        * Mixed selection\n",
      "* How well does the model fit the data?\n",
      "    * Measure of model fit are RSE and $R^2$, and plot the data\n",
      "    * $R^2$ close to 1: explains a large portion of the variance    \n",
      "* Given a set of predictor values, what response value should we predict and how accurate is our prediction? 3 sorts of uncertainty with prediction.\n",
      "    1. The coefficient estimates ($\\hat \\beta_0, \\hat \\beta_1$, etc) is only an ESTIMATE fo the true population regression, therefore there is inaccuracy due to reducible error. We compute a CI to deter how close $\\hat Y$ will be to $f(X)$\n",
      "    * Model bias: Using a wrong model\n",
      "    * Random error ($\\epsilon$): irreducible error as we will never know the true values"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###Considerations of the Regression Model\n",
      "1. **Qualitative Predictors**\n",
      "    * Some predictors are categorical / qualitative\n",
      "    * Create a dummy variable with 2 numerical value\n",
      "        * $x_i$ = 1 (female) and 0 (male)\n",
      "        * $\\beta_0 + \\beta_1 + \\epsilon_i$ (female)\n",
      "        * $\\beta_0 + \\epsilon_i$ (male)\n",
      "        * Interpretation of coeff: \n",
      "            * $\\beta_0$ = male\n",
      "            * $\\beta_0 + \\beta_1$ = female\n",
      "    * Create a dummy variable with 3 numerical value\n",
      "        * $x_i1$ = 1 (asian) and 0 (non-asian)\n",
      "        * $x_i2$ = 1 (caucasian) and 0 (non-caucasian)\n",
      "        * $\\beta_0 + \\beta_1 + \\epsilon_i$ (asian)\n",
      "        * $\\beta_0 + \\beta_2 + \\epsilon_i$ (caucasian)\n",
      "        * $\\beta_0 + \\epsilon_i$ (african)\n",
      "        * Interpretation of coeff: \n",
      "            * $\\beta_0$ = african\n",
      "            * $\\beta_0 + \\beta_1$ = asian\n",
      "            * $\\beta_0 + \\beta_2$ = caucasian\n",
      "* **Extensions of the Linear Model**\n",
      "    * Linear regression has highly restrictive assumptions about the relationship between predictors and response\n",
      "        1. Additive: assumes the predictor X's effect on Y is independent of other predictors\n",
      "        * Linear: assumes the relationship between X and Y is linear (often it can be non-linear)\n",
      "    1. Interactions (removing the additive assumption)\n",
      "        * Solution: include an additional predicot (interaction term) to measure the relationship between 2 predictors\n",
      "        * $Y = \\beta_0 + \\beta_1X_1 +\\beta_2X_2 + \\beta_3X_1X_2 + \\epsilon$\n",
      "        * if the p-value for the interaction term ($X_1X_2$) is LOW, shows the relationship is non-additive. i.e the 2 predictors affects each other\n",
      "    * Non-linear relationships \n",
      "        * Solution: Use polynomial regression\n",
      "        * Compare the $R2$ and p-value to find the best model\n",
      "* **Potential Problems** (when we fit a linear regression model)\n",
      "    1. Non-linearity of response-predictor relationships\n",
      "        * Solution: Residual plots useful tool to identify non-linearity\n",
      "        * Smooth fit: strong indication of non-linearity\n",
      "        * Little pattern (almost horizontal line): the model (e.g quadrative) improves the fit\n",
      "    * Correlation of error terms\n",
      "        * The error terms might have correlation, then estimated standard errors might underestimate the true standard errors\n",
      "    * Non-constant variance of error terms\n",
      "        * Heteroscedasticity: non-contant variances in errors (can be seen by funnel shape in residual plot\n",
      "    * Outliers\n",
      "        * Outlier might have effect on the interpretation of fit (RSE might increase and $R^2$ might decline)\n",
      "        * RSE is used to compute CI and p-values, therefore might affect\n",
      "        * Residual plots can be used to identify outliers\n",
      "        * Solution: Often not easy to see from basic plot, so we can use *studentized residuals* (divide each residual $\\epsilon$ by its estimated standard error\n",
      "    * High-leverage points\n",
      "        * Observations which have unusual high effect on the model\n",
      "        * Very difficult to identify for multiple linear regression\n",
      "        * Solution: Compute the leverage statistics for each observation. Plot leverage against studentized residuals graph.\n",
      "    * Collinearity\n",
      "        * 2 or more predictors are closely related to each other\n",
      "        * Problem: difficult to separate the individual effects of each collinear variables\n",
      "        * To detect collinearity: Look at correlation matrix of predictors\n",
      "        * Correlation matrix only works for a pair of variables\n",
      "        * Multicollinearity: collinearity which exists between 3 or more variables\n",
      "        * Compute the variance inflation factor (VIF)\n",
      "        * VIF = 1 (smallest) - complete absence of collinearity\n",
      "        * VIF > 5 or 10 - problematic amount of collinearity\n",
      "        * 2 solutions: drop one of the problematic variable or combine the collinear variables into 1 single predictor"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###Comparison of Linear Regression with K-Nearest\n",
      "* Linear regression is parametric which has the advantage of easy to fit, need only a small amount of coefficients, simple interpreations, and tests fo statistical significance easy to compute\n",
      "* Disadvantage of parametric: strong assumptions about the form of $f(X)$\n",
      "* Non-parametric: do not assume a parametric form, so more flexible approach\n",
      "* K-nearest neighhors regression (KNN regression), closely related to KNN classifier\n",
      "* Identifies the closest K training observations, takes the average of all training responses\n",
      "\n",
      "$\\hat f(x_0) = \\frac{1}{K}\\sum y_i$\n",
      "\n",
      "* Optimal number of K will depend on bias-variance tradeoff\n",
      "* K = 1 most flexible fit (overfitting)\n",
      "* When will parametric be better than non-parametric? If selected parametric form is very close to the true form. e.g in higher dimensions, KNN often performs worse than linear regression\n",
      "* **Parametric methods outperform non-parametric when there's a small number of observations per predictor"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}