MachineLearning/LinearRegression/linear_regressions_and_simple_relationships.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "heading_collapsed": true,
    "id": "qg86awMSCxbq"
   },
   "source": [
    "# Finding Relationships"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "heading_collapsed": true,
    "hidden": true,
    "id": "rp_aSi1pC4jq"
   },
   "source": [
    "## Covariance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "heading_collapsed": true,
    "hidden": true,
    "id": "hELUGAEwJw7U"
   },
   "source": [
    "### Motivation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "4ghio9TDJyw8"
   },
   "source": [
    "**Variance**: Measure of dispersion from the mean\n",
    "$$\\sigma^2 = \\sum_{i}^{n}\\frac{(x_i -\\mu )^2}{n}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "l6SpCVyJJ2H9"
   },
   "source": [
    "**Covariance**: Measure of how variables differ/relate to one another\n",
    "$$\\sigma_{xy} = \\frac{\\sum_{i,j}^{n} (x_i -\\mu_x )(y_j - \\mu_y)}{n}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "heading_collapsed": true,
    "hidden": true,
    "id": "ho4S7xV5LJuW"
   },
   "source": [
    "### Meaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "FRrqW6cQLLvG"
   },
   "source": [
    "* Positive covariance  --> correlates (together)\n",
    "* Negative covariance --> correlates inversely\n",
    "\n",
    "But ranges (-∞,∞) so what is a \"stronger\" relationship?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "heading_collapsed": true,
    "hidden": true,
    "id": "iGP6OFiqLl3O"
   },
   "source": [
    "## Corelation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "heading_collapsed": true,
    "hidden": true,
    "id": "jitdGexSLri4"
   },
   "source": [
    "### Motivation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "KVIbXYr8L4wi"
   },
   "source": [
    "Pearson's Correlation:\n",
    "\n",
    "$$ r = \\frac{\\sum_{i,j}^{n}(x_i -\\mu_x)(y_j - \\mu_y)} {\\sqrt{\\sum_{i,j}^{n}(x_i - \\mu_x)^2 (y_j-\\mu_y)^2}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "PepMswvMMvCq"
   },
   "source": [
    "* Allows independent of units\n",
    "* Normalized [-1,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "heading_collapsed": true,
    "hidden": true,
    "id": "gSMBvku4L2ya"
   },
   "source": [
    "### Interpretting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "57TmehmaNPYQ"
   },
   "source": [
    "<img src='https://raw.githubusercontent.com/learn-co-students/dsc-0-10-03-cov-corr-online-ds-sp-000/master/images/correlation.png' width=70%/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "heading_collapsed": true,
    "hidden": true,
    "id": "kf4e1fKjNpUu"
   },
   "source": [
    "## Intro to Statistical Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "heading_collapsed": true,
    "hidden": true,
    "id": "8BsOTDQjNuWy"
   },
   "source": [
    "### Intro to the basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "aKgHubfNN-yI"
   },
   "source": [
    "Variables in 3 flavors:\n",
    "* Independent\n",
    "* Dependent\n",
    "* Controlled \n",
    "\n",
    "**Statistical Model** == **Function** (mapping) that defines a relationship"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "FBQn9ZVLOGwh"
   },
   "source": [
    "<img src='https://blogs.sas.com/content/iml/files/2013/02/RegSlopeInt.png' width=400 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "heading_collapsed": true,
    "hidden": true,
    "id": "LmvqUfEpOv4A"
   },
   "source": [
    "### Some Processes & Terms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "5pKlWBV0O0WS"
   },
   "source": [
    "- Use **parameters** to build a model\n",
    "- **Features** vs **Target**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "heading_collapsed": true,
    "hidden": true,
    "id": "G4TGOXAQPKuq"
   },
   "source": [
    "#### Model Validation\n",
    "\n",
    "The \"learning\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "PzYONaIVPR2f"
   },
   "source": [
    "**Validation**: Split data into training and testing\n",
    "\n",
    "***Q: Why do this?***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "TamNyo3FQ2lA"
   },
   "source": [
    "**Loss Function**: Evaluating our model w/ the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "heading_collapsed": true,
    "hidden": true,
    "id": "WDRoua1JCo5l"
   },
   "source": [
    "## Linear Regression\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "oAXv6GUBRt6y"
   },
   "source": [
    "<img src='https://cdn-images-1.medium.com/max/1600/1*iuqVEjdtEMY8oIu3cGwC1g.png' width=50%/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "heading_collapsed": true,
    "hidden": true,
    "id": "3wG-RNilIcCL"
   },
   "source": [
    "### Exploring the Motivation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "umDvUMnnLqkg"
   },
   "source": [
    "* A way to model the data\n",
    "* Data scientist go-to"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "heading_collapsed": true,
    "hidden": true,
    "id": "On1MjirgR8Ay"
   },
   "source": [
    "### Creating Regression Line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "J-cRBlanUWHa"
   },
   "source": [
    "Line of best fit\n",
    "\n",
    "$$ Y = a X + b $$\n",
    "\n",
    "Slope:\n",
    "\n",
    "$$ a = r \\frac{s_y}{s_x} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "8sRbK9jLUMNj"
   },
   "source": [
    "<img src='https://rasbt.github.io/mlxtend/user_guide/regressor/LinearRegression_files/simple_regression.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "Tg-r-sZxVwqf"
   },
   "source": [
    "Can get the intercept with the means of X & Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "heading_collapsed": true,
    "hidden": true,
    "id": "gUIxgjbET5lS"
   },
   "source": [
    "### But is it any good?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "wIk1Q3SeVEDd"
   },
   "source": [
    "**Can think of it the loss function to be the sum of sqaures**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "PqBUVMkkT-gY"
   },
   "source": [
    "**Can think in terms of predictions vs worst model**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "wIk1Q3SeVEDd"
   },
   "source": [
    "<img src='https://github.com/learn-co-students/dsc-0-10-08-coefficient-of-determination-online-ds-ft-031119/raw/master/images/linreg_rsq.png'  width = 60%/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "Au6_duOVWoUB"
   },
   "source": [
    "**Wait, what do you mean worst model...?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "PqBUVMkkT-gY"
   },
   "source": [
    "<img src='https://github.com/learn-co-students/dsc-0-10-08-coefficient-of-determination-online-ds-ft-031119/raw/master/images/rs5.png'  width=60%/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "heading_collapsed": true,
    "hidden": true,
    "id": "50Nc9LgrDCXh"
   },
   "source": [
    "### When Does Linear Regression Work?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "Au6_duOVWoUB"
   },
   "source": [
    "<img src='https://github.com/learn-co-curriculum/dsc-01-10-13-ols-statsmodels-lab/raw/solution/index_files/index_24_1.png'  width=60%/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "heading_collapsed": true,
    "hidden": true,
    "id": "Y7CzTP6dDveR"
   },
   "source": [
    "#### Is normal (normality)\n",
    "\n",
    "<img src='https://raw.githubusercontent.com/learn-co-students/dsc-01-10-11-regression-assumptions-online-ds-ft-031119/master/images/inhouse_histo.png' width=70%/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "zcKUka0QEI7O"
   },
   "source": [
    "<!--\n",
    "**Actually not required for regression**\n",
    "\n",
    "* Allows for stat testin (z-test)\n",
    "* Confidence Interval\n",
    "-->\n",
    "\n",
    "For residuals, except for large samples (central limit theorem)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "Y7CzTP6dDveR"
   },
   "source": [
    "***Q: Why would this matter?***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "1_FY8IKbEoi1"
   },
   "source": [
    "Observing Normality with **Histograms** & **Q-Q Plots**\n",
    "\n",
    "***Remember this is a visual test***\n",
    "\n",
    "<img src='https://github.com/learn-co-students/dsc-01-10-11-regression-assumptions-online-ds-ft-031119/raw/master/images/inhouse_qq_plots.png' width=700/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "DPjXYjtuFtOx"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "heading_collapsed": true,
    "hidden": true,
    "id": "O7r_RnBSGMA6"
   },
   "source": [
    "#### Heteroscedasticity\n",
    "\n",
    "Basically if the residuals are evenly spread through range\n",
    "\n",
    "<img src='https://github.com/learn-co-students/dsc-01-10-11-regression-assumptions-online-ds-ft-031119/raw/master/images/homo_2.png' width=700/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "hidden": true,
    "id": "XCC2CzM6GZpJ"
   },
   "source": [
    "You are looking to make sure the residuals are spread evenly. [In depth explantion](https://statisticsbyjim.com/regression/heteroscedasticity-regression/)\n",
    "\n",
    "<img src='https://github.com/learn-co-students/dsc-01-10-11-regression-assumptions-online-ds-ft-031119/raw/master/images/prob_2.png' width=80%/>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}