linear-regression-derived.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression Derived"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Cost function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$ C = \\sum_i (y_i - mx_i - c)^2 $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate the partial derivative with respect to $m$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\n",
    "\\begin{align}\n",
    "\\frac{\\partial C}{\\partial m} &= \\sum_i 2(y_i - m x_i -c)(-x_i) \\\\\n",
    "&= -2 \\sum_i x_i (y_i - m x_i -c) \\\\\n",
    "\\end{align}\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set derivative to zero"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\n",
    "\\begin{align}\n",
    "& \\frac{\\partial C}{\\partial m} = 0 \\\\\n",
    "\\Rightarrow & -2 \\sum_i x_i (y_i - m x_i -c) = 0 \\\\\n",
    "\\Rightarrow & \\sum_i ( x_i y_i - m x_i x_i - x_i c ) = 0 \\\\\n",
    "\\Rightarrow & \\sum_i x_i y_i - \\sum_i m x_i x_i - \\sum_i x_i c = 0 \\\\\n",
    "\\Rightarrow & \\sum_i x_i y_i - c \\sum_i x_i = m \\sum_i x_i x_i \\\\\n",
    "\\Rightarrow & m = \\frac{\\sum_i x_i y_i - c \\sum_i x_i}{\\sum_i x_i x_i}\n",
    "\\end{align}\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate the partial derivative with respect to $c$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\n",
    "\\begin{align}\n",
    "\\frac{\\partial C}{\\partial c} &= \\sum_i 2(y_i - m x_i -c)(-1) \\\\\n",
    "&= -2 \\sum_i (y_i - m x_i -c) \\\\\n",
    "\\end{align}\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set the derivative to zero"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\n",
    "\\begin{align}\n",
    "& \\frac{\\partial C}{\\partial c} = 0 \\\\\n",
    "\\Rightarrow & -2 \\sum_i (y_i - m x_i - c) = 0 \\\\\n",
    "\\Rightarrow & \\sum_i (y_i - m x_i - c) = 0 \\\\\n",
    "\\Rightarrow & \\sum_i y_i - \\sum_i m x_i - \\sum_i c = 0 \\\\\n",
    "\\Rightarrow & \\sum_i y_i - m \\sum_i x_i - c \\sum_i 1 = 0 \\\\\n",
    "\\Rightarrow & \\sum_i y_i - m \\sum_i x_i = c \\sum_i 1 \\\\\n",
    "\\Rightarrow & c = \\frac{\\sum_i y_i - m \\sum_i x_i}{\\sum_i 1} \\\\\n",
    "\\Rightarrow & c = \\frac{\\sum_i y_i}{\\sum_i 1} - m \\frac{\\sum_i x_i}{\\sum_i 1} \\\\\n",
    "\\Rightarrow & c = \\bar{y} - m \\bar{x} \\\\\n",
    "\\end{align}\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Combine the estimates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\n",
    "\\begin{align}\n",
    "& m = \\frac{\\sum_i x_i y_i - c \\sum_i x_i}{\\sum_i x_i x_i} \\\\\n",
    "& c = \\bar{y} - m \\bar{x} \\\\\n",
    "& \\Rightarrow m = \\frac{\\sum_i x_i y_i - (\\bar{y} - m \\bar{x}) \\sum_i x_i}{\\sum_i x_i x_i} \\\\\n",
    "& \\Rightarrow m = \\frac{\\sum_i x_i y_i - \\bar{y} \\sum_i x_i  + m \\bar{x} \\sum_i x_i}{\\sum_i x_i x_i} \\\\\n",
    "& \\Rightarrow m = \\frac{\\sum_i x_i y_i - \\bar{y} \\sum_i x_i}{\\sum_i x_i x_i}  + m \\frac{\\bar{x} \\sum_i x_i}{\\sum_i x_i x_i} \\\\\n",
    "& \\Rightarrow m - m \\frac{\\bar{x} \\sum_i x_i}{\\sum_i x_i x_i} = \\frac{\\sum_i x_i y_i - \\bar{y} \\sum_i x_i}{\\sum_i x_i x_i} \\\\\n",
    "& \\Rightarrow m(1 - \\frac{\\bar{x} \\sum_i x_i}{\\sum_i x_i x_i}) = \\frac{\\sum_i x_i y_i - \\bar{y} \\sum_i x_i}{\\sum_i x_i x_i} \\\\\n",
    "& \\Rightarrow m(\\frac{\\sum_i x_i x_i - \\bar{x} \\sum_i x_i}{\\sum_i x_i x_i}) = \\frac{\\sum_i x_i y_i - \\bar{y} \\sum_i x_i}{\\sum_i x_i x_i} \\\\\n",
    "& \\Rightarrow m(\\sum_i x_i x_i - \\bar{x} \\sum_i x_i) = \\sum_i x_i y_i - \\bar{y} \\sum_i x_i \\\\\n",
    "& \\Rightarrow m = \\frac{\\sum_i x_i y_i - \\bar{y} \\sum_i x_i}{\\sum_i x_i x_i - \\bar{x} \\sum_i x_i} \\\\\n",
    "\\end{align}\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### End"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}