C1_W1_Lab04_Gradient_Descent_Soln.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optional Lab: Gradient Descent for Linear Regression\n",
    "\n",
    "<figure>\n",
    "    <center> <img src=\"./images/C1_W1_L4_S1_Lecture_GD.png\"  style=\"width:800px;height:200px;\" ></center>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goals\n",
    "In this lab, you will:\n",
    "- automate the process of optimizing $w$ and $b$ using gradient descent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tools\n",
    "In this lab, we will make use of: \n",
    "- NumPy, a popular library for scientific computing\n",
    "- Matplotlib, a popular library for plotting data\n",
    "- plotting routines in the lab_utils.py file in the local directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math, copy\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('./deeplearning.mplstyle')\n",
    "from lab_utils_uni import plt_house_x, plt_contour_wgrad, plt_divergence, plt_gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"toc_40291_2\"></a>\n",
    "# Problem Statement\n",
    "\n",
    "Let's use the same two data points as before - a house with 1000 square feet sold for \\\\$300,000 and a house with 2000 square feet sold for \\\\$500,000.\n",
    "\n",
    "| Size (1000 sqft)     | Price (1000s of dollars) |\n",
    "| ----------------| ------------------------ |\n",
    "| 1               | 300                      |\n",
    "| 2               | 500                      |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load our data set\n",
    "x_train = np.array([1.0, 2.0])   #features\n",
    "y_train = np.array([300.0, 500.0])   #target value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"toc_40291_2.0.1\"></a>\n",
    "### Compute_Cost\n",
    "This was developed in the last lab. We'll need it again here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Function to calculate the cost\n",
    "def compute_cost(x, y, w, b):\n",
    "   \n",
    "    m = x.shape[0] \n",
    "    cost = 0\n",
    "    \n",
    "    for i in range(m):\n",
    "        f_wb = w * x[i] + b\n",
    "        cost = cost + (f_wb - y[i])**2\n",
    "    total_cost = 1 / (2 * m) * cost\n",
    "\n",
    "    return total_cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"toc_40291_2.1\"></a>\n",
    "## Gradient descent summary\n",
    "So far in this course, you have developed a linear model that predicts $f_{w,b}(x^{(i)})$:\n",
    "$$f_{w,b}(x^{(i)}) = wx^{(i)} + b \\tag{1}$$\n",
    "In linear regression, you utilize input training data to fit the parameters $w$,$b$ by minimizing a measure of the error between our predictions $f_{w,b}(x^{(i)})$ and the actual data $y^{(i)}$. The measure is called the $cost$, $J(w,b)$. In training you measure the cost over all of our training samples $x^{(i)},y^{(i)}$\n",
    "$$J(w,b) = \\frac{1}{2m} \\sum\\limits_{i = 0}^{m-1} (f_{w,b}(x^{(i)}) - y^{(i)})^2\\tag{2}$$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "In lecture, *gradient descent* was described as:\n",
    "\n",
    "$$\\begin{align*} \\text{repeat}&\\text{ until convergence:} \\; \\lbrace \\newline\n",
    "\\;  w &= w -  \\alpha \\frac{\\partial J(w,b)}{\\partial w} \\tag{3}  \\; \\newline \n",
    " b &= b -  \\alpha \\frac{\\partial J(w,b)}{\\partial b}  \\newline \\rbrace\n",
    "\\end{align*}$$\n",
    "where, parameters $w$, $b$ are updated simultaneously.  \n",
    "The gradient is defined as:\n",
    "$$\n",
    "\\begin{align}\n",
    "\\frac{\\partial J(w,b)}{\\partial w}  &= \\frac{1}{m} \\sum\\limits_{i = 0}^{m-1} (f_{w,b}(x^{(i)}) - y^{(i)})x^{(i)} \\tag{4}\\\\\n",
    "  \\frac{\\partial J(w,b)}{\\partial b}  &= \\frac{1}{m} \\sum\\limits_{i = 0}^{m-1} (f_{w,b}(x^{(i)}) - y^{(i)}) \\tag{5}\\\\\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Here *simultaniously* means that you calculate the partial derivatives for all the parameters before updating any of the parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"toc_40291_2.2\"></a>\n",
    "## Implement Gradient Descent\n",
    "You will implement gradient descent algorithm for one feature. You will need three functions. \n",
    "- `compute_gradient` implementing equation (4) and (5) above\n",
    "- `compute_cost` implementing equation (2) above (code from previous lab)\n",
    "- `gradient_descent`, utilizing compute_gradient and compute_cost\n",
    "\n",
    "Conventions:\n",
    "- The naming of python variables containing partial derivatives follows this pattern,$\\frac{\\partial J(w,b)}{\\partial b}$  will be `dj_db`.\n",
    "- w.r.t is With Respect To, as in partial derivative of $J(wb)$ With Respect To $b$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"toc_40291_2.3\"></a>\n",
    "### compute_gradient\n",
    "<a name='ex-01'></a>\n",
    "`compute_gradient`  implements (4) and (5) above and returns $\\frac{\\partial J(w,b)}{\\partial w}$,$\\frac{\\partial J(w,b)}{\\partial b}$. The embedded comments describe the operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_gradient(x, y, w, b): \n",
    "    \"\"\"\n",
    "    Computes the gradient for linear regression \n",
    "    Args:\n",
    "      x (ndarray (m,)): Data, m examples \n",
    "      y (ndarray (m,)): target values\n",
    "      w,b (scalar)    : model parameters  \n",
    "    Returns\n",
    "      dj_dw (scalar): The gradient of the cost w.r.t. the parameters w\n",
    "      dj_db (scalar): The gradient of the cost w.r.t. the parameter b     \n",
    "     \"\"\"\n",
    "    \n",
    "    # Number of training examples\n",
    "    m = x.shape[0]    \n",
    "    dj_dw = 0\n",
    "    dj_db = 0\n",
    "    \n",
    "    for i in range(m):  \n",
    "        f_wb = w * x[i] + b \n",
    "        dj_dw_i = (f_wb - y[i]) * x[i] \n",
    "        dj_db_i = f_wb - y[i] \n",
    "        dj_db += dj_db_i\n",
    "        dj_dw += dj_dw_i \n",
    "    dj_dw = dj_dw / m \n",
    "    dj_db = dj_db / m \n",
    "        \n",
    "    return dj_dw, dj_db"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"left\" src=\"./images/C1_W1_Lab03_lecture_slopes.PNG\"   style=\"width:340px;\" > The lectures described how gradient descent utilizes the partial derivative of the cost with respect to a parameter at a point to update that parameter.   \n",
    "Let's use our `compute_gradient` function to find and plot some partial derivatives of our cost function relative to one of the parameters, $w_0$.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt_gradients(x_train,y_train, compute_cost, compute_gradient)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, the left plot shows $\\frac{\\partial J(w,b)}{\\partial w}$ or the slope of the cost curve relative to $w$ at three points. On the right side of the plot, the derivative is positive, while on the left it is negative. Due to the 'bowl shape', the derivatives will always lead gradient descent toward the bottom where the gradient is zero.\n",
    " \n",
    "The left plot has fixed $b=100$. Gradient descent will utilize both $\\frac{\\partial J(w,b)}{\\partial w}$ and $\\frac{\\partial J(w,b)}{\\partial b}$ to update parameters. The 'quiver plot' on the right provides a means of viewing the gradient of both parameters. The arrow sizes reflect the magnitude of the gradient at that point. The direction and slope of the arrow reflects the ratio of $\\frac{\\partial J(w,b)}{\\partial w}$ and $\\frac{\\partial J(w,b)}{\\partial b}$ at that point.\n",
    "Note that the gradient points *away* from the minimum. Review equation (3) above. The scaled gradient is *subtracted* from the current value of $w$ or $b$. This moves the parameter in a direction that will reduce cost."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"toc_40291_2.5\"></a>\n",
    "###  Gradient Descent\n",
    "Now that gradients can be computed,  gradient descent, described in equation (3) above can be implemented below in `gradient_descent`. The details of the implementation are described in the comments. Below, you will utilize this function to find optimal values of $w$ and $b$ on the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_descent(x, y, w_in, b_in, alpha, num_iters, cost_function, gradient_function): \n",
    "    \"\"\"\n",
    "    Performs gradient descent to fit w,b. Updates w,b by taking \n",
    "    num_iters gradient steps with learning rate alpha\n",
    "    \n",
    "    Args:\n",
    "      x (ndarray (m,))  : Data, m examples \n",
    "      y (ndarray (m,))  : target values\n",
    "      w_in,b_in (scalar): initial values of model parameters  \n",
    "      alpha (float):     Learning rate\n",
    "      num_iters (int):   number of iterations to run gradient descent\n",
    "      cost_function:     function to call to produce cost\n",
    "      gradient_function: function to call to produce gradient\n",
    "      \n",
    "    Returns:\n",
    "      w (scalar): Updated value of parameter after running gradient descent\n",
    "      b (scalar): Updated value of parameter after running gradient descent\n",
    "      J_history (List): History of cost values\n",
    "      p_history (list): History of parameters [w,b] \n",
    "      \"\"\"\n",
    "    \n",
    "    # An array to store cost J and w's at each iteration primarily for graphing later\n",
    "    J_history = []\n",
    "    p_history = []\n",
    "    b = b_in\n",
    "    w = w_in\n",
    "    \n",
    "    for i in range(num_iters):\n",
    "        # Calculate the gradient and update the parameters using gradient_function\n",
    "        dj_dw, dj_db = gradient_function(x, y, w , b)     \n",
    "\n",
    "        # Update Parameters using equation (3) above\n",
    "        b = b - alpha * dj_db                            \n",
    "        w = w - alpha * dj_dw                            \n",
    "\n",
    "        # Save cost J at each iteration\n",
    "        if i<100000:      # prevent resource exhaustion \n",
    "            J_history.append( cost_function(x, y, w , b))\n",
    "            p_history.append([w,b])\n",
    "        # Print cost every at intervals 10 times or as many iterations if < 10\n",
    "        if i% math.ceil(num_iters/10) == 0:\n",
    "            print(f\"Iteration {i:4}: Cost {J_history[-1]:0.2e} \",\n",
    "                  f\"dj_dw: {dj_dw: 0.3e}, dj_db: {dj_db: 0.3e}  \",\n",
    "                  f\"w: {w: 0.3e}, b:{b: 0.5e}\")\n",
    " \n",
    "    return w, b, J_history, p_history #return w and J,w history for graphing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize parameters\n",
    "w_init = 0\n",
    "b_init = 0\n",
    "# some gradient descent settings\n",
    "iterations = 10000\n",
    "tmp_alpha = 1.0e-2\n",
    "# run gradient descent\n",
    "w_final, b_final, J_hist, p_hist = gradient_descent(x_train ,y_train, w_init, b_init, tmp_alpha, \n",
    "                                                    iterations, compute_cost, compute_gradient)\n",
    "print(f\"(w,b) found by gradient descent: ({w_final:8.4f},{b_final:8.4f})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"left\" src=\"./images/C1_W1_Lab03_lecture_learningrate.PNG\"  style=\"width:340px; padding: 15px; \" > \n",
    "Take a moment and note some characteristics of the gradient descent process printed above.  \n",
    "\n",
    "- The cost starts large and rapidly declines as described in the slide from the lecture.\n",
    "- The partial derivatives, `dj_dw`, and `dj_db` also get smaller, rapidly at first and then more slowly. As shown in the diagram from the lecture, as the process nears the 'bottom of the bowl' progress is slower due to the smaller value of the derivative at that point.\n",
    "- progress slows though the learning rate, alpha, remains fixed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cost versus iterations of gradient descent \n",
    "A plot of cost versus iterations is a useful measure of progress in gradient descent. Cost should always decrease in successful runs. The change in cost is so rapid initially, it is useful to plot the initial decent on a different scale than the final descent. In the plots below, note the scale of cost on the axes and the iteration step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot cost versus iteration  \n",
    "fig, (ax1, ax2) = plt.subplots(1, 2, constrained_layout=True, figsize=(12,4))\n",
    "ax1.plot(J_hist[:100])\n",
    "ax2.plot(1000 + np.arange(len(J_hist[1000:])), J_hist[1000:])\n",
    "ax1.set_title(\"Cost vs. iteration(start)\");  ax2.set_title(\"Cost vs. iteration (end)\")\n",
    "ax1.set_ylabel('Cost')            ;  ax2.set_ylabel('Cost') \n",
    "ax1.set_xlabel('iteration step')  ;  ax2.set_xlabel('iteration step') \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predictions\n",
    "Now that you have discovered the optimal values for the parameters $w$ and $b$, you can now use the model to predict housing values based on our learned parameters. As expected, the predicted values are nearly the same as the training values for the same housing. Further, the value not in the prediction is in line with the expected value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"1000 sqft house prediction {w_final*1.0 + b_final:0.1f} Thousand dollars\")\n",
    "print(f\"1200 sqft house prediction {w_final*1.2 + b_final:0.1f} Thousand dollars\")\n",
    "print(f\"2000 sqft house prediction {w_final*2.0 + b_final:0.1f} Thousand dollars\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"toc_40291_2.6\"></a>\n",
    "## Plotting\n",
    "You can show the progress of gradient descent during its execution by plotting the cost over iterations on a contour plot of the cost(w,b). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1,1, figsize=(12, 6))\n",
    "plt_contour_wgrad(x_train, y_train, p_hist, ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, the contour plot shows the $cost(w,b)$ over a range of $w$ and $b$. Cost levels are represented by the rings. Overlayed, using red arrows, is the path of gradient descent. Here are some things to note:\n",
    "- The path makes steady (monotonic) progress toward its goal.\n",
    "- initial steps are much larger than the steps near the goal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Zooming in**, we can see that final steps of gradient descent. Note the distance between steps shrinks as the gradient approaches zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1,1, figsize=(12, 4))\n",
    "plt_contour_wgrad(x_train, y_train, p_hist, ax, w_range=[180, 220, 0.5], b_range=[80, 120, 0.5],\n",
    "            contours=[1,5,10,20],resolution=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"toc_40291_2.7.1\"></a>\n",
    "### Increased Learning Rate\n",
    "\n",
    "<figure>\n",
    " <img align=\"left\", src=\"./images/C1_W1_Lab03_alpha_too_big.PNG\"   style=\"width:340px;height:240px;\" >\n",
    "</figure>\n",
    "In the lecture, there was a discussion related to the proper value of the learning rate, $\\alpha$ in equation(3). The larger $\\alpha$ is, the faster gradient descent will converge to a solution. But, if it is too large, gradient descent will diverge. Above you have an example of a solution which converges nicely.\n",
    "\n",
    "Let's try increasing the value of  $\\alpha$ and see what happens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize parameters\n",
    "w_init = 0\n",
    "b_init = 0\n",
    "# set alpha to a large value\n",
    "iterations = 10\n",
    "tmp_alpha = 8.0e-1\n",
    "# run gradient descent\n",
    "w_final, b_final, J_hist, p_hist = gradient_descent(x_train ,y_train, w_init, b_init, tmp_alpha, \n",
    "                                                    iterations, compute_cost, compute_gradient)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, $w$ and $b$ are bouncing back and forth between positive and negative with the absolute value increasing with each iteration. Further, each iteration $\\frac{\\partial J(w,b)}{\\partial w}$ changes sign and cost is increasing rather than decreasing. This is a clear sign that the *learning rate is too large* and the solution is diverging. \n",
    "Let's visualize this with a plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt_divergence(p_hist, J_hist,x_train, y_train)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, the left graph shows $w$'s progression over the first few steps of gradient descent. $w$ oscillates from positive to negative and cost grows rapidly. Gradient Descent is operating on both $w$ and $b$ simultaneously, so one needs the 3-D plot on the right for the complete picture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Congratulations!\n",
    "In this lab you:\n",
    "- delved into the details of gradient descent for a single variable.\n",
    "- developed a routine to compute the gradient\n",
    "- visualized what the gradient is\n",
    "- completed a gradient descent routine\n",
    "- utilized gradient descent to find parameters\n",
    "- examined the impact of sizing the learning rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "dl_toc_settings": {
   "rndtag": "40291"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc-autonumbering": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}