Advance_pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advance Pipeline Land Detection\n",
    "In the Previous project, Canny edge detection and Hough Transformation have limitations on handling shadows, and the constant threshold would not play well with a constantly changing driving environment. A more robust image process logic is require to get a good lane detecting result\n",
    "\n",
    "## Road Map to a More Robust Lane Finding Algorithms \n",
    "The approach would be shown in the following steps\n",
    "1. Camera Calibration and Distortion Correction\n",
    "2. Color and Gradient Threshold Filter\n",
    "3. Perspective Transform Calculation\n",
    "4. Sliding Window Lane Line Finding\n",
    "5. Curvature Measurement\n",
    "\n",
    "## Import\n",
    "I combine all the function from previous notbook into a python script so that I can import all the function here to simplify and only highlight the lane finding logic here\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from frame_process import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Line Class\n",
    "First let's define the a class to hold the all the status of the line.\n",
    "This would be declare as a golbal variables and can be access across functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Line():\n",
    "    def __init__(self):\n",
    "        # was the line detected in the last iteration?\n",
    "        self.detected = False\n",
    "        # x values of the last n fits of the line\\\n",
    "        self.recent_xfitted = []\n",
    "        # average x values of the fitted line over the last n iterations\n",
    "        self.bestx = None\n",
    "        # polynomial coefficients averaged over the last n iterations\n",
    "        self.best_fit = None\n",
    "        # polynomial coefficients for the most recent fit\n",
    "        self.current_fit = [np.array([False])]\n",
    "        # radius of curvature of the line in some units\n",
    "        self.radius_of_curvature = None\n",
    "        # Confident level, calculated after find the line poly\n",
    "        self.confident = 0\n",
    "        # Fail to detect line compare previous result\n",
    "        self.fail_count = 0\n",
    "        # distance in pixels of vehicle center from the line\n",
    "        self.line_to_center = None\n",
    "        # x values for detected line pixels\n",
    "        self.allx = None\n",
    "        # y values for detected line pixels this should never change base on the same resolution\n",
    "        self.ploty = None\n",
    "        # difference in fit coefficients between last and new fits\n",
    "        self.diffs = np.array([0, 0, 0], dtype='float')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Curvature Calculation and polynomial fit the line\n",
    "Here I combined polynomial fit with curvature calcuation into one fucntion. This function will take x and y coordinates and calculate the polynomial coefficients and curvature in meters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_one_line(img_shape, y, x, ploty):\n",
    "    \"\"\"\n",
    "    :param img_shape: process image shape\n",
    "    :param y: collection of y coordinates\n",
    "    :param x: collection of x coordinates\n",
    "    :param ploty: A uniformed y coordinates for plotting\n",
    "    :return: fitted x for easy plotting\n",
    "            and polynomial coefficient both in pixel and meter\n",
    "    \"\"\"\n",
    "    # it a second order polynomial to each with np.polyfit()\n",
    "    fit = np.polyfit(y, x, 2)\n",
    "\n",
    "    # Calcuated X coordinate using ploynomial coefficient and ploty\n",
    "    fitx = fit[0] * ploty ** 2 + fit[1] * ploty + fit[2]\n",
    "\n",
    "    # Define conversions in x and y from pixels space to meters\n",
    "    ym_per_pix = 3 / 80\n",
    "    xm_per_pix = 3.7 / 570\n",
    "\n",
    "    # calculate polynomials in meters\n",
    "    fit_cr = np.polyfit(y * ym_per_pix, x * xm_per_pix, 2)\n",
    "\n",
    "    # Define y-value where we want radius of curvature\n",
    "    # We'll choose the maximum y-value, corresponding to the bottom of the image\n",
    "    y_eval = img_shape[0]\n",
    "\n",
    "    # Implement the calculation of R_curve (radius of curvature)\n",
    "    curve_rad = (1 + (2 * fit_cr[0] * y_eval * ym_per_pix + fit_cr[1]) ** 2) ** 1.5 / np.absolute(\n",
    "        2 * fit_cr[0])\n",
    "    return fitx, fit, curve_rad\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sliding Window Lane Finding\n",
    "Next, I define a fucntion that uses sliding window algorithm to process the binary image, which is the output of our image filter process. \n",
    "`find_lane_pixels` is able to run single lane line and run all lane lines options. In the future code, I will process left and right lane line separately since to reduce run time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_lane_pixels(binary_warped, run_left=False, run_all=False, plot=False):\n",
    "    # Logic to setup which line to run\n",
    "    left_run = run_left or run_all\n",
    "    right_run = not run_left or run_all\n",
    "    \n",
    "    if plot:\n",
    "        # Create an output image to draw on and visualize the result\n",
    "        out_img = np.dstack((binary_warped, binary_warped, binary_warped))\n",
    "\n",
    "    # Create a uniformed y coordinates for plotting\n",
    "    img_shape = binary_warped.shape\n",
    "    ploty = np.linspace(0, img_shape[1] - 1, img_shape[0])\n",
    "    # Take a histogram of the bottom half of the image\n",
    "    histogram = np.sum(binary_warped[binary_warped.shape[0] // 2:, :], axis=0)\n",
    "    # Find the peak of the left and right halves of the histogram\n",
    "    # These will be the starting point for the left and right lines\n",
    "    midpoint = np.int(histogram.shape[0] // 2)\n",
    "    leftx_base = np.argmax(histogram[:midpoint])\n",
    "    rightx_base = np.argmax(histogram[midpoint:]) + midpoint\n",
    "\n",
    "    # HYPERPARAMETERS\n",
    "    # Choose the number of sliding windows\n",
    "    nwindows = 9\n",
    "    # Set the width of the windows +/- margin\n",
    "    margin = 100\n",
    "    # Set minimum number of pixels found to recenter window\n",
    "    minpix = 50\n",
    "\n",
    "    # Set height of windows - based on nwindows above and image shape\n",
    "    window_height = np.int(binary_warped.shape[0] // nwindows)\n",
    "    # Identify the x and y positions of all nonzero pixels in the image\n",
    "    nonzero = binary_warped.nonzero()\n",
    "    nonzeroy = np.array(nonzero[0])\n",
    "    nonzerox = np.array(nonzero[1])\n",
    "    # Current positions to be updated later for each window in nwindows\n",
    "    leftx_current = leftx_base\n",
    "    rightx_current = rightx_base\n",
    "\n",
    "    # Create empty lists to receive left and right lane pixel indices\n",
    "    left_lane_inds = []\n",
    "    right_lane_inds = []\n",
    "\n",
    "    # Step through the windows one by one\n",
    "    for window in range(nwindows):\n",
    "        # Identify window boundaries in x and y (and right and left)\n",
    "        win_y_low = binary_warped.shape[0] - (window + 1) * window_height\n",
    "        win_y_high = binary_warped.shape[0] - window * window_height\n",
    "        if left_run:\n",
    "            win_xleft_low = leftx_current - margin\n",
    "            win_xleft_high = leftx_current + margin\n",
    "            # Identify the nonzero pixels in x and y within the window ###\n",
    "            good_left_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) &\n",
    "                              (nonzerox >= win_xleft_low) & (nonzerox < win_xleft_high)).nonzero()[0]\n",
    "            # Append these indices to the lists\n",
    "            left_lane_inds.append(good_left_inds)\n",
    "            # If found > minpix pixels, recenter next window\n",
    "            # (`right` or `leftx_current`) on their mean position\n",
    "            if len(good_left_inds) > minpix:\n",
    "                leftx_current = np.int(np.mean(nonzerox[good_left_inds]))\n",
    "            if plot:\n",
    "                 # Draw the windows on the visualization image\n",
    "                cv2.rectangle(out_img,(win_xleft_low,win_y_low),\n",
    "                (win_xleft_high,win_y_high),(0,255,0), 2) \n",
    "\n",
    "        if right_run:\n",
    "            win_xright_low = rightx_current - margin\n",
    "            win_xright_high = rightx_current + margin\n",
    "\n",
    "            good_right_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) &\n",
    "                               (nonzerox >= win_xright_low) & (nonzerox < win_xright_high)).nonzero()[0]\n",
    "            right_lane_inds.append(good_right_inds)\n",
    "            if len(good_right_inds) > minpix:\n",
    "                rightx_current = np.int(np.mean(nonzerox[good_right_inds]))\n",
    "            if plot:\n",
    "                 # Draw the windows on the visualization image\n",
    "                cv2.rectangle(out_img,(win_xright_low,win_y_low),\n",
    "                (win_xright_high,win_y_high),(0,255,0), 2) \n",
    "\n",
    "    if left_run:\n",
    "        try:\n",
    "            left_lane_inds = np.concatenate(left_lane_inds)\n",
    "        except ValueError:\n",
    "            # Avoids an error if the above is not implemented fully\n",
    "            pass\n",
    "\n",
    "        # Extract left line pixel positions\n",
    "        leftx = nonzerox[left_lane_inds]\n",
    "        lefty = nonzeroy[left_lane_inds]\n",
    "        left_fitx, left_fit, left_curve = fit_one_line(img_shape, lefty, leftx, ploty)\n",
    "\n",
    "    if right_run:\n",
    "        try:\n",
    "            right_lane_inds = np.concatenate(right_lane_inds)\n",
    "        except ValueError:\n",
    "            # Avoids an error if the above is not implemented fully\n",
    "            pass\n",
    "        # Extract right line pixel position\n",
    "        rightx = nonzerox[right_lane_inds]\n",
    "        righty = nonzeroy[right_lane_inds]\n",
    "        right_fitx, right_fit, right_curve = fit_one_line(img_shape, righty, rightx, ploty)\n",
    "\n",
    "    ## Visualization ##\n",
    "    # Draw the windows on the visualization image if the plot flag is set\n",
    "    if plot:\n",
    "\n",
    "        if left_run:\n",
    "            # Colors in the left and right lane regions\n",
    "            out_img[lefty, leftx] = [255, 0, 0]\n",
    "            # Plots the left and right polynomials on the lane lines\n",
    "            plt.plot(left_fitx, ploty, color='yellow')\n",
    "        if right_run:  \n",
    "            out_img[righty, rightx] = [0, 0, 255]\n",
    "            plt.plot(right_fitx, ploty, color='yellow')\n",
    "        plt.axis('off')\n",
    "        plt.imshow(out_img)\n",
    "        plt.savefig('./test_images/out_img.png',bbox_inches='tight',pad_inches=0)\n",
    "        \n",
    "    # Logic to return correct variables\n",
    "    if run_all:\n",
    "        return left_fitx, right_fitx, ploty, left_fit, right_fit, [left_curve, right_curve]\n",
    "    else:\n",
    "        if run_left:\n",
    "            return left_fitx, ploty, left_fit, left_curve\n",
    "        else:\n",
    "            return right_fitx, ploty, right_fit, right_curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2934.5248026953445, 48941.39860379955]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAADKCAYAAAAGnJP4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAOXklEQVR4nO3dv6ssZx3H8c+cc3bvuRdTiJWFwUqU2IigaBEUIa1WojaCASVgI4ZbSiI2KjYSImnTBBubEPwPbBSxCRKwE5JCJIVwzzn77N6xmJ2z83N35tl9Zj7n3PcLLnNz9iQz2X3ms9/97jPzZHmeCwAwjbO5DwAAniWELgBMiNAFgAkRugAwIUIXACZ0se/BLMsGTW3Isuzg79yrWRIJ/lfy6lP4OUkfSD+Q9M6e3xv26oxw+GVEjImH/v8kvSXpVak+SPITvsCMlb3yvP/JPkmlm+f5wT/3TnbiP1VXxebhy92/d/vXVPvH6Z16vPT+yRU+lha/V/tdmfFigfaCoyfF5uHDeQ8Dd1MI0mIx91GgD6HraFvpPnrU/fA9/NyAk8glZdvQfXnug0EPQtdR2V6g0kWEInTD3IeBHoSuiVqbLJd0Lb3eEbq003AIoeuN0HV1JamnvQB0K96SCV1ve6eMYT4fXknv9bQXqHaxT3/oFj1fzItK19STJ/R0EWe1otJ1Ruiaurrqn70A7BOCtFyu5j4M9CB0TV1dUekiTggvUuka2xu6Qy7vRRpfeUKlizghLHpCl/PZwcFKl+CdTu2ZptJFpP7QhYNB7QWCdwaELqLkhK65wVPGsiy7nzeuMdF6ZhvthfJx3v5Q1xw55WXAL8xyNDhs1BdpVLwT6ql0edvDTvdo4OIIb6NnL2RZRvhOgXm6GK04L5mn640pYyZab2PM08Ve/Z95uufpUii52Bu6+3q4VLuJXUkPHkhnHa8QLQbsQ3vB28FKl+CdCTcyR6R26HKeOoluL9zbZXhmkKtdvf6Ue+oiUhG6H4u1dTwNmjKW53mrqo2ZQnZvpp1lOuln/EztNQOv3ii2D/9T+aWqe/A0PlOmWER0u48gaZFin4eQ74MMrnS7wnJse6ErvO+slAtTSnry/WL76PPF461fSbx/nFDf833q12v7WPiVtNgc+d9n/CQzqr1wqir13gRvQu/QXsAou3MqBOn8XOI08zS6p9sM3phq9160GFIjdBEpbL9DY0VgT1FfpBGcE9jOXmCuLg6rFz6r1W8lEbqujrrhTRm8tAsSoNLFXn3nXHHDG0laLqc7Ggw3KHT3fQFG8CZC6CJSGbpUup4G38R8SPAiXuuZpb2AQaojpzgPCV1vBy8DHhO8VLsndFNsHjzY/YhnF0OE8GNJhK6r0ZcBE7wTuS42l5fdD/PZAn1vw8xe8Db4JuZV+9oJtBpOpKPSrT6zvLWhrRgVhK43bu3o6kB7gbc29CF0vRG6rjaS1v3tBaDPansrXULXE0uwG2k92zf1ShcYoqx0mafriSXYnXWELq8GDqG94G3wFWldt3bEaTX7tB9e19sL5X13ubET9ukO3a67NmMO0asBMz0svRvaC4iwC9335j0QdIpaDbhE8KbVF7rUK9hnF7rlkj2MGCdRsxcI3mlcXzN7AePtQvc7jUc4Tx1ETxmjx5se7QXECOGfkpo9Xc5PF1FXpJUI2rQIXcRo3/CG89TJUaH7zErVImucG9cvSs9V98clafdD3tie2OozxQTd5duS3k6zD8Q76RVpz1Tlm2Bxv+az99K70oO/9zx+6kUFn6GXbnZJF4nMFcLzkqTFK6n3JcZPhJNfBvxMBW9qe9oLFLrow8UR3va2F7hj2MwqsxearwRvbeiWEbrmuOGNs0qlS8hiKELXG6HrjPYCIhC63ghdM7WKlosjEGGzKbaEridC19mN9BzzdBFhtSJ0XRG6Rpr3gXrtRtJS6poQQo8X+xC6vghdI80gfW27OCVXpWGsELiJuStC11ljnTSqWwwVApWuK0LXWcfilMAQhK4vQtdIcxrYj7bthcvLer+XiheHELq+CF1jN1S6iETo+iJ0jZWhe3lJdYtxCF1fhK6RZrCuVsWWkwdjFaH7bbWbVlzLODdC1xiXcyJWCF+urJFWInAdELrG/twTupw6OGS1Wmq5XInGlB9C19m2vVBOcuf0wTC5Qlg0Kl3eql0Qukaap8XXOird6s36gT7t0C0xeuZG6Bqjp4tY9dClynXCwpSxUozjRhGy+luxXb6bYF9NeXv/SCR1BmaV0G2+rqn2zZU7g1HpxqguxpdwYcrwhWK7+N7p99W7wCDSSzR+yoUppTJ0/3H7z7V9M35mRegao72AcXYBG8KfesYNrYa5EbrG/tWYvcDpgn710cEVab4IXWdUuoi0WvXdT5c+wNwIXWeV0KXKxRgh/GT7Zt0M2eb6JJgaoeus0V4Ahiq+SOt7lGp3ToSumdrp0NFeoEbBECG8sR03jBg3hK6Z6ilyQU8Xg9Wr1xAea7HgRsyOCF1jm4309CntBYxXtBduOh6htTA3QtdYrt3UH04VjBHCQufnUpY9nftQ0EDommO+JYZpztMtBg331PVD6Jr7RO98S6DfLnQvZz4SNBG6ZlptBCpdRFitHkuSlsv/Nh6hUTU37jJmpvrhL5P0b0IXg2Sq33uh2C4Wn5rncNCLStdc1+WcdOVwCDdL8kXoGqvOXpD4YIjhCF1fhK6Rrgp2tdqdOOX9qAlf7JdVQvcD8dnIC6FrpCtMQ6i3Fzh90K2+dMMudLvWScOcCF1zzNNFjHro8tnICaFrLJP01RW3dsQQ9cbTLnS/NM/hoBdTxo6ReHHKXJKCtPzWBPvt2D8SS/g6bu8KquVf0+2jhbEzCJVurKkW99tI53+Zad9IK+EikeGbxS4W30i7H8bOeISumdb43UhfP6//nFYDDmHKmC9C191G0vncB4G7htD1ReiaaV4GrLXovGM0QtcXoWssl6h0EYXQ9UUN5W4buvWp78B+hK4vKl0zrVBdi0oXoxG6vghdM62ZCRvVPo8wcwFDELq+CF132/YCbQWMsdpeHcGqI34IXTN97QUqXIxBpeuL0HW3kT7i606MROj6InTN/WEjfZov0jASoeuL0DXTbCO8wjxdRNiF7i8bj9Comhuha6avp8u9FzDGZlNsF4tfVH7KyHFA6LrbThlrXR4MHFBd6onLa3wQuuZ+Q3sBkdqrjhC4Dghdc4/X0orQxShFVbtaMU/XEaFrqFaPbKTlBTUKhirXjK5WurufYX6ErqHa1x0bSWd8BYKh6uukFaFL4DohdM103ntBoq+L0VhJ2hPXOh0jQfmZScqrhcl6u73QLoApe++2vLFNJEhafFbSD9PuB+MQusdK/cmtq9JNtU/CfDoTfOIP70uL9yV9N/2+MBztBXe0FxCJ9oInQtfdtr3wSUIXIxG6nghdQ80pY5L0MY0gjMQ8XU+ErjvaC4hEpeuJ0DVU+z6rnL1A6GIkQtcToWuo1l54ut0SuhiJ0PVE6LorQ7eSxMzswhCEridC11De9Q+VV4qLOjEEoeuJ0DVUC1Vug4pIhK4nQtddR3sBGILQ9UToGjrUXgCGCEG6YH63HU5lQ4faC3yRhiHWaypdR4SuO3q6iER7wROh666jp0v+YgjaC54IXXf0dBFpvf55o9KlMeWAU9kd7QVECuF3HUuwE7xzI3TdMWUMo+WS8m17oTlwGEhzI3Td0V7AKLuPRuv16zo7y3V2tt77b2BanMruGu0F6hQMkyuEorewWISZjwVVfLd5rFQtsjJdmz1dWnJ3X6b040aqhe7NzeX2p3lj5dN0+0c3QvcYUwywRk83zxLulkCfzgRjZ70uTu+Li2p7IeEAYvwMQnvB1O15QU8XkUL4mSTaC244lU3lzb/wsQ0jhfCWJELXDaHrjiljiFT2dOvtBcyN0HVHewGRyp4ula4XTmVTrZ4ulS5GYsqYJ0LX1G1Pl/YComS0F0wRuqb2zV5gZg4Oy2kvmCJ03XW0Fyh6MQTtBU+Erjt6uohEe8EToeuOni4i0V7wROi6Y8oYItFe8MSp7I72AiLtQvel7U8YRA644Y0pLgPGcTKtt63cYp00BpALQtcdt3TEKLtwDduuAisCe6G94I7QRSRC1xOhe1cQuhipDF2WYfdC6LqjFYdIZU+XStcLoeuO9gIi0V7wROiaahW4hC5Gor3giZcj1hQhmKm7vUAA3w+JX8fy4t/Fm5LeTLsvDEele4ws3Z/bBVu72gsJ94sJpXwdMyk8V+xm8Wr6fTF2hiN0TWXNv1DdYiTaC54IXVOtjCV0MRKzFzwRuqZalS4w0mZTbAldL4SuO9oLOMJqRei6IXTvCkIXEUKgp+uG0HVHewFHWK+pdN0Quu4a7QUyGGOEQOi6IXTd0dPFEWgv+CF07wpCFxFoL/ghdN3RT8ARaC/4IXRN3Ra2tBdwBNoLfgjdu4LQRYR2e4GBNDdC1x3tBRxh115ghVMXhK4pbniDU9i1FwhbF4SuKXq6OAVmL/ghdO8KQhcRmL3gh9B1x6dCHIHQ9UPouqO9gCMwZcwPoXtXELqIQE/XD++BY+U9fz+xTNt10sq3xbyxTbVTTGeCN9IgqTZjLDVmph1E6MaYckCdb7ebGfaNtDIVIZXwNQ1/lC6+KOmFdPto4VPZXrQX3DVDFxiB9oIfQtdd+QoRuojA7AU/hK67stJ9OutR4I5i9oIfQtcd7QUcgfaCH0LXHaGLI9Be8EPomrr9ArjS02XiAsaiveCH0DV1G7D0dHEE2gt+CF13tBdwBNoLfghdd4QujkB7wQ+ha+q2p1sJXS70wVghFFuC1weha+q2p8vFETjCel1saTH4IHRNtSrdp8xewHhlpUvo+iB0TbVmL1DpIgLtBT+Erqmuni4wTq71+k1J0mLxkfhWwAOh6658hZiniwghFH2FxSLMfCQoEbruzkXgIkJR1Zahe3Hx/JwHgwo6Pe7ORWsBEYpvBeqzF/gq1gGVrqls++fX59I1911AJGYv+CF0zZ2dSRsqXUQidP1kec43mgAwFSpdAJgQoQsAEyJ0AWBChC4ATIjQBYAJEboAMKH/A1jX3YzMtoPLAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Test Code to show how find_lane_pixels works\n",
    "\n",
    "straight_img = mpimg.imread('test_images/straight_lines1.jpg')\n",
    "test_undist = undistort_img(straight_img)\n",
    "test_binary_wrap = binary_wrap_img(test_undist)\n",
    "\n",
    "fig = plt.figure()\n",
    "\n",
    "left_fitx, right_fitx, ploty, left_fit, right_fit, curvatures = find_lane_pixels(test_binary_wrap,run_all=True,plot=True)\n",
    "\n",
    "# img = cv2.imread('./test_images/out_img.png')\n",
    "\n",
    "print(curvatures)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To save runtime, `find_lane_pixel` would only run when there is no previous result or previous lane finding failed. Once there is a successful lane detection before, we would only run search around previous poly fit lines.Below function is take previous ploynomial coeffieicent and search the pixels around that region instead of defining windows again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def search_around_poly(binary_warped, run_left=False, run_all=False, plot=False):\n",
    "    # HYPERPARAMETER\n",
    "    # Choose the width of the margin around the previous polynomial to search\n",
    "    # The quiz grader expects 100 here, but feel free to tune on your own!\n",
    "    margin = 100\n",
    "    # Create a uniformed y coordinates for plotting\n",
    "    img_shape = binary_warped.shape\n",
    "    ploty = np.linspace(0, img_shape[1] - 1, img_shape[0])\n",
    "\n",
    "    # Logic to setup which line to run\n",
    "\n",
    "    left_run = run_left or run_all\n",
    "    right_run = not run_left or run_all\n",
    "\n",
    "    # Grab activated pixels\n",
    "    nonzero = binary_warped.nonzero()\n",
    "    nonzeroy = np.array(nonzero[0])\n",
    "    nonzerox = np.array(nonzero[1])\n",
    "\n",
    "    if left_run:\n",
    "        left_lane_inds = ((nonzerox >= (left_line.current_fit[0] * nonzeroy ** 2 + left_line.current_fit[1] * nonzeroy +\n",
    "                                        left_line.current_fit[2] - margin)) &\n",
    "                          (nonzerox < (left_line.current_fit[0] * nonzeroy ** 2 + left_line.current_fit[1] * nonzeroy +\n",
    "                                       left_line.current_fit[2] + margin))).nonzero()[0]\n",
    "        # Again, extract left and right line pixel positions\n",
    "        leftx = nonzerox[left_lane_inds]\n",
    "        lefty = nonzeroy[left_lane_inds]\n",
    "        left_fitx, left_fit, left_curve = fit_one_line(img_shape, lefty, leftx, ploty)\n",
    "\n",
    "    if right_run:\n",
    "        right_lane_inds = \\\n",
    "            ((nonzerox >= (right_line.current_fit[0] * nonzeroy ** 2 + right_line.current_fit[1] * nonzeroy +\n",
    "                           right_line.current_fit[2] - margin)) &\n",
    "             (nonzerox < (right_line.current_fit[0] * nonzeroy ** 2 + right_line.current_fit[1] * nonzeroy +\n",
    "                          right_line.current_fit[2] + margin))).nonzero()[0]\n",
    "        rightx = nonzerox[right_lane_inds]\n",
    "        righty = nonzeroy[right_lane_inds]\n",
    "        right_fitx, right_fit, right_curve = fit_one_line(img_shape, righty, rightx, ploty)\n",
    "\n",
    "    ## Visualization ##\n",
    "    # Create an image to draw on and an image to show the selection window\n",
    "    if plot:\n",
    "        out_img = np.dstack((binary_warped, binary_warped, binary_warped)) * 255\n",
    "        window_img = np.zeros_like(out_img)\n",
    "        \n",
    "        \n",
    "        if left_run:\n",
    "            # Color in left and right line pixels\n",
    "            out_img[nonzeroy[left_lane_inds], nonzerox[left_lane_inds]] = [255, 0, 0]\n",
    "            # Generate a polygon to illustrate the search window area\n",
    "            # And recast the x and y points into usable format for cv2.fillPoly()\n",
    "            left_line_window1 = np.array([np.transpose(np.vstack([left_fitx - margin, ploty]))])\n",
    "            left_line_window2 = np.array([np.flipud(np.transpose(np.vstack([left_fitx + margin,\n",
    "                                                                            ploty])))])\n",
    "            left_line_pts = np.hstack((left_line_window1, left_line_window2))\n",
    "\n",
    "            # Draw the lane onto the warped blank image\n",
    "            cv2.fillPoly(window_img, np.int_([left_line_pts]), (0, 255, 0))\n",
    "            # Plot the polynomial lines onto the image\n",
    "            plt.plot(left_fitx, ploty, color='yellow')\n",
    "        \n",
    "        if right_run:\n",
    "        \n",
    "            out_img[nonzeroy[right_lane_inds], nonzerox[right_lane_inds]] = [0, 0, 255]\n",
    "            right_line_window1 = np.array([np.transpose(np.vstack([right_fitx - margin, ploty]))])\n",
    "            right_line_window2 = np.array([np.flipud(np.transpose(np.vstack([right_fitx + margin,\n",
    "                                                                             ploty])))])\n",
    "            right_line_pts = np.hstack((right_line_window1, right_line_window2))\n",
    "\n",
    "            cv2.fillPoly(window_img, np.int_([right_line_pts]), (0, 255, 0))\n",
    "            plt.plot(right_fitx, ploty, color='yellow')\n",
    "        \n",
    "        result = cv2.addWeighted(out_img, 1, window_img, 0.3, 0)\n",
    "        plt.axis('off')\n",
    "        plt.imshow(result)\n",
    "        plt.savefig('./test_images/out_img.png',bbox_inches='tight',pad_inches=0)\n",
    "        \n",
    "\n",
    "    # Logic to return correct variables\n",
    "    if run_all:\n",
    "        return left_fitx, right_fitx, ploty, left_fit, right_fit, [left_curve, right_curve]\n",
    "    else:\n",
    "        if run_left:\n",
    "            return left_fitx, ploty, left_fit, left_curve\n",
    "        else:\n",
    "            return right_fitx, ploty, right_fit, right_curve\n",
    "        \n",
    "\n",
    "def combine_lane_images (left,right):\n",
    "    # Black out right half of left image\n",
    "    cols,rows,chn = left.shape\n",
    "    \n",
    "    left[:,int(rows/2):,:] = 0\n",
    "    \n",
    "    cols,rows,chn = right.shape\n",
    "    # Black out left half of right image\n",
    "    right[:,:int(rows/2),:] = 0\n",
    "    \n",
    "    return cv2.add(left, right)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAADvCAYAAAAJgpRkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAbA0lEQVR4nO3df6xc5X3n8fcHG/+4Tmzzy5YFbEOyzipUalyuQ4jSXdFSGgchmWoTyUhVrArJ1YZUrVRp1+xKm3QltGmlNqv0B11XQTirFsI2ZfEi0tQyqVhVG7DdEmIgDg6Nwq0tvA3FkHvh4h/f/WOeuT6+nus798yP85xzPi9pNDPPPffe78w885lnnvNLEYGZmTXLZVUXYGZmw+dwNzNrIIe7mVkDOdzNzBrI4W5m1kAOdzOzBhpZuEvaJumopGOSdo/q/5iNk/u11YVGsZ27pGXA94HbgSngIHB3RLw49H9mNibu11Ynoxq53wwci4hXIuJd4BFg+4j+l9m4uF9bbYwq3K8FXi3cn0ptZnXmfm21sXxEf1c92i6Y/5G0C9iV7k6OqI7+rOt3wUlYBbxz+Px9Dl9i+eRUqaqyNTl54ct1+HAfz0HFIqJXn1yqRfs1ZNa3+zQ5Ock08L30Wk5OTtbide1bn+/xSeDkumvYcOr/AX29uyt/fy/Ut0c15/4x4AsR8Yl0/75UxH9dYPlqD3BzZ78LBmwCTiwxJ55YYj01UOw30jByc7SGEe5L7ddpmVocvCkiOAZsrsFrWUrf73H44r/99/yHr/8u0PvT/CIVv78X6tujmpY5CGyWdIOkFcAOYN+I/td4vVV1AVah5vZr4MqqC8jEzMqJqksYipFMy0TEGUmfA74JLAMejIgXRvG/xu4nVRdgVWl0vwauqLqATEyvWlN1CUMxsu3cI+LJiPhgRHwgIu6/1LLz53Dz1dCvrNa3pfRrqE/fluTenZya6HslXNay2UO1HseVj3m361Dz8ETE3MX6V4fnq1hjG1/jKFyaMi2TTbhDPd4EduEK1DqsTM2B+3Y9iPPhXveenUW4Fze58pvAmsR9u3485z5mfmPkRZJH7UPivp2X/Ssnaj9qh4zCfbGgkJTZm6AJL7+NQ936dus/tD3nPnzd0eBCHT2bTrd+Y9UVVCab16BmatO3N7a4b3dvrFlfZRlDk1W4d2XT0Rey9qqqK6hUTqPMusm+b1/V8r4NHrm32gXbwbYn6LIPJhvcuvN9u00f4hf07KWG+xIObTBODvcy1jRjJ4eyHPINtq7lfRs8cm+nFGqr11ZbhtmQzX1gr3XfZtmoDpY7XlmE++TkZE2+AqYa128otLV7FFuP1606denbczVuON+32/4NLf9X7dKyCPeuOrwJgHlz7u1Um9cqE7V5vlo+LdMkWYU71OS4Fi2fc4fzo7q2j+6WohZ92+E+p+49O7tw78rzTZBe7gnPS1p5OfZtz7k3z0DhLumHkr4r6TlJh1LblZL2S3o5XS96mOjDhw/P7eRRvIzNUs+kMrGO+s/I2aU0pm8v1bp1WX742NINY+T+8xGxJSK2pvu7gQMRsRk4kO43y7oNnP/S5jdCg7Wvb2/YMPfh45Cvt1FMy2wH9qbbe4G7RvA/KpI6+wVz7hmPwmzYGtu354K8MOee9TcMW9Sg4R7AX0s6nM74DrAxIk4ApOsNC/527XTn3L3SqQVa1bfngtwrVBtj0K31Px4RxyVtAPZL+l6/v5jeMLsWXTAraXQzsZYlTcfcSeVnSLcla1Xfnhu5r13r6ZiGGGjkHhHH0/VJ4DHgZuA1SZsA0vXJBX53T0RsLcxn1seadXg6ptla27fXrfN0TEOUDndJayS9t3sb+CXgCLAP2JkW2wk8PmiR+Uid/vKV1ZaRgeI2200b6bWxb88F+kr3bTj/vbzOPXuQaZmNwGOpUywH/jwi/krSQeBRSfcAPwI+PXiZuZj/Urd7hNM9PnkDR3qt69vzP6Ab+Jr2be30KUTn3V7nZ6F0uEfEK8CHe7T/GLhtkKLMquS+3W5r3pnmzaqLGIJmHP5sbLqf5+3WHa0Xp2XaPNJrgtxO9VeVADQ7c8G0TF17draHH7D6cLBbo8zOzN2sc892uJfW7lFOcbTuEV+ztP31/OjsTCP2P3e4D6TOL/3guiHgkXvztDng10+/MffOrnPPdriXVueXfbjaHARN1PYP64nCtEyde7bD3UopBkDbw8Ca5bGGzLl7axkrzaFujVSYc68zj9yXxDsxFXk6pjm8E1PB7HStp2O6HO5Lkjr8uXPpfhO6gFkhzFPfbvUHd2Faps4c7mXMXEabg93nT22udZdd1upgF8D0KbRqTdWlDMzhXsZpaPuUjDXTSvyhzTvTsHKi6ioG5nAv40zVBZiNho8JSWdaxuHeUqeh3kedMOttBT5WkMO9zc6ADyJmTdSdlmnzvDuzM9CGOXdJD0o6KelIoe1KSfslvZyur0jtkvRlScckPS/pplEWX5nZgzjY6899+2IvHjzY7mAHmH6jEedJ7mfk/hCwbV7bbuBARGwGDqT7AJ8ENqfLLuCB4ZSZmdMfwVMy59U4DB7CffsCH//IR9o9JQMXTMvUtmfTR7hHxNPA6/OatwN70+29wF2F9q9Gx7eB9d1zTjbK6aeo98s+PHWen3XfvtjfPvVUnT+shyOFe93XqpU9/MDGiDgBEBEn0hniAa4FXi0sN5XaTsz/A3U8Q3yH4Ez3ZW/5m4BGbjbX2r4tiW3zTsTSSrPTsGpNrYMdhr9Ctdfz0bOX1PcM8VHYWsZapPF9OyJYSa2n2Yaj5VvLvNb9SpquT6b2KeD6wnLXAcfLl5ep2aoLsBFqdd+uf6QNwfSp1qxQ7WUfsDPd3gk8Xmj/TNqy4BbgVPcrbnPIOzE1W2v7tiRWVF1EDs6eacfIXdLDwP8F/pWkKUn3AF8Ebpf0MnB7ug/wJPAKcAz4U+CzI6m6Ut1pGas79+0LdadljEZs577oCtWIuHuBH93WY9kA7h20qLwJzrZ8TrIh3LcvJInPtn2+vasNI3fr4dzii5jV0bKqC8iFw72NwuFujRQRDveuFaurrmBgDvclk8PdGkmSw71rTXu3lmm3s1UXkI+I8HbRDeJwT1asJqj33iwO9zI8cgfO7+zSwL1UW8vh3rElTcvUuWc73MuY+ziv80s/mOJo3SP35ugGQps/sAOYacOBw6yHuZF7nV/64WlzEDRNd+Te9g/st1d65N5OZ/EzZ420HK9SAnhz9dqqSxiYI6qMoPXPnEfrzXQZDndxflqmzsoe8rfdztH6cG/71/amWobDPQBdXv+j7LQ8oko6R70n48wWcBneGKyUO6su4GIO9zI8creG8si9OTwtY6V4zr252v7Kat51XXn8WYbPsGcN5RnH5ujneO4PSjop6Uih7QuS/lHSc+lyR+Fn90k6JumopE+MqvBKOdwbwX37Yt4QrDn6eR0fArb1aP9SRGxJlycBJN0I7AB+Ov3OH0tq3h7Nl+Fwb4aHcN++gEfuzbFouEfE08Drff697cAjETEbEf9A56w1Nw9QX768SUHtuW9fzCP35hjkdfycpOfTV9srUtu1wKuFZaZS20Uk7ZJ0SNKhAWqohnt/07W2b3vk3hxlY+oB4APAFuAE8HupvVe/6DmBERF7ImJrRGwtWUN1RBq5+23QQK3u292tfL01VP2VCveIeC0izkbEOTonC+5+PZ0Cri8seh1wfLASM+QVqo3V9r7tXTiao9TrKGlT4e4vA92tDfYBOyStlHQDsBl4drASM+Rwb6y29+3A30ebYtGdmCQ9DNwKXC1pCvg8cKukLXT6wg+BXwOIiBckPQq8CJwB7o2I5u3w5qHNnIio7Vd49+2Lec79vLp/0C0a7hFxd4/mr1xi+fuB+wcpyuqjrsEO7tu9+AtpcvrdWgc7eAxajp81a6jGfRXpemKJy7/95kjKGCfHVBl1/0g3W4BH7snsTNUVDMzhXoYPt2YNdbrqAnIx+3bVFQzM4V5G43Y6N+twuCfvOtzbySN3ayiHe+JpmZbyyN0ayuGeeOTeUvU/vaJZT+9UXUAuZry1TDt55G4N5ZF74mmZlvKcO9DZOzXCG881icO945533679ZqEO9zI8cp8L9TrvoWoXc7h3tvWfmJ2p/e4sDvcyln+Dtu/JJMnB3kD/+xvfaP3rKuAPGrCdu3L4Wi3pLeBo1XWUcDXwT1UXUUId6x6k5p+KiGuGWUy/3LfHqo41w4j6di6zx0freGIDSYdc93jUsebEfXtM6lgzjK5uT8uYmTWQw93MrIFyCfc9VRdQkusenzrWDK57nOpYM4yo7ixWqJqZ2XDlMnI3M7MhcribmTVQ5eEuaZuko5KOSdpddT1Fkh6UdFLSkULblZL2S3o5XV+R2iXpy+lxPC/ppopqvl7StyS9JOkFSb+Re92SVkl6VtJ3Us2/ndpvkPRMqvlrklak9pXp/rH08/eNu+Z+5Nq33a/HWnd1fbt7fJAqLnR25P8B8H46x1r8DnBjlTXNq+/fADcBRwptvwvsTrd3A7+Tbt8BdHddvQV4pqKaNwE3pdvvBb4P3Jhz3el/vyfdvhx4JtXyKLAjtf8J8O/S7c8Cf5Ju7wC+VnVf6fGYsu3b7tft6NtVd7KPAd8s3L8PuK/KmnrU+L55b4KjwKZChzuabv934O5ey1Vc/+PA7XWpG5gA/g74KJ299pbP7yvAN4GPpdvL03Kq+rme9ziy7tvu15XUPNa+XfW0zLXAq4X7U6ktZxsj4gRAut6Q2rN7LOkr3c/SGS1kXbekZZKeA04C++mMet+IiDM96pqrOf38FHDVeCteVBbP6xJk3T+K6tSvobq+XXW49zpCUV23zczqsUh6D/B14Dcj4lJnHsii7og4GxFbgOuAm4EP9VosXWdR8yLqUGM/snocdevXUF3frjrcp4DrC/evA45XVEu/XpO0CSBdn0zt2TwWSZfTeQP8WUT8ZWrOvm6AiHgD+Bs685LrJXWPf1Ssa67m9PN1wOvjrXRRWT2vfci+f9S5X8P4+3bV4X4Q2JzWHK+gswJhX8U1LWYfsDPd3kln7q/b/pm0lv4W4FT36+I4SRLwFeCliPj9wo+yrVvSNZLWp9urgV8EXgK+BXxqgZq7j+VTwFORJikzUre+nW3/gHr2a6i4b2ewYuQOOmu+fwD8p6rrmVfbw8AJOucwmALuoTP/dQB4OV1fmZYV8EfpcXwX2FpRzT9H52vc88Bz6XJHznUDPwP8far5CPCfU/v7gWeBY8D/BFam9lXp/rH08/dX3VcWeFxZ9m3363b0bR9+wMysgaqeljEzsxFwuJuZNZDD3cysgRzuZmYN5HA3M2sgh7uZWQM53M3MGsjhbmbWQA53M7MGcribmTWQw93MrIEc7mZmDeRwNzNroJGFe65nfjcza4ORHPJX0jI6x7G+nc7xog/SOVnti0P/Z2ZmdpFRjdxvBo5FxCsR8S7wCLB9RP/LzMzmWb74IqX0OvP4R4sLSNoF7Ep3J0dUR3/W9bvgJEwAM4fP3+fwJZZPTpWqarz6fg6StVcx+eaPgb6egcqfg4jodeJhs8YaVbgvegbviNgD7AGQVO3poP51vwsegg8C39f5+z0f6jxPlKpqvE4Bd/a3aAB/cOfd/PoTfwj09QzU4zkwa5BRTctkd+bxoXmreKedg0EBMysnqi7DzC5hVOFetzO/9++txRdpA4e7Wd5GMi0TEWckfQ74JrAMeDAiXhjF/xq7n1RdQB6mV62pugQzu4SRbeceEU9GxAcj4gMRcf+llp2crHZ9av/aOQ3Ty6mJpa6BNbNxymYP1VFsbz98Me92HWoenihcPC1jlrdswh3qEvBWXKHq7zJmecoi3A8fPr+ltAO+Hjznbpa3LMK9Hw79vOxfOeFRu1nGsgl36dJRISmzgG95tHnO3Sxr2YQ7dAL8UiG+2AfA2KzfWHUFlZl7Bdasr7IMM1tEVuHelU2IL2TtVVVXUKkAj9zNMpdluGfvgm28c5oqGq0LPnKXGu59HrfGzIbD4V7GmnbvwCPwyN0scw73JUlj19Vrqy0jB8tGdUBRMxuGLMJ9cnIysy1hFpJqXL+h0Jb5+oERq8OrZtZGWYR7Vz0Cnnlz7mZm+ckq3KET8NmHfMvn3Iva/b3FLF/ZhXtXngGfomzCc+5mlreBwl3SDyV9V9Jzkg6ltisl7Zf0crq+YrG/c/jw4bkdmIqXsVnqKeAm1uHZZjPL2TBG7j8fEVsiYmu6vxs4EBGbgQPpfrOs28D5CQmHvJnlZxTTMtuBven2XuCuEfyPiqQgv2DO3bPOZpafQcM9gL+WdFjSrtS2MSJOAKTrDb1+UdIuSYe60zn10J1z9wpVM8vboHuifDwijkvaAOyX9L1+fzEi9gB7ACTVZG4jlTmxliVNx9zJ0uf1zcwGMNDIPSKOp+uTwGPAzcBrkjYBpOuTgxaZnTXr8HSMmeWsdLhLWiPpvd3bwC8BR4B9wM602E7g8UGLzEcK9MtXVltGJmLetZnlY5BpmY3AY2mTxeXAn0fEX0k6CDwq6R7gR8CnBy8zF/NjrL2j97XTpxCdZ6S9z4JZvkqHe0S8Any4R/uPgdsGKcryt+adad6suggzW5AP7bck3bFquwWg2ZkLpmU8ejfLS7aHH7DMzc7M3XSwm+XH4V5au0fwH52d8T66ZhlzuA+kvbG2fvqNuUfvkbtZfhzupbU70iYK0zLt/Ygzy5fD3Up5zHPuZlnz1jJWTmHO3czy45H7kngnpjmz056OMcuYw31JUpifO5futzjeCtMyZpYfh3sZM5fR5mAXwPQptGpN1aWY2QIc7mWchlZPyQC8Mw0rJ6quwswW4HAv40zVBWRgdsbhbpYxh3sZp6H1R1RxuJtlbdFwl/SgpJOSjhTarpS0X9LL6fqK1C5JX5Z0TNLzkm4aZfGVOQOtP4jY7Ax4zt0sW/2M3B8Cts1r2w0ciIjNwIF0H+CTwOZ02QU8MJwyMzN7kFYHO8D0Gz6XrFnGFg33iHgaeH1e83Zgb7q9F7ir0P7V6Pg2sL57yr1GOf0RWj0lAxdMy7T8Y84sS2Xn3DdGxAmAdL0htV8LvFpYbiq1Ncvpp2h9pKVwb/maB7NsDfvwA73e5z1TUNIuOlM3NSM40420Fgf87DSsWuNgN8tU2ZH7a93plnR9MrVPAdcXlrsOON7rD0TEnojYGhFbS9ZQkShsLdNi3lrGLGtlw30fsDPd3gk8Xmj/TNpq5hbgVHf6plFmqy4gA9OnvELVLGOLTstIehi4Fbha0hTweeCLwKOS7gF+BHw6Lf4kcAdwDJgBfnUENVesOy3TcmfPeORulrFFwz0i7l7gR7f1WDaAewctKm/daRnzdu5m+fIeqksmOFt1DZnwyN0sWw73Ms4tvkgrONzNsuVwX7JwuHetWF11BWa2AIf7ksnh3rXGW8uY5crhXobn3DtWrCZo/Rb/ZllyuJfhkTsAW9K0jPdSNcuPw72MuaFqe2MtgBkfOMwsWw73MuZG7u2OtbdXeuRuliuHexln8TMHvLl6bdUlmNkChn1UyHYIOuHe4rl3AZd7O3ezbDncyzhH60fuAejyFVWXYWYLaHlElXQOTzSXcWfVBZi1h8O9DI/czSxzjigrRfOuzSwvi4a7pAclnZR0pND2BUn/KOm5dLmj8LP7JB2TdFTSJ0ZVeKVafoY9M8tfPyP3h4BtPdq/FBFb0uVJAEk3AjuAn06/88eSlg2r2Gw43M0sc4uGe0Q8Dbze59/bDjwSEbMR8Q90zsh08wD15ekyHO5mlrVB5tw/J+n5NG1zRWq7Fni1sMxUamueFm/jbmb5KxvuDwAfALYAJ4DfS+291q/1HONK2iXpkKRDJWuojldDm1nmSsVURLwWEWcj4hzwp5yfepkCri8seh1wfIG/sScitkbE1jI1VEqkkbu3FTGzPJUKd0mbCnd/GehuSbMP2CFppaQbgM3As4OVmCGvUDWzzC16+AFJDwO3AldLmgI+D9wqaQudiPsh8GsAEfGCpEeBF4EzwL0R0bxTWzjczSxzi4Z7RNzdo/krl1j+fuD+QYrKnufc5wSenDLLkWPKyjv9roPdLFMO9zKa+qw9scTl335zJGWY2eCaGlOj5eFqx+xM1RWY2QIc7mX4KPgds29XXYGZLcDhXkbzjpZTzrsOd7NcOdzL8Mi9w9MyZtlyuJfhkXuHR+5m2XK4l+FTh3bMeGsZs1w53MvwyL3D0zJm2XK4l+E5dwDuefdtH4XBLFMO9zI8cieAidkZb/JvlimPQctY/g3gjkUXazKBt3M3y5giqv9iLekt4GjVdZRwNfBPVRdRQh3rHqTmn4qIa4ZZjFnuchm5H63jSTskHXLd41HHms2q5Dl3M7MGcribmTVQLuG+p+oCSnLd41PHms0qk8UKVTMzG65cRu5mZjZElYe7pG2Sjko6Jml31fUUSXpQ0klJRwptV0raL+nldH1FapekL6fH8bykmyqq+XpJ35L0kqQXJP1G7nVLWiXpWUnfSTX/dmq/QdIzqeavSVqR2lem+8fSz9837prNcldpuEtaBvwR8EngRuBuSTdWWdM8DwHb5rXtBg5ExGbgQLoPncewOV12AQ+Mqcb5zgC/FREfAm4B7k3Pac51zwK/EBEfBrYA2yTdAvwO8KVU8z8D96Tl7wH+OSL+JfCltJyZFVQ9cr8ZOBYRr0TEu8AjwPaKa5oTEU8Dr89r3g7sTbf3AncV2r8aHd8G1kvaNJ5Kz4uIExHxd+n2W8BLwLVkXHf63z9Jdy9PlwB+AfiLBWruPpa/AG6T5CMhmBVUHe7XAq8W7k+ltpxtjIgT0AlSYENqz+6xpOmKnwWeIfO6JS2T9BxwEtgP/AB4IyLO9Khrrub081PAVeOt2CxvVYd7r9FWXTffyeqxSHoP8HXgNyPiUgdez6LuiDgbEVuA6+h8o/tQr8XSdRY1m+Ws6nCfAq4v3L8OOF5RLf16rTttka5PpvZsHouky+kE+59FxF+m5uzrBoiIN4C/obO+YL2k7iEyinXN1Zx+vo6Lp8/MWq3qcD8IbE5bRawAdgD7Kq5pMfuAnen2TuDxQvtn0tYntwCnutMg45Tmnr8CvBQRv1/4UbZ1S7pG0vp0ezXwi3TWFXwL+NQCNXcfy6eAp8I7bJhdoPKdmCTdAfw3OkdJfzAi7q+0oAJJDwO30jki4WvA54H/BTwK/AvgR8CnI+L1FKp/SGfrmhngVyPiUAU1/xzwf4DvAudS83+kM++eZd2SfobOCtJldAYcj0bEf5H0fjor2a8E/h74lYiYlbQK+B901ie8DuyIiFfGWbNZ7ioPdzMzG76qp2XMzGwEHO5mZg3kcDczayCHu5lZAznczcwayOFuZtZADnczswZyuJuZNdD/B6X63W/Nkgm/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Test Code for search around poly\n",
    "# Delcare the line class and record previous ploy fit values\n",
    "left_line = Line()\n",
    "right_line = Line()\n",
    "left_line.current_fit = left_fit\n",
    "right_line.current_fit = right_fit\n",
    "\n",
    "_ = search_around_poly(test_binary_wrap,run_left = True,plot=True)\n",
    "left_img = cv2.imread('./test_images/out_img.png')\n",
    "_ = search_around_poly(test_binary_wrap,run_left = False,plot=True)\n",
    "right_img = cv2.imread('./test_images/out_img.png')\n",
    "\n",
    "\n",
    "plt.clf()\n",
    "\n",
    "plt.subplot(221)\n",
    "plt.imshow(left_img)\n",
    "plt.subplot(222)\n",
    "plt.imshow(right_img)\n",
    "\n",
    "\n",
    "result = combine_lane_images(left_img,right_img)\n",
    "\n",
    "plt.subplot(223)\n",
    "plt.imshow(result)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Draw_poly_fill uses the binary wrap and undist images and all the lane detection result to plot the final output image.\n",
    "# \n",
    "def draw_poly_fill(binary_wrap, undist):\n",
    "    # Draw unwarped the poly fill onto the image\n",
    "    # Create an image to draw the lines on\n",
    "    warp_zero = np.zeros_like(binary_wrap).astype(np.uint8)\n",
    "    color_warp = np.dstack((warp_zero, warp_zero, warp_zero))\n",
    "\n",
    "    left_fitx = left_line.bestx\n",
    "    right_fitx = right_line.bestx\n",
    "    ploty = left_line.ploty\n",
    "\n",
    "    # Recast the x and y points into usable format for cv2.fillPoly()\n",
    "    pts_left = np.array([np.transpose(np.vstack([left_fitx, ploty]))])\n",
    "    pts_right = np.array([np.flipud(np.transpose(np.vstack([right_fitx, ploty])))])\n",
    "    pts = np.hstack((pts_left, pts_right))\n",
    "\n",
    "    # Draw the lane onto the warped blank image\n",
    "    cv2.fillPoly(color_warp, np.int_([pts]), (0, 255, 0))\n",
    "\n",
    "    # Warp the blank back to original image space using inverse perspective matrix (Minv)\n",
    "    newwarp = perspective_transform(color_warp, inverse=True)\n",
    "    # cv2.warpPerspective(color_warp, Minv, (image.shape[1], image.shape[0]))\n",
    "    # Combine the result with the original image\n",
    "    result = cv2.addWeighted(undist, 1, newwarp, 0.3, 0)\n",
    "    \n",
    "    # Add Curvature results to the image\n",
    "    left_curve_text = 'Left Radius of Curvature: ' + str(round(left_line.radius_of_curvature, 2)) + 'Meters'\n",
    "    right_curve_text = 'Right Radius of Curvature: ' + str(round(right_line.radius_of_curvature, 2)) + 'Meters'\n",
    "   \n",
    "    result = cv2.putText(result, left_curve_text, (300, 100), cv2.FONT_HERSHEY_SIMPLEX,\n",
    "                         1, (0, 255, 255), 2, cv2.LINE_AA)\n",
    "    result = cv2.putText(result, right_curve_text, (300, 150), cv2.FONT_HERSHEY_SIMPLEX,\n",
    "                         1, (0, 255, 255), 2, cv2.LINE_AA)\n",
    "\n",
    "    # To dubug, put fail counter and confident level onto the image\n",
    "    left_line_status = 'left: Fail counter:' + str(left_line.fail_count) + ' confident:' + str(left_line.confident)\n",
    "    right_line_status = 'right: Fail counter:' + str(right_line.fail_count) + ' confident:' + str(right_line.confident)\n",
    "\n",
    "    result = cv2.putText(result, left_line_status, (50, 50), cv2.FONT_HERSHEY_SIMPLEX,\n",
    "                         1, (0, 255, 255), 2, cv2.LINE_AA)\n",
    "    result = cv2.putText(result, right_line_status, (600, 50), cv2.FONT_HERSHEY_SIMPLEX,\n",
    "                         1, (0, 255, 255), 2, cv2.LINE_AA)\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Single Lane Detection logic\n",
    "I decide to detect left and right lane separately. If there only one lane detection yield bad result, I don't have to recalculate the other lane and waste runtime on that. This will speed up the algorithm. \n",
    "This function will run for both left and right lane. To differentiate left and right in the lane finding logic, the flag called run_left is implemented. Below are how this logic works:\n",
    "### Decide the complexity of the lane detection based on previous results\n",
    "if previous result is lane detected and it has high confident, then uses search_around poly.\n",
    "Otherwise, see if the fail counter is between 0 and 10, then perform sliding window search.\n",
    "If the fail counter is greater, use a more complex color filtering logic by flag overdrive on` binary_wrap_img` function\n",
    "### First time running\n",
    "The very first iteration would have no previous result. Therefore, when radius of curvature is None, append the first iteration result to the line class without any checks. Also, during the complexity decision, with fail count = 0 and line.detected =  False, first iteration would use window sliding logic to pick up the lane line.\n",
    "### Confident Check\n",
    "For the confident check/sanity check, I check for radius of curvatures, line distance to the image center and polynomial coefficient against previous results. If the check passes, add 1 to the confident variable. When the confident is higher than 1, meaning the current result is good and then append it to the line result\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def single_lane_detection(line, undist, run_left=False, fail_limit=10):\n",
    "    # If the line detected previous iteration, preform search base on previous polynomial\n",
    "    binary_wrap = binary_wrap_img(undist)\n",
    "    if line.detected and line.confident >= 2:\n",
    "        fitx, ploty, fit, curve = search_around_poly(binary_wrap, run_left)\n",
    "    else:\n",
    "        # If fail counter is less, preform simplify image filter and use sliding window to find lines\n",
    "        if line.fail_count < fail_limit and line.fail_count >= 0:\n",
    "            fitx, ploty, fit, curve = find_lane_pixels(binary_wrap, run_left)\n",
    "\n",
    "        # If fail too many times, preform a more complex image filter and sliding window\n",
    "        else:\n",
    "            binary_wrap = binary_wrap_img(undist, overdrive=True)\n",
    "            # Need to find a way to share this in case of the other line failed to detect as well\n",
    "            fitx, ploty, fit, curve = find_lane_pixels(binary_wrap, run_left)\n",
    "\n",
    "    # Calculate in meters\n",
    "    current_line_to_center = abs(fitx[-1] - binary_wrap.shape[0] / 2)\n",
    "    # if the radius is None meaning first time run, saving all the parameter in line\n",
    "    if line.radius_of_curvature is None:\n",
    "        line.detected = True\n",
    "        line.radius_of_curvature = curve\n",
    "        line.current_fit = fit\n",
    "        line.recent_xfitted = fitx\n",
    "        line.bestx = fitx\n",
    "        line.allx = [fitx]\n",
    "        line.line_to_center = current_line_to_center\n",
    "        line.ploty = ploty\n",
    "    else:\n",
    "        # Here check for current results with previous.\n",
    "        confident = 0\n",
    "        if line.radius_of_curvature * (1 + 0.2) >= curve > line.radius_of_curvature * (1 - 0.2):\n",
    "            confident += 1\n",
    "        # Check line distance to the center\n",
    "        if line.line_to_center * (1 + 0.1) >= current_line_to_center > \\\n",
    "                line.line_to_center * (1 - 0.1):\n",
    "            confident += 1\n",
    "        # Calculate fit coefficients difference\n",
    "        if np.all(abs(fit) <= abs(line.current_fit) * (1 + 0.5)) and np.all(\n",
    "                abs(fit) > abs(line.current_fit) * (1 - 0.5)):\n",
    "            confident += 1\n",
    "        # try not strict one first, can change later\n",
    "        # If the confident level is great than 1, record the result into line\n",
    "        if confident >= 1:\n",
    "            # Append result into line\n",
    "            line.detected = True\n",
    "            line.radius_of_curvature = curve\n",
    "            line.current_fit = fit\n",
    "            line.recent_xfitted = fitx\n",
    "            line.allx.append(fitx)\n",
    "            line.line_to_center = current_line_to_center\n",
    "            line.fail_count = 0\n",
    "\n",
    "            # average x fitted only takes the most recent 20\n",
    "            if len(line.allx) > 20:\n",
    "                line.bestx = np.mean(line.allx[-20:], axis=0)\n",
    "            else:\n",
    "                line.bestx = np.mean(line.allx, axis=0)\n",
    "        # else increase fail counter, discard current result\n",
    "        else:\n",
    "            line.fail_count += 1\n",
    "        line.confident = confident\n",
    "    return binary_wrap\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Video Processing\n",
    "The video lane detection logic will un-distort the image and perform single lane detection logic to left and right lanes perspectively. Finally, it would call `draw_poly_fill` to fill the lane line with green shade and print out curvatures and debug message on the screen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def video_lane_detection(img):\n",
    "    global left_line, right_line\n",
    "    fail_allowed = 5\n",
    "\n",
    "    # Un-distort image using Camera calibration data\n",
    "    undist = undistort_img(img)\n",
    "\n",
    "    # Left line detection\n",
    "    binary_wrap = single_lane_detection(left_line, undist, True, fail_allowed)\n",
    "    # Right line detection\n",
    "    _ = single_lane_detection(right_line, undist, False, fail_allowed)\n",
    "\n",
    "\n",
    "    result_img = draw_poly_fill(binary_wrap, undist )\n",
    "    return result_img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_result = video_lane_detection(straight_img)\n",
    "mpimg.imsave('output_images/poly_final.jpg', test_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "t:   0%|          | 0/1260 [00:00<?, ?it/s, now=None]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Moviepy - Building video output_video/project_video.mp4.\n",
      "Moviepy - Writing video output_video/project_video.mp4\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "t:   1%|          | 7/1260 [00:02<08:18,  2.51it/s, now=None]"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<timed eval>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n",
      "\u001b[0;32m<decorator-gen-180>\u001b[0m in \u001b[0;36mwrite_videofile\u001b[0;34m(self, filename, fps, codec, bitrate, audio, audio_fps, preset, audio_nbytes, audio_codec, audio_bitrate, audio_bufsize, temp_audiofile, rewrite_audio, remove_temp, write_logfile, verbose, threads, ffmpeg_params, logger)\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/Udacity/lib/python3.7/site-packages/moviepy/decorators.py\u001b[0m in \u001b[0;36mrequires_duration\u001b[0;34m(f, clip, *a, **k)\u001b[0m\n\u001b[1;32m     52\u001b[0m         \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Attribute 'duration' not set\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 54\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mclip\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     55\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     56\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<decorator-gen-179>\u001b[0m in \u001b[0;36mwrite_videofile\u001b[0;34m(self, filename, fps, codec, bitrate, audio, audio_fps, preset, audio_nbytes, audio_codec, audio_bitrate, audio_bufsize, temp_audiofile, rewrite_audio, remove_temp, write_logfile, verbose, threads, ffmpeg_params, logger)\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/Udacity/lib/python3.7/site-packages/moviepy/decorators.py\u001b[0m in \u001b[0;36muse_clip_fps_by_default\u001b[0;34m(f, clip, *a, **k)\u001b[0m\n\u001b[1;32m    133\u001b[0m              for (k,v) in k.items()}\n\u001b[1;32m    134\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 135\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mclip\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mnew_a\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mnew_kw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<decorator-gen-178>\u001b[0m in \u001b[0;36mwrite_videofile\u001b[0;34m(self, filename, fps, codec, bitrate, audio, audio_fps, preset, audio_nbytes, audio_codec, audio_bitrate, audio_bufsize, temp_audiofile, rewrite_audio, remove_temp, write_logfile, verbose, threads, ffmpeg_params, logger)\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/Udacity/lib/python3.7/site-packages/moviepy/decorators.py\u001b[0m in \u001b[0;36mconvert_masks_to_RGB\u001b[0;34m(f, clip, *a, **k)\u001b[0m\n\u001b[1;32m     20\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mclip\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mismask\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     21\u001b[0m         \u001b[0mclip\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclip\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_RGB\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 22\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mclip\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     23\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     24\u001b[0m \u001b[0;34m@\u001b[0m\u001b[0mdecorator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdecorator\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/Udacity/lib/python3.7/site-packages/moviepy/video/VideoClip.py\u001b[0m in \u001b[0;36mwrite_videofile\u001b[0;34m(self, filename, fps, codec, bitrate, audio, audio_fps, preset, audio_nbytes, audio_codec, audio_bitrate, audio_bufsize, temp_audiofile, rewrite_audio, remove_temp, write_logfile, verbose, threads, ffmpeg_params, logger)\u001b[0m\n\u001b[1;32m    305\u001b[0m                            \u001b[0mverbose\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mverbose\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mthreads\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mthreads\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    306\u001b[0m                            \u001b[0mffmpeg_params\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mffmpeg_params\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 307\u001b[0;31m                            logger=logger)\n\u001b[0m\u001b[1;32m    308\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    309\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mremove_temp\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mmake_audio\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/Udacity/lib/python3.7/site-packages/moviepy/video/io/ffmpeg_writer.py\u001b[0m in \u001b[0;36mffmpeg_write_video\u001b[0;34m(clip, filename, fps, codec, bitrate, preset, withmask, write_logfile, audiofile, verbose, threads, ffmpeg_params, logger)\u001b[0m\n\u001b[1;32m    219\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    220\u001b[0m         for t,frame in clip.iter_frames(logger=logger, with_times=True,\n\u001b[0;32m--> 221\u001b[0;31m                                         fps=fps, dtype=\"uint8\"):\n\u001b[0m\u001b[1;32m    222\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mwithmask\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    223\u001b[0m                 \u001b[0mmask\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m255\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mclip\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmask\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_frame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/Udacity/lib/python3.7/site-packages/moviepy/Clip.py\u001b[0m in \u001b[0;36miter_frames\u001b[0;34m(self, fps, with_times, logger, dtype)\u001b[0m\n\u001b[1;32m    470\u001b[0m         \u001b[0mlogger\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mproglog\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdefault_bar_logger\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlogger\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    471\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mt\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0miter_bar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mduration\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1.0\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mfps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 472\u001b[0;31m             \u001b[0mframe\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_frame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    473\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mdtype\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mframe\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtype\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    474\u001b[0m                 \u001b[0mframe\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mframe\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mastype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<decorator-gen-136>\u001b[0m in \u001b[0;36mget_frame\u001b[0;34m(self, t)\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/Udacity/lib/python3.7/site-packages/moviepy/decorators.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(f, *a, **kw)\u001b[0m\n\u001b[1;32m     87\u001b[0m         new_kw = {k: fun(v) if k in varnames else v\n\u001b[1;32m     88\u001b[0m                  for (k,v) in kw.items()}\n\u001b[0;32m---> 89\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mnew_a\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mnew_kw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     90\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mdecorator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdecorator\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwrapper\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     91\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/Udacity/lib/python3.7/site-packages/moviepy/Clip.py\u001b[0m in \u001b[0;36mget_frame\u001b[0;34m(self, t)\u001b[0m\n\u001b[1;32m     91\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mframe\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     92\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 93\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmake_frame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     94\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     95\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mfl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mapply_to\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkeep_duration\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/Udacity/lib/python3.7/site-packages/moviepy/Clip.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(t)\u001b[0m\n\u001b[1;32m    134\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    135\u001b[0m         \u001b[0;31m#mf = copy(self.make_frame)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 136\u001b[0;31m         \u001b[0mnewclip\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_make_frame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_frame\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    137\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    138\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mkeep_duration\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Applications/anaconda3/envs/Udacity/lib/python3.7/site-packages/moviepy/video/VideoClip.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(gf, t)\u001b[0m\n\u001b[1;32m    488\u001b[0m         \"\"\"\n\u001b[1;32m    489\u001b[0m         \u001b[0mapply_to\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mapply_to\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 490\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mgf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mimage_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mapply_to\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    491\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    492\u001b[0m     \u001b[0;31m# --------------------------------------------------------------\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-10-e71cf013fbdc>\u001b[0m in \u001b[0;36mvideo_lane_detection\u001b[0;34m(img)\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m     \u001b[0;31m# Left line detection\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m     \u001b[0mbinary_wrap\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msingle_lane_detection\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mleft_line\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mundist\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfail_allowed\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     10\u001b[0m     \u001b[0;31m# Right line detection\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     11\u001b[0m     \u001b[0m_\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msingle_lane_detection\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mright_line\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mundist\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfail_allowed\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-9-a3d2334d079f>\u001b[0m in \u001b[0;36msingle_lane_detection\u001b[0;34m(line, undist, run_left, fail_limit)\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0msingle_lane_detection\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mundist\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrun_left\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfail_limit\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m     \u001b[0;31m# If the line detected previous iteration, preform search base on previous polynomial\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m     \u001b[0mbinary_wrap\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbinary_wrap_img\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mundist\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mline\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdetected\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mline\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconfident\u001b[0m \u001b[0;34m>=\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m         \u001b[0mfitx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mploty\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfit\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcurve\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msearch_around_poly\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbinary_wrap\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrun_left\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Code/Advance_lane_finding/frame_process.py\u001b[0m in \u001b[0;36mbinary_wrap_img\u001b[0;34m(undist, overdrive, debug)\u001b[0m\n\u001b[1;32m    230\u001b[0m     \u001b[0mimg_size\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mundist\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mundist\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    231\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 232\u001b[0;31m     \u001b[0mwarped\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mperspective_transform\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbinary\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    233\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mdebug\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    234\u001b[0m         \u001b[0mf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mplt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfigure\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Code/Advance_lane_finding/frame_process.py\u001b[0m in \u001b[0;36mperspective_transform\u001b[0;34m(undist, inverse, debug)\u001b[0m\n\u001b[1;32m     68\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     69\u001b[0m     \u001b[0mimg_size\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mundist\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mundist\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 70\u001b[0;31m     \u001b[0mwarped\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcv2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarpPerspective\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mundist\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mM\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mimg_size\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     71\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     72\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mdebug\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "from moviepy.editor import VideoFileClip\n",
    "from IPython.display import HTML\n",
    "\n",
    "left_line = Line()\n",
    "right_line = Line()\n",
    "\n",
    "output = 'output_video/project_video.mp4'\n",
    "## To speed up the testing process you may want to try your pipeline on a shorter subclip of the video\n",
    "## To do so add .subclip(start_second,end_second) to the end of the line below\n",
    "## Where start_second and end_second are integer values representing the start and end of the subclip\n",
    "## You may also uncomment the following line for a subclip of the first 5 seconds\n",
    "# clip2 = VideoFileClip('../challenge_video.mp4').subclip(0, 3)\n",
    "clip2 = VideoFileClip('project_video.mp4')\n",
    "project_clip = clip2.fl_image(video_lane_detection)\n",
    "%time project_clip.write_videofile(output, audio=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<video width=\"960\" height=\"540\" controls>\n",
       "  <source src=\"output_video/project_video.mp4\">\n",
       "</video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(\"\"\"\n",
    "<video width=\"960\" height=\"540\" controls>\n",
    "  <source src=\"{0}\">\n",
    "</video>\n",
    "\"\"\".format(output))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_single_lane_detection(line, undist, run_left=False, fail_limit=10):\n",
    "    # If the line detected previous iteration, preform search base on previous polynomial\n",
    "    binary_wrap = binary_wrap_img(undist)\n",
    "    if line.detected and line.confident >= 2:\n",
    "        fitx, ploty, fit, curve = search_around_poly(binary_wrap, run_left,plot=True)\n",
    "    else:\n",
    "        # If fail counter is less, preform simplify image filter and use sliding window to find lines\n",
    "        if line.fail_count < fail_limit and line.fail_count >= 0:\n",
    "            fitx, ploty, fit, curve = find_lane_pixels(binary_wrap, run_left,plot=True)\n",
    "\n",
    "        # If fail too many times, preform a more complex image filter and sliding window\n",
    "        else:\n",
    "            binary_wrap = binary_wrap_img(undist, overdrive=True)\n",
    "            # Need to find a way to share this in case of the other line failed to detect as well\n",
    "            fitx, ploty, fit, curve = find_lane_pixels(binary_wrap, run_left,plot=True)\n",
    "\n",
    "    # Calculate in meters\n",
    "    current_line_to_center = abs(fitx[-1] - binary_wrap.shape[0] / 2)\n",
    "    # if the radius is None meaning first time run, saving all the parameter in line\n",
    "    if line.radius_of_curvature is None:\n",
    "        line.detected = True\n",
    "        line.radius_of_curvature = curve\n",
    "        line.current_fit = fit\n",
    "        line.recent_xfitted = fitx\n",
    "        line.bestx = fitx\n",
    "        line.allx = [fitx]\n",
    "        line.line_to_center = current_line_to_center\n",
    "        line.ploty = ploty\n",
    "    else:\n",
    "        # Here check for current results with previous.\n",
    "        confident = 0\n",
    "        if line.radius_of_curvature * (1 + 0.2) >= curve > line.radius_of_curvature * (1 - 0.2):\n",
    "            confident += 1\n",
    "        # Check line distance to the center\n",
    "        if line.line_to_center * (1 + 0.1) >= current_line_to_center > \\\n",
    "                line.line_to_center * (1 - 0.1):\n",
    "            confident += 1\n",
    "        # Calculate fit coefficients difference\n",
    "        if np.all(abs(fit) <= abs(line.current_fit) * (1 + 0.5)) and np.all(\n",
    "                abs(fit) > abs(line.current_fit) * (1 - 0.5)):\n",
    "            confident += 1\n",
    "        # try not strict one first, can change later\n",
    "        # If the confident level is great than 1, record the result into line\n",
    "        if confident >= 1:\n",
    "            # Append result into line\n",
    "            line.detected = True\n",
    "            line.radius_of_curvature = curve\n",
    "            line.current_fit = fit\n",
    "            line.recent_xfitted = fitx\n",
    "            line.allx.append(fitx)\n",
    "            line.line_to_center = current_line_to_center\n",
    "            line.fail_count = 0\n",
    "\n",
    "            # average x fitted only takes the most recent 20\n",
    "            if len(line.allx) > 20:\n",
    "                line.bestx = np.mean(line.allx[-20:], axis=0)\n",
    "            else:\n",
    "                line.bestx = np.mean(line.allx, axis=0)\n",
    "        # else increase fail counter, discard current result\n",
    "        else:\n",
    "            line.fail_count += 1\n",
    "        line.confident = confident\n",
    "        \n",
    "    return binary_wrap\n",
    "\n",
    "\n",
    "# A function that create video shows lane finding process\n",
    "\n",
    "def plot_video_lane_detection(img):\n",
    "    global left_line, right_line\n",
    "    fail_allowed = 5\n",
    "    plt.clf()\n",
    "    # Un-distort image using Camera calibration data\n",
    "    undist = undistort_img(img)\n",
    "\n",
    "    # Left line detection\n",
    "    binary_wrap = plot_single_lane_detection(left_line, undist, True, fail_allowed)\n",
    "    left_img = cv2.imread('./test_images/out_img.png')\n",
    "    \n",
    "    # Right line detection\n",
    "    _ = plot_single_lane_detection(right_line, undist, False, fail_allowed)\n",
    "    \n",
    "    right_img = cv2.imread('./test_images/out_img.png')\n",
    "\n",
    "    result = combine_lane_images(left_img,right_img)\n",
    "\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "t:   0%|          | 0/500 [00:00<?, ?it/s, now=None]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Moviepy - Building video output_video/lane_find_2.mp4.\n",
      "Moviepy - Writing video output_video/lane_find_2.mp4\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                              \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Moviepy - Done !\n",
      "Moviepy - video ready output_video/lane_find_2.mp4\n",
      "CPU times: user 7min 38s, sys: 53 s, total: 8min 31s\n",
      "Wall time: 5min 41s\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAADKCAYAAAAGnJP4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAZUElEQVR4nO3de2wd1Z0H8O9NTOz4kfiROA4xMbFjl+AQmrQBg0iyahUqaAqbtIXA0gBtF7pQVqxUpKpdaUFardSW0pYssGyrChCPhd1Sigh06fYFLATQJiVgCoHySAIkcR7Ow3k59uwfJ8Mdj2fmztw7M+f1/UjWgO+99rmTe7/359+cOVNwHAdERJSPCbIHQERkE4YuEVGOGLpERDli6BIR5YihS0SUo6qoGwuFgrJTGwqFQuLHlDtTI+x36Trz44MPgF//Gvja12SPhLSg+Mu8F8AmADcCWBv3QcnjIxHHcUJ/g1WVblRQR93mDVfHcT7+0lFDA3DyycBbb8keCWmjoPbXP/0ncHA/8EBLzMdIpm3oxgm9oPtEVa3lVM+66eoS282b5Y6DKA0LFgBf/jLw4x8De/bIHk082oZuXEmCN+o29+foHsw9PWLLSpdMcPPNwOAg8KMfyR5JfFqHbqlq1w3ItFoBurYUvLq7xfbtt+WOg6hSixYBK1cCt90mglcXWoduJYIqVlOq2Sg9PcDWrcCRI+Y+R7LDzTeLlsJPfiJ7JMlYE7ppthl0DuXubvZzSX+LFwNf+ALwwx8C+/fLHk0yWoZuoVD4+CtMqRkG7u3en+X/eWG/Q7c2g/f59fSwtUD6u+UWYPdu4PbbZY8kOS1Dt9zQC3pcnJ+lY2UbNObmZgctLax0SW99fcAFFwDf/z5w8KDs0SSnZejKYELwugfR3JkLOj4noltuAXbuBO64Q/ZIyqNt6MaduRD2OO/turULyuWGLitd0tXixcD55wO33goMDckeTXm0DV0ZdK0M3Q+Vnh5gZAR4913JAyIq03e/K2Ys3HWX7JGUT+vQTbPaNZX3Oc6dKwJ3eNj8503mmT8fuPhiMUVMx16uS+vQjaNUsKZ9AoXKurt5Jhrp6zvfAQ4cANbGXtVGTdqFbtQUL5d/ulic4C31s3Ra5CZo6hvgoKdnfOjaUO2T/ubOBS65BLjzTmDvXtmjqYx2oZtEkuA13YwZYoUxVrqko29/Gzh2TJzyqzsjQzdsZoKNMxZc7kI3nLlAujnlFGDNGuBnPxNTxXSnXejmHZamVMj+ObpEurjpJrH9wQ/kjiMt2oVuXKx2x+rpAY4eBbZsGX+bKR8sZJ7WVuDrXwfuu08s1GQCLUM3bljGCV5bdHUB77wDjI7a99xJXzfcAFRXA9/7nuyRpCfyGmmqinNpHfc+hUIh8Qpj3p+jCv94k46vsxP4y1/SHBFRturqgOuuAx57zKy2mJahG8UNWe8SjEHfi/tzVJVsfA66uoDnnst0SKQjdV/iuBpAM4AfrEK645T8x55xoesVFLy6qmQd35YWYMoU0V4Io/v+oQoo2HGaOBH4h83A/34ErD8v4YMdKPmcXEb2dMPCydZg6ewUW7YXSBcrV4rX7a23yh5J+rQMXSB+T9N/vyTVomoH3Mp9zu4VgKMqXSKV3HST6OM+/rjskaRP29AtxdSpYeU8F7fS5epipIMlS4CzzgJuu+1OjI7KHk36jAndNNdIUH2thaRj6+wEPvwQOHw43uI/RDJ961vAwMA03HvvlbKHkgmtQ7fUNdBMrXbj8D7fOXPYWiA9fOITwEUXAXfccT0OH66VPZxMaB26XlEHz1zlnFRhAvfECCLV3XjjtThypBp33nkdlJ6CUAHtp4xFTQvzXu3XtkrXNWmSg/Z29nNJfY2Ne/CVr7TjgQf+BgMDrbKHkxkjKt2oU3z9Z6jZVO06joOODmDCBE4XI9U5+OpXf466ukNYu/YGmFrlAhpVunFO241T8Zaic0Uc1EqJmi6m83MlkziYMGEE119/B555ZgleeeWTGHsKmlkBrE2lG+eECNsXtQnCEyNIB5///Dp0dr57oso1mzahG0dU8NrUVgCKz2POHODQIWD79vD7EMkj3pc33LAWW7e247HH/tp3u3mvUa1Ct5wzsmwPFu/MBbYTSC3i9Thv3utYvvx/cNddf4fjx0+SPKbsaRW6cQQdMLPlir9Bz6+zc2w/1/R9QPr55jf/FUeOVOOnP/1b3y1mFkzKh26cq/8CwWeR+YM3zsE43UPJP/6gdXRVP+OO7DFlyj6sWXMfHnroMuzaNV32cHKhfOimcUqvrVpbgfp6ztElFYn35VVX3YP6+iErDqC5lA9dIF5wxrmahC3c53vqqeL/GbqkJgff+Ma/Yf36s7Fx4yLfbWa2FgCN5unGEXXmWZxFwE07c80N3ffflzoMIh/xHjvvvOcwb94buPrqn/tuNzdwAU0qXcC+arVS7tloAEOXVFJ8H1977d0YHJyKhx++VOJ48qd86PoPooUdAAr6fiUH4Uwwezawdy+wf7/skRCN1dy8G1/60n/h/vuv8K0mZnaVC2jQXvC3Bcq5yKStTj1VVLmmtU0ogTj/7Hm9NArFX7RmzX2oqTmKu+++1jOOlN7PisdCIerNWCgUlHmnhp1dVuqsM9PXW4iyaZOYo7tyJUPXWkpdpNH5ePv666djcLAR5577guf2GANV6vmEc5zwTxDl2wtB0l6c3NSKuaNDVLoMXJKv+BpcsuRZzJv3xtgqV4ckTYny7YU4V/Ytt8INe6wJGhvFZdffe0/2SIjGuuaaf8fg4FQ88sglsocihZaVbhymhmlcnLlAKgo+gGZPlQtoELq2h2e5GLqkjuJ7+PLLH0RNzdGAdRbsoXzoAuHBG2cthVJM7efyxAhSw9j34JVX3osNGxZi06YzT3zHzPdfFC1Cl5Lr6BDr6O7aJXskZK+xgTt//qv49Kf/z9hLq8el7YG0NB5rUuvC/1w7OhweRCOJxr+3rrzyXgwPV+HBBy+XMB51GF3pmhSqSbnTxUxtn5DKxr/vJk48jiuuuB9PPLHCs4Sjna9N5UM3zrXRymVyIHV0AFu2yB4FkfC5z/032tp2WN9aADQIXaCyitXGare21sH06cWDaCZ/uJAerrrqHgwMTMOTT1544jv2viaVDt1Si9VELXLjv1+cx5qC08VInvHvqaamPbjoosfx4IOXY3h4EmwOXEDx0C2nteA+Jk7wmsoNXR5IIxWsXv0fqK4+hnvuuQq2By6geOgClYVlVPCa/Cf3KaeI7datxe+Z/HxJFcHv1TVr7sOmTWfgT3/6ZM7jUZOyoetf1CbqBIm46+a6P8t07e3AyAjw0UeyR0K2mzPnHfT1vYgHHngVCsdNrpSep+tfBzZoDd2wdkLYz/A+xlTt7SJwR0bSXY2NKFoBxWpXvPZWrxb/9/DDUgakJGU/epL2ZkusC5zewDQwaxawbdvY79m2D0im4mtt9Wrg+ed5UNdL2dAF4gVvnJZC0M8wWXv7+NAlyt7YFcZPPx1YsAB46CF5I1KRsqHrD9Og4PX2em0KVT//c29vBz74QNJgyGJjX4eXXSaOLTzyiKThKErZ0A0SNV+XhIYGB1OmBFe6Nn8wUf4uuwz43e+AnTtlj0Qtyh5IC7vwZCUXWTT1Ao3efdXeLr7H9oKFwl7aEl7yiwF0AfiXLjm/X2VaVbpRTAzTcjB0LVdQ42v1bcCxY8CjTRn8fM0pW+kG4RoMpVsE7okRDF2SpVAALr0UeOopYHBQ9mjUo3SlG7SuQpL7J3msKdxK98MP5Y6D7HX22WLaIg+gBVM6dIPYHrylKvb2dmD7dmB4WP/nSnpatUq0Ftatkz0SNSnZXvCfAuwPS/f/o06UsHXBm1mzxq654LLl+ZN8X/wi8NvfAvv2yR6JmpSsdP0nP4SdxhtVuZq8dGMUnhhBMp15JtDZCfziF7JHoi4lQxcIDl7/OgzubXF+jgmtBVfUhwlPjCCZVq0SJ0Q8/rjskahL+faC93tBwes/cy1swXKThH2A1NU5aGpipUvyrFoFPPssMDAgeyTqUrLSjVrGsdR9Tapok5o1S2wZuiRDTw8wfz7w6KOyR6I2JUMXSBa85dzHRDwxgmRatUpsf/lLueNQnXKhG3cxci/T2gelhD1ft9LlHF2SYeVK4KWXFvNDvwTlQrdUgMYNWBur3ZkzxdYfurZ9KFH+Zsz4CGedBfzqVxfLHorylAvdUrjSWLiZM4H9+4FDh+z7wCG5LrjgKQDAunX/KHkk6lMudJPOvY1qR5ha7YY9r7Y2XheNZHCwYsUT2LYNeOUV2WNRn3Khm/Q0Xhsr3LDnPHMmQ5fyN2nSUZx//qM87Tcm5UK3HKavtxAXQ5dkWLKkBg0NwBNPyB6JHpQ7OaJUe6HS+5ts5kyx2A1RnlasAA4fFustUGnKVbppthdMrnL9z7u+3kF9PStdyt+KFcDvfy+Cl0pTLnSpPO50MYYu5amnB5g7l62FJLRpLyRtLYQ9xlRhoWvTPqD8XXih2D75pNxx6ES50E1yQcqw+0Y9xlSsdC3lf3nn/HJfDuBNAO+/l9EvMLBDqFR7Ier037D5uWFsW0+XoWspiRefnFQN/NUh4Om1Kf5c/3MykFKhm+SgWKlANfkgWpCZM4EjR3ghQMrPuecCtbXAb34jeyR6USp0gXSD13Te5188Gy340kZEaVu+HDh+HPjDH2SPRC/K9XTjzLuNEyS2BTJPjKC8LV8OrF8PHDggeyR60bLSjROotlV4J58cHrq27QvKXnMz8KlPAU8/LXsk+lEudIHoS+7EPYBmY6UbdTZa0jWKiaJ89rPAhAns55ZDydAFgmcrxAlT24LFcRzU1Ihro7G9QHlZvlwctH35ZdkjUUuc4kaZnm5Q66CcatW2CrdQKKC1VTznoErXf1VlojR85jPiANrIiOyRqCNu+1OZSjdu+8B73yA2Bktrq9ju2JHscTbuK6rcrFlAVxdnLbjczIrb1lQmdP0BUOpMMypyQ3fnzuj7+c/gs+2vAkrHsmVi+8wzcsehgiQH913KhG6QcsLVxiCJG7qAnfuH0rV0qejn2n6ViHICF1AodMNKcy7nGM1xnI9Dd2BA7ljIDsuWAc89B4yOyh6JPEnaCX7KhG5YLzfJk7Jhqpj3mnDuV2srcPDg+AtSmr4vKH+trcBpp9ndWnAD15tZUddq9FMmdJNUuu79497XJEHPe/r04NaCDfuD8rV0qdj+8Y9yxyGDt53grXS939fuQFqcSjfqQJANlV3QPmptjdfPJarUsmXir6oNG2SPJD/+sPUvGat1Tzfu/fzBm2RNBhMxdCkvS5cCzz8vFrqxgb99AIyf+5503W5lQjduYHp3gpct/dwgra08iEbZmzoVWLAAePZZQKyWbu77Legvb3/GGDF7IUjUco62VbbB+8hhpUu5WLxYbF94wV1wwbz3X5w2Z1jVG5cyoZvkihFxHmcq//5obAROOomVLmXvnHNuwehoAS+9dNaJ75hT7frD1n2f+duXcQ6aabP2QhCbq9ow/v0QdmKE6a0WypuDvr4L0d/fiwMHpsgeTOrCFthy/zvqvt7HRN3uUqbSDcJTVktLcjYaGcKR8AUHfX3rsX59X7bjUUBY4MapbuPklPaVru1hHBS6tu8TK+T6h5+D7u7NaG7eGxC6hcrH4kCp9rA3cMMyKCyU41Cm0i11skO5/RPT+PcDK13Klni99fWtB4DgStdQpXq5SU+KcClR6ZaqYqPmwdlW1YX1dHftkjAYMlzxvXXOOS9g374p+POf53luN7vgCSv6Kpm5ACgSun5BZ5rZVtHG1doK7N4NHD/O/UPZ6etbjxdfPBuOo8wfx2UrFZpxlpWtpNhTcg9WuvCNyfz7Ydo0ThejLBRfZ9XVR3DGGa/i5ZcXSxxPOsoJXP9f3JVmkfRKN27AstIN3gfTpolK18UPJ0rb/PmvoapqBBs2LJI9lLKU23t1H5NGdeslvdKtdA1dm8I4aB80N48NXaLKjX2dLVokVrfRLXSTXkYnbHZC2ksMSA/dIFEHzWxcWcwV9AHT0sLQpWwtWrQBe/c24r33TvV8V91iJyxsg9ai9vLPTMhqPRcl2wul7mdT0JbS0gLs2SN7FGSyhQs3YuPGhVA5aIHkGZFk2dg0KVnpJllwwmY1NQ5qa1npUnaqqoZx5pmvnAhddUX1bcOq2riPT5uSoeuXdiNbV/7n39Iitgxdysppp72Bmpqjyvdzo8KyVEsy73yR3l7wS+PcZlP5P60ZupSNAtyDaQsXbgSg10G0pC3LvDNFuUq31NKNbCsUNTeLrbeny/1DaVq0aAOGhmqxeXOP57t6vsbyOlBWivTQDevBlLqd4RJe6XLfUGWK77czzngV/f29GB2dKHE88YUdDwpaL0EW6aELhIdE2KV5ADvbDOzpUj6K77fe3n689tp83+16vfdUm/mkXE/X5Zb/DNxwDF3KUnPzbrS17UB/f6/nu3r9FaViVkivdEud60zhWlqAoSHg6NF4Cy0TJdHb2w8AntDV5/2o8lXCla10geDVxRgoRTwFmLJTQO+JrO3vv0DuUGLyZoV3OVjVMkN6pRu29kLQheHc20hg6FKW5s8H9u0Dtm2TPZLSdCrOpIdukLAFg1X9c0EWngJMWertBfr7ZY+itLjXMVOF0u0Flz94bQ3doJMjNm0S/636C41SlsM/dy+Ax3L6XWnQ5T2gZKUbRoU5dirxVrq2fhBZq5Dt17TpwHQA/Tdm/7sq2g0ST3Iol/TQ1akXoxYHTU3s6VI2ek6cgPbmm3LHUYqOeSE9dKN2Gqu3cPX1QFUVMDgoeyRkou5usX3rLbnjMJH00KV4/B9AjY1i6w1dfkhRWrq7gePHgffflz0S8ygRuuUuyWazoNAFGLyUju5u4N13RfBSuqTPXuDBsdKCgjQsdInSMHcuWwtZUaLSBUqvHMZQHmvqVLFl6FIWursZullRJnRL4Z/NY7mV7r59csdB5pkxA2hoAN5+W/ZIzKR06Oo4By8LQc8/qL3AfUVp4MyFbEnv6XoFnf7Lnm8wb+hy31CaurrElpVuNqRXumGL2URdTYJET3doCBgelj0SMk1HBzA6CmzdKnskZpIeukFhGueSyTYJm72wbx8/jCh9s2cD27cDx47JHomZpIduEJsDNkjQ/pg6lTMXKBuzZwNbtsgehbmkhy7bCOVpbCyGLvcZpamjg6GbJemhG9ZeoGjeSpd/GVCaZs/m6b9Zkh66fpz2FI/b0wX4IUXpmT4dqKlhpZsl6aEb1l5g8Ebzhi5RWmbPFluGbnakh27YLAXVrlWvFmdMTxdgtUvp6OgQW7YXsiM9dP3iTiGzWU0NMGkSZy9Q+trbxVaHi1HqSnroxl2+0eZqN2wtXW97web9Q+lpaxPzc3nB0+wodRqwl80XoPTz74uGBrHdv794H+9l68kgYf+kGf1TzwCwE4Azms3PJwVC19+79YcHg2Q8N3QPHhRb7iPD5Vh7tK0DtrcCWJzRL3CQ6/NRkfT2gp//QBqr3fHc0D1wQGy5jygtbW3iFGDKjjKhy2otvvp6sXVDl/uO0tLWBuzYIXsUZpPaXgir0DhdbCz/fvK2F7h/KC2FAtDa6la6/tcV/5pKi9RKN2xZR4rmbS9wv1FaWlqAqipg+/bbfbfwNZYmZdoLFM5fzbrtBfdAGoOX0tDWJrbbt/89RNC6X5Qmhq6GOHuBsjB9utgODMgdh+mkhm5Q75b93NIaGsRVI0ZHWYVQepqaxJYnRmSLla4mvB9A9fWscil9buju3St3HKZT6kAaz6oK5/0LoL6+OF2MKC3NzWLL0M0WK10NNTQwdCl9TU1i3YWhIdkjMZsyPV0APP03Qlh7gSgtTU2scvOgTKUbdCCNivwL3rDSpbQ1NzN086BUT9f7PYZvuIYGVrqUvqYmzlzIg/RKl5frSc7bXuCHE6WF7YV8SA9dhkZpQWsvsL1AaZsyZewazZQN6aHL9ReScjhPlzJRV8eZC3mQHrphGMBF3mCtrgYmTuSbg9LH0M2H9NANO+2XFVyw2lqxPXRI7jjIPLW1fF3lQXrosr2QzOTJYnv4sNxxkFmqqsQVplnpZk966FIyrHQt5GT/VTcsftXQP2f8u+I+H4MxdDXD0LVMIcZX3PtFfNXNEj9m6JrKf1YqXwaTHrpsKSTD9gJloa5ObNleyJ700OUBs2RY6VIW+LrKj/TQDcIgHs/dJ6x0KQvV1WJ79KjccdhAidBlyMbHioSyUHXiuuDHj8sdhw2UCF32dePzh667+DtRJdzQHR6WOw4bKBG6Lla8pbG9QFlgpZsfJULXcZyPA9dxHFZuEYLaC/ywoso4OOkk8V8M3ewpEbpA8c9kXictnOM4gZUuP6TslN6/e4HthQhpt/CUCF2GRny1taIa8b45+CFFlcqnvaDm67RU/nj/Ek+DEqHrxxAON3my21rgPrJdmkGQT3tBzdds3kWLUqHr7etSsNra8QfR+CGlhrz/HdL8fTyQlh8lQtd/II3CFSvdIu4zNeR9EDi9f3cH7o/i53f2lAhdim/yZFa6KtL32n5ivCMjjwAAJkx4TeZgrMDQ1czkycCRI2O/p98b3Tx6/hsUxzw6KqJgwoRRWYOxBkNXM9XV48+PZ6VLlSqG7gJEzzLQ8cNFLQxdzQSFLlGl3NCdOBGInmXAD/hKVckeACVTXQ0MDsoeBZmhGKAjI2I7gWVY5hi6mnErXT17iKSq0ROtXIZu9riLNTNpEtsLlD43dEV7gbJUYMVERJQfVrpERDli6BIR5YihS0SUI4YuEVGOGLpERDli6BIR5ej/AT7KUk1jnuGDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from moviepy.editor import VideoFileClip\n",
    "from IPython.display import HTML\n",
    "left_line = Line()\n",
    "right_line = Line()\n",
    "\n",
    "output = 'output_video/lane_find_2.mp4'\n",
    "## To speed up the testing process you may want to try your pipeline on a shorter subclip of the video\n",
    "## To do so add .subclip(start_second,end_second) to the end of the line below\n",
    "## Where start_second and end_second are integer values representing the start and end of the subclip\n",
    "## You may also uncomment the following line for a subclip of the first 5 seconds\n",
    "# clip2 = VideoFileClip('../challenge_video.mp4').subclip(0, 3)\n",
    "clip2 = VideoFileClip('project_video.mp4').subclip(20, 40)\n",
    "project_clip = clip2.fl_image(plot_video_lane_detection)\n",
    "%time project_clip.write_videofile(output, audio=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}