3 Classification/Regression_tree Taxi Tip Prediction using Scikit-Learn and Snap ML.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Taxi Tip Prediction using Scikit-Learn and Snap ML**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **30** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this exercise session you will consolidate your machine learning (ML) modeling skills by using a popular regression model: Decision Tree. You will use a real dataset to train such a model. The dataset includes information about taxi tip and was collected and provided to the NYC Taxi and Limousine Commission (TLC) by technology providers authorized under the Taxicab & Livery Passenger Enhancement Programs (TPEP/LPEP). You will use the trained model to predict the amount of tip paid. \n",
    "\n",
    "In the current exercise session, you will practice not only the Scikit-Learn Python interface, but also the Python API offered by the Snap Machine Learning (Snap ML) library. Snap ML is a high-performance IBM library for ML modeling. It provides highly-efficient CPU/GPU implementations of linear models and tree-based models. Snap ML not only accelerates ML algorithms through system awareness, but it also offers novel ML algorithms with best-in-class accuracy. For more information, please visit [snapml](https://ibm.biz/BdPfxy?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkML0101ENSkillsNetwork1047-2022-01-01) information page.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After completing this lab you will be able to:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Perform basic data preprocessing using Scikit-Learn\n",
    "* Model a regression task using the Scikit-Learn and Snap ML Python APIs\n",
    "* Train a Decision Tree Regressor model using Scikit-Learn and Snap ML\n",
    "* Run inference and assess the quality of the trained models\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\" style=\"margin-top: 10px\">\n",
    "    <ol>\n",
    "        <li><a href=\"#introduction\">Introduction</a></li>\n",
    "        <li><a href=\"#import_libraries\">Import Libraries</a></li>\n",
    "        <li><a href=\"#dataset_analysis\">Dataset Analysis</a></li>\n",
    "        <li><a href=\"#dataset_preprocessing\">Dataset Preprocessing</a></li>\n",
    "        <li><a href=\"#dataset_split\">Dataset Train/Test Split</a></li>\n",
    "        <li><a href=\"#dt_sklearn\">Build a Decision Tree Regressor model with Scikit-Learn</a></li>\n",
    "        <li><a href=\"#dt_snap\">Build a Decision Tree Regressor model with Snap ML</a></li>\n",
    "        <li><a href=\"#dt_sklearn_snap\">Evaluate the Scikit-Learn and Snap ML Decision Tree Regressors</a></li>\n",
    "    </ol>\n",
    "</div>\n",
    "<br>\n",
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"Introduction\">\n",
    "    <h2>Introduction</h2>\n",
    "    <br>The dataset used in this exercise session is publicly available here: https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page (all rights reserved by Taxi & Limousine Commission(TLC), City of New York). The TLC Yellow Taxi Trip Records of June, 2019 are used in this notebook. The prediction of the tip amount can be modeled as a regression problem. To train the model you can use part of the input dataset and the remaining data can be used to assess the quality of the trained model. First, let's download the dataset.\n",
    "    <br>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2022-11-17 13:03:53--  https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-SkillsNetwork/labs/Module%203/data/yellow_tripdata_2019-06.csv\n",
      "Resolving cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)... 169.63.118.104\n",
      "Connecting to cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)|169.63.118.104|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 364904448 (348M) [text/csv]\n",
      "Saving to: ‘yellow_tripdata_2019-06.csv.2’\n",
      "\n",
      "yellow_tripdata_201 100%[===================>] 348.00M  28.9MB/s    in 12s     \n",
      "\n",
      "2022-11-17 13:04:08 (30.1 MB/s) - ‘yellow_tripdata_2019-06.csv.2’ saved [364904448/364904448]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# download June 2020 TLC Yellow Taxi Trip records\n",
    "# !wget -nc https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2019-06.csv\n",
    "!wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-SkillsNetwork/labs/Module%203/data/yellow_tripdata_2019-06.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Did you know?__ When it comes to Machine Learning, you will most likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](https://ibm.biz/BdPfxf?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkML0101ENSkillsNetwork1047-2022-01-01)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"import_libraries\">\n",
    "    <h2>Import Libraries</h2>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: snapml in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (1.10.1)\n",
      "Requirement already satisfied: numpy>=1.18.5 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from snapml) (1.21.6)\n",
      "Requirement already satisfied: scikit-learn in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from snapml) (0.20.1)\n",
      "Requirement already satisfied: scipy in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from snapml) (1.7.3)\n"
     ]
    }
   ],
   "source": [
    "# Snap ML is available on PyPI. To install it simply run the pip command below.\n",
    "!pip install snapml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jupyterlab/conda/envs/python/lib/python3.7/site-packages/sklearn/utils/validation.py:37: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.\n",
      "  LARGE_SPARSE_SUPPORTED = LooseVersion(scipy_version) >= '0.14.0'\n"
     ]
    }
   ],
   "source": [
    "# Import the libraries we need to use in this lab\n",
    "from __future__ import print_function\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import normalize, StandardScaler, MinMaxScaler\n",
    "from sklearn.utils.class_weight import compute_sample_weight\n",
    "from sklearn.metrics import mean_squared_error\n",
    "import time\n",
    "import warnings\n",
    "import gc, sys\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"dataset_analysis\">\n",
    "    <h2>Dataset Analysis</h2>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section you will read the dataset in a Pandas dataframe and visualize its content. You will also look at some data statistics.\n",
    "\n",
    "Note: A Pandas dataframe is a two-dimensional, size-mutable, potentially heterogeneous tabular data structure. For more information: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 3936004 observations in the dataset.\n",
      "There are 18 variables in the dataset.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>VendorID</th>\n",
       "      <th>tpep_pickup_datetime</th>\n",
       "      <th>tpep_dropoff_datetime</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>trip_distance</th>\n",
       "      <th>RatecodeID</th>\n",
       "      <th>store_and_fwd_flag</th>\n",
       "      <th>PULocationID</th>\n",
       "      <th>DOLocationID</th>\n",
       "      <th>payment_type</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>extra</th>\n",
       "      <th>mta_tax</th>\n",
       "      <th>tip_amount</th>\n",
       "      <th>tolls_amount</th>\n",
       "      <th>improvement_surcharge</th>\n",
       "      <th>total_amount</th>\n",
       "      <th>congestion_surcharge</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2019-06-01 00:55:13</td>\n",
       "      <td>2019-06-01 00:56:17</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>145.0</td>\n",
       "      <td>145.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>4.30</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2019-06-01 00:06:31</td>\n",
       "      <td>2019-06-01 00:06:52</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>262.0</td>\n",
       "      <td>263.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>6.30</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>2019-06-01 00:17:05</td>\n",
       "      <td>2019-06-01 00:36:38</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>74.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>17.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>18.80</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>2019-06-01 00:59:02</td>\n",
       "      <td>2019-06-01 00:59:12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>145.0</td>\n",
       "      <td>145.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>4.30</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2019-06-01 00:03:25</td>\n",
       "      <td>2019-06-01 00:15:42</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.7</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>113.0</td>\n",
       "      <td>148.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>2.65</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>15.95</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   VendorID tpep_pickup_datetime tpep_dropoff_datetime  passenger_count  \\\n",
       "0         1  2019-06-01 00:55:13   2019-06-01 00:56:17              1.0   \n",
       "1         1  2019-06-01 00:06:31   2019-06-01 00:06:52              1.0   \n",
       "2         1  2019-06-01 00:17:05   2019-06-01 00:36:38              1.0   \n",
       "3         1  2019-06-01 00:59:02   2019-06-01 00:59:12              0.0   \n",
       "4         1  2019-06-01 00:03:25   2019-06-01 00:15:42              1.0   \n",
       "\n",
       "   trip_distance  RatecodeID store_and_fwd_flag  PULocationID  DOLocationID  \\\n",
       "0            0.0         1.0                  N         145.0         145.0   \n",
       "1            0.0         1.0                  N         262.0         263.0   \n",
       "2            4.4         1.0                  N          74.0           7.0   \n",
       "3            0.8         1.0                  N         145.0         145.0   \n",
       "4            1.7         1.0                  N         113.0         148.0   \n",
       "\n",
       "   payment_type  fare_amount  extra  mta_tax  tip_amount  tolls_amount  \\\n",
       "0           2.0          3.0    0.5      0.5        0.00           0.0   \n",
       "1           2.0          2.5    3.0      0.5        0.00           0.0   \n",
       "2           2.0         17.5    0.5      0.5        0.00           0.0   \n",
       "3           2.0          2.5    1.0      0.5        0.00           0.0   \n",
       "4           1.0          9.5    3.0      0.5        2.65           0.0   \n",
       "\n",
       "   improvement_surcharge  total_amount  congestion_surcharge  \n",
       "0                    0.3          4.30                   0.0  \n",
       "1                    0.3          6.30                   2.5  \n",
       "2                    0.3         18.80                   0.0  \n",
       "3                    0.3          4.30                   0.0  \n",
       "4                    0.3         15.95                   2.5  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read the input data\n",
    "raw_data = pd.read_csv('yellow_tripdata_2019-06.csv')\n",
    "print(\"There are \" + str(len(raw_data)) + \" observations in the dataset.\")\n",
    "print(\"There are \" + str(len(raw_data.columns)) + \" variables in the dataset.\")\n",
    "\n",
    "# display first rows in the dataset\n",
    "raw_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each row in the dataset represents a taxi trip. As shown above, each row has 18 variables. One variable is called tip_amount and represents the target variable. Your objective will be to train a model that uses the other variables to predict the value of the tip_amount variable. Let's first clean the dataset and retrieve basic statistics about the target variable.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 2712719 observations in the dataset.\n",
      "There are 17 variables in the dataset.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGsCAYAAAAPJKchAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAf7ElEQVR4nO3df2xV9f3H8de1hVvE9jrK+ksKFKPIKCIWpiggDFNGHYaMOXVT0M0/uvC7YYOCCcOpdQkaQiYQHD9CUCGmwKoQQ53Q+gPHKK2iAqJ2lJU2XXXeC6j9QT/fPwz3610L9JbSN733+UjOH/fcz7n3cz6S9um95/Z6nHNOAAAARq6yngAAAIhuxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMNWtYqS0tFRTpkxRWlqaPB6PduzYEfZjOOe0fPly3XjjjfJ6vUpPT9fTTz/d+ZMFAADtEms9gXCcOXNGw4cP16OPPqpp06Z16DHmzp2r3bt3a/ny5Ro2bJj8fr/q6+s7eaYAAKC9PN31i/I8Ho+2b9+uqVOnBvc1Njbq8ccf14svvqivvvpKmZmZ+vOf/6zx48dLkg4fPqybb75ZH374oQYPHmwzcQAAEKJbvU1zMY8++qjeeecdbdmyRR988IHuu+8+/fSnP9WxY8ckSa+++qoGDRqk1157TRkZGRo4cKAee+wxffnll8YzBwAgekVMjHz22Wd6+eWX9corr2js2LG6/vrrtWDBAo0ZM0YbNmyQJH3++ec6fvy4XnnlFW3atEkbN25UWVmZfvGLXxjPHgCA6NWtrhm5kIMHD8o5pxtvvDFkf0NDgxITEyVJLS0tamho0KZNm4Lj1q1bp6ysLB09epS3bgAAMBAxMdLS0qKYmBiVlZUpJiYm5L5rrrlGkpSamqrY2NiQYBkyZIgkqaqqihgBAMBAxMTIiBEjdPbsWdXV1Wns2LFtjrnzzjvV3Nyszz77TNdff70k6ZNPPpEkDRgwoMvmCgAA/l+3+jTN6dOn9emnn0r6Lj6ee+45TZgwQX369FH//v310EMP6Z133tGzzz6rESNGqL6+Xm+++aaGDRumnJwctbS0aNSoUbrmmmu0YsUKtbS0aObMmUpISNDu3buNzw4AgOjUrWJk7969mjBhQqv9M2bM0MaNG9XU1KQnn3xSmzZtUnV1tRITEzV69GgtW7ZMw4YNkySdPHlSs2fP1u7du9W7d29NnjxZzz77rPr06dPVpwMAANTNYgQAAESeiPloLwAA6J6IEQAAYKpbfJqmpaVFJ0+eVHx8vDwej/V0AABAOzjndOrUKaWlpemqq87/+ke3iJGTJ08qPT3dehoAAKADTpw4oX79+p33/m4RI/Hx8ZK+O5mEhATj2QAAgPYIBAJKT08P/h4/n24RI+femklISCBGAADoZi52iQUXsAIAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMBVrPQFrnmUX/lrjy8EtdV3+nAAAXKl4ZQQAAJgiRgAAgKmwYqSgoECjRo1SfHy8kpKSNHXqVB09evSCx+zdu1cej6fVduTIkUuaOAAAiAxhxUhJSYlmzpyp9957T8XFxWpublZ2drbOnDlz0WOPHj2qmpqa4HbDDTd0eNIAACByhHUB6+uvvx5ye8OGDUpKSlJZWZnGjRt3wWOTkpJ07bXXhj1BAAAQ2S7pmhG/3y9J6tOnz0XHjhgxQqmpqZo4caL27NlzwbENDQ0KBAIhGwAAiEwdjhHnnPLy8jRmzBhlZmaed1xqaqrWrl2rwsJCbdu2TYMHD9bEiRNVWlp63mMKCgrk8/mCW3p6ekenCQAArnAe51yH/ujFzJkztXPnTr399tvq169fWMdOmTJFHo9HRUVFbd7f0NCghoaG4O1AIKD09HT5/X4lJCR0ZLrnxd8ZAQDg8ggEAvL5fBf9/d2hV0Zmz56toqIi7dmzJ+wQkaTbb79dx44dO+/9Xq9XCQkJIRsAAIhMYV3A6pzT7NmztX37du3du1cZGRkdetLy8nKlpqZ26FgAABBZwoqRmTNn6qWXXtLf/vY3xcfHq7a2VpLk8/nUq1cvSVJ+fr6qq6u1adMmSdKKFSs0cOBADR06VI2Njdq8ebMKCwtVWFjYyacCAAC6o7BiZPXq1ZKk8ePHh+zfsGGDHnnkEUlSTU2Nqqqqgvc1NjZqwYIFqq6uVq9evTR06FDt3LlTOTk5lzZzAAAQETp8AWtXau8FMB3BBawAAFwel/UCVgAAgM5CjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU2HFSEFBgUaNGqX4+HglJSVp6tSpOnr06EWPKykpUVZWluLi4jRo0CCtWbOmwxMGAACRJawYKSkp0cyZM/Xee++puLhYzc3Nys7O1pkzZ857TGVlpXJycjR27FiVl5dr8eLFmjNnjgoLCy958gAAoPvzOOdcRw/+z3/+o6SkJJWUlGjcuHFtjlm4cKGKiop0+PDh4L7c3Fy9//772rdvX7ueJxAIyOfzye/3KyEhoaPTbZNnmadTH6893NIOLzkAAN1Ge39/X9I1I36/X5LUp0+f847Zt2+fsrOzQ/ZNmjRJBw4cUFNTU5vHNDQ0KBAIhGwAACAydThGnHPKy8vTmDFjlJmZed5xtbW1Sk5ODtmXnJys5uZm1dfXt3lMQUGBfD5fcEtPT+/oNAEAwBWuwzEya9YsffDBB3r55ZcvOtbjCX0r5Nw7Q/+7/5z8/Hz5/f7gduLEiY5OEwAAXOFiO3LQ7NmzVVRUpNLSUvXr1++CY1NSUlRbWxuyr66uTrGxsUpMTGzzGK/XK6/X25GpAQCAbiasV0acc5o1a5a2bdumN998UxkZGRc9ZvTo0SouLg7Zt3v3bo0cOVI9evQIb7YAACDihBUjM2fO1ObNm/XSSy8pPj5etbW1qq2t1TfffBMck5+fr+nTpwdv5+bm6vjx48rLy9Phw4e1fv16rVu3TgsWLOi8swAAAN1WWDGyevVq+f1+jR8/XqmpqcFt69atwTE1NTWqqqoK3s7IyNCuXbu0d+9e3XLLLfrTn/6klStXatq0aZ13FgAAoNu6pL8z0lX4OyMAAHQ/XfJ3RgAAAC4VMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATIUdI6WlpZoyZYrS0tLk8Xi0Y8eOC47fu3evPB5Pq+3IkSMdnTMAAIggseEecObMGQ0fPlyPPvqopk2b1u7jjh49qoSEhODtH/7wh+E+NQAAiEBhx8jkyZM1efLksJ8oKSlJ1157bdjHAQCAyNZl14yMGDFCqampmjhxovbs2XPBsQ0NDQoEAiEbAACITJc9RlJTU7V27VoVFhZq27ZtGjx4sCZOnKjS0tLzHlNQUCCfzxfc0tPTL/c0AQCAEY9zznX4YI9H27dv19SpU8M6bsqUKfJ4PCoqKmrz/oaGBjU0NARvBwIBpaeny+/3h1x30hk8yzyd+njt4ZZ2eMkBAOg2AoGAfD7fRX9/m3y09/bbb9exY8fOe7/X61VCQkLIBgAAIpNJjJSXlys1NdXiqQEAwBUm7E/TnD59Wp9++mnwdmVlpSoqKtSnTx/1799f+fn5qq6u1qZNmyRJK1as0MCBAzV06FA1NjZq8+bNKiwsVGFhYeedBQAA6LbCjpEDBw5owoQJwdt5eXmSpBkzZmjjxo2qqalRVVVV8P7GxkYtWLBA1dXV6tWrl4YOHaqdO3cqJyenE6YPAAC6u0u6gLWrtPcCmI7gAlYAAC6PK/oCVgAAgHOIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgKuwYKS0t1ZQpU5SWliaPx6MdO3Zc9JiSkhJlZWUpLi5OgwYN0po1azoyVwAAEIHCjpEzZ85o+PDh+stf/tKu8ZWVlcrJydHYsWNVXl6uxYsXa86cOSosLAx7sgAAIPLEhnvA5MmTNXny5HaPX7Nmjfr3768VK1ZIkoYMGaIDBw5o+fLlmjZtWrhPDwAAIsxlv2Zk3759ys7ODtk3adIkHThwQE1NTW0e09DQoEAgELIBAIDIdNljpLa2VsnJySH7kpOT1dzcrPr6+jaPKSgokM/nC27p6emXe5oAAMBIl3yaxuPxhNx2zrW5/5z8/Hz5/f7gduLEics+RwAAYCPsa0bClZKSotra2pB9dXV1io2NVWJiYpvHeL1eeb3eyz01AABwBbjsr4yMHj1axcXFIft2796tkSNHqkePHpf76QEAwBUu7Bg5ffq0KioqVFFRIem7j+5WVFSoqqpK0ndvsUyfPj04Pjc3V8ePH1deXp4OHz6s9evXa926dVqwYEHnnAEAAOjWwn6b5sCBA5owYULwdl5eniRpxowZ2rhxo2pqaoJhIkkZGRnatWuX5s+fr+eff15paWlauXIlH+sFAACSJI87dzXpFSwQCMjn88nv9yshIaFTH9uzrO2LaC8nt/SKX3IAAC5Ze39/8900AADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEx1KEZWrVqljIwMxcXFKSsrS2+99dZ5x+7du1cej6fVduTIkQ5PGgAARI6wY2Tr1q2aN2+elixZovLyco0dO1aTJ09WVVXVBY87evSoampqgtsNN9zQ4UkDAIDIEXaMPPfcc/rtb3+rxx57TEOGDNGKFSuUnp6u1atXX/C4pKQkpaSkBLeYmJgOTxoAAESOsGKksbFRZWVlys7ODtmfnZ2td99994LHjhgxQqmpqZo4caL27NlzwbENDQ0KBAIhGwAAiExhxUh9fb3Onj2r5OTkkP3Jycmqra1t85jU1FStXbtWhYWF2rZtmwYPHqyJEyeqtLT0vM9TUFAgn88X3NLT08OZJgAA6EZiO3KQx+MJue2ca7XvnMGDB2vw4MHB26NHj9aJEye0fPlyjRs3rs1j8vPzlZeXF7wdCAQIEgAAIlRYr4z07dtXMTExrV4Fqaura/VqyYXcfvvtOnbs2Hnv93q9SkhICNkAAEBkCitGevbsqaysLBUXF4fsLy4u1h133NHuxykvL1dqamo4Tw0AACJU2G/T5OXl6eGHH9bIkSM1evRorV27VlVVVcrNzZX03Vss1dXV2rRpkyRpxYoVGjhwoIYOHarGxkZt3rxZhYWFKiws7NwzAQAA3VLYMXL//ffriy++0BNPPKGamhplZmZq165dGjBggCSppqYm5G+ONDY2asGCBaqurlavXr00dOhQ7dy5Uzk5OZ13FgAAoNvyOOec9SQuJhAIyOfzye/3d/r1I55lbV94ezm5pVf8kgMAcMna+/ub76YBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApmKtJxCNPMs8Xfp8bqnr0ucDACAcvDICAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwFSHYmTVqlXKyMhQXFycsrKy9NZbb11wfElJibKyshQXF6dBgwZpzZo1HZosAACIPGHHyNatWzVv3jwtWbJE5eXlGjt2rCZPnqyqqqo2x1dWVionJ0djx45VeXm5Fi9erDlz5qiwsPCSJw8AALo/j3POhXPAbbfdpltvvVWrV68O7hsyZIimTp2qgoKCVuMXLlyooqIiHT58OLgvNzdX77//vvbt29eu5wwEAvL5fPL7/UpISAhnuhflWebp1Me7ErmlYf0nBgCgU7T393dsOA/a2NiosrIyLVq0KGR/dna23n333TaP2bdvn7Kzs0P2TZo0SevWrVNTU5N69OjR6piGhgY1NDQEb/v9fknfnVSn+7bzH/JK48nv2uDy5/u79PkAAFemc7+3L/a6R1gxUl9fr7Nnzyo5OTlkf3Jysmpra9s8pra2ts3xzc3Nqq+vV2pqaqtjCgoKtGzZslb709PTw5kujPie8VlPAQBwBTl16pR8vvP/bggrRs7xeEL/T9s512rfxca3tf+c/Px85eXlBW+3tLToyy+/VGJi4gWfJ1yBQEDp6ek6ceJEp7/9052xLq2xJq2xJm1jXVpjTdoWDevinNOpU6eUlpZ2wXFhxUjfvn0VExPT6lWQurq6Vq9+nJOSktLm+NjYWCUmJrZ5jNfrldfrDdl37bXXhjPVsCQkJETsP4RLwbq0xpq0xpq0jXVpjTVpW6Svy4VeETknrE/T9OzZU1lZWSouLg7ZX1xcrDvuuKPNY0aPHt1q/O7duzVy5Mg2rxcBAADRJeyP9ubl5emvf/2r1q9fr8OHD2v+/PmqqqpSbm6upO/eYpk+fXpwfG5uro4fP668vDwdPnxY69ev17p167RgwYLOOwsAANBthX3NyP33368vvvhCTzzxhGpqapSZmaldu3ZpwIABkqSampqQvzmSkZGhXbt2af78+Xr++eeVlpamlStXatq0aZ13Fh3k9Xq1dOnSVm8JRTvWpTXWpDXWpG2sS2usSdtYl/8X9t8ZAQAA6Ex8Nw0AADBFjAAAAFPECAAAMEWMAAAAU1EdI6tWrVJGRobi4uKUlZWlt956y3pKXaagoECjRo1SfHy8kpKSNHXqVB09ejRkjHNOf/zjH5WWlqZevXpp/Pjx+uijj4xm3PUKCgrk8Xg0b9684L5oXZPq6mo99NBDSkxM1NVXX61bbrlFZWVlwfujbV2am5v1+OOPKyMjQ7169dKgQYP0xBNPqKWlJTgm0tektLRUU6ZMUVpamjwej3bs2BFyf3vOv6GhQbNnz1bfvn3Vu3dv3Xvvvfr3v//dhWfR+S60Lk1NTVq4cKGGDRum3r17Ky0tTdOnT9fJkydDHiMS1+WiXJTasmWL69Gjh3vhhRfcxx9/7ObOnet69+7tjh8/bj21LjFp0iS3YcMG9+GHH7qKigp3zz33uP79+7vTp08HxzzzzDMuPj7eFRYWukOHDrn777/fpaamukAgYDjzrrF//343cOBAd/PNN7u5c+cG90fjmnz55ZduwIAB7pFHHnH/+Mc/XGVlpXvjjTfcp59+GhwTbevy5JNPusTERPfaa6+5yspK98orr7hrrrnGrVixIjgm0tdk165dbsmSJa6wsNBJctu3bw+5vz3nn5ub66677jpXXFzsDh486CZMmOCGDx/umpubu/hsOs+F1uWrr75yd999t9u6das7cuSI27dvn7vttttcVlZWyGNE4rpcTNTGyI9//GOXm5sbsu+mm25yixYtMpqRrbq6OifJlZSUOOeca2lpcSkpKe6ZZ54Jjvn222+dz+dza9assZpmlzh16pS74YYbXHFxsbvrrruCMRKta7Jw4UI3ZsyY894fjetyzz33uN/85jch+37+85+7hx56yDkXfWvyv79023P+X331levRo4fbsmVLcEx1dbW76qqr3Ouvv95lc7+c2oq0/7V//34nKfg/wtGwLm2JyrdpGhsbVVZWpuzs7JD92dnZevfdd41mZcvv90uS+vTpI0mqrKxUbW1tyBp5vV7dddddEb9GM2fO1D333KO77747ZH+0rklRUZFGjhyp++67T0lJSRoxYoReeOGF4P3RuC5jxozR3//+d33yySeSpPfff19vv/22cnJyJEXnmnxfe86/rKxMTU1NIWPS0tKUmZkZFWt0jt/vl8fjCX7/WrSuS4e+tbe7q6+v19mzZ1t9uV9ycnKrL/WLBs455eXlacyYMcrMzJSk4Dq0tUbHjx/v8jl2lS1btujgwYP65z//2eq+aF2Tzz//XKtXr1ZeXp4WL16s/fv3a86cOfJ6vZo+fXpUrsvChQvl9/t10003KSYmRmfPntVTTz2lBx98UFL0/ls5pz3nX1tbq549e+oHP/hBqzHR8nP422+/1aJFi/SrX/0q+EV50bouURkj53g8npDbzrlW+6LBrFmz9MEHH+jtt99udV80rdGJEyc0d+5c7d69W3FxcecdF01rIkktLS0aOXKknn76aUnSiBEj9NFHH2n16tUh30MVTeuydetWbd68WS+99JKGDh2qiooKzZs3T2lpaZoxY0ZwXDStSVs6cv7RskZNTU164IEH1NLSolWrVl10fKSvS1S+TdO3b1/FxMS0qsy6urpWJR/pZs+eraKiIu3Zs0f9+vUL7k9JSZGkqFqjsrIy1dXVKSsrS7GxsYqNjVVJSYlWrlyp2NjY4HlH05pIUmpqqn70ox+F7BsyZEjwO6ii8d/K73//ey1atEgPPPCAhg0bpocffljz589XQUGBpOhck+9rz/mnpKSosbFR//3vf887JlI1NTXpl7/8pSorK1VcXBx8VUSK3nWJyhjp2bOnsrKyVFxcHLK/uLhYd9xxh9GsupZzTrNmzdK2bdv05ptvKiMjI+T+jIwMpaSkhKxRY2OjSkpKInaNJk6cqEOHDqmioiK4jRw5Ur/+9a9VUVGhQYMGRd2aSNKdd97Z6mPfn3zySfDLMaPx38rXX3+tq64K/fEZExMT/GhvNK7J97Xn/LOystSjR4+QMTU1Nfrwww8jeo3OhcixY8f0xhtvKDExMeT+aF2XqP00zbmP9q5bt859/PHHbt68ea53797uX//6l/XUusTvfvc75/P53N69e11NTU1w+/rrr4NjnnnmGefz+dy2bdvcoUOH3IMPPhhRH01sj+9/msa56FyT/fv3u9jYWPfUU0+5Y8eOuRdffNFdffXVbvPmzcEx0bYuM2bMcNddd13wo73btm1zffv2dX/4wx+CYyJ9TU6dOuXKy8tdeXm5k+See+45V15eHvxUSHvOPzc31/Xr18+98cYb7uDBg+4nP/lJt/8I64XWpampyd17772uX79+rqKiIuRnb0NDQ/AxInFdLiZqY8Q5555//nk3YMAA17NnT3frrbcGP9YaDSS1uW3YsCE4pqWlxS1dutSlpKQ4r9frxo0b5w4dOmQ3aQP/GyPRuiavvvqqy8zMdF6v1910001u7dq1IfdH27oEAgE3d+5c179/fxcXF+cGDRrklixZEvILJdLXZM+ePW3+DJkxY4Zzrn3n/80337hZs2a5Pn36uF69ermf/exnrqqqyuBsOs+F1qWysvK8P3v37NkTfIxIXJeL8TjnXNe9DgMAABAqKq8ZAQAAVw5iBAAAmCJGAACAKWIEAACYIkYAAIApYgQAAJgiRgAAgCliBAAAmCJGAACAKWIEAACYIkYAAIApYgQAAJj6P2/cF4d9XAhvAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minimum amount value is  0.01\n",
      "Maximum amount value is  130.0\n",
      "90% of the trips have a tip amount less or equal than  5.95\n"
     ]
    }
   ],
   "source": [
    "# some trips report 0 tip. it is assumed that these tips were paid in cash.\n",
    "# for this study we drop all these rows\n",
    "raw_data = raw_data[raw_data['tip_amount'] > 0]\n",
    "\n",
    "# we also remove some outliers, namely those where the tip was larger than the fare cost\n",
    "raw_data = raw_data[(raw_data['tip_amount'] <= raw_data['fare_amount'])]\n",
    "\n",
    "# we remove trips with very large fare cost\n",
    "raw_data = raw_data[((raw_data['fare_amount'] >=2) & (raw_data['fare_amount'] < 200))]\n",
    "\n",
    "# we drop variables that include the target variable in it, namely the total_amount\n",
    "clean_data = raw_data.drop(['total_amount'], axis=1)\n",
    "\n",
    "# release memory occupied by raw_data as we do not need it anymore\n",
    "# we are dealing with a large dataset, thus we need to make sure we do not run out of memory\n",
    "del raw_data\n",
    "gc.collect()\n",
    "\n",
    "# print the number of trips left in the dataset\n",
    "print(\"There are \" + str(len(clean_data)) + \" observations in the dataset.\")\n",
    "print(\"There are \" + str(len(clean_data.columns)) + \" variables in the dataset.\")\n",
    "\n",
    "plt.hist(clean_data.tip_amount.values, 16, histtype='bar', facecolor='g')\n",
    "plt.show()\n",
    "\n",
    "print(\"Minimum amount value is \", np.min(clean_data.tip_amount.values))\n",
    "print(\"Maximum amount value is \", np.max(clean_data.tip_amount.values))\n",
    "print(\"90% of the trips have a tip amount less or equal than \", np.percentile(clean_data.tip_amount.values, 90))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>VendorID</th>\n",
       "      <th>tpep_pickup_datetime</th>\n",
       "      <th>tpep_dropoff_datetime</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>trip_distance</th>\n",
       "      <th>RatecodeID</th>\n",
       "      <th>store_and_fwd_flag</th>\n",
       "      <th>PULocationID</th>\n",
       "      <th>DOLocationID</th>\n",
       "      <th>payment_type</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>extra</th>\n",
       "      <th>mta_tax</th>\n",
       "      <th>tip_amount</th>\n",
       "      <th>tolls_amount</th>\n",
       "      <th>improvement_surcharge</th>\n",
       "      <th>congestion_surcharge</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2019-06-01 00:03:25</td>\n",
       "      <td>2019-06-01 00:15:42</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.70</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>113.0</td>\n",
       "      <td>148.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>2.65</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>2019-06-01 00:28:31</td>\n",
       "      <td>2019-06-01 00:39:23</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.60</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>79.0</td>\n",
       "      <td>125.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1</td>\n",
       "      <td>2019-06-01 00:54:49</td>\n",
       "      <td>2019-06-01 01:02:57</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.20</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>79.0</td>\n",
       "      <td>249.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>7.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>2019-06-01 00:29:12</td>\n",
       "      <td>2019-06-01 01:03:13</td>\n",
       "      <td>1.0</td>\n",
       "      <td>8.60</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>186.0</td>\n",
       "      <td>243.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>31.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>7.05</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2</td>\n",
       "      <td>2019-06-01 00:01:48</td>\n",
       "      <td>2019-06-01 00:16:06</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.74</td>\n",
       "      <td>1.0</td>\n",
       "      <td>N</td>\n",
       "      <td>107.0</td>\n",
       "      <td>148.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>2.96</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    VendorID tpep_pickup_datetime tpep_dropoff_datetime  passenger_count  \\\n",
       "4          1  2019-06-01 00:03:25   2019-06-01 00:15:42              1.0   \n",
       "5          1  2019-06-01 00:28:31   2019-06-01 00:39:23              2.0   \n",
       "7          1  2019-06-01 00:54:49   2019-06-01 01:02:57              2.0   \n",
       "9          1  2019-06-01 00:29:12   2019-06-01 01:03:13              1.0   \n",
       "10         2  2019-06-01 00:01:48   2019-06-01 00:16:06              1.0   \n",
       "\n",
       "    trip_distance  RatecodeID store_and_fwd_flag  PULocationID  DOLocationID  \\\n",
       "4            1.70         1.0                  N         113.0         148.0   \n",
       "5            1.60         1.0                  N          79.0         125.0   \n",
       "7            1.20         1.0                  N          79.0         249.0   \n",
       "9            8.60         1.0                  N         186.0         243.0   \n",
       "10           1.74         1.0                  N         107.0         148.0   \n",
       "\n",
       "    payment_type  fare_amount  extra  mta_tax  tip_amount  tolls_amount  \\\n",
       "4            1.0          9.5    3.0      0.5        2.65           0.0   \n",
       "5            1.0          9.5    3.0      0.5        1.00           0.0   \n",
       "7            1.0          7.5    3.0      0.5        1.00           0.0   \n",
       "9            1.0         31.5    3.0      0.5        7.05           0.0   \n",
       "10           1.0         11.0    0.5      0.5        2.96           0.0   \n",
       "\n",
       "    improvement_surcharge  congestion_surcharge  \n",
       "4                     0.3                   2.5  \n",
       "5                     0.3                   2.5  \n",
       "7                     0.3                   2.5  \n",
       "9                     0.3                   2.5  \n",
       "10                    0.3                   2.5  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# display first rows in the dataset\n",
    "clean_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By looking at the dataset in more detail, we see that it contains information such as pick-up and drop-off dates/times, pick-up and drop-off locations, payment types, driver-reported passenger counts etc. Before actually training a ML model, we will need to preprocess the data. We need to transform the data in a format that will be correctly handled by the models. For instance, we need to encode the categorical features.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"dataset_preprocessing\">\n",
    "    <h2>Dataset Preprocessing</h2>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this subsection you will prepare the data for training. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_data['tpep_dropoff_datetime'] = pd.to_datetime(clean_data['tpep_dropoff_datetime'])\n",
    "clean_data['tpep_pickup_datetime'] = pd.to_datetime(clean_data['tpep_pickup_datetime'])\n",
    "\n",
    "# extract pickup and dropoff hour\n",
    "clean_data['pickup_hour'] = clean_data['tpep_pickup_datetime'].dt.hour\n",
    "clean_data['dropoff_hour'] = clean_data['tpep_dropoff_datetime'].dt.hour\n",
    "\n",
    "# extract pickup and dropoff day of week\n",
    "clean_data['pickup_day'] = clean_data['tpep_pickup_datetime'].dt.weekday\n",
    "clean_data['dropoff_day'] = clean_data['tpep_dropoff_datetime'].dt.weekday\n",
    "\n",
    "# compute trip time in minutes\n",
    "clean_data['trip_time'] = (clean_data['tpep_dropoff_datetime'] - clean_data['tpep_pickup_datetime']).astype('timedelta64[m]')\n",
    "\n",
    "# ideally use the full dataset for this exercise\n",
    "# however if you run into out of memory issues due to the data size, reduce it\n",
    "# for instance, in this example we use only the first 1M samples\n",
    "first_n_rows = 1000000\n",
    "clean_data = clean_data.head(first_n_rows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3811"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# drop the pickup and dropoff datetimes\n",
    "clean_data = clean_data.drop(['tpep_pickup_datetime', 'tpep_dropoff_datetime'], axis=1)\n",
    "\n",
    "# some features are categorical, we need to encode them\n",
    "# to encode them we use one-hot encoding from the Pandas package\n",
    "get_dummy_col = [\"VendorID\",\"RatecodeID\",\"store_and_fwd_flag\",\"PULocationID\", \"DOLocationID\",\"payment_type\", \"pickup_hour\", \"dropoff_hour\", \"pickup_day\", \"dropoff_day\"]\n",
    "proc_data = pd.get_dummies(clean_data, columns = get_dummy_col)\n",
    "\n",
    "# release memory occupied by clean_data as we do not need it anymore\n",
    "# we are dealing with a large dataset, thus we need to make sure we do not run out of memory\n",
    "del clean_data\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extract the labels from the dataframe\n",
    "y = proc_data[['tip_amount']].values.astype('float32')\n",
    "\n",
    "# drop the target variable from the feature matrixn and get the feature matrix used for trainin\n",
    "X = proc_data.drop(['tip_amount'], axis=1).values\n",
    "\n",
    "#X = proc_data.values\n",
    "\n",
    "# normalize the feature matrix\n",
    "X = normalize(X, axis=1, norm='l1', copy=False)\n",
    "\n",
    "# print the shape of the features matrix and the labels vector\n",
    "print('X.shape=', X.shape, 'y.shape=', y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"dataset_split\">\n",
    "    <h2>Dataset Train/Test Split</h2>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the dataset is ready for building the classification models, you need to first divide the pre-processed dataset into a subset to be used for training the model (the train set) and a subset to be used for evaluating the quality of the model (the test set).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n",
    "print('X_train.shape=', X_train.shape, 'Y_train.shape=', y_train.shape)\n",
    "print('X_test.shape=', X_test.shape, 'Y_test.shape=', y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"dt_sklearn\">\n",
    "    <h2>Build a Decision Tree Regressor model with Scikit-Learn</h2>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the Decision Tree Regression Model from scikit-learn\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "\n",
    "# for reproducible output across multiple function calls, set random_state to a given integer value\n",
    "sklearn_dt = DecisionTreeRegressor(max_depth=8, random_state=35)\n",
    "\n",
    "# train a Decision Tree Regressor using scikit-learn\n",
    "t0 = time.time()\n",
    "sklearn_dt.fit(X_train, y_train)\n",
    "sklearn_time = time.time()-t0\n",
    "print(\"[Scikit-Learn] Training time (s):  {0:.5f}\".format(sklearn_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"dt_snapml\">\n",
    "    <h2>Build a Decision Tree Regressor model with Snap ML</h2>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the Decision Tree Regressor Model from Snap ML\n",
    "from snapml import DecisionTreeRegressor\n",
    "\n",
    "# in contrast to sklearn's Decision Tree, Snap ML offers multi-threaded CPU/GPU training \n",
    "# to use the GPU, one needs to set the use_gpu parameter to True\n",
    "# snapml_dt = DecisionTreeRegressor(max_depth=4, random_state=45, use_gpu=True)\n",
    "\n",
    "# to set the number of CPU threads used at training time, one needs to set the n_jobs parameter\n",
    "# for reproducible output across multiple function calls, set random_state to a given integer value\n",
    "snapml_dt = DecisionTreeRegressor(max_depth=8, random_state=45, n_jobs=4)\n",
    "\n",
    "# train a Decision Tree Regressor model using Snap ML\n",
    "t0 = time.time()\n",
    "snapml_dt.fit(X_train, y_train)\n",
    "snapml_time = time.time()-t0\n",
    "print(\"[Snap ML] Training time (s):  {0:.5f}\".format(snapml_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"dt_sklearn_snapml\">\n",
    "    <h2>Evaluate the Scikit-Learn and Snap ML Decision Tree Regressor Models</h2>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Snap ML vs Scikit-Learn training speedup\n",
    "training_speedup = sklearn_time/snapml_time\n",
    "print('[Decision Tree Regressor] Snap ML vs. Scikit-Learn speedup : {0:.2f}x '.format(training_speedup))\n",
    "\n",
    "# run inference using the sklearn model\n",
    "sklearn_pred = sklearn_dt.predict(X_test)\n",
    "\n",
    "# evaluate mean squared error on the test dataset\n",
    "sklearn_mse = mean_squared_error(y_test, sklearn_pred)\n",
    "print('[Scikit-Learn] MSE score : {0:.3f}'.format(sklearn_mse))\n",
    "\n",
    "# run inference using the Snap ML model\n",
    "snapml_pred = snapml_dt.predict(X_test)\n",
    "\n",
    "# evaluate mean squared error on the test dataset\n",
    "snapml_mse = mean_squared_error(y_test, snapml_pred)\n",
    "print('[Snap ML] MSE score : {0:.3f}'.format(snapml_mse))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown above both decision tree models provide the same score on the test dataset. However Snap ML runs the training routine faster than Scikit-Learn. This is one of the advantages of using Snap ML: acceleration of training of classical machine learning models, such as linear and tree-based models. For more Snap ML examples, please visit [snapml-examples](https://ibm.biz/BdPfxP?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkML0101ENSkillsNetwork1047-2022-01-01). Moreover, as shown above, not only is Snap ML seemlessly accelerating scikit-learn applications, but the library's Python API is also compatible with scikit-learn metrics and data preprocessors.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Practice\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Andreea Anghel\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Contributors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sangeeth Keeriyadath \n",
    "\n",
    "Joseph Santarcangelo\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|  Date (YYYY-MM-DD) |  Version | Changed By  |  Change Description |\n",
    "|---|---|---|---|\n",
    "| 2021-08-31  | 0.1  | AAN  |  Created Lab Content |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Copyright &copy; 2021 IBM Corporation. This notebook and its source code are released under the terms of the [MIT License](https://cognitiveclass.ai/mit-license/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkML0101ENSkillsNetwork1047-2022-01-01).\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}