03.DT-RF-LGBM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Y8wPo4yL8xFE"
   },
   "source": [
    "# UNDERSTANDING DECISION TREES & ENSEMBLE METHODS USING NEW YORK CITY TAXI FARE PREDICTION DATASET\n",
    "--- \n",
    "\n",
    "**PROBLEM STATEMENT** \n",
    "\n",
    "*You are tasked with predicting the fare amount (inclusive of tolls) for a taxi ride in New York City given the pickup and dropoff locations. While you can get a basic estimate based on just the distance between the two points, this will result in an RMSE of \\$5-$8, depending on the model used. Your challenge is to do better than this using Machine Learning techniques!*\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 937
    },
    "colab_type": "code",
    "id": "AHczq7978SN5",
    "outputId": "ad92eb80-e071-4747-ac8f-45967aeb3155"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://images.unsplash.com/photo-1573225935973-40b81f6e39e6?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=500&q=60\" width=\"1200\" height=\"900\" align=\"center\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "html1 = '<img src=\"https://images.unsplash.com/photo-1573225935973-40b81f6e39e6?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=500&q=60\" \\\n",
    "width=\"1200\" height=\"900\" align=\"center\"/>'\n",
    "HTML(html1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "19WR-SoRGVsr"
   },
   "source": [
    "### Download the data\n",
    "\n",
    "Let's use Kaggle API to download the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 275
    },
    "colab_type": "code",
    "id": "TvhnZ6dX-B2m",
    "outputId": "fac1323b-a227-430b-9d5b-ed5e03dadb8a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading new-york-city-taxi-fare-prediction.zip to /Users/nikhilkashyap/Downloads/Learning/Github/11-Projects-to-DataScience\n",
      " 25%|█████████▌                             | 392M/1.56G [01:37<03:33, 5.92MB/s]"
     ]
    }
   ],
   "source": [
    "!kaggle competitions download -c new-york-city-taxi-fare-prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Cs1iBF57HgM5"
   },
   "source": [
    "Let's unzip the taxi file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "BTbjDahmHfKb",
    "outputId": "aa499c9c-2051-4312-8940-8f3cd52d6aaf"
   },
   "outputs": [],
   "source": [
    "!unzip new-york-city-taxi-fare-prediction.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "colab_type": "code",
    "id": "bbV4yhAARmvE",
    "outputId": "753d7762-9b43-4e27-8af5-7b2823f2331b"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import datetime as dt\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "plt.style.use('fivethirtyeight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iutO5yKYGmti"
   },
   "source": [
    "### Read the data\n",
    "\n",
    "The taxiing dataset contains 55M rows. Let us read 5M rows for faster computing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "M60-u5DPGIb8",
    "outputId": "299a43fe-7525-4109-e3e1-87b0b60328c9"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "import pandas as pd\n",
    "\n",
    "taxi = pd.read_csv(\"train.csv\",nrows=5000000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 309
    },
    "colab_type": "code",
    "id": "ncEIAlAHGIfy",
    "outputId": "e70de6cb-cbf3-4b4d-fbd6-5d456462dcac"
   },
   "outputs": [],
   "source": [
    "taxi.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 272
    },
    "colab_type": "code",
    "id": "sTvP2w5BGIZG",
    "outputId": "0b6ac147-8e4d-4717-96cc-7d3f81f0ab00"
   },
   "outputs": [],
   "source": [
    "taxi.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GeFl_vBFKOXp"
   },
   "source": [
    "Convert pickup_datetime from Object to Datetime object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "OJbqYjroKPpV",
    "outputId": "001e2f8d-c4b8-4619-ec49-6018e9d09bbd"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "taxi['pickup_datetime']=pd.to_datetime(taxi['pickup_datetime'],format='%Y-%m-%d %H:%M:%S UTC')\n",
    "taxi.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 297
    },
    "colab_type": "code",
    "id": "BMKq6dtSaqZE",
    "outputId": "cac7f407-11ad-42ed-afd6-09d37f23d5f5"
   },
   "outputs": [],
   "source": [
    "taxi.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5EptS720ayJ3"
   },
   "source": [
    "We can observe that\n",
    "\n",
    "1). Min fare amount is negative.\n",
    "\n",
    "2). Min and Max longitude and latitude look unreal.\n",
    "\n",
    "3). Min passenger count is 0.\n",
    "\n",
    "We are fixing them now.\n",
    "\n",
    "1). New York city longitudes are around -74 and latitudes are around 41.\n",
    "\n",
    "2). Remove 0 passenger count.\n",
    "\n",
    "3). The initial charge is $2.5, so we are removing fare amount smaller than this amount."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "v_Bf2YD1a2Vs"
   },
   "outputs": [],
   "source": [
    "taxi = taxi[((taxi['pickup_longitude'] > -78) & (taxi['pickup_longitude'] < -70)) & \n",
    "            ((taxi['dropoff_longitude'] > -78) & (taxi['dropoff_longitude'] < -70)) & \n",
    "            ((taxi['pickup_latitude'] > 37) & (taxi['pickup_latitude'] < 45)) & \n",
    "            ((taxi['dropoff_latitude'] > 37) & (taxi['dropoff_latitude'] < 45)) & \n",
    "            (taxi['passenger_count'] > 0) & (taxi['fare_amount'] >= 2.5)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ND7_ILW3LXWL"
   },
   "source": [
    "Check for Missing Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 170
    },
    "colab_type": "code",
    "id": "Gy6ULYLIKf-5",
    "outputId": "df085b22-b73d-4963-ba8b-023aa12a5f2e"
   },
   "outputs": [],
   "source": [
    "taxi[pd.isnull(taxi)].sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2bdc9waULkoh"
   },
   "source": [
    "### EDA\n",
    "\n",
    "##### Distribution of Trip Fare"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 325
    },
    "colab_type": "code",
    "id": "CRrqATc6KgCy",
    "outputId": "36fc9c31-7b98-453a-f3fd-2cbf092fd9ef"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize = (14, 4))\n",
    "n, bins, patches = plt.hist(taxi.fare_amount, 1000, facecolor='blue', alpha=0.75)\n",
    "plt.xlabel('Fare amount')\n",
    "plt.title('Histogram of fare amount')\n",
    "plt.xlim(0, 200)\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "w1I09GfaU-ty"
   },
   "source": [
    "The above graph also shows that most of the fare amount is small."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 221
    },
    "colab_type": "code",
    "id": "TKvO8iexVECF",
    "outputId": "5bb17bca-d13b-4104-f0c7-a1c5ff5a16da"
   },
   "outputs": [],
   "source": [
    "taxi.groupby('fare_amount').size().nlargest(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0jaSs7FmVUrd"
   },
   "source": [
    "Interesting, the most common fare amount are very small at only 6.5 and 4.5, they are very short rides."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dUGofy_dWMuo"
   },
   "source": [
    "Passenger Count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 337
    },
    "colab_type": "code",
    "id": "6v1HWb1GVXUE",
    "outputId": "b6a655ae-09a0-4c64-ea39-715041cd1a7a"
   },
   "outputs": [],
   "source": [
    "taxi['passenger_count'].value_counts().plot.bar(color = 'b', edgecolor = 'k');\n",
    "plt.title('Histogram of passenger counts'); plt.xlabel('Passenger counts'); plt.ylabel('Count');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 221
    },
    "colab_type": "code",
    "id": "0oq3w7BcWdxJ",
    "outputId": "7593e140-8130-44f0-b736-f38dc6a00014"
   },
   "outputs": [],
   "source": [
    "taxi.groupby('passenger_count').size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "562kxKR-WhDB"
   },
   "source": [
    "Based on the above discovery, we are going to remove taxi rides with passenger_count > 6."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IXmCXtxRWhpw"
   },
   "outputs": [],
   "source": [
    "taxi = taxi.loc[taxi['passenger_count'] <= 6]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 153
    },
    "colab_type": "code",
    "id": "VT0oaDgcWk7w",
    "outputId": "91b52bb5-bcd3-4b12-9edb-063b91b26445"
   },
   "outputs": [],
   "source": [
    "taxi.groupby('passenger_count').size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 297
    },
    "colab_type": "code",
    "id": "RdJpEhOgWqMd",
    "outputId": "35f1fc52-947b-46b6-9510-cfb052a4aeab"
   },
   "outputs": [],
   "source": [
    "taxi.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NGfgXK95W1AQ"
   },
   "source": [
    "### Baseline Model\n",
    "\n",
    "To be quick, let's create a baseline model, without Machine learning, just a simple rate calculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1uszAkHHW33S"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train, test = train_test_split(taxi, test_size=0.3, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YRNhMnPWYa5h"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import shutil\n",
    "\n",
    "def distance_between(lat1, lon1, lat2, lon2):\n",
    "  # Haversine formula to compute distance \n",
    "  dist = np.degrees(np.arccos(np.sin(np.radians(lat1)) * np.sin(np.radians(lat2)) + np.cos(np.radians(lat1)) * np.cos(np.radians(lat2)) * np.cos(np.radians(lon2 - lon1)))) * 60 * 1.515 * 1.609344\n",
    "  return dist\n",
    "\n",
    "def estimate_distance(df):\n",
    "  return distance_between(df['pickup_latitude'], df['pickup_longitude'], df['dropoff_latitude'], df['dropoff_longitude'])\n",
    "\n",
    "def compute_rmse(actual, predicted):\n",
    "  return np.sqrt(np.mean((actual - predicted)**2))\n",
    "\n",
    "def print_rmse(df, rate, name):\n",
    "  print(\"{1} RMSE = {0}\".format(compute_rmse(df['fare_amount'], rate * estimate_distance(df)), name))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "id": "a9aUN1buYarE",
    "outputId": "2bab9be0-e8b1-4bca-8c6a-67f363c37206"
   },
   "outputs": [],
   "source": [
    "rate = train['fare_amount'].mean() / estimate_distance(train).mean()\n",
    "\n",
    "print(\"Rate = ${0}/km\".format(rate))\n",
    "print_rmse(train, rate, 'Train')\n",
    "print_rmse(test, rate, 'Test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "o3ChaakyY2ky"
   },
   "source": [
    "This baseline model gets us RMSE for test set at $10.0. We expect ML achieve better than this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AQMHdrCSfNJc"
   },
   "source": [
    "### Feature Engineering\n",
    "\n",
    "1). Extract information from datetime (day of week, month, hour, day). Taxi fares change day/night or on weekdays/holidays.\n",
    "\n",
    "2). The distance from pickup to dropoff. The longer the trip, the higher the price.\n",
    "\n",
    "3). Add columns indicating distance from pickup or dropoff coordinates to JFK. Trips from/to JFK have a flat fare at $52.\n",
    "\n",
    "Getting distance between two points based on latitude and longitude using haversine formula. https://stackoverflow.com/questions/29545704/fast-haversine-approximation-python-pandas/29546836#29546836"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WG_2-Z_qY5dw"
   },
   "outputs": [],
   "source": [
    "taxi['year'] = taxi.pickup_datetime.dt.year\n",
    "taxi['month'] = taxi.pickup_datetime.dt.month\n",
    "taxi['day'] = taxi.pickup_datetime.dt.day\n",
    "taxi['weekday'] = taxi.pickup_datetime.dt.weekday\n",
    "taxi['hour'] = taxi.pickup_datetime.dt.hour"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 309
    },
    "colab_type": "code",
    "id": "_H3wW2-3fgJL",
    "outputId": "cdb2d568-16d5-4096-85e3-ff03beac80d0"
   },
   "outputs": [],
   "source": [
    "taxi.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "drnx5wNDfjU9"
   },
   "outputs": [],
   "source": [
    "from math import radians, cos, sin, asin, sqrt\n",
    "import numpy as np\n",
    "\n",
    "def haversine_np(lon1, lat1, lon2, lat2):\n",
    "    \"\"\"\n",
    "    Calculate the great circle distance between two points\n",
    "    on the earth (specified in decimal degrees)\n",
    "\n",
    "    All args must be of equal length.    \n",
    "\n",
    "    \"\"\"\n",
    "    lon1, lat1, lon2, lat2 = map(np.radians, [lon1, lat1, lon2, lat2])\n",
    "\n",
    "    dlon = lon2 - lon1\n",
    "    dlat = lat2 - lat1\n",
    "\n",
    "    a = np.sin(dlat/2.0)**2 + np.cos(lat1) * np.cos(lat2) * np.sin(dlon/2.0)**2\n",
    "\n",
    "    c = 2 * np.arcsin(np.sqrt(a))\n",
    "    km = 6371 * c  # 6371 is Radius of earth in kilometers. Use 3956 for miles\n",
    "    return km\n",
    "\n",
    "taxi['distance'] = haversine_np(taxi['pickup_latitude'], taxi['pickup_longitude'], taxi['dropoff_latitude'] , taxi['dropoff_longitude'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 309
    },
    "colab_type": "code",
    "id": "M3P98VnzflL-",
    "outputId": "c41ff775-cec0-4d66-825f-64716086e75f"
   },
   "outputs": [],
   "source": [
    "taxi.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 325
    },
    "colab_type": "code",
    "id": "tDFv9Qo_flI4",
    "outputId": "dac86174-d431-4285-88e1-e48d5d6fa992"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize = (14, 4))\n",
    "n, bins, patches = plt.hist(taxi.distance, 1000, facecolor='blue', alpha=0.75)\n",
    "plt.xlabel('distance')\n",
    "plt.title('Histogram of ride distance')\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 170
    },
    "colab_type": "code",
    "id": "i_2UK4opfxFN",
    "outputId": "e9b4dd53-ed0a-4921-a7fd-d9d116c51fae"
   },
   "outputs": [],
   "source": [
    "taxi['distance'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bz1xL9i5fzsP"
   },
   "outputs": [],
   "source": [
    "taxi = taxi.loc[taxi['distance'] > 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9mMOu0eof05k"
   },
   "source": [
    "Official NYC yellow taxis have a flat rate fee from JFK to Manhattan for $52 (plus tolls and tip), Add columns indicating distance from pickup or dropoff coordinates to JFK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "K29LxygXf3f6"
   },
   "outputs": [],
   "source": [
    "JFK_coord = (40.6413, -73.7781)\n",
    "\n",
    "pickup_JFK = haversine_np(taxi['pickup_latitude'], taxi['pickup_longitude'], JFK_coord[0], JFK_coord[1]) \n",
    "dropoff_JFK = haversine_np(JFK_coord[0], JFK_coord[1], taxi['dropoff_latitude'], taxi['dropoff_longitude'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Qnqa0HVEf8dE"
   },
   "outputs": [],
   "source": [
    "taxi['JFK_distance'] = pd.concat([pickup_JFK, dropoff_JFK], axis=1).min(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 170
    },
    "colab_type": "code",
    "id": "gSZbuNdgf8Z6",
    "outputId": "637b7efd-28f0-44ec-b5e5-bfd78e9850b1"
   },
   "outputs": [],
   "source": [
    "taxi['JFK_distance'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 309
    },
    "colab_type": "code",
    "id": "HjCVCZBbgA8X",
    "outputId": "bbea047a-f25b-49ea-99cc-78c3c3f987e7"
   },
   "outputs": [],
   "source": [
    "taxi.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oMqQybSVgA5M"
   },
   "outputs": [],
   "source": [
    "del taxi['pickup_datetime']\n",
    "del taxi['key']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Building\n",
    "\n",
    "**Linear Regression**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "19xvYatsgH5C"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "y = taxi['fare_amount']\n",
    "X = taxi.drop(columns=['fare_amount'])\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "k_OSvE93gIF0"
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "lr = LinearRegression()\n",
    "lr.fit(X_train, y_train)\n",
    "y_pred = lr.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "_OvdQx9qgICH",
    "outputId": "bc7fb47d-9445-4eb0-da3e-1bfce739120b"
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "print(\"Test RMSE: %.3f\" % mean_squared_error(y_test, y_pred) ** 0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Decision Trees**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "\n",
    "dt = DecisionTreeRegressor(max_depth=2)\n",
    "dt.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = dt.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Test RMSE: %.3f\" % mean_squared_error(y_test, y_pred) ** 0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wqLzAZC-gkFo"
   },
   "source": [
    "Ensemble methods combine several decision trees classifiers to produce better predictive performance than a single decision tree classifier. The main principle behind the ensemble model is that a group of weak learners come together to form a strong learner, thus increasing the accuracy of the model.\n",
    "\n",
    "Bagging is a way to decrease the variance in the prediction by generating additional data for training from dataset using combinations with repetitions to produce multi-sets of the original data. \n",
    "\n",
    "Boosting is an iterative technique which adjusts the weight of an observation based on the last classification. If an observation was classified incorrectly, it tries to increase the weight of this observation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RCgkDwYHsSTH"
   },
   "source": [
    "**Random Forest**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "Q4MgNuKAgH-y",
    "outputId": "93dae84d-629a-4804-d98f-4bdc670652a7"
   },
   "outputs": [],
   "source": [
    "# %%time\n",
    "# from sklearn.ensemble import RandomForestRegressor\n",
    "\n",
    "# rf = RandomForestRegressor(max_depth=2, random_state=0, n_estimators=100)\n",
    "# rf.fit(X_train, y_train)\n",
    "# y_pred = rf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "dGVTnfNclCFc",
    "outputId": "b2bd77a8-3df4-489d-9252-7294c8e1258a"
   },
   "outputs": [],
   "source": [
    "# print(\"Test RMSE: %.3f\" % mean_squared_error(y_test, y_pred) ** 0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9yZHs4gLgyh9"
   },
   "source": [
    "#### Light GBM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HOgSXefhgH1v"
   },
   "outputs": [],
   "source": [
    "import lightgbm as lgb\n",
    "\n",
    "params = {\n",
    "        'learning_rate': 0.75,\n",
    "        'application': 'regression',\n",
    "        'max_depth': 3,\n",
    "        'num_leaves': 100,\n",
    "        'verbosity': -1,\n",
    "        'metric': 'RMSE',\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qiPxzR7yg4kv"
   },
   "outputs": [],
   "source": [
    "train_set = lgb.Dataset(X_train, y_train, silent=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "C2C9HSevg4Wi",
    "outputId": "05fdd736-82af-4718-9579-b604d1ce2d87"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "lb = lgb.train(params, train_set = train_set, num_boost_round=300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VKYQmrlqg9YJ"
   },
   "outputs": [],
   "source": [
    "y_pred = lb.predict(X_test, num_iteration = lb.best_iteration)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "XL4Rto_Dg9Ua",
    "outputId": "bf9a45ea-cde2-4002-8153-eed1269e305b"
   },
   "outputs": [],
   "source": [
    "print(\"Test RMSE: %.3f\" % mean_squared_error(y_test, y_pred) ** 0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from explainx import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "explainx.ai(X_Data, Y_Data, lb, model_name=\"xgboost\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "TaxiFarePrediction.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}