CaseStudy NYC Taxi Demand.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# New York City Taxi Demand Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# -----------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>\n",
    "Ge the data from : http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml (2016 data)\n",
    "The data used in the attached datasets were collected and provided to the NYC Taxi and Limousine Commission (TLC) \n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Information on taxis:\n",
    "\n",
    "<h5> Yellow Taxi: Yellow Medallion Taxicabs</h5>\n",
    "<p> These are the famous NYC yellow taxis that provide transportation exclusively through street-hails. The number of taxicabs is limited by a finite number of medallions issued by the TLC. You access this mode of transportation by standing in the street and hailing an available taxi with your hand. The pickups are not pre-arranged.</p>\n",
    "\n",
    "<h5> For Hire Vehicles (FHVs) </h5>\n",
    "<p> FHV transportation is accessed by a pre-arrangement with a dispatcher or limo company. These FHVs are not permitted to pick up passengers via street hails, as those rides are not considered pre-arranged. </p>\n",
    "\n",
    "<h5> Green Taxi: Street Hail Livery (SHL) </h5>\n",
    "<p>  The SHL program will allow livery vehicle owners to license and outfit their vehicles with green borough taxi branding, meters, credit card machines, and ultimately the right to accept street hails in addition to pre-arranged rides. </p>\n",
    "<p> Credits: Quora</p>\n",
    "\n",
    "<h5>Footnote:</h5>\n",
    "In the given notebook we are considering only the yellow taxis for the time period between Jan - Mar 2015 & Jan - Mar 2016"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ML Problem Formulation\n",
    "<p><b> Time-series forecasting and Regression</b></p>\n",
    "<br>\n",
    "-<i> To find number of pickups, given location cordinates(latitude and longitude) and time, in the query reigion and surrounding regions.</i>\n",
    "<p> \n",
    "To solve the above we would be using data collected in Jan - Mar 2015 to predict the pickups in Jan - Mar 2016.\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Performance metrics\n",
    "1. Mean Absolute percentage error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# -----------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Objectives:\n",
    "    1.Apply Regression Models with Fourier features\n",
    "    2.Hyper Parameter Tuning For Regression Models\n",
    "    3.Feature Engineering To Bring MAPE less than 0.12"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Findings :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1: Regression Models With Foureir Features\n",
    "     \n",
    "     XGB Mape\n",
    "        Train:0.1279.Test:0.1257\n",
    "        \n",
    "     Random Forest Mape\n",
    "        Train:0.0935.Test:0.1246\n",
    "     \n",
    "     Linear Regression Mape\n",
    "        Train:0.1327.Test:0.1286"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2: Regression Models With HyperParam Tunning\n",
    "     \n",
    "     XGB Mape\n",
    "        Train:0.1203.Test:0.1238\n",
    "        \n",
    "     Random Forest Mape\n",
    "        Train:0.1175.Test:0.1245\n",
    "     \n",
    "     Linear Regression Mape\n",
    "        Train:0.1330.Test:0.1292"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 3: Feature Engineering\n",
    "     \n",
    "     Random Forest Mape\n",
    "        Train:0.1124.Test:0.1193"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# -----------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dask.dataframe as dd#similar to pandas\n",
    "import pandas as pd#pandas to create small dataframes \n",
    "import folium #open street map\n",
    "import datetime #Convert to unix time\n",
    "import time #Convert to unix time\n",
    "import numpy as np#Do aritmetic operations on arrays\n",
    "\n",
    "import matplotlib\n",
    "# matplotlib.use('nbagg') : matplotlib uses this protocall which makes plots more user intractive like zoom in and zoom out\n",
    "matplotlib.use('nbagg')\n",
    "import matplotlib.pylab as plt\n",
    "import seaborn as sns#Plots\n",
    "from matplotlib import rcParams#Size of plots  \n",
    "\n",
    "import gpxpy.geo #Get the haversine distance\n",
    "\n",
    "from sklearn.cluster import MiniBatchKMeans, KMeans#Clustering\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn.linear_model import SGDRegressor\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "\n",
    "import math\n",
    "import pickle\n",
    "import os\n",
    "import xgboost as xgb\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['VendorID', 'tpep_pickup_datetime', 'tpep_dropoff_datetime',\n",
      "       'passenger_count', 'trip_distance', 'pickup_longitude',\n",
      "       'pickup_latitude', 'RateCodeID', 'store_and_fwd_flag',\n",
      "       'dropoff_longitude', 'dropoff_latitude', 'payment_type', 'fare_amount',\n",
      "       'extra', 'mta_tax', 'tip_amount', 'tolls_amount',\n",
      "       'improvement_surcharge', 'total_amount'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "#Looking at the features\n",
    "# dask dataframe  : # https://github.com/dask/dask-tutorial/blob/master/07_dataframe.ipynb\n",
    "month = dd.read_csv('yellow_tripdata_2015-01.csv')\n",
    "print(month.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# -----------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Cleaning\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> Based on analysis done on 2015 Jan data we have the following findings </h3>\n",
    "<ol>\n",
    "    <li> Trim outliers for lat and long lying outside NY boundaries </li>\n",
    "    <li> Trim outliers for trip time between  0 and 720</li>\n",
    "    <li> Trim outliers for trip distance between 0 and 23 </li>\n",
    "    <li> Trim outliers for speed between 0 and 65 </li>\n",
    "    <li> Trim outliers for fare between 0 and 1000  </li>\n",
    "</ol>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# -----------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h5>Function to convert to Unix time</h5>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_to_unix(s):\n",
    "    return time.mktime(datetime.datetime.strptime(s, \"%Y-%m-%d %H:%M:%S\").timetuple())\n",
    "\n",
    "def return_with_trip_times(month):\n",
    "    duration = month[['tpep_pickup_datetime','tpep_dropoff_datetime']].compute()\n",
    "    #pickups and dropoffs to unix time\n",
    "    duration_pickup = [convert_to_unix(x) for x in duration['tpep_pickup_datetime'].values]\n",
    "    duration_drop = [convert_to_unix(x) for x in duration['tpep_dropoff_datetime'].values]\n",
    "    #calculate duration of trips\n",
    "    durations = (np.array(duration_drop) - np.array(duration_pickup))/float(60)\n",
    "\n",
    "    #append durations of trips and speed in miles/hr to a new dataframe\n",
    "    new_frame = month[['passenger_count','trip_distance','pickup_longitude','pickup_latitude','dropoff_longitude','dropoff_latitude','total_amount']].compute()\n",
    "    \n",
    "    new_frame['trip_times'] = durations\n",
    "    new_frame['pickup_times'] = duration_pickup\n",
    "    new_frame['Speed'] = 60*(new_frame['trip_distance']/new_frame['trip_times'])\n",
    "    \n",
    "    return new_frame\n",
    "frame_with_durations = return_with_trip_times(month)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h5>Function to Remove Outliers </h5>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "#removing all outliers based on our univariate analysis above\n",
    "def remove_outliers(new_frame):\n",
    "\n",
    "    \n",
    "    a = new_frame.shape[0]\n",
    "    print (\"Number of pickup records = \",a)\n",
    "    temp_frame = new_frame[((new_frame.dropoff_longitude >= -74.15) & (new_frame.dropoff_longitude <= -73.7004) &\\\n",
    "                       (new_frame.dropoff_latitude >= 40.5774) & (new_frame.dropoff_latitude <= 40.9176)) & \\\n",
    "                       ((new_frame.pickup_longitude >= -74.15) & (new_frame.pickup_latitude >= 40.5774)& \\\n",
    "                       (new_frame.pickup_longitude <= -73.7004) & (new_frame.pickup_latitude <= 40.9176))]\n",
    "    b = temp_frame.shape[0]\n",
    "    print (\"Number of outlier coordinates lying outside NY boundaries:\",(a-b))\n",
    "\n",
    "    \n",
    "    temp_frame = new_frame[(new_frame.trip_times > 0) & (new_frame.trip_times < 720)]\n",
    "    c = temp_frame.shape[0]\n",
    "    print (\"Number of outliers from trip times analysis:\",(a-c))\n",
    "    \n",
    "    \n",
    "    temp_frame = new_frame[(new_frame.trip_distance > 0) & (new_frame.trip_distance < 23)]\n",
    "    d = temp_frame.shape[0]\n",
    "    print (\"Number of outliers from trip distance analysis:\",(a-d))\n",
    "    \n",
    "    temp_frame = new_frame[(new_frame.Speed <= 65) & (new_frame.Speed >= 0)]\n",
    "    e = temp_frame.shape[0]\n",
    "    print (\"Number of outliers from speed analysis:\",(a-e))\n",
    "    \n",
    "    temp_frame = new_frame[(new_frame.total_amount <1000) & (new_frame.total_amount >0)]\n",
    "    f = temp_frame.shape[0]\n",
    "    print (\"Number of outliers from fare analysis:\",(a-f))\n",
    "    \n",
    "    \n",
    "    new_frame = new_frame[((new_frame.dropoff_longitude >= -74.15) & (new_frame.dropoff_longitude <= -73.7004) &\\\n",
    "                       (new_frame.dropoff_latitude >= 40.5774) & (new_frame.dropoff_latitude <= 40.9176)) & \\\n",
    "                       ((new_frame.pickup_longitude >= -74.15) & (new_frame.pickup_latitude >= 40.5774)& \\\n",
    "                       (new_frame.pickup_longitude <= -73.7004) & (new_frame.pickup_latitude <= 40.9176))]\n",
    "    \n",
    "    new_frame = new_frame[(new_frame.trip_times > 0) & (new_frame.trip_times < 720)]\n",
    "    new_frame = new_frame[(new_frame.trip_distance > 0) & (new_frame.trip_distance < 23)]\n",
    "    new_frame = new_frame[(new_frame.Speed < 45.31) & (new_frame.Speed > 0)]\n",
    "    new_frame = new_frame[(new_frame.total_amount <1000) & (new_frame.total_amount >0)]\n",
    "    \n",
    "    print (\"Total outliers removed\",a - new_frame.shape[0])\n",
    "    print (\"---\")\n",
    "    return new_frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# -----------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data-preperation\n",
    "## Clustering/Segmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference:\n",
    "- The main objective was to find a optimal min. distance(Which roughly estimates to the radius of a cluster) between the clusters which we got was 40"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "On choosing a cluster size of  10 \n",
      "Avg. Number of Clusters within the vicinity (i.e. intercluster-distance < 2): 2.0 \n",
      "Avg. Number of Clusters outside the vicinity (i.e. intercluster-distance > 2): 8.0 \n",
      "Min inter-cluster distance =  1.0945442325142543 \n",
      "---\n",
      "On choosing a cluster size of  20 \n",
      "Avg. Number of Clusters within the vicinity (i.e. intercluster-distance < 2): 4.0 \n",
      "Avg. Number of Clusters outside the vicinity (i.e. intercluster-distance > 2): 16.0 \n",
      "Min inter-cluster distance =  0.7131298007387813 \n",
      "---\n",
      "On choosing a cluster size of  30 \n",
      "Avg. Number of Clusters within the vicinity (i.e. intercluster-distance < 2): 8.0 \n",
      "Avg. Number of Clusters outside the vicinity (i.e. intercluster-distance > 2): 22.0 \n",
      "Min inter-cluster distance =  0.5185088176172206 \n",
      "---\n",
      "On choosing a cluster size of  40 \n",
      "Avg. Number of Clusters within the vicinity (i.e. intercluster-distance < 2): 8.0 \n",
      "Avg. Number of Clusters outside the vicinity (i.e. intercluster-distance > 2): 32.0 \n",
      "Min inter-cluster distance =  0.5069768450363973 \n",
      "---\n",
      "On choosing a cluster size of  50 \n",
      "Avg. Number of Clusters within the vicinity (i.e. intercluster-distance < 2): 12.0 \n",
      "Avg. Number of Clusters outside the vicinity (i.e. intercluster-distance > 2): 38.0 \n",
      "Min inter-cluster distance =  0.365363025983595 \n",
      "---\n",
      "On choosing a cluster size of  60 \n",
      "Avg. Number of Clusters within the vicinity (i.e. intercluster-distance < 2): 14.0 \n",
      "Avg. Number of Clusters outside the vicinity (i.e. intercluster-distance > 2): 46.0 \n",
      "Min inter-cluster distance =  0.34704283494187155 \n",
      "---\n",
      "On choosing a cluster size of  70 \n",
      "Avg. Number of Clusters within the vicinity (i.e. intercluster-distance < 2): 16.0 \n",
      "Avg. Number of Clusters outside the vicinity (i.e. intercluster-distance > 2): 54.0 \n",
      "Min inter-cluster distance =  0.30502203163244707 \n",
      "---\n",
      "On choosing a cluster size of  80 \n",
      "Avg. Number of Clusters within the vicinity (i.e. intercluster-distance < 2): 18.0 \n",
      "Avg. Number of Clusters outside the vicinity (i.e. intercluster-distance > 2): 62.0 \n",
      "Min inter-cluster distance =  0.29220324531738534 \n",
      "---\n",
      "On choosing a cluster size of  90 \n",
      "Avg. Number of Clusters within the vicinity (i.e. intercluster-distance < 2): 21.0 \n",
      "Avg. Number of Clusters outside the vicinity (i.e. intercluster-distance > 2): 69.0 \n",
      "Min inter-cluster distance =  0.18257992857034985 \n",
      "---\n"
     ]
    }
   ],
   "source": [
    "#trying different cluster sizes to choose the right K in K-means\n",
    "coords = frame_with_durations_outliers_removed[['pickup_latitude', 'pickup_longitude']].values\n",
    "neighbours=[]\n",
    "\n",
    "def find_min_distance(cluster_centers, cluster_len):\n",
    "    nice_points = 0\n",
    "    wrong_points = 0\n",
    "    less2 = []\n",
    "    more2 = []\n",
    "    min_dist=1000\n",
    "    for i in range(0, cluster_len):\n",
    "        nice_points = 0\n",
    "        wrong_points = 0\n",
    "        for j in range(0, cluster_len):\n",
    "            if j!=i:\n",
    "                distance = gpxpy.geo.haversine_distance(cluster_centers[i][0], cluster_centers[i][1],cluster_centers[j][0], cluster_centers[j][1])\n",
    "                min_dist = min(min_dist,distance/(1.60934*1000))\n",
    "                if (distance/(1.60934*1000)) <= 2:\n",
    "                    nice_points +=1\n",
    "                else:\n",
    "                    wrong_points += 1\n",
    "        less2.append(nice_points)\n",
    "        more2.append(wrong_points)\n",
    "    neighbours.append(less2)\n",
    "    print (\"On choosing a cluster size of \",cluster_len,\"\\nAvg. Number of Clusters within the vicinity (i.e. intercluster-distance < 2):\", np.ceil(sum(less2)/len(less2)), \"\\nAvg. Number of Clusters outside the vicinity (i.e. intercluster-distance > 2):\", np.ceil(sum(more2)/len(more2)),\"\\nMin inter-cluster distance = \",min_dist,\"\\n---\")\n",
    "\n",
    "def find_clusters(increment):\n",
    "    kmeans = MiniBatchKMeans(n_clusters=increment, batch_size=10000,random_state=42).fit(coords)\n",
    "    frame_with_durations_outliers_removed['pickup_cluster'] = kmeans.predict(frame_with_durations_outliers_removed[['pickup_latitude', 'pickup_longitude']])\n",
    "    cluster_centers = kmeans.cluster_centers_\n",
    "    cluster_len = len(cluster_centers)\n",
    "    return cluster_centers, cluster_len\n",
    "\n",
    "# we need to choose number of clusters so that, there are more number of cluster regions \n",
    "#that are close to any cluster center\n",
    "# and make sure that the minimum inter cluster should not be very less\n",
    "for increment in range(10, 100, 10):\n",
    "    cluster_centers, cluster_len = find_clusters(increment)\n",
    "    find_min_distance(cluster_centers, cluster_len)            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if check for the 50 clusters you can observe that there are two clusters with only 0.3 miles apart from each other\n",
    "# so we choose 40 clusters for solve the further problem\n",
    "\n",
    "# Getting 40 clusters using the kmeans \n",
    "kmeans = MiniBatchKMeans(n_clusters=40, batch_size=10000,random_state=0).fit(coords)\n",
    "frame_with_durations_outliers_removed['pickup_cluster'] = kmeans.predict(frame_with_durations_outliers_removed[['pickup_latitude', 'pickup_longitude']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# -----------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1> Time-binning </h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> We make 10 min time bins as we had discovered in EDA that the average trip time based on distance is 10 min </h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h5>Function to convert to 10 min time bins</h5>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_pickup_bins(frame,month,year):\n",
    "    unix_pickup_times=[i for i in frame['pickup_times'].values]\n",
    "    unix_times = [[1420070400,1422748800,1425168000,1427846400,1430438400,1433116800],\\\n",
    "                    [1451606400,1454284800,1456790400,1459468800,1462060800,1464739200]]\n",
    "    \n",
    "    start_pickup_unix=unix_times[year-2015][month-1]\n",
    "    # https://www.timeanddate.com/time/zones/est\n",
    "    # (int((i-start_pickup_unix)/600)+33) : our unix time is in gmt to we are converting it to est\n",
    "    tenminutewise_binned_unix_pickup_times=[(int((i-start_pickup_unix)/600)+33) for i in unix_pickup_times]\n",
    "    frame['pickup_bins'] = np.array(tenminutewise_binned_unix_pickup_times)\n",
    "    return frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# -----------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data preparation for months of 2016 Jan Feb Mar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Return with trip times..\n",
      "Remove outliers..\n",
      "Number of pickup records =  10906858\n",
      "Number of outlier coordinates lying outside NY boundaries: 214677\n",
      "Number of outliers from trip times analysis: 27190\n",
      "Number of outliers from trip distance analysis: 79742\n",
      "Number of outliers from speed analysis: 21047\n",
      "Number of outliers from fare analysis: 4991\n",
      "Total outliers removed 297784\n",
      "---\n",
      "Estimating clusters..\n",
      "Final groupbying..\n",
      "Return with trip times..\n",
      "Remove outliers..\n",
      "Number of pickup records =  11382049\n",
      "Number of outlier coordinates lying outside NY boundaries: 223161\n",
      "Number of outliers from trip times analysis: 27670\n",
      "Number of outliers from trip distance analysis: 81902\n",
      "Number of outliers from speed analysis: 22437\n",
      "Number of outliers from fare analysis: 5476\n",
      "Total outliers removed 308177\n",
      "---\n",
      "Estimating clusters..\n",
      "Final groupbying..\n",
      "Return with trip times..\n",
      "Remove outliers..\n",
      "Number of pickup records =  12210952\n",
      "Number of outlier coordinates lying outside NY boundaries: 232444\n",
      "Number of outliers from trip times analysis: 30868\n",
      "Number of outliers from trip distance analysis: 87318\n",
      "Number of outliers from speed analysis: 23889\n",
      "Number of outliers from fare analysis: 5859\n",
      "Total outliers removed 324635\n",
      "---\n",
      "Estimating clusters..\n",
      "Final groupbying..\n"
     ]
    }
   ],
   "source": [
    "def datapreparation(month,kmeans,month_no,year_no):\n",
    "    \n",
    "    print (\"Return with trip times..\")\n",
    "\n",
    "    frame_with_durations = return_with_trip_times(month)\n",
    "    \n",
    "    print (\"Remove outliers..\")\n",
    "    frame_with_durations_outliers_removed = remove_outliers(frame_with_durations)\n",
    "    \n",
    "    print (\"Estimating clusters..\")\n",
    "    frame_with_durations_outliers_removed['pickup_cluster'] = kmeans.predict(frame_with_durations_outliers_removed[['pickup_latitude', 'pickup_longitude']])\n",
    "    #frame_with_durations_outliers_removed_2016['pickup_cluster'] = kmeans.predict(frame_with_durations_outliers_removed_2016[['pickup_latitude', 'pickup_longitude']])\n",
    "\n",
    "    print (\"Final groupbying..\")\n",
    "    final_updated_frame = add_pickup_bins(frame_with_durations_outliers_removed,month_no,year_no)\n",
    "    final_groupby_frame = final_updated_frame[['pickup_cluster','pickup_bins','trip_distance']].groupby(['pickup_cluster','pickup_bins']).count()\n",
    "    \n",
    "    return final_updated_frame,final_groupby_frame\n",
    "    \n",
    "month_jan_2016 = dd.read_csv('yellow_tripdata_2016-01.csv')\n",
    "month_feb_2016 = dd.read_csv('yellow_tripdata_2016-02.csv')\n",
    "month_mar_2016 = dd.read_csv('yellow_tripdata_2016-03.csv')\n",
    "\n",
    "jan_2016_frame,jan_2016_groupby = datapreparation(month_jan_2016,kmeans,1,2016)\n",
    "feb_2016_frame,feb_2016_groupby = datapreparation(month_feb_2016,kmeans,2,2016)\n",
    "mar_2016_frame,mar_2016_groupby = datapreparation(month_mar_2016,kmeans,3,2016)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# -----------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Smoothing And Filling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> We did smoothing for 2015 data </h3>\n",
    "<h3> We will use filling for 2016 data so as to avoid data leakage </h3>\n",
    "<h3> Removed smoothing code as it is not required here </h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gets the unique bins where pickup values are present for each each reigion\n",
    "\n",
    "# for each cluster region we will collect all the indices of 10min intravels in which the pickups are happened\n",
    "# we got an observation that there are some pickpbins that doesnt have any pickups\n",
    "def return_unq_pickup_bins(frame):\n",
    "    values = []\n",
    "    for i in range(0,40):\n",
    "        new = frame[frame['pickup_cluster'] == i]\n",
    "        list_unq = list(set(new['pickup_bins']))\n",
    "        list_unq.sort()\n",
    "        values.append(list_unq)\n",
    "    return values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for every month we get all indices of 10min intravels in which atleast one pickup got happened\n",
    "\n",
    "#jan\n",
    "jan_2016_unique = return_unq_pickup_bins(jan_2016_frame)\n",
    "\n",
    "#feb\n",
    "feb_2016_unique = return_unq_pickup_bins(feb_2016_frame)\n",
    "\n",
    "#march\n",
    "mar_2016_unique = return_unq_pickup_bins(mar_2016_frame)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fill_missing_1(count_values,values,bins):\n",
    "    smoothed_regions=[]\n",
    "    ind=0\n",
    "    for r in range(0,40):\n",
    "        smoothed_bins=[]\n",
    "        for i in range(bins):\n",
    "            if i in values[r]:\n",
    "                smoothed_bins.append(count_values[ind])\n",
    "                ind+=1\n",
    "            else:\n",
    "                smoothed_bins.append(0)\n",
    "        smoothed_regions.extend(smoothed_bins)\n",
    "    return smoothed_regions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "jan_2016_smooth = fill_missing_1(jan_2016_groupby['trip_distance'].values,jan_2016_unique,4464)\n",
    "feb_2016_smooth = fill_missing_1(feb_2016_groupby['trip_distance'].values,feb_2016_unique,4176)\n",
    "mar_2016_smooth = fill_missing_1(mar_2016_groupby['trip_distance'].values,mar_2016_unique,4464)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "178560\n",
      "167040\n",
      "178560\n"
     ]
    }
   ],
   "source": [
    "print(len(jan_2016_smooth))\n",
    "print(len(feb_2016_smooth))\n",
    "print(len(mar_2016_smooth))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Making list of all the values of pickup data in every bin for a period of 3 months and storing them region-wise \n",
    "regions_cum = []\n",
    "\n",
    "# a =[1,2,3]\n",
    "# b = [2,3,4]\n",
    "# a+b = [1, 2, 3, 2, 3, 4]\n",
    "\n",
    "# number of 10min indices for jan 2015= 24*31*60/10 = 4464\n",
    "# number of 10min indices for jan 2016 = 24*31*60/10 = 4464\n",
    "# number of 10min indices for feb 2016 = 24*29*60/10 = 4176\n",
    "# number of 10min indices for march 2016 = 24*31*60/10 = 4464\n",
    "# regions_cum: it will contain 40 lists, each list will contain 4464+4176+4464 values which represents the number of pickups \n",
    "# that are happened for three months in 2016 data\n",
    "\n",
    "for i in range(0,40):\n",
    "    regions_cum.append(jan_2016_smooth[4464*i:4464*(i+1)]+feb_2016_smooth[4176*i:4176*(i+1)]+mar_2016_smooth[4464*i:4464*(i+1)])\n",
    "\n",
    "# print(len(regions_cum))\n",
    "# 40\n",
    "# print(len(regions_cum[0]))\n",
    "# 13104"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13104\n"
     ]
    }
   ],
   "source": [
    "print(len(regions_cum[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# -----------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Time series and Fourier Transforms "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> Calculation top 5 Amp and Freq </h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def applyFft(month_data, noOfBins,m):\n",
    "    amp=[]\n",
    "    frequency=[]\n",
    "    for i in range(0,40):\n",
    "        if(m=='jan'):\n",
    "            amp_cluster    = np.abs( np.fft.fft(np.array(month_data)[ (noOfBins * i ) +5: noOfBins* (i+1) ]) )\n",
    "            freq_cluster = np.fft.fftfreq(noOfBins, 1)\n",
    "        else:\n",
    "            amp_cluster    = np.abs( np.fft.fft(np.array(month_data)[ (noOfBins *i ): (noOfBins* (i+1)) ]) )\n",
    "            freq_cluster = np.fft.fftfreq(noOfBins, 1)\n",
    "        #Get top 5 amp and its frequencies\n",
    "        top_indices=np.array(np.argsort(amp_cluster)[::-1])[0:5]\n",
    "        #amp_cluster = [amp_cluster[top_indices]] * noOfBins\n",
    "        #freq_cluster = [freq_cluster[top_indices]] * noOfBins\n",
    "        \n",
    "        amp_cluster = amp_cluster[top_indices]\n",
    "        freq_cluster = freq_cluster[top_indices]\n",
    "        \n",
    "        amp.append(amp_cluster)\n",
    "        frequency.append(freq_cluster)\n",
    "    \n",
    "    return amp,frequency"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "jan_2016_amp,jan_2016_freq= applyFft(jan_2016_smooth,4464,'jan')\n",
    "feb_2016_amp,feb_2016_freq= applyFft(feb_2016_smooth,4176,'feb')\n",
    "mar_2016_amp,mar_2016_freq= applyFft(mar_2016_smooth,4464,'mar')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "178555"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(jan_2016_smooth[5:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n",
      "40\n"
     ]
    }
   ],
   "source": [
    "print(len(jan_2016_amp[0]))\n",
    "print(len(jan_2016_amp))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> amp1_allTimeBins has the top amplitutde for cluster and months. Same for others also </h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "amp1_allTimeBins=[]\n",
    "amp2_allTimeBins=[]\n",
    "amp3_allTimeBins=[]\n",
    "amp4_allTimeBins=[]\n",
    "amp5_allTimeBins=[]\n",
    "freq1_allTimeBins=[]\n",
    "freq2_allTimeBins=[]\n",
    "freq3_allTimeBins=[]\n",
    "freq4_allTimeBins=[]\n",
    "freq5_allTimeBins=[]\n",
    "\n",
    "for i in range(0,40):\n",
    "    amp1_allTimeBins.append( [jan_2016_amp[i][0]] * 4459 + [feb_2016_amp[i][0]] * 4176 + [mar_2016_amp[i][0]] * 4464 )\n",
    "    amp2_allTimeBins.append( [jan_2016_amp[i][1]] * 4459 + [feb_2016_amp[i][1]] * 4176 + [mar_2016_amp[i][1]] * 4464 )\n",
    "    amp3_allTimeBins.append( [jan_2016_amp[i][2]] * 4459 + [feb_2016_amp[i][2]] * 4176 + [mar_2016_amp[i][2]] * 4464 )\n",
    "    amp4_allTimeBins.append( [jan_2016_amp[i][3]] * 4459 + [feb_2016_amp[i][3]] * 4176 + [mar_2016_amp[i][3]] * 4464 )\n",
    "    amp5_allTimeBins.append( [jan_2016_amp[i][4]] * 4459 + [feb_2016_amp[i][4]] * 4176 + [mar_2016_amp[i][4]] * 4464 )\n",
    "    \n",
    "    freq1_allTimeBins.append( [jan_2016_freq[i][0]] * 4459 + [feb_2016_freq[i][0]] * 4176 + [mar_2016_freq[i][0]] * 4464 )\n",
    "    freq2_allTimeBins.append( [jan_2016_freq[i][1]] * 4459 + [feb_2016_freq[i][1]] * 4176 + [mar_2016_freq[i][1]] * 4464 )\n",
    "    freq3_allTimeBins.append( [jan_2016_freq[i][2]] * 4459 + [feb_2016_freq[i][2]] * 4176 + [mar_2016_freq[i][2]] * 4464 )\n",
    "    freq4_allTimeBins.append( [jan_2016_freq[i][3]] * 4459 + [feb_2016_freq[i][3]] * 4176 + [mar_2016_freq[i][3]] * 4464 )\n",
    "    freq5_allTimeBins.append( [jan_2016_freq[i][4]] * 4459 + [feb_2016_freq[i][4]] * 4176 + [mar_2016_freq[i][4]] * 4464 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# -----------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train-Test Split\n",
    "Before we start predictions using the tree based regression models we take 3 months of 2016 pickup data and split it such that for every region we have 70% data in train and 30% in test, ordered date-wise for every region"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Preparing data to be split into train and test, The below prepares data in cumulative form which will be later split into test and train\n",
    "# number of 10min indices for jan 2015= 24*31*60/10 = 4464\n",
    "# number of 10min indices for jan 2016 = 24*31*60/10 = 4464\n",
    "# number of 10min indices for feb 2016 = 24*29*60/10 = 4176\n",
    "# number of 10min indices for march 2016 = 24*31*60/10 = 4464\n",
    "# regions_cum: it will contain 40 lists, each list will contain 4464+4176+4464 values which represents the number of pickups \n",
    "# that are happened for three months in 2016 data\n",
    "\n",
    "# print(len(regions_cum))\n",
    "# 40\n",
    "# print(len(regions_cum[0]))\n",
    "# 12960\n",
    "\n",
    "# we take number of pickups that are happened in last 5 10min intravels\n",
    "number_of_time_stamps = 5\n",
    "\n",
    "# output varaible\n",
    "# it is list of lists\n",
    "# it will contain number of pickups 13099 for each cluster\n",
    "output = []\n",
    "\n",
    "\n",
    "# tsne_lat will contain 13104-5=13099 times lattitude of cluster center for every cluster\n",
    "# Ex: [[cent_lat 13099times],[cent_lat 13099times], [cent_lat 13099times].... 40 lists]\n",
    "# it is list of lists\n",
    "tsne_lat = []\n",
    "\n",
    "\n",
    "# tsne_lon will contain 13104-5=13099 times logitude of cluster center for every cluster\n",
    "# Ex: [[cent_long 13099times],[cent_long 13099times], [cent_long 13099times].... 40 lists]\n",
    "# it is list of lists\n",
    "tsne_lon = []\n",
    "\n",
    "# we will code each day \n",
    "# sunday = 0, monday=1, tue = 2, wed=3, thur=4, fri=5,sat=6\n",
    "# for every cluster we will be adding 13099 values, each value represent to which day of the week that pickup bin belongs to\n",
    "# it is list of lists\n",
    "tsne_weekday = []\n",
    "\n",
    "# its an numbpy array, of shape (523960, 5)\n",
    "# each row corresponds to an entry in out data\n",
    "# for the first row we will have [f0,f1,f2,f3,f4] fi=number of pickups happened in i+1th 10min intravel(bin)\n",
    "# the second row will have [f1,f2,f3,f4,f5]\n",
    "# the third row will have [f2,f3,f4,f5,f6]\n",
    "# and so on...\n",
    "tsne_feature = []\n",
    "\n",
    "\n",
    "tsne_feature = [0]*number_of_time_stamps\n",
    "for i in range(0,40):\n",
    "    tsne_lat.append([kmeans.cluster_centers_[i][0]]*13099)\n",
    "    tsne_lon.append([kmeans.cluster_centers_[i][1]]*13099)\n",
    "    # jan 1st 2016 is thursday, so we start our day from 4: \"(int(k/144))%7+4\"\n",
    "    # our prediction start from 5th 10min intravel since we need to have number of pickups that are happened in last 5 pickup bins\n",
    "    tsne_weekday.append([int(((int(k/144))%7+4)%7) for k in range(5,4464+4176+4464)])\n",
    "    # regions_cum is a list of lists [[x1,x2,x3..x13104], [x1,x2,x3..x13104], [x1,x2,x3..x13104], [x1,x2,x3..x13104], [x1,x2,x3..x13104], .. 40 lsits]\n",
    "    tsne_feature = np.vstack((tsne_feature, [regions_cum[i][r:r+number_of_time_stamps] for r in range(0,len(regions_cum[i])-number_of_time_stamps)]))\n",
    "    output.append(regions_cum[i][5:])\n",
    "tsne_feature = tsne_feature[1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(523960, 5)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tsne_feature.shape\n",
    "#tsne_feature[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tsne_lat[0])*len(tsne_lat) == tsne_feature.shape[0] == len(tsne_weekday)*len(tsne_weekday[0]) == 40*13099 == len(output)*len(output[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Getting the predictions of exponential moving averages to be used as a feature in cumulative form\n",
    "\n",
    "# upto now we computed 8 features for every data point that starts from 50th min of the day\n",
    "# 1. cluster center lattitude\n",
    "# 2. cluster center longitude\n",
    "# 3. day of the week \n",
    "# 4. f_t_1: number of pickups that are happened previous t-1th 10min intravel\n",
    "# 5. f_t_2: number of pickups that are happened previous t-2th 10min intravel\n",
    "# 6. f_t_3: number of pickups that are happened previous t-3th 10min intravel\n",
    "# 7. f_t_4: number of pickups that are happened previous t-4th 10min intravel\n",
    "# 8. f_t_5: number of pickups that are happened previous t-5th 10min intravel\n",
    "\n",
    "# from the baseline models we said the exponential weighted moving avarage gives us the best error\n",
    "# we will try to add the same exponential weighted moving avarage at t as a feature to our data\n",
    "# exponential weighted moving avarage => p'(t) = alpha*p'(t-1) + (1-alpha)*P(t-1) \n",
    "alpha=0.1\n",
    "\n",
    "# it is a temporary array that store exponential weighted moving avarage for each 10min intravel, \n",
    "# for each cluster it will get reset\n",
    "# for every cluster it contains 13104 values\n",
    "predicted_values=[]\n",
    "\n",
    "# it is similar like tsne_lat\n",
    "# it is list of lists\n",
    "# predict_list is a list of lists [[x5,x6,x7..x13104], [x5,x6,x7..x13104], [x5,x6,x7..x13104], [x5,x6,x7..x13104], [x5,x6,x7..x13104], .. 40 lsits]\n",
    "predict_list = []\n",
    "tsne_flat_exp_avg = []\n",
    "for r in range(0,40):\n",
    "    for i in range(0,13104):\n",
    "        if i==0:\n",
    "            predicted_value= regions_cum[r][0]\n",
    "            predicted_values.append(0)\n",
    "            continue\n",
    "        predicted_values.append(predicted_value)\n",
    "        predicted_value =int((alpha*predicted_value) + (1-alpha)*(regions_cum[r][i]))\n",
    "    predict_list.append(predicted_values[5:])\n",
    "    predicted_values=[]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "size of train data : 9169\n",
      "size of test data : 3929\n"
     ]
    }
   ],
   "source": [
    "# train, test split : 70% 30% split\n",
    "# Before we start predictions using the tree based regression models we take 3 months of 2016 pickup data \n",
    "# and split it such that for every region we have 70% data in train and 30% in test,\n",
    "# ordered date-wise for every region\n",
    "print(\"size of train data :\", int(13099*0.7))\n",
    "print(\"size of test data :\", int(13099*0.3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extracting first 9169 timestamp values i.e 70% of 13099 (total timestamps) for our training data\n",
    "train_features =  [tsne_feature[i*13099:(13099*i+9169)] for i in range(0,40)]\n",
    "# temp = [0]*(12955 - 9068)\n",
    "test_features = [tsne_feature[(13099*(i))+9169:13099*(i+1)] for i in range(0,40)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of data clusters 40 Number of data points in trian data 9169 Each data point contains 5 features\n",
      "Number of data clusters 40 Number of data points in test data 3930 Each data point contains 5 features\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of data clusters\",len(train_features), \"Number of data points in trian data\", len(train_features[0]), \"Each data point contains\", len(train_features[0][0]),\"features\")\n",
    "print(\"Number of data clusters\",len(train_features), \"Number of data points in test data\", len(test_features[0]), \"Each data point contains\", len(test_features[0][0]),\"features\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extracting first 9169 timestamp values i.e 70% of 13099 (total timestamps) for our training data\n",
    "tsne_train_flat_lat = [i[:9169] for i in tsne_lat]\n",
    "tsne_train_flat_lon = [i[:9169] for i in tsne_lon]\n",
    "tsne_train_flat_weekday = [i[:9169] for i in tsne_weekday]\n",
    "tsne_train_flat_output = [i[:9169] for i in output]\n",
    "tsne_train_flat_exp_avg = [i[:9169] for i in predict_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extracting the rest of the timestamp values i.e 30% of 12956 (total timestamps) for our test data\n",
    "tsne_test_flat_lat = [i[9169:] for i in tsne_lat]\n",
    "tsne_test_flat_lon = [i[9169:] for i in tsne_lon]\n",
    "tsne_test_flat_weekday = [i[9169:] for i in tsne_weekday]\n",
    "tsne_test_flat_output = [i[9169:] for i in output]\n",
    "tsne_test_flat_exp_avg = [i[9169:] for i in predict_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the above contains values in the form of list of lists (i.e. list of values of each region), here we make all of them in one list\n",
    "train_new_features = []\n",
    "for i in range(0,40):\n",
    "    train_new_features.extend(train_features[i])\n",
    "test_new_features = []\n",
    "for i in range(0,40):\n",
    "    test_new_features.extend(test_features[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# converting lists of lists into sinle list i.e flatten\n",
    "# a  = [[1,2,3,4],[4,6,7,8]]\n",
    "# print(sum(a,[]))\n",
    "# [1, 2, 3, 4, 4, 6, 7, 8]\n",
    "\n",
    "tsne_train_lat = sum(tsne_train_flat_lat, [])\n",
    "tsne_train_lon = sum(tsne_train_flat_lon, [])\n",
    "tsne_train_weekday = sum(tsne_train_flat_weekday, [])\n",
    "tsne_train_output = sum(tsne_train_flat_output, [])\n",
    "tsne_train_exp_avg = sum(tsne_train_flat_exp_avg,[])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# converting lists of lists into sinle list i.e flatten\n",
    "# a  = [[1,2,3,4],[4,6,7,8]]\n",
    "# print(sum(a,[]))\n",
    "# [1, 2, 3, 4, 4, 6, 7, 8]\n",
    "\n",
    "tsne_test_lat = sum(tsne_test_flat_lat, [])\n",
    "tsne_test_lon = sum(tsne_test_flat_lon, [])\n",
    "tsne_test_weekday = sum(tsne_test_flat_weekday, [])\n",
    "tsne_test_output = sum(tsne_test_flat_output, [])\n",
    "tsne_test_exp_avg = sum(tsne_test_flat_exp_avg,[])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "40"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tsne_train_flat_lat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 70 30 split of time bins for Amp and Freq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsne_train_flat_amp1 = [i[:9169] for i in amp1_allTimeBins]\n",
    "tsne_train_flat_amp2 = [i[:9169] for i in amp2_allTimeBins]\n",
    "tsne_train_flat_amp3 = [i[:9169] for i in amp3_allTimeBins]\n",
    "tsne_train_flat_amp4 = [i[:9169] for i in amp4_allTimeBins]\n",
    "tsne_train_flat_amp5 = [i[:9169] for i in amp5_allTimeBins]\n",
    "\n",
    "tsne_train_flat_freq1 = [i[:9169] for i in freq1_allTimeBins]\n",
    "tsne_train_flat_freq2 = [i[:9169] for i in freq2_allTimeBins]\n",
    "tsne_train_flat_freq3 = [i[:9169] for i in freq3_allTimeBins]\n",
    "tsne_train_flat_freq4 = [i[:9169] for i in freq4_allTimeBins]\n",
    "tsne_train_flat_freq5 = [i[:9169] for i in freq5_allTimeBins]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsne_test_flat_amp1 = [i[9169:] for i in amp1_allTimeBins]\n",
    "tsne_test_flat_amp2 = [i[9169:] for i in amp2_allTimeBins]\n",
    "tsne_test_flat_amp3 = [i[9169:] for i in amp3_allTimeBins]\n",
    "tsne_test_flat_amp4 = [i[9169:] for i in amp4_allTimeBins]\n",
    "tsne_test_flat_amp5 = [i[9169:] for i in amp5_allTimeBins]\n",
    "\n",
    "tsne_test_flat_freq1 = [i[9169:] for i in freq1_allTimeBins]\n",
    "tsne_test_flat_freq2 = [i[9169:] for i in freq2_allTimeBins]\n",
    "tsne_test_flat_freq3 = [i[9169:] for i in freq3_allTimeBins]\n",
    "tsne_test_flat_freq4 = [i[9169:] for i in freq4_allTimeBins]\n",
    "tsne_test_flat_freq5 = [i[9169:] for i in freq5_allTimeBins]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsne_train_amp1 = sum(tsne_train_flat_amp1, [])\n",
    "tsne_train_amp2 = sum(tsne_train_flat_amp2, [])\n",
    "tsne_train_amp3 = sum(tsne_train_flat_amp3, [])\n",
    "tsne_train_amp4 = sum(tsne_train_flat_amp4, [])\n",
    "tsne_train_amp5 = sum(tsne_train_flat_amp5, [])\n",
    "\n",
    "tsne_train_freq1 = sum(tsne_train_flat_freq1, [])\n",
    "tsne_train_freq2 = sum(tsne_train_flat_freq2, [])\n",
    "tsne_train_freq3 = sum(tsne_train_flat_freq3, [])\n",
    "tsne_train_freq4 = sum(tsne_train_flat_freq4, [])\n",
    "tsne_train_freq5 = sum(tsne_train_flat_freq5, [])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsne_test_amp1 = sum(tsne_test_flat_amp1, [])\n",
    "tsne_test_amp2 = sum(tsne_test_flat_amp2, [])\n",
    "tsne_test_amp3 = sum(tsne_test_flat_amp3, [])\n",
    "tsne_test_amp4 = sum(tsne_test_flat_amp4, [])\n",
    "tsne_test_amp5 = sum(tsne_test_flat_amp5, [])\n",
    "\n",
    "tsne_test_freq1 = sum(tsne_test_flat_freq1, [])\n",
    "tsne_test_freq2 = sum(tsne_test_flat_freq2, [])\n",
    "tsne_test_freq3 = sum(tsne_test_flat_freq3, [])\n",
    "tsne_test_freq4 = sum(tsne_test_flat_freq4, [])\n",
    "tsne_test_freq5 = sum(tsne_test_flat_freq5, [])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def standardizeValues(val,isStd):\n",
    "    if isStd == 'Y':\n",
    "        z=np.array(val)\n",
    "        z=z.reshape(-1,1)\n",
    "        std = StandardScaler().fit_transform(z)\n",
    "        return std\n",
    "    else:\n",
    "        return val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "isStd='Y'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# -----------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final Data Massaging To Be Fed Into The Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(366760, 20)\n"
     ]
    }
   ],
   "source": [
    "# Preparing the data frame for our train data\n",
    "columns = ['ft_5','ft_4','ft_3','ft_2','ft_1']\n",
    "df_train = pd.DataFrame(data=train_new_features, columns=columns) \n",
    "\n",
    "df_train['lat'] = tsne_train_lat\n",
    "df_train['lon'] = tsne_train_lon\n",
    "df_train['weekday'] = tsne_train_weekday\n",
    "df_train['exp_avg'] = tsne_train_exp_avg\n",
    "\n",
    "df_train['amp1'] = standardizeValues(tsne_train_amp1,isStd)\n",
    "df_train['amp2'] = standardizeValues(tsne_train_amp2,isStd)\n",
    "df_train['amp3'] = standardizeValues(tsne_train_amp3,isStd)\n",
    "df_train['amp4'] = standardizeValues(tsne_train_amp4,isStd)\n",
    "df_train['amp5'] = standardizeValues(tsne_train_amp5,isStd)\n",
    "df_train['freq1'] = standardizeValues(tsne_train_freq1,isStd)\n",
    "df_train['freq2'] = standardizeValues(tsne_train_freq2,isStd)\n",
    "df_train['freq3'] = standardizeValues(tsne_train_freq3,isStd)\n",
    "df_train['freq4'] = standardizeValues(tsne_train_freq4,isStd)\n",
    "df_train['freq5'] = standardizeValues(tsne_train_freq5,isStd)\n",
    "\n",
    "print(df_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(157200, 20)\n"
     ]
    }
   ],
   "source": [
    "# Preparing the data frame for our test data\n",
    "df_test = pd.DataFrame(data=test_new_features, columns=columns) \n",
    "#df_test = pd.DataFrame()\n",
    "df_test['lat'] = tsne_test_lat\n",
    "df_test['lon'] = tsne_test_lon\n",
    "df_test['weekday'] = tsne_test_weekday\n",
    "df_test['exp_avg'] = tsne_test_exp_avg\n",
    "\n",
    "df_test['amp1'] = standardizeValues(tsne_test_amp1,isStd)\n",
    "df_test['amp2'] = standardizeValues(tsne_test_amp2,isStd)\n",
    "df_test['amp3'] = standardizeValues(tsne_test_amp3,isStd)\n",
    "df_test['amp4'] = standardizeValues(tsne_test_amp4,isStd)\n",
    "df_test['amp5'] = standardizeValues(tsne_test_amp5,isStd)\n",
    "df_test['freq1'] = standardizeValues(tsne_test_freq1,isStd)\n",
    "df_test['freq2'] = standardizeValues(tsne_test_freq2,isStd)\n",
    "df_test['freq3'] = standardizeValues(tsne_test_freq3,isStd)\n",
    "df_test['freq4'] = standardizeValues(tsne_test_freq4,isStd)\n",
    "df_test['freq5'] = standardizeValues(tsne_test_freq5,isStd)\n",
    "\n",
    "print(df_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ft_5</th>\n",
       "      <th>ft_4</th>\n",
       "      <th>ft_3</th>\n",
       "      <th>ft_2</th>\n",
       "      <th>ft_1</th>\n",
       "      <th>lat</th>\n",
       "      <th>lon</th>\n",
       "      <th>weekday</th>\n",
       "      <th>exp_avg</th>\n",
       "      <th>amp1</th>\n",
       "      <th>amp2</th>\n",
       "      <th>amp3</th>\n",
       "      <th>amp4</th>\n",
       "      <th>amp5</th>\n",
       "      <th>freq1</th>\n",
       "      <th>freq2</th>\n",
       "      <th>freq3</th>\n",
       "      <th>freq4</th>\n",
       "      <th>freq5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>63</td>\n",
       "      <td>217</td>\n",
       "      <td>189</td>\n",
       "      <td>137</td>\n",
       "      <td>40.776228</td>\n",
       "      <td>-73.982119</td>\n",
       "      <td>4</td>\n",
       "      <td>142</td>\n",
       "      <td>0.542451</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.180805</td>\n",
       "      <td>1.12074</td>\n",
       "      <td>0.900268</td>\n",
       "      <td>-0.950465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>63</td>\n",
       "      <td>217</td>\n",
       "      <td>189</td>\n",
       "      <td>137</td>\n",
       "      <td>135</td>\n",
       "      <td>40.776228</td>\n",
       "      <td>-73.982119</td>\n",
       "      <td>4</td>\n",
       "      <td>135</td>\n",
       "      <td>0.542451</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.180805</td>\n",
       "      <td>1.12074</td>\n",
       "      <td>0.900268</td>\n",
       "      <td>-0.950465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>217</td>\n",
       "      <td>189</td>\n",
       "      <td>137</td>\n",
       "      <td>135</td>\n",
       "      <td>129</td>\n",
       "      <td>40.776228</td>\n",
       "      <td>-73.982119</td>\n",
       "      <td>4</td>\n",
       "      <td>129</td>\n",
       "      <td>0.542451</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.180805</td>\n",
       "      <td>1.12074</td>\n",
       "      <td>0.900268</td>\n",
       "      <td>-0.950465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>189</td>\n",
       "      <td>137</td>\n",
       "      <td>135</td>\n",
       "      <td>129</td>\n",
       "      <td>150</td>\n",
       "      <td>40.776228</td>\n",
       "      <td>-73.982119</td>\n",
       "      <td>4</td>\n",
       "      <td>147</td>\n",
       "      <td>0.542451</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.180805</td>\n",
       "      <td>1.12074</td>\n",
       "      <td>0.900268</td>\n",
       "      <td>-0.950465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>137</td>\n",
       "      <td>135</td>\n",
       "      <td>129</td>\n",
       "      <td>150</td>\n",
       "      <td>164</td>\n",
       "      <td>40.776228</td>\n",
       "      <td>-73.982119</td>\n",
       "      <td>4</td>\n",
       "      <td>162</td>\n",
       "      <td>0.542451</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.180805</td>\n",
       "      <td>1.12074</td>\n",
       "      <td>0.900268</td>\n",
       "      <td>-0.950465</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   ft_5  ft_4  ft_3  ft_2  ft_1        lat        lon  weekday  exp_avg  \\\n",
       "0     0    63   217   189   137  40.776228 -73.982119        4      142   \n",
       "1    63   217   189   137   135  40.776228 -73.982119        4      135   \n",
       "2   217   189   137   135   129  40.776228 -73.982119        4      129   \n",
       "3   189   137   135   129   150  40.776228 -73.982119        4      147   \n",
       "4   137   135   129   150   164  40.776228 -73.982119        4      162   \n",
       "\n",
       "       amp1     amp2     amp3      amp4      amp5  freq1     freq2    freq3  \\\n",
       "0  0.542451  0.80106  0.80106  0.375128  0.375128    0.0 -1.180805  1.12074   \n",
       "1  0.542451  0.80106  0.80106  0.375128  0.375128    0.0 -1.180805  1.12074   \n",
       "2  0.542451  0.80106  0.80106  0.375128  0.375128    0.0 -1.180805  1.12074   \n",
       "3  0.542451  0.80106  0.80106  0.375128  0.375128    0.0 -1.180805  1.12074   \n",
       "4  0.542451  0.80106  0.80106  0.375128  0.375128    0.0 -1.180805  1.12074   \n",
       "\n",
       "      freq4     freq5  \n",
       "0  0.900268 -0.950465  \n",
       "1  0.900268 -0.950465  \n",
       "2  0.900268 -0.950465  \n",
       "3  0.900268 -0.950465  \n",
       "4  0.900268 -0.950465  "
      ]
     },
     "execution_count": 209,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# -----------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task 1:  Applying Regression Models With Foureir Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> Linear Regression </h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "lr_reg=LinearRegression().fit(df_train, tsne_train_output)\n",
    "\n",
    "y_pred = lr_reg.predict(df_test)\n",
    "lr_test_predictions = [round(value) for value in y_pred]\n",
    "y_pred = lr_reg.predict(df_train)\n",
    "lr_train_predictions = [round(value) for value in y_pred]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [],
   "source": [
    "trMape=(mean_absolute_error(tsne_train_output, lr_train_predictions))/(sum(tsne_train_output)/len(tsne_train_output))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {},
   "outputs": [],
   "source": [
    "tstMape=(mean_absolute_error(tsne_test_output, lr_test_predictions))/(sum(tsne_test_output)/len(tsne_test_output))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train MAPE : 0.13274292592069048 . Test MAPE : 0.12868880189916243 \n"
     ]
    }
   ],
   "source": [
    "print('Train MAPE : %s . Test MAPE : %s ' %(trMape,tstMape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> Random Forest </h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,\n",
       "           max_features='sqrt', max_leaf_nodes=None,\n",
       "           min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "           min_samples_leaf=4, min_samples_split=3,\n",
       "           min_weight_fraction_leaf=0.0, n_estimators=40, n_jobs=-1,\n",
       "           oob_score=False, random_state=None, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regr1 = RandomForestRegressor(max_features='sqrt',min_samples_leaf=4,min_samples_split=3,n_estimators=40, n_jobs=-1)\n",
    "regr1.fit(df_train, tsne_train_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predicting on test data using our trained random forest model \n",
    "\n",
    "# the models regr1 is already hyper parameter tuned\n",
    "# the parameters that we got above are found using grid search\n",
    "\n",
    "y_pred = regr1.predict(df_test)\n",
    "rndf_test_predictions = [round(value) for value in y_pred]\n",
    "y_pred = regr1.predict(df_train)\n",
    "rndf_train_predictions = [round(value) for value in y_pred]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "fImp=regr1.feature_importances_\n",
    "col = df_train.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "index=(np.argsort(fImp))[::-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Weights: [0.23256047 0.20371421 0.16139968 0.13044209 0.12451193 0.05210638\n",
      " 0.0292042  0.02135776 0.01701342 0.01196076 0.00992593 0.00170521\n",
      " 0.0015913  0.00114796 0.00041639 0.00039203 0.00030693 0.00024335\n",
      " 0.        ] \n",
      "col: Index(['exp_avg', 'ft_3', 'ft_2', 'ft_1', 'ft_5', 'ft_4', 'amp5', 'amp1',\n",
      "       'amp4', 'amp3', 'amp2', 'lat', 'weekday', 'lon', 'freq4', 'freq5',\n",
      "       'freq3', 'freq2', 'freq1'],\n",
      "      dtype='object') \n"
     ]
    }
   ],
   "source": [
    "print('Weights: %s ' %(fImp[index]))\n",
    "print('col: %s ' %(col[index]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "trMape=(mean_absolute_error(tsne_train_output, rndf_train_predictions))/(sum(tsne_train_output)/len(tsne_train_output))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "tstMape=(mean_absolute_error(tsne_test_output, rndf_test_predictions))/(sum(tsne_test_output)/len(tsne_test_output))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train MAPE : 0.0935870325176211 . Test MAPE : 0.1246361658938258 \n"
     ]
    }
   ],
   "source": [
    "print('Train MAPE : %s . Test MAPE : %s ' %(trMape,tstMape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> XGBOOST </h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "       colsample_bytree=0.8, gamma=0, learning_rate=0.1, max_delta_step=0,\n",
       "       max_depth=3, min_child_weight=3, missing=None, n_estimators=1000,\n",
       "       n_jobs=1, nthread=4, objective='reg:linear', random_state=0,\n",
       "       reg_alpha=200, reg_lambda=200, scale_pos_weight=1, seed=None,\n",
       "       silent=True, subsample=0.8)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_model = xgb.XGBRegressor(\n",
    " learning_rate =0.1,\n",
    " n_estimators=1000,\n",
    " max_depth=3,\n",
    " min_child_weight=3,\n",
    " gamma=0,\n",
    " subsample=0.8,\n",
    " reg_alpha=200, reg_lambda=200,\n",
    " colsample_bytree=0.8,nthread=4)\n",
    "x_model.fit(df_train, tsne_train_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "#predicting with our trained Xg-Boost regressor\n",
    "# the models x_model is already hyper parameter tuned\n",
    "# the parameters that we got above are found using grid search\n",
    "\n",
    "y_pred = x_model.predict(df_test)\n",
    "xgb_test_predictions = [round(value) for value in y_pred]\n",
    "y_pred = x_model.predict(df_train)\n",
    "xgb_train_predictions = [round(value) for value in y_pred]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'exp_avg': 751,\n",
       " 'ft_2': 943,\n",
       " 'ft_1': 954,\n",
       " 'ft_3': 743,\n",
       " 'amp1': 340,\n",
       " 'ft_4': 634,\n",
       " 'amp3': 54,\n",
       " 'amp2': 310,\n",
       " 'lon': 312,\n",
       " 'ft_5': 930,\n",
       " 'lat': 253,\n",
       " 'amp4': 174,\n",
       " 'weekday': 303,\n",
       " 'freq2': 45,\n",
       " 'freq4': 63,\n",
       " 'amp5': 27,\n",
       " 'freq5': 45,\n",
       " 'freq3': 23}"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#feature importances\n",
    "x_model.get_booster().get_score(importance_type='weight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "trMape=(mean_absolute_error(tsne_train_output, xgb_train_predictions))/(sum(tsne_train_output)/len(tsne_train_output))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "tstMape=(mean_absolute_error(tsne_test_output, xgb_test_predictions))/(sum(tsne_test_output)/len(tsne_test_output))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train MAPE : 0.12796546250024254 . Test MAPE : 0.12572155174720648 \n"
     ]
    }
   ],
   "source": [
    "print('Train MAPE : %s . Test MAPE : %s ' %(trMape,tstMape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# -----------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task 2: Hyperparam tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> Error Function To Be Used In GridSearchCV and RandomSearchCV </h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "def myError(y_orig,y_pred):\n",
    "    error=mean_absolute_error(y_orig, y_pred)/(sum(y_orig)/len(y_orig))\n",
    "    return error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics.scorer import make_scorer\n",
    "my_scorer = make_scorer(myError, greater_is_better=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1> Linear Regression </h1>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "tuned_parameters = [{'alpha': [0.001,0.01,0.1,1,10] , 'eta0': [10 ** -4,10 ** -5,10 ** -6,10 ** -7,10 ** -8,10 ** -9,10 ** -10]}]\n",
    "gridSearchModel = GridSearchCV(SGDRegressor(random_state=42,loss='squared_loss',penalty='l2'),tuned_parameters,scoring = my_scorer, cv=5)\n",
    "gridSearchModel.fit(df_train, tsne_train_output)\n",
    "results = gridSearchModel.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best estimator :SGDRegressor(alpha=0.01, average=False, epsilon=0.1, eta0=1e-06,\n",
      "       fit_intercept=True, l1_ratio=0.15, learning_rate='invscaling',\n",
      "       loss='squared_loss', max_iter=None, n_iter=None, penalty='l2',\n",
      "       power_t=0.25, random_state=42, shuffle=True, tol=None, verbose=0,\n",
      "       warm_start=False)\n",
      "\n",
      "Score on train data :0.13320983056258268\n",
      "\n",
      "Score on test data :0.12939730292966478\n"
     ]
    }
   ],
   "source": [
    "print('Best estimator :{}'.format(gridSearchModel.best_estimator_))\n",
    "print('\\nScore on train data :{}'.format(np.abs(gridSearchModel.score(df_train, tsne_train_output))))\n",
    "print('\\nScore on test data :{}'.format(np.abs(gridSearchModel.score(df_test, tsne_test_output))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> Training the model with best hyper params </h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "bestAlpha=gridSearchModel.best_estimator_.alpha\n",
    "bestEta=gridSearchModel.best_estimator_.eta0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best alpha and eta from hyper param tunning is : 0.01 ,  1e-06 \n",
      "Mape for best alpha and eta is -> Test : 12.927407115742634  Train : 13.30796878507937 \n"
     ]
    }
   ],
   "source": [
    "clf=SGDRegressor(random_state=42,loss='squared_loss',penalty='l2',alpha=bestAlpha,eta0=bestEta)\n",
    "clf.fit(df_train,tsne_train_output)\n",
    "\n",
    "y_pred = clf.predict(df_test)\n",
    "test_predictions = [round(value) for value in y_pred]\n",
    "y_pred = clf.predict(df_train)\n",
    "train_predictions = [round(value) for value in y_pred]\n",
    "\n",
    "train_pred=(mean_absolute_error(tsne_train_output,train_predictions))/(sum(tsne_train_output)/len(tsne_train_output))\n",
    "test_pred=(mean_absolute_error(tsne_test_output,test_predictions))/(sum(tsne_test_output)/len(tsne_test_output))\n",
    "\n",
    "print('The best alpha and eta from hyper param tunning is : %s ,  %s ' %(bestAlpha,bestEta))\n",
    "print('Mape for best alpha and eta is -> Test : %s  Train : %s ' %(test_pred*100,train_pred*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[154.0, 141.0, 129.0, 141.0, 156.0, 152.0, 137.0, 136.0, 141.0, 130.0]\n",
      "[135, 129, 150, 164, 152, 131, 138, 147, 127, 138]\n",
      "**************************************************\n",
      "[99.0, 99.0, 113.0, 124.0, 151.0, 152.0, 139.0, 144.0, 161.0, 154.0]\n",
      "[101, 120, 131, 164, 154, 133, 148, 172, 153, 162]\n"
     ]
    }
   ],
   "source": [
    "print(train_predictions[0:10])\n",
    "print(tsne_train_output[0:10])\n",
    "print(\"*\" * 50)\n",
    "print(test_predictions[0:10])\n",
    "print(tsne_test_output[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1> Random Forest </h1>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "min_samples_leaf: [53 17 20] \n",
      "min_samples_split: [9 8 2] \n",
      "n_estimators: [50 79 21] \n"
     ]
    }
   ],
   "source": [
    "min_samples_leaf = np.random.uniform(2, 80, 3).astype(int)\n",
    "min_samples_split = np.random.uniform(2, 10, 3).astype(int)\n",
    "n_estimators = np.random.uniform(20, 80 ,3).astype(int)\n",
    "\n",
    "print('min_samples_leaf: %s ' %(min_samples_leaf))\n",
    "print('min_samples_split: %s ' %(min_samples_split))\n",
    "print('n_estimators: %s ' %(n_estimators))\n",
    "\n",
    "tuned_parameters = {'min_samples_leaf':min_samples_leaf, 'min_samples_split':min_samples_split,'n_estimators': n_estimators}\n",
    "rndSearchRF = RandomizedSearchCV(RandomForestRegressor(max_features='sqrt'),tuned_parameters,scoring = my_scorer, cv=5)\n",
    "rndSearchRF.fit(df_train, tsne_train_output)\n",
    "results = rndSearchRF.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best estimator :RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,\n",
      "           max_features='sqrt', max_leaf_nodes=None,\n",
      "           min_impurity_decrease=0.0, min_impurity_split=None,\n",
      "           min_samples_leaf=17, min_samples_split=2,\n",
      "           min_weight_fraction_leaf=0.0, n_estimators=50, n_jobs=1,\n",
      "           oob_score=False, random_state=None, verbose=0, warm_start=False)\n",
      "\n",
      "Score on train data :0.11773815194192055\n",
      "\n",
      "Score on test data :0.12472312005155084\n"
     ]
    }
   ],
   "source": [
    "print('Best estimator :{}'.format(rndSearchRF.best_estimator_))\n",
    "print('\\nScore on train data :{}'.format(np.abs(rndSearchRF.score(df_train, tsne_train_output))))\n",
    "print('\\nScore on test data :{}'.format(np.abs(rndSearchRF.score(df_test, tsne_test_output))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> Training the model with best hyper params </h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "bestMinSampleLeaf=rndSearchRF.best_estimator_.min_samples_leaf\n",
    "bestSplit=rndSearchRF.best_estimator_.min_samples_split\n",
    "bestEstimators=rndSearchRF.best_estimator_.n_estimators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best min_samples_leaf from hyper param tunning is : 17 \n",
      "The best bestEstimators from hyper param tunning is : 50 \n",
      "The best bestSplit from hyper param tunning is : 2 \n",
      "Mape for best hyperparameters is -> Test : 0.12459220380061853  Train : 0.11759747386976876 \n"
     ]
    }
   ],
   "source": [
    "rf=RandomForestRegressor(max_features='sqrt',min_samples_leaf=bestMinSampleLeaf,min_samples_split=bestSplit,n_estimators=bestEstimators)\n",
    "rf.fit(df_train,tsne_train_output)\n",
    "\n",
    "y_pred = rf.predict(df_test)\n",
    "test_predictions = [round(value) for value in y_pred]\n",
    "y_pred = rf.predict(df_train)\n",
    "train_predictions = [round(value) for value in y_pred]\n",
    "\n",
    "train_pred=(mean_absolute_error(tsne_train_output,train_predictions))/(sum(tsne_train_output)/len(tsne_train_output))\n",
    "test_pred=(mean_absolute_error(tsne_test_output,test_predictions))/(sum(tsne_test_output)/len(tsne_test_output))\n",
    "\n",
    "print('The best min_samples_leaf from hyper param tunning is : %s ' %(bestMinSampleLeaf))\n",
    "print('The best bestEstimators from hyper param tunning is : %s ' %(bestEstimators))\n",
    "print('The best bestSplit from hyper param tunning is : %s ' %(bestSplit))\n",
    "print('Mape for best hyperparameters is -> Test : %s  Train : %s ' %(test_pred,train_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[128.0, 133.0, 133.0, 145.0, 150.0, 143.0, 140.0, 138.0, 140.0, 132.0]\n",
      "[135, 129, 150, 164, 152, 131, 138, 147, 127, 138]\n",
      "**************************************************\n",
      "[104.0, 105.0, 111.0, 125.0, 151.0, 146.0, 138.0, 143.0, 156.0, 151.0]\n",
      "[101, 120, 131, 164, 154, 133, 148, 172, 153, 162]\n"
     ]
    }
   ],
   "source": [
    "print(train_predictions[0:10])\n",
    "print(tsne_train_output[0:10])\n",
    "print(\"*\" * 50)\n",
    "print(test_predictions[0:10])\n",
    "print(tsne_test_output[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>XGBOOST</h1>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_child_weight = np.random.uniform(2, 8, 2).astype(int)\n",
    "max_depth = np.random.uniform(2, 8, 2).astype(int)\n",
    "n_estimators = np.random.uniform(100, 1000 ,2).astype(int)\n",
    "reg_alpha = np.random.uniform(100, 500 ,2).astype(int)\n",
    "reg_lambda = np.random.uniform(100, 500 ,2).astype(int)\n",
    "\n",
    "tuned_parameters = {\n",
    "    'min_child_weight':min_child_weight,\n",
    "    'max_depth':max_depth,\n",
    "    'n_estimators': n_estimators,\n",
    "    'reg_alpha':reg_alpha,\n",
    "    'reg_lambda': reg_lambda\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_model = xgb.XGBRegressor(\n",
    " learning_rate =0.1,\n",
    " gamma=0,\n",
    " subsample=0.8,\n",
    " colsample_bytree=0.8,nthread=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=5, error_score='raise',\n",
       "          estimator=XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
       "       colsample_bytree=0.8, gamma=0, learning_rate=0.1, max_delta_step=0,\n",
       "       max_depth=3, min_child_weight=1, missing=None, n_estimators=100,\n",
       "       n_jobs=1, nthread=4, objective='reg:linear', random_state=0,\n",
       "       reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,\n",
       "       silent=True, subsample=0.8),\n",
       "          fit_params=None, iid=True, n_iter=10, n_jobs=1,\n",
       "          param_distributions={'min_child_weight': array([4, 7]), 'max_depth': array([7, 2]), 'n_estimators': array([877, 795]), 'reg_alpha': array([333, 463]), 'reg_lambda': array([104, 229])},\n",
       "          pre_dispatch='2*n_jobs', random_state=None, refit=True,\n",
       "          return_train_score='warn',\n",
       "          scoring=make_scorer(myError, greater_is_better=False), verbose=0)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgbModelRS = RandomizedSearchCV(x_model,param_distributions=tuned_parameters,scoring = my_scorer, cv=5)\n",
    "xgbModelRS.fit(df_train,tsne_train_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best estimator :XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
      "       colsample_bytree=0.8, gamma=0, learning_rate=0.1, max_delta_step=0,\n",
      "       max_depth=7, min_child_weight=7, missing=None, n_estimators=795,\n",
      "       n_jobs=1, nthread=4, objective='reg:linear', random_state=0,\n",
      "       reg_alpha=333, reg_lambda=104, scale_pos_weight=1, seed=None,\n",
      "       silent=True, subsample=0.8)\n",
      "\n",
      "Score on train data :0.12044026413149364\n",
      "\n",
      "Score on test data :0.12391724860838926\n"
     ]
    }
   ],
   "source": [
    "print('Best estimator :{}'.format(xgbModelRS.best_estimator_))\n",
    "print('\\nScore on train data :{}'.format(np.abs(xgbModelRS.score(df_train1, tsne_train_output))))\n",
    "print('\\nScore on test data :{}'.format(np.abs(xgbModelRS.score(df_test1, tsne_test_output))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> Training the model with best hyper params </h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "bestdepth=xgbModelRS.best_estimator_.max_depth\n",
    "bestChildWeight=xgbModelRS.best_estimator_.min_child_weight\n",
    "bestEstimators=xgbModelRS.best_estimator_.n_estimators\n",
    "bestlambda=xgbModelRS.best_estimator_.reg_lambda\n",
    "bestalpha=xgbModelRS.best_estimator_.reg_alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgbBest = xgb.XGBRegressor(\n",
    " learning_rate =0.1,\n",
    " max_depth=bestdepth,\n",
    " min_child_weight = bestChildWeight,\n",
    " n_estimators = bestEstimators,\n",
    " reg_alpha = bestalpha,\n",
    " reg_lambda = bestlambda,   \n",
    " gamma=0,\n",
    " subsample=0.8,\n",
    " colsample_bytree=0.8,nthread=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best bestdepth from hyper param tunning is : 7 \n",
      "The best bestEstimators from hyper param tunning is : 795 \n",
      "The best bestChildWeight from hyper param tunning is : 7 \n",
      "The best bestlambda from hyper param tunning is : 104 \n",
      "The best bestalpha from hyper param tunning is : 333 \n",
      "Mape for best hyperparameters is -> Test : 0.12384628176261234  Train : 0.1203148256321562 \n"
     ]
    }
   ],
   "source": [
    "xgbBest.fit(df_train,tsne_train_output)\n",
    "\n",
    "y_pred = xgbBest.predict(df_test)\n",
    "test_predictions = [round(value) for value in y_pred]\n",
    "y_pred = xgbBest.predict(df_train)\n",
    "train_predictions = [round(value) for value in y_pred]\n",
    "\n",
    "train_pred=(mean_absolute_error(tsne_train_output,train_predictions))/(sum(tsne_train_output)/len(tsne_train_output))\n",
    "test_pred=(mean_absolute_error(tsne_test_output,test_predictions))/(sum(tsne_test_output)/len(tsne_test_output))\n",
    "\n",
    "print('The best bestdepth from hyper param tunning is : %s ' %(bestdepth))\n",
    "print('The best bestEstimators from hyper param tunning is : %s ' %(bestEstimators))\n",
    "print('The best bestChildWeight from hyper param tunning is : %s ' %(bestChildWeight))\n",
    "print('The best bestlambda from hyper param tunning is : %s ' %(bestlambda))\n",
    "print('The best bestalpha from hyper param tunning is : %s ' %(bestalpha))\n",
    "print('Mape for best hyperparameters is -> Test : %s  Train : %s ' %(test_pred,train_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'exp_avg': 5797,\n",
       " 'ft_5': 8570,\n",
       " 'amp1': 2933,\n",
       " 'ft_3': 7474,\n",
       " 'ft_4': 7913,\n",
       " 'amp2': 3004,\n",
       " 'amp4': 2481,\n",
       " 'ft_1': 7321,\n",
       " 'ft_2': 7732,\n",
       " 'lat': 2231,\n",
       " 'lon': 2364,\n",
       " 'freq4': 1057,\n",
       " 'amp3': 695,\n",
       " 'freq5': 668,\n",
       " 'amp5': 497,\n",
       " 'freq2': 776,\n",
       " 'weekday': 3550,\n",
       " 'freq3': 407}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#feature importances\n",
    "xgbBest.get_booster().get_score(importance_type='weight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[97.0, 137.0, 132.0, 138.0, 149.0, 149.0, 142.0, 139.0, 138.0, 133.0]\n",
      "[135, 129, 150, 164, 152, 131, 138, 147, 127, 138]\n",
      "**************************************************\n",
      "[105.0, 104.0, 113.0, 121.0, 148.0, 147.0, 142.0, 143.0, 151.0, 150.0]\n",
      "[101, 120, 131, 164, 154, 133, 148, 172, 153, 162]\n"
     ]
    }
   ],
   "source": [
    "print(train_predictions[0:10])\n",
    "print(tsne_train_output[0:10])\n",
    "print(\"*\" * 50)\n",
    "print(test_predictions[0:10])\n",
    "print(tsne_test_output[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# -----------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task 3 : Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding hour of  day  to each row \n",
    "\n",
    "This will improve the model as the number of pickups have a dependency on the time of the day\n",
    "\n",
    "During morning and evening hours the pickups will be more in comaprison to the afternoon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = pd.read_pickle('df_train.pkl')\n",
    "df_test = pd.read_pickle('df_test.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('tsne_test_output.pkl', 'rb') as f:\n",
    "    tsne_test_output = pickle.load(f)\n",
    "with open('tsne_train_output.pkl', 'rb') as f:\n",
    "    tsne_train_output = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ft_5</th>\n",
       "      <th>ft_4</th>\n",
       "      <th>ft_3</th>\n",
       "      <th>ft_2</th>\n",
       "      <th>ft_1</th>\n",
       "      <th>lat</th>\n",
       "      <th>lon</th>\n",
       "      <th>weekday</th>\n",
       "      <th>exp_avg</th>\n",
       "      <th>amp1</th>\n",
       "      <th>amp2</th>\n",
       "      <th>amp3</th>\n",
       "      <th>amp4</th>\n",
       "      <th>amp5</th>\n",
       "      <th>freq1</th>\n",
       "      <th>freq2</th>\n",
       "      <th>freq3</th>\n",
       "      <th>freq4</th>\n",
       "      <th>freq5</th>\n",
       "      <th>hours</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>63</td>\n",
       "      <td>217</td>\n",
       "      <td>189</td>\n",
       "      <td>137</td>\n",
       "      <td>40.776228</td>\n",
       "      <td>-73.982119</td>\n",
       "      <td>4</td>\n",
       "      <td>150</td>\n",
       "      <td>0.542451</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.180805</td>\n",
       "      <td>1.12074</td>\n",
       "      <td>0.900268</td>\n",
       "      <td>-0.950465</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>63</td>\n",
       "      <td>217</td>\n",
       "      <td>189</td>\n",
       "      <td>137</td>\n",
       "      <td>135</td>\n",
       "      <td>40.776228</td>\n",
       "      <td>-73.982119</td>\n",
       "      <td>4</td>\n",
       "      <td>139</td>\n",
       "      <td>0.542451</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.180805</td>\n",
       "      <td>1.12074</td>\n",
       "      <td>0.900268</td>\n",
       "      <td>-0.950465</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>217</td>\n",
       "      <td>189</td>\n",
       "      <td>137</td>\n",
       "      <td>135</td>\n",
       "      <td>129</td>\n",
       "      <td>40.776228</td>\n",
       "      <td>-73.982119</td>\n",
       "      <td>4</td>\n",
       "      <td>132</td>\n",
       "      <td>0.542451</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.180805</td>\n",
       "      <td>1.12074</td>\n",
       "      <td>0.900268</td>\n",
       "      <td>-0.950465</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>189</td>\n",
       "      <td>137</td>\n",
       "      <td>135</td>\n",
       "      <td>129</td>\n",
       "      <td>150</td>\n",
       "      <td>40.776228</td>\n",
       "      <td>-73.982119</td>\n",
       "      <td>4</td>\n",
       "      <td>144</td>\n",
       "      <td>0.542451</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.180805</td>\n",
       "      <td>1.12074</td>\n",
       "      <td>0.900268</td>\n",
       "      <td>-0.950465</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>137</td>\n",
       "      <td>135</td>\n",
       "      <td>129</td>\n",
       "      <td>150</td>\n",
       "      <td>164</td>\n",
       "      <td>40.776228</td>\n",
       "      <td>-73.982119</td>\n",
       "      <td>4</td>\n",
       "      <td>158</td>\n",
       "      <td>0.542451</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.180805</td>\n",
       "      <td>1.12074</td>\n",
       "      <td>0.900268</td>\n",
       "      <td>-0.950465</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   ft_5  ft_4  ft_3  ft_2  ft_1        lat        lon  weekday  exp_avg  \\\n",
       "0     0    63   217   189   137  40.776228 -73.982119        4      150   \n",
       "1    63   217   189   137   135  40.776228 -73.982119        4      139   \n",
       "2   217   189   137   135   129  40.776228 -73.982119        4      132   \n",
       "3   189   137   135   129   150  40.776228 -73.982119        4      144   \n",
       "4   137   135   129   150   164  40.776228 -73.982119        4      158   \n",
       "\n",
       "       amp1     amp2     amp3      amp4      amp5  freq1     freq2    freq3  \\\n",
       "0  0.542451  0.80106  0.80106  0.375128  0.375128    0.0 -1.180805  1.12074   \n",
       "1  0.542451  0.80106  0.80106  0.375128  0.375128    0.0 -1.180805  1.12074   \n",
       "2  0.542451  0.80106  0.80106  0.375128  0.375128    0.0 -1.180805  1.12074   \n",
       "3  0.542451  0.80106  0.80106  0.375128  0.375128    0.0 -1.180805  1.12074   \n",
       "4  0.542451  0.80106  0.80106  0.375128  0.375128    0.0 -1.180805  1.12074   \n",
       "\n",
       "      freq4     freq5  hours  \n",
       "0  0.900268 -0.950465      0  \n",
       "1  0.900268 -0.950465      1  \n",
       "2  0.900268 -0.950465      1  \n",
       "3  0.900268 -0.950465      1  \n",
       "4  0.900268 -0.950465      1  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "oneDay=[]\n",
    "for i in range(24):\n",
    "    oneDay.extend([i] *6)  # As there are 144 bins in a day and 6 bins in a hour \n",
    "oneRegionTotal = oneDay * 31 + oneDay * 29 + oneDay * 31  # Jan-31 Feb -29 Mar-31\n",
    "oneRegionTotal = oneRegionTotal[5:]  #Removing the first 5 bin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "totalHourClusters=[]\n",
    "for i in range(40):\n",
    "    totalHourClusters.append(oneRegionTotal)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsne_train_hours_flat = [i[:9169] for i in totalHourClusters]\n",
    "tsne_test_hours_flat = [i[9169:] for i in totalHourClusters]\n",
    "\n",
    "tsne_train_hours = sum(tsne_train_hours_flat, [])\n",
    "tsne_test_hours = sum(tsne_test_hours_flat, [])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train['hours'] = tsne_train_hours\n",
    "df_test['hours'] = tsne_test_hours"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ft_5</th>\n",
       "      <th>ft_4</th>\n",
       "      <th>ft_3</th>\n",
       "      <th>ft_2</th>\n",
       "      <th>ft_1</th>\n",
       "      <th>lat</th>\n",
       "      <th>lon</th>\n",
       "      <th>weekday</th>\n",
       "      <th>exp_avg</th>\n",
       "      <th>amp1</th>\n",
       "      <th>amp2</th>\n",
       "      <th>amp3</th>\n",
       "      <th>amp4</th>\n",
       "      <th>amp5</th>\n",
       "      <th>freq1</th>\n",
       "      <th>freq2</th>\n",
       "      <th>freq3</th>\n",
       "      <th>freq4</th>\n",
       "      <th>freq5</th>\n",
       "      <th>hours</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>63</td>\n",
       "      <td>217</td>\n",
       "      <td>189</td>\n",
       "      <td>137</td>\n",
       "      <td>40.776228</td>\n",
       "      <td>-73.982119</td>\n",
       "      <td>4</td>\n",
       "      <td>150</td>\n",
       "      <td>0.542451</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.180805</td>\n",
       "      <td>1.12074</td>\n",
       "      <td>0.900268</td>\n",
       "      <td>-0.950465</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>63</td>\n",
       "      <td>217</td>\n",
       "      <td>189</td>\n",
       "      <td>137</td>\n",
       "      <td>135</td>\n",
       "      <td>40.776228</td>\n",
       "      <td>-73.982119</td>\n",
       "      <td>4</td>\n",
       "      <td>139</td>\n",
       "      <td>0.542451</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.180805</td>\n",
       "      <td>1.12074</td>\n",
       "      <td>0.900268</td>\n",
       "      <td>-0.950465</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>217</td>\n",
       "      <td>189</td>\n",
       "      <td>137</td>\n",
       "      <td>135</td>\n",
       "      <td>129</td>\n",
       "      <td>40.776228</td>\n",
       "      <td>-73.982119</td>\n",
       "      <td>4</td>\n",
       "      <td>132</td>\n",
       "      <td>0.542451</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.180805</td>\n",
       "      <td>1.12074</td>\n",
       "      <td>0.900268</td>\n",
       "      <td>-0.950465</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>189</td>\n",
       "      <td>137</td>\n",
       "      <td>135</td>\n",
       "      <td>129</td>\n",
       "      <td>150</td>\n",
       "      <td>40.776228</td>\n",
       "      <td>-73.982119</td>\n",
       "      <td>4</td>\n",
       "      <td>144</td>\n",
       "      <td>0.542451</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.180805</td>\n",
       "      <td>1.12074</td>\n",
       "      <td>0.900268</td>\n",
       "      <td>-0.950465</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>137</td>\n",
       "      <td>135</td>\n",
       "      <td>129</td>\n",
       "      <td>150</td>\n",
       "      <td>164</td>\n",
       "      <td>40.776228</td>\n",
       "      <td>-73.982119</td>\n",
       "      <td>4</td>\n",
       "      <td>158</td>\n",
       "      <td>0.542451</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.80106</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.375128</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.180805</td>\n",
       "      <td>1.12074</td>\n",
       "      <td>0.900268</td>\n",
       "      <td>-0.950465</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   ft_5  ft_4  ft_3  ft_2  ft_1        lat        lon  weekday  exp_avg  \\\n",
       "0     0    63   217   189   137  40.776228 -73.982119        4      150   \n",
       "1    63   217   189   137   135  40.776228 -73.982119        4      139   \n",
       "2   217   189   137   135   129  40.776228 -73.982119        4      132   \n",
       "3   189   137   135   129   150  40.776228 -73.982119        4      144   \n",
       "4   137   135   129   150   164  40.776228 -73.982119        4      158   \n",
       "\n",
       "       amp1     amp2     amp3      amp4      amp5  freq1     freq2    freq3  \\\n",
       "0  0.542451  0.80106  0.80106  0.375128  0.375128    0.0 -1.180805  1.12074   \n",
       "1  0.542451  0.80106  0.80106  0.375128  0.375128    0.0 -1.180805  1.12074   \n",
       "2  0.542451  0.80106  0.80106  0.375128  0.375128    0.0 -1.180805  1.12074   \n",
       "3  0.542451  0.80106  0.80106  0.375128  0.375128    0.0 -1.180805  1.12074   \n",
       "4  0.542451  0.80106  0.80106  0.375128  0.375128    0.0 -1.180805  1.12074   \n",
       "\n",
       "      freq4     freq5  hours  \n",
       "0  0.900268 -0.950465      0  \n",
       "1  0.900268 -0.950465      1  \n",
       "2  0.900268 -0.950465      1  \n",
       "3  0.900268 -0.950465      1  \n",
       "4  0.900268 -0.950465      1  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf=RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,\n",
    "           max_features='sqrt', max_leaf_nodes=None,\n",
    "           min_impurity_decrease=0.0, min_impurity_split=None,\n",
    "           min_samples_leaf=17, min_samples_split=2,\n",
    "           min_weight_fraction_leaf=0.0, n_estimators=50, n_jobs=1,\n",
    "           oob_score=False, random_state=None, verbose=0, warm_start=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mape for best hyperparameters is -> Test : 0.119325545034386  Train : 0.11240538098445352 \n"
     ]
    }
   ],
   "source": [
    "rf.fit(df_train,tsne_train_output)\n",
    "\n",
    "y_pred = rf.predict(df_test)\n",
    "test_predictions = [round(value) for value in y_pred]\n",
    "y_pred = rf.predict(df_train)\n",
    "train_predictions = [round(value) for value in y_pred]\n",
    "\n",
    "train_pred=(mean_absolute_error(tsne_train_output,train_predictions))/(sum(tsne_train_output)/len(tsne_train_output))\n",
    "test_pred=(mean_absolute_error(tsne_test_output,test_predictions))/(sum(tsne_test_output)/len(tsne_test_output))\n",
    "\n",
    "print('Mape for best hyperparameters is -> Test : %s  Train : %s ' %(test_pred,train_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Weights: [3.14459270e-01 1.99040490e-01 1.71786422e-01 9.81518248e-02\n",
      " 7.33022740e-02 4.85752397e-02 2.38011340e-02 1.69503480e-02\n",
      " 1.57608877e-02 1.40856037e-02 1.02099195e-02 9.63251971e-03\n",
      " 1.48677887e-03 1.38449254e-03 7.53425742e-04 2.73784471e-04\n",
      " 1.35848483e-04 1.15373473e-04 9.43631081e-05 0.00000000e+00] \n",
      "col: Index(['exp_avg', 'ft_1', 'ft_3', 'ft_2', 'ft_5', 'ft_4', 'amp1', 'amp5',\n",
      "       'amp2', 'amp4', 'amp3', 'hours', 'lon', 'lat', 'weekday', 'freq5',\n",
      "       'freq4', 'freq3', 'freq2', 'freq1'],\n",
      "      dtype='object') \n"
     ]
    }
   ],
   "source": [
    "fImp=rf.feature_importances_\n",
    "col = df_train.columns\n",
    "index=(np.argsort(fImp))[::-1]\n",
    "print('Weights: %s ' %(fImp[index]))\n",
    "print('col: %s ' %(col[index]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}