NYC Taxi Data/01 Download Raw Parquet Files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "11f516d3-dd0c-4a3d-80cd-5a01437efec8",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "![NYC Logo](https://www.nyc.gov/assets/tlc/images/content/pages/home/nyc-tlc-logo.svg)\n",
    "\n",
    "# Downloading NYC Taxi Data\n",
    "\n",
    "When working on a Spark project, it's often handy to have a really big dataset to work with.  One very popular, very large dataset is the\n",
    "[NYC Taxi](https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page) dataset.  This data is provided by the New York City Taxi & Limousine Commission.  It contains data\n",
    "on individual taxi trips taken between 2009 and 2023.  The data is provided as Parquet files and is stored in a public Amazon S3 bucket.\n",
    "\n",
    "To use the data in Spark, you will need to download it to your own HDFS-compliant storage.  The script below assumes that you've mounted your target storage location in the DBFS.\n",
    "\n",
    "Note that you can run this code multiple times.  It will skip any files that you've already downloaded and only add new files that it discovers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "83892f07-7448-4364-b624-eb974b1e0d58",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "# The location where you've mounted the target storage\n",
    "# IMPORTANT: This script will be accessing the files using native Python libraries, NOT Spark libraries.  Therefore, the path must start with \"/dbfs/\"\n",
    "output_path = \"/dbfs/mnt/taxi/parquet-raw\"\n",
    "\n",
    "# The most recent month for which data is available.  (Check the web site linked above to see if new data has been made available.)\n",
    "max_month = \"2023-02\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "37600e88-07aa-4777-be99-af880602784f",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Next, we build a Python list of all possible year/month combinations.  We'll filter this down to the exact range of dates that we need later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "ee066340-7026-4a54-a18d-728e4175dae6",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "all_months = []\n",
    "for y in range(2000, 2050):\n",
    "  for m in range(1, 13):\n",
    "    month = \"{0}-{1:02d}\".format(y, m)\n",
    "    all_months.append(month)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "8e594cde-43ec-4b91-aa55-53fb1b9fdef0",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "The NYC Taxi dataset is divided by taxi type:\n",
    " - \"yellow\" taxicabs can be hailed by people on the street\n",
    " - \"fhv\" (or \"For-Hire Vehicles\") are ride-sharing services, black car services, or luxury limo services.  Rides in these vehicles must be pre-arranged with a dispatching service.  They cannot pickup street hails.\n",
    " - \"green\" taxicabs were introduced in 2013.  They are a hybrid between yellow taxis and FHV's.  They can accept street hails in certain parts of the city that are often under-served by yellow taxis.  They can also be pre-arranged by a dispatching service.\n",
    " - \"fhvhv\" (or \"High Volume For-Hire Vehicles\")\n",
    "\n",
    "Each taxi type has a different schema and covers a different range of months.  For example, data for yellow taxis goes back to 2009, but FHV data is only provided from 2015 to the present.\n",
    "\n",
    "Below, we declare the types of taxis and the date ranges for which their files are available.  If a new type of taxi data becomes available, just add it here, and the rest of\n",
    "this script should pick it up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "41e30c0f-e152-4686-966e-6fb2e900b19b",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "taxi_types = {\n",
    "  'yellow': (\"2009-01\", max_month),\n",
    "  'green':  (\"2013-08\", max_month),\n",
    "  'fhv':    (\"2015-01\", max_month),\n",
    "  'fhvhv':  (\"2019-02\", max_month)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "c9fe6ce8-f611-4fc4-970b-6815c4819371",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "## Downloading the Data\n",
    "\n",
    "Using Python, we now loop through the different types of taxis and download each type's monthly CSV files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "0fc7c278-5039-411d-ad9c-038581519e65",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing \"yellow\"  (2009-01  through  2023-02)\n   Skipped 2009-01 (file already exists)\n   Skipped 2009-02 (file already exists)\n   Skipped 2009-03 (file already exists)\n   Skipped 2009-04 (file already exists)\n   Skipped 2009-05 (file already exists)\n   Skipped 2009-06 (file already exists)\n   Skipped 2009-07 (file already exists)\n   Skipped 2009-08 (file already exists)\n   Skipped 2009-09 (file already exists)\n   Skipped 2009-10 (file already exists)\n   Skipped 2009-11 (file already exists)\n   Skipped 2009-12 (file already exists)\n   Skipped 2010-01 (file already exists)\n   Skipped 2010-02 (file already exists)\n   Skipped 2010-03 (file already exists)\n   Skipped 2010-04 (file already exists)\n   Skipped 2010-05 (file already exists)\n   Skipped 2010-06 (file already exists)\n   Skipped 2010-07 (file already exists)\n   Skipped 2010-08 (file already exists)\n   Skipped 2010-09 (file already exists)\n   Skipped 2010-10 (file already exists)\n   Skipped 2010-11 (file already exists)\n   Skipped 2010-12 (file already exists)\n   Skipped 2011-01 (file already exists)\n   Skipped 2011-02 (file already exists)\n   Skipped 2011-03 (file already exists)\n   Skipped 2011-04 (file already exists)\n   Skipped 2011-05 (file already exists)\n   Skipped 2011-06 (file already exists)\n   Skipped 2011-07 (file already exists)\n   Skipped 2011-08 (file already exists)\n   Skipped 2011-09 (file already exists)\n   Skipped 2011-10 (file already exists)\n   Skipped 2011-11 (file already exists)\n   Skipped 2011-12 (file already exists)\n   Skipped 2012-01 (file already exists)\n   Skipped 2012-02 (file already exists)\n   Skipped 2012-03 (file already exists)\n   Skipped 2012-04 (file already exists)\n   Skipped 2012-05 (file already exists)\n   Skipped 2012-06 (file already exists)\n   Skipped 2012-07 (file already exists)\n   Skipped 2012-08 (file already exists)\n   Skipped 2012-09 (file already exists)\n   Skipped 2012-10 (file already exists)\n   Skipped 2012-11 (file already exists)\n   Skipped 2012-12 (file already exists)\n   Skipped 2013-01 (file already exists)\n   Skipped 2013-02 (file already exists)\n   Skipped 2013-03 (file already exists)\n   Skipped 2013-04 (file already exists)\n   Skipped 2013-05 (file already exists)\n   Skipped 2013-06 (file already exists)\n   Skipped 2013-07 (file already exists)\n   Skipped 2013-08 (file already exists)\n   Skipped 2013-09 (file already exists)\n   Skipped 2013-10 (file already exists)\n   Skipped 2013-11 (file already exists)\n   Skipped 2013-12 (file already exists)\n   Skipped 2014-01 (file already exists)\n   Skipped 2014-02 (file already exists)\n   Skipped 2014-03 (file already exists)\n   Skipped 2014-04 (file already exists)\n   Skipped 2014-05 (file already exists)\n   Skipped 2014-06 (file already exists)\n   Skipped 2014-07 (file already exists)\n   Skipped 2014-08 (file already exists)\n   Skipped 2014-09 (file already exists)\n   Skipped 2014-10 (file already exists)\n   Skipped 2014-11 (file already exists)\n   Skipped 2014-12 (file already exists)\n   Skipped 2015-01 (file already exists)\n   Skipped 2015-02 (file already exists)\n   Skipped 2015-03 (file already exists)\n   Skipped 2015-04 (file already exists)\n   Skipped 2015-05 (file already exists)\n   Skipped 2015-06 (file already exists)\n   Skipped 2015-07 (file already exists)\n   Skipped 2015-08 (file already exists)\n   Skipped 2015-09 (file already exists)\n   Skipped 2015-10 (file already exists)\n   Skipped 2015-11 (file already exists)\n   Skipped 2015-12 (file already exists)\n   Skipped 2016-01 (file already exists)\n   Skipped 2016-02 (file already exists)\n   Skipped 2016-03 (file already exists)\n   Skipped 2016-04 (file already exists)\n   Skipped 2016-05 (file already exists)\n   Skipped 2016-06 (file already exists)\n   Skipped 2016-07 (file already exists)\n   Skipped 2016-08 (file already exists)\n   Skipped 2016-09 (file already exists)\n   Skipped 2016-10 (file already exists)\n   Skipped 2016-11 (file already exists)\n   Skipped 2016-12 (file already exists)\n   Skipped 2017-01 (file already exists)\n   Skipped 2017-02 (file already exists)\n   Skipped 2017-03 (file already exists)\n   Skipped 2017-04 (file already exists)\n   Skipped 2017-05 (file already exists)\n   Skipped 2017-06 (file already exists)\n   Skipped 2017-07 (file already exists)\n   Skipped 2017-08 (file already exists)\n   Skipped 2017-09 (file already exists)\n   Skipped 2017-10 (file already exists)\n   Skipped 2017-11 (file already exists)\n   Skipped 2017-12 (file already exists)\n   Skipped 2018-01 (file already exists)\n   Skipped 2018-02 (file already exists)\n   Skipped 2018-03 (file already exists)\n   Skipped 2018-04 (file already exists)\n   Skipped 2018-05 (file already exists)\n   Skipped 2018-06 (file already exists)\n   Skipped 2018-07 (file already exists)\n   Skipped 2018-08 (file already exists)\n   Skipped 2018-09 (file already exists)\n   Skipped 2018-10 (file already exists)\n   Skipped 2018-11 (file already exists)\n   Skipped 2018-12 (file already exists)\n   Skipped 2019-01 (file already exists)\n   Skipped 2019-02 (file already exists)\n   Skipped 2019-03 (file already exists)\n   Skipped 2019-04 (file already exists)\n   Skipped 2019-05 (file already exists)\n   Skipped 2019-06 (file already exists)\n   Skipped 2019-07 (file already exists)\n   Skipped 2019-08 (file already exists)\n   Skipped 2019-09 (file already exists)\n   Skipped 2019-10 (file already exists)\n   Skipped 2019-11 (file already exists)\n   Skipped 2019-12 (file already exists)\n   Skipped 2020-01 (file already exists)\n   Skipped 2020-02 (file already exists)\n   Skipped 2020-03 (file already exists)\n   Skipped 2020-04 (file already exists)\n   Skipped 2020-05 (file already exists)\n   Skipped 2020-06 (file already exists)\n   Skipped 2020-07 (file already exists)\n   Skipped 2020-08 (file already exists)\n   Skipped 2020-09 (file already exists)\n   Skipped 2020-10 (file already exists)\n   Skipped 2020-11 (file already exists)\n   Skipped 2020-12 (file already exists)\n   Skipped 2021-01 (file already exists)\n   Skipped 2021-02 (file already exists)\n   Skipped 2021-03 (file already exists)\n   Skipped 2021-04 (file already exists)\n   Skipped 2021-05 (file already exists)\n   Skipped 2021-06 (file already exists)\n   Skipped 2021-07 (file already exists)\n   Skipped 2021-08 (file already exists)\n   Skipped 2021-09 (file already exists)\n   Skipped 2021-10 (file already exists)\n   Skipped 2021-11 (file already exists)\n   Skipped 2021-12 (file already exists)\n   Skipped 2022-01 (file already exists)\n   Skipped 2022-02 (file already exists)\n   Skipped 2022-03 (file already exists)\n   Skipped 2022-04 (file already exists)\n   Skipped 2022-05 (file already exists)\n   Skipped 2022-06 (file already exists)\n   Skipped 2022-07 (file already exists)\n   Skipped 2022-08 (file already exists)\n   Skipped 2022-09 (file already exists)\n   Skipped 2022-10 (file already exists)\n   Skipped 2022-11 (file already exists)\n   Downloaded 2022-12\n   Downloaded 2023-01\n   Downloaded 2023-02\n============================================================\nProcessing \"green\"  (2013-08  through  2023-02)\n !!Could not download 2013-08\n      HTTP Error 403: Forbidden\n !!Could not download 2013-09\n      HTTP Error 403: Forbidden\n !!Could not download 2013-10\n      HTTP Error 403: Forbidden\n !!Could not download 2013-11\n      HTTP Error 403: Forbidden\n !!Could not download 2013-12\n      HTTP Error 403: Forbidden\n   Skipped 2014-01 (file already exists)\n   Skipped 2014-02 (file already exists)\n   Skipped 2014-03 (file already exists)\n   Skipped 2014-04 (file already exists)\n   Skipped 2014-05 (file already exists)\n   Skipped 2014-06 (file already exists)\n   Skipped 2014-07 (file already exists)\n   Skipped 2014-08 (file already exists)\n   Skipped 2014-09 (file already exists)\n   Skipped 2014-10 (file already exists)\n   Skipped 2014-11 (file already exists)\n   Skipped 2014-12 (file already exists)\n   Skipped 2015-01 (file already exists)\n   Skipped 2015-02 (file already exists)\n   Skipped 2015-03 (file already exists)\n   Skipped 2015-04 (file already exists)\n   Skipped 2015-05 (file already exists)\n   Skipped 2015-06 (file already exists)\n   Skipped 2015-07 (file already exists)\n   Skipped 2015-08 (file already exists)\n   Skipped 2015-09 (file already exists)\n   Skipped 2015-10 (file already exists)\n   Skipped 2015-11 (file already exists)\n   Skipped 2015-12 (file already exists)\n   Skipped 2016-01 (file already exists)\n   Skipped 2016-02 (file already exists)\n   Skipped 2016-03 (file already exists)\n   Skipped 2016-04 (file already exists)\n   Skipped 2016-05 (file already exists)\n   Skipped 2016-06 (file already exists)\n   Skipped 2016-07 (file already exists)\n   Skipped 2016-08 (file already exists)\n   Skipped 2016-09 (file already exists)\n   Skipped 2016-10 (file already exists)\n   Skipped 2016-11 (file already exists)\n   Skipped 2016-12 (file already exists)\n   Skipped 2017-01 (file already exists)\n   Skipped 2017-02 (file already exists)\n   Skipped 2017-03 (file already exists)\n   Skipped 2017-04 (file already exists)\n   Skipped 2017-05 (file already exists)\n   Skipped 2017-06 (file already exists)\n   Skipped 2017-07 (file already exists)\n   Skipped 2017-08 (file already exists)\n   Skipped 2017-09 (file already exists)\n   Skipped 2017-10 (file already exists)\n   Skipped 2017-11 (file already exists)\n   Skipped 2017-12 (file already exists)\n   Skipped 2018-01 (file already exists)\n   Skipped 2018-02 (file already exists)\n   Skipped 2018-03 (file already exists)\n   Skipped 2018-04 (file already exists)\n   Skipped 2018-05 (file already exists)\n   Skipped 2018-06 (file already exists)\n   Skipped 2018-07 (file already exists)\n   Skipped 2018-08 (file already exists)\n   Skipped 2018-09 (file already exists)\n   Skipped 2018-10 (file already exists)\n   Skipped 2018-11 (file already exists)\n   Skipped 2018-12 (file already exists)\n   Skipped 2019-01 (file already exists)\n   Skipped 2019-02 (file already exists)\n   Skipped 2019-03 (file already exists)\n   Skipped 2019-04 (file already exists)\n   Skipped 2019-05 (file already exists)\n   Skipped 2019-06 (file already exists)\n   Skipped 2019-07 (file already exists)\n   Skipped 2019-08 (file already exists)\n   Skipped 2019-09 (file already exists)\n   Skipped 2019-10 (file already exists)\n   Skipped 2019-11 (file already exists)\n   Skipped 2019-12 (file already exists)\n   Skipped 2020-01 (file already exists)\n   Skipped 2020-02 (file already exists)\n   Skipped 2020-03 (file already exists)\n   Skipped 2020-04 (file already exists)\n   Skipped 2020-05 (file already exists)\n   Skipped 2020-06 (file already exists)\n   Skipped 2020-07 (file already exists)\n   Skipped 2020-08 (file already exists)\n   Skipped 2020-09 (file already exists)\n   Skipped 2020-10 (file already exists)\n   Skipped 2020-11 (file already exists)\n   Skipped 2020-12 (file already exists)\n   Skipped 2021-01 (file already exists)\n   Skipped 2021-02 (file already exists)\n   Skipped 2021-03 (file already exists)\n   Skipped 2021-04 (file already exists)\n   Skipped 2021-05 (file already exists)\n   Skipped 2021-06 (file already exists)\n   Skipped 2021-07 (file already exists)\n   Skipped 2021-08 (file already exists)\n   Skipped 2021-09 (file already exists)\n   Skipped 2021-10 (file already exists)\n   Skipped 2021-11 (file already exists)\n   Skipped 2021-12 (file already exists)\n   Skipped 2022-01 (file already exists)\n   Skipped 2022-02 (file already exists)\n   Skipped 2022-03 (file already exists)\n   Skipped 2022-04 (file already exists)\n   Skipped 2022-05 (file already exists)\n   Skipped 2022-06 (file already exists)\n   Skipped 2022-07 (file already exists)\n   Skipped 2022-08 (file already exists)\n   Skipped 2022-09 (file already exists)\n   Skipped 2022-10 (file already exists)\n   Skipped 2022-11 (file already exists)\n   Downloaded 2022-12\n   Downloaded 2023-01\n   Downloaded 2023-02\n============================================================\nProcessing \"fhv\"  (2015-01  through  2023-02)\n   Skipped 2015-01 (file already exists)\n   Skipped 2015-02 (file already exists)\n   Skipped 2015-03 (file already exists)\n   Skipped 2015-04 (file already exists)\n   Skipped 2015-05 (file already exists)\n   Skipped 2015-06 (file already exists)\n   Skipped 2015-07 (file already exists)\n   Skipped 2015-08 (file already exists)\n   Skipped 2015-09 (file already exists)\n   Skipped 2015-10 (file already exists)\n   Skipped 2015-11 (file already exists)\n   Skipped 2015-12 (file already exists)\n   Skipped 2016-01 (file already exists)\n   Skipped 2016-02 (file already exists)\n   Skipped 2016-03 (file already exists)\n   Skipped 2016-04 (file already exists)\n   Skipped 2016-05 (file already exists)\n   Skipped 2016-06 (file already exists)\n   Skipped 2016-07 (file already exists)\n   Skipped 2016-08 (file already exists)\n   Skipped 2016-09 (file already exists)\n   Skipped 2016-10 (file already exists)\n   Skipped 2016-11 (file already exists)\n   Skipped 2016-12 (file already exists)\n   Skipped 2017-01 (file already exists)\n   Skipped 2017-02 (file already exists)\n   Skipped 2017-03 (file already exists)\n   Skipped 2017-04 (file already exists)\n   Skipped 2017-05 (file already exists)\n   Skipped 2017-06 (file already exists)\n   Skipped 2017-07 (file already exists)\n   Skipped 2017-08 (file already exists)\n   Skipped 2017-09 (file already exists)\n   Skipped 2017-10 (file already exists)\n   Skipped 2017-11 (file already exists)\n   Skipped 2017-12 (file already exists)\n   Skipped 2018-01 (file already exists)\n   Skipped 2018-02 (file already exists)\n   Skipped 2018-03 (file already exists)\n   Skipped 2018-04 (file already exists)\n   Skipped 2018-05 (file already exists)\n   Skipped 2018-06 (file already exists)\n   Skipped 2018-07 (file already exists)\n   Skipped 2018-08 (file already exists)\n   Skipped 2018-09 (file already exists)\n   Skipped 2018-10 (file already exists)\n   Skipped 2018-11 (file already exists)\n   Skipped 2018-12 (file already exists)\n   Skipped 2019-01 (file already exists)\n   Skipped 2019-02 (file already exists)\n   Skipped 2019-03 (file already exists)\n   Skipped 2019-04 (file already exists)\n   Skipped 2019-05 (file already exists)\n   Skipped 2019-06 (file already exists)\n   Skipped 2019-07 (file already exists)\n   Skipped 2019-08 (file already exists)\n   Skipped 2019-09 (file already exists)\n   Skipped 2019-10 (file already exists)\n   Skipped 2019-11 (file already exists)\n   Skipped 2019-12 (file already exists)\n   Skipped 2020-01 (file already exists)\n   Skipped 2020-02 (file already exists)\n   Skipped 2020-03 (file already exists)\n   Skipped 2020-04 (file already exists)\n   Skipped 2020-05 (file already exists)\n   Skipped 2020-06 (file already exists)\n   Skipped 2020-07 (file already exists)\n   Skipped 2020-08 (file already exists)\n   Skipped 2020-09 (file already exists)\n   Skipped 2020-10 (file already exists)\n   Skipped 2020-11 (file already exists)\n   Skipped 2020-12 (file already exists)\n   Skipped 2021-01 (file already exists)\n   Skipped 2021-02 (file already exists)\n   Skipped 2021-03 (file already exists)\n   Skipped 2021-04 (file already exists)\n   Skipped 2021-05 (file already exists)\n   Skipped 2021-06 (file already exists)\n   Skipped 2021-07 (file already exists)\n   Skipped 2021-08 (file already exists)\n   Skipped 2021-09 (file already exists)\n   Skipped 2021-10 (file already exists)\n   Skipped 2021-11 (file already exists)\n   Skipped 2021-12 (file already exists)\n   Skipped 2022-01 (file already exists)\n   Skipped 2022-02 (file already exists)\n   Skipped 2022-03 (file already exists)\n   Skipped 2022-04 (file already exists)\n   Skipped 2022-05 (file already exists)\n   Skipped 2022-06 (file already exists)\n   Skipped 2022-07 (file already exists)\n   Skipped 2022-08 (file already exists)\n   Skipped 2022-09 (file already exists)\n   Skipped 2022-10 (file already exists)\n   Skipped 2022-11 (file already exists)\n   Downloaded 2022-12\n   Downloaded 2023-01\n   Downloaded 2023-02\n============================================================\nProcessing \"fhvhv\"  (2019-02  through  2023-02)\n   Skipped 2019-02 (file already exists)\n   Skipped 2019-03 (file already exists)\n   Skipped 2019-04 (file already exists)\n   Skipped 2019-05 (file already exists)\n   Skipped 2019-06 (file already exists)\n   Skipped 2019-07 (file already exists)\n   Skipped 2019-08 (file already exists)\n   Skipped 2019-09 (file already exists)\n   Skipped 2019-10 (file already exists)\n   Skipped 2019-11 (file already exists)\n   Skipped 2019-12 (file already exists)\n   Skipped 2020-01 (file already exists)\n   Skipped 2020-02 (file already exists)\n   Skipped 2020-03 (file already exists)\n   Skipped 2020-04 (file already exists)\n   Skipped 2020-05 (file already exists)\n   Skipped 2020-06 (file already exists)\n   Skipped 2020-07 (file already exists)\n   Skipped 2020-08 (file already exists)\n   Skipped 2020-09 (file already exists)\n   Skipped 2020-10 (file already exists)\n   Skipped 2020-11 (file already exists)\n   Skipped 2020-12 (file already exists)\n   Skipped 2021-01 (file already exists)\n   Skipped 2021-02 (file already exists)\n   Skipped 2021-03 (file already exists)\n   Skipped 2021-04 (file already exists)\n   Skipped 2021-05 (file already exists)\n   Skipped 2021-06 (file already exists)\n   Skipped 2021-07 (file already exists)\n   Skipped 2021-08 (file already exists)\n   Skipped 2021-09 (file already exists)\n   Skipped 2021-10 (file already exists)\n   Skipped 2021-11 (file already exists)\n   Skipped 2021-12 (file already exists)\n   Skipped 2022-01 (file already exists)\n   Skipped 2022-02 (file already exists)\n   Skipped 2022-03 (file already exists)\n   Skipped 2022-04 (file already exists)\n   Skipped 2022-05 (file already exists)\n   Skipped 2022-06 (file already exists)\n   Skipped 2022-07 (file already exists)\n   Skipped 2022-08 (file already exists)\n   Skipped 2022-09 (file already exists)\n   Skipped 2022-10 (file already exists)\n   Skipped 2022-11 (file already exists)\n   Downloaded 2022-12\n   Downloaded 2023-01\n   Downloaded 2023-02\n============================================================\n"
     ]
    }
   ],
   "source": [
    "import urllib.request\n",
    "import os\n",
    "\n",
    "for k in taxi_types.keys():\n",
    "  print(\"Processing \\\"{0}\\\"  ({1}  through  {2})\".format(k, taxi_types[k][0], taxi_types[k][1]))\n",
    "  months = [x for x in all_months if x >= taxi_types[k][0] and x <= taxi_types[k][1]]\n",
    "  \n",
    "  type_path = \"{0}/{1}\".format(output_path, k)\n",
    "  if not os.path.exists(type_path):\n",
    "    os.makedirs(type_path)\n",
    "\n",
    "  for m in months:\n",
    "    url = f\"https://d37ci6vzurychx.cloudfront.net/trip-data/{k}_tripdata_{m}.parquet\"\n",
    "    filename = f\"{type_path}/{m}.parquet\"\n",
    "\n",
    "    # Do not download the file if we already have a copy of it\n",
    "    if not os.path.exists(filename):\n",
    "      try:\n",
    "        urllib.request.urlretrieve(url, filename)\n",
    "        print(f\"   Downloaded {m}\")\n",
    "      except Exception as err:\n",
    "        print(f\" !!Could not download {m}\\n      {err}\")\n",
    "    else:\n",
    "      print(\"   Skipped {} (file already exists)\".format(m))\n",
    "    \n",
    "  print(\"============================================================\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "6dbbdf3c-1fd0-4d81-bf6c-3455176442f0",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "application/vnd.databricks.v1+notebook": {
   "dashboards": [],
   "language": "python",
   "notebookMetadata": {
    "pythonIndentUnit": 2
   },
   "notebookName": "01 Download Raw Parquet Files",
   "widgets": {}
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}