SFFireDeptDataAnalysis_Azure.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download data from \n",
    "\n",
    "https://data.sfgov.org/Public-Safety/Fire-Department-Calls-for-Service/nuek-vuh3\n",
    "\n",
    "Press the export button to export as csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. What were all the different types of fire calls in 2018?\n",
    "\n",
    "2. What months within the year 2018 saw for the highest number of fire calls?\n",
    "\n",
    "3. Which neighborhood in SF generated the most fire calls in 2018?\n",
    "\n",
    "4. Which neighborhoods in SF had the worst response time to fire calls in 2018?\n",
    "\n",
    "5. Which week in the year in 2018 had the most fire calls?\n",
    "\n",
    "6. Is there a correlation between neighborhood, zip code, and fire calls?\n",
    "\n",
    "7. How can we use Parquet files or SQL tables to store this data and read it back?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Number of ways to do this\n",
    "\n",
    "1. Local machine. Jupyter notebook or spark submit\n",
    "\n",
    "    File is too large. Not possible\n",
    "\n",
    "2. [Cloud - Azure HDinsight](https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-jupyter-spark-sql-use-portal)\n",
    "    \n",
    "    Number of ways to submit spark job - [Azure documentation](https://docs.microsoft.com/en-us/archive/blogs/azuredatalake/spark-job-submission-on-hdinsight-101)\n",
    "    \n",
    "    1. [Interactive jupyter shell in HDI cluster](https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-load-data-run-query)\n",
    "\n",
    "    Quickest way, but I was not able to create Jupyter notbooks in HDI cluster\n",
    "\n",
    "    2. [Run jupyter locally and connect to HDI cluster](https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-jupyter-notebook-install-locally)\n",
    "\n",
    "    Lots of steps - but can be one time effort if everything works.\n",
    "\n",
    "    3. Write code and submit batch job through API\n",
    "\n",
    "    Requires JAR creation. No instruction for python\n",
    "\n",
    "    4. [Visual studio code for pyspark](https://azure.microsoft.com/en-us/blog/run-your-pyspark-interactive-query-and-batch-job-in-visual-studio-code/)\n",
    "\n",
    "    Probably the second most easiest way. A lot is abstracted out and you are dependent a lot on visual studio code.\n",
    "    But considering VScode is cross platform and actively develped and maintained, I will try this today.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('./sffiredata.csv', <http.client.HTTPMessage at 0x16bf56060c8>)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#download data\n",
    "import urllib\n",
    "from urllib import request\n",
    "\n",
    "out = './sffiredata.csv'\n",
    "url = 'https://data.sfgov.org/api/views/nuek-vuh3/rows.csv?accessType=DOWNLOAD'\n",
    "\n",
    "urllib.request.urlretrieve(url, out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "jupyter core     : 4.6.1\n",
      "jupyter-notebook : 6.0.2\n",
      "qtconsole        : 4.6.0\n",
      "ipython          : 7.11.1\n",
      "ipykernel        : 5.1.3\n",
      "jupyter client   : 5.3.4\n",
      "jupyter lab      : not installed\n",
      "nbconvert        : 5.6.1\n",
      "ipywidgets       : 7.5.1\n",
      "nbformat         : 5.0.3\n",
      "traitlets        : 4.3.3\n"
     ]
    }
   ],
   "source": [
    "!jupyter --version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "file_extension": ".py",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  },
  "mimetype": "text/x-python",
  "name": "python",
  "npconvert_exporter": "python",
  "pygments_lexer": "ipython3",
  "version": 3
 },
 "nbformat": 4,
 "nbformat_minor": 2
}