Big data course - Spark tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8be3d2c1-699c-4e26-b6f4-a2f24fd739b7",
   "metadata": {},
   "source": [
    "# Attention\n",
    "\n",
    "Download your notebook changes (File > Download) before you are leaving or when you will be inactive for a while, Binder instances close after 30min of inactivity!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19b726fe-dfa0-4ed1-ac8d-f5f9e6053116",
   "metadata": {},
   "source": [
    "# PySpark data exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3fd4790b-c5ce-4f0e-b3b5-1b3afc54a9ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark.sql import functions as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c464c222-e3db-4984-b285-ca8f600aeaea",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n",
      "23/04/27 08:51:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "            <div>\n",
       "                <p><b>SparkSession - in-memory</b></p>\n",
       "                \n",
       "        <div>\n",
       "            <p><b>SparkContext</b></p>\n",
       "\n",
       "            <p><a href=\"http://jupyter-pythonpredictio-2d-2dpyspark-2dbinder-2dbynl6fxc:4040\">Spark UI</a></p>\n",
       "\n",
       "            <dl>\n",
       "              <dt>Version</dt>\n",
       "                <dd><code>v3.4.0</code></dd>\n",
       "              <dt>Master</dt>\n",
       "                <dd><code>local[*]</code></dd>\n",
       "              <dt>AppName</dt>\n",
       "                <dd><code>big data course</code></dd>\n",
       "            </dl>\n",
       "        </div>\n",
       "        \n",
       "            </div>\n",
       "        "
      ],
      "text/plain": [
       "<pyspark.sql.session.SparkSession at 0x7fabde7faef0>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spark = (\n",
    "    SparkSession.builder\n",
    "                        # Name to identify your experiment in the cluster's dashboard:\n",
    "                        .appName(\"big data course\")\n",
    "                        # Connect with the cluster's orchestrator:\n",
    "                        .master(\"local[*]\") # Cannot be \"local\" if you want to use your company's cluster.\n",
    "                        # Maximum memory any result dataframe will take up in driver memory:\n",
    "                        .config(\"spark.driver.maxResultSize\", \"4g\")\n",
    "                        # How much memory can be allocated to the driver (master/orchestrator):\n",
    "                        .config(\"spark.driver.memory\", \"1g\")\n",
    "                        # How much executors will be needed for the experiment:\n",
    "                        .config(\"spark.executor.instances\", \"4\")\n",
    "                        # Alternatively, allow spinning up more executors when there is more computation load, and discard them when less load:\n",
    "                        # .config(\"spark.dynamicAllocation.enabled\", True)\n",
    "                        # .config(\"spark.dynamicAllocation.minExecutors\", 1)\n",
    "                        # .config(\"spark.dynamic Allocation.maxExecutors\", 4)\n",
    "                        # How much memory can be allocated to each executor:\n",
    "                        .config(\"spark.executor.memory\", \"1g\")\n",
    "                        # How much CPU cores can be used to optimize parallellization within the executors (can be useful for shuffling data etc):\n",
    "                        #.config(\"spark.executor.cores\", 4)\n",
    "                        .config(\"spark.serializer\", \"org.apache.spark.serializer.KryoSerializer\")\n",
    "                        .getOrCreate()\n",
    ")\n",
    "spark"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f1f3afc-87d8-4479-a1d1-adaf9bd06ac6",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Clarification on the \"Spark UI\" link that doesn't work here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b6b32f71-f9ff-49c8-82e3-54f4632a6e4a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://hub.gke2.mybinder.org/user/pythonpredictio--pyspark-binder-bynl6fxc/'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "binder_url = 'https://hub.mybinder.org' + os.environ['JUPYTERHUB_SERVICE_PREFIX']\n",
    "binder_url = \"https://hub.gke2.mybinder.org\" + os.environ[\"JUPYTERHUB_SERVICE_PREFIX\"]\n",
    "binder_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3f66b686-f5e7-44a0-9f06-de477c53cdcc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://hub.gke2.mybinder.org/user/pythonpredictio--pyspark-binder-bynl6fxc/proxy/4040'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "external_spark_ui_url = binder_url + \"proxy/4040\"\n",
    "external_spark_ui_url"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47d1a00d-b9b1-4e3f-ae2f-4fb93740087b",
   "metadata": {},
   "source": [
    "Normally, you should be able to visit the Spark UI when entering the above URL in a new browser tab. If a token is requested, you can copy-paste it from the command output of the next cell. But you'll probably get a HTTP error 4xx or 5xx.\n",
    "\n",
    "This *should* work, since the docker container of this binder project uses jupyter-server-proxy to make locally-listened ports available externally on the binder URL. See the documentation: https://jupyter-server-proxy.readthedocs.io/en/latest/arbitrary-ports-hosts.html\n",
    "\n",
    "It of course *doesn't* work though.\n",
    "There seems to be an issue with the proxying of the pyspark UI port 4040. We cannot navigate to it externally, but we can internally from within our container (see next cells). This is due to how the binder container is configured.\n",
    "\n",
    "So, let's settle with the fact that with this binder container, we will not be able to check the Spark UI when experimenting. Check that on your company's cluster, when moving from this toy environment towards \"the real thing\".\n",
    "Let's just use the binder container for now to play around with pySpark."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5599c5a7-3328-4528-8d08-25ddc809f9e7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Currently running servers:\n",
      "http://jupyter-pythonpredictio-2d-2dpyspark-2dbinder-2dbynl6fxc:8888/user/pythonpredictio--pyspark-binder-bynl6fxc/?token=1SS4JF0XSPKnw3L-cMBe2Q :: /home/jovyan\n"
     ]
    }
   ],
   "source": [
    "! jupyter server list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a3e5a387-8d2d-4309-b2aa-9ae854934c87",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://jupyter-pythonpredictio-2d-2dpyspark-2dbinder-2dbynl6fxc:4040'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# url used internally within this binder's docker container:\n",
    "spark.sparkContext.uiWebUrl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "06aecc93-a1fc-4047-a03c-3a5226f072aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://jupyter-pythonpredictio-2d-2dpyspark-2dbinder-2dbynl6fxc:4040/jobs\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<!DOCTYPE html><html>\n",
       "      <head>\n",
       "        <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\"/><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"/><link rel=\"stylesheet\" href=\"/static/bootstrap.min.css\" type=\"text/css\"/><link rel=\"stylesheet\" href=\"/static/vis-timeline-graph2d.min.css\" type=\"text/css\"/><link rel=\"stylesheet\" href=\"/static/webui.css\" type=\"text/css\"/><link rel=\"stylesheet\" href=\"/static/timeline-view.css\" type=\"text/css\"/><script src=\"/static/sorttable.js\"></script><script src=\"/static/jquery-3.5.1.min.js\"></script><script src=\"/static/vis-timeline-graph2d.min.js\"></script><script src=\"/static/bootstrap.bundle.min.js\"></script><script src=\"/static/initialize-tooltips.js\"></script><script src=\"/static/table.js\"></script><script src=\"/static/timeline-view.js\"></script><script src=\"/static/log-view.js\"></script><script src=\"/static/webui.js\"></script><script>setUIRoot('')</script>\n",
       "        <script>setAppBasePath('')</script>\n",
       "        \n",
       "        \n",
       "        <link rel=\"shortcut icon\" href=\"/static/spark-logo-77x50px-hd.png\"></link>\n",
       "        <title>big data course - Spark Jobs</title>\n",
       "      </head>\n",
       "      <body>\n",
       "        <nav class=\"navbar navbar-expand-md navbar-light bg-light mb-4\">\n",
       "          <div class=\"navbar-header\">\n",
       "            <div class=\"navbar-brand\">\n",
       "              <a href=\"/\">\n",
       "                <img src=\"/static/spark-logo-77x50px-hd.png\"/>\n",
       "                <span class=\"version\">3.4.0</span>\n",
       "              </a>\n",
       "            </div>\n",
       "          </div>\n",
       "          <button class=\"navbar-toggler\" type=\"button\" data-toggle=\"collapse\" data-target=\"#navbarCollapse\" aria-controls=\"navbarCollapse\" aria-expanded=\"false\" aria-label=\"Toggle navigation\">\n",
       "            <span class=\"navbar-toggler-icon\"></span>\n",
       "          </button>\n",
       "          <div class=\"collapse navbar-collapse\" id=\"navbarCollapse\">\n",
       "            <ul class=\"navbar-nav mr-auto\"><li class=\"nav-item active\">\n",
       "        <a class=\"nav-link\" href=\"/jobs/\">Jobs</a>\n",
       "      </li><li class=\"nav-item\">\n",
       "        <a class=\"nav-link\" href=\"/stages/\">Stages</a>\n",
       "      </li><li class=\"nav-item\">\n",
       "        <a class=\"nav-link\" href=\"/storage/\">Storage</a>\n",
       "      </li><li class=\"nav-item\">\n",
       "        <a class=\"nav-link\" href=\"/environment/\">Environment</a>\n",
       "      </li><li class=\"nav-item\">\n",
       "        <a class=\"nav-link\" href=\"/executors/\">Executors</a>\n",
       "      </li><li class=\"nav-item\">\n",
       "        <a class=\"nav-link\" href=\"/SQL/\">SQL / DataFrame</a>\n",
       "      </li></ul>\n",
       "            <span class=\"navbar-text navbar-right d-none d-md-block\">\n",
       "              <strong title=\"big data course\" class=\"text-nowrap\">big data course</strong>\n",
       "              <span class=\"text-nowrap\">application UI</span>\n",
       "            </span>\n",
       "          </div>\n",
       "        </nav>\n",
       "        <div class=\"container-fluid\">\n",
       "          <div class=\"row\">\n",
       "            <div class=\"col-12\">\n",
       "              <h3 style=\"vertical-align: bottom; white-space: nowrap; overflow: hidden;\n",
       "                text-overflow: ellipsis;\">\n",
       "                Spark Jobs\n",
       "                <sup>\n",
       "      (<a data-toggle=\"tooltip\" data-placement=\"top\" title=\"A job is triggered by an action, like count() or saveAsTextFile(). Click on a job to see information about the stages of tasks inside it.\">?</a>)\n",
       "    </sup>\n",
       "              </h3>\n",
       "            </div>\n",
       "          </div>\n",
       "          <div class=\"row\">\n",
       "            <div class=\"col-12\">\n",
       "              <div>\n",
       "        <ul class=\"list-unstyled\">\n",
       "          <li>\n",
       "\n",
       "            <strong>User:</strong>\n",
       "            jovyan\n",
       "          </li>\n",
       "          <li>\n",
       "            <strong>Total Uptime:</strong>\n",
       "            2.3 min\n",
       "          </li>\n",
       "          <li>\n",
       "            <strong>Scheduling Mode: </strong>\n",
       "            FIFO\n",
       "          </li>\n",
       "          \n",
       "          \n",
       "          \n",
       "        </ul>\n",
       "      </div><span class=\"expand-application-timeline\">\n",
       "      <span class=\"expand-application-timeline-arrow arrow-closed\"></span>\n",
       "      <a data-toggle=\"tooltip\" title=\"Shows when jobs started and ended and when executors joined or left. Drag to scroll.\n",
       "       Click Enable Zooming and use mouse wheel to zoom in/out.\" data-placement=\"top\">\n",
       "        Event Timeline\n",
       "      </a>\n",
       "    </span><div id=\"application-timeline\" class=\"collapsed\">\n",
       "      \n",
       "      \n",
       "      <div class=\"control-panel\">\n",
       "        <div id=\"application-timeline-zoom-lock\">\n",
       "          <input type=\"checkbox\"></input>\n",
       "          <span>Enable zooming</span>\n",
       "        </div>\n",
       "      </div>\n",
       "    </div><script type=\"text/javascript\">\n",
       "      drawApplicationTimeline(\n",
       "[\n",
       "  {\n",
       "    'id': 'executors',\n",
       "    'content': '<div>Executors</div><div class=\"legend-area\"><svg width=\"150px\" height=\"55px\">      <rect class=\"executor-added-legend\" x=\"5px\" y=\"5px\" width=\"20px\" height=\"15px\" rx=\"2px\" ry=\"2px\"></rect>      <text x=\"35px\" y=\"17px\">Added</text>      <rect class=\"executor-removed-legend\" x=\"5px\" y=\"30px\" width=\"20px\" height=\"15px\" rx=\"2px\" ry=\"2px\"></rect>      <text x=\"35px\" y=\"42px\">Removed</text>    </svg></div>',\n",
       "  },\n",
       "  {\n",
       "    'id': 'jobs',\n",
       "    'content': '<div>Jobs</div><div class=\"legend-area\"><svg width=\"150px\" height=\"85px\">      <rect class=\"succeeded-job-legend\" x=\"5px\" y=\"5px\" width=\"20px\" height=\"15px\" rx=\"2px\" ry=\"2px\"></rect>      <text x=\"35px\" y=\"17px\">Succeeded</text>      <rect class=\"failed-job-legend\" x=\"5px\" y=\"30px\" width=\"20px\" height=\"15px\" rx=\"2px\" ry=\"2px\"></rect>      <text x=\"35px\" y=\"42px\">Failed</text>      <rect class=\"running-job-legend\" x=\"5px\" y=\"55px\" width=\"20px\" height=\"15px\" rx=\"2px\" ry=\"2px\"></rect>      <text x=\"35px\" y=\"67px\">Running</text>    </svg></div>',\n",
       "  }\n",
       "]\n",
       "        ,[\n",
       "{\n",
       "  'className': 'executor added',\n",
       "  'group': 'executors',\n",
       "  'start': new Date(1682583245506),\n",
       "  'content': '<div class=\"executor-event-content\"' +\n",
       "    'data-toggle=\"tooltip\" data-placement=\"top\"' +\n",
       "    'data-title=\"Executor driver<br>' +\n",
       "    'Added at 2023/04/27 08:14:05\"' +\n",
       "    'data-html=\"true\">Executor driver added</div>'\n",
       "}\n",
       "         ], 1682583242934, 0);\n",
       "    </script>\n",
       "            </div>\n",
       "          </div>\n",
       "        </div>\n",
       "      </body>\n",
       "    </html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This isn't great, but with this hack we can see that the Spark UI *is* live:\n",
    "import requests\n",
    "from IPython.display import HTML\n",
    "\n",
    "def render_local_spark_ui(subpage=\"jobs\"):\n",
    "    subpage_url = spark.sparkContext.uiWebUrl + (\"\" if subpage is None else \"/\" + subpage)\n",
    "    print(subpage_url)\n",
    "    response = requests.get(subpage_url)\n",
    "    return HTML(data=bytes.decode(response.content))\n",
    "    \n",
    "render_local_spark_ui(subpage=\"jobs\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a002b835-272e-4323-b108-9d8a6e10a233",
   "metadata": {},
   "source": [
    "## Exploratory data analysis\n",
    "\n",
    "On a small sample of the popular flights dataset: https://github.com/ozlerhakan/datacamp/blob/master/Introduction%20to%20PySpark/flights_small.csv."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0a94b675-98e4-4f3d-95b4-4e8495f040c2",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pyspark.sql.dataframe.DataFrame'>\n",
      "root\n",
      " |-- year: integer (nullable = true)\n",
      " |-- month: integer (nullable = true)\n",
      " |-- day: integer (nullable = true)\n",
      " |-- dep_time: string (nullable = true)\n",
      " |-- dep_delay: string (nullable = true)\n",
      " |-- arr_time: string (nullable = true)\n",
      " |-- arr_delay: string (nullable = true)\n",
      " |-- carrier: string (nullable = true)\n",
      " |-- tailnum: string (nullable = true)\n",
      " |-- flight: integer (nullable = true)\n",
      " |-- origin: string (nullable = true)\n",
      " |-- dest: string (nullable = true)\n",
      " |-- air_time: string (nullable = true)\n",
      " |-- distance: integer (nullable = true)\n",
      " |-- hour: string (nullable = true)\n",
      " |-- minute: string (nullable = true)\n",
      "\n",
      "+----+-----+---+--------+---------+--------+---------+-------+-------+------+------+----+--------+--------+----+------+\n",
      "|year|month|day|dep_time|dep_delay|arr_time|arr_delay|carrier|tailnum|flight|origin|dest|air_time|distance|hour|minute|\n",
      "+----+-----+---+--------+---------+--------+---------+-------+-------+------+------+----+--------+--------+----+------+\n",
      "|2014|   12|  8|     658|       -7|     935|       -5|     VX| N846VA|  1780|   SEA| LAX|     132|     954|   6|    58|\n",
      "|2014|    1| 22|    1040|        5|    1505|        5|     AS| N559AS|   851|   SEA| HNL|     360|    2677|  10|    40|\n",
      "|2014|    3|  9|    1443|       -2|    1652|        2|     VX| N847VA|   755|   SEA| SFO|     111|     679|  14|    43|\n",
      "|2014|    4|  9|    1705|       45|    1839|       34|     WN| N360SW|   344|   PDX| SJC|      83|     569|  17|     5|\n",
      "|2014|    3|  9|     754|       -1|    1015|        1|     AS| N612AS|   522|   SEA| BUR|     127|     937|   7|    54|\n",
      "|2014|    1| 15|    1037|        7|    1352|        2|     WN| N646SW|    48|   PDX| DEN|     121|     991|  10|    37|\n",
      "|2014|    7|  2|     847|       42|    1041|       51|     WN| N422WN|  1520|   PDX| OAK|      90|     543|   8|    47|\n",
      "|2014|    5| 12|    1655|       -5|    1842|      -18|     VX| N361VA|   755|   SEA| SFO|      98|     679|  16|    55|\n",
      "|2014|    4| 19|    1236|       -4|    1508|       -7|     AS| N309AS|   490|   SEA| SAN|     135|    1050|  12|    36|\n",
      "|2014|   11| 19|    1812|       -3|    2352|       -4|     AS| N564AS|    26|   SEA| ORD|     198|    1721|  18|    12|\n",
      "+----+-----+---+--------+---------+--------+---------+-------+-------+------+------+----+--------+--------+----+------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "flights = spark.read.csv(\"flights_small.csv\", header=True, inferSchema=True) # if inferSchema=True is not set, all columns are just string.\n",
    "print(type(flights))\n",
    "flights.printSchema()\n",
    "flights.show(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "fc69f6f7-89b3-489a-8b7c-2a911877a1c7",
   "metadata": {},
   "outputs": [
    {
     "ename": "ImportError",
     "evalue": "Pandas >= 1.0.5 must be installed; however, it was not found.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.10/site-packages/pyspark/sql/pandas/utils.py:27\u001b[0m, in \u001b[0;36mrequire_minimum_pandas_version\u001b[0;34m()\u001b[0m\n\u001b[1;32m     26\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 27\u001b[0m     \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mpandas\u001b[39;00m\n\u001b[1;32m     29\u001b[0m     have_pandas \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mTrue\u001b[39;00m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'pandas'",
      "\nThe above exception was the direct cause of the following exception:\n",
      "\u001b[0;31mImportError\u001b[0m                               Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[10], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mflights\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mlimit\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m10\u001b[39;49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtoPandas\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.10/site-packages/pyspark/sql/pandas/conversion.py:89\u001b[0m, in \u001b[0;36mPandasConversionMixin.toPandas\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     85\u001b[0m \u001b[38;5;28;01massert\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(\u001b[38;5;28mself\u001b[39m, DataFrame)\n\u001b[1;32m     87\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpyspark\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01msql\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mpandas\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mutils\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m require_minimum_pandas_version\n\u001b[0;32m---> 89\u001b[0m \u001b[43mrequire_minimum_pandas_version\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     91\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mnumpy\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mnp\u001b[39;00m\n\u001b[1;32m     92\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mpandas\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mpd\u001b[39;00m\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.10/site-packages/pyspark/sql/pandas/utils.py:34\u001b[0m, in \u001b[0;36mrequire_minimum_pandas_version\u001b[0;34m()\u001b[0m\n\u001b[1;32m     32\u001b[0m     raised_error \u001b[38;5;241m=\u001b[39m error\n\u001b[1;32m     33\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m have_pandas:\n\u001b[0;32m---> 34\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mImportError\u001b[39;00m(\n\u001b[1;32m     35\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mPandas >= \u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m must be installed; however, \u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mit was not found.\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;241m%\u001b[39m minimum_pandas_version\n\u001b[1;32m     36\u001b[0m     ) \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mraised_error\u001b[39;00m\n\u001b[1;32m     37\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m LooseVersion(pandas\u001b[38;5;241m.\u001b[39m__version__) \u001b[38;5;241m<\u001b[39m LooseVersion(minimum_pandas_version):\n\u001b[1;32m     38\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mImportError\u001b[39;00m(\n\u001b[1;32m     39\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mPandas >= \u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m must be installed; however, \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m     40\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124myour version was \u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m.\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;241m%\u001b[39m (minimum_pandas_version, pandas\u001b[38;5;241m.\u001b[39m__version__)\n\u001b[1;32m     41\u001b[0m     )\n",
      "\u001b[0;31mImportError\u001b[0m: Pandas >= 1.0.5 must be installed; however, it was not found."
     ]
    }
   ],
   "source": [
    "# if lots of columns, this plots nicer than show():\n",
    "flights.limit(10).toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "344e8ea7-6db6-4187-9fab-de0c88dd5919",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+\n",
      "|max(year)|\n",
      "+---------+\n",
      "|     2014|\n",
      "+---------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Showing the latest available year:\n",
    "flights.select(F.max(F.col(\"year\"))).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "a44aa904-bfd0-4b66-9669-3d5949ccae5f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2014"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Selecting all results to a python variable can be done like this:\n",
    "max_year = flights.select(F.max(F.col(\"year\"))).collect()[0][0]\n",
    "max_year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "e489ad0c-52bb-40c7-8a06-7f4ade0fd30f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 19:>                                                         (0 + 1) / 1]\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+-----+\n",
      "|month|count|\n",
      "+-----+-----+\n",
      "|    1|  748|\n",
      "|    2|  639|\n",
      "|    3|  790|\n",
      "|    4|  817|\n",
      "|    5|  844|\n",
      "|    6|  889|\n",
      "|    7|  976|\n",
      "|    8| 1065|\n",
      "|    9|  836|\n",
      "|   10|  814|\n",
      "|   11|  780|\n",
      "|   12|  802|\n",
      "+-----+-----+\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "# The full year 2014 is available (though probably a sample of ALL flights happening in the US, since our CSV file is small, there is a larger one that is used often):\n",
    "flights.groupby(\"month\").count().orderBy(\"month\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "869cd1a1-5dc0-4976-972f-7a00b7f6bb56",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+-----+---+--------+---------+--------+---------+-------+-----------+------+------+----+--------+--------+----+------+--------------------+\n",
      "|year|month|day|dep_time|dep_delay|arr_time|arr_delay|carrier|tail_number|flight|origin|dest|air_time|distance|hour|minute|              source|\n",
      "+----+-----+---+--------+---------+--------+---------+-------+-----------+------+------+----+--------+--------+----+------+--------------------+\n",
      "|2014|   12|  8|     658|       -7|     935|       -5|     VX|     N846VA|  1780|   SEA| LAX|     132|     954|   6|    58|https://github.co...|\n",
      "|2014|    1| 22|    1040|        5|    1505|        5|     AS|     N559AS|   851|   SEA| HNL|     360|    2677|  10|    40|https://github.co...|\n",
      "|2014|    3|  9|    1443|       -2|    1652|        2|     VX|     N847VA|   755|   SEA| SFO|     111|     679|  14|    43|https://github.co...|\n",
      "|2014|    4|  9|    1705|       45|    1839|       34|     WN|     N360SW|   344|   PDX| SJC|      83|     569|  17|     5|https://github.co...|\n",
      "|2014|    3|  9|     754|       -1|    1015|        1|     AS|     N612AS|   522|   SEA| BUR|     127|     937|   7|    54|https://github.co...|\n",
      "|2014|    1| 15|    1037|        7|    1352|        2|     WN|     N646SW|    48|   PDX| DEN|     121|     991|  10|    37|https://github.co...|\n",
      "|2014|    7|  2|     847|       42|    1041|       51|     WN|     N422WN|  1520|   PDX| OAK|      90|     543|   8|    47|https://github.co...|\n",
      "|2014|    5| 12|    1655|       -5|    1842|      -18|     VX|     N361VA|   755|   SEA| SFO|      98|     679|  16|    55|https://github.co...|\n",
      "|2014|    4| 19|    1236|       -4|    1508|       -7|     AS|     N309AS|   490|   SEA| SAN|     135|    1050|  12|    36|https://github.co...|\n",
      "|2014|   11| 19|    1812|       -3|    2352|       -4|     AS|     N564AS|    26|   SEA| ORD|     198|    1721|  18|    12|https://github.co...|\n",
      "|2014|   11|  8|    1653|       -2|    1924|       -1|     AS|     N323AS|   448|   SEA| LAX|     130|     954|  16|    53|https://github.co...|\n",
      "|2014|    8|  3|    1120|        0|    1415|        2|     AS|     N305AS|   656|   SEA| PHX|     154|    1107|  11|    20|https://github.co...|\n",
      "|2014|   10| 30|     811|       21|    1038|       29|     AS|     N433AS|   608|   SEA| LAS|     127|     867|   8|    11|https://github.co...|\n",
      "|2014|   11| 12|    2346|       -4|     217|      -28|     AS|     N765AS|   121|   SEA| ANC|     183|    1448|  23|    46|https://github.co...|\n",
      "|2014|   10| 31|    1314|       89|    1544|      111|     AS|     N713AS|   306|   SEA| SFO|     129|     679|  13|    14|https://github.co...|\n",
      "|2014|    1| 29|    2009|        3|    2159|        9|     UA|     N27205|  1458|   PDX| SFO|      90|     550|  20|     9|https://github.co...|\n",
      "|2014|   12| 17|    2015|       50|    2150|       41|     AS|     N626AS|   368|   SEA| SMF|      76|     605|  20|    15|https://github.co...|\n",
      "|2014|    8| 11|    1017|       -3|    1613|       -7|     WN|     N8634A|   827|   SEA| MDW|     216|    1733|  10|    17|https://github.co...|\n",
      "|2014|    1| 13|    2156|       -9|     607|      -15|     AS|     N597AS|    24|   SEA| BOS|     290|    2496|  21|    56|https://github.co...|\n",
      "|2014|    6|  5|    1733|      -12|    1945|      -10|     OO|     N215AG|  3488|   PDX| BUR|     111|     817|  17|    33|https://github.co...|\n",
      "+----+-----+---+--------+---------+--------+---------+-------+-----------+------+------+----+--------+--------+----+------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "flights = (\n",
    "    flights.withColumn(\"source\", F.lit(\"https://github.com/ozlerhakan/datacamp/blob/master/Introduction%20to%20PySpark/flights_small.csv\"))\n",
    "           .withColumnRenamed(\"tailnum\", \"tail_number\")\n",
    ")\n",
    "flights.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2a244551-f6ae-48ea-9d63-8eb204109185",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+-----+---+--------+---------+--------+---------+-------+-------+------+------+----+--------+--------+----+------+\n",
      "|year|month|day|dep_time|dep_delay|arr_time|arr_delay|carrier|tailnum|flight|origin|dest|air_time|distance|hour|minute|\n",
      "+----+-----+---+--------+---------+--------+---------+-------+-------+------+------+----+--------+--------+----+------+\n",
      "|2014|   10| 30|     811|       21|    1038|       29|     AS| N433AS|   608|   SEA| LAS|     127|     867|   8|    11|\n",
      "|2014|    2| 25|     555|       -5|     819|       -3|     AS| N549AS|   604|   SEA| LAS|     121|     867|   5|    55|\n",
      "|2014|   11| 15|    1858|        8|    2107|        2|     AS| N587AS|   616|   SEA| LAS|     106|     867|  18|    58|\n",
      "|2014|    9| 28|    2041|       -9|    2251|       -7|     AS| N552AS|   624|   PDX| LAS|     106|     763|  20|    41|\n",
      "|2014|    7| 27|    1020|       -5|    1251|        2|     AS| N587AS|   586|   SEA| LAS|     130|     867|  10|    20|\n",
      "|2014|    3| 18|    1832|       -3|    2039|      -15|     AS| N320AS|   616|   SEA| LAS|     112|     867|  18|    32|\n",
      "|2014|    6| 28|    1829|       -1|    2023|      -17|     WN| N453WN|  2593|   PDX| LAS|     102|     763|  18|    29|\n",
      "|2014|   10|  8|    1909|       -6|    2137|        1|     OO| N806SK|  4536|   SEA| LAS|     118|     867|  19|     9|\n",
      "|2014|    5| 30|    1407|       -3|    1635|        8|     AS| N594AS|   606|   SEA| LAS|     127|     867|  14|     7|\n",
      "|2014|    6| 30|    1736|        6|    1949|      -12|     OO| N805SK|  4820|   SEA| LAS|     115|     867|  17|    36|\n",
      "|2014|   10| 14|     742|       -8|    1016|        7|     AS| N306AS|   608|   SEA| LAS|     132|     867|   7|    42|\n",
      "|2014|    7| 11|    2043|       -2|    2307|        5|     AS| N585AS|   614|   SEA| LAS|     126|     867|  20|    43|\n",
      "|2014|    6| 14|    1833|        8|    2047|        2|     AS| N706AS|   616|   SEA| LAS|     118|     867|  18|    33|\n",
      "|2014|    2| 27|    1426|       -4|    1645|       -6|     AS| N609AS|   608|   SEA| LAS|     113|     867|  14|    26|\n",
      "|2014|    8| 22|    1311|        6|    1528|       -4|     AS| N459AS|   578|   SEA| LAS|     108|     867|  13|    11|\n",
      "|2014|    5| 23|    1850|       -5|    2102|        7|     AS| N552AS|   620|   PDX| LAS|     109|     763|  18|    50|\n",
      "|2014|    7| 19|    1739|       -1|    2004|       -7|     OO| N810SK|  4738|   SEA| LAS|     121|     867|  17|    39|\n",
      "|2014|    7|  3|    1927|       37|    2144|       44|     WN| N491WN|   203|   PDX| LAS|     106|     763|  19|    27|\n",
      "|2014|    7| 12|    1833|        3|    2035|       -5|     WN| N287WN|  2593|   PDX| LAS|     109|     763|  18|    33|\n",
      "|2014|   11| 29|    1318|       53|    1538|       48|     OO| N631SK|  4608|   SEA| LAS|     121|     867|  13|    18|\n",
      "+----+-----+---+--------+---------+--------+---------+-------+-------+------+------+----+--------+--------+----+------+\n",
      "only showing top 20 rows\n",
      "\n",
      "CPU times: user 2.91 ms, sys: 1.12 ms, total: 4.03 ms\n",
      "Wall time: 1.01 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Which flights flew to Las Vegas?\n",
    "flights.filter(\"dest = 'LAS'\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "9fa2caed-47ac-4451-8b96-dd04552e9f95",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://jupyter-pythonpredictio-2d-2dpyspark-2dbinder-2dbynl6fxc:4040/jobs\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<!DOCTYPE html><html>\n",
       "      <head>\n",
       "        <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\"/><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"/><link rel=\"stylesheet\" href=\"/static/bootstrap.min.css\" type=\"text/css\"/><link rel=\"stylesheet\" href=\"/static/vis-timeline-graph2d.min.css\" type=\"text/css\"/><link rel=\"stylesheet\" href=\"/static/webui.css\" type=\"text/css\"/><link rel=\"stylesheet\" href=\"/static/timeline-view.css\" type=\"text/css\"/><script src=\"/static/sorttable.js\"></script><script src=\"/static/jquery-3.5.1.min.js\"></script><script src=\"/static/vis-timeline-graph2d.min.js\"></script><script src=\"/static/bootstrap.bundle.min.js\"></script><script src=\"/static/initialize-tooltips.js\"></script><script src=\"/static/table.js\"></script><script src=\"/static/timeline-view.js\"></script><script src=\"/static/log-view.js\"></script><script src=\"/static/webui.js\"></script><script>setUIRoot('')</script>\n",
       "        <script>setAppBasePath('')</script>\n",
       "        \n",
       "        \n",
       "        <link rel=\"shortcut icon\" href=\"/static/spark-logo-77x50px-hd.png\"></link>\n",
       "        <title>big data course - Spark Jobs</title>\n",
       "      </head>\n",
       "      <body>\n",
       "        <nav class=\"navbar navbar-expand-md navbar-light bg-light mb-4\">\n",
       "          <div class=\"navbar-header\">\n",
       "            <div class=\"navbar-brand\">\n",
       "              <a href=\"/\">\n",
       "                <img src=\"/static/spark-logo-77x50px-hd.png\"/>\n",
       "                <span class=\"version\">3.4.0</span>\n",
       "              </a>\n",
       "            </div>\n",
       "          </div>\n",
       "          <button class=\"navbar-toggler\" type=\"button\" data-toggle=\"collapse\" data-target=\"#navbarCollapse\" aria-controls=\"navbarCollapse\" aria-expanded=\"false\" aria-label=\"Toggle navigation\">\n",
       "            <span class=\"navbar-toggler-icon\"></span>\n",
       "          </button>\n",
       "          <div class=\"collapse navbar-collapse\" id=\"navbarCollapse\">\n",
       "            <ul class=\"navbar-nav mr-auto\"><li class=\"nav-item active\">\n",
       "        <a class=\"nav-link\" href=\"/jobs/\">Jobs</a>\n",
       "      </li><li class=\"nav-item\">\n",
       "        <a class=\"nav-link\" href=\"/stages/\">Stages</a>\n",
       "      </li><li class=\"nav-item\">\n",
       "        <a class=\"nav-link\" href=\"/storage/\">Storage</a>\n",
       "      </li><li class=\"nav-item\">\n",
       "        <a class=\"nav-link\" href=\"/environment/\">Environment</a>\n",
       "      </li><li class=\"nav-item\">\n",
       "        <a class=\"nav-link\" href=\"/executors/\">Executors</a>\n",
       "      </li><li class=\"nav-item\">\n",
       "        <a class=\"nav-link\" href=\"/SQL/\">SQL / DataFrame</a>\n",
       "      </li></ul>\n",
       "            <span class=\"navbar-text navbar-right d-none d-md-block\">\n",
       "              <strong title=\"big data course\" class=\"text-nowrap\">big data course</strong>\n",
       "              <span class=\"text-nowrap\">application UI</span>\n",
       "            </span>\n",
       "          </div>\n",
       "        </nav>\n",
       "        <div class=\"container-fluid\">\n",
       "          <div class=\"row\">\n",
       "            <div class=\"col-12\">\n",
       "              <h3 style=\"vertical-align: bottom; white-space: nowrap; overflow: hidden;\n",
       "                text-overflow: ellipsis;\">\n",
       "                Spark Jobs\n",
       "                <sup>\n",
       "      (<a data-toggle=\"tooltip\" data-placement=\"top\" title=\"A job is triggered by an action, like count() or saveAsTextFile(). Click on a job to see information about the stages of tasks inside it.\">?</a>)\n",
       "    </sup>\n",
       "              </h3>\n",
       "            </div>\n",
       "          </div>\n",
       "          <div class=\"row\">\n",
       "            <div class=\"col-12\">\n",
       "              <div>\n",
       "        <ul class=\"list-unstyled\">\n",
       "          <li>\n",
       "\n",
       "            <strong>User:</strong>\n",
       "            jovyan\n",
       "          </li>\n",
       "          <li>\n",
       "            <strong>Total Uptime:</strong>\n",
       "            19 min\n",
       "          </li>\n",
       "          <li>\n",
       "            <strong>Scheduling Mode: </strong>\n",
       "            FIFO\n",
       "          </li>\n",
       "          \n",
       "          <li id=\"completed-summary\">\n",
       "                <a href=\"#completed\"><strong>Completed Jobs:</strong></a>\n",
       "                21\n",
       "              </li>\n",
       "          \n",
       "        </ul>\n",
       "      </div><span class=\"expand-application-timeline\">\n",
       "      <span class=\"expand-application-timeline-arrow arrow-closed\"></span>\n",
       "      <a data-toggle=\"tooltip\" title=\"Shows when jobs started and ended and when executors joined or left. Drag to scroll.\n",
       "       Click Enable Zooming and use mouse wheel to zoom in/out.\" data-placement=\"top\">\n",
       "        Event Timeline\n",
       "      </a>\n",
       "    </span><div id=\"application-timeline\" class=\"collapsed\">\n",
       "      \n",
       "      \n",
       "      <div class=\"control-panel\">\n",
       "        <div id=\"application-timeline-zoom-lock\">\n",
       "          <input type=\"checkbox\"></input>\n",
       "          <span>Enable zooming</span>\n",
       "        </div>\n",
       "      </div>\n",
       "    </div><script type=\"text/javascript\">\n",
       "      drawApplicationTimeline(\n",
       "[\n",
       "  {\n",
       "    'id': 'executors',\n",
       "    'content': '<div>Executors</div><div class=\"legend-area\"><svg width=\"150px\" height=\"55px\">      <rect class=\"executor-added-legend\" x=\"5px\" y=\"5px\" width=\"20px\" height=\"15px\" rx=\"2px\" ry=\"2px\"></rect>      <text x=\"35px\" y=\"17px\">Added</text>      <rect class=\"executor-removed-legend\" x=\"5px\" y=\"30px\" width=\"20px\" height=\"15px\" rx=\"2px\" ry=\"2px\"></rect>      <text x=\"35px\" y=\"42px\">Removed</text>    </svg></div>',\n",
       "  },\n",
       "  {\n",
       "    'id': 'jobs',\n",
       "    'content': '<div>Jobs</div><div class=\"legend-area\"><svg width=\"150px\" height=\"85px\">      <rect class=\"succeeded-job-legend\" x=\"5px\" y=\"5px\" width=\"20px\" height=\"15px\" rx=\"2px\" ry=\"2px\"></rect>      <text x=\"35px\" y=\"17px\">Succeeded</text>      <rect class=\"failed-job-legend\" x=\"5px\" y=\"30px\" width=\"20px\" height=\"15px\" rx=\"2px\" ry=\"2px\"></rect>      <text x=\"35px\" y=\"42px\">Failed</text>      <rect class=\"running-job-legend\" x=\"5px\" y=\"55px\" width=\"20px\" height=\"15px\" rx=\"2px\" ry=\"2px\"></rect>      <text x=\"35px\" y=\"67px\">Running</text>    </svg></div>',\n",
       "  }\n",
       "]\n",
       "        ,[\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682583403729),\n",
       "  'end': new Date(1682583405518),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"csv at NativeMethodAccessorImpl.java:0 (Job 0)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:16:43' +\n",
       "     '<br>Completed: 2023/04/27 08:16:45\">' +\n",
       "    'csv at NativeMethodAccessorImpl.java:0 (Job 0)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682583406300),\n",
       "  'end': new Date(1682583407714),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"csv at NativeMethodAccessorImpl.java:0 (Job 1)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:16:46' +\n",
       "     '<br>Completed: 2023/04/27 08:16:47\">' +\n",
       "    'csv at NativeMethodAccessorImpl.java:0 (Job 1)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682583408411),\n",
       "  'end': new Date(1682583408745),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"showString at NativeMethodAccessorImpl.java:0 (Job 2)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:16:48' +\n",
       "     '<br>Completed: 2023/04/27 08:16:48\">' +\n",
       "    'showString at NativeMethodAccessorImpl.java:0 (Job 2)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682583438844),\n",
       "  'end': new Date(1682583438930),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"csv at NativeMethodAccessorImpl.java:0 (Job 3)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:17:18' +\n",
       "     '<br>Completed: 2023/04/27 08:17:18\">' +\n",
       "    'csv at NativeMethodAccessorImpl.java:0 (Job 3)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682583439117),\n",
       "  'end': new Date(1682583439443),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"csv at NativeMethodAccessorImpl.java:0 (Job 4)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:17:19' +\n",
       "     '<br>Completed: 2023/04/27 08:17:19\">' +\n",
       "    'csv at NativeMethodAccessorImpl.java:0 (Job 4)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682583439833),\n",
       "  'end': new Date(1682583439925),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"showString at NativeMethodAccessorImpl.java:0 (Job 5)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:17:19' +\n",
       "     '<br>Completed: 2023/04/27 08:17:19\">' +\n",
       "    'showString at NativeMethodAccessorImpl.java:0 (Job 5)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682583647438),\n",
       "  'end': new Date(1682583647923),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"showString at NativeMethodAccessorImpl.java:0 (Job 6)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:20:47' +\n",
       "     '<br>Completed: 2023/04/27 08:20:47\">' +\n",
       "    'showString at NativeMethodAccessorImpl.java:0 (Job 6)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682583861826),\n",
       "  'end': new Date(1682583862236),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"showString at NativeMethodAccessorImpl.java:0 (Job 7)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:24:21' +\n",
       "     '<br>Completed: 2023/04/27 08:24:22\">' +\n",
       "    'showString at NativeMethodAccessorImpl.java:0 (Job 7)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682583862504),\n",
       "  'end': new Date(1682583862720),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"showString at NativeMethodAccessorImpl.java:0 (Job 8)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:24:22' +\n",
       "     '<br>Completed: 2023/04/27 08:24:22\">' +\n",
       "    'showString at NativeMethodAccessorImpl.java:0 (Job 8)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682583904916),\n",
       "  'end': new Date(1682583905134),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"collect at \\/tmp\\/ipykernel_67\\/2350353133.py:1 (Job 9)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:25:04' +\n",
       "     '<br>Completed: 2023/04/27 08:25:05\">' +\n",
       "    'collect at \\/tmp\\/ipykernel_67\\/2350353133.py:1 (Job 9)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682583905253),\n",
       "  'end': new Date(1682583905407),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"collect at \\/tmp\\/ipykernel_67\\/2350353133.py:1 (Job 10)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:25:05' +\n",
       "     '<br>Completed: 2023/04/27 08:25:05\">' +\n",
       "    'collect at \\/tmp\\/ipykernel_67\\/2350353133.py:1 (Job 10)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682583935203),\n",
       "  'end': new Date(1682583935411),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"collect at \\/tmp\\/ipykernel_67\\/2462410233.py:1 (Job 11)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:25:35' +\n",
       "     '<br>Completed: 2023/04/27 08:25:35\">' +\n",
       "    'collect at \\/tmp\\/ipykernel_67\\/2462410233.py:1 (Job 11)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682583935500),\n",
       "  'end': new Date(1682583935535),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"collect at \\/tmp\\/ipykernel_67\\/2462410233.py:1 (Job 12)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:25:35' +\n",
       "     '<br>Completed: 2023/04/27 08:25:35\">' +\n",
       "    'collect at \\/tmp\\/ipykernel_67\\/2462410233.py:1 (Job 12)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682583941127),\n",
       "  'end': new Date(1682583941326),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"collect at \\/tmp\\/ipykernel_67\\/2491323169.py:1 (Job 13)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:25:41' +\n",
       "     '<br>Completed: 2023/04/27 08:25:41\">' +\n",
       "    'collect at \\/tmp\\/ipykernel_67\\/2491323169.py:1 (Job 13)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682583941411),\n",
       "  'end': new Date(1682583941439),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"collect at \\/tmp\\/ipykernel_67\\/2491323169.py:1 (Job 14)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:25:41' +\n",
       "     '<br>Completed: 2023/04/27 08:25:41\">' +\n",
       "    'collect at \\/tmp\\/ipykernel_67\\/2491323169.py:1 (Job 14)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682584105125),\n",
       "  'end': new Date(1682584105923),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"showString at NativeMethodAccessorImpl.java:0 (Job 15)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:28:25' +\n",
       "     '<br>Completed: 2023/04/27 08:28:25\">' +\n",
       "    'showString at NativeMethodAccessorImpl.java:0 (Job 15)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682584106330),\n",
       "  'end': new Date(1682584106518),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"showString at NativeMethodAccessorImpl.java:0 (Job 16)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:28:26' +\n",
       "     '<br>Completed: 2023/04/27 08:28:26\">' +\n",
       "    'showString at NativeMethodAccessorImpl.java:0 (Job 16)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682584228960),\n",
       "  'end': new Date(1682584229114),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"showString at NativeMethodAccessorImpl.java:0 (Job 17)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:30:28' +\n",
       "     '<br>Completed: 2023/04/27 08:30:29\">' +\n",
       "    'showString at NativeMethodAccessorImpl.java:0 (Job 17)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682584251419),\n",
       "  'end': new Date(1682584251624),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"showString at NativeMethodAccessorImpl.java:0 (Job 18)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:30:51' +\n",
       "     '<br>Completed: 2023/04/27 08:30:51\">' +\n",
       "    'showString at NativeMethodAccessorImpl.java:0 (Job 18)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682584270833),\n",
       "  'end': new Date(1682584271012),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"showString at NativeMethodAccessorImpl.java:0 (Job 19)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:31:10' +\n",
       "     '<br>Completed: 2023/04/27 08:31:11\">' +\n",
       "    'showString at NativeMethodAccessorImpl.java:0 (Job 19)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'job application-timeline-object succeeded',\n",
       "  'group': 'jobs',\n",
       "  'start': new Date(1682584309344),\n",
       "  'end': new Date(1682584309429),\n",
       "  'content': '<div class=\"application-timeline-content\"' +\n",
       "     'data-html=\"true\" data-placement=\"top\" data-toggle=\"tooltip\"' +\n",
       "     'data-title=\"showString at NativeMethodAccessorImpl.java:0 (Job 20)<br>' +\n",
       "     'Status: SUCCEEDED<br>' +\n",
       "     'Submitted: 2023/04/27 08:31:49' +\n",
       "     '<br>Completed: 2023/04/27 08:31:49\">' +\n",
       "    'showString at NativeMethodAccessorImpl.java:0 (Job 20)</div>'\n",
       "}\n",
       "         ,\n",
       "{\n",
       "  'className': 'executor added',\n",
       "  'group': 'executors',\n",
       "  'start': new Date(1682583245506),\n",
       "  'content': '<div class=\"executor-event-content\"' +\n",
       "    'data-toggle=\"tooltip\" data-placement=\"top\"' +\n",
       "    'data-title=\"Executor driver<br>' +\n",
       "    'Added at 2023/04/27 08:14:05\"' +\n",
       "    'data-html=\"true\">Executor driver added</div>'\n",
       "}\n",
       "         ], 1682583242934, 0);\n",
       "    </script><span id=\"completed\" class=\"collapse-aggregated-completedJobs collapse-table\" onClick=\"collapseTable('collapse-aggregated-completedJobs','aggregated-completedJobs')\">\n",
       "          <h4>\n",
       "            <span class=\"collapse-table-arrow arrow-open\"></span>\n",
       "            <a>Completed Jobs (21)</a>\n",
       "          </h4>\n",
       "        </span><div class=\"aggregated-completedJobs collapsible-table\">\n",
       "          <div>\n",
       "        <div>\n",
       "      <div>\n",
       "        <form id=\"form-completedJob-table-top-page\" method=\"get\" action=\"/jobs/?&completedJob.sort=Job+Id&completedJob.desc=true#completed\" class=\"form-inline float-right justify-content-end\" style=\"margin-bottom: 0px;\">\n",
       "          <input type=\"hidden\" name=\"completedJob.sort\" value=\"Job Id\"/><input type=\"hidden\" name=\"completedJob.desc\" value=\"true\"/>\n",
       "          <label>1 Pages. Jump to</label>\n",
       "          <input type=\"text\" name=\"completedJob.page\" id=\"form-completedJob-table-top-page-no\" value=\"1\" class=\"col-1 form-control\"/>\n",
       "\n",
       "          <label>. Show </label>\n",
       "          <input type=\"text\" id=\"form-completedJob-table-top-page-size\" name=\"completedJob.pageSize\" value=\"100\" class=\"col-1 form-control\"/>\n",
       "          <label>items in a page.</label>\n",
       "\n",
       "          <button type=\"submit\" class=\"btn btn-spark\">Go</button>\n",
       "        </form>\n",
       "      </div>\n",
       "      <div>\n",
       "        <span style=\"float: left; padding-top: 4px; padding-right: 4px;\">Page: </span>\n",
       "        <ul class=\"pagination\">\n",
       "          \n",
       "          \n",
       "          <li class=\"page-item disabled\"><a href=\"#\" class=\"page-link\">1</a></li>\n",
       "          \n",
       "          \n",
       "        </ul>\n",
       "      </div>\n",
       "    </div>\n",
       "        <table class=\"table table-bordered table-sm table-striped table-head-clickable table-cell-width-limited\" id=\"completedJob-table\">\n",
       "          <thead>\n",
       "      <tr><th>\n",
       "            <a href=\"/jobs/?&completedJob.sort=Job+Id&completedJob.desc=false&completedJob.pageSize=100#completed\">\n",
       "              <span data-toggle=\"tooltip\" data-placement=\"top\" title=\"\">\n",
       "                Job Id&nbsp;&#x25BE;\n",
       "              </span>\n",
       "            </a>\n",
       "          </th><th>\n",
       "              <a href=\"/jobs/?&completedJob.sort=Description&completedJob.pageSize=100#completed\">\n",
       "                <span data-toggle=\"tooltip\" data-placement=\"top\" title=\"\">\n",
       "                  Description\n",
       "                </span>\n",
       "              </a>\n",
       "            </th><th>\n",
       "              <a href=\"/jobs/?&completedJob.sort=Submitted&completedJob.pageSize=100#completed\">\n",
       "                <span data-toggle=\"tooltip\" data-placement=\"top\" title=\"\">\n",
       "                  Submitted\n",
       "                </span>\n",
       "              </a>\n",
       "            </th><th>\n",
       "              <a href=\"/jobs/?&completedJob.sort=Duration&completedJob.pageSize=100#completed\">\n",
       "                <span data-toggle=\"tooltip\" data-placement=\"top\" title=\"Elapsed time since the job was submitted until execution completion of all its stages.\">\n",
       "                  Duration\n",
       "                </span>\n",
       "              </a>\n",
       "            </th><th>\n",
       "              <span data-toggle=\"tooltip\" data-placement=\"top\" title=\"\">\n",
       "                Stages: Succeeded/Total\n",
       "              </span>\n",
       "            </th><th>\n",
       "              <span data-toggle=\"tooltip\" data-placement=\"top\" title=\"\">\n",
       "                Tasks (for all stages): Succeeded/Total\n",
       "              </span>\n",
       "            </th></tr>\n",
       "    </thead>\n",
       "          <tbody>\n",
       "            <tr id=\"job-20\">\n",
       "      <td>\n",
       "        20 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">showString at NativeMethodAccessorImpl.java:0</span> \n",
       "        <a href=\"/jobs/job/?id=20\" class=\"name-link\">showString at NativeMethodAccessorImpl.java:0</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:31:49\n",
       "      </td>\n",
       "      <td>85 ms</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        \n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-19\">\n",
       "      <td>\n",
       "        19 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">showString at NativeMethodAccessorImpl.java:0</span> \n",
       "        <a href=\"/jobs/job/?id=19\" class=\"name-link\">showString at NativeMethodAccessorImpl.java:0</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:31:10\n",
       "      </td>\n",
       "      <td>0.2 s</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        \n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-18\">\n",
       "      <td>\n",
       "        18 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">showString at NativeMethodAccessorImpl.java:0</span> \n",
       "        <a href=\"/jobs/job/?id=18\" class=\"name-link\">showString at NativeMethodAccessorImpl.java:0</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:30:51\n",
       "      </td>\n",
       "      <td>0.2 s</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        \n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-17\">\n",
       "      <td>\n",
       "        17 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">showString at NativeMethodAccessorImpl.java:0</span> \n",
       "        <a href=\"/jobs/job/?id=17\" class=\"name-link\">showString at NativeMethodAccessorImpl.java:0</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:30:28\n",
       "      </td>\n",
       "      <td>0.2 s</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        \n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-16\">\n",
       "      <td>\n",
       "        16 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">showString at NativeMethodAccessorImpl.java:0</span> \n",
       "        <a href=\"/jobs/job/?id=16\" class=\"name-link\">showString at NativeMethodAccessorImpl.java:0</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:28:26\n",
       "      </td>\n",
       "      <td>0.2 s</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        (1 skipped)\n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        (1 skipped)\n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-15\">\n",
       "      <td>\n",
       "        15 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">showString at NativeMethodAccessorImpl.java:0</span> \n",
       "        <a href=\"/jobs/job/?id=15\" class=\"name-link\">showString at NativeMethodAccessorImpl.java:0</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:28:25\n",
       "      </td>\n",
       "      <td>0.8 s</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        \n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-14\">\n",
       "      <td>\n",
       "        14 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">collect at /tmp/ipykernel_67/2491323169.py:1</span> \n",
       "        <a href=\"/jobs/job/?id=14\" class=\"name-link\">collect at /tmp/ipykernel_67/2491323169.py:1</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:25:41\n",
       "      </td>\n",
       "      <td>28 ms</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        (1 skipped)\n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        (1 skipped)\n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-13\">\n",
       "      <td>\n",
       "        13 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">collect at /tmp/ipykernel_67/2491323169.py:1</span> \n",
       "        <a href=\"/jobs/job/?id=13\" class=\"name-link\">collect at /tmp/ipykernel_67/2491323169.py:1</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:25:41\n",
       "      </td>\n",
       "      <td>0.2 s</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        \n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-12\">\n",
       "      <td>\n",
       "        12 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">collect at /tmp/ipykernel_67/2462410233.py:1</span> \n",
       "        <a href=\"/jobs/job/?id=12\" class=\"name-link\">collect at /tmp/ipykernel_67/2462410233.py:1</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:25:35\n",
       "      </td>\n",
       "      <td>35 ms</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        (1 skipped)\n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        (1 skipped)\n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-11\">\n",
       "      <td>\n",
       "        11 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">collect at /tmp/ipykernel_67/2462410233.py:1</span> \n",
       "        <a href=\"/jobs/job/?id=11\" class=\"name-link\">collect at /tmp/ipykernel_67/2462410233.py:1</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:25:35\n",
       "      </td>\n",
       "      <td>0.2 s</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        \n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-10\">\n",
       "      <td>\n",
       "        10 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">collect at /tmp/ipykernel_67/2350353133.py:1</span> \n",
       "        <a href=\"/jobs/job/?id=10\" class=\"name-link\">collect at /tmp/ipykernel_67/2350353133.py:1</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:25:05\n",
       "      </td>\n",
       "      <td>0.2 s</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        (1 skipped)\n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        (1 skipped)\n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-9\">\n",
       "      <td>\n",
       "        9 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">collect at /tmp/ipykernel_67/2350353133.py:1</span> \n",
       "        <a href=\"/jobs/job/?id=9\" class=\"name-link\">collect at /tmp/ipykernel_67/2350353133.py:1</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:25:04\n",
       "      </td>\n",
       "      <td>0.2 s</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        \n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-8\">\n",
       "      <td>\n",
       "        8 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">showString at NativeMethodAccessorImpl.java:0</span> \n",
       "        <a href=\"/jobs/job/?id=8\" class=\"name-link\">showString at NativeMethodAccessorImpl.java:0</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:24:22\n",
       "      </td>\n",
       "      <td>0.2 s</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        (1 skipped)\n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        (1 skipped)\n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-7\">\n",
       "      <td>\n",
       "        7 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">showString at NativeMethodAccessorImpl.java:0</span> \n",
       "        <a href=\"/jobs/job/?id=7\" class=\"name-link\">showString at NativeMethodAccessorImpl.java:0</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:24:21\n",
       "      </td>\n",
       "      <td>0.4 s</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        \n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-6\">\n",
       "      <td>\n",
       "        6 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">showString at NativeMethodAccessorImpl.java:0</span> \n",
       "        <a href=\"/jobs/job/?id=6\" class=\"name-link\">showString at NativeMethodAccessorImpl.java:0</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:20:47\n",
       "      </td>\n",
       "      <td>0.5 s</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        \n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-5\">\n",
       "      <td>\n",
       "        5 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">showString at NativeMethodAccessorImpl.java:0</span> \n",
       "        <a href=\"/jobs/job/?id=5\" class=\"name-link\">showString at NativeMethodAccessorImpl.java:0</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:17:19\n",
       "      </td>\n",
       "      <td>92 ms</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        \n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-4\">\n",
       "      <td>\n",
       "        4 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">csv at NativeMethodAccessorImpl.java:0</span> \n",
       "        <a href=\"/jobs/job/?id=4\" class=\"name-link\">csv at NativeMethodAccessorImpl.java:0</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:17:19\n",
       "      </td>\n",
       "      <td>0.3 s</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        \n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-3\">\n",
       "      <td>\n",
       "        3 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">csv at NativeMethodAccessorImpl.java:0</span> \n",
       "        <a href=\"/jobs/job/?id=3\" class=\"name-link\">csv at NativeMethodAccessorImpl.java:0</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:17:18\n",
       "      </td>\n",
       "      <td>86 ms</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        \n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-2\">\n",
       "      <td>\n",
       "        2 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">showString at NativeMethodAccessorImpl.java:0</span> \n",
       "        <a href=\"/jobs/job/?id=2\" class=\"name-link\">showString at NativeMethodAccessorImpl.java:0</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:16:48\n",
       "      </td>\n",
       "      <td>0.3 s</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        \n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-1\">\n",
       "      <td>\n",
       "        1 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">csv at NativeMethodAccessorImpl.java:0</span> \n",
       "        <a href=\"/jobs/job/?id=1\" class=\"name-link\">csv at NativeMethodAccessorImpl.java:0</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:16:46\n",
       "      </td>\n",
       "      <td>1 s</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        \n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr><tr id=\"job-0\">\n",
       "      <td>\n",
       "        0 \n",
       "      </td>\n",
       "      <td>\n",
       "        <span class=\"description-input\">csv at NativeMethodAccessorImpl.java:0</span> \n",
       "        <a href=\"/jobs/job/?id=0\" class=\"name-link\">csv at NativeMethodAccessorImpl.java:0</a>\n",
       "      </td>\n",
       "      <td>\n",
       "        2023/04/27 08:16:43\n",
       "      </td>\n",
       "      <td>2 s</td>\n",
       "      <td class=\"stage-progress-cell\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "      </td>\n",
       "      <td class=\"progress-cell\">\n",
       "        <div class=\"progress\">\n",
       "      <span style=\"text-align:center; position:absolute; width:100%;\">\n",
       "        1/1\n",
       "        \n",
       "        \n",
       "        \n",
       "        \n",
       "      </span>\n",
       "      <div class=\"progress-bar progress-completed\" style=\"width: 100.0%\"></div>\n",
       "      <div class=\"progress-bar progress-started\" style=\"width: 0.0%\"></div>\n",
       "    </div>\n",
       "      </td>\n",
       "    </tr>\n",
       "          </tbody>\n",
       "        </table>\n",
       "        <div>\n",
       "      <div>\n",
       "        <form id=\"form-completedJob-table-bottom-page\" method=\"get\" action=\"/jobs/?&completedJob.sort=Job+Id&completedJob.desc=true#completed\" class=\"form-inline float-right justify-content-end\" style=\"margin-bottom: 0px;\">\n",
       "          <input type=\"hidden\" name=\"completedJob.sort\" value=\"Job Id\"/><input type=\"hidden\" name=\"completedJob.desc\" value=\"true\"/>\n",
       "          <label>1 Pages. Jump to</label>\n",
       "          <input type=\"text\" name=\"completedJob.page\" id=\"form-completedJob-table-bottom-page-no\" value=\"1\" class=\"col-1 form-control\"/>\n",
       "\n",
       "          <label>. Show </label>\n",
       "          <input type=\"text\" id=\"form-completedJob-table-bottom-page-size\" name=\"completedJob.pageSize\" value=\"100\" class=\"col-1 form-control\"/>\n",
       "          <label>items in a page.</label>\n",
       "\n",
       "          <button type=\"submit\" class=\"btn btn-spark\">Go</button>\n",
       "        </form>\n",
       "      </div>\n",
       "      <div>\n",
       "        <span style=\"float: left; padding-top: 4px; padding-right: 4px;\">Page: </span>\n",
       "        <ul class=\"pagination\">\n",
       "          \n",
       "          \n",
       "          <li class=\"page-item disabled\"><a href=\"#\" class=\"page-link\">1</a></li>\n",
       "          \n",
       "          \n",
       "        </ul>\n",
       "      </div>\n",
       "    </div>\n",
       "      </div>\n",
       "        </div>\n",
       "            </div>\n",
       "          </div>\n",
       "        </div>\n",
       "      </body>\n",
       "    </html>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# On your Spark UI, you will see the operations as they are being executed. Here we see that the above operations have been successfully executed:\n",
    "render_local_spark_ui(subpage=\"jobs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "ccffe51c-ea61-488b-8b2d-47fcc59c0517",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "== Physical Plan ==\n",
      "AdaptiveSparkPlan isFinalPlan=false\n",
      "+- Sort [month#149 ASC NULLS FIRST], true, 0\n",
      "   +- Exchange rangepartitioning(month#149 ASC NULLS FIRST, 200), ENSURE_REQUIREMENTS, [plan_id=332]\n",
      "      +- HashAggregate(keys=[month#149], functions=[count(1)])\n",
      "         +- Exchange hashpartitioning(month#149, 200), ENSURE_REQUIREMENTS, [plan_id=329]\n",
      "            +- HashAggregate(keys=[month#149], functions=[partial_count(1)])\n",
      "               +- FileScan csv [month#149] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex(1 paths)[file:/home/jovyan/flights_small.csv], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<month:int>\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Visualizing the DAG for an operation: (read it bottom-up.)\n",
    "flight_stats = flights.groupby(\"month\").count().orderBy(\"month\")\n",
    "flight_stats.explain()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16bf12ce-ee38-4103-849f-9869c34d7fd0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feel free to explore a bit more!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c15701b-d643-4005-97a7-0e7ed37bdaa3",
   "metadata": {},
   "source": [
    "## SparkML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "58b09100-6477-42ba-823a-6d0a91b89b3f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- year: integer (nullable = true)\n",
      " |-- month: integer (nullable = true)\n",
      " |-- day: integer (nullable = true)\n",
      " |-- dep_time: string (nullable = true)\n",
      " |-- dep_delay: string (nullable = true)\n",
      " |-- arr_time: string (nullable = true)\n",
      " |-- arr_delay: string (nullable = true)\n",
      " |-- carrier: string (nullable = true)\n",
      " |-- tailnum: string (nullable = true)\n",
      " |-- flight: integer (nullable = true)\n",
      " |-- origin: string (nullable = true)\n",
      " |-- dest: string (nullable = true)\n",
      " |-- air_time: string (nullable = true)\n",
      " |-- distance: integer (nullable = true)\n",
      " |-- hour: string (nullable = true)\n",
      " |-- minute: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "flights.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b52b70ef-9910-4b10-91d2-eaf435031d8d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['year',\n",
       " 'month',\n",
       " 'day',\n",
       " 'dep_time',\n",
       " 'dep_delay',\n",
       " 'arr_time',\n",
       " 'arr_delay',\n",
       " 'carrier',\n",
       " 'tailnum',\n",
       " 'flight',\n",
       " 'origin',\n",
       " 'dest',\n",
       " 'air_time',\n",
       " 'distance',\n",
       " 'hour',\n",
       " 'minute']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flights.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8f4fc8ff-017c-46f8-911c-dfd76e9de110",
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'numpy'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[8], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpyspark\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mml\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mclassification\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m DecisionTreeClassifier\n\u001b[1;32m      3\u001b[0m clf \u001b[38;5;241m=\u001b[39m DecisionTreeClassifier(inputCols\u001b[38;5;241m=\u001b[39m[col \u001b[38;5;28;01mfor\u001b[39;00m col \u001b[38;5;129;01min\u001b[39;00m flights\u001b[38;5;241m.\u001b[39mcolumns \u001b[38;5;28;01mif\u001b[39;00m col \u001b[38;5;241m!=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124marr_delay\u001b[39m\u001b[38;5;124m\"\u001b[39m], outputCol\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124marr_delay\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m      4\u001b[0m df_train, df_test \u001b[38;5;241m=\u001b[39m flights\u001b[38;5;241m.\u001b[39mrandomSplit([\u001b[38;5;241m0.8\u001b[39m, \u001b[38;5;241m0.2\u001b[39m], seed\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m42\u001b[39m)\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.10/site-packages/pyspark/ml/__init__.py:22\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;66;03m#\u001b[39;00m\n\u001b[1;32m      2\u001b[0m \u001b[38;5;66;03m# Licensed to the Apache Software Foundation (ASF) under one or more\u001b[39;00m\n\u001b[1;32m      3\u001b[0m \u001b[38;5;66;03m# contributor license agreements.  See the NOTICE file distributed with\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[38;5;66;03m# limitations under the License.\u001b[39;00m\n\u001b[1;32m     16\u001b[0m \u001b[38;5;66;03m#\u001b[39;00m\n\u001b[1;32m     18\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m     19\u001b[0m \u001b[38;5;124;03mDataFrame-based machine learning APIs to let users quickly assemble and configure practical\u001b[39;00m\n\u001b[1;32m     20\u001b[0m \u001b[38;5;124;03mmachine learning pipelines.\u001b[39;00m\n\u001b[1;32m     21\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m---> 22\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpyspark\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mml\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mbase\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m (\n\u001b[1;32m     23\u001b[0m     Estimator,\n\u001b[1;32m     24\u001b[0m     Model,\n\u001b[1;32m     25\u001b[0m     Predictor,\n\u001b[1;32m     26\u001b[0m     PredictionModel,\n\u001b[1;32m     27\u001b[0m     Transformer,\n\u001b[1;32m     28\u001b[0m     UnaryTransformer,\n\u001b[1;32m     29\u001b[0m )\n\u001b[1;32m     30\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpyspark\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mml\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mpipeline\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Pipeline, PipelineModel\n\u001b[1;32m     31\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpyspark\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mml\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m (\n\u001b[1;32m     32\u001b[0m     classification,\n\u001b[1;32m     33\u001b[0m     clustering,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     44\u001b[0m     param,\n\u001b[1;32m     45\u001b[0m )\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.10/site-packages/pyspark/ml/base.py:40\u001b[0m\n\u001b[1;32m     23\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mtyping\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m (\n\u001b[1;32m     24\u001b[0m     Any,\n\u001b[1;32m     25\u001b[0m     Callable,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     36\u001b[0m     TYPE_CHECKING,\n\u001b[1;32m     37\u001b[0m )\n\u001b[1;32m     39\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpyspark\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m since\n\u001b[0;32m---> 40\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpyspark\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mml\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mparam\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m P\n\u001b[1;32m     41\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpyspark\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mml\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcommon\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m inherit_doc\n\u001b[1;32m     42\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpyspark\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mml\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mparam\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mshared\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m (\n\u001b[1;32m     43\u001b[0m     HasInputCol,\n\u001b[1;32m     44\u001b[0m     HasOutputCol,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     48\u001b[0m     Params,\n\u001b[1;32m     49\u001b[0m )\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.10/site-packages/pyspark/ml/param/__init__.py:32\u001b[0m\n\u001b[1;32m     19\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mcopy\u001b[39;00m\n\u001b[1;32m     20\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mtyping\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m (\n\u001b[1;32m     21\u001b[0m     Any,\n\u001b[1;32m     22\u001b[0m     Callable,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     29\u001b[0m     TYPE_CHECKING,\n\u001b[1;32m     30\u001b[0m )\n\u001b[0;32m---> 32\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mnumpy\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mnp\u001b[39;00m\n\u001b[1;32m     33\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpy4j\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mjava_gateway\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m JavaObject\n\u001b[1;32m     35\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpyspark\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mml\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mlinalg\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m DenseVector, Vector, Matrix\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'numpy'"
     ]
    }
   ],
   "source": [
    "from pyspark.ml.classification import DecisionTreeClassifier\n",
    "\n",
    "clf = DecisionTreeClassifier(inputCols=[col for col in flights.columns if col != \"arr_delay\"], outputCol=\"arr_delay\")\n",
    "df_train, df_test = flights.randomSplit([0.8, 0.2], seed=42)\n",
    "clf.fit(df_train)\n",
    "df_test_with_predictions = clf.transform(df_test)\n",
    "df_test_with_predictions.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d1fc57e-b0d6-418d-a302-08d20b4ec2ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feel free to experiment some more! You can use the mindmap shared with this course for inspiration."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "973b0da2-f295-47d1-ab8d-79b848f60d5c",
   "metadata": {},
   "source": [
    "## Efficient querying"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "73279439-b0a1-4275-bbd1-497c07fdece0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m\n",
       "\u001b[0mflights\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwrite\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparquet\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mpath\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mmode\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mpartitionBy\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mcompression\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Saves the content of the :class:`DataFrame` in Parquet format at the specified path.\n",
       "\n",
       ".. versionadded:: 1.4.0\n",
       "\n",
       ".. versionchanged:: 3.4.0\n",
       "    Supports Spark Connect.\n",
       "\n",
       "Parameters\n",
       "----------\n",
       "path : str\n",
       "    the path in any Hadoop supported file system\n",
       "mode : str, optional\n",
       "    specifies the behavior of the save operation when data already exists.\n",
       "\n",
       "    * ``append``: Append contents of this :class:`DataFrame` to existing data.\n",
       "    * ``overwrite``: Overwrite existing data.\n",
       "    * ``ignore``: Silently ignore this operation if data already exists.\n",
       "    * ``error`` or ``errorifexists`` (default case): Throw an exception if data already                 exists.\n",
       "partitionBy : str or list, optional\n",
       "    names of partitioning columns\n",
       "\n",
       "Other Parameters\n",
       "----------------\n",
       "Extra options\n",
       "    For the extra options, refer to\n",
       "    `Data Source Option <https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#data-source-option>`_\n",
       "    for the version you use.\n",
       "\n",
       "    .. # noqa\n",
       "\n",
       "Examples\n",
       "--------\n",
       "Write a DataFrame into a Parquet file and read it back.\n",
       "\n",
       ">>> import tempfile\n",
       ">>> with tempfile.TemporaryDirectory() as d:\n",
       "...     # Write a DataFrame into a Parquet file\n",
       "...     spark.createDataFrame(\n",
       "...         [{\"age\": 100, \"name\": \"Hyukjin Kwon\"}]\n",
       "...     ).write.parquet(d, mode=\"overwrite\")\n",
       "...\n",
       "...     # Read the Parquet file as a DataFrame.\n",
       "...     spark.read.format(\"parquet\").load(d).show()\n",
       "+---+------------+\n",
       "|age|        name|\n",
       "+---+------------+\n",
       "|100|Hyukjin Kwon|\n",
       "+---+------------+\n",
       "\u001b[0;31mFile:\u001b[0m      /srv/conda/envs/notebook/lib/python3.10/site-packages/pyspark/sql/readwriter.py\n",
       "\u001b[0;31mType:\u001b[0m      method"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "?flights.write.parquet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c3a33ad6-244e-4365-9b86-322c5df77f14",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "flights.write.parquet(\"flights_results.parquet\", partitionBy=[\"year\", \"month\", \"day\"]) # you could have specified even extra columns: [\"origin\", \"year\", \"month\", \"day\", \"hour\"]... Note the order! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12bfb5f2-ad88-4ed1-8b78-44abf6256ae4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# check the folder structure that appeared, in the folder explorer! (on the left in this window.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "af788d32-a903-4d9b-aac3-9128ac74ccf8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------+---------+--------+---------+-------+-------+------+------+----+--------+--------+----+------+\n",
      "|dep_time|dep_delay|arr_time|arr_delay|carrier|tailnum|flight|origin|dest|air_time|distance|hour|minute|\n",
      "+--------+---------+--------+---------+-------+-------+------+------+----+--------+--------+----+------+\n",
      "|    1719|       -6|    2130|      -15|     AS| N590AS|   875|   SEA| KOA|     342|    2688|  17|    19|\n",
      "|    1326|       -4|    1600|       -9|     AS| N706AS|   494|   SEA| SAN|     129|    1050|  13|    26|\n",
      "|     950|      -10|    1358|       -7|     HA| N373HA|    29|   SEA| OGG|     333|    2640|   9|    50|\n",
      "|     806|       -4|    1352|       -3|     AA| N4XBAA|  1608|   PDX| DFW|     202|    1616|   8|     6|\n",
      "|    1729|       24|    1928|       23|     WN| N602SW|   894|   SEA| OAK|      97|     671|  17|    29|\n",
      "|    1205|       30|    1434|       29|     B6| N662JB|   107|   SEA| LGB|     134|     965|  12|     5|\n",
      "|    2120|       -5|       6|       -9|     AS| N768AS|   121|   SEA| ANC|     201|    1448|  21|    20|\n",
      "|     758|       -2|    1043|      -12|     AS| N626AS|    83|   SEA| ANC|     200|    1448|   7|    58|\n",
      "|     824|       24|     920|       20|     AS| N769AS|   690|   SEA| GEG|      41|     224|   8|    24|\n",
      "|    1402|       -8|    2149|       -6|     AS| N517AS|     2|   SEA| DCA|     260|    2329|  14|     2|\n",
      "|     641|        1|     815|       -5|     VX| N851VA|   805|   PDX| SFO|      79|     550|   6|    41|\n",
      "|    1319|       -1|    1610|        6|     OO| N919SW|  6371|   SEA| LAX|     136|     954|  13|    19|\n",
      "|    2257|       -3|       6|        9|     AS| N532AS|   698|   SEA| GEG|      43|     224|  22|    57|\n",
      "|     825|        0|     955|      -10|     WN| N600WN|  4564|   PDX| OAK|      78|     543|   8|    25|\n",
      "|     512|       -3|     904|        4|     US| N653AW|   480|   SEA| PHX|     154|    1107|   5|    12|\n",
      "|     546|        1|     916|       -4|     WN| N8323C|   757|   PDX| DEN|     126|     991|   5|    46|\n",
      "|    2054|       -1|    2326|        7|     AS| N519AS|   534|   SEA| ONT|     127|     956|  20|    54|\n",
      "|     551|       -4|    1202|       12|     AA| N3HXAA|  1094|   SEA| DFW|     211|    1660|   5|    51|\n",
      "|    1755|        0|    2149|       10|     US| N536UW|   576|   SEA| PHX|     149|    1107|  17|    55|\n",
      "|    1345|        0|    1549|       -7|     AS| N461AS|   570|   PDX| LAX|     111|     834|  13|    45|\n",
      "+--------+---------+--------+---------+-------+-------+------+------+----+--------+--------+----+------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# You can load a small partition to play around!\n",
    "flights_31dec = spark.read.parquet(\"flights_results.parquet/year=2014/month=12/day=31\")\n",
    "flights_31dec.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25573a0f-c5b9-40c7-bde6-b18bec7feaf7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feel free to play around a bit more! We did not demonstrate all functionality that we presented in the course!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "802cc119-5661-4f88-8e78-86eead8d053c",
   "metadata": {},
   "outputs": [],
   "source": [
    "spark.stop()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50782219-ea38-45a0-99ab-d0967a840989",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}