Add a notebook to demonstrate how to run notebooks from another notebook

notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/meta.toml

@@ -0,0 +1,13 @@
+[meta]
+authors=["singlestore"]
+title="Executing Notebooks from Another Notebook with Fusion SQL"
+description="""\
+    Learn how to execute Notebooks from another Notebook
+    in SingleStoreDB Cloud using Fusion SQL.
+    """
+icon="files"
+difficulty="intermediate"
+tags=["notebooks", "jobs", "python", "fusion", "starter"]
+lesson_areas=["Python SDK"]
+destinations=["spaces"]
+minimum_tier="standard"

notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb

@@ -0,0 +1,336 @@
+{
+  "cells": [
+    {
+      "id": "55bb0386",
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<div id=\"singlestore-header\" style=\"display: flex; background-color: rgba(255, 224, 129, 0.25); padding: 5px;\">\n",
+        "    <div id=\"icon-image\" style=\"width: 90px; height: 90px;\">\n",
+        "        <img width=\"100%\" height=\"100%\" src=\"https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/common/images/header-icons/files.png\" />\n",
+        "    </div>\n",
+        "    <div id=\"text\" style=\"padding: 5px; margin-left: 10px;\">\n",
+        "        <div id=\"badge\" style=\"display: inline-block; background-color: rgba(0, 0, 0, 0.15); border-radius: 4px; padding: 4px 8px; align-items: center; margin-top: 6px; margin-bottom: -2px; font-size: 80%\">SingleStore Notebooks</div>\n",
+        "        <h1 style=\"font-weight: 500; margin: 8px 0 0 4px;\">Executing Notebooks from Another Notebook with Fusion SQL</h1>\n",
+        "    </div>\n",
+        "</div>"
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "id": "18ed6021",
+      "metadata": {},
+      "source": [
+        "In this notebook, we demonstrate how to use **[Fusion SQL](https://www.singlestore.com/spaces/getting-started-with-fusion-sql/)** to execute a notebook from another notebook, either within the **same session** (useful to avoid code duplication) or in a **new session** (useful for parallel execution)."
+      ]
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Creating the Sample Notebook\n",
+        "In this section, we will create a sample notebook to be used in our examples."
+      ],
+      "id": "eb9a8449"
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Create the sample notebook in the local filesystem:"
+      ],
+      "id": "4a8ca733"
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 1,
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": "Notebook 'sample_notebook.ipynb' created successfully in the local filesystem.\n"
+        }
+      ],
+      "source": [
+        "import nbformat as nbf\n",
+        "\n",
+        "nb = nbf.v4.new_notebook()\n",
+        "\n",
+        "cell = nbf.v4.new_code_cell(\"\"\"# This is a code cell\n",
+        "if 'sample_var' not in globals():\n",
+        "    sample_var = 'sample value'\n",
+        "print('Sample Notebook has been executed!')\"\"\")\n",
+        "\n",
+        "cell.metadata = {\n",
+        "    \"language\": \"python\"\n",
+        "}\n",
+        "\n",
+        "nb.cells.append(cell)\n",
+        "\n",
+        "# Save the notebook to a file\n",
+        "with open('sample_notebook.ipynb', 'w') as f:\n",
+        "    nbf.write(nb, f)\n",
+        "\n",
+        "print(\"Notebook 'sample_notebook.ipynb' created successfully in the local filesystem.\")"
+      ],
+      "id": "470af5d3"
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Upload it to the **Data Studio** shared files for later download:"
+      ],
+      "id": "7c9e8aa3"
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 2,
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": "Notebook 'Sample Notebook 1733503812823709.ipynb' has been created in the Data Studio shared files.\n"
+        }
+      ],
+      "source": [
+        "import time\n",
+        "\n",
+        "sample_notebook_name='Sample Notebook {}.ipynb'.format(int(time.time() * 1_000_000))\n",
+        "\n",
+        "%sql UPLOAD SHARED FILE TO '{{ sample_notebook_name }}' FROM 'sample_notebook.ipynb';\n",
+        "\n",
+        "print(\"Notebook '{}' has been created in the Data Studio shared files.\".format(sample_notebook_name))"
+      ],
+      "id": "ca167d27"
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Executing the Sample Notebook in the Current Session\n",
+        "\n",
+        "In this example, we will execute the previously created sample notebook within the current session. This approach is useful for avoiding code duplication in your notebooks, such as environment setup or reusable functions."
+      ],
+      "id": "e1b704e8"
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "First, we download the notebook to the local filesystem:"
+      ],
+      "id": "81099b51"
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 3,
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": "Notebook 'sample_notebook.ipynb' has been created in the local filesystem.\n"
+        }
+      ],
+      "source": [
+        "%sql DOWNLOAD SHARED FILE '{{ sample_notebook_name }}' TO 'sample_notebook.ipynb' OVERWRITE\n",
+        "\n",
+        "print(\"Notebook 'sample_notebook.ipynb' has been created in the local filesystem.\")"
+      ],
+      "id": "400b350f"
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Next, we execute the notebook and confirm that the `sample_var` variable set in the sample notebook is accessible in the current session:"
+      ],
+      "id": "ca693ad1"
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 4,
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": "Sample Notebook has been executed!\nThe value of 'sample_var' is 'sample value'.\n\n"
+        }
+      ],
+      "source": [
+        "if 'sample_var' in globals():\n",
+        "    del sample_var\n",
+        "\n",
+        "%run sample_notebook.ipynb\n",
+        "\n",
+        "print(\"The value of 'sample_var' is '{}'.\\n\".format(sample_var))"
+      ],
+      "id": "f4903151"
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Executing the Sample Notebook in a New Session\n",
+        "\n",
+        "Instead of executing a notebook in the current session, we can run it in a new session \u2014 either synchronously or asynchronously \u2014 using jobs. This approach is useful for parallel execution or running code in a separate runtime environment.\n",
+        "\n",
+        "<div class=\"alert alert-block alert-warning\">\n",
+        "    <b class=\"fa fa-solid fa-exclamation-circle\"></b>\n",
+        "    <div>\n",
+        "        <p><b>Note</b></p>\n",
+        "        <p>SingleStore enforces <a href=\"https://docs.singlestore.com/cloud/developer-resources/notebooks/using-notebooks/#rate-limits\">limits</a> on the number of compute sessions that can be created, which can be increased upon request. Please contact <a href=\"https://support.singlestore.com/hc/en-us\">Support</a> for more information.</p>\n",
+        "    </div>\n",
+        "</div>"
+      ],
+      "id": "41eabdd5"
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Run **two jobs** in parallel and wait for their completion (each job will run on a separate session):"
+      ],
+      "id": "002acbb1"
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 5,
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": "Running job for 0...\nRunning job for 1...\nWaiting for jobs to complete... ['09d29031-6002-4025-8fed-01623758ebc2', 'd201a83a-a62f-4f71-8c13-34063779a4e3']\nAll jobs completed with success: 1\n"
+        }
+      ],
+      "source": [
+        "job_ids = []\n",
+        "for x in range(2):\n",
+        "    print(\"Running job for {}...\".format(x))\n",
+        "    job_res = %sql RUN JOB USING NOTEBOOK '{{ sample_notebook_name }}' WITH PARAMETERS {\"sample_var\": \"{{x}}\"}\n",
+        "    job_ids.append(job_res[0].JobID)\n",
+        "\n",
+        "print(f'Waiting for jobs to complete... {job_ids}')\n",
+        "success = %sql WAIT ON JOBS {{ job_ids }} WITH TIMEOUT 60 MINUTES\n",
+        "\n",
+        "print(f'All jobs completed with success: {success[0].Success}')"
+      ],
+      "id": "d6986f67"
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "View the executions of the jobs we run:"
+      ],
+      "id": "fd82ec27"
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 6,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "for job_id in job_ids:\n",
+        "  execs = %sql SHOW JOB EXECUTIONS FOR {{ job_id }} from 1 to 1\n",
+        "  print(execs)"
+      ],
+      "id": "bde4b8e0"
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "The jobs can now be dropped:"
+      ],
+      "id": "f0f5ea0e"
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 7,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "for id in job_ids:\n",
+        "    print(f\"Dropping job '{id}'...\")\n",
+        "    %sql DROP JOBS {{id}}"
+      ],
+      "id": "48a77d34"
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Cleanup"
+      ],
+      "id": "ce14b492"
+    },
+    {
+      "attachments": {},
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Delete the sample notebook from the **Data Studio** shared files:"
+      ],
+      "id": "b9ba7ccc"
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 8,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%%sql\n",
+        "DROP SHARED FILE '{{ sample_notebook_name }}'"
+      ],
+      "id": "83aac6db"
+    },
+    {
+      "id": "2c0f6025",
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "<div id=\"singlestore-footer\" style=\"background-color: rgba(194, 193, 199, 0.25); height:2px; margin-bottom:10px\"></div>\n",
+        "<div><img src=\"https://raw.githubusercontent.com/singlestore-labs/spaces-notebooks/master/common/images/singlestore-logo-grey.png\" style=\"padding: 0px; margin: 0px; height: 24px\"/></div>"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "Python 3 (ipykernel)",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.11.9"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 5
+}