00 - Welcome.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# Welcome to the Kedro-Ibis tutorial!\n",
    "\n",
    "## Outline\n",
    "\n",
    "- Introduction\n",
    "  - Who we are\n",
    "  - Workshop material\n",
    "  - Setup\n",
    "  - Motivation\n",
    "- Expressive analytics at any scale: Introduction to Ibis\n",
    "- From prototype to production: Introduction to Kedro\n",
    "- Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "source": [
    "These are the notebooks for the tutorial: 👇\n",
    "\n",
    "1. [Getting Started with Ibis](./01%20-%20Getting%20Started%20with%20Ibis.ipynb)\n",
    "2. [Switching Backends](./02%20-%20Switching%20Backends.ipynb)\n",
    "3. [First Steps with Kedro](./03%20-%20First%20Steps%20with%20Kedro.ipynb)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Introduction\n",
    "\n",
    "### Who we are\n",
    "\n",
    "|  |  |\n",
    "|--------|------|\n",
    "| ![Deepyaman](static/deepyaman.jpg) | **Deepyaman Datta**<br><br>Deepyaman is a software engineer at Voltron Data. Before their acquisition by Voltron Data, he was a Founding Machine Learning Engineer at  Claypot AI, working on their real-time feature engineering platform. Prior to that, he led data engineering teams and asset development across a range of  industries at QuantumBlack, AI by McKinsey. |\n",
    "| ![Juan Luis](static/juanluis.png) | **Juan Luis Cano Rodríguez**<br><br>Juan Luis (he/him/él) is an Aerospace Engineer with a passion for tech communities, outreach, and sustainability. He works at QuantumBlack, AI by McKinsey, as Product Manager for Kedro, an  opinionated Python framework for creating reproducible, maintainable and modular data science code. He has worked as Developer Advocate at Read  the Docs, as software engineer in the space, consulting, and banking industries, and as a Python trainer for several private and public entities. |"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Workshop material\n",
    "\n",
    "**https://github.com/ibis-project/kedro-ibis-tutorial**\n",
    "\n",
    "![QR Code](static/qr.png)\n",
    "\n",
    "_Note: This will be a lot of material for a 90-minute tutorial; we'll go fast and not go too much in depth, but will be more than happy to answer questions later!_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "1. Open URL above\n",
    "2. Hit 🟩 \"Create codespace on main\"\n",
    "3. Open `00 - Welcome.ipynb` notebook and follow instructions\n",
    "\n",
    "<img src=\"static/codespaces.png\" width=\"400\" alt=\"Codespaces\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Setup\n",
    "\n",
    "Let's start by downloading the [nycflights13 data](https://github.com/hadley/nycflights13); we'll use this dataset throughout the tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import ibis\n",
    "\n",
    "con = ibis.connect(\"duckdb://nycflights13.ddb\")\n",
    "con.create_table(\n",
    "    \"flights\", ibis.examples.nycflights13_flights.fetch().to_pyarrow(), overwrite=True\n",
    ")\n",
    "con.create_table(\n",
    "    \"weather\", ibis.examples.nycflights13_weather.fetch().to_pyarrow(), overwrite=True\n",
    ")\n",
    "con.disconnect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Next, we'll load the data into a local PostgreSQL database using DuckDB—[yes, you can do that](https://duckdb.org/docs/extensions/postgres.html#writing-data-to-postgres)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "!psql < sql/create_nycflights13.sql"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "!duckdb nycflights13.ddb < sql/load_nycflights13.sql"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "We can confirm that our PostgreSQL database contains the tables we just populated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "!psql < sql/verify_nycflights13.sql"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Motivation\n",
    "\n",
    "In your experience doing data analytics/building data pipelines, have you ever..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- ...slurped up large amounts of data into memory, instead of pushing execution down to the source database/engine?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- ...prototyped code in pandas, and then rewritten it in PySpark/Snowpark/some other native dataframe API?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- ...implemented a proof-of-concept solution on data extracts, and then struggled massively when you needed to move to running against the production databases and scale out?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "- ...insisted on using Python across the full data engineering/data science workflow for consistency (fair enough), although dbt would have been the much better fit for non-ML pipelines, because you essentially needed a SQL workflow?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}