Skip to content

Commit

Permalink
docs(linkedin-talk): update some streaming slides based on the dry ru…
Browse files Browse the repository at this point in the history
…n feedback (#26)

<!--
Thanks for taking the time to contribute to Ibis!

Please ensure that your pull request title matches the conventional
commits
specification: https://www.conventionalcommits.org/en/v1.0.0/
-->

## Description of changes

<!--
Write a description of the changes commensurate with the pull request's
scope.

Extremely small changes such as fixing typos do not need a description.
-->

## Issues closed

<!--
Please add Resolves #<issue number> (no angle brackets) if this pull
request
resolves any outstanding issues.

For example, if your pull requests resolves issues 1000, 2000 and 3000
write:

* Resolves ibis-project#1000
* Resolves ibis-project#2000
* Resolves ibis-project#3000

If your pull request doesn't resolve any issues, you can delete this
section
entirely, including the `## Issues closed` section header.
-->

---------

Co-authored-by: Chloe He <chloe@chloe-mac.lan>
  • Loading branch information
chloeh13q and Chloe He authored Apr 24, 2024
1 parent bc404bd commit 9029672
Show file tree
Hide file tree
Showing 6 changed files with 50 additions and 14 deletions.
Binary file added docs/presentations/cds.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/presentations/cds_manifesto.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/presentations/engines.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/presentations/growth.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
64 changes: 50 additions & 14 deletions docs/presentations/linkedin-meetup-2024-04-24.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -512,20 +512,28 @@ ibis.to_sql(expr)
- 54% of Databricks’ customers are using Spark Structured Streaming
- The stream processing market is expected to grow at a compound annual growth rate (CAGR) of 21.5% from 2022 to 2028 (IDC)

![](./growth.png){fig-align="center"}

## Batch and streaming {background-image="./qrcode.svg" background-size="5%" background-position="96% 96%"}

```{mermaid}
%%| fig-width: 20
%%| fig-height: 10
%%| fig-align: center
graph LR
classDef white color:white;
A[data] --> B[batch processing] & C[stream processing] --> D[downstream]
class A white;
class B white;
class C white;
class D white;
subgraph " "
direction LR
A[data] --> B[batch processing] & C[stream processing] --> D[downstream]
end
```

::: {.fragment}
![](./engines.png){width="60%" fig-align="center"}
:::

## In the machine learning world... {background-image="./qrcode.svg" background-size="5%" background-position="96% 96%"}

::: {.fragment}
```{mermaid}
graph TB
proddata --> sampled
Expand All @@ -541,10 +549,11 @@ graph TB
proddata[production data] --> prodpipeline[production pipelines]
end
```
:::

## {#local-experimentation-to-production-image}

[Insert image here]
![](./throwing.png)

## In the machine learning world... {background-image="./qrcode.svg" background-size="5%" background-position="96% 96%"}

Expand All @@ -568,9 +577,9 @@ graph TB

## A real-world example {background-image="./qrcode.svg" background-size="5%" background-position="96% 96%"}

:::: {.columns}
:::: {.r-hstack}

::: {.column width="50%"}
::: {.fragment style="margin-right: 300px;"}
pandas
```python
return (
Expand All @@ -582,9 +591,8 @@ return (
```
:::

::: {.column width="50%"}
::: {.fragment}
Flink SQL

```sql
SELECT
user,
Expand All @@ -603,20 +611,28 @@ FROM clicks

## Code rewrites {background-image="./qrcode.svg" background-size="5%" background-position="96% 96%"}

::: {.incremental}
- From batch to streaming
- From local experimentation to production
- Backfilling a streaming feature on a batch backing table
-
:::

::: {.notes}
Examples for each
:::

## The solution... {background-image="./qrcode.svg" background-size="5%" background-position="96% 96%"}

::: {.fragment}
Stream-batch unified API

- Flink SQL
- Spark DataFrame API
-
:::

## Stream-batch unification {.nostretch} {background-image="./qrcode.svg" background-size="5%" background-position="96% 96%"}
## Stream-batch unification {background-image="./qrcode.svg" background-size="5%" background-position="96% 96%"}

:::: {.columns}

Expand Down Expand Up @@ -649,8 +665,11 @@ FROM clicks
:::
::::

![](./arrow.png){fig-align="center" width=5%}
::: {.fragment}
![](./arrow.png){fig-align="center" width="5%"}
:::

::: {.fragment}
Ibis
```python
agged = clicks.select(
Expand All @@ -662,15 +681,18 @@ agged = clicks.select(
),
)
```
:::

## Challenges of creating a unified API {background-image="./qrcode.svg" background-size="5%" background-position="96% 96%"}
## But it's hard... {background-image="./qrcode.svg" background-size="5%" background-position="96% 96%"}

::: {.incremental}
- Streaming is different
- Time semantics
- Long-running queries
- Sources and sinks
-
- Less established standards in streaming syntax
:::

## Ibis streaming today {background-image="./qrcode.svg" background-size="5%" background-position="96% 96%"}

Expand All @@ -683,6 +705,20 @@ agged = clicks.select(
- Continuously iterate on a stream-batch unified API
- More streaming backends (Spark Structured Streaming)

## Towards composable data systems

:::: {.columns}

::: {.column width="40%"}
![](./cds_manifesto.png)
:::
::: {.column width="15%"}
:::
::: {.column width="40%"}
![](./cds.png)
:::
::::

## Try it out now! {background-image="./qrcode.svg" background-size="5%" background-position="96% 96%"}

::: {.panel-tabset}
Expand Down
Binary file added docs/presentations/throwing.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 9029672

Please sign in to comment.