Skip to content

Commit

Permalink
Adds image to spark example README
Browse files Browse the repository at this point in the history
To make it clearer what's going on when people
land in the directory.
  • Loading branch information
skrawcz committed Mar 5, 2024
1 parent 66cc3ba commit 92650cf
Showing 1 changed file with 5 additions and 0 deletions.
5 changes: 5 additions & 0 deletions examples/LLM_Workflows/scraping_and_chunking/spark/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,11 @@ Otherwise:
python spark_pipeline.py
```

This is what it'll run:
![pipeline](pipeline.png)
As you can see it'll linearize the process of adding UDFs in the right order to the dataframe based
on the DAG we defined in `doc_pipeline.py`.

# Changes to run on PySpark
Here's why we need to make some minor adjustments to the code.

Expand Down

0 comments on commit 92650cf

Please sign in to comment.