Renames example folder to scraping and chunking

Since that reflects what is happening, rather than creating embeddings. TODOs: - leave notes on how this connects with embeddings.
DAGWorks-Inc · Mar 3, 2024 · 7c5468d · 7c5468d
1 parent 859e881
commit 7c5468d
Show file tree

Hide file tree

Showing 5 changed files with 7 additions and 0 deletions.
diff --git a/examples/LLM_Workflows/scraping_and_chunking/README.md b/examples/LLM_Workflows/scraping_and_chunking/README.md
@@ -0,0 +1,7 @@
+# Scraping and Chunking
+Scraping and chunking are an important part of any RAG dataflow.
+
+Here we show you how you can scale the scraping and chunking dataflow to run in parallel
+locally, as well as with Ray, Dask, and even PySpark.
+
+We'll use creating chunks of text from the Hamilton documentation as an example.
diff --git a/...Workflows/embedding_creation/pipeline.png → ...kflows/scraping_and_chunking/pipeline.png b/...Workflows/embedding_creation/pipeline.png → ...kflows/scraping_and_chunking/pipeline.png
diff --git a/..._Workflows/embedding_creation/pipeline.py → ...rkflows/scraping_and_chunking/pipeline.py b/..._Workflows/embedding_creation/pipeline.py → ...rkflows/scraping_and_chunking/pipeline.py
diff --git a/...flows/embedding_creation/requirements.txt → ...ws/scraping_and_chunking/requirements.txt b/...flows/embedding_creation/requirements.txt → ...ws/scraping_and_chunking/requirements.txt
diff --git a/...s/LLM_Workflows/embedding_creation/run.py → ...LM_Workflows/scraping_and_chunking/run.py b/...s/LLM_Workflows/embedding_creation/run.py → ...LM_Workflows/scraping_and_chunking/run.py