Skip to content

Commit

Permalink
Removes unused code.
Browse files Browse the repository at this point in the history
  • Loading branch information
skrawcz committed Aug 19, 2023
1 parent ce2abec commit 7db102c
Showing 1 changed file with 0 additions and 12 deletions.
12 changes: 0 additions & 12 deletions examples/LLM_Workflows/pdf_summarizer/run_on_spark/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,6 @@
from hamilton.experimental import h_spark


def create_hamilton_driver(config: dict, modules: list) -> driver.Driver:
"""Helper function to create a Hamilton driver.
:param config: any configuration required to instantiate a DAG.
:param modules: the modules to crawl to get functions from to build the DAG.
:return: an instantiated Hamilton driver.
"""
adapter = h_spark.PySparkUDFGraphAdapter()
dr = driver.Driver(config, *modules, adapter=adapter) # can pass in multiple modules
return dr


def my_spark_job(spark: SparkSession, openai_gpt_model: str, content_type: str, user_query: str):
"""Template for a Spark job that uses Hamilton for their featuring engineering, i.e. any map, operations.
Expand Down

0 comments on commit 7db102c

Please sign in to comment.