Full spark integration #249
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
elijahbenizzy
force-pushed
the
full-spark-integration
branch
18 times, most recently
from
8699370 to
8ca4ef6
Compare
elijahbenizzy
force-pushed
the
materialization
branch
from
cb12bde to
d370f28
Compare
elijahbenizzy
force-pushed
the
full-spark-integration
branch
from
4adca5d to
8ca4ef6
Compare
elijahbenizzy
force-pushed
the
materialization
branch
9 times, most recently
from
2587e09 to
14a32e1
Compare
elijahbenizzy
force-pushed
the
full-spark-integration
branch
11 times, most recently
from
7cb43c0 to
5b64362
Compare
See issue for more design. Basic overview is:
1. We use `with_columns` to group together map transforms
2. Map transforms can be UDFs (normal) or pandas UDFs
3. THese all get run and "linearized" -- this means that we have two
sets of edges:
-> edges that form the "physical" dependency -- the dataframe geting
passed through and consistently appended to
-> edges that represent logical dependencies -- these are the
original edges in the with_columns group`
While this muddles the edges, this allows us to visualize both the
structure and execution of the DAG. We will likely be adding metadata to
edges to help with visualization.
This has basic documentation in the README, a notebook, and some simple hello_world code/a script. There are a few caveats that are noted in the code.
Dependencies within a with_columns group can come from one of three places: 1. Other columns inside that group 2. The upsteram dataframe 3. External places in the DAG Previously we were requiring users to specify (2) with the initial_schema kwarg, but we now allow them to specify (3) with the external_inputs kwarg.
append will just append to that dataframe select will just select the specified columns from that dataframe
This enables you to extract columns from a cetnral dataframe. Note there are two available approaches: 1. Specify initial_schema to get a set of columns extracted already 2. Specify dataframe_subdag_param to extract them yourself
1. Removes dataframe parameter in favor of using the function's first parameter 2. Renames the dataframe_subdag_param parameter to pass_dataframe_as 3. Renames the initial_schema parameter as columns_to_pass We also update the README to be a little easier to run.
These are in an example. We don't have a notebook yet (as getting the data is a pain and I don't want to link our copy due to licensing), but its great to demonstrate how it works.
elijahbenizzy
force-pushed
the
full-spark-integration
branch
from
5b64362 to
a1ac01d
Compare
skrawcz
reviewed
Aug 22, 2023
hamilton/experimental/h_spark.py
Outdated
| if return_type in (int, float, bool, str, bytes): | ||
| return python_to_spark_type(return_type) | ||
| elif return_type in (list[int], list[float], list[bool], list[str], list[bytes]): |
There was a problem hiding this comment.
this should reference _list above.
There was a problem hiding this comment.
else this will break for python < 3.9
There was a problem hiding this comment.
Yeah this was lost in the merge
elijahbenizzy
force-pushed
the
full-spark-integration
branch
from
5696e81 to
49ed8d6
Compare
skrawcz
approved these changes
Aug 22, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[Short description explaining the high-level reason for the pull request]
Changes
How I tested this
Notes
Checklist