Skip to content

Commit

Permalink
Upgrades dependency to use a more compatible version of spark
Browse files Browse the repository at this point in the history
The new spark integration relies heavily on the new pandas UDF API (3+).
3.2+ also brings in pandas by default, so I htink it makes sense to
update the dependency. If someone wants, they can always bypass it or
reach out to us.
  • Loading branch information
elijahbenizzy committed Aug 5, 2023
1 parent 8ca4ef6 commit 4adca5d
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ def load_requirements():
"dask-diagnostics": ["dask[diagnostics]"],
"dask-distributed": ["dask[distributed]"],
"ray": ["ray>=2.0.0", "pyarrow"],
"pyspark": ["pyspark[pandas_on_spark]", "pandas<2.0"], # I'm sure they'll add support soon,
"pyspark": ["pyspark>3.2", "pandas<2.0"], # I'm sure they'll add support soon,
# but for now its not compatible
"pandera": ["pandera"],
},
Expand Down

0 comments on commit 4adca5d

Please sign in to comment.