Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unit testing the code with Spark Connect #87

Open
MrPowers opened this issue Feb 2, 2024 · 0 comments
Open

Unit testing the code with Spark Connect #87

MrPowers opened this issue Feb 2, 2024 · 0 comments

Comments

@MrPowers
Copy link
Owner

MrPowers commented Feb 2, 2024

I created a pull request and confirmed that chispa is fully compliant wth Spark Connect: #86

We don't want to make chispa depend on Spark connect because then it'd have to depend on PySpark >= 3.5. We want chispa to support many old Spark versions for users that are still on legacy Spark runtimes.

It would be nice to test chispa+Spark Connect in the CI tho. We don't want to ever add any chispa code that wouldn't work with Spark Connect. We want all Spark Connnect users to always have access to chispa as well.

Any thoughts on how to add Spark Connect to the CI test suite?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant