Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(pyspark): suppress errors for unsupported Databricks serverless compute properties #9830

Merged
merged 9 commits into from
Aug 23, 2024

Conversation

ArtnerC
Copy link
Contributor

@ArtnerC ArtnerC commented Aug 13, 2024

Description of changes

Suppress errors when setting spark properties on Databricks serverless compute that are not supported.

Issues closed

@ArtnerC ArtnerC marked this pull request as ready for review August 22, 2024 16:28
@gforsyth
Copy link
Member

Hey @ArtnerC, thanks for putting this in!

The Risingwave failure is spurious, github actions just had a bad morning. But I think the ModuleNotFound error in the pyspark job is legit -- and you can run pre-commit run --all-files to handle the lint error.

@cpcloud
Copy link
Member

cpcloud commented Aug 22, 2024

Will clean up the imports here and merge!

Copy link
Member

@cpcloud cpcloud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for powering through!

@cpcloud cpcloud added bug Incorrect behavior inside of ibis pyspark The Apache PySpark backend labels Aug 23, 2024
@ArtnerC
Copy link
Contributor Author

ArtnerC commented Aug 23, 2024

I think this one is going to do it. Sorry for the spam!

@cpcloud
Copy link
Member

cpcloud commented Aug 23, 2024

Don't worry about the patch failure, the pyspark coverage hasn't been uploaded yet.

@cpcloud cpcloud merged commit 57f5ff6 into ibis-project:main Aug 23, 2024
81 checks passed
@ArtnerC ArtnerC deleted the bug/databricks-serverless-pyspark branch August 23, 2024 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Incorrect behavior inside of ibis pyspark The Apache PySpark backend
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Config Value spark.sql.mapKeyDedupPolicy not Supported by Databricks SQL Warehouse
3 participants