-
Notifications
You must be signed in to change notification settings - Fork 234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AUDIT] test the RAPIDS plugin with the spark-connect plugin #9375
Comments
@mattahrens @sameerz |
I presume the main driver for connect will be support for an LTS of Databricks 14 https://docs.databricks.com/en/release-notes/runtime/14.0.html. 14.3 LTS was released on Feb 1 https://docs.databricks.com/en/release-notes/runtime/index.html |
What about Apache Spark 3.5.1? Since 14.3 is branching off of it. |
Upstream users have a choice, whereas it does not look like Shared Cluster users can opt out https://docs.databricks.com/en/release-notes/runtime/14.0.html#introducing-spark-connect-in-shared-cluster-architecture |
We can revisit when we work on 4.0 or Databricks 14.x shims. For 3.5.1 it is not critical. |
Related user question #10611 |
Spark connect modifies how jars are associated with the spark context / session apache/spark@caa3df4
Add tests ensuring, spark-rapids works with connect.
The text was updated successfully, but these errors were encountered: