-
Notifications
You must be signed in to change notification settings - Fork 234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve Databricks runtime shim detection #8587
Comments
We can also improve the reliability of the released spark-rapids jars using the nightly pipeline of the pending release. We know that semi-monthly/bi-weekly maintenance updates to Databricks Runtimes can break released spark-rapids plugin code. Ideally we want to retest the jar version that is already used by customers upon every maintenance update. However, testing is time consuming. So we do not want to retest last However, we can utilize the fact that our pending release runs nightly tests on DBR to detect whether we need to kick off released artifacts tests. We can maintain a table mapping DB buildver to last tested build hashes
Somewhere in the source code we will have a test or then the CI can compare it to the last known value for the DB shim based on the table and kick off a pipeline for released test jars automatically, then update the table. This should shorten the window of detection to a couple of a days. |
Update: The P0 part of this issue is to log details org.apache.spark.BuildInfo and potentially more details as documented for the SQL function This should be logged via Databricks shim service providers com.nvidia.spark.rapids.shims.spark3XYdb.SparkShimServiceProvider and in the CI logs |
link to #11184 |
Fixes NVIDIA#8587 - Match Version from the binaries - Log build info exposed in current_version Signed-off-by: Gera Shegalov <gera@apache.org>
We currently rely on the prefix of the version strings
spark-rapids/sql-plugin/src/main/spark332db/scala/com/nvidia/spark/rapids/shims/spark332db/SparkShimServiceProvider.scala
Lines 34 to 36 in 8b75245
whose values are documented in the spark-versions API. These versions represent wildcards for the latest patch of the major.minor such as
11.3.x
.Thus, a user of an older rapids-4-spark artifact may hit a runtime bug or worse, a silent defect, instead of a clear actionable message as implemented in #8521
Spark UI on DBR displays "Build Properties" in the Environment Tab:
These values correspond to
in a Scala notebook
which we can be utilized in the patch version detection.
The text was updated successfully, but these errors were encountered: