-
Notifications
You must be signed in to change notification settings - Fork 228
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Integration tests run with default configs that don't match RapidsConf defaults #11134
Comments
Related code in |
So, enforcing the default configs were added in #4917 for integration tests. This was to address #4893. I see how irritating one would feel if some cluster setting sneaks into the integration tests. However, the problem I see with the current approach is that we have two places where we keep default configs, one in scala and another in python. As long as we have them separate, this issue of inconsistent defaults will keep coming back to us whenever we change the default of some config only in one place but not in another. I'd like to fix this issue once and for all. Two approaches in my mind. Both of them will use the scala code as the source of truth.
Both look feasible and not hard to me. I wonder though if I'm missing something and they are actually not possible or hard. |
+1 for the single source of truth RapidsConf generating the default conf dictionary for PyTests. It could be a generated file, just like we do for the doc to be loaded to init |
That would work too. Alright, I will try a couple of things and see what's best. |
Reopening as we have reverted the fix in #11347. |
@pmattione-nvidia discovered that integration_tests/src/main/python/spark_session.py has default configurations that will set spark.rapids.sql.castFloatToDecimal.enabled to false despite the config being defaulted to true in RapidsConf. That means we're not really testing what we're shipping for most tests if they happen to involve operations that would require a cast of a float to a decimal.
Similarly there are many other cast-related configs being setup for integration test default that don't match what the defaults are in RapidsConf. If we need to disable these operations because too many tests fail when they're enabled, that implies we should reconsider whether these configs should be enabled by default.
The text was updated successfully, but these errors were encountered: