You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
In RapidsConf.scala, there are some ConfigBuiler object. For example,
valMULTITHREAD_READ_NUM_THREADS= conf("spark.rapids.sql.multiThreadedRead.numThreads")
.doc("The maximum number of threads on each executor to use for reading small "+"files in parallel. This can not be changed at runtime after the executor has "+
...
".")
.integerConf
.checkValue(v => v >0, "The thread count must be greater than zero.")
.createWithDefault(0)
Here checkValue does not work. createWithDefault(-1) or createWithDefault(0) will not throw any exception.
Even if "spark.rapids.sql.multiThreadedRead.numThreads" = -1 is set explicitly by user, there is no exception, either.
Steps/Code to reproduce bug
Run spark-rapids locally via:
Expected behavior
When spark.rapids.sql.multiThreadedRead.numThreads <= 0, we should throw an IllegalArgument exception.
In original spark, if we set an invalid parameter, there is an IllegalArgument.
$ spark-shell --conf spark.sql.shuffle.partitions=-1
scala> spark.conf.get("spark.sql.shuffle.partitions")
java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1178)
at org.apache.spark.sql.SparkSession.$anonfun$sessionState$2(SparkSession.scala:162)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:160)
at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:157)
at org.apache.spark.sql.SparkSession.conf$lzycompute(SparkSession.scala:185)
at org.apache.spark.sql.SparkSession.conf(SparkSession.scala:185)
... 47 elided
Caused by: java.lang.IllegalArgumentException: '-1'in spark.sql.shuffle.partitions is invalid. The value of spark.sql.shuffle.partitions must be positive
...
The text was updated successfully, but these errors were encountered:
Describe the bug
In
RapidsConf.scala
, there are someConfigBuiler
object. For example,Here
checkValue
does not work.createWithDefault(-1)
orcreateWithDefault(0)
will not throw any exception.Even if
"spark.rapids.sql.multiThreadedRead.numThreads" = -1
is set explicitly by user, there is no exception, either.Steps/Code to reproduce bug
Run spark-rapids locally via:
Expected behavior
When spark.rapids.sql.multiThreadedRead.numThreads <= 0, we should throw an
IllegalArgument
exception.In original spark, if we set an invalid parameter, there is an
IllegalArgument
.The text was updated successfully, but these errors were encountered: