You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Caused by: java.lang.UnsupportedOperationException: Use getFilePartitions() instead.
at com.databricks.sql.transaction.tahoe.files.TahoeFileIndexWithStaticPartitions.inputFiles(TahoeFileIndexWithStaticPartitions.scala:98)
at org.apache.spark.sql.execution.datasources.HadoopFsRelation.inputFiles(HadoopFsRelation.scala:77)
at com.nvidia.spark.rapids.GpuOverrides.$anonfun$isDeltaLakeMetadataQuery$1(GpuOverrides.scala:4641)
at com.nvidia.spark.rapids.GpuOverrides.$anonfun$isDeltaLakeMetadataQuery$1$adapted(GpuOverrides.scala:4628)
Steps/Code to reproduce bug
import spark.implicits._
// try disabling log query for delta knob, fails either way
spark.conf.set("spark.rapids.sql.detectDeltaLogQueries", "false")
val df0 = sc.parallelize(0 until 10).toDF
val df1 = sc.parallelize(0 until 10).toDF
val df9 = df0.unionAll(df1)
val df00 = sc.parallelize(0 until 100).toDF
val df011 = sc.parallelize(0 until 100).toDF
val df099 = df00.unionAll(df011)
val fin = df9.unionAll(df099)
fin.explain()
sqlContext.sql("CREATE TABLE IF NOT EXISTS test00 (value int) USING delta")
fin.write.mode(SaveMode.Overwrite).format("delta").insertInto("test00")
import io.delta.tables._
val deltaTable = DeltaTable.forName(spark, "test00")
deltaTable.optimize().executeCompaction()
Expected behavior
Non fatal execution.
Environment details (please complete the following information)
Databricks AWS 13.3, Spark Rapids plugin version 24.06.0,
13.3 LTS ML (includes Apache Spark 3.4.1, GPU, Scala 2.12)
The text was updated successfully, but these errors were encountered:
Describe the bug
A query that involves a delta table with the
optimize
call https://docs.databricks.com/en/delta/optimize.html#syntax-examples fails with the Rapids for Spark Plugin, with:Steps/Code to reproduce bug
Expected behavior
Non fatal execution.
Environment details (please complete the following information)
The text was updated successfully, but these errors were encountered: