[VL] Slow scan with Spark runtime filter when split-preloading is turned on #6074
Comments
weixiuli
pushed a commit
to weixiuli/gluten
that referenced
this issue
Jul 10, 2024
Rebase Velox 2023-12-20 ``` 5144f14 Add next_day Spark function (apache#7333) b57f14b Fix static initialization order of default memory manager (apache#8098) ef47305 Revert D51867634: Multisect successfully blamed "D51867634: [velox][PR] Scatter struct nulls when deserializing Presto wire format" for otest failure (apache#8105) eb71c70 Back out "Extract files used in ReaderTest to helper file" 7c42ddc Expose memory leak metrics (apache#8095) ce212f1 Specify units of individual counters in ExchangeSource.stats (apache#8005) f0b1ae0 Add per-driver cpu time slicing support (apache#8083) 0f32527 Move velox_aggregation_fuzzer_test and spark_aggregation_fuzzer_test to the fuzzer directory (apache#8007) 2ada2ec Add schema to parquet writer options (apache#6074) 050c16c Extract files used in ReaderTest to helper file (apache#7740) f08cb12 Optimize Presto Serialization (apache#8029) ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In some cases when runtime filter is added,
spark.gluten.sql.columnar.backend.velox.IOThreads=0(which turns Velox's preload / prefetch off) could gain better performance than the default settings (spark.gluten.sql.columnar.backend.velox.IOThreads=<task slot num>).Caused by facebookincubator/velox#10173
To temporarily bypass the issue if found, just set
spark.gluten.sql.columnar.backend.velox.IOThreads=0.The text was updated successfully, but these errors were encountered: