Previously valid queries now fail with "does not satisfy parent distribution requirements: SinglePartition"

[adamfaulkner-at]

Environment

Delta-rs version: 0.19.0 (also tested against main)

Binding: Rust

Environment: MacOS, linux

• Cloud provider: AWS
• OS: MacOS, Linux
• Other:

---

Bug

What happened:

After performing the following operations:

1. Create a table
2. Perform two writes to the table
3. Re-open the table.
4. Perform a query that filters all data, and sorts on the column used for filtering

The query fails with this error:

called `Result::unwrap()` on an `Err` value: Context("SanityCheckPlan", Plan("Child: [\"CoalesceBatchesExec: target_batch_size=8192\", \"  FilterExec: a@0 > s\", \"    DeltaScan\", \"      ParquetExec: file_groups={0 groups: []}, projection=[a], predicate=a@0 > s, pruning_predicate=CASE WHEN a_null_count@1 = a_row_count@2 THEN false ELSE a_max@0 > s END, required_guarantees=[]\"] does not satisfy parent distribution requirements: SinglePartition"))

What you expected to happen:

I expected this query to continue to work like it did prior to 0.19.0.

How to reproduce it:

I've added a new failing test in this PR: #2830

More details:

I suspect this is somehow related to predicate pushdown excluding all parquet files from the DeltaScan

[adamfaulkner-at]

Hello! I actually have a fix now:
#2830