Add support for LIMIT push down during query planning to reduce number of files scanned

[scottsand-db]

Description

This PR adds support for limit pushdown, where we will "push down" any LIMITs during query planning so that we scan the minimum number of files necessary.

How was this patch tested?

New test suite.

Does this PR introduce any user-facing changes?

No.

[tdas]

extra line

[tdas]

Please make the title better. LimitPushdown into what?

[tdas]

Lgtm

[ciurlaro42]

@scottsand-db Is OFFSET also affected by this change? It seems plausible for optimization to occur, as files without the required data could be skipped.

For example, if there were two files with 1000 rows each, and an attempt was made to read them with OFFSET 1001, the first file could be easily skipped using the metadata information.

[zsxwing]

Good question. However, using OFFSET on a Delta table is an undefined behavior. There is no ordering in a Delta table, hence using OFFSET on a Delta table may get non-deterministic results.

[ciurlaro42]

@zsxwing What about z-ordered tables? I would assume that's a fairly common optimisation

[camlee]

I'm wondering about a possible enhancement on top of this limit pushdown support. If the query specifies an ORDER BY, that isn't currently pushed down: the whole table is scanned. On either a partitioned or z-ordered table, couldn't some data skipping kick in? Using file statistics, could delta pick which partitions or files to check first and stop when enough records to satisfy the LIMIT are found? Or otherwise prune out substantial amounts of data. I'm wondering if this would make sense as a new Feature Request or if it's technically very challenging or something like that.
I think this is a common use case, for example, selecting the oldest or most recent records from a table that is partitioned on day or week or z-ordered on the timestamp column.