feat: add z-order optimize #1429
Conversation
github-actions
bot
added
binding/python
wjones127
requested review from
houqp,
xianwill,
fvaleye,
roeap,
rtyler and
mosyp
as code owners
|
took the liberty of resolving the merge conflicts I caused :) |
rust/src/operations/optimize.rs
Outdated
| read_stream: impl Future< | ||
| Output = Result< | ||
| BoxStream<'static, Result<RecordBatch, ParquetError>>, | ||
| DeltaTableError, | ||
| >, | ||
| > + Send | ||
| + 'static, |
There was a problem hiding this comment.
nit: should this be defined as a separate type?
There was a problem hiding this comment.
LMK what you think of the type I added.
rust/src/operations/optimize.rs
Outdated
| @@ -414,6 +439,56 @@ impl MergePlan { | |||
| Ok((partial_actions, partial_metrics)) | |||
| } | |||
|
|
|||
| /// Creates a stream of batches that are zordered | |||
| /// | |||
| /// Currently requires loading all the data into memory. | |||
There was a problem hiding this comment.
sounds a bit like loading all data from the table - for a partitioned table, this would be loading a single partition?
There was a problem hiding this comment.
Yeah if it's non-partitioned, it's everything. If it's partitioned then only max_concurrent_tasks number of partitions are loaded at once.
There was a problem hiding this comment.
Added a comment for this.
rust/src/operations/optimize.rs
Outdated
| .as_slice(), | ||
| ) | ||
| .unwrap(); | ||
| let indices = arrow::compute::lexsort_to_indices( |
There was a problem hiding this comment.
I may be completely off here, but we may just get away with sorting an enumerate()'ed vector of values here. IIUC, we should not have null values at this point since this is already the encoded array and they byte values can be compared using the native methods. If that is true, we could avoid the non-trivial internals of the lexsort_to_indices function.
I guess the optimizations on the CPU level mentioned in the arrow row blog posts should take effekt either way since we are comparing bytes nonetheless. So leave it to your expertise if this is worthwhile :)
There was a problem hiding this comment.
Oh I think you are right! I'll give that a shot.
There was a problem hiding this comment.
Nicely simplified :)
Description
Implements Z-order in Rust. This is a very basic version that requires loading the whole partition into memory for sorting. In the future, we can implement a DataFusion-based code path that allows sorting with spilling to disk for even larger optimize jobs.
The Z-order function here is based on Arrow's row format. We truncate to take the first 16 bytes of data (padding with zeros at end as necessary). So for variable-width columns like strings, this means that we are only using the first 15 bytes of the string (the first byte is used to differentiate null and empty strings, see row format docs). If a user has a string column where they all share the same prefix, this z-order function won't work well for them. But in many common cases it will work.
We'll also expose this in Python as a follow up.
Related Issue(s)
Documentation