Support bucket table for Iceberg

[jerryshao]

Current Iceberg doesn't support "bucket" semantics both in read and write, so we cannot leverage this to do bucketed join. We should add such support in Iceberg.

[aokolnychyi]

This would be a great feature to have. I think Spark might have to be adapted as well.

[jerryshao]

@rdblue @aokolnychyi I wrote a simple design doc about bucketing support in Iceberg, would you please help to review, appreciate your time.

https://docs.google.com/document/d/1X3tpcJFz8Fd9m2SixHP4psBFHc39Y21TXev4i26ve-I/edit?usp=sharing

[rdblue]

@jerryshao, thanks for posting this! I'll take a look as soon as I can, but I'm going to be at a conference next week so it may not be quick.

[jerryshao]

Sure, no problem, take your time :).

[jerryshao]

I'm roughly dividing this issue into 3 ongoing PRs:

1. Add Bucket Spec support in Iceberg API and metadata.
2. Support bucketing write in Iceberg Spark data source.
3. Support bucketing read in Iceberg Spark data source.

Currently I'm working on the first task.

[rdblue]

Thanks @jerryshao! I had a look at the doc and made some comments.

The main thing is that Iceberg already supports bucketing and has solved many of the challenges you identified, like schema evolution. There are two remaining problems:

1. Writing requires users to cluster data into buckets using a UDF.
2. Bucketed joins can't take advantage of Iceberg bucket values.

For problem 1, we need to allow Iceberg to control the requiredChildDistribution and requiredChildOrdering returned by WriteToDataSourceV2Exec. Here's a gist that shows what we use to automatically insert distribution/ordering requirements that allow automatic bucketing writes. That also depends on #317.

We also need a FunctionCatalog that allows us to return Iceberg transforms as UDFs that Spark can use.

For problem 2, we are planning to add support for Spark to be able to use bucket values to speed up joins. We aren't quite sure how to do this yet, but we know that Spark will need to recognize that a table is bucketed (using the Table's partitioning), get the bucket function from the table's catalog (using FunctionCatalog) and use that function to prepare data for the other side of the join. If the other side of the join uses the same partition function, then we can avoid a shuffle for that side of the join as well.

Hopefully this short write-up and the comments I left on the doc give you an idea of the current status of bucketed joins. Thanks for working on this!

[jerryshao]

Thanks a lot @rdblue and sorry for late response. I see from your point that you want to reuse bucket transform, and the key thing is that current Spark doesn't aware of child distribution regarding to #274 , let me revamp the whole design. Thanks again for your comments.

[jerryshao]

Also for #274, IIUC most of the partition transformations requires data sorted by this transformation functions, otherwise data cannot be grouped together, and will throw exceptions in partitioned writer, am I right? @rdblue

[rdblue]

@jerryshao, yes that's correct.

That's why we need to expose the transformation functions to Spark via FunctionCatalog, and add the ability for DSv2 sources to set distribution and ordering requirements with those functions.

[aokolnychyi]

@dbtsai, FYI

[aokolnychyi]

I am preparing a few optimizations for metadata compaction and will work on sort spec next.

[yupbank]

is there any progress on this ?

[rdblue]

@yupbank, @sunchao has a design doc for this that he's planning on sharing with the Spark community soon. I think that there is also some work going on to enable bucketed joins in Trino.

[sunchao]

Hi @yupbank , yes like @rdblue said I'm in the process of wrapping up a design doc for this feature. Will cross link it here soon!

[yupbank]

Nice! let me know if you need extra eyes, would love to help, as we run into the issue of shuffling big records recently

[sunchao]

@yupbank sure, here is the design doc. It'd be great to get more comments & feedback on it!

[SinghAsDev]

@sunchao @rdblue is this still being worked upon?

[sunchao]

Yes I'm working on this right now, and the bulk of the work is on Spark side. Please track https://issues.apache.org/jira/browse/SPARK-37375 for progress.

BTW good to see you here @SinghAsDev !

[SinghAsDev]

Great to e-see you too as well Chao!

On Thu, Dec 9, 2021 at 2:43 PM Chao Sun ***@***.***> wrote: Yes I'm working on this right now, and the bulk of the work is on Spark side. Please track https://issues.apache.org/jira/browse/SPARK-37375 for progress. BTW good to see you here @SinghAsDev <https://github.com/SinghAsDev> ! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#430 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABFQCZJLT6TG7KVST6AL5D3UQEWJ3ANCNFSM4ISKXHYQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- - Ashish

[Hoeze]

Any updates on this? Storage Partitioned Join landed in Spark v3.3.

[sunchao]

I think this is ongoing - first Iceberg needs to support function catalog which is tracked (partially) by #5305.

[kbendick]

I have truncate in #5305 and am working on bucket as well for once that is reviewed.

I will open an issue for the FunctionCatalog and link it to this. 👍

[Hoeze]

Since these were merged, is this working now in 0.14.1?

[sunchao]

There are at least the following work need to be done:

• Core: Add a util method to combine tasks by partition #2276 (allow Iceberg to combine input splits based on partition boundaries)
• implement SupportsReportPartitioning and HasPartitionKey from Spark side

Will update here once the feature is fully available.

[aokolnychyi]

I am excited to announce that support for storage-partitioned joins has been merged into master.
It will be shipped in 1.2.0. Thanks everyone involved, especially @sunchao. I am going to resolve this issue.
See PR #6371.