feat(storage): distributed execution of recluster

[zhyass]

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

enable_distributed_recluster: Enable distributed execution of table recluster, default is 0.

> set global enable_distributed_recluster=1;

SET
  GLOBAL enable_distributed_recluster = 1

0 row read in 0.601 sec. Processed 0 row, 0 B (0 row/s, 0 B/s)

> alter table test_order recluster final;

ALTER TABLE
  test_order recluster final

202563631 rows written in 431.972 sec. Processed 202.56 million rows, 115.82 GiB (468.93 thousand rows/s, 274.56 MiB/s)

>select * from clustering_information('zzq','test_order');

SELECT
  *
FROM
  clustering_information('zzq', 'test_order')

┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│    cluster_key    │ total_block_count │ constant_block_count │ unclustered_block_count │ average_overlaps │ average_depth │   block_depth_histogram  │
│       String      │       UInt64      │        UInt64        │          UInt64         │      Float64     │    Float64    │          Variant         │
├───────────────────┼───────────────────┼──────────────────────┼─────────────────────────┼──────────────────┼───────────────┼──────────────────────────┤
│ (id, insert_time) │               529 │                   91 │                       0 │          15.8261 │       16.8261 │ {"00001":437,"00128":92} │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
1 row read in 1.334 sec. Processed 1 row, 448 B (0.75 row/s, 335 B/s)

> set global enable_distributed_recluster=0;

SET
  GLOBAL enable_distributed_recluster = 0

0 row read in 0.601 sec. Processed 0 row, 0 B (0 row/s, 0 B/s)

> alter table test_order recluster final;

ALTER TABLE
  test_order recluster final

222679181 rows written in 939.860 sec. Processed 222.68 million rows, 129.19 GiB (236.93 thousand rows/s, 140.76 MiB/s)

> select * from clustering_information('zzq','test_order');

SELECT
  *
FROM
  clustering_information('zzq', 'test_order')

┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│    cluster_key    │ total_block_count │ constant_block_count │ unclustered_block_count │ average_overlaps │ average_depth │   block_depth_histogram  │
│       String      │       UInt64      │        UInt64        │          UInt64         │      Float64     │    Float64    │          Variant         │
├───────────────────┼───────────────────┼──────────────────────┼─────────────────────────┼──────────────────┼───────────────┼──────────────────────────┤
│ (id, insert_time) │               543 │                   80 │                       0 │          11.9337 │       12.9337 │ {"00001":462,"00128":81} │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
1 row read in 1.886 sec. Processed 1 row, 448 B (0.03 row/s, 12 B/s)

• Closes #issue

---

This change is 

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
databend	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 24, 2023 3:26am

[github-actions]

Docker Image for PR

• tag: pr-13048-01d54a0

note: this image tag is only available for internal use,
please check the internal doc for more details.