-
Notifications
You must be signed in to change notification settings - Fork 753
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add table function fuse_vacuum2()
#16049
base: main
Are you sure you want to change the base?
Conversation
# Conflicts: # src/query/service/src/pipelines/builders/builder_commit.rs # src/query/service/src/pipelines/builders/builder_recluster.rs # src/query/service/tests/it/storages/fuse/conflict.rs # src/query/storages/fuse/src/operations/common/generators/mutation_generator.rs
Docker Image for PR
|
Docker Image for PR
|
Docker Image for PR
|
Docker Image for PR
|
Docker Image for PR
|
# Conflicts: # src/query/catalog/src/catalog/interface.rs # src/query/ee/Cargo.toml # src/query/service/src/catalogs/default/database_catalog.rs # src/query/service/src/catalogs/default/mutable_catalog.rs # src/query/service/src/catalogs/default/session_catalog.rs # src/query/service/tests/it/storages/fuse/operations/commit.rs # src/query/storages/fuse/src/fuse_table.rs # src/query/storages/fuse/src/io/locations.rs # tests/sqllogictests/suites/base/06_show/06_0014_show_table_functions.test
# Conflicts: # src/query/catalog/src/table_context.rs # src/query/service/src/pipelines/builders/builder_column_mutation.rs # src/query/service/src/sessions/query_ctx.rs # src/query/service/tests/it/sql/exec/get_table_bind_test.rs # src/query/settings/src/settings_getter_setter.rs
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
This PR introduces a new table function
fuse_vacuum2()
aimed at enhancing performance.Object Key of Snapshot v5, Segment, and Block
For segments/blocks that are newly created in an operation based on snapshot
s
, the timestamp embedded in their object keys equalss.timestamp
. If there is no base snapshot, letts
be0
. For snapshots, the timestamp embedded in their object keys equals to their own timestamp.New Field
lvt
in Snapshot v5For all snapshots v5 (
s
), the candidate value (lvt_candidate
) ofs.lvt
is calculated ass.timestamp - settings.get_retention_period()
.s.lvt
may be adjusted to larger values based onlvt_candidate
, in cases where the retention period has been tweaked or due to clock skews. For example, iflvt_candidate <= s.prev.lvt
, thens.lvt
could be set tos.prev.lvt + 1
.s.lvt
should respect the retention period setting at the time. Decreasing or settings.lvt
equal to thelvt
of the previous snapshot is not allowed.Properties
s.lvt <= s.timestamp
s.lvt > s.prev.lvt
s
is based on another snapshots'
:s.lvt <= s'.timestamp
(enforced during commitment: transactions with stale base snapshots are not allowed).Notes
Some(_) > None
s.prev
may not equals'
Steps of Vacuum2
Tests
Type of change
This change is