Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cloud_storage: remote labels #20778

Merged
merged 34 commits into from
Jul 8, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
44978ef
offline_log_viewer: support remote_labels in topic properties
andrwng Jun 20, 2024
7d38dba
cluster: add remote label as topic property
andrwng Jun 12, 2024
fab93f1
archival_stm: add remote path provider to archival stm
andrwng Jun 29, 2024
cda86a5
ntp_archiver: use path provider for naming
andrwng Jun 20, 2024
160a40e
cloud_storage: plumb path provider into async manifest view
andrwng Jun 20, 2024
291ed3f
cluster: use topic_manifest_downloader in topic recovery
andrwng Jun 17, 2024
b6d1e16
cluster: use topic_manifest_downloader in list-based recovery
andrwng Jun 25, 2024
41df563
archival_stm: use partition_manifest_downloader for snapshot recovery
andrwng Jun 18, 2024
e16c247
ntp_archiver: use partition_manifest_downlaoder for read replicas
andrwng Jun 17, 2024
13a1567
cluster: use partition_manifest_downloader in partition recovery
andrwng Jun 17, 2024
fe640c9
cluster: use partition_manifest_downloader in topic recovery validation
andrwng Jun 20, 2024
9d97ff6
cluster: use partition_manifest_downloader for unsafe reset from cloud
andrwng Jun 17, 2024
8cb5db2
cloud_storage: use partition_manifest_downloader in scrubber/anomaly …
andrwng Jun 20, 2024
ebf2fa1
remote_partition: use partition_manifest_downloader for manifest fina…
andrwng Jun 17, 2024
653c9ec
cloud_storage: remove try_download_partition_manifest
andrwng Jun 17, 2024
eaa7b95
cloud_storage: remove remote::partition_manifest_exists()
andrwng Jun 20, 2024
849aef7
cloud_storage: use path provider for spillovers in manifest view
andrwng Jun 19, 2024
5248547
remote_partition: use path provider for segment paths
andrwng Jun 18, 2024
3937454
cloud_storage: add utils for lifecycle marker paths
andrwng Jun 29, 2024
d10a60d
archival: use path provider to generate lifecycle marker paths
andrwng Jun 29, 2024
5d30320
archival: use the path provider throughout the purger
andrwng Jun 18, 2024
a0be06a
archival/segment_merger: use path provider for remote runs
andrwng Jun 18, 2024
d2681ba
admin: use path provider for anomalies report
andrwng Jun 19, 2024
1066344
features: add flag for remote labels
andrwng Jul 4, 2024
2d7e19a
config: property for disabling remote labels for tests
andrwng Jun 20, 2024
e7669a9
cloud_storage: clean up manifest path generation from manifests
andrwng Jun 20, 2024
46ed453
cloud_storage: clean up remote segment name generation
andrwng Jun 29, 2024
459106c
topic_manifest: remove feature table
andrwng Jun 22, 2024
902f522
rptest/services: make get_cluster_uuid() node optional
andrwng Jun 24, 2024
245b4f2
rptest: support remote labels in BucketView
andrwng Jun 24, 2024
df36562
rptest: option to change si_settings bucket
andrwng Jul 1, 2024
45c62a4
cluster: plug cluster uuid into new topics
andrwng Jun 29, 2024
b6ae520
config: enable remote labels by default
andrwng Jul 3, 2024
263d4b3
rptest: add remote_label_test
andrwng Jul 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions src/v/cluster/topic_properties.cc
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,8 @@ std::ostream& operator<<(std::ostream& o, const topic_properties& properties) {
"mpx_virtual_cluster_id: {}, "
"write_caching: {}, "
"flush_ms: {}, "
"flush_bytes: {}}}",
"flush_bytes: {}, "
"remote_label: {}}}",
properties.compression,
properties.cleanup_policy_bitflags,
properties.compaction_strategy,
Expand Down Expand Up @@ -67,7 +68,8 @@ std::ostream& operator<<(std::ostream& o, const topic_properties& properties) {
properties.mpx_virtual_cluster_id,
properties.write_caching,
properties.flush_ms,
properties.flush_bytes);
properties.flush_bytes,
properties.remote_label);
WillemKauf marked this conversation as resolved.
Show resolved Hide resolved

return o;
}
Expand Down Expand Up @@ -101,7 +103,7 @@ bool topic_properties::has_overrides() const {
|| initial_retention_local_target_bytes.is_engaged()
|| initial_retention_local_target_ms.is_engaged()
|| write_caching.has_value() || flush_ms.has_value()
|| flush_bytes.has_value();
|| flush_bytes.has_value() || remote_label.has_value();
}

bool topic_properties::requires_remote_erase() const {
Expand Down
19 changes: 17 additions & 2 deletions src/v/cluster/topic_properties.h
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@

#pragma once

#include "cloud_storage/remote_label.h"
#include "cluster/remote_topic_properties.h"
#include "model/compression.h"
#include "model/fundamental.h"
Expand All @@ -28,7 +29,7 @@ namespace cluster {
*/
struct topic_properties
: serde::
envelope<topic_properties, serde::version<8>, serde::compat_version<0>> {
envelope<topic_properties, serde::version<9>, serde::compat_version<0>> {
topic_properties() noexcept = default;
topic_properties(
std::optional<model::compression> compression,
Expand Down Expand Up @@ -152,6 +153,19 @@ struct topic_properties
std::optional<std::chrono::milliseconds> flush_ms;
std::optional<size_t> flush_bytes;

// Label to be used when generating paths of remote objects (manifests,
// segments, etc) of this topic.
//
// The topic's data is associated with exactly one label: as a topic is
// removed and recovered across different clusters, its label will be the
// same, even though the clusters' UUIDs hosting it will be different. This
// allows recovered topics and read replica topics to download with just
// one label in mind.
//
// std::nullopt indicates this topic was created before labels were
// supported, in which case objects will use a legacy naming scheme.
std::optional<cloud_storage::remote_label> remote_label;

bool is_compacted() const;
bool has_overrides() const;
bool requires_remote_erase() const;
Expand Down Expand Up @@ -191,7 +205,8 @@ struct topic_properties
mpx_virtual_cluster_id,
write_caching,
flush_ms,
flush_bytes);
flush_bytes,
remote_label);
}

friend bool operator==(const topic_properties&, const topic_properties&)
Expand Down