Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rpk: Add Support for Topic Mounting and Unmounting #23575

Merged
merged 1 commit into from
Oct 16, 2024
Merged

rpk: Add Support for Topic Mounting and Unmounting #23575

merged 1 commit into from
Oct 16, 2024

Conversation

gene-redpanda
Copy link
Contributor

@gene-redpanda gene-redpanda commented Sep 30, 2024 •
edited
Loading

Adds support for mounting and unmounting a topic to and from tiered storage with RPK. Also adds list, status and cancel commands.

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v24.2.x
  • v24.1.x
  • v23.3.x

Release Notes

Features

  • adds rpk cluster storage topic mount, unmount, list-mount, status-mount, cancel-mount

@gene-redpanda gene-redpanda marked this pull request as ready for review October 11, 2024 02:45
@gene-redpanda gene-redpanda force-pushed the add-mount-unmount branch 2 times, most recently from 3f33a71 to a3b1800 Compare October 11, 2024 14:52
r-vasquez
r-vasquez reviewed Oct 11, 2024
View reviewed changes
src/go/rpk/pkg/cli/cluster/storage/cancel-mount.go Outdated Show resolved Hide resolved
src/go/rpk/pkg/cli/cluster/storage/cancel-mount.go Outdated Show resolved Hide resolved
src/go/rpk/pkg/cli/cluster/storage/cancel-mount.go Outdated Show resolved Hide resolved
src/go/rpk/pkg/cli/cluster/storage/list-mount.go Outdated Show resolved Hide resolved
src/go/rpk/pkg/cli/cluster/storage/list-mount.go Show resolved Hide resolved
src/go/rpk/pkg/cli/cluster/storage/cancel-mount.go Outdated Show resolved Hide resolved
src/go/rpk/pkg/cli/cluster/storage/list-mount.go Outdated Show resolved Hide resolved
src/go/rpk/pkg/cli/cluster/storage/list-mount.go Outdated Show resolved Hide resolved
src/go/rpk/pkg/cli/cluster/storage/list-mount.go Outdated Show resolved Hide resolved
src/go/rpk/pkg/cli/cluster/storage/list-mount.go Outdated Show resolved Hide resolved
@dotnwat dotnwat changed the title Add Support for Topic Mounting and Unmounting rpk: Add Support for Topic Mounting and Unmounting Oct 11, 2024
@gene-redpanda gene-redpanda force-pushed the add-mount-unmount branch 4 times, most recently from 6e0879a to 5afc378 Compare October 11, 2024 21:16
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Oct 12, 2024 •
edited
Loading

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56392#01927db5-11d8-448d-b8f2-48b136c29ca8
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56392#01927db5-11da-4538-90ab-18e2d2f65897
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56392#01927db5-11e0-4e82-ad57-d61a1c0aba47
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56392#01927db5-11dd-471f-b022-9606ca71c21f
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56493#01928d0c-8dad-48b8-87c2-40805fb726e0
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56493#01928d0c-8db5-4044-925a-a3e3a4f1ce57
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/56576#0192929e-d4a4-4a03-8eb7-3f7d92ca260d

@vbotbuildovich
Copy link
Collaborator

Retry command for Build#56392

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/cloud_storage_timing_stress_test.py::CloudStorageTimingStressTest.test_cloud_storage@{"cleanup_policy":"delete"}

r-vasquez
r-vasquez reviewed Oct 12, 2024
View reviewed changes
src/go/rpk/pkg/cli/cluster/storage/cancel-mount.go Outdated Show resolved Hide resolved
src/go/rpk/pkg/cli/cluster/storage/cancel-mount.go Outdated Show resolved Hide resolved
src/go/rpk/pkg/cli/cluster/storage/mount.go Show resolved Hide resolved
src/go/rpk/pkg/cli/cluster/storage/mount.go Outdated Show resolved Hide resolved
src/go/rpk/pkg/cli/cluster/storage/mount.go Outdated Show resolved Hide resolved
src/go/rpk/pkg/cli/cluster/storage/status-mount.go Outdated Show resolved Hide resolved
src/go/rpk/pkg/cli/cluster/storage/unmount.go Outdated
Comment on lines 37 to 40
The unmount process ensures data safety and cluster integrity. Every
acknowledged message before the initiation of unmount is guaranteed to be
persisted in Tiered Storage, subject to Redpanda's existing data safety
guarantees.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mattschumpert Do we want this in the rpk docs or would we have it elsewhere? It is important, but I wonder if rpk is the right place for this information.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed it

src/go/rpk/pkg/cli/cluster/storage/unmount.go Outdated Show resolved Hide resolved
src/go/rpk/pkg/cli/cluster/storage/unmount.go Outdated Show resolved Hide resolved
src/go/rpk/pkg/cli/cluster/storage/unmount.go Outdated Show resolved Hide resolved
kbatuigas
kbatuigas reviewed Oct 13, 2024
View reviewed changes
src/go/rpk/pkg/cli/cluster/storage/mount.go

Mount topic my-topic from Tiered Storage to the cluster in the my-namespace
with my-new-topic as the new topic name
rpk cluster storage mount my-namespace/my-topic --to my-namespace/my-new-topic
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are other namespaces supported? It appears that at least the Admin API call only supports kafka

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well there's support for namespace in the spec for migration/mount and we auto-define the kafka namespace if they don't provide one

dotnwat
dotnwat reviewed Oct 14, 2024
View reviewed changes
Copy link
Member

@dotnwat dotnwat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR needs to have bazel run //:gazelle run

@gene-redpanda gene-redpanda force-pushed the add-mount-unmount branch 2 times, most recently from 640545d to 792effe Compare October 14, 2024 20:49
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Oct 14, 2024 •
edited
Loading

Retry command for Build#56493

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/cloud_storage_timing_stress_test.py::CloudStorageTimingStressTest.test_cloud_storage@{"cleanup_policy":"delete"}
tests/rptest/transactions/transactions_test.py::TxUpgradeRevertTest.test_snapshot_compatibility

@vbotbuildovich
Copy link
Collaborator

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/56493#01928d0c-8db0-4949-880a-aba194a4eb7f:

"rptest.transactions.transactions_test.TxUpgradeRevertTest.test_snapshot_compatibility"

@gene-redpanda gene-redpanda force-pushed the add-mount-unmount branch 6 times, most recently from 0b3db92 to 0067cd9 Compare October 15, 2024 22:36
r-vasquez
r-vasquez approved these changes Oct 15, 2024
View reviewed changes
Copy link
Contributor

@r-vasquez r-vasquez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@gene-redpanda
feat: add support for mount management
75c20ea 
Adds mount/unmount support
@gene-redpanda gene-redpanda merged commit b91e736 into redpanda-data:dev Oct 16, 2024
23 checks passed
@gene-redpanda gene-redpanda deleted the add-mount-unmount branch October 16, 2024 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kbatuigas kbatuigas kbatuigas left review comments

@dotnwat dotnwat dotnwat left review comments

@r-vasquez r-vasquez r-vasquez approved these changes

@twmb twmb Awaiting requested review from twmb twmb is a code owner

@Deflaimun Deflaimun Awaiting requested review from Deflaimun Deflaimun is a code owner

Assignees
No one assigned
Labels
area/build area/rpk
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@gene-redpanda @vbotbuildovich @dotnwat @kbatuigas @r-vasquez