operator: Add the option to immediately decommission ghosts #13298
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
joejulian
force-pushed
the
boo_a_ghost
branch
from
8e04088
to
757656a
Compare
joejulian
force-pushed
the
boo_a_ghost
branch
4 times, most recently
from
fabca06
to
51ff50c
Compare
joejulian
changed the title
alejandroEsc
previously approved these changes
Sep 7, 2023
joejulian
force-pushed
the
boo_a_ghost
branch
from
51ff50c
to
3c3edd8
Compare
RafalKorepta
reviewed
Sep 7, 2023
Comment on lines
+136
to
+137
| flag.BoolVar(&ghostbuster, "unsafe-decommission-failed-brokers", false, "Set to enable decommissioning a failed broker that is configured but does not exist in the StatefulSet (ghost broker). This may result in invalidating valid data") | ||
| _ = flag.CommandLine.MarkHidden("unsafe-decommission-failed-brokers") |
There was a problem hiding this comment.
Can this property be moved to soon deprecated cluster custom resource? I don't see a value to deprecate flags along with the cluster custom resource.
RafalKorepta
reviewed
Sep 7, 2023
RafalKorepta
reviewed
Sep 7, 2023
When a broker's storage is deleted, it comes back with a new broker id. There's no indication as to why the data is gone or whether it can be recovered. Since the Cluster resource is deprecated and the cloud team would just like to have a method to delete it regardless, the consensus is that we should assume it's not recoverable and decommission the old broker id immediately. This PR adds a hidden flag, `unsafe-decommission-failed-brokers` that, if set to true, will offer this behavior. This is dangerous and shouldn't be used. There are circumstances in which brokers with valid data can be decommissioned and even the potential for all the brokers to be decommissioned.
joejulian
force-pushed
the
boo_a_ghost
branch
from
3c3edd8
to
ea40014
Compare
alejandroEsc
approved these changes
Sep 7, 2023
|
/backport v23.2.x |
This was referenced Sep 8, 2023
RafalKorepta
added a commit
to redpanda-data/redpanda-operator
that referenced
this pull request
Jun 19, 2024
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
RafalKorepta
added a commit
to redpanda-data/redpanda-operator
that referenced
this pull request
Jun 21, 2024
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
RafalKorepta
added a commit
to redpanda-data/redpanda-operator
that referenced
this pull request
Jun 28, 2024
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
RafalKorepta
added a commit
to redpanda-data/redpanda-operator
that referenced
this pull request
Jul 2, 2024
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
RafalKorepta
added a commit
to redpanda-data/redpanda-operator
that referenced
this pull request
Jul 2, 2024
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a broker's storage is deleted, it comes back with a new broker id. There's no indication as to why the data is gone or whether it can be recovered. Since the Cluster resource is deprecated and the cloud team would just like to have a method to delete it regardless, the consensus is that we should assume it's not recoverable and decommission the old broker id immediately.
This PR adds a hidden flag,
unsafe-decommission-failed-brokersthat, if set to true, will offer this behavior.This is dangerous and shouldn't be used. There are circumstances in which brokers with valid data can be decommissioned and even the potential for all the brokers to be decommissioned.
Fixes #13132
Backports Required
Release Notes