-
Notifications
You must be signed in to change notification settings - Fork 573
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
operator: Add the option to immediately decommission ghosts #13298
Conversation
8e04088
to
757656a
Compare
fabca06
to
51ff50c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few questions and one suggestion, nothing blocking, leaving it up to you.
51ff50c
to
3c3edd8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm, thanks!
flag.BoolVar(&ghostbuster, "unsafe-decommission-failed-brokers", false, "Set to enable decommissioning a failed broker that is configured but does not exist in the StatefulSet (ghost broker). This may result in invalidating valid data") | ||
_ = flag.CommandLine.MarkHidden("unsafe-decommission-failed-brokers") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can this property be moved to soon deprecated cluster custom resource? I don't see a value to deprecate flags along with the cluster custom resource.
When a broker's storage is deleted, it comes back with a new broker id. There's no indication as to why the data is gone or whether it can be recovered. Since the Cluster resource is deprecated and the cloud team would just like to have a method to delete it regardless, the consensus is that we should assume it's not recoverable and decommission the old broker id immediately. This PR adds a hidden flag, `unsafe-decommission-failed-brokers` that, if set to true, will offer this behavior. This is dangerous and shouldn't be used. There are circumstances in which brokers with valid data can be decommissioned and even the potential for all the brokers to be decommissioned.
3c3edd8
to
ea40014
Compare
/backport v23.2.x |
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
When a broker's storage is deleted, it comes back with a new broker id. There's no indication as to why the data is gone or whether it can be recovered. Since the Cluster resource is deprecated and the cloud team would just like to have a method to delete it regardless, the consensus is that we should assume it's not recoverable and decommission the old broker id immediately.
This PR adds a hidden flag,
unsafe-decommission-failed-brokers
that, if set to true, will offer this behavior.This is dangerous and shouldn't be used. There are circumstances in which brokers with valid data can be decommissioned and even the potential for all the brokers to be decommissioned.
Fixes #13132
Backports Required
Release Notes