-
Notifications
You must be signed in to change notification settings - Fork 98
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Decommission via Helm #253
Comments
Step 5 changes a specific parameter, but it doesn't seem to matter which parameter is changed in the upgrade as long as it requires a node restart. |
|
I see on redpanda-2:
indicating the seeds talked to a node at redpanda-0 at 14:37. OTOH, redpanda-0's first logs start later than that:
...at which point it loaded a different UUID from disk than the one seen by redpanda-2:
|
The underlying issue here was that in this example, we were using One thought was to add a decommission step in the preStop phase, but that won't work for 3-node clusters, since they will be unable to complete the decommission unless another node is added prior, given a replication factor of 3. So it seems the path forward is that, after starting up, a node should check whether there are duplicates of its hostname in the cluster with other node IDs assigned to it. If so, we should decommission the other, presumably down, node ID. |
Just to associate this log message to this issue, the following message appears constantly after a node is replaced but not decommissioned:
The following commands shows that the node count went from 3 to 4:
I think there is a time frame after which an unresponsive node gets decommissioned, but node 0 was not automatically decommissioned after 30 minutes. The cluster continued to function normally but in unhealthy state. The following rpk command to decommission the node fails:
But this admin API request from the controller broker resolved the issue:
|
I've also seen the cluster come back up after |
Redpanda moves data away from an unavailable node, but it won't officially decommission (kick the node out of the group). The mismatch between the admin API and rpk seems strange though.
There being no leader for the controller I think is explainable by each node having a new node ID after having all its data wiped, and the Raft group effectively being twice the size it should be. That said, it's odd that you're seeing mismatched cluster sizes, and that the sizes are small. When in this state, what are the node IDs? Are they new or old IDs? |
So I think the gist of all this is that the pod roll is leaving the cluster split-brained. Somehow the pod data's being deleted (pvc deleted by something outside of helm or redpanda?). |
When using emptyDir, whenever a Pod is terminated, when the pod starts up its storage is empty so it's assigned a new broker id. The old id is never decommissioned. The fix is adding a process to decommission the broker that is no longer in the cluster. |
@joejulian could this be done by a preStop script addition? Or is something that needs to happen at some other stage? |
There is some work done in old operator. |
@joejulian this says a core estimate is 'pending' but if this is a noop for core (I think it is), can you change sizing-core to 'N/A' please |
This issue says it requires docs, but from the discussion, it looks like this will be an automated process. What needs adding to the docs? |
@alejandroEsc @joejulian Correct me if I'm wrong, but I think docs should describe 2 modes of operations:
It is maybe irrelevant what automated decommission is actually doing behind the scenes, but at least describe options how to enable such functionality. |
💥 |
@mattschumpert this is only partially completed. The changes on redpanda still need to be reviewed and merged, I'll fix this |
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
Calling decommission in the case of changing Pod annotation might be not possible if Pod was removed along with its annotation where previous Redpanda ID was stored. There is dedicated function to handle Ghost brokers. Reference redpanda-data/redpanda#9750 redpanda-data/redpanda#13298 redpanda-data/redpanda#13132 redpanda-data/helm-charts#253 redpanda-data/redpanda#12847
Add an explicit decommission action to our Helm chart
Note: Decommission in Redpanda can fail if not all partitions can be reassigned to other brokers (and maybe be rejected by other future guardrails), so the command must fail gracefully with an error if Redpanda doesn't allow the decommission to succeed. Polling the Decommission status Admin API may be helpful here.
Historical context:
The text was updated successfully, but these errors were encountered: