-
Notifications
You must be signed in to change notification settings - Fork 24
lost master during upgrade #123
Comments
Hmm, might be a configuration error? the etcd pod is giving me some weird logs:
|
Which etcd pod is that - the new one, or one of the good ones? |
t's the one from a new node that failed to kubeadm join successfully |
Is something missing some certs somewhere? |
@chuckha if you still have it, could you please gist the full cloud-init log and the full etcd pod logs? |
@ncdc and I paired on this one for a bit but as expected everything looked fine. It seems to have been a transient issue that We wanted to tag @fabriziopandini, @timothysc and @mauilion to start a larger conversation around the robustness of Are there things we could do to improve the robustness of kubeadm? is that outside the scope of the kubeadm project? |
Also, specifically, have any of you encountered issues where the apiserver static pod for some reason tries to connect over a period of time with a bad client cert to etcd, and then it magically clears itself up? |
|
@chuckha I don't have full context on how the upgrade tool works, so I might be fully out of the track, but there is something that seems strange to me in this sequence. From what I understand from the logs, kubeadm is running an init sequence, so it generates new certificates for all components, etcd included.
Then the new etcd instance starts on
What it is not clear to me is if/how the existing and new member are expected to communicate, considering the new member has a newly generated set of PKI More generally. Is it correct to start an upgrade sequence with a plain
Happy to start this discussion. Lately, we are adding more and more retry trying to make kubeadm resilient to possible conditions that might happen in the wild, but we should always consider that those are a workaround that in some case retries might hide real problems (see e.g discussion on kubernetes/kubeadm#1793 that was fixed by kubernetes-retired/cluster-api-bootstrap-provider-kubeadm#250) Also, this might be relevant for this issue kubernetes/kubeadm#1793 |
It definitely should be doing a join. That's what it's coded to do. We might have a bug somewhere. |
After further deep dive (thanks for help!): There are no problems in the sequence and in certificates. Most probably, the problem is in a combination of:
As a temporary workaround, a possible solution is to make the upgrade tool to wait for all the etcd members to report status before moving to the next node #124 A more long term solution is to review kubeadm wait and make it wait for the status of the joining member-only (not for the status of all the members) |
One of my master nodes was unable to bring up etcd fast enough:
Here are the relevant lines in /var/log/cloud-init-output.log
This kinda sucks because it's not like we can re-run kubeadm init :/ I don't know if there's anything we can really do in this situation, at least not in this repo. Feel free to close if this is too inactionable.
The text was updated successfully, but these errors were encountered: