-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provision 3 control plane nodes using kubeadm got errors etcd errors #8250
Comments
This issue is currently awaiting triage. If CAPI contributors determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
I think you should look into this:
Just for my understanding. Your are using your own control plane and bootstrap provider? As far as I'm aware you have to create the first control plane node and after that you can create further control plane nodes in parallel (KCP should do it ~ like that) |
That's red herring, whenever kubeadm join failed, it will have this error, I do have the same config.toml and credential setup for the first control plane nodes. Yes, I'm using own control plane and bootstrap provider. I know cluster-api guarantees it will create the first control plane node, then create the other control plane nodes in parallel, but I wonder whether this is the root cause, or should we always provision the control plane nodes serially, based on kubernetes/kubeadm#1846 (comment), we need to provision the control plane nodes serially due to etcd issues, the error match exactly what I saw. |
I'm not sure, I'm not aware that we had any reports about that issue for KCP in the last few years. So I'm very hesitant to change the KCP implementation Probably I was wrong and KCP actually joins nodes sequentially. Someone would have to take a closer look at the KCP code |
I think it works by always running prechecks before adding a machine: cluster-api/controlplane/kubeadm/internal/controllers/scale.go Lines 68 to 87 in bbe7c2f
Essentially:
|
Only the very first control plane has this lock:
Once this machine is ready, then
Preflight check will also run in the same time, they have no idea about each other, so essentially they all finish the preflightcheck successfully without knowing each other provision status. |
@fabriziopandini WDYT? |
/remove-kind bug In KCP we prefer stability over speed when dealing with the control plane machines, and thus we are joining control plane machines sequentially, one at time, and waiting for the entire control plane to be stable before creating the next machine. If you are building your own provider using kubeadm internally you can follow KCP as a reference or engage the kubeadm community to discuss making kubeadm join to work consistently (according to https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/#steps-for-the-rest-of-the-control-plane-nodes parallel join of control plane machines is supported, but if I look at kubernetes/kubeadm#1846 (comment) or #2050 (comment) it doesn't work consistently) Note: I'm closing this issue because given my understanding it is due to a kubeadm issue, not to a CAPI/KCP/CABPK problem, but feel free to continue the discussion here if you need more information about how KCP or CAPBK works. |
@fabriziopandini: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Are you saying current kubeadm controller is creating kubeadm secrets sequentially? I'm using official kubeadm controller not my own. Based on my observation, it will use the lock to create the first kubeadm secret, but the remaining control plane nodes secrets are created in parallel, not sequentially: Once
|
/reopen I would really like to clarify if KCP behaves correct today. @fabriziopandini The question is if KCP actually creates Machines sequentially (and how this is enforced) As far as I can tell the following could happen:
See also
|
@sbueringer: Reopened this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
@yongxiu I talked to Fabrizio and KCP actually creates Machines sequentially. The lock in CABPK is only for the very first control plane machine. KCP is then creating additional Machines sequentially:
The preflight checks in 3. will fail until the Machine from 2 is sucessfully up. There is no way that (1+2) and (3+4) can run concurrently as the KCP controller only has one worker go routine per KCP object. So basically controller-runtime ensures that we only ever reconcile a specific KCP object in one worker. Can you please clarify with providers you are using. I'm a bit confused as the following statements seem to contradict each other:
|
I see, thanks. This is new logic added after 0.3.6, I was using an old version. Good to know cluster-api is already provisioning control plane nodes sequentially. |
What steps did you take and what happened:
I use cluster-api with my own cluster api provider to provision the cluster, my own cluster api provider will use kubeadm to initialize the control plane nodes.
What did you expect to happen:
It should provision the cluster successfully
Anything else you would like to add:
The provision is flaky, sometimes it failed, when failed, the 2nd contorl plane node will have below errors:
According to kubernetes/kubeadm#1846 (comment), we need to provision the control plane nodes serially due to etcd issues, can anyone help confirm this is the right resolution? Then we need to change the lock logic.
Environment:
kubectl version
): v1.26.0/etc/os-release
): Ubuntu 20.04/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]
The text was updated successfully, but these errors were encountered: