-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
KubeadmControlPlane should adopt existing controlplane Machines #2214
Comments
Also needs to make sure the bootstrap ref is to a KubeadmConfig. |
/cc @chuckha who definitely has ideas on this one |
We should be able to automagically fill in the required kubeadm config info (at least the join bits, init/clusterconfig bits may be missing if the cluster was upgraded) using one of the referenced bootstrap configs, and even be able to automagically create the required infrastructure template using one of the referenced infra configs. That said, anything we are likely to do here could potentially conflict with the validation we have on incoming KubeadmControlPlane resources. I think the main thing we have to define is what do we expect the migration (from non-KCP to KCP) to look like during the adoption process. I think we have a few different choices here:
|
Does the KCP today rely on some kind of selector that could be exploited for the purpose of option 3? I think it would be a viable option to mark existing controlplane Machines in a way that tells the KCP controller to adopt. Curious about
Wouldn't this imply that a user wants a controlplane that is partially managed by a KCP? Is that a supportable pattern? |
I'm thinking more along the lines of opting not to use KubeadmControlPlane initially (at some point, I expect we'll make control plane required, but we specifically stated that it would not be required for v1alpha3), So I would expect this to be something like a command line flag on the controller manager (or whatever implements the automated kcp creation/adoption) that would disable the creation/adoption workflow. If they choose to create a new cluster that uses KCP, it should still work, but any existing clusters would remain as they were. |
Yes, currently we use the same labels that we previously did manually: cluster-api/controlplane/kubeadm/controllers/kubeadm_control_plane_controller.go Lines 488 to 493 in 065eb53
Once upgrade handling is in place (prior to v1alpha3 release), this will also include another label that will act as a configuration "hash" as well. This is something that we could add during the adoption process, though. |
I'm digging option two:
To be more specific about some potential gotchas:
|
Anything we do here could potentially cause issues for pivoting workflows |
Why's that? Naturally, I'm assuming all clusters are pivoted from my perspective. 🍕 |
Assuming resources are pivoted, the resources are created on the remote cluster without ordering guarantees. If we try to enforce this as part of a defaulting webhook, then the KCP resource creation might fail. If trying to handle it in the controller, then we'd have to provide asynchronous feedback to the user that the adoption process failed and would need to differentiate the adoption workflow from the reconciliation workflow for populated resources. We could potentially make the constraint that we do not accept populated KCP resources that have the label that signals the adoption workflow set. That would allow us to validate that KCP resources without it have the fields we expect and would allow us to block populated KCP resources that have it set relatively easily in the admission webhooks. |
Ah, yep, understood.
This makes sense to me. I think this label may have to be unset once ownership is granted to protect future "move" workflows? Or, as part of a move, it may need to be added... 🤔 I need to diagram... |
@detiber what are the potential failure modes of purely relying on the label selector and pushing adoption logic to the top of the reconcile loop? Looks like today we're only looking at ownerRefs: cluster-api/controlplane/kubeadm/controllers/kubeadm_control_plane_controller.go Lines 180 to 186 in 7cdea78
What if, after checking ownerRefs, we just checked for controlplane machines that match the label selector before checking replica count and adopt those that match? Assorted random thoughts:
|
I think my biggest concern is that I would expect the adoption process to provide some assurances and halt reconciliation if it hits a scenario it's not expecting. The worst case scenario in my mind is that we attempt adoption/reconciliation against a working control plane causing that control plane to get into a bad state (loss of quorum or deletion) causing cluster downtime. |
Two potential paths may be:
as for the replicas, i was thinking the controller would aggregate all possible children (ownerRefs OR labels) and then determine whether a scale-up or scale-down is needed. |
/assign Working on some kind of visual/proposal based on the above discussion. |
/assign @sethp-nr |
We've got a proof of concept working for machine adoption over here: newrelic-forks@7931678 Currently our thinking is for the controller to compute the hash of a Machine as it's adopted, and then let the rest of the reconciliation happen as appropriate. That has the effect of leaving the control plane in its current state if the KCP would produce the same config, or using the existing update mechanisms to gradually converge the existing control plane to the KCP's spec. My feeling is that I shouldn't open the PR until after the upgrade work from #2334 is in – all the interesting test cases depend on that work being done first, I think, and there's likely to be a bunch of conflicts to resolve once that settles. |
Nice, this is great work!
+1, with the upgrade PR this definitely seems like a simple solution. Would require some docs since it could be a somewhat disruptive operation. This did get me thinking, if our KCP defines a version > 1 minor version ahead of what version the machines are using, this could get weird since k8s requires upgrading through each minor version. |
We can always add something in the validation webhook to validate the upgrade version limitations. Will look through in more depth when I can access from a computer and not just my phone |
Oh, yeah, a validating webhook / adoption-time check to ensure the version changes are reasonable is a super good call. The other thing our steel thread doesn't do yet is make sure that the Machines it's claiming ownership over are actually Kubeadm bootstrapped. I'll address those two issues once I've got a few test runs under my belt – I'd like to see it work once :) |
/milestone v0.3.x |
User Story
As a user I would like to be able to migrate from a v1alpha2 cluster to a v1alpha3 cluster that utilizes the KubeadmControlPlane object. To facilitate this, the KubeadmControlPlane object should be able to adopt existing v1alpha3 controlplane machines. To simplify implementation for now, this should only occur when:
/kind feature
The text was updated successfully, but these errors were encountered: