-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Re-test restore of CAPI clusters with upstream fix to allow machine adoption #2554
Comments
@wjun Thanks for reporting this. More specifically the issue seems to be in the kubeadm controlplane controller that fails to identify machines owned by it as the UID in the owner ref is different. |
@ashish-amarnath I checked the owner references after restore, and noticed the existing control plane node object's owner reference is set to the current workload Cluster object's uid, which is incorrect, because its owner should be the kubeadmcontrolplane, while all other capi objects' owner references have been upated to the new uids correctly. Do you know if Velero updated the owner references during restore or CAPI did that? |
@wjun |
@ashish-amarnath My question is which component should care about the ownerference update during restore as the owners' UIDs have changed. This looks a generic question that Velero may consider to process. |
kubernetes-sigs/cluster-api#2489 should allow for control plane adoption. My understanding is that once it's in a CAPI release, Velero will be in a better position to restore CAPI objects so that they will be adopted. |
Sounds like the action item for now is simply a re-test to determine if the CAPI change fixes this or not. Are we able to re-test now that the fix has merged, or do we have to wait for a release here? |
Moved this into the 1.5 milestone for retest. |
I can re-test with the fix. It's on my list. If there is no release with the fix by the time I get to it, I can build my own images to test it out. |
I have re-tested with the cluster api v0.3.7, and the issue has been fixed. |
What steps did you take and what happened:
I have 1 capv management cluster created by tkg 1.0 and create a workload cluster from this management cluster. I use Velero to backup only the namespace these workload cluster related objects belong to. When this tkg management cluster crashes, I create a new management cluster, and restore the workload cluster objects to this new cluster with the backup, so I can continue to manage the existing workload clusters(scale nodes, etc).
The major functions work, but I noticed when restore starts, a new control plane node creation for the workload cluster is triggered, which is unnecessary, since my workload cluster is already there.
What did you expect to happen:
I expect only those workload cluster objects are restored, and there is no any change to the targeting workload cluster during restore.
Anything else you would like to add:
I guess this is caused by workload cluster object continue to reconcile when it is restored but other objects are not restored yet. CAPI v1apha3 enhanced cluster reconcile with "Cluster.Spec.Paused" field (https://cluster-api.sigs.k8s.io/developer/providers/v1alpha2-to-v1alpha3.html#support-the-clusterx-k8siopaused-annotation-and-clusterspecpaused-field) so if we set this field to 'true' , it pauses reconciliation on the cluster and all associated objects. by this way, during restore, cluster will not reconcile new objects. Once restore completes, we can reset the field to 'false'.
Environment:
velero version
): 1.3.2kubectl version
): 1.17.3/etc/os-release
): photon-3The text was updated successfully, but these errors were encountered: