CAPI controllers should keep ownerRefs on the current apiVersion #7224
Comments
sbueringer
added
the
kind/feature
k8s-ci-robot
added
the
needs-triage
|
cc @killianmuldoon Just in case you're interested |
|
/assign I'll take a look at this |
|
/triage accepted |
k8s-ci-robot
added
triage/accepted
|
It seems there is no upstream solution to this problem - it's been tracked in this issue for the last couple of years: It would be great if we could get a solution that would also work in Controller Runtime, or in k/k itself to solve this problem across the community, but the first step here might just be to solve it in CAPI e.g. by upgrading references on every reconcile if required. |
|
Yup - as discussed, I agree I think we have to start with a solution for CAPI. |
|
Replicating this in CAPV as an investigation issue. 💯 |
|
/milestone v1.4 Given we are planning to implement this for v1.4 |
During the VM reconciliation, the controllers track down the parent of the Machine objects using the owner reference hierarchy. The owner reference checks were comparing the Group as well as the API version to determins and fetch the owner. Currently, the API version used for comparison was selected from the storageVersion of the CAPI objects on the API server. Due to this, any objects created via an earlier CAPI version, say v1alpha3, which sets the owner reference API version to group name + v1alpha3 were not being found since no such object would exist. A similar object with the updated API type is stored in the api server. This drify happens since CAPI does not update the owner references of the objects when moving to newer API versions. There is an issue tracking this kubernetes-sigs/cluster-api#7224 which will solve the stale owner ref API version problem. Until then we are dropping the version check and just relying on the group info and kind combination to fetch the parent object. Signed-off-by: Sagar Muchhal <muchhals@vmware.com>
During the VM reconciliation, the controllers track down the parent of the Machine objects using the owner reference hierarchy. The owner reference checks were comparing the Group as well as the API version to determins and fetch the owner. Currently, the API version used for comparison was selected from the storageVersion of the CAPI objects on the API server. Due to this, any objects created via an earlier CAPI version, say v1alpha3, which sets the owner reference API version to group name + v1alpha3 were not being found since no such object would exist. A similar object with the updated API type is stored in the api server. This drify happens since CAPI does not update the owner references of the objects when moving to newer API versions. There is an issue tracking this kubernetes-sigs/cluster-api#7224 which will solve the stale owner ref API version problem. Until then we are dropping the version check and just relying on the group info and kind combination to fetch the parent object. Signed-off-by: Sagar Muchhal <muchhals@vmware.com>
During the VM reconciliation, the controllers track down the parent of the Machine objects using the owner reference hierarchy. The owner reference checks were comparing the Group as well as the API version to determins and fetch the owner. Currently, the API version used for comparison was selected from the storageVersion of the CAPI objects on the API server. Due to this, any objects created via an earlier CAPI version, say v1alpha3, which sets the owner reference API version to group name + v1alpha3 were not being found since no such object would exist. A similar object with the updated API type is stored in the api server. This drify happens since CAPI does not update the owner references of the objects when moving to newer API versions. There is an issue tracking this kubernetes-sigs/cluster-api#7224 which will solve the stale owner ref API version problem. Until then we are dropping the version check and just relying on the group info and kind combination to fetch the parent object. Signed-off-by: Sagar Muchhal <muchhals@vmware.com>
During the VM reconciliation, the controllers track down the parent of the Machine objects using the owner reference hierarchy. The owner reference checks were comparing the Group as well as the API version to determins and fetch the owner. Currently, the API version used for comparison was selected from the storageVersion of the CAPI objects on the API server. Due to this, any objects created via an earlier CAPI version, say v1alpha3, which sets the owner reference API version to group name + v1alpha3 were not being found since no such object would exist. A similar object with the updated API type is stored in the api server. This drify happens since CAPI does not update the owner references of the objects when moving to newer API versions. There is an issue tracking this kubernetes-sigs/cluster-api#7224 which will solve the stale owner ref API version problem. Until then we are dropping the version check and just relying on the group info and kind combination to fetch the parent object. Signed-off-by: Sagar Muchhal <muchhals@vmware.com>
During the VM reconciliation, the controllers track down the parent of the Machine objects using the owner reference hierarchy. The owner reference checks were comparing the Group as well as the API version to determins and fetch the owner. Currently, the API version used for comparison was selected from the storageVersion of the CAPI objects on the API server. Due to this, any objects created via an earlier CAPI version, say v1alpha3, which sets the owner reference API version to group name + v1alpha3 were not being found since no such object would exist. A similar object with the updated API type is stored in the api server. This drify happens since CAPI does not update the owner references of the objects when moving to newer API versions. There is an issue tracking this kubernetes-sigs/cluster-api#7224 which will solve the stale owner ref API version problem. Until then we are dropping the version check and just relying on the group info and kind combination to fetch the parent object. Signed-off-by: Sagar Muchhal <muchhals@vmware.com>
|
Afaik will be fixed by #8256 |
|
/reopen This is no longer tested in CAPI e2e tests on main as we are no longer upgrading apiVersions in the clusterctl_upgrade tests. We should consider whether to try to continuously test this - e.g. by create a v1beta2 apiVersion for testing, or whether it's alright to leave this until there is a future apiVersion to upgrade to. |
|
@killianmuldoon: Reopened this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
I think I wouldn't introduce an artificial apiVersion just to test this. Seems like a lot of effort Especially as we still test the re-reconcile case and the only gap is that we don't verify that we would bump an apiVersion on an existing ref |
|
Hm or wait, stupid idea. Could we just do the same as for the re-reconcile case, but instead of deleting the ownerRefs we just set them to some non-existent v1alpha1 version? |
Not stupid at all - I think it should work perfectly. |
I took a closer look at our ownerRefs and how they evolve (or not) after ClusterAPI upgrades. In some cases we bump the apiVersions automatically in some cases we don’t. Our cluster deletion reconciliation works independent of the apiVersion, but the Kubernetes garbage collection does not.
An Example:
Now let's take a look at how MachineDeployment deletion usually works:
Let's now assume the case from above. MS has an ownerRef from MD with an apiVersion that doesn't exist anymore (e.g. v1alpha3 in the example). The MD deletion now won't work anymore:
This seems like a bug to me and a blocker for removal of v1alpha3 apiVersions. I don't know for which resources exactly we depend on the Kubernetes garbage collection but I think we should keep all ownerRefs we set on the current apiVersion.
Proposed tasks:
It would be nice to have this implemented for v1.3.0 so we can consider dropping v1alpha3/v1alpha3 in the next releases.
/kind feature
The text was updated successfully, but these errors were encountered: