🐛 [WIP] Update MachineSet and MachineDeployment ownerRef versions on reconcile #7509

killianmuldoon · 2022-11-07T16:26:34Z

Signed-off-by: killianmuldoon kmuldoon@vmware.com

Currently if the apiVersion of an ownerReference doesn't match any version that is served by the Kubernetes apiServer garbage collection of resources based on ownerReferences fails. Ref: kubernetes/kubernetes#96650

When Cluster API stops serving deprecated APIVersions - i.e. v1alpha3 and v1alpha4 when they're removed - we want to ensure garbage collection continues to work as expected. This change ensures that these ownerReferences are kept up to date on each reconcile for a subset of resources.

Specifically this changes ensures updates for the apiVersion in OwnerReferences of:

Machines owned by MachineSets
MachineSets owned by MachineDeployments
MachineDeployments owned by Clusters

Fixes #7224

Signed-off-by: killianmuldoon <kmuldoon@vmware.com>

k8s-ci-robot · 2022-11-07T16:26:42Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign cecilerobertmichon for approval by writing /assign @cecilerobertmichon in a comment. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

srm09 · 2022-11-07T18:29:00Z

This is an interesting one, and I think there might be some cases of this in the CAPV codebase. The gist here is we should always update the OwnerRefs to use the latest API version served by the api server.

sbueringer · 2022-11-08T10:42:32Z

This is an interesting one, and I think there might be some cases of this in the CAPV codebase. The gist here is we should always update the OwnerRefs to use the latest API version served by the api server.

Yup exactly!

In general every controller which sets ownerReferences in a CRD is responsible to keep them up-to-date. Otherwise there's a chance garbage collection is broken when apiVersions are removed.

@killianmuldoon Would you mind surfacing this general topic in the office hours and pointing folks to the issue? Just that everyone has a chance to be aware of it.

sbueringer · 2022-11-08T11:01:42Z

internal/controllers/machinedeployment/machinedeployment_controller.go

+		if metav1.GetControllerOf(ms).APIVersion != md.APIVersion {
+			err := r.updateControllerRef(ctx, md, ms)
+			if err != nil {
+				log.Error(err, "Failed to update Machine controller reference.")


I wonder if we should return the error in this case.

In general this request should work anyway, right?

sbueringer · 2022-11-08T11:04:47Z

internal/controllers/machinedeployment/machinedeployment_controller.go

@@ -165,9 +165,9 @@ func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (_ ctrl.Re
 	return result, err
 }

-func patchMachineDeployment(ctx context.Context, patchHelper *patch.Helper, d *clusterv1.MachineDeployment, options ...patch.Option) error {
+func patchMachineDeployment(ctx context.Context, patchHelper *patch.Helper, md *clusterv1.MachineDeployment, options ...patch.Option) error {


nit: I would highly recommend to split out a general m=>md rename into a separate PR next time.

Just helps to de-obfuscate the actual changes in this PR and keeps everything nicely separated

(just resolve for this PR)

sbueringer · 2022-11-08T11:05:55Z

internal/controllers/machineset/machineset_controller.go

+		if metav1.GetControllerOf(machine).APIVersion != machineSet.APIVersion {
+			err := r.updateControllerReference(ctx, machineSet, machine)
+			if err != nil {
+				log.Error(err, "Failed to update Machine controller reference.")


I would also return the error here

sbueringer · 2022-11-08T11:07:41Z

internal/controllers/machineset/machineset_controller.go

@@ -278,6 +279,15 @@ func (r *Reconciler) reconcile(ctx context.Context, cluster *clusterv1.Cluster,
 			r.recorder.Eventf(machineSet, corev1.EventTypeNormal, "SuccessfulAdopt", "Adopted Machine %q", machine.Name)
 		}

+		// Ensure that the version of the controllerReference is up-to-date.
+		if metav1.GetControllerOf(machine).APIVersion != machineSet.APIVersion {


Are sure the machineSet.APIVersion is set? (afaik per default it's not in CR / client-go)

I think we should use the same source of truth as updateControllerReference (which is machineSetKind)

sbueringer · 2022-11-08T11:08:31Z

internal/controllers/machinedeployment/machinedeployment_controller.go

 			continue
 		}

+		// Ensure that the version of the controllerReference is up-to-date.
+		if metav1.GetControllerOf(ms).APIVersion != md.APIVersion {


Are sure the md.APIVersion is set? (afaik per default it's not in CR / client-go)

I think we should use the same source of truth as updateControllerReference (which is machineDeploymentKind)

sbueringer · 2022-11-08T11:08:52Z

internal/controllers/machineset/machineset_controller.go

-// adoptOrphan sets the MachineSet as a controller OwnerReference to the Machine.
-func (r *Reconciler) adoptOrphan(ctx context.Context, machineSet *clusterv1.MachineSet, machine *clusterv1.Machine) error {
+// updateControllerReference sets the MachineSet as a controller OwnerReference to the Machine.
+func (r *Reconciler) updateControllerReference(ctx context.Context, machineSet *clusterv1.MachineSet, machine *clusterv1.Machine) error {


nit: let's be consistent between MD and MS and pick either updateControllerRef or updateControllerReference

sbueringer · 2022-11-08T11:10:50Z

util/util.go

@@ -437,6 +437,20 @@ func HasOwner(refList []metav1.OwnerReference, apiVersion string, kinds []string
 	return false
 }

+// HasOwnerWithSameVersion checks if any of the references in the passed list match the given group from apiVersion and one of the given kinds.
+func HasOwnerWithSameVersion(refList []metav1.OwnerReference, apiVersion string, kinds []string) bool {


I know it's relatively trivial but let's ~ copy&paste the unit tests of HasOwner

sbueringer · 2022-11-08T11:11:20Z

Overall I really like the simplicity of the change, nice work!

k8s-ci-robot · 2022-11-18T23:35:48Z

@killianmuldoon: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sbueringer · 2022-11-24T04:56:23Z

@killianmuldoon

Q: How does this PR relate now to #7575?

Is #7575 fixing all kinds of ownerRef reconcile issues including that the version is correct and the current PR is then fixing the version issue for MS/MD?

killianmuldoon · 2022-11-24T10:22:02Z

Q: How does this PR relate now to #7575?

Should be considered entirely unrelated.

killianmuldoon

/close

k8s-ci-robot · 2023-03-08T13:15:07Z

@killianmuldoon: Closed this PR.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Update MachineSet and MachineDeployment ownerRef versions on reconcile

4528a6a

Signed-off-by: killianmuldoon <kmuldoon@vmware.com>

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Nov 7, 2022

k8s-ci-robot requested review from sbueringer and stmcginnis November 7, 2022 16:26

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 7, 2022

sbueringer reviewed Nov 8, 2022

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 18, 2022

fabriziopandini added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 11, 2023

killianmuldoon commented Mar 8, 2023

View reviewed changes

k8s-ci-robot closed this Mar 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 [WIP] Update MachineSet and MachineDeployment ownerRef versions on reconcile #7509

🐛 [WIP] Update MachineSet and MachineDeployment ownerRef versions on reconcile #7509

killianmuldoon commented Nov 7, 2022 •

edited

Loading

k8s-ci-robot commented Nov 7, 2022

srm09 commented Nov 7, 2022

sbueringer commented Nov 8, 2022

sbueringer Nov 8, 2022

sbueringer Nov 8, 2022

sbueringer Nov 8, 2022

sbueringer Nov 8, 2022

sbueringer Nov 8, 2022

sbueringer Nov 8, 2022

sbueringer Nov 8, 2022

sbueringer commented Nov 8, 2022

k8s-ci-robot commented Nov 18, 2022

sbueringer commented Nov 24, 2022

killianmuldoon commented Nov 24, 2022 •

edited

Loading

killianmuldoon left a comment

k8s-ci-robot commented Mar 8, 2023

🐛 [WIP] Update MachineSet and MachineDeployment ownerRef versions on reconcile #7509

🐛 [WIP] Update MachineSet and MachineDeployment ownerRef versions on reconcile #7509

Conversation

killianmuldoon commented Nov 7, 2022 • edited Loading

k8s-ci-robot commented Nov 7, 2022

srm09 commented Nov 7, 2022

sbueringer commented Nov 8, 2022

sbueringer Nov 8, 2022

Choose a reason for hiding this comment

sbueringer Nov 8, 2022

Choose a reason for hiding this comment

sbueringer Nov 8, 2022

Choose a reason for hiding this comment

sbueringer Nov 8, 2022

Choose a reason for hiding this comment

sbueringer Nov 8, 2022

Choose a reason for hiding this comment

sbueringer Nov 8, 2022

Choose a reason for hiding this comment

sbueringer Nov 8, 2022

Choose a reason for hiding this comment

sbueringer commented Nov 8, 2022

k8s-ci-robot commented Nov 18, 2022

sbueringer commented Nov 24, 2022

killianmuldoon commented Nov 24, 2022 • edited Loading

killianmuldoon left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Mar 8, 2023

killianmuldoon commented Nov 7, 2022 •

edited

Loading

killianmuldoon commented Nov 24, 2022 •

edited

Loading