GC only if allowed to update DV owner finalizers #2416

arnongilboa · 2022-08-30T16:25:28Z

Signed-off-by: Arnon Gilboa agilboa@redhat.com

What this PR does / why we need it:
Eliminate errors like persistentvolumeclaims SOME_PVC is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers.

We have this error in OpenShift where VM and DV are created. However, the PVC is created by CDI which is not allowed to update finalizers of the VM, triggered when PVC is updated with its DV ownerRef to the VM with blockOwnerDeletion: true.

PVC gets its DV ownerRefs, which can be any CR. If BlockOwnerDeletion is true, we allow GC only if CDI has RBAC to update the owner finalizers (we explicitly added it for VirtualMachines). BlockOwnerDeletions gets validated with this admission plugin which is enabled in OpenShift, but disabled in our kubevirtci clusters.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Release note:

Garbage collect a DataVolume only if RBAC allows to update its owner finalizers

brybacki · 2022-08-31T10:30:09Z

pkg/controller/datavolume-controller.go

+		for _, pvcOwnerRef := range pvc.OwnerReferences {
+			if dvOwnerRef.UID == pvcOwnerRef.UID {
+				refExists = true
+				break
+			}
+		}


maybe extratc this as
findRef(pvc.OwnerReferences, dvOwnerRef.UID)
or
existsRef(dvOwnerRef.UID, pvc.OwnerReferences)

It's already a compact function, so extracting this loop to another func seems unnecessary.

but it will be obvious what it does without reading two levels of for loop and break statement;

for _, dvOwnerRef := range dv.OwnerReferences { if !refExists(dvOwnerRef.UID, pvc.OwnerReferences) { dvOwnerRef.BlockOwnerDeletion = nil refs = append(refs, dvOwnerRef) } }

btw: this is square complexity, but I suppose we should never see more that a few references so it is ok.

Right, in the OpenShift case it's actually 1x1 😂
Anyway, I'll extract it as I really want to make you happy today, bro.

pkg/controller/datavolume-controller.go

brybacki · 2022-08-31T10:31:30Z

You say "Eliminate errors like persistentvolumeclaims SOME_PVC is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers".

But I do not understand what does it mean. Why do we have such error?

brybacki · 2022-08-31T10:35:42Z

Ok, so we are trying to copy ownerReferences from DV to PVC, before deleting DV. And we cannot copy blockOwnerDeletion.

So why it was possible to have blockOwnerDeletion on DV and not on PVC? Don't we want to have it also on PVC?

arnongilboa · 2022-08-31T10:59:06Z

You say "Eliminate errors like persistentvolumeclaims SOME_PVC is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers".

But I do not understand what does it mean. Why do we have such error?

We have this error in OpenShift where VM and DV are created. However, the PVC is created by CDI which is not allowed to set finalizers of the VM, triggered when PVC is updated with its DV ownerRef to the VM with blockOwnerDeletion: true. This gets validated with this admission controller which is enabled in OpenShift, but disabled in our kubevirtci clusters.

arnongilboa · 2022-08-31T11:00:15Z

Ok, so we are trying to copy ownerReferences from DV to PVC, before deleting DV. And we cannot copy blockOwnerDeletion.

So why it was possible to have blockOwnerDeletion on DV and not on PVC? Don't we want to have it also on PVC?

See answer to your other comment. Adding it to the PR description.

akalenyu · 2022-08-31T11:33:59Z

pkg/controller/datavolume-controller.go

+			}
+		}
+		if !refExists {
+			dvOwnerRef.BlockOwnerDeletion = nil


maybe a comment above this with the admission plugin name and short explanation that we're taking ownerrefs that can be anything in the world?

Added in the PR description. Looks ok?

yeah, the PR description is awesome I just wanted to make sure we have something in the code that screams it

Sure, adding.

akalenyu · 2022-08-31T11:36:36Z

pkg/controller/datavolume-controller.go

+			}
+		}
+		if !refExists {
+			dvOwnerRef.BlockOwnerDeletion = nil


from https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/:

Kubernetes automatically sets this field to true if a [controller](https://kubernetes.io/docs/concepts/architecture/controller/) (for example, the Deployment controller) sets the value of the metadata.ownerReferences field.

I think we may be better with an explicit false value here, but maybe I got this wrong

Sure, fixing.

We should probably try to test this somehow that I am not lying :D
Maybe push the cdi-controller image to quay and we'll try to swap to it in a running cluster?

Tested it already on @kgoldbla cluster.

brybacki · 2022-08-31T11:54:57Z

Nice description, thanks.

Thinking more about the second part of my question: Don't we want to have blockOwnerDeletion also on PVC?

I see that it is impossible. Does it mean we loose part of functionality - namely deletion protection - by doing DV GC?

arnongilboa · 2022-08-31T12:34:19Z

Nice description, thanks.

Thinking more about the second part of my question: Don't we want to have blockOwnerDeletion also on PVC?

I see that it is impossible. Does it mean we loose part of functionality - namely deletion protection - by doing DV GC?

After DV is GC'ed, PVC is still protected from deletion:

  finalizers:
  - kubernetes.io/pvc-protection

awels · 2022-08-31T13:17:30Z

So the documentation says this about blockOwnerDeletion: true

Kubernetes also adds finalizers to an owner resource when you use either [foreground or orphan cascading deletion](https://kubernetes.io/docs/concepts/architecture/garbage-collection/#cascading-deletion). In foreground deletion, it adds the foreground finalizer so that the controller must delete dependent resources that also have ownerReferences.blockOwnerDeletion=true before it deletes the owner.

For our purposes we use the ownerRefs for cascading deletion of the dependents and we don't really care about the blockOwnerDeletion. So right now if you have a VM with DVTemplates that created the DVs, and then the DVs created the PVCs, if you delete the VM, it also cascade deletes the DVs and PVCs. Now if we do a foreground delete of the VM, I guess it makes sure that the DVs and PVCs are deleted before the VM is deleted. I don't think we care about the order in which the objects are deleted in that scenario.

arnongilboa · 2022-08-31T13:43:49Z

So the documentation says this about blockOwnerDeletion: true
Kubernetes also adds finalizers to an owner resource when you use either [foreground or orphan cascading deletion](https://kubernetes.io/docs/concepts/architecture/garbage-collection/#cascading-deletion). In foreground deletion, it adds the foreground finalizer so that the controller must delete dependent resources that also have ownerReferences.blockOwnerDeletion=true before it deletes the owner.
For our purposes we use the ownerRefs for cascading deletion of the dependents and we don't really care about the blockOwnerDeletion. So right now if you have a VM with DVTemplates that created the DVs, and then the DVs created the PVCs, if you delete the VM, it also cascade deletes the DVs and PVCs. Now if we do a foreground delete of the VM, I guess it makes sure that the DVs and PVCs are deleted before the VM is deleted. I don't think we care about the order in which the objects are deleted in that scenario.

I also think the deletion order is not critical here, so blockOwnerDeletion=false doesn't seem that risky.

mhenriks · 2022-08-31T13:55:59Z

What is the minimal rbac that would be required to get this to work (keep block owner deletion) in the KubeVirt universe? My preference would be go that route. Then if we encounter a DV that is owned by some other resource (prob very rare in practice) we don't garbage collect it.

akalenyu · 2022-08-31T14:09:29Z

What is the minimal rbac that would be required to get this to work (keep block owner deletion) in the KubeVirt universe? My preference would be go that route. Then if we encounter a DV that is owned by some other resource (prob very rare in practice) we don't garbage collect it.

its "virtualmachines/finalizers" update if I'm not mistaken, but I am not sure I got

Then if we encounter a DV that is owned by some other resource (prob very rare in practice) we don't garbage collect it

Wouldn't we GC it? I thought we would, but would just lose the order mechanism with blockOwnerDeletion set to false
If we go this route people from outside of CDI will need to submit patches for their "specialoutsideofkubevirtresource/finalizers" in our RBAC

arnongilboa · 2022-08-31T14:23:39Z

What is the minimal rbac that would be required to get this to work (keep block owner deletion) in the KubeVirt universe? My preference would be go that route. Then if we encounter a DV that is owned by some other resource (prob very rare in practice) we don't garbage collect it.

Not sure if I like this tightly coupled KubeVirt-CDI behavior and different/no-GC for other envs. Why keeping blockOwnerDeletion: true for PVC DV-inherited ownerRefs is that dramatic?

akalenyu · 2022-08-31T16:43:20Z

/test pull-containerized-data-importer-e2e-ceph
#1988

awels · 2022-08-31T16:56:19Z

Just so I understand the problem properly. When KubeVirt created a VM with some DVTemplates, it creates the VM with an ownerRef of the VM, and it has blockOwnerDeletion: true on it. Now when we garbage collect the DV, we attempt to set the ownerRef of the PVC to be whatever is on the DV, and that is failing due to the ownerreferencepermissionenforement controller being enabled by default in 1.24 or 1.25?

There is no issue because when KubeVirt creates the DV it has the permissions to put a finalizer on the VM. But when CDI tries to set the same ownerRef it can't because it has no permissions to put a finalizer on the VM. Is there a possibility for us to push the responsibility of setting blockOwnerDeletion: true into KubeVirt (since it has the right RBAC to do so). Maybe have it detect PVCs without that and matching DVTemplates?

aglitke · 2022-09-02T14:47:33Z

pkg/controller/datavolume-controller.go

+// https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#ownerreferencespermissionenforcement
+func isGarbageCollectionAllowed(dv *cdiv1.DataVolume, log logr.Logger) bool {
+	for _, dvOwnerRef := range dv.OwnerReferences {
+		if dvOwnerRef.Kind != "VirtualMachine" &&


I would suggest using the authorization APIs to check if you are able to update the finalizer for the parent object rather than checking for a specific type. If you do this, then someone can make garbage collection work for any parent type so long as the provide the required rbac rules in the deployment.

provide the required rbac rules in the deployment

As in, they would provide the RBAC to the cdi controller service account? through their projects?

authorization APIs

Using these?https://github.com/kubernetes/kubernetes/blob/272e245f06e425fc912b1e477e8b6763850b161e/staging/src/k8s.io/kubectl/pkg/cmd/auth/cani.go#L256-L301

Thanks @aglitke, now using authorization API.

Great idea @aglitke!

pkg/controller/datavolume-controller.go

PVC gets its DV ownerRefs, which can be any CR. If BlockOwnerDeletion is true, we allow GC only if CDI has RBAC to update the owner finalizers (we explicitly added it for VirtualMachines). BlockOwnerDeletions gets validated with this admission plugin which is enabled in OpenShift, but disabled in our kubevirtci clusters: https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#ownerreferencespermissionenforcement Signed-off-by: Arnon Gilboa <agilboa@redhat.com>

mhenriks · 2022-09-06T14:41:39Z

/lgtm
/approve

kubevirt-bot · 2022-09-06T14:41:47Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mhenriks

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mhenriks]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

arnongilboa · 2022-09-06T15:18:11Z

/test pull-cdi-generate-verify

kubevirt-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Aug 30, 2022

kubevirt-bot requested review from aglitke and mhenriks August 30, 2022 16:25

kubevirt-bot added the size/XS label Aug 30, 2022

arnongilboa force-pushed the dv_gc_no_pvc_block_owner_deletion branch from b29d5fe to 9dd985a Compare August 31, 2022 07:12

kubevirt-bot added size/S and removed size/XS labels Aug 31, 2022

brybacki reviewed Aug 31, 2022

View reviewed changes

pkg/controller/datavolume-controller.go Outdated Show resolved Hide resolved

akalenyu reviewed Aug 31, 2022

View reviewed changes

arnongilboa force-pushed the dv_gc_no_pvc_block_owner_deletion branch 2 times, most recently from b7aaf62 to c89fad0 Compare September 2, 2022 05:59

kubevirt-bot added size/M and removed size/S labels Sep 2, 2022

aglitke reviewed Sep 2, 2022

View reviewed changes

arnongilboa force-pushed the dv_gc_no_pvc_block_owner_deletion branch 3 times, most recently from 406530b to 1f90644 Compare September 5, 2022 08:11

arnongilboa changed the title ~~Do not add PVC BlockOwnerDeletion on DV GC~~ GC only if allowed to update DV owner finalizers Sep 5, 2022

arnongilboa force-pushed the dv_gc_no_pvc_block_owner_deletion branch 4 times, most recently from 1c2edab to 4ab4594 Compare September 5, 2022 09:00

brybacki reviewed Sep 5, 2022

View reviewed changes

pkg/controller/datavolume-controller.go Show resolved Hide resolved

arnongilboa force-pushed the dv_gc_no_pvc_block_owner_deletion branch from 4ab4594 to 572bcf6 Compare September 5, 2022 13:18

arnongilboa force-pushed the dv_gc_no_pvc_block_owner_deletion branch from 572bcf6 to 1f80b98 Compare September 6, 2022 12:05

kubevirt-bot assigned mhenriks Sep 6, 2022

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Sep 6, 2022

kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 6, 2022

kubevirt-bot merged commit 119a67a into kubevirt:main Sep 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GC only if allowed to update DV owner finalizers #2416

GC only if allowed to update DV owner finalizers #2416

arnongilboa commented Aug 30, 2022 •

edited

Loading

brybacki Aug 31, 2022

arnongilboa Aug 31, 2022

brybacki Aug 31, 2022

brybacki Aug 31, 2022

arnongilboa Aug 31, 2022

brybacki commented Aug 31, 2022

brybacki commented Aug 31, 2022

arnongilboa commented Aug 31, 2022

arnongilboa commented Aug 31, 2022

akalenyu Aug 31, 2022

arnongilboa Aug 31, 2022

akalenyu Aug 31, 2022

arnongilboa Aug 31, 2022

akalenyu Aug 31, 2022

arnongilboa Aug 31, 2022

akalenyu Aug 31, 2022

arnongilboa Aug 31, 2022

akalenyu Aug 31, 2022

brybacki commented Aug 31, 2022

arnongilboa commented Aug 31, 2022

awels commented Aug 31, 2022

arnongilboa commented Aug 31, 2022

mhenriks commented Aug 31, 2022

akalenyu commented Aug 31, 2022

arnongilboa commented Aug 31, 2022

akalenyu commented Aug 31, 2022

awels commented Aug 31, 2022

aglitke Sep 2, 2022

akalenyu Sep 4, 2022 •

edited

Loading

arnongilboa Sep 5, 2022

mhenriks Sep 6, 2022

mhenriks commented Sep 6, 2022

kubevirt-bot commented Sep 6, 2022

arnongilboa commented Sep 6, 2022

GC only if allowed to update DV owner finalizers #2416

GC only if allowed to update DV owner finalizers #2416

Conversation

arnongilboa commented Aug 30, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brybacki commented Aug 31, 2022

brybacki commented Aug 31, 2022

arnongilboa commented Aug 31, 2022

arnongilboa commented Aug 31, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brybacki commented Aug 31, 2022

arnongilboa commented Aug 31, 2022

awels commented Aug 31, 2022

arnongilboa commented Aug 31, 2022

mhenriks commented Aug 31, 2022

akalenyu commented Aug 31, 2022

arnongilboa commented Aug 31, 2022

akalenyu commented Aug 31, 2022

awels commented Aug 31, 2022

Choose a reason for hiding this comment

akalenyu Sep 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mhenriks commented Sep 6, 2022

kubevirt-bot commented Sep 6, 2022

arnongilboa commented Sep 6, 2022

arnongilboa commented Aug 30, 2022 •

edited

Loading

akalenyu Sep 4, 2022 •

edited

Loading