Patch verb for persistentvolumes resources in the external-provisioner-runner clusterrole is not effective

[sameshai]

What happened:
As per PR below PR If the feature-gate HonorPVReclaimPolicy is enabled, create a pvc with a delete relaim policy, then delete the pvc, the pv stuck in deleting status, the error message is: #1155

csi-mockplugin-0/csi-provisioner@kind..lane: I0206 11:08:41.536819       1 controller.go:1523] delete "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38": failed to remove finalizer for persistentvolume: persistentvolumes "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38" is forbidden: User "system:serviceaccount:csi-mock-honor-pv-reclaim-policy-99-996:csi-mock" cannot update resource "persistentvolumes" in API group "" at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io "cluster-driver-registrar-runner-csi-mock-honor-pv-reclaim-policy-99" not found, clusterrole.rbac.authorization.k8s.io "e2e-test-privileged-psp" not found]
csi-mockplugin-0/csi-provisioner@kind..lane: W0206 11:08:41.536850       1 controller.go:989] Retrying syncing volume "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38", failure 0
csi-mockplugin-0/csi-provisioner@kind..lane: E0206 11:08:41.536880       1 controller.go:1007] error syncing volume "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38": persistentvolumes "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38" is forbidden: User "system:serviceaccount:csi-mock-honor-pv-reclaim-policy-99-996:csi-mock" cannot update resource "persistentvolumes" in API group "" at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io "cluster-driver-registrar-runner-csi-mock-honor-pv-reclaim-policy-99" not found, clusterrole.rbac.authorization.k8s.io "e2e-test-privileged-psp" not found]
I0206 19:08:

I did try this with IBM VPC Block CSI driver and 5.0.2 provisioner but seems without added the patch permission I am still able to delete PVC/PV and no error.

sameershaikh@Sameers-MBP Downloads % k describe pvc
Name:     pvc-block
Namespace:   default
StorageClass: ibmc-vpc-block-5iops-tier
Status:    Bound
Volume:    pvc-351e6980-ac05-4441-a482-6ea544b3aad5
Labels:    <none>
Annotations:  pv.kubernetes.io/bind-completed: yes
        pv.kubernetes.io/bound-by-controller: yes
        volume.beta.kubernetes.io/storage-provisioner: vpc.block.csi.ibm.io
        volume.kubernetes.io/storage-provisioner: vpc.block.csi.ibm.io
Finalizers:  [kubernetes.io/pvc-protection]
Capacity:   10Gi
Access Modes: RWO
VolumeMode:  Filesystem
Used By:    <none>
Events:    <none>
sameershaikh@Sameers-MBP Downloads % k describe pv 
Name:       pvc-351e6980-ac05-4441-a482-6ea544b3aad5
Labels:      <none>
Annotations:    pv.kubernetes.io/provisioned-by: vpc.block.csi.ibm.io
          volume.kubernetes.io/provisioner-deletion-secret-name: 
          volume.kubernetes.io/provisioner-deletion-secret-namespace: 
Finalizers:    [kubernetes.io/pv-protection external-provisioner.volume.kubernetes.io/finalizer]
StorageClass:   ibmc-vpc-block-5iops-tier
Status:      Bound
Claim:       default/pvc-block
Reclaim Policy:  Delete
Access Modes:   RWO
VolumeMode:    Filesystem
Capacity:     10Gi
Node Affinity:   
 Required Terms:  
  Term 0:    failure-domain.beta.kubernetes.io/region in [us-south]
          failure-domain.beta.kubernetes.io/zone in [us-south-3]
Message:      
Source:
  Type:       CSI (a Container Storage Interface (CSI) volume source)
  Driver:      vpc.block.csi.ibm.io
  FSType:      ext4
  VolumeHandle:   r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4
  ReadOnly:     false
  VolumeAttributes:   clusterID=cqsr5km10bghfqv414rg
              failure-domain.beta.kubernetes.io/region=us-south
              failure-domain.beta.kubernetes.io/zone=us-south-3
              iops=3000
              storage.kubernetes.io/csiProvisionerIdentity=1723447099267-9715-vpc.block.csi.ibm.io
              tags=
              volumeCRN=crn:v1:staging:public:is:us-south-3:a/77f2bceddaeb577dcaddb4073fe82c1c::volume:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4
              volumeId=r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4
Events:
 Type  Reason        Age  From                      Message
 ----  ------        ----  ----                      -------
 Normal VolumeMetaDataSaved 2m20s ibm-vpc-block-csi-controller-588c75b988-xpkbt Success

sameershaikh@Sameers-MBP file-csi-driver % k describe clusterRole vpc-block-provisioner-role
Name:     vpc-block-provisioner-role
Labels:    addonmanager.kubernetes.io/mode=Reconcile
       app=ibm-vpc-block-csi-driver
Annotations: razee.io/build-url: https://travis.ibm.com/alchemy-containers/addon-vpc-block-csi-driver/builds/22121223
       razee.io/source-url: https://github.ibm.com/alchemy-containers/addon-vpc-block-csi-driver/commit/d7c3406c7f992ef727f3e9e3ce60f4e33c4f20c2
       version: 5.2.20_579
PolicyRule:
 Resources                    Non-Resource URLs Resource Names Verbs
 ---------                    ----------------- -------------- -----
 persistentvolumes                []         []       [get list watch create delete]
 persistentvolumeclaims             []         []       [get list watch update]
 nodes                      []         []       [get list watch]
 secrets                     []         []       [get list watch]
 csinodes.storage.k8s.io             []         []       [get list watch]
 storageclasses.storage.k8s.io          []         []       [get list watch]
 volumeattachments.storage.k8s.io        []         []       [get list watch]
 configmaps                   []         []       [get list]
 volumesnapshotcontents.snapshot.storage.k8s.io []         []       [get list]
 volumesnapshots.snapshot.storage.k8s.io     []         []       [get list]
 events                     []         []       [list watch create update patch]
sameershaikh@Sameers-MBP file-csi-driver %

},\"f:failure-domain.beta.kubernetes.io/region\":{},\"f:failure-domain.beta.kubernetes.io/zone\":{},\"f:iops\":{},\"f:storage.kubernetes.io/csiProvisionerIdentity\":{},\"f:tags\":{},\"f:volumeCRN\":{},\"f:volumeId\":{}},\"f:volumeHandle\":{}},\"f:nodeAffinity\":{\".\":{},\"f:required\":{}},\"f:persistentVolumeReclaimPolicy\":{},\"f:storageClassName\":{},\"f:volumeMode\":{}}} } {csi-attacher Update v1 2024-08-12 13:05:53 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:finalizers\":{\"v:\\\"external-attacher/vpc-block-csi-ibm-io\\\"\":{}}}} } {csi-resizer Update v1 2024-08-12 13:07:05 +0000 UTC FieldsV1 {\"f:spec\":{\"f:capacity\":{\"f:storage\":{}}}} } {kube-controller-manager Update v1 2024-08-12 13:08:58 +0000 UTC FieldsV1 {\"f:status\":{\"f:phase\":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{21474836480 0} {<nil>} 20Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:vpc.block.csi.ibm.io,VolumeHandle:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,ReadOnly:false,FSType:ext4,VolumeAttributes:map[string]string{clusterID: cqsr5km10bghfqv414rg,failure-domain.beta.kubernetes.io/region: us-south,failure-domain.beta.kubernetes.io/zone: us-south-3,iops: 3000,storage.kubernetes.io/csiProvisionerIdentity: 1723447099267-9715-vpc.block.csi.ibm.io,tags: ,volumeCRN: crn:v1:staging:public:is:us-south-3:a/77f2bceddaeb577dcaddb4073fe82c1c::volume:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,volumeId: r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:pvc-block,UID:351e6980-ac05-4441-a482-6ea544b3aad5,APIVersion:v1,ResourceVersion:2679,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:ibmc-vpc-block-5iops-tier,MountOptions:[],VolumeMode:*Filesystem,NodeAffinity:&VolumeNodeAffinity{Required:&NodeSelector{NodeSelectorTerms:[]NodeSelectorTerm{NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:failure-domain.beta.kubernetes.io/region,Operator:In,Values:[us-south],},NodeSelectorRequirement{Key:failure-domain.beta.kubernetes.io/zone,Operator:In,Values:[us-south-3],},},MatchFields:[]NodeSelectorRequirement{},},},},},VolumeAttributesClassName:nil,},Status:PersistentVolumeStatus{Phase:Released,Message:,Reason:,LastPhaseTransitionTime:<nil>,},}"
I0812 13:09:13.994031    1 controller.go:1258] "shouldDelete" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:13.994053    1 controller.go:1288] "shouldDelete is true" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:13.994064    1 controller.go:1132] "shouldDelete" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:13.994078    1 controller.go:1548] "Started" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:13.994098    1 controller.go:1312] volume pvc-351e6980-ac05-4441-a482-6ea544b3aad5 does not need any deletion secrets
I0812 13:09:13.994456    1 connection.go:264] "GRPC call" method="/csi.v1.Controller/DeleteVolume" request="{\"volume_id\":\"r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4\"}"
I0812 13:09:19.569875    1 reflector.go:808] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: Watch close - *v1.PersistentVolume total 13 items received
I0812 13:09:24.904270    1 connection.go:270] "GRPC response" response="{}" err=null
I0812 13:09:24.904325    1 controller.go:1563] "Volume deleted" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
W0812 13:09:24.966351    1 warnings.go:70] spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].key: failure-domain.beta.kubernetes.io/region is deprecated since v1.17; use "topology.kubernetes.io/region" instead
W0812 13:09:24.966364    1 warnings.go:70] spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[1].key: failure-domain.beta.kubernetes.io/zone is deprecated since v1.17; use "topology.kubernetes.io/zone" instead
I0812 13:09:24.966561    1 controller.go:1606] "PersistentVolume deleted succeeded" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966783    1 controller.go:1171] "handleProtectionFinalizer" PV="&PersistentVolume{ObjectMeta:{pvc-351e6980-ac05-4441-a482-6ea544b3aad5  3f0a9a59-3442-4171-b67a-efdd232c3664 19236 0 2024-08-12 07:31:18 +0000 UTC 2024-08-12 13:09:24 +0000 UTC 0xc0005b1210 map[] map[pv.kubernetes.io/provisioned-by:vpc.block.csi.ibm.io volume.kubernetes.io/provisioner-deletion-secret-name: volume.kubernetes.io/provisioner-deletion-secret-namespace:] [] [kubernetes.io/pv-protection external-provisioner.volume.kubernetes.io/finalizer] [{csi-provisioner Update v1 2024-08-12 13:01:59 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pv.kubernetes.io/provisioned-by\":{},\"f:volume.kubernetes.io/provisioner-deletion-secret-name\":{},\"f:volume.kubernetes.io/provisioner-deletion-secret-namespace\":{}},\"f:finalizers\":{\"v:\\\"external-provisioner.volume.kubernetes.io/finalizer\\\"\":{}}},\"f:spec\":{\"f:accessModes\":{},\"f:capacity\":{},\"f:claimRef\":{\".\":{},\"f:apiVersion\":{},\"f:kind\":{},\"f:name\":{},\"f:namespace\":{},\"f:resourceVersion\":{},\"f:uid\":{}},\"f:csi\":{\".\":{},\"f:driver\":{},\"f:fsType\":{},\"f:volumeAttributes\":{\".\":{},\"f:clusterID\":{},\"f:failure-domain.beta.kubernetes.io/region\":{},\"f:failure-domain.beta.kubernetes.io/zone\":{},\"f:iops\":{},\"f:storage.kubernetes.io/csiProvisionerIdentity\":{},\"f:tags\":{},\"f:volumeCRN\":{},\"f:volumeId\":{}},\"f:volumeHandle\":{}},\"f:nodeAffinity\":{\".\":{},\"f:required\":{}},\"f:persistentVolumeReclaimPolicy\":{},\"f:storageClassName\":{},\"f:volumeMode\":{}}} } {csi-resizer Update v1 2024-08-12 13:07:05 +0000 UTC FieldsV1 {\"f:spec\":{\"f:capacity\":{\"f:storage\":{}}}} } {kube-controller-manager Update v1 2024-08-12 13:08:58 +0000 UTC FieldsV1 {\"f:status\":{\"f:phase\":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{21474836480 0} {<nil>} 20Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:vpc.block.csi.ibm.io,VolumeHandle:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,ReadOnly:false,FSType:ext4,VolumeAttributes:map[string]string{clusterID: cqsr5km10bghfqv414rg,failure-domain.beta.kubernetes.io/region: us-south,failure-domain.beta.kubernetes.io/zone: us-south-3,iops: 3000,storage.kubernetes.io/csiProvisionerIdentity: 1723447099267-9715-vpc.block.csi.ibm.io,tags: ,volumeCRN: crn:v1:staging:public:is:us-south-3:a/77f2bceddaeb577dcaddb4073fe82c1c::volume:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,volumeId: r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:pvc-block,UID:351e6980-ac05-4441-a482-6ea544b3aad5,APIVersion:v1,ResourceVersion:2679,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:ibmc-vpc-block-5iops-tier,MountOptions:[],VolumeMode:*Filesystem,NodeAffinity:&VolumeNodeAffinity{Required:&NodeSelector{NodeSelectorTerms:[]NodeSelectorTerm{NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:failure-domain.beta.kubernetes.io/region,Operator:In,Values:[us-south],},NodeSelectorRequirement{Key:failure-domain.beta.kubernetes.io/zone,Operator:In,Values:[us-south-3],},},MatchFields:[]NodeSelectorRequirement{},},},},},VolumeAttributesClassName:nil,},Status:PersistentVolumeStatus{Phase:Released,Message:,Reason:,LastPhaseTransitionTime:<nil>,},}"
I0812 13:09:24.966803    1 controller.go:1258] "shouldDelete" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966813    1 controller.go:1288] "shouldDelete is true" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966820    1 controller.go:1132] "shouldDelete" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966830    1 controller.go:1548] "Started" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966844    1 controller.go:1312] volume pvc-351e6980-ac05-4441-a482-6ea544b3aad5 does not need any deletion secrets
I0812 13:09:24.966899    1 connection.go:264] "GRPC call" method="/csi.v1.Controller/DeleteVolume" request="{\"volume_id\":\"r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4\"}"
I0812 13:09:25.194175    1 connection.go:270] "GRPC response" response="{}" err=null
I0812 13:09:25.194222    1 controller.go:1563] "Volume deleted" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:25.207594    1 controller.go:1569] "Failed to delete persistentvolume" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5" err="persistentvolumes \"pvc-351e6980-ac05-4441-a482-6ea544b3aad5\" not found"
I0812 13:09:25.207641    1 controller.go:1007] "Retrying syncing volume" key="pvc-351e6980-ac05-4441-a482-6ea544b3aad5" failures=0
E0812 13:09:25.207795    1 controller.go:1025] error syncing volume "pvc-351e6980-ac05-4441-a482-6ea544b3aad5": persistentvolumes "pvc-351e6980-ac05-4441-a482-6ea544b3aad5" not found
I0812 13:09:48.412100    1 reflector.go:808] k8s.io/client-go/informers/factory.go:160: Watch close - *v1.CSINode total 9 items received
I0812 13:10:11.430774    1 reflector.go:808] k8s.io/client-go/informers/factory.go:160: Watch close - *v1.PersistentVolumeClaim total 15 items received
I0812 13:11:05.631537    1 reflector.go:808] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: Watch close - *v1.StorageClass total 11 items received
I0812 13:11:58.431219    1 reflector.go:808] k8s.io/client-go/informers/factory.go:160: Watch close - *v1.Node total 21 items received

What you expected to happen:
I was expecting RBAC error

How to reproduce it:

1. Create PVC with external provisioner
2. Delete PVC which will trigger the PV deletion
3. There is no error even if we dont have patch permission.

Anything else we need to know?:

Environment:

• Driver version:
• Kubernetes version (use kubectl version):
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others: