[BUG]: Not able to take volumesnapshots

[N1K68]

Bug Description

When trying to create a VolumeSnapshots the creation hanging in pending state forever. According to the csm-isilon-controller log it it missing patch privileges.

After updating the ClusterRole csm-isilon-controller with verb "patch" for
apiGroups:

• snapshot.storage.k8s.io
  resources:
• volumesnapshotcontents

and

• apiGroups:
  • snapshot.storage.k8s.io
    resources:
  • volumesnapshotcontents/status

it start working. But the operator seems to remove these configuration after a while at which point the VolumeSnapshots fails again.

Logs

2023-09-06T17:39:15.830379848Z I0906 17:39:15.830342 1 snapshot_controller.go:190] updating VolumeSnapshotContent[snapcontent-d673d5a9-b354-4c93-9a73-280a3521d733] error status failed volumesnapshotcontents.snapshot.storage.k8s.io "snapcontent-d673d5a9-b354-4c93-9a73-280a3521d733" is forbidden: User "system:serviceaccount:sebshift-powerscale-csi:csm-isilon-controller" cannot patch resource "volumesnapshotcontents/status" in API group "snapshot.storage.k8s.io" at the cluster scope
2023-09-06T17:39:15.830379848Z E0906 17:39:15.830365 1 snapshot_controller.go:107] createSnapshot for content [snapcontent-d673d5a9-b354-4c93-9a73-280a3521d733]: error occurred in createSnapshotWrapper: failed to add VolumeSnapshotBeingCreated annotation on the content snapcontent-d673d5a9-b354-4c93-9a73-280a3521d733: "snapshot controller failed to update snapcontent-d673d5a9-b354-4c93-9a73-280a3521d733 on API server: volumesnapshotcontents.snapshot.storage.k8s.io "snapcontent-d673d5a9-b354-4c93-9a73-280a3521d733" is forbidden: User "system:serviceaccount:sebshift-powerscale-csi:csm-isilon-controller" cannot patch resource "volumesnapshotcontents" in API group "snapshot.storage.k8s.io" at the cluster scope"
2023-09-06T17:39:17.430232402Z I0906 17:39:17.430226 1 snapshot_controller_base.go:185] Failed to sync content "snapcontent-dab19496-b06a-4e96-b790-d60fc3aee8b7", will retry again: failed to add VolumeSnapshotBeingCreated annotation on the content snapcontent-dab19496-b06a-4e96-b790-d60fc3aee8b7: "snapshot controller failed to update snapcontent-dab19496-b06a-4e96-b790-d60fc3aee8b7 on API server: volumesnapshotcontents.snapshot.storage.k8s.io "snapcontent-dab19496-b06a-4e96-b790-d60fc3aee8b7" is forbidden: User "system:serviceaccount:sebshift-powerscale-csi:csm-isilon-controller" cannot patch resource "volumesnapshotcontents" in API group "snapshot.storage.k8s.io" at the cluster scope"
2023-09-06T17:39:17.430341585Z I0906 17:39:17.430319 1 event.go:285] Event(v1.ObjectReference{Kind:"VolumeSnapshotContent", Namespace:"", Name:"snapcontent-dab19496-b06a-4e96-b790-d60fc3aee8b7", UID:"03e793cd-d9c0-41b3-ad21-a9da77e83468", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"3668603121", FieldPath:""}): type: 'Warning' reason: 'SnapshotCreationFailed' Failed to create snapshot: failed to add VolumeSnapshotBeingCreated annotation on the content snapcontent-dab19496-b06a-4e96-b790-d60fc3aee8b7: "snapshot controller failed to update snapcontent-dab19496-b06a-4e96-b790-d60fc3aee8b7 on API server: volumesnapshotcontents.snapshot.storage.k8s.io "snapcontent-dab19496-b06a-4e96-b790-d60fc3aee8b7" is forbidden: User "system:serviceaccount:sebshift-powerscale-csi:csm-isilon-controller" cannot patch resource "volumesnapshotcontents" in API group "snapshot.storage.k8s.io" at the cluster scope"

Screenshots

No response

Additional Environment Information

The privileges that seems to be missing in ClusterRole csm-isilon-controller are:

• apiGroups:
  • snapshot.storage.k8s.io
    resources:
  • volumesnapshotcontents
    verbs:
  • create
  • get
  • list
  • watch
  • update
  • patch <---
  • delete
• apiGroups:
  • snapshot.storage.k8s.io
    resources:
  • volumesnapshotcontents/status
    verbs:
  • update
  • patch <---

Steps to Reproduce

Install the Dell Container Storage Modules 1.2.0 and the PowerScale v2.7.0 module.

Expected Behavior

We expect to be able to take a volumesnapshot.

CSM Driver(s)

CSIDRIVERTYPE: isilon CONFIGVERSION : v2.7.0

Installation Type

Operator 1.2.0

Container Storage Modules Enabled

isilon v2.7.0
resiliency v1.6.0
observability v1.5.0

Container Orchestrator

OpenShift 4.11.37

Operating System

RHEL 8.6

[N1K68]

[jooseppi-luna]

Hi N1K68, thanks so much for bringing this issue to our attention -- we will take this up asap and look into a fix.

[bortek]

Hi

I am facing the same issue. Were you able to reproduce?

Running on Openshift 4.12.31

[bortek]

We are syncing permission (RBAC) using ArgoCD but it looks like the operator is changing it back permission.

[shanmydell]

This bug is being addressed in 1.9.0, thanks for reporting the issue