Update the capabilities needed for client.csi-cephfs-node in case of encryption

[NymanRobin]

Describe the bug

Currently the standard capabilities set in the capabilities.md is not enough to acquire a lock in rados that is needed when running. This can be seen when the LockExclusive is called rados returns the following error code

Failed to lock volume ID 0001-0009-rook-ceph-0000000000000001-489ccf33-13a5-40fc-8460-7dd866bc44de: rados: ret=-1, Operation not permitted

Some discussion and a naive solution how to circumvent this can be found, in this PR discussion

The Solution

It needs to be figured out what to add to the OSD caps to give only the necessary permission to do the lock operation. Further this should then be documented in the capabilities.md .

[NymanRobin]

I tried adding this osd 'allow command "rbd lock"' to the already preexisting spell, as I understood the ceph documentation this should do it, but it complained about syntax. Maybe @Madhu-1 or @nixpanic has some ideas for this? Also how bad is it to simply change to rw without tag or is it somehow possible to tag the lock? As you can hear I am quite lost on this one 😄

I guess after the way forward is decided this can also be set in rook as standard?

Also follow up question is it so that I need to restart the pods everytime I change the auth caps for them to take effect or should it be immediate?

[NymanRobin]

For your information @z2000l here is the issue regarding the additional configurations needed for cephfs encryption

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.