Unable to restrict cephfs-csi access to single filesystem

[marco-venuti]

Describe the bug

I have deployed ceph-csi with helm on a k3s cluster. I can succesfully provision volumes on cephfs using a user with the following capabilities, provided in the documentation

ceph auth caps client.kubernetes \
     mgr 'allow rw' \
     mds 'allow rw' \
     mon 'allow r' \
     osd 'allow rw tag cephfs *=*'

However, I'd like to restrict this user access to the single filesystem hosting the pvcs.

I have tried restricting to the filesystem as per ceph's documentation

ceph auth caps client.kubernetes \
     mds 'allow rw' \
     mgr 'allow rw' \
     mon 'allow r' \
     osd 'allow rw tag cephfs data=KubeVolumes'

or to a single pool

ceph auth caps client.kubernetes \
     mgr 'allow rw' \
     mds 'allow rw' \
     mon 'allow r' \
     osd 'allow rw tag cephfs pool=cephfs.KubeVolumes.data'

but in both cases the provisioner fails. In particular, I can only see the following requests repeating on ceph's side (with journalctl)

Nov 30 00:40:50 seppia bash[3867]: audit 2021-11-29T23:40:50.388954+0000 mon.polposition (mon.2) 5007 : audit [DBG] from='client.? 192.168.81.113:0/3588022304' entity='client.kubernetes' cmd=[{"prefix":"fs dump","format":"json"}]: dispatch
Nov 30 00:40:50 seppia bash[3867]: audit 2021-11-29T23:40:50.390970+0000 mon.polposition (mon.2) 5008 : audit [DBG] from='client.? 192.168.81.113:0/3588022304' entity='client.kubernetes' cmd=[{"prefix":"fs ls","format":"json"}]: dispatch

(whereas with sufficient permissions I also see the creation of subvolumegroup and subvolumes).

On kubernetes' side, I get the following logs

# kubectl describe pvc <pvc>
[...]
Type     Reason                Age               From                                                                                                   Message
  ----     ------                ----              ----                                                                                                   -------
  Normal   Provisioning          8s (x5 over 15s)  cephfs.csi.ceph.com_ceph-csi-cephfs-provisioner-5f465c8b64-nfb5q_258b3cf6-dc48-4ecd-a2c0-22e8bc949187  External provisioner is provisioning volume for claim "default/pvc-cephfs-test"
  Warning  ProvisioningFailed    8s (x5 over 15s)  cephfs.csi.ceph.com_ceph-csi-cephfs-provisioner-5f465c8b64-nfb5q_258b3cf6-dc48-4ecd-a2c0-22e8bc949187  failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Internal desc = rados: ret=-1, Operation not permitted
  Normal   ExternalProvisioning  1s (x4 over 26s)  persistentvolume-controller                                                                            waiting for a volume to be created, either by external provisioner "cephfs.csi.ceph.com" or manually created by system administrator

and

# kubectl -n cephfs logs <provisioner> -c csi-provisioner
[...]
I1130 17:07:54.914106       1 connection.go:182] GRPC call: /csi.v1.Controller/CreateVolume
I1130 17:07:54.914133       1 connection.go:183] GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-b423f6b2-993f-47a2-848b-ccc212d08324","parameters":{"clusterID":"2bf78d6c-4b1d-11ec-85e7-003048dfae38","csi.storage.k8s.io/pv/name":"pvc-b423f6b2-993f-47a2-848b-ccc212d08324","csi.storage.k8s.io/pvc/name":"pvc-cephfs-test","csi.storage.k8s.io/pvc/namespace":"default","fsName":"KubeVolumes","mounter":"kernel","pool":"cephfs.KubeVolumes.data","volumeNamePrefix":"cephfs-"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":1}}]}
I1130 17:07:54.920852       1 connection.go:185] GRPC response: {}
I1130 17:07:54.920897       1 connection.go:186] GRPC error: rpc error: code = Internal desc = rados: ret=-1, Operation not permitted
I1130 17:07:54.920935       1 controller.go:753] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Finished: rpc error: code = Internal desc = rados: ret=-1, Operation not permitted
I1130 17:07:54.920984       1 controller.go:1099] Final error received, removing PVC b423f6b2-993f-47a2-848b-ccc212d08324 from claims in progress
W1130 17:07:54.921001       1 controller.go:958] Retrying syncing claim "b423f6b2-993f-47a2-848b-ccc212d08324", failure 6
E1130 17:07:54.921032       1 controller.go:981] error syncing claim "b423f6b2-993f-47a2-848b-ccc212d08324": failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Internal desc = rados: ret=-1, Operation not permitted
I1130 17:07:54.921056       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pvc-cephfs-test", UID:"b423f6b2-993f-47a2-848b-ccc212d08324", APIVersion:"v1", ResourceVersion:"294541", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "csi-cephfs-sc": rpc error: code = Internal desc = rados: ret=-1, Operation not permitted
I1130 17:08:26.921665       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"pvc-cephfs-test", UID:"b423f6b2-993f-47a2-848b-ccc212d08324", APIVersion:"v1", ResourceVersion:"294541", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/pvc-cephfs-test"

However, with the caps specified above, in all cases I can manually create subvolumegroups and subvolumes with

ceph --id kubernetes fs <command>

Environment details

• Helm chart version : 3.4.0
• Kernel version : 5.10.0-9-amd64
• Mounter used for mounting PVC: kernel
• Kubernetes cluster version : k3s version v1.21.5+k3s2 (724ef700), go version go1.16.8
• Ceph cluster version : ceph version 15.2.15 octopus (stable) deployed with cephadm

[nixpanic]

Can you check the logs of the csi-cephfsplugin container in the provisioner Pod? Hopefully that shows more details about the operations that were performed, and where the failure occurs.

[marco-venuti]

Here it is, but the information seems to be the same

I1201 10:07:02.657087       1 utils.go:176] ID: 17 Req-ID: pvc-74abffa3-ffd6-479a-87cb-4449591bfd5b GRPC call: /csi.v1.Controller/CreateVolume
I1201 10:07:02.657691       1 utils.go:180] ID: 17 Req-ID: pvc-74abffa3-ffd6-479a-87cb-4449591bfd5b GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-74abffa3-ffd6-479a-87cb-4449591bfd5b","parameters":{"clusterID":"2bf78d6c-4b1d-11ec-85e7-003048dfae38","csi.storage.k8s.io/pv/name":"pvc-74abffa3-ffd6-479a-87cb-4449591bfd5b","csi.storage.k8s.io/pvc/name":"pvc-cephfs-test","csi.storage.k8s.io/pvc/namespace":"default","fsName":"KubeVolumes","mounter":"kernel","pool":"cephfs.KubeVolumes.data","volumeNamePrefix":"cephfs-"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":1}}]}
E1201 10:07:02.688499       1 utils.go:185] ID: 17 Req-ID: pvc-74abffa3-ffd6-479a-87cb-4449591bfd5b GRPC error: rpc error: code = Internal desc = rados: ret=-1, Operation not permitted

The logs are the same for both sets of caps specified in my last message.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[fhufenreuter]

Hello,

we see the same errors in our Env:

• manual deploy of 3.4.0
• Kernel version: 5.13.19-1-pve
• Kubernetes Version: Major:"1", Minor:"21", GitVersion:"v1.21.6", GitCommit:"d921bc6d1810da51177fbd0ed61dc811c5228097", GitTreeState:"clean", BuildDate:"2021-10-27T17:44:26Z", GoVersion:"go1.16.9"
• Ceph Cluster version: 16.2.6

[Madhu-1]

@marco-venuti @fhufenreuter have you tried manually creating a subvolume/subvolumegroup and rados objects with this restricted user access? you can create a rados object with the below command and check is it working

rados create test -p <metadata-pool-name> --id=<restricted-user-name> --key=<restricted-user-key> --namespace=csi

[marco-venuti]

With

[client.test]
	caps mds = "allow rw"
	caps mgr = "allow rw"
	caps mon = "allow r"
	caps osd = "allow rw tag cephfs data=KubeVolumes"

subvolume(group) creation succeds, but it actually succeds even for filesystems other than KubeVolumes. Deletion also succeds.

Object creation instead fails on the metadata pool, but succeds on data pool:

# rados create test -p cephfs.KubeVolumes.meta --id=test --key=... --namespace=csi
error creating cephfs.KubeVolumes.meta/test: (1) Operation not permitted

[Madhu-1]

cephcsi needs access to create the objects in the cephfs metadata pool (to store some informations in the rados objects) without that PVC creation fails.

[marco-venuti]

Ok, the following seems to be working indeed

caps mds = "allow rw"
caps mgr = "allow rw"
caps mon = "allow r"
caps osd = "allow rw tag cephfs data=KubeVolumes, allow rw tag cephfs metadata=KubeVolumes"

However, I have not tried letting cephcsi create the subvolumegroup, since that had already been done in my cluster with the allow rw tag cephfs *=* privileges. But I tested with ceph fs --id=kubernetes subvolumegroup create ... and that still works with these privileges (for all filesystems).

If you think this is a clean and useful solution, maybe it's worth adding a few lines here (?)

[fhufenreuter]

Following caps also work for us:

caps: [mds] allow rw fsname=cephfs path=/volumes
caps: [mgr] allow rw
caps: [mon] allow r fsname=cephfs
caps: [osd] allow rw tag cephfs data=cephfs, allow rw tag cephfs metadata=cephfs

Update of the docs to make this clear would be great.

[Madhu-1]

@marco-venuti @fhufenreuter if you can contribute we are happy to review the changes. if not let us know anyone from @ceph/contributors can take this up/

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[Madhu-1]

good to get this documented, keeping it alive

[dragoangel]

Following caps also work for us:
...

@fhufenreuter thanks, would add a bit more:

ceph fs volume ls
FS_NAME=cephfs
ceph fs subvolumegroup ls $FS_NAME
SUB_VOL=csi
ceph fs subvolumegroup create $FS_NAME $SUB_VOL
USER=cephfs-csi
ceph auth get-or-create client.$USER mgr "allow rw" osd "allow rw tag cephfs metadata=$FS_NAME, allow rw tag cephfs data=$FS_NAME" mds "allow r fsname=$FS_NAME path=/volumes, allow rws fsname=$FS_NAME path=/volumes/$SUB_VOL" mon "allow r fsname=$FS_NAME"

This setup will add /volumes only read access to protect other volumes from overriding, while provide rw and s - snapshooting capabilities. Honestly don't know why csi doesn't work without access to /volumes as it only need to work with own subvolume and that's it.