doc: add proposal for fetching ceph credentials from KMS

[saiprashanth173]

Add proposal to support fetching Ceph credentials from KMS

Signed-off-by: Prashanth Dintyala vdintyala@nvidia.com

Describe what this PR does

Ceph-CSI currently relies on K8s secrets for ceph user (provisioner, node-stage, expand-secret) credentials. Given that ceph-csi already supports managing volume encryption keys through vault and other KMS, it should be possible to extend the existing implementation to fetch Ceph user credentials as well.

This PR has a high level design proposal for this feature.

Related slack discussion with @Madhu-1 @nixpanic: https://cephcsi.slack.com/archives/CJV6G2DFT/p1662485493014329

---

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

• /retest ci/centos/<job-name>: retest the <job-name> after unrelated
  failure (please report the failure too!)
• /retest all: run this in case the CentOS CI failed to start/report any test
  progress or results

[humblec]

@saiprashanth173 thanks for the proposal!

Looking into this, I am really confused and here is why.

Ceph-CSI relies on K8s CSI components for fetching Ceph user credentials.
K8s CSI uses the following reserved StorageClass parameters for fetching
credentials and passes them to Ceph-CSI.
csi.storage.k8s.io/controller-expand-secret-name
csi.storage.k8s.io/controller-expand-secret-namespace
csi.storage.k8s.io/provisioner-secret-name
csi.storage.k8s.io/provisioner-secret-namespace
csi.storage.k8s.io/node-stage-secret-name
csi.storage.k8s.io/node-stage-secret-namespace

These secrets and its management has nothing to do with the CSI driver itself. This is the mechanism co-ordinated from CO ( ex: Kube) based on the CSI spec . We ( as in Ceph CSI drivers) just consume it in the path! It is not something like "go and fetch" from "secrets" from CSI driver pov. These are provided as part of the RPC call to the driver like any other SC-> CSI driver combo.
These entries in the SC is common for all CSI drivers and nothing specific to Ceph CSI driver thus Ceph CSI driver has no role here.

StorageClass parameters cannot be used for credentials as
they are not passed in all CSI requests e.g DeleteVolumeRequest.

This is well intended and its not supposed to be used and it works.

Regardless, keeping the Ceph user information in KMS/Kube object is something we have to get the discussions on and we dont have to clobber it with above secrets..etc. In that light, adding more details on "use case" , "advantages compared to existing model" , "security aspects" (if any), "backward compatibliity" , "integration to rook as it comes in the workflow" ...etc would be value added to the proposal.

[saiprashanth173]

@humblec thanks for your feedback.

Sorry it took me some time to get back, I was out of the office last week.

These secrets and its management has nothing to do with the CSI driver itself. This is the mechanism co-ordinated from CO ( ex: Kube) based on the CSI spec . We ( as in Ceph CSI drivers) just consume it in the path! It is not something like "go and fetch" from "secrets" from CSI driver pov. These are provided as part of the RPC call to the driver like any other SC-> CSI driver combo.
These entries in the SC is common for all CSI drivers and nothing specific to Ceph CSI driver thus Ceph CSI driver has no role here.

I initially added the background section to give some context on how the current flow works. My intention was not to comment on the mechanism, it was just to summarize it. I removed the background section for now as it appears to add confusion and isn't solving the intended purpose.

This is well intended and its not supposed to be used and it works.

Just to clarify, I was trying to set context for why kmsID in case of credentials should be a part expand-secret, provisioner-secret or node-stage-secret instead of specifying it as a StorageClass parameter, like we do in the case of volume encryption. If you feel this isn't very clear, I can restructure this section a bit. Please let me know.

Regardless, keeping the Ceph user information in KMS/Kube object is something we have to get the discussions on and we dont have to clobber it with above secrets..etc. In that light, adding more details on "use case" , "advantages compared to existing model" , "security aspects" (if any), "backward compatibliity" , "integration to rook as it comes in the workflow" ...etc would be value added to the proposal.

I tried to add some more details to the proposal. I will try to refine the proposal further. Please let me know if you have any other suggestions on the latest version. Happy to make further changes based on your feedback :)

[saiprashanth173]

@Madhu-1 @humblec @nixpanic do you have any suggestion/comments on the proposal? Please let me know.

[Madhu-1]

@saiprashanth173 is this still WIP design doc?

[Madhu-1]

Overall looks good but as mention this is kind of overhead and have performance impact if someone chooses security over performance its good for them.

IMHO again it wont solve this issue, if the kubernetes secrets are compromised anyone can gain access to KMS(as all the KMS details are stored in kubernetes configmap and secret) and get the ceph cluster access.

[Madhu-1]

we dont need to stick with RBD volumes here as this can be used with cephfs also?

[Madhu-1]

This is true only if CephCSI is depoyed externally, when Rook deploys the ceph cluster it creates the required secrets for cephcsi.

[saiprashanth173]

I agree that this feature would directly fit into external setup scenario. However, I would see this as a valid case even when Rook deploys Ceph cluster. I see that Rook creates the following secrets by default, and the user caps are quite broad:

rook-csi-cephfs-node
rook-csi-cephfs-provisioner
rook-csi-rbd-node
rook-csi-rbd-provisioner

However, if we were to support more advanced use case such as soft multi-tenancy. It might make sense to have a ceph users per tenant and provide permissions to only access specific pool(s).

Also, if Rook plans to add support for rook/rook#6374. It would be a valid case even in normal scenario.

[Madhu-1]

However, if we were to support more advanced use case such as soft multi-tenancy. It might make sense to have a ceph users per tenant and provide permissions to only access specific pool(s).

Admin can still create new users with different caps using cephclient CRD and use that with custom storageclasses.

[saiprashanth173]

This is true only if CephCSI is depoyed externally, when Rook deploys the ceph cluster it creates the required secrets for cephcsi.

Sorry, I might have misunderstood your comment.

In this point, I was only referring CephCSI being deployed externally:

For scenarios where Ceph is deployed externally, either as
a standalone or through Rook,

For scenarios where Ceph is deployed externally, by this, I mean Ceph cluster is not in the same K8s cluster as Ceph-CSI. either as a standalone or through Rook by this I mean the external Ceph cluster may have been deployed using Rook or some other tool.

If you feel the sentence isn't clear enough, I can change that a bit. Please let me know.

Admin can still create new users with different caps using cephclient CRD and use that with custom storageclasses.

Thanks for letting me know. You are right, for Rook based Ceph and CephCSI deployments this approach might not be applicable, unless Rook supports storing keyrings directly in a KMS.

[Madhu-1]

Wont this be a major drawback?

[saiprashanth173]

I feel KMS rate limit is a valid concern for anything, including RBD volume encryption, using the KMS implementation. One possible approach to mitigate this would be to use an in-memory cache of Ceph keyrings, given that keyrings don't change too often we only fetch them when the keyring doesn't exist already or is invalid (i.e. on failed ceph auth).

[Madhu-1]

👍 the in-memory cache need to be deleted for few cases like

• auth issues
• connection issues

[saiprashanth173]

I added a point on in-memory cache under Code Changes sections.

Do you mind elaborating a bit on what you mean by connection issues in this context?

[Madhu-1]

why we need extra kms configmap cant we use existing kms configuration?

[saiprashanth173]

Technically, we don't need a new config here. Given that structure of encryption and creds KMS config entries slight differ, specially due to the newly added tenantNamespace key and missing tenant sub config in case of creds, I thought it might make sense to keep them separate.

I can change this section to use the same configuration if that makes sense. Please let me know what you think.

[Madhu-1]

Yes please

[saiprashanth173]

I updated this section for reusing the existing KMS configuration.

[Madhu-1]

this is not something specific to this feature right?

[saiprashanth173]

Sorry, I don't follow the question completely.

The need to add "tenantNamespace" in config is specific to this feature, as we don't have the tenant namespace available in all RPC calls. RBD volume encryption uses the tenants' namespace for all its operations.

[Madhu-1]

add the new field to the existing struct and reuse it, lets avoid too many configurations

[saiprashanth173]

Makes sense. I updated the section.

[Madhu-1]

@travisn can you please check this one?

[Rakshith-R]

Overall looks good but as mention this is kind of overhead and have performance impact if someone chooses security over performance its good for them.

IMHO again it wont solve this issue, if the kubernetes secrets are compromised anyone can gain access to KMS(as all the KMS details are stored in kubernetes configmap and secret) and get the ceph cluster access.

+1

[Madhu-1]

Overall looks good but as mention this is kind of overhead and have performance impact if someone chooses security over performance its good for them.
IMHO again it wont solve this issue, if the kubernetes secrets are compromised anyone can gain access to KMS(as all the KMS details are stored in kubernetes configmap and secret) and get the ceph cluster access.

+1

@saiprashanth173 can you please comment on this one, this helps to take this forward.

[nixpanic]

Overall looks good but as mention this is kind of overhead and have performance impact if someone chooses security over performance its good for them.
IMHO again it wont solve this issue, if the kubernetes secrets are compromised anyone can gain access to KMS(as all the KMS details are stored in kubernetes configmap and secret) and get the ceph cluster access.

+1

@saiprashanth173 can you please comment on this one, this helps to take this forward.

This is also what I wonder mostly about. The advantage of placing the Ceph credentials in a KMS is not clear to me. Kubernetes still stores credentials to access the KMS, so getting to the Ceph credentials is still possible without much additional effort?

[saiprashanth173]

IMHO again it wont solve this issue, if the kubernetes secrets are compromised anyone can gain access to KMS(as all the KMS details are stored in kubernetes configmap and secret) and get the ceph cluster access.

Sorry, I missed commenting on this.

I mostly agree that this solution wouldn't solve the issue completely for the KMS integrations that require storing auth tokens in K8s secrets. If we take vault service account based approach, it would make it more difficult (if not impossible) to access credentials compared to K8s secrets, for following reasons:

1. In K8s 1.24+, service account token secrets are not auto-generated and starting Ceph-CSI 3.7 we always attempt to create a service account token and use it with vault. These tokens are short-lived, and creating them would require additional privileges, e.g. RBAC with serviceaccounts/token|create . Related: rbd: create token and use it for vault SA #3174 rbd: create token and use it for vault SA everytime possible #3377
2. Even with cases where a service account token is compromised, the impact will be restricted only to the vault path which the vault role linked to the service account has access to.

I believe storing secrets in KMS can provide additional level of security. Even with compromised access tokens, most KMS support additional security tightening measures which imposes restrictions on who can access the KMS endpoints, making it slightly more difficult to access sensitive secrets.

Additionally, I see this feature solving another issue, specifically for external Ceph setups, where the process of sharing and rotating keyrings can be made more seamless and secure.

Please let me know what you think. Happy to discuss this further.

[nixpanic]

I agree that this proposal could make it easier to deny access (or rotate tokens) for certain users/accounts if credentials are required to access a KMS and fetch the Ceph credentials from there. Careful auditing of the access logs in the KMS might prevent requiring rotating the Ceph credentials after someone obtained the short-living KMS credentials.

It would also become possible (not sure about the smart or recommended practice) to share Ceph credentials from a single location (KMS) over multiple Kubernetes clusters.

With that, I'm supportive of the feature and some users might be interested in it.

[nixpanic]

The parameters in the StorageClass are copied into the PersistentVolume.CSIDriver parameters. I think it would be cleaner to not use the common secrets in the StorageClass for this, but directly refer to a kmsID with potentially additional parameters like userID.

[saiprashanth173]

It is not very clear to me on how we can obtain the PersistentVolume.CSIDriver parameters from calls like DeleteVolume.

Looking at the DeleteVolume method and DeleteVolumeRequest struct:

func (cs *ControllerServer) DeleteVolume(
	ctx context.Context,
	req *csi.DeleteVolumeRequest,
) (*csi.DeleteVolumeResponse, error)

// ....

type DeleteVolumeRequest struct {
	// The ID of the volume to be deprovisioned.
	// This field is REQUIRED.
	VolumeId string `protobuf:"bytes,1,opt,name=volume_id,json=volumeId,proto3" json:"volume_id,omitempty"`
	// Secrets required by plugin to complete volume deletion request.
	// This field is OPTIONAL. Refer to the `Secrets Requirements`
	// section on how to use this field.
	Secrets              map[string]string `protobuf:"bytes,2,rep,name=secrets,proto3" json:"secrets,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
	XXX_NoUnkeyedLiteral struct{}          `json:"-"`
	XXX_unrecognized     []byte            `json:"-"`
	XXX_sizecache        int32             `json:"-"`
}

I only see that we can get VolumeId as part of this request. Are you suggesting we get the PersistentVolume object here to obtain these parameters? If yes, wouldn't that require an additional API call?

Do you mind helping me understand this better?

[nixpanic]

I think this is not very practical in the end. Why have a default user for some operations, and use the tenant for others? Using different Ceph credentials for different operations on a single volume looks rather confusing. Usually you would also try to prevent normal users from accessing the Ceph cluster, so those users should not need to have access to Ceph credentials.

[saiprashanth173]

@nixpanic to clarify my point:

1. the tenantNamespace here is only used for getting everything required by the KMS to fetch the Ceph user credentials. One such example is obtaining the KMS access token, which could be creating a service account token for vault SA or fetching tokens from K8s secret for some other KMS.
2. Any volume associated with a storage class would always use same Ceph credentials for all its operations. I see Ceph credentials as pair of userID and userKey, where userID is obtained from csi.storage.k8s.io/*-secret-name and userKey is fetched from the KMS. The user will never change until we change userID and KMS config.
3. The KMS will only use one namespace for all the operations. If the tenantNamespace is present in the KMS config, it will always use that. If not, it always looks in Ceph-CSI namespace as default. Again, this is only for KMS authentication so that we can fetch the required Ceph credentials.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

[github-actions]

This pull request has been automatically closed due to inactivity. Please re-open if these changes are still required.