From e65390f70686d66be37fb994e2684cf6a0862218 Mon Sep 17 00:00:00 2001 From: Humble Chirammal Date: Wed, 23 Mar 2022 18:27:30 +0530 Subject: [PATCH] Add csi nodeexpandsecret blog article Signed-off-by: Humble Chirammal --- .../_posts/2022-04-19-csi-nodeexpandsecret.md | 190 ++++++++++++++++++ 1 file changed, 190 insertions(+) create mode 100644 content/en/blog/_posts/2022-04-19-csi-nodeexpandsecret.md diff --git a/content/en/blog/_posts/2022-04-19-csi-nodeexpandsecret.md b/content/en/blog/_posts/2022-04-19-csi-nodeexpandsecret.md new file mode 100644 index 0000000000000..6a1a4481dba78 --- /dev/null +++ b/content/en/blog/_posts/2022-04-19-csi-nodeexpandsecret.md @@ -0,0 +1,190 @@ +--- +layout: blog +title: "Kubernetes 1.25: Use Secrets for Node-Driven Resize of CSI Volumes." +date: 2022-07-26T10:00:00-08:00 +slug: kubernetes-1-25-use-secrets-while-expanding-csi-volumes-on-node-alpha +--- + +**Author:** Humble Chirammal (Red Hat), Louis Koo (deeproute.ai) + +Kubernetes v1.25, released earlier this month, introduced a new feature +that lets your cluster expand storage volumes, even when access to those +volumes requires a secret (for example: a credential for accessing a SAN fabric) +to perform node expand operation. This new behavior is in alpha and you +must enable a feature gate (`CSINodeExpandSecret`) to make use of it. +You must also be using [CSI](https://kubernetes-csi.github.io/docs/) +storage; this change isn't relevant to storage drivers that are built in to Kubernetes. + +TO turn on this new, alpha feature, you enable the `CSINodeExpandSecret` feature +gate for the kube api server and kubelet, provide a mechanism to receive secretRef +configuration as part of NodeExpansion by the CSI drivers thus make use of +the same to perform node side expansion operation with the underlying +storage system. + +## What is this all about? + +Before Kubernetes v1.24, you were able to define a cluster-level StorageClass +that made use of [StorageClass Secrets](https://kubernetes-csi.github.io/docs/secrets-and-credentials-storage-class.html), +but you didn't have any mechanism to specify the credentials that would be used for +operations that take place when the storage was mounted onto a node and when +when the volume has to be expanded at node side. +Even-though Kubernetes CSI have implemented similar mechanism for Controller +side operations, ie secretRef field available in the csi PV source and making +use of it while controllerExpand request has been sent to the CSI driver, similar +field was missing in the nodeExpandVolume request. This was causing some trouble +for different CSI drivers to fulfill below scenarios while it perform NodeExpansion +operation. + +- At times, the CSI driver needs to check the actual size of the backend block storage (or image) + before proceeding with a node-level filesystem expand operation. This avoids false positive returns + from the backend storage cluster during filesystem expands. + +- When a PersistentVolume represents encrypted block storage (for example using LUKS) + you need to provide a passphrase in order to expand the device, and also to make it possible + to grow the filesystem on that device. + +- For various validations at time of node expansion the CSI driver has to be connected + to the backend storage cluster, if the secretRef is part of the + nodeExpandVolume request the CSI driver can make use of the same and connect to + the storage cluster to perform the cluster operations. + +## How does it work? + +To enable this functionality from this version of Kubernetes, Kubernetes Storage SIG have introduced +a new feature gate called `CSINodeExpandSecret`. Once the feature gate is enabled +in the cluster, NodeExpandVolume requests can include a `secretRef` field. The NodeExpandVolume request +is part of CSI; for example, in a request which has been sent from the Kubernetes +control plane to the CSI driver. +The admin can specify these secrets as an opaque parameter in storage class as +he/she specifies other CSI secrets today. The secrets mention in the storage class +has to carry the key name as shown below + +``` +csi.storage.k8s.io/node-expand-secret-name: test-secret +csi.storage.k8s.io/node-expand-secret-namespace: default` +``` + +If feature gates are enabled and storage class carries the above secret configuration, +the subject CSI provisioner receives the credentials from the Secret as part of the NodeExpansion request. + +CSI volumes that require secrets for online expansion will have NodeExpandSecretRef +field set. If not set, the NodeExpandVolume CSI RPC call will be made without a secret. + + + +## Trying it out + + +Step 1: enable `CSINodeExpandSecret` feature gate (Please refer to +[Feature Gates](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/)) + +Step 2: Secret and storage class creation + +Ex: Here's an example manifest for a Secret that holds credentials: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: test-secret + namespace: default +data: + username: bXktYXBw + password: Mzk1MjgkdmRnN0pi +``` + +Here's an example manifest for a StorageClass that refers to those credentials: + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: csi-hostpath-sc +parameters: + csi.storage.k8s.io/node-expand-secret-name: test-secret # the name of the Secret + csi.storage.k8s.io/node-expand-secret-namespace: default # the namespace that the Secret is in +provisioner: hostpath.csi.k8s.io +reclaimPolicy: Delete +volumeBindingMode: Immediate +allowVolumeExpansion: true +``` + + +## Example output + +If the persistentVolumeClaim(`PVC`) was created successfully, Kubernetes Storage SIGcan see it in the `pv.spec.csi` with command `kubectl get persistentvolume -o yaml` + +```yaml +apiVersion: v1 +kind: PersistentVolume +metadata: + annotations: + pv.kubernetes.io/provisioned-by: hostpath.csi.k8s.io + creationTimestamp: "2022-07-23T15:14:07Z" + finalizers: + - kubernetes.io/pv-protection + name: pvc-95eb531a-d675-49f6-940b-9bc3fde83eb0 + resourceVersion: "420263" + uid: 6fa824d7-8a06-4e0c-b722-d3f897dcbd65 +spec: + accessModes: + - ReadWriteOnce + capacity: + storage: 6Gi + claimRef: + apiVersion: v1 + kind: PersistentVolumeClaim + name: csi-pvc + namespace: default + resourceVersion: "419862" + uid: 95eb531a-d675-49f6-940b-9bc3fde83eb0 + csi: + driver: hostpath.csi.k8s.io + nodeExpandSecretRef: + name: test-secret + namespace: default + volumeAttributes: + storage.kubernetes.io/csiProvisionerIdentity: 1648042783218-8081-hostpath.csi.k8s.io + volumeHandle: e21c7809-aabb-11ec-917a-2e2e254eb4cf + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: topology.hostpath.csi/node + operator: In + values: + - rook-node01 + persistentVolumeReclaimPolicy: Delete + storageClassName: csi-hostpath-sc + volumeMode: Filesystem +status: + phase: Bound +``` + +If you then trigger online storage expansion, the kubelet passes the appropriate credentials +to the CSI driver, by loading that Secret and passing the data to the storage driver. + +Here's an example debug log: + +```console +I0330 03:29:51.966241 1 server.go:101] GRPC call: /csi.v1.Node/NodeExpandVolume +I0330 03:29:51.966261 1 server.go:105] GRPC request: {"capacity_range":{"required_bytes":7516192768},"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/hostpath.csi.k8s.io/f7c62e6e08ce21e9b2a95c841df315ed4c25a15e91d8fcaf20e1c2305e5300ab/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_id":"e21c7809-aabb-11ec-917a-2e2e254eb4cf","volume_path":"/var/lib/kubelet/pods/bcb1b2c4-5793-425c-acf1-47163a81b4d7/volumes/kubernetes.io~csi/pvc-95eb531a-d675-49f6-940b-9bc3fde83eb0/mount"} +I0330 03:29:51.966360 1 nodeserver.go:459] req:volume_id:"e21c7809-aabb-11ec-917a-2e2e254eb4cf" volume_path:"/var/lib/kubelet/pods/bcb1b2c4-5793-425c-acf1-47163a81b4d7/volumes/kubernetes.io~csi/pvc-95eb531a-d675-49f6-940b-9bc3fde83eb0/mount" capacity_range: staging_target_path:"/var/lib/kubelet/plugins/kubernetes.io/csi/hostpath.csi.k8s.io/f7c62e6e08ce21e9b2a95c841df315ed4c25a15e91d8fcaf20e1c2305e5300ab/globalmount" volume_capability: access_mode: > secrets: secrets: +``` + +## The future + +As this feature is still in alpha, Kubernetes Storage SIG expect to update or get feedback from CSI driver +authors with more tests and implementation. The community plans to eventually +promote the feature to Beta in upcoming releases. + +## Get involved or learn more? + +The enhancement proposal includes lots of detail about the history and technical +implementation of this feature. + +To know more about Storage Class based dynamic provisioning in Kubernetes please refer: +[Storage Classes](/docs/concepts/storage/storage-classes/) and +[Persistent Volumes](/docs/concepts/storage/persistent-volumes/). + +Please get involved by joining the Kubernetes [Storage SIG](https://github.com/kubernetes/community/blob/master/sig-storage/README.md) (Special Interest Group) to help us enhance this feature. There are a lot of good ideas already and we'd be thrilled to have more!