CSI Volume Plugins in Kubernetes Design Doc #1258

saad-ali · 2017-10-23T08:53:54Z

Design documenting all the requirements for enabling an a CSI compliant volume plugin (a CSI volume driver) in Kubernetes. This design was drafted here, and is now ready for publishing.

This feature is targeted as alpha in v1.9.

jsafrane · 2017-10-23T10:39:32Z

contributors/design-proposals/storage/container-storage-interface.md

+
+More specifically, an external “attacher” must watch the Kubernetes API on behalf of the external CSI volume driver to handle attach/detach requests.
+
+Upon initialization the `external-attacher` should call `GetNodeId` for each node in the cluster and create a mapping from cluster node ID to storage node ID. When issuing `ControllerPublishVolume` calls it should use the mapping.


external attacher operates on "Controller plugin" part of CSI, while GetNodeId is part of "Node plugin" functionality. The attacher would need to run on every node to get ID of all of them.

this is already resolved

jsafrane · 2017-10-23T10:45:04Z

contributors/design-proposals/storage/container-storage-interface.md

+1. A new `VolumeAttachment` Kubernetes API objects is created by Kubernetes.
+2. The `VolumeAttachment.Spec.Attacher` value in that object corresponds to the name of the external attacher.
+3. The `VolumeAttachment.Status.IsAttached` value is not yet set to true.
+


The VolumeAttachment.Metadata.DeletionTimestamp is not set.

jsafrane · 2017-10-23T10:47:53Z

contributors/design-proposals/storage/container-storage-interface.md

+
+The external-attacher may implement it’s own error recovery strategy, and retry as long as conditions specified for attachment above are valid. It is strongly recommended that the external-attacher implement an exponential backoff strategy for retries.
+
+The detach operation will be triggered by the deletion of the `VolumeAttachment` Kubernetes API objects. The `VolumeAttachment` object will have an `external-attacher` finalizer which will wait for confirmation from the external-attacher before deleting the object.


It's not clear who sets the finalizer. Will Kubernetes A/D controller add it there when creating VolumeAttachment? Will external attacher add it there before attaching a volume? Will external attacher implement an external admission hook ("admission plugin") to inject it to all VolumeAttachment objects automatically?

I propose that the external attacher adds it there before calling CSI ControllerPublishVolume - the attacher has full control over its name and lifetime.

Agreed. Added clarification

jsafrane · 2017-10-23T10:52:05Z

contributors/design-proposals/storage/container-storage-interface.md

+
+Once all the following conditions are true, the external-attacher should call `ControllerUnpublishVolume` against the CSI volume driver to detach the volume from the specified node:
+1. A `VolumeAttachment` Kubernetes API object is deleted by Kubernetes: the value for the `VolumeAttachment.metadata.deletionTimestamp` field is set.
+2. The `VolumeAttachment.Status.IsAttached` value in that object is set to `true` (indicating the volume is attached).


It can happen that the external attacher was terminated after attaching a volume and before saving VolumeAttachment.Status.IsAttached to API server. Newly started attacher should try to detach also volumes that have IsAttached==false if the attacher is not sure.

Good point. removed this requirement.

jsafrane · 2017-10-23T10:55:49Z

contributors/design-proposals/storage/container-storage-interface.md

+  Attacher string `json:"attacher" protobuf:"bytes,1,opt,name=attacher"`
+
+  // VolumeSource indicates the volume to attach.
+  Volume VolumeSource `json:",inline" protobuf:"bytes,2,opt,name=volumeSource"`


VolumeSource can have non-empty CSIVolumeSource.SecretRef. This secret ref contains only name of a secret, inheriting the namespace from referring PVC (or pod in case of inline volume). Ergo we need the namespace somewhere in VolumeAttachmentSpec. Or full blown ObjectReference.

Good point. But this gets messy. We end up adding a namespace to a non-namespaced object. And the problem with that is multiple pods may reference the same volume, and the secret required for Attach (ControllerPublishVolume) should not be tied to the namespace of any one of those pods.

Really what we need is separate secrets for Attach and Mount in CSIVolumeSource:

// Optional credentials passed to `NodePublishVolume` (secret must exist in the pod namespace). MountSecretRef *LocalObjectReference `json:"secretRef,omitempty" protobuf:"bytes,3,opt,name=secretRef"` // Optional credentials passed to `ControllerPublishVolume`. AttachSecretRef *ObjectReference `json:"secretRef,omitempty" protobuf:"bytes,4,opt,name=secretRef"`

This way it is clear that there are a set of credentials to attach the volume (potentially shared between multiple pods), and a set of credentials to mount the volume to the pod (potentially different for each pod).

The problem with this is that a malicious user could point AttachSecretRef at any secret in the cluster. But, because CSI volume drivers are responsible for fetching the secret, meaning they have access to all secrets anyway, this may be ok--it gives non-cluster admin no additional privileges.

Thoughts?

in the case of a PV, the credential needed to mount a volume to a pod may not live in the same namespace as the pod (generally, I wouldn't expect it to)

CSI volume drivers are responsible for fetching the secret, meaning they have access to all secrets anyway

Not necessarily. If the CSI driver fetches from the API, you can partition credentials by volume type / provisioner and only give the provisioner access to that namespace.

Alternately, if credentials are passed into the CSI API, the node can obtain the secret and provide it as part of the CSI API call.

Reading @liggitt's comment #1258 (comment), this will actually work very nicely:

Provisioner secrets

Specified with namespace inside StorageClass

external-provisioner access to secrets limited to a specific namespace for that provisioner.

Attacher secrets

Specified with namespace inside CSIVolumeSource (as defined above)

external-attacher access to secrets limited to a specific namespace for that attacher.

Mount secrets

Specified without namespace inside CSIVolumeSource (as defined above) -- Mount secrets must live in the user's namespace,

Since these will be dereferenced by kubelet, kubelet already has access to pod secrets when preparing (and mounting volumes for) pods.

I will update the spec to reflect this.

We end up adding a namespace to a non-namespaced object. And the problem with that is multiple pods may reference the same volume and the secret required for Attach (ControllerPublishVolume) should not be tied to the namespace of any one of those pods.

If multiple pods use the same PV they are all from the same namespace. I would not worry about inline volumes much here - let's use credentials from namespace of the first pod that uses the volume.

I see a different problem: we have two sets of secrets.

Generic CSI secret(s) so CSI driver can authenticate+authorize itself to a storage backend. For example AWS API keys as we use them today - one key for whole AWS cloud provider. We do not need to worry about these secrets, a pod with CSI driver can live in a dedicated namespace and have its own Secrets instance there that gets propagated to the pod as usual (e.g. via env. variables). No problem here.

"End user credentials used to authenticate/authorize" various operations (provisioning, attachment, mounting) from CSI spec. The spec does not cover any details about them, "end user credentials" is all we have.

These credentials can either authenticate Kubernetes user - but do we have a place to store secrets for various CSI drivers for each user? AFAIK we do not tie objects to users. In addition, how can an external provisioner reliably find the right user? All an external provisioner sees is a PVC with some annotations and they can be set/modified by the user.

Or they can authenticate Kubernetes namespaces - which is IMO the case here. We need either a namespaced secret or some dedicated namespace where all secrets will be stored and we can map them to individual namespaces.

Or we won't support end user credentials in Kubernetes. IMO, it's good for alpha - that's how all internal volume plugins work now.

Or we won't support end user credentials in Kubernetes. IMO, it's good for alpha - that's how all internal volume plugins work now.

Except Flex. It can use secret in pod's namespace for mounting :-(.

I want to make sure we captured the notes from our October 24 meeting, so here they are:

For security reasons (to prevent a user from pointing to any secret and getting access to attach and potentially mount a volume they don’t have rights to), only have CSI Volume Source for PVs-- NO inline CSI volume source (can reassess after alpha)

Rename CSIVolumeSource -> CSIPersistentVolumeSource

Call out that if a malicious provisioner can obtain an arbitrary secret by setting the mount secret in PV object to whatever secret it wants.

Have both AttachSecret and MountSecret specify both namespace and secret name.

Use SecretReference instead of ObjectReference.

Make sure that as part of CSI Volume Source work that the NodeAuthorizer is updated to allow kubelet read access to Mount Secrets.

For the future consider for sig-storage

Easier UX for pre provision “but only bind to certain namespaces”

The document has been updated to capture the above.

As to the concern @jsafrane raised, agreed, we can hold off for alpha. Opened container-storage-interface/spec#140 to track this on it on the CSI side.

jsafrane · 2017-10-23T10:57:33Z

contributors/design-proposals/storage/container-storage-interface.md

+  // This is a local object reference so that only secrets within the
+  // same namespace as the pod referencing this volume can be referenced.
+  // +optional
+  SecretRef *LocalObjectReference `json:"secretRef,omitempty" protobuf:"bytes,3,opt,name=secretRef"`


External attacher and provisioner must have access to all secrets in all namespaces in Kubernetes. That sounds scary. Or is there a way how to limit their access only to secrets with certain SecretType?

you cannot limit to a specific secret type. you can limit to a specific namespace. I'd expect the following:

external provisioner is granted read/write access to secrets in a specific namespace (e.g. myfs-provisioner)

storage class for that provisioner sets the config parameter controller the location of secrets to that namespace ,"secretNamespace":"myfs-provisioner",...

the provisioner creates persistent volumes referencing secrets in that namespace

if it provisions secrets, it provisons them in that namespace

@liggitt yes, this works for provisioners! And it works with the proposal in #1258 (comment), which adds two additional sets of secrets: one for attaching and one for mounting.

For the attaching secrets, the same thing you outlined here should work: i.e. a my-external-attacher namespace, and limiting the external-attacher's access to just that namespace.

For the mounting secrets, which must live in the user's namespace, since those will be dereferenced by kubelet, kubelet already has access to pod secrets.

For the mounting secrets, which must live in the user's namespace

Some mount secrets are not appropriate for namespaced users to have access to.

since those will be dereferenced by kubelet, kubelet already has access to pod secrets.

as PV API objects are being refactored to allow specifying namespaces of secrets, the node authorizer automatically allows kubelets to access those secrets needed for PVs used by its pods

if this is going to be added to pods and pv's, we need the same split here as is being added for the other PV sources... restricting to the local namespace when used in a Pod, allowing specifying a secret namespace when used in a PV

Some mount secrets are not appropriate for namespaced users to have access to.

I think it's worth clarifying if this will still be true with CSI. Mount should only require end user credentials. Any credentials required to access the storage control plane must be injected directly into the CSI driver. Any credentials required to access (mount) the volume must be made available to the end user consuming the volume.

it's possible I'm confusing current phases of volume setup with new CSI-specific concepts. as long as it's possible for a CSI PV to be set up for a user's pod without them ever having PV-related secrets in their namespace, that's the main thing I care about

That will depend on what the secret is for. As spec'd, if a volume plugin requires credentials on mount, they must exist in the user's namespace. The assumption there is that a plugin would only require them, if it needs to validate that the user has access to that volume, and therefore it makes sense to require the secret to be in the user's namespace. For credentials that grant permissions to the entire control plane, those can be injected into the 3rd party volume plugin driver. For provisioner and attach permission, they would work as mentioned above.

I set up a meeting for us to sort this out tomorrow.

saad-ali · 2017-10-23T22:27:22Z

@erictune for the AttachError VolumeError and DetachError VolumeError

vladimirvivien · 2017-10-23T23:49:08Z

contributors/design-proposals/storage/container-storage-interface.md

+
+Kubelet (responsible for mount and unmount) will communicate with an external “CSI volume driver” running on the same host machine (whether containerized or not) via a Unix Domain Socket.
+
+The Unix Domain Socket will be registered with kubelet using the [Device Plugin Unix Domain Socket Registration](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-management/device-plugin.md#unix-socket) mechanism. This mechanism will need to be extended to support registration of CSI volume drivers.


Will the k8s CSI impl simply copy how Device Plugin communicate using Unix Domain Sock or are you envisioning acutually extending Device plugin to include k8s volumes plugins as devices ?

I'd like to extend the device plugin to support k8s volume plugins. But this is not a blocker for alpha.

vladimirvivien · 2017-10-24T01:38:22Z

contributors/design-proposals/storage/container-storage-interface.md

+      * The operation terminates with an error indicating a detach operation is in progress.
+      * The `VolumeAttachment.Status.IsAttached` value must not be trusted. The attach/detach controller has to wait until the object is deleted by the external-attacher before creating a new instance of the object.
+
+When the controller decides to detach a CSI volume, it will call the in-tree CSI volume plugin’s detach method. The in-tree CSI volume plugin’s attach method will do the following:


s/attach/detach in that paragraph

chakri-nelluri · 2017-10-24T15:55:58Z

contributors/design-proposals/storage/container-storage-interface.md

+
+To support CSI Compliant Volume plugins, a new in-tree CSI Volume plugin will be introduced in Kubernetes. This new volume plugin will be the mechanism by which Kubernetes users (application developers and cluster admins) interact with external CSI volume drivers.
+
+The mount/unmount calls for the new in-tree CSI volume plugin will directly invoke `NodePublishVolume` and `NodeUnpublishVolume` CSI RPCs through a unix domain socket on the node machine.


MountDevice/UnmountDevice calls?

Mount/Unmount should be triggered by the Setup() and TearDown on the VolumePlugin Impl of the plugin.

Changed to SetUp and TearDown to be more precise.

In one of the previous calls we wanted "NodePublish/Unpublish" to actually do "MountDevice & UnmountDevice" so that we call mount only once per volume per node. @saad-ali would know the final conclusion on it.

Yep, tracking that in container-storage-interface/spec#119

chakri-nelluri · 2017-10-24T15:56:51Z

contributors/design-proposals/storage/container-storage-interface.md

+
+Provision/delete and attach/detach must be handled by some external component that monitors the Kubernetes API on behalf of a CSI volume driver and invokes the appropriate CSI RPCs against it.
+
+To simplify integration, the Kubernetes team will offer a container that captures all the Kubernetes specific logic and acts as a bridge between third-party containerized CSI volume driver and Kubernetes (each deployment of a volume plugin would have it’s own “bridge”).


Isn't this more of an adapter pattern than a bridge pattern? Use Adapter instead of bridge.

chakri-nelluri · 2017-10-24T16:07:32Z

contributors/design-proposals/storage/container-storage-interface.md

+
+The Unix Domain Socket will be registered with kubelet using the [Device Plugin Unix Domain Socket Registration](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-management/device-plugin.md#unix-socket) mechanism. This mechanism will need to be extended to support registration of CSI volume drivers.
+
+Upon initialization of the external “CSI volume driver”, some external component should call `GetNodeId` to get the mapping from Kubernetes NodeId to CSI driver NodeId. It should then add the CSI driver NodeId as a label to the Kubernetes Node API object. The key of the label should be `csi.volume.kubernetes.io/<CsiDriverName>/nodeID`. This will enable the component that will issue `ControllerPublishVolume` calls to use the label as a mapping from cluster node ID to storage node ID.


May be this should be part of CSI discussion. This NodeID is typically property of the Node, but not of the storage subsystem/plugin. Ideally, all NodeID's from all components & plugins should be the same. If not, this will be very fragmented to use and impossible to maintain.

Indeed, that belongs to CSI to resolve. Right now CSI allows for different IDs of the same machine on Kubernetes side and storage side.

Thinking more this would be needed for storage systems like FC, which do not have a network identity. WWPN returned by NodeID can be used by controller for LUN Masking.

Opened container-storage-interface/spec#141 based on our discussion https://youtu.be/7SK1VZk9jro?t=23m37s

chakri-nelluri · 2017-10-24T16:08:48Z

contributors/design-proposals/storage/container-storage-interface.md

+
+Upon initialization of the external “CSI volume driver”, some external component should call `GetNodeId` to get the mapping from Kubernetes NodeId to CSI driver NodeId. It should then add the CSI driver NodeId as a label to the Kubernetes Node API object. The key of the label should be `csi.volume.kubernetes.io/<CsiDriverName>/nodeID`. This will enable the component that will issue `ControllerPublishVolume` calls to use the label as a mapping from cluster node ID to storage node ID.
+
+The Kubernetes team will provide a helper container that can manage the UDS registration and NodeId initialization (see “Recommended Mechanism for Deployment” below for details).


UDS -> Unix Domain Socket or please add it in brackets before it is mentioned.

chakri-nelluri · 2017-10-24T16:13:27Z

contributors/design-proposals/storage/container-storage-interface.md

+
+In short, to dynamically provision a new CSI volume, a cluster admin would create a `StorageClass` with the provisioner corresponding to the name of the external provisioner handling provisioning requests on behalf of the CSI volume driver.
+
+To provision a new CSI volume, an end user would create a `PersistentVolumeClaim` object referencing this `StorageClass`. The external provisioner will react to the creation of the PVC and issue the `CreateVolume` call against the CSI volume driver to provision the volume. The `CreateVolume` name will be auto-generated as it is for other dynamically provisioned volumes. The `CreateVolume` capacity will be take from the `PersistentVolumeClaim` object. The `CreateVolume` parameters will be passed through from the `StorageClass` parameters (opaque to Kubernetes). Once the operation completes successfully, the external provisioner creates a `PersistentVolume` object to represent the volume using the information returned in the `CreateVolume` call. The `PersistentVolume` object is bound to the `PersistentVolumeClaim` and available for use.


"using the information passed in the CreateVolume call? Instead of returned in the CreateVolume Call?

I think that "returned in the CreateVolume call" is correct - the provisioner parses CreateVolumeResponse returned from CreateVolume call and puts this info (namely VolumeHandle) into the created PV.

Oh got it. Shall we rephrase it as "using the information returned in the CreateVolume response" to avoid confusion?

chakri-nelluri · 2017-10-24T16:14:56Z

contributors/design-proposals/storage/container-storage-interface.md

+
+To provision a new CSI volume, an end user would create a `PersistentVolumeClaim` object referencing this `StorageClass`. The external provisioner will react to the creation of the PVC and issue the `CreateVolume` call against the CSI volume driver to provision the volume. The `CreateVolume` name will be auto-generated as it is for other dynamically provisioned volumes. The `CreateVolume` capacity will be take from the `PersistentVolumeClaim` object. The `CreateVolume` parameters will be passed through from the `StorageClass` parameters (opaque to Kubernetes). Once the operation completes successfully, the external provisioner creates a `PersistentVolume` object to represent the volume using the information returned in the `CreateVolume` call. The `PersistentVolume` object is bound to the `PersistentVolumeClaim` and available for use.
+
+To delete a CSI volume, an end user would delete the corresponding `PersistentVolumeClaim` object. The external provisioner will react to the deletion of the PVC and issue the `DeleteVolume` call against the CSI volume driver commands to delete the volume. It will then delete the `PersistentVolume` object.


Mention based on reclaim policy?

chakri-nelluri · 2017-10-24T16:28:10Z

contributors/design-proposals/storage/container-storage-interface.md

+
+Notably, `ProvisionableVolumePlugin` and `DeletableVolumePlugin` are not implemented because provisioning and deleting for CSI volumes is handled by an external provisioner. 
+
+#### Mount and Unmount


MountDevice/unmountDevice?

No, it's filesystem mount and unmount. We don't do block devices at this point.

container-storage-interface/spec#119

chakri-nelluri · 2017-10-24T16:31:13Z

contributors/design-proposals/storage/container-storage-interface.md

+When the controller decides to attach a CSI volume, it will call the in-tree CSI volume plugin’s attach method. The in-tree CSI volume plugin’s attach method will do the following:
+
+1. Create a new `VolumeAttachment` object (defined in the “Communication Channels” section) to attach the volume.
+    * The name of the of the `VolumeAttachment` object will be the `<UniqueVolumeName>-<NodeId>`.


Do we have any other naming patterns already used by K8S components?

None that would apply here. Usually it's user who chooses object name. Kubernetes adds random suffix or sequence number to Pod.Name created from a ReplicaSet or StatefulSet. We need the name of VolumeAttachment to be fixed to catch collisions when something wants to attach the same volume to the same node twice - VolumeAttachment objects will have the same name and thus only one can be successfully created.

Are UniqueVolumeName or NodeId able to contain a - character?

It doesn't matter if they do. The - character is not used as a delimiter. The generated name is simply used as a deterministic way of ensuring that attach operations are not triggered for the same volume more then once.

It could matter if one unique volume name could be a prefix of another.

volumename foo and namespace bar-baz
volumename foo-bar and namespace baz

Those would conflict and mean that one volume attach would not be able to be satisfied

erictune · 2017-10-24T23:10:07Z

use of Status looks good.

vladimirvivien · 2017-10-25T03:39:37Z

Needs updates regarding CSIVolumeSource and the secret referencs. Other than that LGTM.

edisonxiang · 2017-10-25T08:21:49Z

contributors/design-proposals/storage/container-storage-interface.md

+
+### Links
+
+* Container Storage Interface (CSI) Spec


It's better to add CSI Spec Link (https://github.com/container-storage-interface/spec/blob/master/spec.md)

edisonxiang · 2017-10-25T08:28:31Z

contributors/design-proposals/storage/container-storage-interface.md

+To deploy a containerized third-party CSI volume driver, it is recommended that storage vendors:
+
+  * Create a “CSI volume driver” container that implements the volume plugin behavior and exposes a gRPC interface via a unix domain socket, as defined in the CSI spec (including Controller, Node, and Identity services).
+  * The Kubernetes team will provide helper containers (external-attacher, external-provisioner, kubelet registration) which will assist the “CSI volume driver” container in interacting with the Kubernetes system.


Does "Kubernetes CSI Helper Container" in the diagram only include "kubelet registration"?

Renamed "kubelet registration" to "Kubernetes CSI Helper", which is then described below as "responsible for registering the unix domain socket with kubelet and NodeId initialization"

edisonxiang · 2017-10-25T08:35:11Z

contributors/design-proposals/storage/container-storage-interface.md

+
+The in-tree volume plugin’s mount and unmount methods will trigger the `NodePublishVolume` and `NodeUnpublishVolume` CSI calls via Unix Domain Socket. Kubernetes will generate a unique `target_path` (unique per pod per volume) to pass via `NodePublishVolume` for the CSI plugin to mount the volume. Upon successful completion of the `NodeUnpublishVolume` call (once volume unmount has been verified), Kubernetes will delete the directory.
+
+The Kubernetes volume sub-system does not currently support block volumes (only file), so for alpha, the Kubernetes CSI volume plugin will only support file.


I am a little confused with the definitions between "raw block volume" and "block volume" in Kubernetes. So does it mean that in the alpha version Kubernetes will not support the block volume drivers like EBS and Ceph RBD?

I agree it's confusing, but in Kubernetes context "raw block volume" and "block volume" usually means the same - access to raw device in pods - and we won't support it in Alpha implementation of CSI plugin.

On the other hand, we will support filesystems on block devices as we do now, i.e. one can implement a CSI volume driver that attaches a block device (say EBS) and mounts it and a pod can use it.

@jsafrane Thank your reply. It is very useful for me.

edisonxiang

Great work, saad, Just list some questions and waiting for your reply. Thank you!

jsafrane · 2017-10-25T11:24:46Z

Pushed some fixes there on behalf of Saad. No updates regarding handling of Secrets yet.

jsafrane · 2017-10-25T12:43:56Z

Updated with result of discussion with @liggitt about secrets, I hope I got it right.

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Kubernetes CSI - Persistent Volume Source Type **What this PR does / why we need it**: This PR is to track the addition of new API type `CSIPersistentVolumeSource` that will be used as PersistentVolume for storage sources managed by CSI drivers. **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: xref kubernetes/enhancements#178 **Special notes for your reviewer**: - Implements API `PersistentVolume` type `CSIPersistentVolumeSource` - Part of implementation for kubernetes/enhancements#178 - Designed at kubernetes/community#1258 Other CSI Volume Plugin PRs: - Plugin Mounter/Unmounter kubernetes/kubernetes#54529 - Plugin Attacher/Detacher kubernetes/kubernetes#55809 **Release note**: ```release-note NONE ``` Kubernetes-commit: 928c85fc997a49e607a94960f1680d9a87b8934f

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Introduce new `VolumeAttachment` API Object **What this PR does / why we need it**: Introduce a new `VolumeAttachment` API Object. This object will be used by the CSI volume plugin to enable external attachers (see design [here](kubernetes/community#1258). In the future, existing volume plugins can be refactored to use this object as well. **Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: Part of issue kubernetes/enhancements#178 **Special notes for your reviewer**: None **Release note**: ```release-note NONE ``` Kubernetes-commit: ebe8ea73fd1a961779242dfbb629befa153e96fc

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Kubernetes CSI - Persistent Volume Source Type **What this PR does / why we need it**: This PR is to track the addition of new API type `CSIPersistentVolumeSource` that will be used as PersistentVolume for storage sources managed by CSI drivers. **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: xref kubernetes/enhancements#178 **Special notes for your reviewer**: - Implements API `PersistentVolume` type `CSIPersistentVolumeSource` - Part of implementation for kubernetes/enhancements#178 - Designed at kubernetes/community#1258 Other CSI Volume Plugin PRs: - Plugin Mounter/Unmounter kubernetes/kubernetes#54529 - Plugin Attacher/Detacher kubernetes/kubernetes#55809 **Release note**: ```release-note NONE ``` Kubernetes-commit: 928c85fc997a49e607a94960f1680d9a87b8934f

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Introduce new `VolumeAttachment` API Object **What this PR does / why we need it**: Introduce a new `VolumeAttachment` API Object. This object will be used by the CSI volume plugin to enable external attachers (see design [here](kubernetes/community#1258). In the future, existing volume plugins can be refactored to use this object as well. **Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: Part of issue kubernetes/enhancements#178 **Special notes for your reviewer**: None **Release note**: ```release-note NONE ``` Kubernetes-commit: ebe8ea73fd1a961779242dfbb629befa153e96fc

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Kubernetes CSI - Persistent Volume Source Type **What this PR does / why we need it**: This PR is to track the addition of new API type `CSIPersistentVolumeSource` that will be used as PersistentVolume for storage sources managed by CSI drivers. **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: xref kubernetes/enhancements#178 **Special notes for your reviewer**: - Implements API `PersistentVolume` type `CSIPersistentVolumeSource` - Part of implementation for kubernetes/enhancements#178 - Designed at kubernetes/community#1258 Other CSI Volume Plugin PRs: - Plugin Mounter/Unmounter kubernetes/kubernetes#54529 - Plugin Attacher/Detacher kubernetes/kubernetes#55809 **Release note**: ```release-note NONE ``` Kubernetes-commit: 928c85fc997a49e607a94960f1680d9a87b8934f

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Introduce new `VolumeAttachment` API Object **What this PR does / why we need it**: Introduce a new `VolumeAttachment` API Object. This object will be used by the CSI volume plugin to enable external attachers (see design [here](kubernetes/community#1258). In the future, existing volume plugins can be refactored to use this object as well. **Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: Part of issue kubernetes/enhancements#178 **Special notes for your reviewer**: None **Release note**: ```release-note NONE ``` Kubernetes-commit: ebe8ea73fd1a961779242dfbb629befa153e96fc

Automatic merge from submit-queue (batch tested with PRs 57266, 58187, 58186, 46245, 56509). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. e2e: CSI Tests **What this PR does / why we need it**: This e2e test tests the CSI external attacher with a mock CSI plugin driver. **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: kubernetes/enhancements#178 **Special notes for your reviewer**: * Tests features in kubernetes/enhancements#178 * Tests implementation of kubernetes/community#1258 * Tests VolumeAttachment Object: #54463 **Release note**: ```release-note NONE ```

hairesis · 2019-07-12T09:23:55Z

contributors/design-proposals/storage/container-storage-interface.md

+
+##### Provisioning and Deleting
+
+Provisioning and deletion operations are handled using the existing [external provisioner mechanism](https://github.com/kubernetes-incubator/external-storage/tree/master/docs), where the external component watching the Kubernetes API on behalf of the external CSI volume driver will act as an external provisioner.


broken link

Automatic merge from submit-queue. CSI Volume Plugins in Kubernetes Design Doc Design documenting all the requirements for enabling an a CSI compliant volume plugin (a CSI volume driver) in Kubernetes. This design was drafted [here](https://docs.google.com/document/d/10GDyPWbFE5tQunKMlTXbcWysUttMFhBFJRX8ntaS_4Y/edit?usp=sharing), and is now ready for publishing. This feature is targeted as alpha in v1.9.

saad-ali added the sig/storage Categorizes an issue or PR as relevant to SIG Storage. label Oct 23, 2017

saad-ali assigned childsb, jsafrane and chakri-nelluri Oct 23, 2017

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Oct 23, 2017

saad-ali assigned thockin Oct 23, 2017

jsafrane reviewed Oct 23, 2017

View reviewed changes

saad-ali force-pushed the csi-design-doc branch from cb65248 to ff5f7b1 Compare October 23, 2017 18:18

saad-ali mentioned this pull request Oct 23, 2017

Out-of-tree CSI Volume Plugins kubernetes/enhancements#178

Closed

saad-ali force-pushed the csi-design-doc branch from ff5f7b1 to e2a057e Compare October 23, 2017 22:08

saad-ali assigned erictune Oct 23, 2017

vladimirvivien reviewed Oct 23, 2017

View reviewed changes

vladimirvivien reviewed Oct 24, 2017

View reviewed changes

saad-ali force-pushed the csi-design-doc branch from e2a057e to 534beed Compare October 24, 2017 02:44

saad-ali mentioned this pull request Oct 24, 2017

Introduce new VolumeAttachment API Object kubernetes/kubernetes#54463

Merged

chakri-nelluri reviewed Oct 24, 2017

View reviewed changes

vladimirvivien mentioned this pull request Oct 25, 2017

Kubernetes CSI - in-tree Plugin Implementation kubernetes/kubernetes#54529

Merged

edisonxiang reviewed Oct 25, 2017

View reviewed changes

jsafrane force-pushed the csi-design-doc branch 2 times, most recently from ab357e5 to 3215005 Compare October 25, 2017 12:42

cblecker deleted the csi-design-doc branch November 30, 2017 19:44

hairesis reviewed Jul 12, 2019

View reviewed changes


		More specifically, an external “attacher” must watch the Kubernetes API on behalf of the external CSI volume driver to handle attach/detach requests.

		Upon initialization the `external-attacher` should call `GetNodeId` for each node in the cluster and create a mapping from cluster node ID to storage node ID. When issuing `ControllerPublishVolume` calls it should use the mapping.


		The external-attacher may implement it’s own error recovery strategy, and retry as long as conditions specified for attachment above are valid. It is strongly recommended that the external-attacher implement an exponential backoff strategy for retries.

		The detach operation will be triggered by the deletion of the `VolumeAttachment` Kubernetes API objects. The `VolumeAttachment` object will have an `external-attacher` finalizer which will wait for confirmation from the external-attacher before deleting the object.


		Kubelet (responsible for mount and unmount) will communicate with an external “CSI volume driver” running on the same host machine (whether containerized or not) via a Unix Domain Socket.

		The Unix Domain Socket will be registered with kubelet using the [Device Plugin Unix Domain Socket Registration](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-management/device-plugin.md#unix-socket) mechanism. This mechanism will need to be extended to support registration of CSI volume drivers.


		To support CSI Compliant Volume plugins, a new in-tree CSI Volume plugin will be introduced in Kubernetes. This new volume plugin will be the mechanism by which Kubernetes users (application developers and cluster admins) interact with external CSI volume drivers.

		The mount/unmount calls for the new in-tree CSI volume plugin will directly invoke `NodePublishVolume` and `NodeUnpublishVolume` CSI RPCs through a unix domain socket on the node machine.


		Provision/delete and attach/detach must be handled by some external component that monitors the Kubernetes API on behalf of a CSI volume driver and invokes the appropriate CSI RPCs against it.

		To simplify integration, the Kubernetes team will offer a container that captures all the Kubernetes specific logic and acts as a bridge between third-party containerized CSI volume driver and Kubernetes (each deployment of a volume plugin would have it’s own “bridge”).


		The Unix Domain Socket will be registered with kubelet using the [Device Plugin Unix Domain Socket Registration](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-management/device-plugin.md#unix-socket) mechanism. This mechanism will need to be extended to support registration of CSI volume drivers.

		Upon initialization of the external “CSI volume driver”, some external component should call `GetNodeId` to get the mapping from Kubernetes NodeId to CSI driver NodeId. It should then add the CSI driver NodeId as a label to the Kubernetes Node API object. The key of the label should be `csi.volume.kubernetes.io/<CsiDriverName>/nodeID`. This will enable the component that will issue `ControllerPublishVolume` calls to use the label as a mapping from cluster node ID to storage node ID.


		Upon initialization of the external “CSI volume driver”, some external component should call `GetNodeId` to get the mapping from Kubernetes NodeId to CSI driver NodeId. It should then add the CSI driver NodeId as a label to the Kubernetes Node API object. The key of the label should be `csi.volume.kubernetes.io/<CsiDriverName>/nodeID`. This will enable the component that will issue `ControllerPublishVolume` calls to use the label as a mapping from cluster node ID to storage node ID.

		The Kubernetes team will provide a helper container that can manage the UDS registration and NodeId initialization (see “Recommended Mechanism for Deployment” below for details).


		In short, to dynamically provision a new CSI volume, a cluster admin would create a `StorageClass` with the provisioner corresponding to the name of the external provisioner handling provisioning requests on behalf of the CSI volume driver.

		To provision a new CSI volume, an end user would create a `PersistentVolumeClaim` object referencing this `StorageClass`. The external provisioner will react to the creation of the PVC and issue the `CreateVolume` call against the CSI volume driver to provision the volume. The `CreateVolume` name will be auto-generated as it is for other dynamically provisioned volumes. The `CreateVolume` capacity will be take from the `PersistentVolumeClaim` object. The `CreateVolume` parameters will be passed through from the `StorageClass` parameters (opaque to Kubernetes). Once the operation completes successfully, the external provisioner creates a `PersistentVolume` object to represent the volume using the information returned in the `CreateVolume` call. The `PersistentVolume` object is bound to the `PersistentVolumeClaim` and available for use.


		To provision a new CSI volume, an end user would create a `PersistentVolumeClaim` object referencing this `StorageClass`. The external provisioner will react to the creation of the PVC and issue the `CreateVolume` call against the CSI volume driver to provision the volume. The `CreateVolume` name will be auto-generated as it is for other dynamically provisioned volumes. The `CreateVolume` capacity will be take from the `PersistentVolumeClaim` object. The `CreateVolume` parameters will be passed through from the `StorageClass` parameters (opaque to Kubernetes). Once the operation completes successfully, the external provisioner creates a `PersistentVolume` object to represent the volume using the information returned in the `CreateVolume` call. The `PersistentVolume` object is bound to the `PersistentVolumeClaim` and available for use.

		To delete a CSI volume, an end user would delete the corresponding `PersistentVolumeClaim` object. The external provisioner will react to the deletion of the PVC and issue the `DeleteVolume` call against the CSI volume driver commands to delete the volume. It will then delete the `PersistentVolume` object.


		Notably, `ProvisionableVolumePlugin` and `DeletableVolumePlugin` are not implemented because provisioning and deleting for CSI volumes is handled by an external provisioner.

		#### Mount and Unmount


		The in-tree volume plugin’s mount and unmount methods will trigger the `NodePublishVolume` and `NodeUnpublishVolume` CSI calls via Unix Domain Socket. Kubernetes will generate a unique `target_path` (unique per pod per volume) to pass via `NodePublishVolume` for the CSI plugin to mount the volume. Upon successful completion of the `NodeUnpublishVolume` call (once volume unmount has been verified), Kubernetes will delete the directory.

		The Kubernetes volume sub-system does not currently support block volumes (only file), so for alpha, the Kubernetes CSI volume plugin will only support file.


		##### Provisioning and Deleting

		Provisioning and deletion operations are handled using the existing [external provisioner mechanism](https://github.com/kubernetes-incubator/external-storage/tree/master/docs), where the external component watching the Kubernetes API on behalf of the external CSI volume driver will act as an external provisioner.

CSI Volume Plugins in Kubernetes Design Doc #1258

CSI Volume Plugins in Kubernetes Design Doc #1258

Conversation

saad-ali commented Oct 23, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saad-ali Oct 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saad-ali commented Oct 23, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erictune commented Oct 24, 2017

vladimirvivien commented Oct 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edisonxiang left a comment

Choose a reason for hiding this comment

jsafrane commented Oct 25, 2017

jsafrane commented Oct 25, 2017

Choose a reason for hiding this comment

saad-ali Oct 23, 2017 •

edited

Loading