Skip to content
forked from spdk/spdk-csi

CSI driver to bring SPDK to Kubernetes storage through NVMe-oF or iSCSI. Supports dynamic volume provisioning and enables Pods to use SPDK storage transparently.

License

Notifications You must be signed in to change notification settings

lijhnihaoa/spdk-csi

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

59 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

SPDK CSI

About

This repo contains SPDK CSI (Container Storage Interface) plugin for Kubernetes.

SPDK CSI plugin brings SPDK to Kubernetes. It provisions SPDK logical volumes on storage node dynamically and enables Pods to access SPDK storage backend through NVMe-oF or iSCSI.

Please see SPDK CSI Design Document for detailed introduction.

Supported platforms

This plugin conforms to CSI Spec v1.1.0. It is currently developed and tested only on Kubernetes.

This plugin supports x86_64 and Arm64 architectures.

Project status

Status: Beta

Prerequisites

SPDK-CSI is currently developed and tested with Go 1.14, Docker 19.03 and Kubernetes 1.19.3 on Ubuntu 18.04.

Minimal requirement: Go 1.12+(supports Go module), Docker 18.03+ and Kubernetes 1.13+(supports CSI spec 1.0).

Setup

Build

  • $ make all Build targets spdkcsi, lint, test.

  • $ make spdkcsi Build SPDK-CSI binary _out/spdkcsi.

  • $ make lint Lint code and scripts.

    • $ make golangci Install golangci-lint and perform various go code static checks.
    • $ make yamllint Lint yaml files if yamllint is installed. Requires yamllint 1.10+.
  • $ make test Verify go modules and run unit tests. Requires SPDK target and JsonRPC HTTP proxy running on localhost. See deploy/spdk/README for details.

  • $ make e2e-test Verify core features through Kubernetes end-to-end (e2e) test.

  • $ make image Build SPDK-CSI docker image.

Parameters

spdkcsi executable accepts several command line parameters.

Parameter Type Description Default
--controller - enable controller service -
--node - enable node service -
--endpoint string communicate with sidecars /tmp/spdkcsi.sock
--drivername string driver name csi.spdk.io
--nodeid string node id -

Usage

Example deployment files can be found in deploy/kubernetes directory.

File Name Usage
storageclass.yaml StorageClass of provisioner "csi.spdk.io"
controller.yaml StatefulSet running CSI Controller service
node.yaml DaemonSet running CSI Node service
controller-rbac.yaml Access control for CSI Controller service
node-rbac.yaml Access control for CSI Node service
config-map.yaml SPDK storage cluster configurations
secret.yaml SPDK storage cluster access tokens
snapshotclass.yaml SnapshotClass of provisioner "csi.spdk.io"

NOTE:

Below example is a simplest test system running in a single host or VM. No NVMe device is required, memory based bdev is used instead. docs/multi-node.md introduces how to deploy SPDKCSI on multiple nodes with NVMe devices.


Prepare SPDK storage node

Follow deploy/spdk/README to deploy SPDK storage service on localhost.

Deploy SPDKCSI services

  1. Launch Minikube test cluster
$ cd scripts
$ sudo ./minikube.sh up

# Create kubectl shortcut (assume kubectl version 1.19.3)
$ sudo ln -s /var/lib/minikube/binaries/v1.19.3/kubectl /usr/local/bin/kubectl

# Wait for Kubernetes ready
$ kubectl get pods --all-namespaces
NAMESPACE     NAME                          READY   STATUS    RESTARTS   AGE
kube-system   coredns-6955765f44-dlb88      1/1     Running   0          81s
......                                              ......
kube-system   kube-apiserver-spdkcsi-dev    1/1     Running   0          67s
......                                              ......
  1. Install snapshot controller and CRD
SNAPSHOT_VERSION="v3.0.3" ./scripts/install-snapshot.sh install

# Check status
$ kubectl get pod snapshot-controller-0
NAME                    READY   STATUS    RESTARTS   AGE
snapshot-controller-0   1/1     Running   0          6m14s
  1. Deploy SPDK-CSI services
$ cd deploy/kubernetes
$ ./deploy.sh

# Check status
$ kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
spdkcsi-controller-0   3/3     Running   0          3m16s
spdkcsi-node-lzvg5     2/2     Running   0          3m16s
  1. Deploy test pod
$ cd deploy/kubernetes
$ kubectl apply -f testpod.yaml

# Check status
$ kubectl get pv
NAME                       CAPACITY   ...    STORAGECLASS   REASON   AGE
persistentvolume/pvc-...   256Mi      ...    spdkcsi-sc              43s

$ kubectl get pvc
NAME                                ...   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/spdkcsi-pvc   ...   256Mi      RWO            spdkcsi-sc     44s

$ kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
spdkcsi-test           1/1     Running   0          1m31s

# Check attached spdk volume in test pod
$ kubectl exec spdkcsi-test mount | grep spdkcsi
/dev/disk/by-id/nvme-..._spdkcsi-sn on /spdkvol type ext4 (rw,relatime)
  1. Deploy PVC snapshot
  # Create snapshot of the bound PVC
  $ cd deploy/kubernetes
  $ kubectl apply -f snapshot.yaml

  # Get details about the snapshot
  $ kubectl get volumesnapshot spdk-snapshot
  NAME            READYTOUSE   SOURCEPVC   ... SNAPSHOTCLASS         AGE
  spdk-snapshot   false        spdkcsi-pvc ... csi-spdk-snapclass    29s

  # Get details about the volumesnapshotcontent
  kubectl get volumesnapshotcontent
  $ kubectl get volumesnapshotcontent
  NAME        ...   READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER        VOLUMESNAPSHOTCLASS   VOLUMESNAPSHOT   AGE
  snapcontent-...   true         268435456     Delete           csi.spdk.io   csi-spdk-snapclass    spdk-snapshot    29s

Teardown

  1. Delete PVC snapshot
cd deploy/kubernetes
kubectl delete -f snapshot.yaml
  1. Delete test pod
$ cd deploy/kubernetes
$ kubectl delete -f testpod.yaml
  1. Delete SPDK-CSI services
$ cd deploy/kubernetes
$ ./deploy.sh teardown
  1. Delete snapshot controller and CRD
SNAPSHOT_VERSION="v3.0.3" ./scripts/install-snapshot.sh cleanup
  1. Teardown Kubernetes test cluster
$ cd scripts
$ sudo ./minikube.sh clean

Communication and Contribution

Please join SPDK community for communication and contribution.

Project progress is tracked in Trello board.

About

CSI driver to bring SPDK to Kubernetes storage through NVMe-oF or iSCSI. Supports dynamic volume provisioning and enables Pods to use SPDK storage transparently.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Go 85.5%
  • Shell 11.6%
  • Makefile 2.1%
  • Other 0.8%