Merge pull request ceph#116 from ceph/devel

Sync devel branch with upstream
pkalever · Aug 4, 2022 · 7cf881f · 7cf881f
2 parents 3db764f + 83df1ea
commit 7cf881f
Show file tree

Hide file tree

Showing 146 changed files with 9,280 additions and 1,099 deletions.
diff --git a/README.md b/README.md
@@ -21,15 +21,15 @@ Card](https://goreportcard.com/badge/github.com/ceph/ceph-csi)](https://goreport
   - [Dev standup](#dev-standup)
   - [Contact](#contact)
 
-This repo contains Ceph
+This repo contains the Ceph
 [Container Storage Interface (CSI)](https://github.com/container-storage-interface/)
-driver for RBD, CephFS and kubernetes sidecar deployment yamls of provisioner,
-attacher, resizer, driver-registrar and snapshotter for supporting CSI functionalities.
+driver for RBD, CephFS and Kubernetes sidecar deployment YAMLs to support CSI
+functionality:  provisioner, attacher, resizer, driver-registrar and snapshotter.
 
 ## Overview
 
-Ceph CSI plugins implement an interface between CSI enabled Container Orchestrator
-(CO) and Ceph cluster. It allows dynamically provisioning Ceph volumes and
+Ceph CSI plugins implement an interface between a CSI-enabled Container Orchestrator
+(CO) and Ceph clusters. They enable dynamically provisioning Ceph volumes and
 attaching them to workloads.
 
 Independent CSI plugins are provided to support RBD and CephFS backed volumes,
@@ -38,7 +38,7 @@ Independent CSI plugins are provided to support RBD and CephFS backed volumes,
   [rbd doc](https://github.com/ceph/ceph-csi/blob/devel/docs/deploy-rbd.md) and
   for CephFS plugin configuration and deployment please
   refer [cephFS doc](https://github.com/ceph/ceph-csi/blob/devel/docs/deploy-cephfs.md).
-- For example usage of RBD and CephFS CSI plugins, see examples in `examples/`.
+- For example usage of the RBD and CephFS CSI plugins, see examples in `examples/`.
 - Stale resource cleanup, please refer [cleanup doc](docs/resource-cleanup.md).
 
 NOTE:
@@ -51,7 +51,7 @@ Status: **GA**
 
 ## Known to work CO platforms
 
-Ceph CSI drivers are currently developed and tested **exclusively** on Kubernetes
+Ceph CSI drivers are currently developed and tested **exclusively** in Kubernetes
 environments.
 
 | Ceph CSI Version | Container Orchestrator Name | Version Tested|
@@ -62,18 +62,17 @@ environments.
 | v3.5.0 | Kubernetes | v1.21, v1.22, v1.23|
 | v3.4.0 | Kubernetes | v1.20, v1.21, v1.22|
 
-There is work in progress to make this CO independent and thus
-support other orchestration environments (Nomad, Mesos..etc) in the future.
+There is work in progress to make this CO-independent and thus
+support other orchestration environments (Nomad, Mesos..etc).
 
 NOTE:
 
-The supported window of Ceph CSI versions  is known as "N.(x-1)":
+The supported window of Ceph CSI versions is "N.(x-1)":
 (N (Latest major release) . (x (Latest minor release) - 1)).
 
-For example, if Ceph CSI latest major version is `3.6.0` today, support is
+For example, if the Ceph CSI latest major version is `3.6.0` today, support is
 provided for the versions above `3.5.0`. If users are running an unsupported
-Ceph CSI version, they will be asked to upgrade when requesting support for the
-cluster.
+Ceph CSI version, they will be asked to upgrade when requesting support.
 
 ## Support Matrix
 

diff --git a/charts/ceph-csi-cephfs/README.md b/charts/ceph-csi-cephfs/README.md
@@ -86,6 +86,7 @@ charts and their default values.
 | `serviceAccounts.provisioner.name`             | The name of the provisioner ServiceAccount of provisioner to use. If not set and create is true, a name is generated using the fullname              | ""                                                 |
 | `csiConfig`                                    | Configuration for the CSI to connect to the cluster                                                                                                  | []                                                 |
 | `logLevel`                                     | Set logging level for csi containers. Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity.                          | `5`                                                |
+| `sidecarLogLevel`                              | Set logging level for csi sidecar containers. Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity.                  | `1`                                                |
 | `nodeplugin.name`                              | Specifies the nodeplugin name                                                                                                                        | `nodeplugin`                                       |
 | `nodeplugin.updateStrategy`                    | Specifies the update Strategy. If you are using ceph-fuse client set this value to OnDelete                                                          | `RollingUpdate`                                    |
 | `nodeplugin.priorityClassName`                 | Set user created priorityclassName for csi plugin pods. default is system-node-critical which is highest priority                                    | `system-node-critical`                             |
@@ -105,6 +106,8 @@ charts and their default values.
 | `provisioner.name`                             | Specifies the name of provisioner                                                                                                                    | `provisioner`                                      |
 | `provisioner.replicaCount`                     | Specifies the replicaCount                                                                                                                           | `3`                                                |
 | `provisioner.timeout`                          | GRPC timeout for waiting for creation or deletion of a volume                                                                                        | `60s`                                              |
+| `provisioner.clustername`                      | Cluster name to set on the subvolume                                                                                                                 | ""                                                 |
+| `provisioner.setmetadata`                      | Set metadata on volume                                                                                                                               | `true`                                             |
 | `provisioner.priorityClassName`                | Set user created priorityclassName for csi provisioner pods. Default is `system-cluster-critical` which is less priority than `system-node-critical` | `system-cluster-critical`                          |
 | `provisioner.enableHostNetwork`                | Specifies whether hostNetwork is enabled for provisioner pod.                                                                                        | `false`                                            |
 | `provisioner.profiling.enabled`                | Specifies whether profiling should be enabled                                                                                                        | `false`                                            |

diff --git a/charts/ceph-csi-cephfs/templates/provisioner-deployment.yaml b/charts/ceph-csi-cephfs/templates/provisioner-deployment.yaml
@@ -58,7 +58,7 @@ spec:
           imagePullPolicy: {{ .Values.provisioner.provisioner.image.pullPolicy }}
           args:
             - "--csi-address=$(ADDRESS)"
-            - "--v={{ .Values.logLevel }}"
+            - "--v={{ .Values.sidecarLogLevel }}"
             - "--timeout={{ .Values.provisioner.timeout }}"
             - "--leader-election=true"
             - "--retry-interval-start=500ms"
@@ -78,9 +78,10 @@ spec:
           imagePullPolicy: {{ .Values.provisioner.snapshotter.image.pullPolicy }}
           args:
             - "--csi-address=$(ADDRESS)"
-            - "--v={{ .Values.logLevel }}"
+            - "--v={{ .Values.sidecarLogLevel }}"
             - "--timeout={{ .Values.provisioner.timeout }}"
             - "--leader-election=true"
+            - "--extra-create-metadata=true"
           env:
             - name: ADDRESS
               value: "unix:///csi/{{ .Values.provisionerSocketFile }}"
@@ -94,7 +95,7 @@ spec:
           image: "{{ .Values.provisioner.resizer.image.repository }}:{{ .Values.provisioner.resizer.image.tag }}"
           imagePullPolicy: {{ .Values.provisioner.resizer.image.pullPolicy }}
           args:
-            - "--v={{ .Values.logLevel }}"
+            - "--v={{ .Values.sidecarLogLevel }}"
             - "--csi-address=$(ADDRESS)"
             - "--timeout={{ .Values.provisioner.timeout }}"
             - "--leader-election"
@@ -124,6 +125,10 @@ spec:
 {{- if .Values.provisioner.profiling.enabled }}
             - "--enableprofiling={{ .Values.provisioner.profiling.enabled }}"
 {{- end }}
+{{- if .Values.provisioner.clustername }}
+            - "--clustername={{ .Values.provisioner.clustername }}"
+{{- end }}
+            - "--setmetadata={{ .Values.provisioner.setmetadata }}"
           env:
             - name: POD_IP
               valueFrom:

diff --git a/charts/ceph-csi-cephfs/values.yaml b/charts/ceph-csi-cephfs/values.yaml
@@ -33,7 +33,10 @@ csiConfig: []
 # Set logging level for csi containers.
 # Supported values from 0 to 5. 0 for general useful logs,
 # 5 for trace level verbosity.
+# logLevel is the variable for CSI driver containers's log level
 logLevel: 5
+# sidecarLogLevel is the variable for Kubernetes sidecar container's log level
+sidecarLogLevel: 1
 
 nodeplugin:
   name: nodeplugin
@@ -125,6 +128,8 @@ provisioner:
       maxUnavailable: 50%
   # Timeout for waiting for creation or deletion of a volume
   timeout: 60s
+  # cluster name to set on the subvolume
+  # clustername: "k8s-cluster-1"
 
   # set user created priorityclassName for csi provisioner pods. default is
   # system-cluster-critical which is less priority than system-node-critical
@@ -175,6 +180,9 @@ provisioner:
       pullPolicy: IfNotPresent
     resources: {}
 
+  # set metadata on volume
+  setmetadata: true
+
   resizer:
     name: resizer
     enabled: true

diff --git a/charts/ceph-csi-rbd/README.md b/charts/ceph-csi-rbd/README.md
@@ -88,6 +88,7 @@ charts and their default values.
 | `csiMapping`                                   | Configuration details of clusterID,PoolID,FscID mapping                                                                                              | []                                                 |
 | `encryptionKMSConfig`                          | Configuration for the encryption KMS                                                                                                                 | `{}`                                               |
 | `logLevel`                                     | Set logging level for csi containers. Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity.                          | `5`                                                |
+| `sidecarLogLevel`                              | Set logging level for csi sidecar containers. Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity.                  | `1`                                                |
 | `nodeplugin.name`                              | Specifies the nodeplugins name                                                                                                                       | `nodeplugin`                                       |
 | `nodeplugin.updateStrategy`                    | Specifies the update Strategy. If you are using ceph-fuse client set this value to OnDelete                                                          | `RollingUpdate`                                    |
 | `nodeplugin.priorityClassName`                 | Set user created priorityclassName for csi plugin pods. default is system-node-critical which is highest priority                                    | `system-node-critical`                             |

diff --git a/charts/ceph-csi-rbd/templates/provisioner-deployment.yaml b/charts/ceph-csi-rbd/templates/provisioner-deployment.yaml
@@ -58,7 +58,7 @@ spec:
           imagePullPolicy: {{ .Values.provisioner.provisioner.image.pullPolicy }}
           args:
             - "--csi-address=$(ADDRESS)"
-            - "--v={{ .Values.logLevel }}"
+            - "--v={{ .Values.sidecarLogLevel }}"
             - "--timeout={{ .Values.provisioner.timeout }}"
             - "--leader-election=true"
             - "--retry-interval-start=500ms"
@@ -82,7 +82,7 @@ spec:
           image: "{{ .Values.provisioner.resizer.image.repository }}:{{ .Values.provisioner.resizer.image.tag }}"
           imagePullPolicy: {{ .Values.provisioner.resizer.image.pullPolicy }}
           args:
-            - "--v={{ .Values.logLevel }}"
+            - "--v={{ .Values.sidecarLogLevel }}"
             - "--csi-address=$(ADDRESS)"
             - "--timeout={{ .Values.provisioner.timeout }}"
             - "--leader-election"
@@ -103,7 +103,7 @@ spec:
           imagePullPolicy: {{ .Values.provisioner.snapshotter.image.pullPolicy }}
           args:
             - "--csi-address=$(ADDRESS)"
-            - "--v={{ .Values.logLevel }}"
+            - "--v={{ .Values.sidecarLogLevel }}"
             - "--timeout={{ .Values.provisioner.timeout }}"
             - "--leader-election=true"
             - "--extra-create-metadata=true"
@@ -120,7 +120,7 @@ spec:
           image: "{{ .Values.provisioner.attacher.image.repository }}:{{ .Values.provisioner.attacher.image.tag }}"
           imagePullPolicy: {{ .Values.provisioner.attacher.image.pullPolicy }}
           args:
-            - "--v={{ .Values.logLevel }}"
+            - "--v={{ .Values.sidecarLogLevel }}"
             - "--csi-address=$(ADDRESS)"
             - "--leader-election=true"
             - "--retry-interval-start=500ms"

diff --git a/charts/ceph-csi-rbd/values.yaml b/charts/ceph-csi-rbd/values.yaml
@@ -56,7 +56,10 @@ encryptionKMSConfig: {}
 # Set logging level for csi containers.
 # Supported values from 0 to 5. 0 for general useful logs,
 # 5 for trace level verbosity.
+# logLevel is the variable for CSI driver containers's log level
 logLevel: 5
+# sidecarLogLevel is the variable for Kubernetes sidecar container's log level
+sidecarLogLevel: 1
 
 nodeplugin:
   name: nodeplugin

diff --git a/deploy/cephfs/kubernetes/csi-cephfsplugin-provisioner.yaml b/deploy/cephfs/kubernetes/csi-cephfsplugin-provisioner.yaml
@@ -46,7 +46,7 @@ spec:
           image: registry.k8s.io/sig-storage/csi-provisioner:v3.2.1
           args:
             - "--csi-address=$(ADDRESS)"
-            - "--v=5"
+            - "--v=1"
             - "--timeout=150s"
             - "--leader-election=true"
             - "--retry-interval-start=500ms"
@@ -65,7 +65,7 @@ spec:
           image: registry.k8s.io/sig-storage/csi-resizer:v1.5.0
           args:
             - "--csi-address=$(ADDRESS)"
-            - "--v=5"
+            - "--v=1"
             - "--timeout=150s"
             - "--leader-election"
             - "--retry-interval-start=500ms"
@@ -82,9 +82,10 @@ spec:
           image: registry.k8s.io/sig-storage/csi-snapshotter:v6.0.1
           args:
             - "--csi-address=$(ADDRESS)"
-            - "--v=5"
+            - "--v=1"
             - "--timeout=150s"
             - "--leader-election=true"
+            - "--extra-create-metadata=true"
           env:
             - name: ADDRESS
               value: unix:///csi/csi-provisioner.sock
@@ -104,6 +105,7 @@ spec:
             - "--drivername=cephfs.csi.ceph.com"
             - "--pidlimit=-1"
             - "--enableprofiling=false"
+            - "--setmetadata=true"
           env:
             - name: POD_IP
               valueFrom:

diff --git a/deploy/cephfs/kubernetes/csi-cephfsplugin.yaml b/deploy/cephfs/kubernetes/csi-cephfsplugin.yaml
@@ -29,7 +29,7 @@ spec:
             allowPrivilegeEscalation: true
           image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1
           args:
-            - "--v=5"
+            - "--v=1"
             - "--csi-address=/csi/csi.sock"
             - "--kubelet-registration-path=/var/lib/kubelet/plugins/cephfs.csi.ceph.com/csi.sock"
           env:

diff --git a/deploy/nfs/kubernetes/csi-nfsplugin-provisioner.yaml b/deploy/nfs/kubernetes/csi-nfsplugin-provisioner.yaml
@@ -43,13 +43,11 @@ spec:
           image: registry.k8s.io/sig-storage/csi-provisioner:v3.2.1
           args:
             - "--csi-address=$(ADDRESS)"
-            - "--v=5"
+            - "--v=1"
             - "--timeout=150s"
             - "--leader-election=true"
             - "--retry-interval-start=500ms"
-            # TODO: enable the feature-gate once error has been handled in nfs.
-            # Issue #3230
-            # - "--feature-gates=HonorPVReclaimPolicy=true"
+            - "--feature-gates=HonorPVReclaimPolicy=true"
             - "--prevent-volume-mode-conversion=true"
           env:
             - name: ADDRESS
@@ -62,7 +60,7 @@ spec:
           image: registry.k8s.io/sig-storage/csi-resizer:v1.5.0
           args:
             - "--csi-address=$(ADDRESS)"
-            - "--v=5"
+            - "--v=1"
             - "--timeout=150s"
             - "--leader-election"
             - "--retry-interval-start=500ms"
@@ -78,7 +76,7 @@ spec:
           image: registry.k8s.io/sig-storage/csi-snapshotter:v6.0.1
           args:
             - "--csi-address=$(ADDRESS)"
-            - "--v=5"
+            - "--v=1"
             - "--timeout=150s"
             - "--leader-election=true"
           env:

diff --git a/deploy/nfs/kubernetes/csi-nfsplugin.yaml b/deploy/nfs/kubernetes/csi-nfsplugin.yaml
@@ -33,7 +33,7 @@ spec:
             - mountPath: /csi
               name: socket-dir
         - args:
-            - --v=5
+            - --v=1
             - --csi-address=/csi/csi.sock
             - --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
           env:
@@ -74,7 +74,7 @@ spec:
             - mountPath: /registration
               name: registration-dir
         - args:
-            - -v=5
+            - -v=1
             - --drivername=nfs.csi.ceph.com
             - --nodeid=$(NODE_ID)
             - --endpoint=$(CSI_ENDPOINT)

diff --git a/deploy/nfs/kubernetes/csi-provisioner-rbac.yaml b/deploy/nfs/kubernetes/csi-provisioner-rbac.yaml
@@ -20,7 +20,7 @@ rules:
     verbs: ["list", "watch", "create", "update", "patch"]
   - apiGroups: [""]
     resources: ["persistentvolumes"]
-    verbs: ["get", "list", "watch", "create", "delete", "patch"]
+    verbs: ["get", "list", "watch", "create", "update", "delete", "patch"]
   - apiGroups: [""]
     resources: ["persistentvolumeclaims"]
     verbs: ["get", "list", "watch", "update"]

diff --git a/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml b/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
@@ -50,7 +50,7 @@ spec:
           image: k8s.gcr.io/sig-storage/csi-provisioner:v3.2.1
           args:
             - "--csi-address=$(ADDRESS)"
-            - "--v=5"
+            - "--v=1"
             - "--timeout=150s"
             - "--retry-interval-start=500ms"
             - "--leader-election=true"
@@ -72,7 +72,7 @@ spec:
           image: registry.k8s.io/sig-storage/csi-snapshotter:v6.0.1
           args:
             - "--csi-address=$(ADDRESS)"
-            - "--v=5"
+            - "--v=1"
             - "--timeout=150s"
             - "--leader-election=true"
             - "--extra-create-metadata=true"
@@ -86,7 +86,7 @@ spec:
         - name: csi-attacher
           image: registry.k8s.io/sig-storage/csi-attacher:v3.5.0
           args:
-            - "--v=5"
+            - "--v=1"
             - "--csi-address=$(ADDRESS)"
             - "--leader-election=true"
             - "--retry-interval-start=500ms"
@@ -101,7 +101,7 @@ spec:
           image: registry.k8s.io/sig-storage/csi-resizer:v1.5.0
           args:
             - "--csi-address=$(ADDRESS)"
-            - "--v=5"
+            - "--v=1"
             - "--timeout=150s"
             - "--leader-election"
             - "--retry-interval-start=500ms"

diff --git a/deploy/rbd/kubernetes/csi-rbdplugin.yaml b/deploy/rbd/kubernetes/csi-rbdplugin.yaml
@@ -31,7 +31,7 @@ spec:
             allowPrivilegeEscalation: true
           image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1
           args:
-            - "--v=5"
+            - "--v=1"
             - "--csi-address=/csi/csi.sock"
             - "--kubelet-registration-path=/var/lib/kubelet/plugins/rbd.csi.ceph.com/csi.sock"
           env: