ceph-csi and kubelet 1.25.4 - rbdplugin-provisioner not starting

[rsoika]

I have installed kubernetes cluster on the latest version 1.25.4. Now I try to install the ceph-csi plugins but the pods are not starting. I tried the dev version from here and also version 3.7.2 without success.

I only see log messages like this:

stream logs failed container "csi-rbdplugin" in pod "csi-rbdplugin-ld4ng" is waiting to start: ContainerCreating for ceph-system/csi-rbdplugin-ld4ng (csi-rbdplugin)  
stream logs failed container "driver-registrar" in pod "csi-rbdplugin-ld4ng" is waiting to start: ContainerCreating for ceph-system/csi-rbdplugin-ld4ng (driver-registrar)   
stream logs failed container "liveness-prometheus" in pod "csi-rbdplugin-ld4ng" is waiting to start: ContainerCreating for ceph-system/csi-rbdplugin-ld4ng (liveness-prometheus) 

Now I have read that there are some changes in kubernetes 1.25.4 regarding the flag --allow-privileged and a change in the csi api here.

My question is: should it be possible to deploy ceph-csi in Kubernetes 1.25.4 or should I install Kubernetes 1.24.6 instead?

[Madhu-1]

@rsoika it should not matter it's kubernetes 1.24 or 1.25 its all the same. looks like it's something in your kubernetes cluster setting. cephcsi needs to run as privileged, kube-apiserver and kubelet should allow running privileged pods https://kubernetes-csi.github.io/docs/deploying.html#enable-privileged-pods

[rsoika]

@Madhu-1 yes it looks like it is something about the 'run as privileged'.
Regarding v1.25 if found this change about pod security and thought it could be related.

How can I verify if my kube-apiserver/kubelet support the flag allow-privileged ?

[Madhu-1]

Yes, that could be one reason. Can you set pod security to allow privileged pods https://kubernetes.io/docs/concepts/security/pod-security-admission/#pod-security-admission-labels-for-namespaces , let the level to warn to just event warning not to enforce the rules for now.

How can I verify if my kube-apiserver/kubelet support the flag allow-privileged ?

you can check this with running process with docker ps or ps and check the flags.

[rsoika]

I am running cri-o as container runtime (maybe an issue?)

I now changed my namespace settings regarding the pod-security like explained here

apiVersion: v1
kind: Namespace
metadata:
  name: ceph-system
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/enforce-version: v1.25

    # We are setting these to our _desired_ `enforce` level.  (restricted|privileged|baseline)
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: v1.25
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.25

The same result, but the following warnings are now shown when I create the ceph-csi plugins:

$ kubectl apply -f management/ceph/
namespace/ceph-system created
configmap/ceph-csi-config created
configmap/ceph-config created
serviceaccount/rbd-csi-nodeplugin created
clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created
clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created
serviceaccount/rbd-csi-provisioner created
clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created
role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created
rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created
service/csi-rbdplugin-provisioner created
Warning: would violate PodSecurity "restricted:v1.25": allowPrivilegeEscalation != false (containers "csi-provisioner", "csi-snapshotter", "csi-attacher", "csi-resizer", "csi-rbdplugin", "csi-rbdplugin-controller", "liveness-prometheus" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "csi-provisioner", "csi-snapshotter", "csi-attacher", "csi-resizer", "csi-rbdplugin", "csi-rbdplugin-controller", "liveness-prometheus" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volumes "host-dev", "host-sys", "lib-modules" use restricted volume type "hostPath"), runAsNonRoot != true (pod or containers "csi-provisioner", "csi-snapshotter", "csi-attacher", "csi-resizer", "csi-rbdplugin", "csi-rbdplugin-controller", "liveness-prometheus" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "csi-provisioner", "csi-snapshotter", "csi-attacher", "csi-resizer", "csi-rbdplugin", "csi-rbdplugin-controller", "liveness-prometheus" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
deployment.apps/csi-rbdplugin-provisioner created
Warning: would violate PodSecurity "restricted:v1.25": host namespaces (hostNetwork=true, hostPID=true), privileged (containers "driver-registrar", "csi-rbdplugin", "liveness-prometheus" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (containers "driver-registrar", "csi-rbdplugin", "liveness-prometheus" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "driver-registrar", "csi-rbdplugin", "liveness-prometheus" must set securityContext.capabilities.drop=["ALL"]; container "csi-rbdplugin" must not include "SYS_ADMIN" in securityContext.capabilities.add), restricted volume types (volumes "socket-dir", "plugin-dir", "mountpoint-dir", "ceph-logdir", "registration-dir", "host-dev", "host-sys", "etc-selinux", "host-mount", "lib-modules" use restricted volume type "hostPath"), runAsNonRoot != true (pod or containers "driver-registrar", "csi-rbdplugin", "liveness-prometheus" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "driver-registrar", "csi-rbdplugin", "liveness-prometheus" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
daemonset.apps/csi-rbdplugin created
service/csi-metrics-rbdplugin created
csidriver.storage.k8s.io/rbd.csi.ceph.com created
storageclass.storage.k8s.io/ceph-abpua created
storageclass.storage.k8s.io/ceph-i created

[Madhu-1]

CRIO should not be an issue. can you please share kubectl get po and describe po outputs?

[rsoika]

I have posted the data on gist:

https://gist.github.com/rsoika/13514c75fef3d26b3fabc05acbff2351

[Madhu-1]

Warning FailedMount 102s (x9 over 3m49s) kubelet MountVolume.SetUp failed for volume "ceph-csi-encryption-kms-config" : configmap "ceph-csi-encryption-kms-config" not found

You need to create this empty configmap kubectl create cm ceph-csi-encryption-kms-config -n<namepsace-name>

The above should fix your problem

[rsoika]

Oh my God, yes, you're absolutely right, of course! I did uncomment the config map during I tried to align my deployment settings to the latest version. Now it works ! thanks!

Why is this empty config map not part of the csi-config-map.yaml file?

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ceph-csi-encryption-kms-config
  # replace with non-default namespace name
  namespace: default
data:
  config.json: |-
    {}

[Madhu-1]

Yes, we can have an empty configmap. Not everyone uses encryption by default.