Persistent Volumes permission issue for 2+ node clutsters #12360

albert-salman · 2021-08-27T10:01:48Z

Hello,
If minikube cluster is 2+ nodes then initContainers cannot set volume permissions inside PVCs and pods fail to access data there. Single node cluster do not have this issue.

RA489 · 2021-09-01T03:07:51Z

/kind support

spowelljr · 2021-10-06T18:47:33Z

Hi @albert-salman, thanks for reporting your issue with minikube!

To help me debug the issue you're experiencing, could you please provide me with a list of instructions to follow to allow me to reproduce this issue?

Thanks!

albert-salman · 2021-10-07T11:59:04Z

To help me debug the issue you're experiencing, could you please provide me with a list of instructions to follow to allow me to reproduce this issue?

Hi @spowelljr

Create 2-node cluster
Deploy some manifest or helm chart (e.g. bitnami's elasticstack) which use initContainers to setup permission for containers' PVCs
Path which had to have correct permissions (set by initContainers section) is denied for any running app in container

i have a suspicion that PVC is not correctly mapped at node which starts the initContainers's temporary container

i am using
minikube version: v1.22.0
commit: a03fbcf

in bitnami/elasticstack file master-statefulset.yaml

 initContainers:
    - name: volume-permissions
      image: {{ include "elasticsearch.volumePermissions.image" . }}
      imagePullPolicy: {{ .Values.volumePermissions.image.pullPolicy | quote }}
      command:
        - /bin/bash
        - -ec
        - |
          chown -R {{ .Values.master.securityContext.runAsUser }}:{{ .Values.master.securityContext.fsGroup }} /bitnami/elasticsearch/data
      securityContext:
        runAsUser: 0
      {{- if .Values.volumePermissions.resources }}
      resources: {{- toYaml .Values.volumePermissions.resources | nindent 12 }}
      {{- end }}
      volumeMounts:
        - name: data
          mountPath: "/bitnami/elasticsearch/data"

victor-sudakov · 2021-12-05T07:21:23Z

I think my problem is related. On a 2-node cluster, the spec.template.spec.securityContext.fsGroup=65534 setting is not respected and volume membership is not changed. It remains 755 root:root instead of 2775 root:nobody.

How to reproduce

Apply the following manifest

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: ubuntu
  name: ubuntu
spec:
  replicas: 1
  serviceName: ubuntu-headless
  selector:
    matchLabels:
      app: ubuntu
  template:
    metadata:
      labels:
        app: ubuntu
    spec:
      securityContext:
        fsGroup: 65534
        runAsGroup: 65534
        runAsNonRoot: true
        runAsUser: 65534
      containers:
        - image: ubuntu:jammy-20211122
          name: ubuntu
          command: ["/bin/sleep", "365d"]
          volumeMounts:
            - mountPath: /d01
              name: bigdisk
  volumeClaimTemplates:
    - metadata:
        name: bigdisk
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 5G

Shell into the ubuntu-0 Pod and check the permissions on the /d01 mountpoint. They will not belong to root:nobody as they should according to the fsGroup setting, but will remain root:root.

Minikube version
minikube version: v1.24.0 commit: 76b94fb3c4e8ac5062daf70d60cf03ddcc0a741b-dirty

Minikube was started as
minikube start --disk-size=50g --nodes=2 --insecure-registry "192.168.88.0/24" --mount --mount-string ~/.volumes:/volumes

victor-sudakov · 2021-12-05T07:27:15Z

@albert-salman Do you see the problem if you use the fsGroup security context instead of initContainers? I do.

BTW I think using initContainers just for setting permissions is a bit of overkill.

victor-sudakov · 2021-12-06T02:27:55Z

On a single-node minikube cluster, the setting of spec.template.spec.securityContext.fsGroup=65534 results in volume permissions 777 root:root which is not correct either. fsGroup should not work this way IMHO.

nobody@ubuntu-0:/$ ls -ald /d01
drwxrwxrwx 2 root root 6 Dec  6 02:25 /d01
nobody@ubuntu-0:/$

Expected: root:nobody

@albert-salman can you confirm this?

victor-sudakov · 2022-01-07T04:32:44Z

@albert-salman ping!

k8s-triage-robot · 2022-04-07T05:12:10Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

victor-sudakov · 2022-04-07T18:41:05Z

/remove-lifecycle stale

denisok · 2022-05-11T06:44:30Z

Hi all,

I have similar issue with minikube and I think it is related to host-pathprovisioner.

minikube start --nodes=4 --cpus=3
$ minikube version
minikube version: v1.24.0
commit: 76b94fb3c4e8ac5062daf70d60cf03ddcc0a741b

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"archive", BuildDate:"2021-07-22T00:00:00Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:35:25Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}

I use security context for a pod and pvc dynamic claim.

  securityContext:
    runAsNonRoot: true
    runAsUser: 1001

Same issue when accessing volume - access denied.

I found out checking different docs, PRs and code, that actually host-pathprovisioner doesn't support securityContext at all, to workaround it provisioner just sets 777 permissions to the provisioned directories so when they mounted - they could be accessed by any pod with any permission:
https://github.com/kubernetes/minikube/blob/master/pkg/storage/storage_provisioner.go#L67

minikube runs standard controller as enabled by default addon:

$ kubectl get pods -o wide -n kube-system
NAME                               READY   STATUS    RESTARTS   AGE   IP               NODE           NOMINATED NODE   READINESS GATES
coredns-78fcd69978-f7g4z           1/1     Running   0          21m   10.244.0.2       minikube       <none>           <none>
etcd-minikube                      1/1     Running   0          21m   192.168.39.233   minikube       <none>           <none>
kindnet-2rzwl                      1/1     Running   0          20m   192.168.39.39    minikube-m04   <none>           <none>
kindnet-qd4cz                      1/1     Running   0          21m   192.168.39.233   minikube       <none>           <none>
kindnet-r7xtb                      1/1     Running   0          20m   192.168.39.247   minikube-m03   <none>           <none>
kindnet-rp24h                      1/1     Running   0          21m   192.168.39.226   minikube-m02   <none>           <none>
kube-apiserver-minikube            1/1     Running   0          21m   192.168.39.233   minikube       <none>           <none>
kube-controller-manager-minikube   1/1     Running   0          21m   192.168.39.233   minikube       <none>           <none>
kube-proxy-7w222                   1/1     Running   0          21m   192.168.39.226   minikube-m02   <none>           <none>
kube-proxy-nftpm                   1/1     Running   0          20m   192.168.39.247   minikube-m03   <none>           <none>
kube-proxy-pzn7l                   1/1     Running   0          20m   192.168.39.39    minikube-m04   <none>           <none>
kube-proxy-tvxxq                   1/1     Running   0          21m   192.168.39.233   minikube       <none>           <none>
kube-scheduler-minikube            1/1     Running   0          21m   192.168.39.233   minikube       <none>           <none>
storage-provisioner                1/1     Running   0          21m   192.168.39.233   minikube       <none>           <none>

And it indeed provisions host path on minikube node it is running with 777:

$ minikube ssh -n minikube
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ ls -la /tmp/hostpath-provisioner/
total 8
drwxr-xr-x  3 root root 4096 May 11 06:20 .
drwxrwxrwt 13 root root  300 May 11 06:18 ..
drwxr-xr-x  5 root root 4096 May 11 06:24 default
$ ls -la /tmp/hostpath-provisioner/default/
total 20
drwxr-xr-x 5 root root 4096 May 11 06:24 .
drwxr-xr-x 3 root root 4096 May 11 06:20 ..
drwxrwxrwx 2 root root 4096 May 11 06:24 mongod-data-minikube-cfg-0
drwxrwxrwx 2 root root 4096 May 11 06:24 mongod-data-minikube-rs0-0
drwxrwxrwx 2 root root 4096 May 11 06:20 pmm-storage-pmm-0

but on other nodes it provisions with 775:

$ minikube ssh -n minikube-m03
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ ls -la /tmp/hostpath-provisioner/default/
total 12
drwxr-xr-x 3 root root 4096 May 11 06:25 .
drwxr-xr-x 3 root root 4096 May 11 06:25 ..
drwxr-xr-x 2 root root 4096 May 11 06:25 mongod-data-minikube-rs0-0

And thus as all other nodes has 775 only privileged containers could start on those volumes and on those nodes.

Looks like a bug in provisioner code.

denisok · 2022-05-11T08:50:11Z

So my theory on whats happens: provisioner runs only on master node and responds correctly with pvc to the k8s. It creates PVC only locally on master node.

When scheduler schedules pod on some node kubelet checks for PVC, finds it, attaches to the container that runs under the docker with the path from that PVC that was created only on master node.
Docker takes that path, it doesn't exists and it creates it locally with default permissions:

$ ls -la /tmp
total 16
drwxrwxrwt 13 root   root    300 May 11 06:19 .
drwxr-xr-x 18 root   root    480 May 11 06:18 ..
drwxrwxrwt  2 root   root     40 May 11 06:18 .ICE-unix
drwxrwxrwt  2 root   root     40 May 11 06:18 .Test-unix
drwxrwxrwt  2 root   root     40 May 11 06:18 .X11-unix
drwxrwxrwt  2 root   root     40 May 11 06:18 .XIM-unix
drwxrwxrwt  2 root   root     40 May 11 06:18 .font-unix
drwxr-xr-x  2 root   root     40 May 11 06:18 gvisor
-rw-r--r--  1 docker docker   79 May 11 06:18 h.2529
-rw-r--r--  1 docker docker  126 May 11 06:18 h.2574
drwxr-xr-x  3 root   root   4096 May 11 06:25 hostpath-provisioner
drwxr-xr-x  2 root   root   4096 May 11 06:18 hostpath_pv
drwx------  3 root   root     60 May 11 06:18 systemd-private-4b03b55590774907b3007965a5da2612-systemd-logind.service-SctAZg
drwx------  3 root   root     60 May 11 06:18 systemd-private-4b03b55590774907b3007965a5da2612-systemd-resolved.service-bjwybg
drwx------  3 root   root     60 May 11 06:18 systemd-private-4b03b55590774907b3007965a5da2612-systemd-timesyncd.service-XtzY7f

$ docker run -it --rm -v /tmp/new/dir:/tmp/dir busybox
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
50e8d59317eb: Pull complete
Digest: sha256:d2b53584f580310186df7a2055ce3ff83cc0df6caacf1e3489bff8cf5d0af5d8
Status: Downloaded newer image for busybox:latest

/ # ls -la /tmp/
total 8
drwxrwxrwt    1 root     root          4096 May 11 08:45 .
drwxr-xr-x    1 root     root          4096 May 11 08:45 ..
drwxr-xr-x    2 root     root            40 May 11 08:45 dir
/ # ls -la /tmp/dir/
total 4
drwxr-xr-x    2 root     root            40 May 11 08:45 .
drwxrwxrwt    1 root     root          4096 May 11 08:45 ..
/ # stat /tmp/dir
  File: /tmp/dir
  Size: 40        	Blocks: 0          IO Block: 4096   directory
Device: 24h/36d	Inode: 151276      Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-05-11 08:45:50.933769106 +0000
Modify: 2022-05-11 08:45:42.786726156 +0000
Change: 2022-05-11 08:45:42.786726156 +0000
/ # id
uid=0(root) gid=0(root) groups=10(wheel)
/ # exit

$ ls -la /tmp/new/dir/
total 0
drwxr-xr-x 2 root root 40 May 11 08:45 .
drwxr-xr-x 3 root root 60 May 11 08:45 ..
$ stat /tmp/new/dir
  File: /tmp/new/dir
  Size: 40        	Blocks: 0          IO Block: 4096   directory
Device: 24h/36d	Inode: 151276      Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-05-11 08:46:12.110882020 +0000
Modify: 2022-05-11 08:45:42.786726156 +0000
Change: 2022-05-11 08:45:42.786726156 +0000
 Birth: -

denisok · 2022-05-11T09:08:23Z

that theory confirmed by couple of facts:

directories for all pods that scheduled on different nodes do exists on master node, even those pods weren't run on master
nothing runs on the nodes to actually create those directories
the only thing in the logs for one of the node:

May 11 06:24:56 minikube-m03 kubelet[2643]: I0511 06:24:56.400670    2643 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-371b5e4e-a080-4a23-9014-ef6b87ce0b41\" (UniqueName: \"kubernetes.io/host-path/18998f87-1463-4335-a996-f304d5bf78c9-pvc-371b5e4e-a080-4a23-9014-ef6b87ce0b41\") pod \"minikube-rs0-0\" (UID: \"18998f87-1463-4335-a996-

when starting pod that would fail with access.

So I think that current host-pathprovisioner doesn't account nodes somehow: SelectedNode is not used Provision.
Or maybe those pods should run on all nodes to handle provision locally, for example upstream kinda says about it: https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/blob/master/examples/hostpath-provisioner/hostpath-provisioner.go#L51
https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/blob/a35bec6eb688ef7a0e31b340ed77bdb84010ace7/examples/hostpath-provisioner/pod.yaml#L11

denisok · 2022-05-11T09:27:41Z

/remove-lifecycle stale

denisok · 2022-05-11T09:42:50Z

/triage discuss

denisok · 2022-05-11T13:54:27Z

So looks like the host-pathprovisioner in minikube doesn't support multinode at all, and can provision only master node.

There are other host provisioner that support multi node:

I tried kubevirt legacy and it works for me:

$ minikube start --nodes=4 --cpus=3
😄  minikube v1.24.0 on Fedora 35
...
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

$ minikube addons disable storage-provisioner
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌑  "The 'storage-provisioner' addon is disabled

$ kubectl delete storageclass standard
storageclass.storage.k8s.io "standard" deleted

$ kubectl apply -f kubevirt-hostpath-provisioner.yaml
storageclass.storage.k8s.io/standard created
clusterrolebinding.rbac.authorization.k8s.io/kubevirt-hostpath-provisioner created
clusterrole.rbac.authorization.k8s.io/kubevirt-hostpath-provisioner created
serviceaccount/kubevirt-hostpath-provisioner-admin created
daemonset.apps/kubevirt-hostpath-provisioner created

here is kubevirt-hostpath-provisioner.yaml :

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubevirt.io/hostpath-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubevirt-hostpath-provisioner
subjects:
- kind: ServiceAccount
  name: kubevirt-hostpath-provisioner-admin
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: kubevirt-hostpath-provisioner
  apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubevirt-hostpath-provisioner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]

  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]

  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubevirt-hostpath-provisioner-admin
  namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kubevirt-hostpath-provisioner
  labels:
    k8s-app: kubevirt-hostpath-provisioner
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: kubevirt-hostpath-provisioner
  template:
    metadata:
      labels:
        k8s-app: kubevirt-hostpath-provisioner
    spec:
      serviceAccountName: kubevirt-hostpath-provisioner-admin
      containers:
        - name: kubevirt-hostpath-provisioner
          image: quay.io/kubevirt/hostpath-provisioner
          imagePullPolicy: Always
          env:
            - name: USE_NAMING_PREFIX
              value: "false" # change to true, to have the name of the pvc be part of the directory
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: PV_DIR
              value: /tmp/hostpath-provisioner
          volumeMounts:
            - name: pv-volume # root dir where your bind mounts will be on the node
              mountPath: /tmp/hostpath-provisioner/
              #nodeSelector:
              #- name: xxxxxx
      volumes:
        - name: pv-volume
          hostPath:
            path: /tmp/hostpath-provisioner/

denisok · 2022-05-11T13:57:29Z

$ kubectl get pods -o wide -A
NAMESPACE     NAME                                  READY   STATUS    RESTARTS   AGE     IP               NODE           NOMINATED NODE   READINESS GATES
kube-system   etcd-minikube                         1/1     Running   0          9m45s   192.168.39.100   minikube       <none>           <none>
kube-system   kindnet-7kzq4                         1/1     Running   0          9m33s   192.168.39.100   minikube       <none>           <none>
kube-system   kindnet-8kpws                         1/1     Running   0          8m26s   192.168.39.235   minikube-m04   <none>           <none>
kube-system   kindnet-nvczc                         1/1     Running   0          8m54s   192.168.39.177   minikube-m03   <none>           <none>
kube-system   kindnet-tnhh6                         1/1     Running   0          9m22s   192.168.39.33    minikube-m02   <none>           <none>
kube-system   kube-apiserver-minikube               1/1     Running   0          9m45s   192.168.39.100   minikube       <none>           <none>
kube-system   kube-controller-manager-minikube      1/1     Running   0          9m45s   192.168.39.100   minikube       <none>           <none>
kube-system   kube-proxy-5hbth                      1/1     Running   0          8m26s   192.168.39.235   minikube-m04   <none>           <none>
kube-system   kube-proxy-gfhz9                      1/1     Running   0          9m22s   192.168.39.33    minikube-m02   <none>           <none>
kube-system   kube-proxy-m7wmm                      1/1     Running   0          8m54s   192.168.39.177   minikube-m03   <none>           <none>
kube-system   kube-proxy-r29lm                      1/1     Running   0          9m33s   192.168.39.100   minikube       <none>           <none>
kube-system   kube-scheduler-minikube               1/1     Running   0          9m47s   192.168.39.100   minikube       <none>           <none>
kube-system   kubevirt-hostpath-provisioner-5lqbc   1/1     Running   0          7m20s   10.244.0.3       minikube       <none>           <none>
kube-system   kubevirt-hostpath-provisioner-g5z7k   1/1     Running   0          7m20s   10.244.1.2       minikube-m02   <none>           <none>
kube-system   kubevirt-hostpath-provisioner-gxgpm   1/1     Running   0          7m20s   10.244.2.2       minikube-m03   <none>           <none>
kube-system   kubevirt-hostpath-provisioner-s6nwx   1/1     Running   0          7m20s   10.244.3.2       minikube-m04   <none>           <none>

and deployment with different securityContext did work for me, as well as the directory were created on a right nodes with 777 permissions.

victor-sudakov · 2022-05-11T21:04:01Z

@denisok Thanks for the links to the alternative hostpath provisioners.

denisok · 2022-05-18T17:20:14Z

https://github.com/kubernetes-csi/csi-driver-host-path could be a good solution option as it is already available in Addons.

But it also has the bug with permissions. I tried to:

disable storage-provisioner addon
install https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy/kubernetes-distributed with deploy.sh
it indeed installs provisioner on all nodes
kubectl patch storageclass standard -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
kubectl patch storageclass csi-hostpath-slow -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
it creates and mounts volumes on correct nodes, but still volumes have 0750 perm

I think there might be a bug in csi-driver-host-path, it might mount provisioned volume and probably resets permissions or something:
https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/pkg/hostpath/nodeserver.go#L144
https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/pkg/hostpath/nodeserver.go#L179

I think the way to go might be to update csi-driver-host-path in minikube, make it distributed and document how to use it in case of multinode. Or maybe get it enabled by default when we have more than 1 node.

Also csi-driver-host-path needs to be foxed to get 777perm on mounted dirs.

I can try to fix it myself if someone checks if that correct path, and what is the way - documenting, automatic detection what addons to enable, or maybe there some reason to not have distributed csi.

k8s-triage-robot · 2022-08-16T18:02:59Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

RA489 · 2022-08-17T01:31:59Z

/remove-lifecycle stale

medyagh · 2022-09-21T18:20:52Z

@denisok thank you for the detailed debugging and investigations, nice work here.

minikube comes with a very basic in-house storage provisioner and I wouldn't be surprised if it doesn't cover corner cases or multi-nodes.

if it helps here is the link to the code in these two files

I would love it if you or anyone make a PR to either add the functionality to cover this use case or add a tutorial on our website to how to use it with this workaround ?

also adding @presztak

k8s-triage-robot · 2022-12-20T19:06:12Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2023-01-19T20:03:20Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

denisok · 2023-02-14T19:07:00Z

Fix:

minikube addons disable storage-provisioner
minikube addons disable default-storageclass
minikube addons enable volumesnapshots
minikube addons enable csi-hostpath-driver
kubectl patch storageclass csi-hostpath-sc -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

As suggested here: #15829 (review)

Will be good to have it automatic for the multi-node.

luispabon · 2023-03-21T12:55:13Z

The csi-hostpath-driver workaround above doesn't work for me. Folders are created with 750 permissions and as such only privileged containers can access them.

gryphon2411 · 2023-08-28T06:40:17Z

@denisok I confirm your fix fixed me the following issues:

Run type: RabbitmqCluster resource with replica: 3 on multi-node Minikube.
Run type: Elasticsearch resource with count: 3 on multi-node Minikube.
Run type: Kibana resource with count: 3 on multi-node Minikube.
Run type: Logstash resource with count:3 on multi-node Minikube

Before your fix, neither worked.

Much thanks, I do agree it should be automatic for the multi-node.

k8s-ci-robot added the kind/support Categorizes issue or PR as a support question. label Sep 1, 2021

spowelljr added co/multinode Issues related to multinode clusters triage/needs-information Indicates an issue needs more information in order to work on it. long-term-support Long-term support issues that can't be fixed in code labels Oct 6, 2021

spowelljr removed the triage/needs-information Indicates an issue needs more information in order to work on it. label Oct 7, 2021

victor-sudakov mentioned this issue Dec 5, 2021

A StatefulSet does not respect fsGroup when creating a volume from a volumeClaimTemplate kubernetes/kubernetes#106806

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 7, 2022

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 7, 2022

k8s-ci-robot added the triage/discuss Items for discussion label May 11, 2022

denisok mentioned this issue May 28, 2022

fix volume dir mode for unprivileged containers kubernetes-csi/csi-driver-host-path#356

Closed

denisok mentioned this issue Jul 4, 2022

Allow Kubernetes to change permission of provisioned volumes kubernetes-csi/csi-driver-host-path#362

Merged

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 16, 2022

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 17, 2022

presztak mentioned this issue Oct 3, 2022

Add local-path-provisioner addon #15062

Merged

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 20, 2022

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 19, 2023

denisok mentioned this issue Feb 10, 2023

multi-node: Add support for volumes using CSI addon #15829

Merged

medyagh closed this as completed in #15829 Feb 14, 2023

AlanGreene mentioned this issue May 3, 2023

Unable to import pipeline and tasks while following the Minikube tutorial tektoncd/dashboard#2898

Closed

jiayouxujin mentioned this issue Jun 10, 2023

fix: use the default storageclass when the storageclass is nil RocksLabs/kvrocks-operator#34

Merged

Noxsios mentioned this issue Sep 3, 2023

zarf init fails on multi-node minikube cluster zarf-dev/zarf#1998

Closed

soenkeliebau mentioned this issue Sep 14, 2023

bash: /stackable/data/myid: Permission denied stackabletech/zookeeper-operator#727

Closed

KrKOo mentioned this issue Dec 12, 2023

permission denied error in hub container CERIT-SC/nbgrader-k8s#2

Closed

harshavardhana mentioned this issue May 10, 2024

Server init fail on minikube when multiple nodes are set minio/minio#19692

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Persistent Volumes permission issue for 2+ node clutsters #12360

Persistent Volumes permission issue for 2+ node clutsters #12360

albert-salman commented Aug 27, 2021

RA489 commented Sep 1, 2021

spowelljr commented Oct 6, 2021

albert-salman commented Oct 7, 2021

victor-sudakov commented Dec 5, 2021

victor-sudakov commented Dec 5, 2021

victor-sudakov commented Dec 6, 2021 •

edited

Loading

victor-sudakov commented Jan 7, 2022

k8s-triage-robot commented Apr 7, 2022

victor-sudakov commented Apr 7, 2022

denisok commented May 11, 2022 •

edited

Loading

denisok commented May 11, 2022 •

edited

Loading

denisok commented May 11, 2022

denisok commented May 11, 2022

denisok commented May 11, 2022

denisok commented May 11, 2022 •

edited

Loading

denisok commented May 11, 2022

victor-sudakov commented May 11, 2022

denisok commented May 18, 2022 •

edited

Loading

k8s-triage-robot commented Aug 16, 2022

RA489 commented Aug 17, 2022

medyagh commented Sep 21, 2022

k8s-triage-robot commented Dec 20, 2022

k8s-triage-robot commented Jan 19, 2023

denisok commented Feb 14, 2023

luispabon commented Mar 21, 2023

gryphon2411 commented Aug 28, 2023 •

edited

Loading

Persistent Volumes permission issue for 2+ node clutsters #12360

Persistent Volumes permission issue for 2+ node clutsters #12360

Comments

albert-salman commented Aug 27, 2021

RA489 commented Sep 1, 2021

spowelljr commented Oct 6, 2021

albert-salman commented Oct 7, 2021

victor-sudakov commented Dec 5, 2021

victor-sudakov commented Dec 5, 2021

victor-sudakov commented Dec 6, 2021 • edited Loading

victor-sudakov commented Jan 7, 2022

k8s-triage-robot commented Apr 7, 2022

victor-sudakov commented Apr 7, 2022

denisok commented May 11, 2022 • edited Loading

denisok commented May 11, 2022 • edited Loading

denisok commented May 11, 2022

denisok commented May 11, 2022

denisok commented May 11, 2022

denisok commented May 11, 2022 • edited Loading

denisok commented May 11, 2022

victor-sudakov commented May 11, 2022

denisok commented May 18, 2022 • edited Loading

k8s-triage-robot commented Aug 16, 2022

RA489 commented Aug 17, 2022

medyagh commented Sep 21, 2022

k8s-triage-robot commented Dec 20, 2022

k8s-triage-robot commented Jan 19, 2023

denisok commented Feb 14, 2023

luispabon commented Mar 21, 2023

gryphon2411 commented Aug 28, 2023 • edited Loading

victor-sudakov commented Dec 6, 2021 •

edited

Loading

denisok commented May 11, 2022 •

edited

Loading

denisok commented May 11, 2022 •

edited

Loading

denisok commented May 11, 2022 •

edited

Loading

denisok commented May 18, 2022 •

edited

Loading

gryphon2411 commented Aug 28, 2023 •

edited

Loading