[BUG] nfs-client not able to mount NFS on macOS

[lopez-]

What did you do

• How was the cluster created?

  • k3d cluster create --k3s-arg "--disable=traefik@server:*" home

• What did you do afterwards?

Setup nfs-client to persist volumes.

helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/

helm install nfs nfs-subdir-external-provisioner/nfs-subdir-external-provisioner --set nfs.server=192.168.1.115 --set nfs.path=/volume1/k8s

What did you expect to happen

The nfs provider is ready to manage my volumes. On the terminal I'm successfully able to mount my volume via

mount -t nfs 192.168.1.115:/volume1/k8s ./local

Screenshots or terminal output

nfs is stuck on ContainerCreating. pod description can be found below

FYI /var/lib/kubelet doesn't exist on my system

Name:             nfs-nfs-subdir-external-provisioner-864fd7b5fd-q55mr
Namespace:        default
Priority:         0
Service Account:  nfs-nfs-subdir-external-provisioner
Node:             k3d-home-server-0/172.18.0.3
Start Time:       Thu, 14 Dec 2023 22:32:01 -0500
Labels:           app=nfs-subdir-external-provisioner
                  pod-template-hash=864fd7b5fd
                  release=nfs
Annotations:      <none>
Status:           Pending
IP:               
IPs:              <none>
Controlled By:    ReplicaSet/nfs-nfs-subdir-external-provisioner-864fd7b5fd
Containers:
  nfs-subdir-external-provisioner:
    Container ID:   
    Image:          registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      PROVISIONER_NAME:  cluster.local/nfs-nfs-subdir-external-provisioner
      NFS_SERVER:        192.168.1.115
      NFS_PATH:          /volume1/k8s
    Mounts:
      /persistentvolumes from nfs-subdir-external-provisioner-root (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wzjhw (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  nfs-subdir-external-provisioner-root:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    192.168.1.115
    Path:      /volume1/k8s
    ReadOnly:  false
  kube-api-access-wzjhw:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                   From               Message
  ----     ------       ----                  ----               -------
  Normal   Scheduled    26m                   default-scheduler  Successfully assigned default/nfs-nfs-subdir-external-provisioner-864fd7b5fd-q55mr to k3d-home-server-0
  Warning  FailedMount  6m57s (x7 over 24m)   kubelet            Unable to attach or mount volumes: unmounted volumes=[nfs-subdir-external-provisioner-root], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition
  Warning  FailedMount  5m24s (x17 over 26m)  kubelet            MountVolume.SetUp failed for volume "nfs-subdir-external-provisioner-root" : mount failed: exit status 255
Mounting command: mount
Mounting arguments: -t nfs 192.168.1.115:/volume1/k8s /var/lib/kubelet/pods/7cef225b-dc8b-4952-8c97-62228aa4f8e1/volumes/kubernetes.io~nfs/nfs-subdir-external-provisioner-root
Output: mount: mounting 192.168.1.115:/volume1/k8s on /var/lib/kubelet/pods/7cef225b-dc8b-4952-8c97-62228aa4f8e1/volumes/kubernetes.io~nfs/nfs-subdir-external-provisioner-root failed: Connection refused
  Warning  FailedMount  62s                 kubelet  Unable to attach or mount volumes: unmounted volumes=[nfs-subdir-external-provisioner-root], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition
  Warning  FailedMount  56s (x9 over 3m5s)  kubelet  MountVolume.SetUp failed for volume "nfs-subdir-external-provisioner-root" : mount failed: exit status 255
Mounting command: mount
Mounting arguments: -t nfs 192.168.1.115:/volume1/k8s /var/lib/kubelet/pods/7cef225b-dc8b-4952-8c97-62228aa4f8e1/volumes/kubernetes.io~nfs/nfs-subdir-external-provisioner-root
Output: mount: mounting 192.168.1.115:/volume1/k8s on /var/lib/kubelet/pods/7cef225b-dc8b-4952-8c97-62228aa4f8e1/volumes/kubernetes.io~nfs/nfs-subdir-external-provisioner-root failed: Connection refused

Which OS & Architecture

• output of k3d runtime-info

arch: aarch64
cgroupdriver: cgroupfs
cgroupversion: "2"
endpoint: /var/run/docker.sock
filesystem: extfs
infoname: docker-desktop
name: docker
os: Docker Desktop
ostype: linux
version: 24.0.6

Which version of k3d

• output of k3d version

k3d version v5.6.0
k3s version v1.27.5-k3s1 (default)

Which version of docker

• output of docker version

Client:
 Cloud integration: v1.0.35+desktop.4
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:28:49 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.23.0 (120376)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:36 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

• output of docker info

Client:
 Version:    24.0.6
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.4
    Path:     /Users/myuser/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0-desktop.1
    Path:     /Users/myuser/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/myuser/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/myuser/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.7
    Path:     /Users/myuser/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/myuser/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/myuser/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  0.24.1
    Path:     /Users/myuser/.docker/cli-plugins/docker-scout

Server:
 Containers: 3
  Running: 3
  Paused: 0
  Stopped: 0
 Images: 7
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.3.13-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 5
 Total Memory: 7.667GiB
 Name: docker-desktop
 ID: 58a32658-9654-4378-8f7a-56e32a003246
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile