Cannot create kind cluster with --config flag #2959

aliabbasjaffri · 2022-10-07T09:50:19Z

What happened:
I am trying to make my kind environment declarative and creating a kind config to create my cluster. Cluster creation with --config consistently fails, whereas without it succeeds.

What you expected to happen:
Kind cluster should be able to create with config provided to it.

How to reproduce it (as minimally and precisely as possible):
The config that i am creating:

# three node (two workers) cluster config
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node:v1.25.14@sha256:6e0f9005eba4010e364aa1bb25c8d7c64f050f744258eb68c4eb40c284c3c0dd
- role: worker
  image: kindest/node:v1.25.14@sha256:6e0f9005eba4010e364aa1bb25c8d7c64f050f744258eb68c4eb40c284c3c0dd
- role: worker
  image: kindest/node:v1.25.14@sha256:6e0f9005eba4010e364aa1bb25c8d7c64f050f744258eb68c4eb40c284c3c0dd

the command that i'm using:

~ ❯❯❯ kind create cluster --config .kind/kind-v1.25.2.yml
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.25.14) 🖼
 ✗ Preparing nodes 📦 📦 📦
ERROR: failed to create cluster: could not find a log line that matches "Reached target .*Multi-User System.*|detected cgroup v1"

Anything else we need to know?:

Environment:

kind version: (use kind version):

~ ❯❯❯ kind version
kind v0.16.0 go1.19.1 darwin/arm64

Kubernetes version: (use kubectl version):
Docker version: (use docker info):

~ ❯❯❯ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.9.1)
  compose: Docker Compose (Docker Inc., v2.10.2)
  extension: Manages Docker extensions (Docker Inc., v0.2.9)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.19.0)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 20.10.17
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.124-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 5
 Total Memory: 9.718GiB
 Name: docker-desktop
 ID: D3KZ:YKVP:6HJT:GUDW:DEVL:ZG44:RA7T:QRKA:YHN2:YHLK:GZZT:LPAT
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

OS (e.g. from /etc/os-release): macOS Monterey M1 macbook pro

#2959 (comment)

The text was updated successfully, but these errors were encountered:

stmcginnis · 2022-10-07T10:46:44Z

You may need to raise the amount of memory allocated for Docker Desktop in order to create a three node cluster.

Have you tried/verified that you are able to create a single node cluster? That would be a good sanity check of your environment to make sure there isn't some other underlying issue.

Another thing to try would be to run kind create cluster --config .kind/kind-v1.25.2.yml --retain. That will keep what had been created so far around after the failure. You can then inspect the logs to see if they point to a reason for the failure.

aliabbasjaffri · 2022-10-08T13:46:10Z

I believe i have provided more than enough resources to docker (see attached image)

Yes, I'm able to create and run a single node cluster, but if i try to create the same cluster via config file, it doesn't work.

I created a cluster with --retain flag and exported the logs. This is the output

~ ❯❯❯ kind export logs -n kind
Exporting logs for cluster "kind" to:
/private/var/folders/ns/22fnl3dj65n2745_r2g1kkjh0000gn/T/3520188974
ERROR: [command "docker exec --privileged kind-worker sh -c 'tar --hard-dereference -C /var/log/ -chf - . || (r=$?; [ $r -eq 1 ] || exit $r)'" failed with error: exit status 1, command "docker exec --privileged kind-worker2 sh -c 'tar --hard-dereference -C /var/log/ -chf - . || (r=$?; [ $r -eq 1 ] || exit $r)'" failed with error: exit status 1, command "docker exec --privileged kind-control-plane sh -c 'tar --hard-dereference -C /var/log/ -chf - . || (r=$?; [ $r -eq 1 ] || exit $r)'" failed with error: exit status 1, [[command "docker exec --privileged kind-worker journalctl --no-pager -u kubelet.service" failed with error: exit status 1, command "docker exec --privileged kind-worker journalctl --no-pager -u containerd.service" failed with error: exit status 1, command "docker exec --privileged kind-worker journalctl --no-pager" failed with error: exit status 1, command "docker exec --privileged kind-worker cat /kind/version" failed with error: exit status 1, command "docker exec --privileged kind-worker crictl images" failed with error: exit status 1], [command "docker exec --privileged kind-control-plane crictl images" failed with error: exit status 1, command "docker exec --privileged kind-control-plane cat /kind/version" failed with error: exit status 1, command "docker exec --privileged kind-control-plane journalctl --no-pager -u kubelet.service" failed with error: exit status 1, command "docker exec --privileged kind-control-plane journalctl --no-pager -u containerd.service" failed with error: exit status 1, command "docker exec --privileged kind-control-plane journalctl --no-pager" failed with error: exit status 1], [command "docker exec --privileged kind-worker2 journalctl --no-pager -u kubelet.service" failed with error: exit status 1, command "docker exec --privileged kind-worker2 journalctl --no-pager" failed with error: exit status 1, command "docker exec --privileged kind-worker2 cat /kind/version" failed with error: exit status 1, command "docker exec --privileged kind-worker2 journalctl --no-pager -u containerd.service" failed with error: exit status 1, command "docker exec --privileged kind-worker2 crictl images" failed with error: exit status 1]]]

BenTheElder · 2022-10-08T16:27:52Z

Can we see docker logs for the node containers?
Sounds like they failed in the entrypoint which is unusual.

BenTheElder · 2022-10-08T16:31:27Z

image: kindest/node:v1.25.14@sha256:6e0f9005eba4010e364aa1bb25c8d7c64f050f744258eb68c4eb40c284c3c0dd

1.25.14 isn't a real Kubernetes version (latest is 1.25.2) This digest also doesn't appear to match any from any recent KIND releases ... what is this image?

aliabbasjaffri · 2022-10-08T17:28:16Z

Oh, my bad there. That was a typo. I updated the version and tried again, but the same error.
I managed to pull the logs from exited docker containers for control-plane and workers:

~ ❯❯❯ docker logs 23f22a62e390
INFO: ensuring we can execute mount/umount even with userns-remap
INFO: remounting /sys read-only
INFO: making mounts shared
INFO: detected cgroup v2
INFO: clearing and regenerating /etc/machine-id
Initializing machine ID from random generator.
INFO: setting iptables to detected mode: legacy
iptables-save v1.8.7 (legacy): Cannot initialize: iptables who? (do you need to insmod?)

INFO: ensuring we can execute mount/umount even with userns-remap
INFO: remounting /sys read-only
INFO: making mounts shared
INFO: detected cgroup v2
INFO: clearing and regenerating /etc/machine-id
Initializing machine ID from random generator.
INFO: setting iptables to detected mode: legacy
iptables-save v1.8.7 (legacy): Cannot initialize: iptables who? (do you need to insmod?)

~ ❯❯❯ docker logs efd024650c25
INFO: ensuring we can execute mount/umount even with userns-remap
INFO: remounting /sys read-only
INFO: making mounts shared
INFO: detected cgroup v2
INFO: clearing and regenerating /etc/machine-id
Initializing machine ID from random generator.
INFO: setting iptables to detected mode: legacy
iptables-save v1.8.7 (legacy): Cannot initialize: iptables who? (do you need to insmod?)

INFO: ensuring we can execute mount/umount even with userns-remap
INFO: remounting /sys read-only
INFO: making mounts shared
INFO: detected cgroup v2
INFO: clearing and regenerating /etc/machine-id
Initializing machine ID from random generator.
INFO: setting iptables to detected mode: legacy
iptables-save v1.8.7 (legacy): Cannot initialize: iptables who? (do you need to insmod?)

~ ❯❯❯ docker ps -a
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS                      PORTS                       NAMES
23f22a62e390   kindest/node:v1.25.2   "/usr/local/bin/entr…"   55 seconds ago   Exited (1) 51 seconds ago                               kind-worker2
cbedeaa30970   kindest/node:v1.25.2   "/usr/local/bin/entr…"   55 seconds ago   Exited (1) 51 seconds ago                               kind-worker
efd024650c25   kindest/node:v1.25.2   "/usr/local/bin/entr…"   55 seconds ago   Exited (1) 51 seconds ago                               kind-control-plane

BenTheElder · 2022-10-08T17:36:55Z

Is this the arm64 image or amd64?

the fact that we can't detect or use iptables smells like wrong architecture
#2718

BenTheElder · 2022-10-08T17:37:59Z

docker inspect kind-control-plane will tell us IIRC

aliabbasjaffri · 2022-10-08T17:49:09Z

My bad! That was the real issue!
I was selecting amd64 architecture all along!
thank you so much @BenTheElder

~ ❯❯❯ kubectl get nodes
NAME                    STATUS   ROLES           AGE   VERSION
kind-ha-control-plane   Ready    control-plane   38s   v1.25.2
kind-ha-worker          Ready    <none>          18s   v1.25.2
kind-ha-worker2         Ready    <none>          18s   v1.25.2

aliabbasjaffri added the kind/bug Categorizes issue or PR as related to a bug. label Oct 7, 2022

aliabbasjaffri closed this as completed Oct 8, 2022

BenTheElder added kind/support Categorizes issue or PR as a support question. and removed kind/bug Categorizes issue or PR as related to a bug. labels Oct 8, 2022

BenTheElder self-assigned this Oct 8, 2022

aliabbasjaffri mentioned this issue Oct 10, 2022

Config file should validate if the image is suitable for arm or amd architecture #2963

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot create kind cluster with --config flag #2959

Cannot create kind cluster with --config flag #2959

aliabbasjaffri commented Oct 7, 2022 •

edited

Loading

stmcginnis commented Oct 7, 2022

aliabbasjaffri commented Oct 8, 2022

BenTheElder commented Oct 8, 2022

BenTheElder commented Oct 8, 2022 •

edited

Loading

aliabbasjaffri commented Oct 8, 2022

BenTheElder commented Oct 8, 2022

BenTheElder commented Oct 8, 2022

aliabbasjaffri commented Oct 8, 2022

Cannot create kind cluster with --config flag #2959

Cannot create kind cluster with --config flag #2959

Comments

aliabbasjaffri commented Oct 7, 2022 • edited Loading

stmcginnis commented Oct 7, 2022

aliabbasjaffri commented Oct 8, 2022

BenTheElder commented Oct 8, 2022

BenTheElder commented Oct 8, 2022 • edited Loading

aliabbasjaffri commented Oct 8, 2022

BenTheElder commented Oct 8, 2022

BenTheElder commented Oct 8, 2022

aliabbasjaffri commented Oct 8, 2022

aliabbasjaffri commented Oct 7, 2022 •

edited

Loading

BenTheElder commented Oct 8, 2022 •

edited

Loading