kind v0.13.0 does not work with Colima 0.4.1

[longwuyuan]

What happened:

• kind create cluster failed

  • with kind version 0.13.0
  • on Colima v0.4.1
  • On macOS Monterey v12.4
  • on Intel Mac Pro

• The logs are here kind 0.13.0 does not work with Colima v0.4.1 abiosoft/colima#291

• Kind version 0.12.0 works. I found out the hard way after finally brew uninstall kind and then manually downloading the binary https://github.com/kubernetes-sigs/kind/releases/download/v0.12.0/kind-darwin-amd64 and use the download kind v0.12.0

% kind version && kind get clusters && which kind
kind v0.12.0 go1.17.8 darwin/amd64
ingress-nginx-dev
/Users/m/bin/kind

What you expected to happen:

• kind installed by brew on MacOS to work just like kind v0.12.0 works

How to reproduce it (as minimally and precisely as possible):

• On macOS Monterey, on a Intel Mac, make sure brew is updated
• Install Colima with brew install Colima
• Start Colima with colima start --cpu 2 --memory 4 --disk 20
• install kind with brew install kind
• Check kind version and make sure its the latest v0.13.0 (NOT v0.12.0)
• Try to create a cluster with kind kind create cluster --name dev

Anything else we need to know?:

• Not sure if its only me or happens to others as well

Environment:

• macOS Monterey on Intel Mac with brew update && brew upgrade
• kind version: (use kind version):

kind v0.13.0 go1.18.2 darwin/amd64

• Kubernetes version: (use kubectl version):

v1.24.0

• Docker version: (use docker info):

% docker version
Client: Docker Engine - Community
 Version:           20.10.16
 API version:       1.41
 Go version:        go1.18.2
 Git commit:        aa7e414fdc
 Built:             Wed May 11 16:22:17 2022
 OS/Arch:           darwin/amd64
 Context:           colima
 Experimental:      true

Server:
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.9
  Git commit:       87a90dc786bda134c9eb02adbae2c6a7342fb7f6
  Built:            Fri Apr 15 00:05:05 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.2
  GitCommit:        52b36a2dd837e8462de8e01458bf02cf9eea47dd
 docker-init:
  Version:          0.19.0
  GitCommit:        

• OS (e.g. from /etc/os-release):

macOS Monterey v12.4 on Intel Mac

[BenTheElder]

I can't run this myself (we are not approved to run VMs on our work machines) -- does colima use an init other than systemd?

if so this is likely the same issue as #2769, #2765

[BenTheElder]

See also: workarounds in the meantime in the other issues, if this is indeed the same root issue (i'd guess it is, it's the main change in that version).

[longwuyuan]

oh ok. extremely likely because Colima uses alpine (rc). Verifying the other issues

[BenTheElder]

I dug out a personal mac and installed colima tonight to reproduce.

For reference: inside colima openrc is in use like #2765 (because both use lima / alpine I think), and systemd is running in "hybrid" cgroups mode (both cgroupv1 and cgroupv2)
https://systemd.io/CGROUP_DELEGATION/

I have a working fix, will update the PR and push an image for testing in the other environments.

[longwuyuan]

@BenTheElder gratitude.
"other environment" relevance is fixed kind v0.13.0 image in systemd+non-systemd environments ?

[BenTheElder]

@BenTheElder gratitude.

🙏

"other environment" relevance is fixed kind v0.13.0 image in systemd+non-systemd environments ?

Yes, I've already tested the fix with colima as instructed in this issue, but I want to ask folks to test in WSL2 and rancher desktop. #2765 #2769 #2766 (which, hopefully are also fixed by this fix)

[longwuyuan]

Yes, I've already tested the fix with colima as instructed in this issue, but I want to ask folks to test in WSL2 and rancher desktop. #2765 #2769 #2766 (which, hopefully are also fixed by this fix)

No words to express gratitude. Explains the beautiful usefullness of kind. 🙏

[BenTheElder]

Sorry for the problems here, clearly should have checked the driver change on a non-systemd host.

This image should work for v1.24 with kind v0.13.0 (not with kind v0.12.0)
kind create cluster --image=bentheelder/kind-node:v1.24.0@sha256:cf92db251fb3f420634ec46d9c222812fee4d75f1e0105a325e57679b0f57370

We'll push another release soon with this fix. #2767

[longwuyuan]

@BenTheElder Simply awesome awesome help 🙏 . Tested your image working with kind v0.13.0 on Colima 0.4.1

[longwuyuan]

@rikatz make dev-env, make build image and make kind-e2e-test all happily singing & dancing on kind+colima :-)

[BenTheElder]

https://github.com/kubernetes-sigs/kind/releases/tag/v0.14.0

[rosscdh]

Sorry seems to have regressed

kind version
kind v0.18.0 go1.20.2 darwin/amd64


kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.26.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✗ Starting control-plane 🕹️
Deleted nodes: ["kind-control-plane"]
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged kind-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 137
Command Output: I0423 21:48:43.246975     158 initconfiguration.go:254] loading configuration from "/kind/kubeadm.conf"
W0423 21:48:43.267098     158 initconfiguration.go:331] [config] WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta3, Kind=JoinConfiguration
[init] Using Kubernetes version: v1.26.3
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I0423 21:48:43.306763     158 certs.go:112] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I0423 21:48:43.984573     158 certs.go:519] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kind-control-plane kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost] and IPs [10.96.0.1 172.20.0.2 127.0.0.1]

[BenTheElder]

Please file a new issue and fill out the bug template and include the logs export, this isn't enough to diagnose and we can't run this in CI or on our work machines.

[BenTheElder]

In general I don't recommend trying to run Kubernetes under openRC, to my knowledge there's zero CI for this in Kubernetes, containerd, cri-o, or dockershim and cgroupsv2 under systemd is slowly becoming the default in the container ecosystem. IIRC kubeadm didn't support openrc at all for a very long time and still has limited to no support.

we do our best to make kind hackery work anyhow, but it's not in CI