Unable to create multi-node Kind cluster in Podman container

[8-cm]

Hi,

I'm trying to create a multi-node Kind cluster inside a Podman container, but I'm encountering issues. I'm wondering if anyone has successfully created a multi-node cluster in a similar setup.

Goal:
Ideally, I'd like to create a cluster with 3 control-plane nodes and 3 worker nodes. However, at this point, successfully creating any cluster with more than 1 node would be progress.

Current issue:
When trying to create a multi-node cluster, I get the following error:

# kind-create-cluster --verbosity 10 --name test

+ cp /etc/kind-cluster.yaml /var/lib/containers/kind-cluster.yaml
+ kind create cluster --config /var/lib/containers/kind-cluster.yaml --verbosity 10 --name test
using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
Creating cluster "test" ...
DEBUG: podman/images.go:58] Image: docker.io/kindest/node@sha256:c48c62eac5da28cdadcf560d1d8616cfa6783b58f0d94cf63ad1bf49600cb027 present locally
 ✓ Ensuring node image (kindest/node:v1.32.0) 🖼 
 ✗ Preparing nodes 📦 📦 📦 📦 📦 📦  

Deleted nodes: ["test-external-load-balancer" "test-worker" "test-control-plane2" "test-worker3" "test-control-plane3" "test-worker2" "test-control-plane"]
ERROR: failed to create cluster: could not find a log line that matches "Reached target .*Multi-User System.*|detected cgroup v1"

Stack Trace: 
sigs.k8s.io/kind/pkg/errors.Errorf
        sigs.k8s.io/kind/pkg/errors/errors.go:41
sigs.k8s.io/kind/pkg/cluster/internal/providers/common.WaitUntilLogRegexpMatches
        sigs.k8s.io/kind/pkg/cluster/internal/providers/common/cgroups.go:84
sigs.k8s.io/kind/pkg/cluster/internal/providers/podman.createContainerWithWaitUntilSystemdReachesMultiUserSystem
        sigs.k8s.io/kind/pkg/cluster/internal/providers/podman/provision.go:435
sigs.k8s.io/kind/pkg/cluster/internal/providers/podman.planCreation.func2
        sigs.k8s.io/kind/pkg/cluster/internal/providers/podman/provision.go:111
sigs.k8s.io/kind/pkg/errors.UntilErrorConcurrent.func1
        sigs.k8s.io/kind/pkg/errors/concurrent.go:30
runtime.goexit
        runtime/asm_amd64.s:1700
+ podman stop --all

# cat /etc/kind-cluster.yaml 


kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: control-plane
- role: control-plane
- role: worker
- role: worker
- role: worker
networking:
  apiServerPort: 6443
kubeadmConfigPatchesJSON6902:
- group: kubeadm.k8s.io
  version: v1beta3
  kind: ClusterConfiguration
  patch: |
    - op: add
      path: /apiServer/certSANs/-
      value: 127.0.0.1

Questions:

1. Has anyone successfully created a multi-node Kind cluster inside a Podman container?
2. Are there any specific configurations or workarounds needed for this setup?

Any insights or suggestions would be greatly appreciated. Thank you!

[adelton]

May we assume that you are able to create a single-node Kind cluster in podman fine? If that's the case and if you are on a rootful podman (judging by the fact that you don't have the KubeletInUserNamespace=true configuration in your kind-cluster.yaml), then merely adding the node to the nodes list should work. In 9af5685 I show adding a worker node, and https://github.com/adelton/kind-in-pod/actions/runs/12398023111 shows that the tests pass.

For debugging, I'd recommend to create the Kind cluster with --retain and then

[root@container /]# podman exec kind-worker journalctl -l

to see what errors you'll find there. That's how I figured out that a InitConfiguration does not work on the second node which led me to find JoinConfiguration.

I do see the cluster failing when I have more than one control plane node, so hopefully once you've debugged your specific setup with a control-plane + worker, you will be able to report what needs to be done for multiple control planes.

We could then amend the documentation and CI with your findings.

[8-cm]

Well - Single node cluster is working - except error in picture with self-signed certificate (default with your script probably).

Same for coredns pods

[adelton]

Please don't do images to show the outputs, they are not very accessible. Use ``` blocks to paste text.

[adelton]

It seems the multi-control-plane setups are failing even in some docker scenarios: kubernetes-sigs/kind#2858, probably related to the networking and the external load balancer which gets created in such setup.

[8-cm]

Sorry about the images.

I forgot to mention that I use newer version of kind than your Dockerfile.

Also I am running it in Talos Linux from SideroLabs.

I will try it with your exact version.

[8-cm]

It seems the multi-control-plane setups are failing even in some docker scenarios: kubernetes-sigs/kind#2858, probably related to the networking and the external load balancer which gets created in such setup.

It was not issue only for multiple control-planes. I tried one control-plane with two workers and the failure was same as in case of 3 control-planes with multiple workers.