Creating a kind cluster as a AMD64 on a host ARM 64 machine

[gowrav27]

What am I trying to do:
Trying to build a Service,, lets say Service A, as an image in AMD on a ARM hostmachine
And running that machine on K8 which is started by Kind which is possible should be AMD 64.

They were other articles on using UTM etc... But I thought of giving this a shot

The process is as follows,

1. Service --> Built my service A, as AMD 64 by using the docker command with argument docker build --platform linux/amd64 -t servicea .
2. Kind creates a control plane --> then, tried to create a cluster, using the command [ ```
   kind create cluster
   --config=./kind-cluster.yaml
   --name=ServiceA]

This step (2) creates a machine in ARM 64

So, I thought of creating the machine itself as a AMD 64 by replacing the kind create cluster command with this

docker run --platform linux/amd64 --hostname servicea-control-plane --name servicea-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro --detach --tty --label io.x-k8s.kind.cluster=servicea --net kind --restart=on-failure:1 --init=false --publish=0.0.0.0:9090:30000/TCP --publish=127.0.0.1:61145:6443/TCP -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6

What happened:
When I tried the above command and started the container from the image, I got the below

INFO: setting iptables to detected mode: legacy
iptables-save v1.8.7 (legacy): Cannot initialize: iptables who? (do you need to insmod?)

What you expected to happen:
I was expecting a AMD. 64 image would have been created

Environment:

• kind version: (use kind version): kind v0.11.1 go1.16.4 darwin/amd64
• Kubernetes version: (use kubectl version): Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.13", GitCommit:"53c7b65d4531a749cd3a7004c5212d23daa044a9", GitTreeState:"clean", BuildDate:"2021-07-15T20:58:11Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"darwin/amd64"}
• Docker version: (use docker info):

-     - `Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.8.2)
  compose: Docker Compose (Docker Inc., v2.4.1)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 2
 Server Version: 20.10.14
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.104-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 5
 Total Memory: 17.55GiB
 Name: docker-desktop
 ID: 2ZN7:DBMC:E52N:O5UN:OI4S:2U23:BS2K:FMLL:KPKQ:KCW5:3PPV:E4CD
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false`

• OS
  

[BenTheElder]

Creating a kind cluster as a AMD64 on a host ARM 64 machine #2738

This is not supported and will not work, you need to use the architecture on which the docker daemon is running.

iptables-save v1.8.7 (legacy): Cannot initialize: iptables who? (do you need to insmod?)

... and that is why, qemu userspace / binfmt_misc is a cool trick, but it's not sufficient for running something complex like a kind node.

We do not intend to support this. You need to run a virtual or real machine of the desired architecture and run docker in it.

We've had recent issues discussing this in #2718 #2735

[BenTheElder]

2. Kind creates a control plane --> then, tried to create a cluster, using the command [ ```
   kind create cluster
   --config=./kind-cluster.yaml
   --name=ServiceA]

This step (2) creates a machine in ARM 64

That's not accurate. KIND creates docker containers running multiple processes inside them.

So, I thought of creating the machine itself as a AMD 64 by replacing the kind create cluster command with this

We don't support creating clusters by running docker commands without the kind tool. There's a lot that needs to be orchestrated at runtime.

I was expecting a AMD. 64 image would have been created

It would be an amd64 container (not image), and there is one, but not in a way that can run complex workloads like kind.

docker run --platform multiarch support is basically https://wiki.debian.org/QemuUserEmulation which is good enough to do builds but not to run a kind node.

For more on what docker is doing and some idea why it won't work: https://ownyourbits.com/2018/06/13/transparently-running-binaries-from-any-architecture-in-linux-with-qemu-and-binfmt_misc/

[gowrav27]

Thanks @BenTheElder for the information,
Sorry about my statements, didnt explain myself clearly in the problem statement

[BenTheElder]

not at all! 😅

[linkous8]

@BenTheElder any chance running an amd64 kind node on m1 mac is more feasible now that they've added rosetta support?

https://levelup.gitconnected.com/docker-on-apple-silicon-mac-how-to-run-x86-containers-with-rosetta-2-4a679913a0d5

[BenTheElder]

It should either work without us doing anything, or it won't work.

I don't have docker desktop so someone else will have to test is Rosetta is being used in a way that works for this purpose.

You can do so most clearly by explicitly using an amd64 image digest. So for example crane manifest $image => pick out the amd64 entry and use kindest/node@sha256:.... in `--image

[linkous8]

Quick followup. I've just tried it and it didn't work unfortunately 😞 It timed out waiting for the kubelet to become healthy. Thanks for the steps on how to try this out

% kind create cluster --image kindest/node@sha256:f1de3b0670462f43280114eccceab8bf1b9576d2afe0582f8f74529da6fd0365
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✗ Starting control-plane 🕹️ 
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged kind-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
Command Output: I0208 20:24:53.917401     129 initconfiguration.go:254] loading configuration from "/kind/kubeadm.conf"
...
I0208 20:26:51.865035     129 round_trippers.go:553] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I0208 20:26:52.369306     129 round_trippers.go:553] GET https://kind-control-plane:6443/healthz?timeout=10s  in 2 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.

[arumugamsubramanian]

Hi all, I also get the same results. Is there any other solution available? Most of my app images are in amd64 and I would like to use that in local KIND cluster in m1 machine. KIND cluster can able to run the app amd64 images but performance is not so good.

[BenTheElder]

Unfortunately this is not going to work great, I highly recommend porting images to the host architecture.