kind create cluster fails to remove control plane taint #2867
Comments
|
Can you share the logs folder from Just to confirm: Everything is running in arm64 mode, no amd64 images / binaries? (See: #2718 which has been a somewhat common issue for M1 users) |
|
Maybe to clarify: running I'm trying to run the command from inside a container, that has access to the docker-socket of the host. This container is also built for arm64 and can run other containers without issues. Alternatively if I create the cluster on the host, how can I access it from inside a container? Is there some way to access |
|
I switched from This is probably less performant, but it works for now. Closing the issue, since I probably tried something that is not officially supported. |
you can put the other container on the “kind” docker network either with the net flag or with “docker network connect” then you can use “kind export kubeconfig —internal” |
Thanks! I'll try that and compare performance. |
|
It’s weird that this is working from the host but not via docker socket in a container. I’m not sure why we’d see that. Maybe proxy config differences? |
|
I'm not sure. I need to test more to find out. I had issues with volumes/mounts before when using the hosts docker socket. I had to give the path how it is on the host instead of how it would be inside the container. |
|
I am also having this issue on my m1 macbook air when using a |
|
Nevermind, it's a problem with the name resolution from within the control-plane itself, not the caller of |
|
After some investigating. The problem is, the name resolution in the control-plane doesn't work immediately when the container is started. Adding a sleep before running the remove taint command works as a hacky fix. diff --git a/pkg/cluster/internal/create/actions/kubeadminit/init.go b/pkg/cluster/internal/create/actions/kubeadminit/init.go
index cc587940..e9778ce9 100644
--- a/pkg/cluster/internal/create/actions/kubeadminit/init.go
+++ b/pkg/cluster/internal/create/actions/kubeadminit/init.go
@@ -19,6 +19,7 @@ package kubeadminit
import (
"strings"
+ "time"
"sigs.k8s.io/kind/pkg/errors"
"sigs.k8s.io/kind/pkg/exec"
@@ -135,6 +136,7 @@ func (a *action) Execute(ctx *actions.ActionContext) error {
taintArgs := []string{"--kubeconfig=/etc/kubernetes/admin.conf", "taint", "nodes", "--all"}
taintArgs = append(taintArgs, taints...)
+ time.Sleep(5 * time.Second)
if err := node.Command(
"kubectl", taintArgs...,
).Run(); err != nil {IDK why this is the case with our environments. We are both running Thoughts @BenTheElder ? |
|
I was also able to reproduce this issue with a |
Uh, that shouldn't be the case. Sounds like a docker bug? Can you explain what you mean by " |
Just a minimal setup mounting |
Maybe... I tested with older versions of kind, and they have the same issue. Seems odd that we are only just finding it now, unless something got broken in docker. |
|
Sorry, too many things going on 😅 Does this happen if you use the docker client from the host, running on the host, instead? It sounds like this environment is broken, and I'd rather not add a sleep to hack around it (I mean it could take longer elsewhere as well) but we could consider, for example, using loopback instead. But I'd like to know more about why this is failing, something is off with the network in this setup. |
No it does not. It may be something to do with the added latency or overhead when mounting the socket somehow? I agree, the sleep is a bad hacky solution. I think using the loopback addr is likely best. |
|
Facing the same issue here - it seems highly error prone. I've observed that it seems to start working randomly after several retries. I also don't see how the socket should be an issue here, as the error is DNS resolution from within the container 🤔 Note that in my case, I've been using the Kind go library, rather than the An additional note on my setup is that I've been calling docker over TCP rather than a unix socket (running |
So the DNS resolution in the container comes from a (different) socket docker embeds. See the note about custom networks in: https://docs.docker.com/config/containers/container-networking/#dns-services If you're using kind via a container that mounts the docker socket, it's possible docker behaves differently here (?) This is not really a use case we've been focused on, kind is a statically linked go binary meant to be run on the host.
No the CLI code is a very small CLI wrapper over the public APIs, there's no special retry logic.
FYI this is unfortunately also known to have issues, off the top of my head there's no way for kind to reserve a random TCP port for the api server reliably. We do permit setting an explicit 0 port and let docker pick instead, but then on restart docker will assign another port. (don't have this handy but past discussion in the issue tracker) |
By only triggering reconciliation for spec updates, we reduce the risk of our cluster never starting due to network errors. We only need it to work once rather than twice. See kubernetes-sigs/kind#2867
|
@BenTheElder to clarify, I run a docker socket on my local and use Do you think it would make sense to add some retry mechanism to this taint call? What's happening now is that the whole creation gets rolled back after the cluster is already created, effectively due to networking. I'm wondering if it might make sense to add an exponential backoff retry (with a fairly low max) to make this more reliable. As you said, kind isn't really built for this usecase, so I'd also get if you don't want to add maintenance burden for an edge case. Thoughts? Also, just to clarify what's happening: the error here isn't coming from a call to the docker daemon. Rather, the error we're getting is coming from the |
Tentatively, that's just patching over one particular symptom of the networking being broken.
Well yes, we can't very well run a functional cluster with broken networking.
I don't think that's reasonable after the API server is up, this is a local API call executed itself on one of the control plane nodes itself. We already have an exponential retry waiting for the api server to be ready in kubeadm. Perhaps one retry, but again, this should not flake, it should be a very cheap local call, if it's failing, it's a symptom of the cluster being in a bad state of some sort. I suggested a possible solution above, but I'd like to understand what / why this is actually broken before I jump on making any changes. There's no reason resolving the container names should fail, docker is responsible for this and I've
Yes, but it only seems to be failing when the docker socket is mounted when using kind, and it seems to be related to DNS issues, which makes me think mounting the docker socket when creating the cluster is leading to somewhat broken DNS in the cluster, which doesn't make sense given my understanding of how docker implements DNS, but none of this makes sense ... The dns response for the node name should be local from docker and should be quick and reliable ™️ So far we've had no reports of this with standard local docker socket without containerizing kind itself of using docker over TCP, though I can't fathom why those are relevant. Unfortunately, without a way to replicate this, I'm reliant on you all to identify why docker containers are not reliably able to resolve themselves or what else is making this call fail. |
What happened:
I tried to create a cluster with
kind create clusterand received the error "failed to remove control plane taint"What you expected to happen:
Successfully creating a cluster
How to reproduce it (as minimally and precisely as possible):
Install
kindversionv0.14.0-arm64and callkind create clusterAnything else we need to know?:
I'm running kind inside a container with the hosts (MacOS M1 Max) docker socket mounted and I'm able to run other containers with
docker run.Logs:
Environment:
kind version):kind v0.14.0 go1.18.2 linux/arm64
kubectl version):Client Version: v1.24.3
Kustomize Version: v4.5.4
docker info):/etc/os-release):The text was updated successfully, but these errors were encountered: