Cluster creation hangs or fails on "joining worker nodes" if custom node role is set

[carreter]

What happened: Creation of a cluster hangs or fails on the "Joining worker nodes" step if node-role.kubernetes.io/worker: "" is added as a label. It seems kubelet is failing to start:

I0613 17:42:08.596458     154 kubelet.go:158] [kubelet-start] Checking for an existing Node in the cluster with name "bug-test-worker" and status "Ready"
I0613 17:42:08.599004     154 round_trippers.go:553] GET https://bug-test-control-plane:6443/api/v1/nodes/bug-test-worker?timeout=10s 404 Not Found in 2 milliseconds
I0613 17:42:08.599295     154 kubelet.go:173] [kubelet-start] Stopping the kubelet
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet. This can take up to 4m0s
[kubelet-check] The kubelet is not healthy after 4m0.000408189s

Unfortunately, an error has occurred:
        The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' returned error: Get "http://localhost:10248/healthz": context deadline exceeded

What you expected to happen: Cluster should start and label nodes with appropriate role.

How to reproduce it (as minimally and precisely as possible):
kind create with the following config:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  ipFamily: ipv4
nodes:
- role: control-plane
- role: worker
  labels:
    node-role.kubernetes.io/worker: ""

Anything else we need to know?:
I believe it's something specifically about the node-role.kubernetes.io label that's breaking it. kind create with another label succeeds:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  ipFamily: ipv4
nodes:
- role: control-plane
- role: worker
  labels:
    test: test

Environment:

• kind version: (use kind version): v0.23.0
• Runtime info: (use docker info, podman info or nerdctl info): Server Version: 20.10.21
• OS (e.g. from /etc/os-release): gLinux Rodete (Debian), kernel: 6.6.15-2rodete2-amd64
• Kubernetes version: (use kubectl version): v1.30.0
• Any proxies or other special environment settings?: N/A

[carreter]

kind-bug.tar.gz

Log dump here!

[BenTheElder]

Yes, this isn't supported, you're using a namespace reserved for Kubernetes and Kubelet will not allow you to do this if it's not an approved value.

You should use a different namespace for your custom labels / values.

https://kubernetes.io/docs/reference/labels-annotations-taints/

Kubernetes reserves all labels and annotations in the kubernetes.io and k8s.io namespaces.

/remove-kind bug
/kind support
/close

[k8s-ci-robot]

@BenTheElder: Closing this issue.

In response to this:

Yes, this isn't supported, you're using a namespace reserved for Kubernetes and Kubelet will not allow you to do this if it's not an approved value.

You should use a different namespace for your custom labels / values.

https://kubernetes.io/docs/reference/labels-annotations-taints/

/remove-kind bug
/kind support
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[BenTheElder]

(also kubernetes-sigs/apiserver-network-proxy#631 (comment), and if you check the kubelet log kubelet should have a message about this)

[carreter]

Got it, makes sense. Thanks for the fast response!

[BenTheElder]

this is also #3536, we may attempt to validate these earlier