Description

What problem are you trying to solve?

Karpenter currently doesn't taint nodes in its userData, but it chooses to propagate some labels, annotations, etc. down to nodes when they join the cluster. There is currently a race that is present between the Karpenter controller's apply of the labels to the node and the kube-scheduler being able to bind pods against the node. It's possible that Karpenter may not have applied labels yet when the kube-scheduler determines that it can start to schedule pods against the node.

This problem is more prevalent with DaemonSets, where they can tolerate the node.kubernetes.io/not-ready taint, meaning that they can be scheduled against the node as soon as it joins. This is definitely less of a problem for application pods that have to wait for the node to go ready, but can still be an issue if the Karpenter controller is down.

This also isn't a large issue, in general, since the Karpenter cloudproviders will generally pass all of the labels down from a NodeClaim into the relevant startup script that is passed to the kubelet. This means that those labels will be there atomically on kubelet start, which, again, removes the chance that this race can occur.

Realistically, we shoudn't have to rely on this behavior and should just take a stance, similar to the external cloudprovider implementation with its taint that a node gets tainted on startup with a known taint and Karpenter removes the taint from the node once it has finished its registration process.

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment