-
Notifications
You must be signed in to change notification settings - Fork 170
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Karpenter should taint nodes on startup with karpenter.sh/unregistered:NoSchedule
and remove once the node is registered
#1049
Karpenter should taint nodes on startup with karpenter.sh/unregistered:NoSchedule
and remove once the node is registered
#1049
Comments
karpenter.sh/unregistered
and remove once the node is registeredkarpenter.sh/unregistered:NoSchedule
and remove once the node is registered
There is some relevant conversation around this topic that I found here as I was exploring what it would look like if Karpenter were to allow users to specify the |
|
Do we know why the external cloudprovider implementation uses the |
Hi @jonathan-innis . Can I offer help to work on this task ? |
As far as I can tell, the |
I'm not sure it really makes a huge difference because the taint exists on creation so nothing should schedule anyways. |
Fair enough, in that case then I think 'NoEffect' still works because if a cx would want a pod to tolerate the taint then they have more control over how with access to 'tolerationSeconds'. WDYT? |
If the pod is never scheduled to the node in the first place, I don't think it makes any difference. Seems fine either way to me. |
Description
What problem are you trying to solve?
Karpenter currently doesn't taint nodes in its userData, but it chooses to propagate some labels, annotations, etc. down to nodes when they join the cluster. There is currently a race that is present between the Karpenter controller's apply of the labels to the node and the kube-scheduler being able to bind pods against the node. It's possible that Karpenter may not have applied labels yet when the kube-scheduler determines that it can start to schedule pods against the node.
This problem is more prevalent with DaemonSets, where they can tolerate the
node.kubernetes.io/not-ready
taint, meaning that they can be scheduled against the node as soon as it joins. This is definitely less of a problem for application pods that have to wait for the node to go ready, but can still be an issue if the Karpenter controller is down.This also isn't a large issue, in general, since the Karpenter cloudproviders will generally pass all of the labels down from a NodeClaim into the relevant startup script that is passed to the kubelet. This means that those labels will be there atomically on kubelet start, which, again, removes the chance that this race can occur.
Realistically, we shoudn't have to rely on this behavior and should just take a stance, similar to the external cloudprovider implementation with its taint that a node gets tainted on startup with a known taint and Karpenter removes the taint from the node once it has finished its registration process.
The text was updated successfully, but these errors were encountered: