-
Notifications
You must be signed in to change notification settings - Fork 904
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Karpenter pod capacity exceeds available IPs from ENIs with other CNIs #5780
Comments
Can you explain more about how Cilium's (and maybe other CNIs too) calculation of maxPods differs from the AWS calculation? Is it also a static value like a |
My interpretation of:
is that the final "+2" comes from this comment:
We don't have kube-proxy pods, only cilium pods are using host networking. So should this be a |
I disagree that this is a feature request. Without this resolved, using ENI IPAM, you can enter states where pods can schedule to a node and never start due to a node's pod capacity being set higher than a node's ENI capacity. If that is intended Karpenter operation this behavior should be documented somewhere. |
Description
Observed Behavior: We've observed Karpenter creates nodes with a pod capacity higher than is actually supported by the instance type's ENI's. We observed this most recently with a "g4ad.4xlarge" instance node, which with our Karpenter configuration (reserved ENIs = 1) was assigned a pod capacity of "20" pods, however, we had a pod that was not able to start, and after investigating, it was unable to be assigned an IP from Cilium.
We run clusters with Cilium and use Cilium to replace kube-proxy, so there are no kube-proxy pods that run on our nodes. When we were debugging this behavior we noticed that the calculation Karpenter uses for the node's pod capacity is based on this:
This does come out to 20 when supplying the variables for the g4ad.4xlarge:
(3 ENIs - 1 Reserved) * (10 - 1) + 2 = 20
. So Karpenter is following this formula, however, if there's no kube-proxy pod and only 1 CNI pod, then would that make this math wrong?Expected Behavior: Expected the node's pod capacity to match the number of pods that can run on the node.
Reproduction Steps (Please include YAML): N/A
Versions:
kubectl version
): v1.26.12-eks-5e0fddeThe text was updated successfully, but these errors were encountered: