Description

Observed Behavior: We've observed Karpenter creates nodes with a pod capacity higher than is actually supported by the instance type's ENI's. We observed this most recently with a "g4ad.4xlarge" instance node, which with our Karpenter configuration (reserved ENIs = 1) was assigned a pod capacity of "20" pods, however, we had a pod that was not able to start, and after investigating, it was unable to be assigned an IP from Cilium.

We run clusters with Cilium and use Cilium to replace kube-proxy, so there are no kube-proxy pods that run on our nodes. When we were debugging this behavior we noticed that the calculation Karpenter uses for the node's pod capacity is based on this:

#
# Mapping is calculated from AWS EC2 API using the following formula:
# * First IP on each ENI is not used for pods
# * +2 for the pods that use host-networking (AWS CNI and kube-proxy)
#
#   # of ENI * (# of IPv4 per ENI - 1) + 2
#
# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI
#

This does come out to 20 when supplying the variables for the g4ad.4xlarge: (3 ENIs - 1 Reserved) * (10 - 1) + 2 = 20. So Karpenter is following this formula, however, if there's no kube-proxy pod and only 1 CNI pod, then would that make this math wrong?

Expected Behavior: Expected the node's pod capacity to match the number of pods that can run on the node.

Reproduction Steps (Please include YAML): N/A

Versions:

• Chart Version: v0.34.0
• Kubernetes Version (kubectl version): v1.26.12-eks-5e0fdde

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment