Labels match but Cluster Autoscaler says "are not similar, labels do not match" when trying to balance similar node groups.

[nicksecurity]

Which component are you using?:
cluster-autoscaler

What version of the component are you using?:
Component version: 1.28.5

What k8s version are you using (kubectl version)?:

kubectl version Output

$ kubectl version
Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.9-eks-036c24b

What environment is this in?:
AWS, EKS using EC2

What did you expect to happen?:
When new nodes are added I expect them to be balanced on to 3 similar node groups, which have the same labels.

What happened instead?:
I have 3 node groups, one per AZ, the new nodes are only added to 1 node group.

The error says the labels are different, but I've checked them all, and except for a couple I've excluded, they are similar.

I0621 14:19:48.338960 1 compare_nodegroups.go:157] nodes template-node-for-eks-nodegroup-4-128-1ec80b90 and template-node-for-eks-nodegroup-3-128-f0c80b8d are not similar, labels do not match

How to reproduce it (as minimally and precisely as possible):
I scale up the pods so it will add several new nodes and then check to see which nodegroup it added them to.

Anything else we need to know?:
No

[adrianmoisey]

/area cluster-autoscaler