-
Notifications
You must be signed in to change notification settings - Fork 87
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
EKS tries to create system processes (kube-system namespace) incorrectly on Admiralty virtual nodes #158
Comments
It looks like the virtual nodes have taints, but that the kubectl describe node admiralty-default-multi-cloud-test-cluster-c962bad2df | grep Taints
virtual-kubelet.io/provider=admiralty:NoSchedule |
I added a label to the EKS nodes and patched the |
You can exclude labels from being picked up, cf. #115. |
Thanks for the recommendation @adrienjt. I should point out that my attempts to tweak Maybe I'm misunderstanding, but I think this conflict between Admiralty and EKS is something Admiralty needs to address, even if it's in documentation. Altering the Or am I missing something? |
Scenario
Problem Description
Certain pods get stuck in "Terminating" status, apparently because K8s is trying to schedule processes on the Admiralty virtual nodes. Not sure if this is connected to the use of spot instances (where nodes disappear). If it is connected, I'd expect this would also affect autoscaling groups (unconfirmed).
Issue
Admiralty appears to be confusing EKS. EKS is trying to schedule pods like
kube-proxy
,ebs-csi-node
, andaws-node
on Admiralty virtual nodes and failing.Is there a taint we should be putting on the EKS-supplied nodes?
Observations
Representative (truncated output) pod states after nodes disappear:
Investigate one of the stuck pods (note 10250 is the port used by Admiralty) The x.x.x.114 address no longer exists in the cluster, so I'm guessing it previously belonged to a pod on a node that no longer exists (spot instance).
Force kill the pod to clean up, and K8s spawns a new one, but it stays stuck in Pending:
K8s is trying to run start the pod on a virtual node for some reason:
kubectl -n kube-system describe pod aws-node-jcrqf |tail -4 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 40m default-scheduler Successfully assigned kube-system/aws-node-jcrqf to admiralty-default-app-gke-cluster-1998d80ea2
Other pods are running on EKS nodes. The "pending" and "terminating" pods are scheduled on Admiralty virtual nodes.
Double check that the kube-system namespace does not have the Admiralty label:
The text was updated successfully, but these errors were encountered: