-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Node groups get "stuck" during node deletion #3949
Comments
+1 It is now a recurring issue. Looking for a resolution. |
Looking at the code the problem seems to be that here:
readiness.Deleted is not considered. However, just adding this to the equation will most likely only solve half of the problem. The other half of the problem is that the ToBeDeleted taint is not removed from a node when the autoscaler is interrupted after it has added this taint but not yet deleted the node. There are a couple of defer sections that are supposed to do this in case the node could not be drained successfully, but this obviously doesn't run if the autoscaler is killed. Old ToBeDeleted taints are ignored and the affected nodes are eligible for deletion again, however, it can happen that in the mean time the overall cluster utilization situation has shifted and the node is no longer "unneeded", in which case there will be no more retries to deleted it unless the utilization situation changes again.
|
Cluster Autoscaler is expected to remove all ToBeDeletedByClusterAutoscaler taints when starting up: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/core/static_autoscaler.go#L174. The new CA leader should execute this logic on its first autoscaling loop. |
@MaciekPytel I cannot find this error message:
that should have been logged if this would have happened in any of the logs, so I suppose failing at this stage wouldn't help much. Is it possible that the lister didn't fail but just returned an empty list because the cache wasn't populated yet? |
I think I have finally figured out the mechanisms of action for this bug:
|
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
/remove-lifecycle stale |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
/remove-lifecycle stale |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
/remove-lifecycle stale |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
Which component are you using?: cluster-autoscaler
What version of the component are you using?: 1.17.4
Component version: 1.17.4
What k8s version are you using (
kubectl version
)?:kubectl version
OutputWhat environment is this in?: AWS EKS with self-managed node groups
What happened?:
The leading cluster-autoscaler pod was apparently disrupted in the middle of draining and deleting a node. The affected node had both the
ToBeDeletedByClusterAutoscaler
andDeletionCandidateOfClusterAutoscaler
taints but there were still pods running and it was not deleted from EC2. Afterwards, the node group that this node is part of was apparently kind of "stuck" as cluster-autoscaler didn't scale it up anymore although there was a pending pod that would have been able to run on it. According to the logs, cluster-autoscaler believed that there was a node coming up to run this pod, but this was not the case, the Auto Scaling Group was not scaled up, all nodes in the ASG were registered and ready in Kubernetes:The
cluster-autoscaler-status
config map inkube-system
showed the following for the affected node group:So it didn't consider the affected node as
ready
and (I assume) thought that there was a node coming up becausecloudProviderTarget - ready > 0
. At the same time, it didn't delete the node because there were still pods running on it.Manually removing the
ToBeDeletedByClusterAutoscaler
taint from the affected node resolved the situation, causing the node group to be scaled up as expected.The text was updated successfully, but these errors were encountered: