-
Notifications
You must be signed in to change notification settings - Fork 497
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[coscheduling] Infinite scheduling loop of distributed tasks block all other pods #429
Comments
minResource is optional and pre-gates the group's res req in a best-efforts manner, so it's per design. The root cause is two-folded:
|
Note that the 2nd cause is not a regression of #408 - the |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
/remove-lifecycle stale |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
/remove-lifecycle stale |
@Huang-Wei Hi, if no one working on this, I can help to add podGroup backoff time. |
@KunWuLuan sure, appreciate that. And please write a test to reproduce the issue and ensure the fix can at least remediate this issue. |
thanks, I will do that. : ) |
I have added the test for this case. 😄 |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
#559 has been merged. |
Yes. thanks for the reminder. |
Recently, we encountered some infinite scheduling loop issues when using Coscheduling in our production environment.
Suppose the following scenario, there are 3 instances of the distributed task, and each instance requires 4 GPU resources. The resources in the current cluster are as follows:
Obviously, there is resource fragmentation problem in the cluster. Since the total resources of the cluster can satisfy 3 instances, instance will succeeded in the PreFilter stage, the scheduling process is as follows:
Worse, whenever an instance is added to the ActiveQueue, according to the QueueSort extension point, the distributed instance will be queued in front of all later created Pods, and the above loop will block all other later created Pods, these pods will stuck in Pending state and don't have schedule opportunities at all.
In order to quickly solve the above problem, we removed the Queue extension point of Coscheduling and used the default PrioritySort plugin, but the scheduling loop will still block all lower priority tasks.
The root cause seems to be that minResource verification cannot fundamentally guarantee that all distributed instances can run normally, and whenever an instance is successfully scheduled in the Wait phase, all Sibling instances will be added to ActiveQueue through PodsToActivateKey.
The text was updated successfully, but these errors were encountered: