-
Notifications
You must be signed in to change notification settings - Fork 497
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[coscheduling] support match policy for podgroup #560
Comments
/assign |
IMO a field in PodGroup API is a better place? cc @denkensk |
A new property in spec is also feasible I think |
Isn't that what we have now?
What value should we fill in? @KunWuLuan |
yes, we have already support this case, and I think this can be the default behavior
I think we can support three values for three cases I mentioned above, like: "only-waiting", "waiting-and-running" and "oncesatisfied". Or we can let user choose the pod status they need, like: "scheduled", "running", "succeed", "failed", "oncesatisfied". For example, they can do nothing or use "scheduling.sigs.x-k8s.io/pod-group-matchpolicy"="scheduled,running" to achieve the same behavior as what we have now. @denkensk |
Can you give more details about your scenario? It will help me understand what you want cleanly. As I know, for distributed training like in PyTorch, we need the |
For example, in asynchronous training with DLRover, if a worker failed in training, maybe because of OOM or other reason, Etjob-operator will run a new worker to continue training. And if there are some workers who have complete the training job, the new worker will be blocked in Permit. In this case, we may need to include In tfjob training process, if any of the workers failed, we must delete all workers and rebuild them to continue training, and new coming workers may come before the workers being deleted. If we use |
FYI @RongbinZ @Hanyu96 @Xiaoaier-Z-L Do we also have these issues in the asynchronous training? |
Thanks for your explanation. These scenarios are mainly focused on the part of failure recovery. But I was wondering if we need to keep coscheduling in effect in failure recovery. @Huang-Wei WDYT? Because most of the pods are scheduled. coscheduling has been less effective in saving resources. For the asynchronous training part, you said |
Do you mean Is there an issue of untimely state synchronization here? And Will you delete all workers by yourself and recreate it directly in the job? Why not resubmit a new one to continue training? @KunWuLuan |
This sounds reasonable as In a real-world case, unknown pod failure is not uncommon. So providing an additional option to count
In this case, is the intention to delete all workers but keeping PS pod? and is PS and Worker pods are treated as a PodGroup? or, only Worker pods are treated as a PodGroup? |
I think |
Thanks for replying 😄. Yes, currently they both belong to the same pod group |
It's reasonable. But we can analyze the problem in depth. If you include |
Apologies, I didn't explain it clearly 😄. Currently, if any of the workers in tfjob failed, the whole job will be marked failed and we need to submit a new tfjob to restart training like you said. |
Thanks for your explanation. @KunWuLuan |
Yes. I think deleting and rebuilding a resource with the same name is a duplicate action. And Reusing an existing podgroup can preserve some historical information. @denkensk |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close not-planned |
@k8s-triage-robot: Closing this issue, marking it as "Not Planned". In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
For GPU topology, all my tasks need to be scheduled at the same time to ensure the best performance, so I hope that gang can only be satisfied when the number of pods waiting on permit is larger than the min-available.
For task of elastic training, the new workers'' arrival is random. So I hope the gang can be satisfied when the total number of the running pods and waiting pods is large than the min-available.
In some specific scenarios, the completed task Pod may not exist due to node recycling. At this time, it is impossible to count the number of completed tasks. In this case, OnceResourceSatisfied can be used.
We can use
scheduling.sigs.x-k8s.io/pod-group-matchpolicy
to declare which pods should be considered in permit.The text was updated successfully, but these errors were encountered: