-
Notifications
You must be signed in to change notification settings - Fork 228
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for max runtime for workloads #405
Comments
What do you need in addition to terminating the job once the deadline is hit? Such field already exists in Job: https://kubernetes.io/docs/concepts/workloads/controllers/job/#job-termination-and-cleanup |
Yea I saw that as I posted this. In the Kueue design docs, it talks about having workloads as an abstraction for Jobs/CustomJobObjects. That link is obviously specific to Jobs but I was wondering for other custom controllers. But maybe we can call it solved for the JobController at the moment. And other Job implementations would need their own deadline implemented. And are you okay with the addition to CQ? Even if most users puts a deadline on their job, it may be good to have default limits on Queues for those that do not. |
#83 discusses this topic a bit, however, as a signal for preemption, instead of a hard deadline. We can certainly discuss making it more aggressive. As for defaults, probably LocalQueue might be enough. I'm working on a KEP for preemption that I hope to share next week. |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
/lifecycle rotten |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close not-planned |
@k8s-triage-robot: Closing this issue, marking it as "Not Planned". In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
@kannon92 please have a look at the preemption KEP https://github.com/kubernetes-sigs/kueue/tree/main/keps/83-workload-preemption #83 Please reopen if you would like to still have some form of max runtime even with preemption. |
Also you could have a look at #477 |
Yea no worries. I created this when I was thinking a bit about interactive jobs. But not sure if there was a strong interest from anyone on that one. I think it'd be nice to eventually have a max walltime on a queueing solution but I think there are higher priority items that are more useful atm |
What would you like to be added:
It would be useful if ClusterQueues had a way to enforce walltime limit.
This would enforce a walltime limit for any workload that is admitted into the ClusterQueue.
Why is this needed:
In HPC schedulers, it is common to have multiple layers where users can request a job with a set walltime limit. It is also common for admins to set up walltime limits.
A Batch user can submit a workload and should also be able to request a walltime limit for their job. This can be useful because it allows a user to have greater control over how long their job will run for.
An admin can design queues with various lengths for run time. I've seen this in some HPC centers as a way to have dedicated queues for short lived jobs.
One question I have is where should the user requested walltime limit live?
Completion requirements:
This enhancement requires the following artifacts:
The artifacts should be linked in subsequent comments.
The text was updated successfully, but these errors were encountered: