-
Notifications
You must be signed in to change notification settings - Fork 198
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Better scheduling with low-priority pods #1437
Comments
This issue is currently awaiting triage. If Karpenter contributors determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Can you share all of the DaemonSet manifests, workload pods, and NodePool/NodeClass configuration that you have on your cluster? Ideally, we can repro this behavior as well as see all of the scheduling constraints for the manifests that you have so we can see if Karpenter is thinking all of the DS are compatible for some reason when they shouldn't be. |
You can see the specs in aws/karpenter-provider-aws#6507. |
@WxFang that issue does not seem to include DaemonSet pods, and only the low priority pod, no? |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
Description
What problem are you trying to solve?
We're trying to migrate gpu workloads to karpenter. We have also deployed a daemonset called cost-placeholder. The idea is to deploy low-priority pods for better cost attribution of idle cost. And the spec looks as below:
But Karpenter is unable to schedule pods onto those gpu nodes because it counts the low-priority pod into workload thus the requested resources is over capacity:
Related issue: aws/karpenter-provider-aws#6507
How important is this feature to you?
This is really important for our cost attribution.
GPU machines are expansive and have low availability and we use ODCR to reserve them. That means sometimes the machines can be empty and it's hard to attribute the (idle) cost since no workloads are running.
So, we launch low-priority pods for this purpose. Without this feature, low-priority placeholder pods will block any pods to schedule onto those machines.
The text was updated successfully, but these errors were encountered: