Better scheduling with low-priority pods

[WxFang]

Description

What problem are you trying to solve?
We're trying to migrate gpu workloads to karpenter. We have also deployed a daemonset called cost-placeholder. The idea is to deploy low-priority pods for better cost attribution of idle cost. And the spec looks as below:

    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: nodepool
                operator: In
                values:
                - gpu-p5-48-llm
                       
       resources:
          limits:
            cpu: "191"
            memory: 1995G
            nvidia.com/gpu: "8"
          requests:
            cpu: "191"
            memory: 1995G
            nvidia.com/gpu: "8"
      priorityClassName: low-priority
      restartPolicy: Always
      schedulerName: default-scheduler

But Karpenter is unable to schedule pods onto those gpu nodes because it counts the low-priority pod into workload thus the requested resources is over capacity:

daemonset overhead={"cpu":"191250m","memory":"1995268435456","nvidia.com/gpu":"8","pods":"7"},
no instance type satisfied resources 
{"cpu":"271250m","memory":"1995268437456","nvidia.com/gpu":"16","pods":"8"} and 
requirements karpenter.sh/capacity-type In [on-demand], 
karpenter.sh/nodepool In [gpu-p5-48-llm], 
kubernetes.io/arch In [amd64], 
kubernetes.io/os In [linux], 
node.kubernetes.io/instance-type In [p5.48xlarge], 
nodepool In [gpu-p5-48-llm], 
nvidia.com/gpu In [true] (no instance type has enough resources)

Related issue: aws/karpenter-provider-aws#6507

How important is this feature to you?
This is really important for our cost attribution.
GPU machines are expansive and have low availability and we use ODCR to reserve them. That means sometimes the machines can be empty and it's hard to attribute the (idle) cost since no workloads are running.
So, we launch low-priority pods for this purpose. Without this feature, low-priority placeholder pods will block any pods to schedule onto those machines.

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[k8s-ci-robot]

This issue is currently awaiting triage.

If Karpenter contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[jonathan-innis]

Can you share all of the DaemonSet manifests, workload pods, and NodePool/NodeClass configuration that you have on your cluster? Ideally, we can repro this behavior as well as see all of the scheduling constraints for the manifests that you have so we can see if Karpenter is thinking all of the DS are compatible for some reason when they shouldn't be.

[WxFang]

Can you share all of the DaemonSet manifests, workload pods, and NodePool/NodeClass configuration that you have on your cluster? Ideally, we can repro this behavior as well as see all of the scheduling constraints for the manifests that you have so we can see if Karpenter is thinking all of the DS are compatible for some reason when they shouldn't be.

You can see the specs in aws/karpenter-provider-aws#6507.

[engedaam]

@WxFang that issue does not seem to include DaemonSet pods, and only the low priority pod, no?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale