Disruption budgets not workings as expected/documented

[komapa]

Description

Observed Behavior:

Hello, we have the the following disruption budget on a NodePool:

budgets:
    - nodes: "1"
    - duration: 60m
      nodes: 1%
      schedule: '@hourly'
    - duration: 17h
      nodes: "0"
      schedule: 0 9 * * *

Expected Behavior:

We expect, for a NodePool with around 200 nodes, to disrupt at most 2 nodes per hour out side of the 17h where we define no disruptions (nodes: 0). What we see though is disruptions happening at the a much higher rate. see the attached graph of karpenter.karpenter_interruption_actions_performed.count metric.

Reproduction Steps (Please include YAML):

budgets:
    - nodes: "1"
    - duration: 60m
      nodes: 1%
      schedule: '@hourly'
    - duration: 17h
      nodes: "0"
      schedule: 0 9 * * *

Versions:

• Chart Version: 0.35.2
• Kubernetes Version (kubectl version): 1.28.x

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[k8s-ci-robot]

This issue is currently awaiting triage.

If Karpenter contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[miadabrin]

might also be related to this #1421

[engedaam]

@komapa Are you using Spot instances? karpenter_interruption_actions_performed is a metric that is produced by the karpenter-provider-aws. That metric is emitted due to Spot interruption events, which is not a form of voluntary disruption. karpenter_interruption_actions_performed is expected not to respect disruption budgets, since spot interupation events can occur within a given disruption window. https://karpenter.sh/docs/concepts/disruption/#interruption.