[metrics] Add quota_reserved_wait_time_seconds #1977

mbobrovskyi · 2024-04-12T10:45:59Z

What type of PR is this?

/kind bug
/kind feature

What this PR does / why we need it:

Fixed metrics admitted_workloads_total, admission_wait_time_seconds.
Added metrics quota_reserved_workloads_total, quota_reserved_wait_time_seconds.

Which issue(s) this PR fixes:

Fixes #1961

Does this PR introduce a user-facing change?

Improve metrics related to workload's quota reservation and admission:
- fix admission_wait_time_seconds - to measure the time to "Admitted" condition since creation time or last requeue (as opposed to the "QuotaReserved" condition as before)
- add quota_reserved_wait_time_seconds - measures time to "QuotaReserved" condition since creation time, or last eviction time
- add quota_reserved_workloads_total - counts the number of workloads that got admitted
- admission_checks_wait_time_seconds - measures the time to admit a workload with admission checks since quota reservation
- use longer buckets (up to 10240s) for histogram metrics: admission_wait_time_seconds, quota_reserved_wait_time_seconds, admission_checks_wait_time_seconds

…me metrics.

k8s-ci-robot · 2024-04-12T10:46:08Z

Hi @mbobrovskyi. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

netlify · 2024-04-12T10:46:15Z

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	`a33f33e`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/66265ea2242094000847962a

trasc · 2024-04-12T15:31:32Z

/assign
/ok-to-test
/test all

k8s-ci-robot · 2024-04-12T15:37:25Z

@trasc: /release-note-edit must be used with a release note block.

In response to this:

/release-note-edit

Improve metrics relating to workload's quota reservation and admission

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2024-04-12T15:38:50Z

@trasc: /release-note-edit must be used with a single release note block.

In response to this:

/release-note-edit

Improve metrics relating to workload's quota reservation and admission

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2024-04-12T16:43:07Z

@trasc: /release-note-edit must be used with a release note block.

In response to this:

/release-note-edit

Improve metrics relating to workload's quota reservation and admission

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

trasc · 2024-04-15T14:22:01Z

/lgtm
/assign @mimowo

trasc

/lgtm

k8s-ci-robot · 2024-04-22T07:19:19Z

LGTM label has been added.

Git tree hash: c9c1f4ba99a1d94dc4b267733da06e284f80195a

mimowo

Almost lgtm

mimowo · 2024-04-22T07:51:14Z

pkg/controller/core/workload_controller.go

+			queuedTime := wl.CreationTimestamp.Time
+			if c := apimeta.FindStatusCondition(wl.Status.Conditions, kueue.WorkloadRequeued); c != nil {
+				queuedTime = c.LastTransitionTime.Time
+			}
+			queuedWaitTime := time.Since(queuedTime)


nit: extract this as a helper in workload.go to commonize the analogous code with scheduler.go. Maybe QueuedWaitTime(workload)

mimowo · 2024-04-22T07:52:00Z

pkg/workload/workload.go

+	condition := metav1.Condition{
+		Type:               kueue.WorkloadRequeued,
+		Status:             metav1.ConditionTrue,
+		LastTransitionTime: metav1.Now(),


nit: this is set inside apimeta.SetStatusCondition(

mimowo · 2024-04-22T07:53:25Z

apis/kueue/v1beta1/workload_types.go

+	// WorkloadRequeued means that the Workload was requeued.
+	// The possible reasons for this condition are:
+	// - On setting QuotaReserved=False


Suggested change

// WorkloadRequeued means that the Workload was requeued.

// The possible reasons for this condition are:

// - On setting QuotaReserved=False

// WorkloadRequeued means that the Workload was requeued due to eviction.

Let's make the comment meaningful for the users who may observe the conditions.
Kueue will set QuotaReserved=False for evicted workloads.

mimowo · 2024-04-22T08:00:30Z

test/integration/scheduler/scheduler_test.go

+				Request(corev1.ResourceCPU, "2").
+				Obj()
+
+			ginkgo.By("checking the first workload gets created and admitted", func() {


Suggested change

ginkgo.By("checking the first workload gets created and admitted", func() {

ginkgo.By("checking the first workload gets created and gets quota reserved", func() {

mimowo · 2024-04-22T08:03:42Z

apis/kueue/v1beta1/workload_types.go

@@ -302,6 +302,11 @@ const (
 	// more detailed information. The more detailed reasons should be prefixed
 	// by one of the "base" reasons.
 	WorkloadPreempted = "Preempted"
+
+	// WorkloadRequeued means that the Workload was requeued.


Please make sure we have an integration test which checks that:

Requeued=True when QuotaReserved=True and Admitted=False

Requeued=False when the workload is evicted (IIUC this may still need implementation)

Requeued=False when the workload is evicted (IIUC this may still need implementation)

Could you please explain this case? Is it possible to set Requeued to false?

Sure, we could set it to false, and when we set Admitted=True condition. We do something similar for the other conditions when settting QuotaReserved: true:

kueue/pkg/workload/workload.go

Lines 405 to 417 in 7880819

if evictedCond := apimeta.FindStatusCondition(w.Status.Conditions, kueue.WorkloadEvicted); evictedCond != nil {

evictedCond.Status = metav1.ConditionFalse

evictedCond.Reason = "QuotaReserved"

evictedCond.Message = "Previously: " + evictedCond.Message

evictedCond.LastTransitionTime = metav1.Now()

}

// reset Preempted condition if present.

if preemptedCond := apimeta.FindStatusCondition(w.Status.Conditions, kueue.WorkloadPreempted); preemptedCond != nil {

preemptedCond.Status = metav1.ConditionFalse

preemptedCond.Reason = "QuotaReserved"

preemptedCond.Message = "Previously: " + preemptedCond.Message

preemptedCond.LastTransitionTime = metav1.Now()

}

OTOH, setting it to False is not required to compute the metric, so maybe it does not need to be in this PR. WDYT @alculquicondor ?

As discussed offline, we don't want to change the condition to False on admission.

Could you just add an assert the it remains True when QuotaReserved=True, and Admitted=True?

The condition should transition to False in the following scenarios:

When workload.spec.active=false. It should return to True when it's reactivated.

When evicted due to WaitForPodsReady, the workload will temporarily be in backoff. During this time, Requeued should be false.

Although I think we can leave these for a follow up.

…ndition.

alculquicondor

/approve
Leaving lgtm to @mimowo

alculquicondor · 2024-04-22T15:01:45Z

pkg/controller/jobs/job/job_controller_test.go

@@ -633,6 +633,12 @@ func TestReconciler(t *testing.T) {
 						Reason:  "Pending",
 						Message: "The workload is deactivated",
 					}).
+					Condition(metav1.Condition{
+						Type:    kueue.WorkloadRequeued,
+						Status:  metav1.ConditionTrue,


This is not correct. If the workload is deactivated, Requeued should be False.

alculquicondor · 2024-04-22T15:22:54Z

apis/kueue/v1beta1/workload_types.go

@@ -302,6 +302,11 @@ const (
 	// more detailed information. The more detailed reasons should be prefixed
 	// by one of the "base" reasons.
 	WorkloadPreempted = "Preempted"
+
+	// WorkloadRequeued means that the Workload was requeued.


The condition should transition to False in the following scenarios:

When workload.spec.active=false. It should return to True when it's reactivated.

When evicted due to WaitForPodsReady, the workload will temporarily be in backoff. During this time, Requeued should be false.

Although I think we can leave these for a follow up.

k8s-ci-robot · 2024-04-22T15:26:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, mbobrovskyi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [alculquicondor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mimowo · 2024-04-23T06:29:18Z

/lgtm
Let's address the remaining comments (test and transitions) in follow up(s).

k8s-ci-robot · 2024-04-23T06:29:23Z

LGTM label has been added.

Git tree hash: d813f5fde2705e8a3177eeb00b5269ec4a993645

mimowo · 2024-04-23T06:42:24Z

/lgtm Let's address the remaining comments (test and transitions) in follow up(s).

I opened the issue to track the remaining work for transitioning: #2038.

mimowo · 2024-04-23T07:11:33Z

/release-note-edit

Improve metrics related to workload's quota reservation and admission:
- fix admission_wait_time_seconds - to measure the time to "Admitted" condition since creation time or last requeue (as opposed to the "QuotaReserved" condition as before)
- add quota_reserved_wait_time_seconds - measures time to "QuotaReserved" condition since creation time, or last eviction time
- add quota_reserved_workloads_total - counts the number of workloads that got admitted
- admission_checks_wait_time_seconds - measures the time to admit a workload with admission checks since quota reservation
- use longer buckets (up to 10240s) for histogram metrics: admission_wait_time_seconds, quota_reserved_wait_time_seconds, admission_checks_wait_time_seconds

tenzen-y · 2024-04-24T08:12:04Z

@mimowo As I can see this PR, the semantics of the existing metric (admitted_workloads_total and admission_wait_time_seconds ) are reworked with breaking. So, shouldn't add the ACTION REQUIRED in the release note?

mimowo · 2024-04-24T08:24:33Z

@mimowo As I can see this PR, the semantics of the existing metric (admitted_workloads_total and admission_wait_time_seconds ) are reworked with breaking. So, shouldn't add the ACTION REQUIRED in the release note?

I don't think admitted_workloads_total changed, we just added quota_reserved_workloads_total.
Regarding admission_wait_time_seconds I would say it was fixed for workloads with admission checks.
It does not seem to me user action is needed.

tenzen-y · 2024-04-24T17:28:06Z

I don't think admitted_workloads_total changed, we just added quota_reserved_workloads_total.

Oh, I see. Thank you for the clarifications. I was thinking that the metric had been changed.

Regarding admission_wait_time_seconds I would say it was fixed for workloads with admission checks.
It does not seem to me user action is needed.

That makes sense.

mbobrovskyi added 2 commits April 12, 2024 10:54

[metrics] Created QuotaReservedWorkloadsTotal and quotaReservedWaitTi…

82c5ab8

…me metrics.

[metrics] Added integration tests.

9937357

k8s-ci-robot requested review from denkensk and kerthcet April 12, 2024 10:46

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 12, 2024

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Apr 12, 2024

[metrics] Fixed imports.

a5896fa

k8s-ci-robot assigned trasc Apr 12, 2024

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 12, 2024

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Apr 15, 2024

mbobrovskyi added 3 commits April 15, 2024 15:07

[metrics] Added new metric doc.

a3ef4a6

[metrics] Added new metric doc.

6c4abed

[metrics] Added new metric doc.

eef84ac

mbobrovskyi marked this pull request as ready for review April 15, 2024 14:19

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 15, 2024

k8s-ci-robot requested a review from trasc April 15, 2024 14:19

mbobrovskyi added 2 commits April 17, 2024 14:58

[metrics] Added generateExponentialBuckets test.

3dffb30

[metrics] Added WorkloadRequeued condition.

8a9b478

trasc reviewed Apr 22, 2024

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 22, 2024

mimowo reviewed Apr 22, 2024

View reviewed changes

mbobrovskyi added 4 commits April 22, 2024 11:16

[metrics] Change explanation for WorkloadRequeued.

e8e9a89

[metrics] Remove extra argument (LastTransitionTime) on SetRequeuedCo…

ccf69ad

…ndition.

[metrics] Added QueuedWaitTime helper.

96ec56b

[metrics] Rename test doc.

a33f33e

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 22, 2024

k8s-ci-robot requested review from mimowo and trasc April 22, 2024 12:57

alculquicondor reviewed Apr 22, 2024

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 22, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 23, 2024

mimowo mentioned this pull request Apr 23, 2024

Fix transitions of Requeued condition #2038

Closed

k8s-ci-robot merged commit 92baacd into kubernetes-sigs:main Apr 23, 2024
15 checks passed

k8s-ci-robot added this to the v0.7 milestone Apr 23, 2024

mbobrovskyi deleted the fix/admission_wait_time_seconds branch April 23, 2024 18:38

mbobrovskyi mentioned this pull request Apr 26, 2024

REQUEST: New membership for mbobrovskyi kubernetes/org#4915

Closed

11 tasks

woehrl01 mentioned this pull request May 29, 2024

Add the possibility to define bucket sizes for kueue_admission_wait_time_seconds_bucket #1622

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[metrics] Add quota_reserved_wait_time_seconds #1977

[metrics] Add quota_reserved_wait_time_seconds #1977

mbobrovskyi commented Apr 12, 2024 •

edited by k8s-ci-robot

Loading

k8s-ci-robot commented Apr 12, 2024

netlify bot commented Apr 12, 2024 •

edited

Loading

trasc commented Apr 12, 2024

k8s-ci-robot commented Apr 12, 2024

k8s-ci-robot commented Apr 12, 2024

k8s-ci-robot commented Apr 12, 2024

trasc commented Apr 15, 2024

trasc left a comment

k8s-ci-robot commented Apr 22, 2024

mimowo left a comment

mimowo Apr 22, 2024

mimowo Apr 22, 2024

mimowo Apr 22, 2024

mimowo Apr 22, 2024

mimowo Apr 22, 2024

mbobrovskyi Apr 22, 2024

mimowo Apr 22, 2024

mimowo Apr 22, 2024

mimowo Apr 22, 2024

mimowo Apr 22, 2024

alculquicondor Apr 22, 2024

alculquicondor left a comment

alculquicondor Apr 22, 2024

alculquicondor Apr 22, 2024

k8s-ci-robot commented Apr 22, 2024

mimowo commented Apr 23, 2024

k8s-ci-robot commented Apr 23, 2024

mimowo commented Apr 23, 2024

mimowo commented Apr 23, 2024

tenzen-y commented Apr 24, 2024

mimowo commented Apr 24, 2024

tenzen-y commented Apr 24, 2024

	ginkgo.By("checking the first workload gets created and admitted", func() {
	ginkgo.By("checking the first workload gets created and gets quota reserved", func() {

	if evictedCond := apimeta.FindStatusCondition(w.Status.Conditions, kueue.WorkloadEvicted); evictedCond != nil {
	evictedCond.Status = metav1.ConditionFalse
	evictedCond.Reason = "QuotaReserved"
	evictedCond.Message = "Previously: " + evictedCond.Message
	evictedCond.LastTransitionTime = metav1.Now()
	}
	// reset Preempted condition if present.
	if preemptedCond := apimeta.FindStatusCondition(w.Status.Conditions, kueue.WorkloadPreempted); preemptedCond != nil {
	preemptedCond.Status = metav1.ConditionFalse
	preemptedCond.Reason = "QuotaReserved"
	preemptedCond.Message = "Previously: " + preemptedCond.Message
	preemptedCond.LastTransitionTime = metav1.Now()
	}

[metrics] Add quota_reserved_wait_time_seconds #1977

[metrics] Add quota_reserved_wait_time_seconds #1977

Conversation

mbobrovskyi commented Apr 12, 2024 • edited by k8s-ci-robot Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Does this PR introduce a user-facing change?

k8s-ci-robot commented Apr 12, 2024

netlify bot commented Apr 12, 2024 • edited Loading

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

trasc commented Apr 12, 2024

k8s-ci-robot commented Apr 12, 2024

k8s-ci-robot commented Apr 12, 2024

k8s-ci-robot commented Apr 12, 2024

trasc commented Apr 15, 2024

trasc left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Apr 22, 2024

mimowo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Apr 22, 2024

mimowo commented Apr 23, 2024

k8s-ci-robot commented Apr 23, 2024

mimowo commented Apr 23, 2024

mimowo commented Apr 23, 2024

tenzen-y commented Apr 24, 2024

mimowo commented Apr 24, 2024

tenzen-y commented Apr 24, 2024

mbobrovskyi commented Apr 12, 2024 •

edited by k8s-ci-robot

Loading

netlify bot commented Apr 12, 2024 •

edited

Loading