Make ordering timestamp configurable when Workload evicted due to Pods-ready #1542

nstogner · 2024-01-03T16:28:10Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Allow admins to choose the timestamp used for sorting workloads after eviction due to wait-for-pods-ready. See KEP-1282.

Which issue(s) this PR fixes:

Fixes #1282

Special notes for your reviewer:

Does this PR introduce a user-facing change?

WaitForPodsReady: Add a config field to allow admins to configure the timestamp used when sorting workloads that were evicted due to their Pods not becoming ready on time.

netlify · 2024-01-03T16:28:17Z

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	`edc8d7d`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/65a84a311158a60008dbd044

k8s-ci-robot · 2024-01-03T16:28:19Z

Hi @nstogner. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

nstogner · 2024-01-03T16:29:58Z

PS: I still need to add the user-story test case.

tenzen-y · 2024-01-03T17:51:24Z

/ok-to-test

tenzen-y · 2024-01-05T19:03:25Z

/assign

alculquicondor · 2024-01-10T14:10:43Z

/assign @mimowo

mimowo · 2024-01-10T14:24:27Z

/assign @mimowo

ack

apis/config/v1beta1/configuration_types.go

mimowo · 2024-01-11T09:27:57Z

apis/config/v1beta1/defaults.go

@@ -111,12 +110,15 @@ func SetDefaults_Configuration(cfg *Configuration) {
 			}
 			cfg.WaitForPodsReady.BlockAdmission = &defaultBlockAdmission
 		}
+		if cfg.WaitForPodsReady.RequeuingTimestamp == "" {


Let's use checking the nil here

mimowo · 2024-01-11T09:30:02Z

apis/config/v1beta1/defaults.go

 	DefaultQueueVisibilityUpdateIntervalSeconds int32   = 5
 	DefaultClusterQueuesMaxCount                int32   = 10
+	DefaultJobFrameworkName                             = "batch/job"


This change might be ok, but seems unrelated, so I would suggest reverting it so that the PR is focused.

I think I added this to avoid an import cycle. I updated the tests to be in a dedicated _test package to make sure we keep the constants in sync.

sgtm, if required to avoid the cycle

Reverted package change

mimowo · 2024-01-11T09:31:14Z

cmd/kueue/main.go

@@ -370,6 +371,13 @@ func waitForPodsReady(cfg *configapi.Configuration) bool {
 	return cfg.WaitForPodsReady != nil && cfg.WaitForPodsReady.Enable
 }

+func podsReadyRequeuingTimestamp(cfg *configapi.Configuration) configapi.RequeuingTimestamp {
+	if cfg.WaitForPodsReady != nil {
+		return cfg.WaitForPodsReady.RequeuingTimestamp


If we go for ptr, I think we will also need to check it is not nil here.

Or use ptr.Deref

Updated to pointer and added nil check

mimowo · 2024-01-11T09:31:38Z

cmd/kueue/main.go

+	if cfg.WaitForPodsReady != nil {
+		return cfg.WaitForPodsReady.RequeuingTimestamp
+	}
+	return ""


let's return Eviction here

pkg/queue/cluster_queue_strict_fifo_test.go

mimowo · 2024-01-11T09:50:35Z

pkg/scheduler/scheduler.go


-func (e entryOrdering) Len() int {


nit: I would suggest reverting the renames, but feel free to keep

I also think that the previous receiver name, e would be better.

mimowo · 2024-01-11T09:52:22Z

pkg/workload/workload.go

-	if c := apimeta.FindStatusCondition(w.Status.Conditions, kueue.WorkloadEvicted); c != nil && c.Status == metav1.ConditionTrue && c.Reason == kueue.WorkloadEvictedByPodsReadyTimeout {
-		return &c.LastTransitionTime
+func (o Ordering) GetQueueOrderTimestamp(w *kueue.Workload) *metav1.Time {
+	if o.PodsReadyRequeuingTimestamp == config.Eviction || o.PodsReadyRequeuingTimestamp == "" {


Suggested change

if o.PodsReadyRequeuingTimestamp == config.Eviction || o.PodsReadyRequeuingTimestamp == "" {

if o.PodsReadyRequeuingTimestamp == config.Eviction {

This should not be needed, if we adjust fallback in podsReadyRequeuingTimestamp

mimowo · 2024-01-11T09:56:49Z

test/integration/controller/jobs/job/job_controller_test.go

+		gomega.Expect(createdEquivJob2.Spec.Template.Spec.NodeSelector[instanceKey]).Should(gomega.Equal(equivalentFlavorB.Name))
+		util.ExpectPendingWorkloadsMetric(equivalentClusterQ, 0, 0)
+
+		time.Sleep(time.Second)


I see the first sleep is needed to make sure the creation timestamps differ.

However, can we simplify the scenario, so that it captures the change, but uses at most one sleep(1s)?

Removing these tests as per Aldo's comment

mimowo · 2024-01-11T09:57:38Z

test/integration/scheduler/podsready/scheduler_test.go

+				gomega.Expect(k8sClient.Create(ctx, wl1)).Should(gomega.Succeed())
+				time.Sleep(time.Second)
+				gomega.Expect(k8sClient.Create(ctx, wl2)).Should(gomega.Succeed())
+				time.Sleep(time.Second)


Similarly here, can we capture the essence with just 1 sleep?

The expectation here is workload 1 gets admitted (need spacing between 1 & 2), then workload 2 gets admitted (need spacing between 2 & 3 to make this consistent), then workload 1 gets admitted again. I think we need the workloads to be spaced out in time to have trigger this behavior and maintain the conditions being asserted on.

Turns out that my test was wrong and wl2 was getting admitted on borrowed capacity. I simplified the test to get rid of borrowed capacity and use 2 workloads and 1 sleep.

alculquicondor

I can think of an alternative approach in which we add a timestamp field in Workload.Info that is populated by the queue manager as the workloads are added or updated. Then the rest of the system can use this cached timestamp without having to understand where it's coming from.

It might simplify some code, but I'll leave it up to you to decide which approach to keep.

alculquicondor · 2024-01-12T15:25:49Z

apis/config/v1beta1/configuration_types.go

+
+const (
+	// Creation timestamp (from Workload .metadata.creationTimestamp).
+	Creation RequeuingTimestamp = "Creation"


Suggested change

Creation RequeuingTimestamp = "Creation"

CreationTimestamp RequeuingTimestamp = "Creation"

alculquicondor · 2024-01-12T15:26:07Z

apis/config/v1beta1/configuration_types.go

+	Creation RequeuingTimestamp = "Creation"
+
+	// Eviction timestamp (from Workload .status.conditions).
+	Eviction RequeuingTimestamp = "Eviction"


Suggested change

Eviction RequeuingTimestamp = "Eviction"

EvictionTimestamp RequeuingTimestamp = "Eviction"

apis/config/v1beta1/defaults_test.go

alculquicondor · 2024-01-12T15:30:44Z

cmd/kueue/main.go

@@ -370,6 +371,13 @@ func waitForPodsReady(cfg *configapi.Configuration) bool {
 	return cfg.WaitForPodsReady != nil && cfg.WaitForPodsReady.Enable
 }

+func podsReadyRequeuingTimestamp(cfg *configapi.Configuration) configapi.RequeuingTimestamp {
+	if cfg.WaitForPodsReady != nil {
+		return cfg.WaitForPodsReady.RequeuingTimestamp


Or use ptr.Deref

pkg/queue/manager.go

pkg/scheduler/scheduler.go

test/integration/controller/jobs/job/job_controller_test.go

tenzen-y · 2024-01-15T09:15:40Z

apis/config/v1beta1/defaults_test.go

 					UpdateIntervalSeconds: 10,
-					ClusterQueues: &ClusterQueueVisibility{
+					ClusterQueues: &v1beta1.ClusterQueueVisibility{
 						MaxCount: 0,
 					},
 				},


Should we add a case, "defaulting waitForPodsReady.RequeuingTimestamp"?

The assertion is being captured in the renamed defaulting waitForPodsReady.* case ... it was already asserting on other waitForPodsReady fields.

It makes sense.

tenzen-y · 2024-01-15T09:35:12Z

pkg/workload/workload.go

-func GetQueueOrderTimestamp(w *kueue.Workload) *metav1.Time {
-	if c := apimeta.FindStatusCondition(w.Status.Conditions, kueue.WorkloadEvicted); c != nil && c.Status == metav1.ConditionTrue && c.Reason == kueue.WorkloadEvictedByPodsReadyTimeout {
-		return &c.LastTransitionTime
+func (o Ordering) GetQueueOrderTimestamp(w *kueue.Workload) *metav1.Time {


Should we add a unit test since this function is exported?

Test is there, updated it just now

pkg/scheduler/scheduler.go

tenzen-y · 2024-01-15T10:01:44Z

pkg/scheduler/scheduler.go


-func (e entryOrdering) Len() int {


I also think that the previous receiver name, e would be better.

test/integration/controller/jobs/job/job_controller_test.go

tenzen-y · 2024-01-15T10:50:24Z

/kind api-change

nstogner · 2024-01-15T20:23:05Z

I can think of an alternative approach in which we add a timestamp field in Workload.Info that is populated by the queue manager as the workloads are added or updated. Then the rest of the system can use this cached timestamp without having to understand where it's coming from.

It might simplify some code, but I'll leave it up to you to decide which approach to keep.

Ah, I missed you comment and implemented all of the other requested changes. I am looking at what you suggested now.

nstogner · 2024-01-15T20:29:47Z

I can think of an alternative approach in which we add a timestamp field in Workload.Info that is populated by the queue manager as the workloads are added or updated. Then the rest of the system can use this cached timestamp without having to understand where it's coming from.
It might simplify some code, but I'll leave it up to you to decide which approach to keep.

Ah, I missed you comment and implemented all of the other requested changes. I am looking at what you suggested now.

While I think this would remove some of the configuration propagation that is occurring at the moment, it seems a bit unnatural to carry a global config value around in each instance of a Workload (via type Info). This value determines how multiple workloads get sorted against eachother. It might be a little confusing to pull this value from a given instance of a workload itself to then compare the rest of the workloads for sorting.

alculquicondor · 2024-01-15T21:26:24Z

apis/config/v1beta1/defaults.go

+			defaultRequeuingTimestamp := EvictionTimestamp
+			cfg.WaitForPodsReady.RequeuingTimestamp = &defaultRequeuingTimestamp


Suggested change

defaultRequeuingTimestamp := EvictionTimestamp

cfg.WaitForPodsReady.RequeuingTimestamp = &defaultRequeuingTimestamp

cfg.WaitForPodsReady.RequeuingTimestamp = ptr.To(EvictionTimestamp)

alculquicondor · 2024-01-15T21:28:28Z

pkg/queue/cluster_queue_strict_fifo_test.go

@@ -38,7 +39,10 @@ func TestFIFOClusterQueue(t *testing.T) {
 		Spec: kueue.ClusterQueueSpec{


I actually preferred putting &kueue.ClusterQueue in a new line.

Changing back to new line

alculquicondor · 2024-01-15T21:33:07Z

test/integration/scheduler/podsready/scheduler_test.go

+			})
+
+			ginkgo.By("waiting the timeout, the first workload should be evicted and readmitted", func() {
+				time.Sleep(podsReadyTimeout)


Use Eventually to detect that the Workload gets the Eviction condition.

util.FinishEvictionForWorkloads() has a retry loop built in - using that and removing the sleep.

alculquicondor · 2024-01-15T21:33:25Z

test/integration/scheduler/podsready/scheduler_test.go

+			})
+
+			ginkgo.By("waiting the timeout, the first workload should be evicted and the second should be admitted", func() {
+				time.Sleep(podsReadyTimeout)


Use Eventually to detect that the Workload gets the Eviction condition.

Removing the sleep

alculquicondor · 2024-01-15T21:34:05Z

/approve
I'm leaving LGTM to @mimowo or @tenzen-y

tenzen-y

just some nits.

tenzen-y · 2024-01-16T08:16:50Z

apis/config/v1beta1/defaults.go

@@ -41,6 +39,7 @@ const (
 	defaultPodsReadyTimeout                             = 5 * time.Minute
 	DefaultQueueVisibilityUpdateIntervalSeconds int32   = 5
 	DefaultClusterQueuesMaxCount                int32   = 10
+	DefaultJobFrameworkName                             = "batch/job"


Suggested change

DefaultJobFrameworkName = "batch/job"

defaultJobFrameworkName = "batch/job"

Is this constant used from external packages?

Good catch, un-exporting

tenzen-y · 2024-01-16T08:21:12Z

apis/config/v1beta1/defaults_test.go

 					UpdateIntervalSeconds: 10,
-					ClusterQueues: &ClusterQueueVisibility{
+					ClusterQueues: &v1beta1.ClusterQueueVisibility{
 						MaxCount: 0,
 					},
 				},


It makes sense.

tenzen-y · 2024-01-16T08:21:30Z

apis/config/v1beta1/defaults_test.go

@@ -312,7 +310,7 @@ func TestSetDefaults_Configuration(t *testing.T) {
 				QueueVisibility:  defaultQueueVisibility,
 			},
 		},
-		"defaulting waitForPodsReady.timeout": {
+		"defaulting waitForPodsReady.*": {


Suggested change

"defaulting waitForPodsReady.*": {

"defaulting waitForPodsReady values": {

tenzen-y · 2024-01-16T08:23:51Z

pkg/queue/cluster_queue_impl_test.go

@@ -35,8 +36,14 @@ const (
 	defaultNamespace = "default"
 )

+var (
+	defaultQueueOrdering = queueOrderingFunc(workload.Ordering{


Suggested change

defaultQueueOrdering = queueOrderingFunc(workload.Ordering{

defaultQueueOrderingFunc = queueOrderingFunc(workload.Ordering{

tenzen-y · 2024-01-16T08:47:57Z

test/integration/scheduler/podsready/scheduler_test.go

+		})
+
+		ginkgo.It("Should prioritize workloads submitted earlier", func() {
+			localQueueName := "eviction-creation-lq"


Why do we need to define a new lq? Should we use prodQueue?

Good question, removing

tenzen-y · 2024-01-16T08:48:38Z

test/integration/scheduler/podsready/scheduler_test.go

+		})
+
+		ginkgo.It("Should keep the evicted workload at the front of the queue", func() {
+			localQueueName := "eviction-creation-lq"


Here is the same comment.

mimowo · 2024-01-16T10:31:49Z

test/integration/scheduler/podsready/scheduler_test.go

+			ginkgo.By("waiting the timeout, the first workload should be evicted and readmitted", func() {
+				time.Sleep(podsReadyTimeout)
+				util.FinishEvictionForWorkloads(ctx, k8sClient, wl1)
+				util.ExpectWorkloadsToHaveQuotaReservation(ctx, k8sClient, prodClusterQ.Name, wl1)


Should we also verify what happens to wl2, does it remain pending?

Good catch, it gets admitted via borrowed capacity from the dev queue. I am now creating a standalone queue and asserting that it does not get admitted.

tenzen-y · 2024-01-17T05:50:47Z

test/integration/scheduler/podsready/scheduler_test.go

+			defer func() {
+				gomega.Expect(util.DeleteClusterQueue(ctx, k8sClient, standaloneClusterQ)).Should(gomega.Succeed())
+			}()


Should we perform this in ginkgo.AfterEach in this Context node instead of the defer?

IIUC this is because the queues are declared only within the It, while the AfterEach is for all tests.
Alternatively, we could declare the variable for all tests in the context, and in afterEach delete in not nil, but I don't hold a view which one is better.

Alternatively, we could declare the variable for all tests in the context, and in afterEach delete in not nil

Yes, if we want to use AfterEach, we must put standaloneClusterQ on the line 548.

but I don't hold a view which one is better.

I would suggest using AfterEach.

I updated to use a JustBeforeEach() (cant use BeforeEach() b/c you need the k8sClient) and an AfterEach().

Ah, we set up k8client in the JustBeforeEach() node...
Ideally, we want to set up the k8sclient in the BeforeAll() or BeforeEach(). However, I think the investigation is out of scope in this PR.

tenzen-y · 2024-01-17T05:51:04Z

test/integration/scheduler/podsready/scheduler_test.go

+			defer func() {
+				gomega.Expect(util.DeleteLocalQueue(ctx, k8sClient, standaloneQueue)).Should(gomega.Succeed())
+			}()


tenzen-y · 2024-01-17T05:53:55Z

Due to #1593
/test pull-kueue-test-e2e-main-1-28

mimowo · 2024-01-17T08:46:06Z

apis/config/v1beta1/configuration_types.go

@@ -192,8 +192,22 @@ type WaitForPodsReady struct {
 	// until the jobs reach the PodsReady=true condition. It defaults to false if Enable is false
 	// and defaults to true otherwise.
 	BlockAdmission *bool `json:"blockAdmission,omitempty"`
+
+	// RequeuingTimestamp defines the timestamp used for requeuing a Workload
+	// that was evicted due to Pod readiness. Defaults to Eviction (back of queue).


I'm wondering if "(back of queue)" could be misleading, there could be lower priority workloads that are even more behind in the queue order, because priority is checked first:

kueue/pkg/queue/cluster_queue_strict_fifo.go

Lines 50 to 55 in 2790286

p1 := utilpriority.Priority(objA.Obj)

p2 := utilpriority.Priority(objB.Obj)

if p1 != p2 {

return p1 > p2

}

. I would suggest to just remove the text in the brackets.

Agree, done

alculquicondor

/approve
Please address the suggestions for the integration test.

alculquicondor · 2024-01-17T18:02:27Z

test/integration/scheduler/podsready/scheduler_test.go

+		ginkgo.BeforeEach(func() {
+			podsReadyTimeout = 3 * time.Second
+			requeuingTimestamp = config.CreationTimestamp
+		})


there should be a corresponding AfterEach resetting the original values.

This is not done in the other tests today (they all set these values in their BeforeEach()s). Would you like me to modify the others?

Ideally, yes, but we can leave it for a follow up.

Just add an AfterEach for this new case in this PR.

Gotcha, I guess it makes sense to add some variables for tracking the original values?

Take a look at this commit to see if that is what you prefer.

k8s-ci-robot · 2024-01-17T18:05:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, nstogner

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [alculquicondor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mimowo · 2024-01-18T08:38:19Z

LGTM

tenzen-y

@nstogner Thank you!

/lgtm
/hold cancel

k8s-ci-robot · 2024-01-18T13:35:09Z

LGTM label has been added.

Git tree hash: 2d4a963ffcad71f03c8030b210b27ecb3b7f716e

alculquicondor · 2024-02-13T19:35:04Z

/release-note-edit

Add a config field to `.waitForPodsReady` to allow admins to configure the timestamp used when sorting workloads that were evicted due to their Pods not becoming ready in time.

alculquicondor · 2024-02-13T19:35:21Z

/release-note-edit

Add a config field to `.waitForPodsReady` to allow admins to configure the timestamp used when sorting workloads that were evicted due to their Pods not becoming ready on time.

alculquicondor · 2024-02-13T19:42:31Z

/release-note-edit

WaitForPodsReady: Add a config field to allow admins to configure the timestamp used when sorting workloads that were evicted due to their Pods not becoming ready on time.

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jan 3, 2024

k8s-ci-robot requested review from kerthcet and trasc January 3, 2024 16:28

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jan 3, 2024

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jan 3, 2024

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 3, 2024

nstogner force-pushed the pods-ready-requeue-strategy branch 3 times, most recently from 432f4d7 to 6a1ed38 Compare January 5, 2024 16:20

k8s-ci-robot assigned tenzen-y Jan 5, 2024

nstogner force-pushed the pods-ready-requeue-strategy branch 2 times, most recently from 62414b3 to 8d32336 Compare January 9, 2024 23:54

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 9, 2024

k8s-ci-robot assigned mimowo Jan 10, 2024

mimowo reviewed Jan 11, 2024

View reviewed changes

alculquicondor reviewed Jan 12, 2024

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 12, 2024

tenzen-y reviewed Jan 15, 2024

View reviewed changes

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jan 15, 2024

nstogner force-pushed the pods-ready-requeue-strategy branch 2 times, most recently from 24ec9d2 to 64404b5 Compare January 15, 2024 20:17

Address PR comments

3f738ee

nstogner force-pushed the pods-ready-requeue-strategy branch from 64404b5 to 3f738ee Compare January 15, 2024 20:44

Regenerate

dbe3017

alculquicondor reviewed Jan 15, 2024

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 15, 2024

tenzen-y reviewed Jan 16, 2024

View reviewed changes

mimowo reviewed Jan 16, 2024

View reviewed changes

Address more comments

9a228c7

tenzen-y reviewed Jan 17, 2024

View reviewed changes

mimowo reviewed Jan 17, 2024

View reviewed changes

alculquicondor reviewed Jan 17, 2024

View reviewed changes

nstogner added 2 commits January 17, 2024 16:24

Address more comments

e4dac11

Reset mutated test values

edc8d7d

tenzen-y reviewed Jan 18, 2024

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 18, 2024

k8s-ci-robot merged commit 36c2451 into kubernetes-sigs:main Jan 18, 2024
14 checks passed

k8s-ci-robot added this to the v0.6 milestone Jan 18, 2024

	if o.PodsReadyRequeuingTimestamp == config.Eviction \|\| o.PodsReadyRequeuingTimestamp == "" {
	if o.PodsReadyRequeuingTimestamp == config.Eviction {

	Creation RequeuingTimestamp = "Creation"
	CreationTimestamp RequeuingTimestamp = "Creation"

	Eviction RequeuingTimestamp = "Eviction"
	EvictionTimestamp RequeuingTimestamp = "Eviction"

		defaultRequeuingTimestamp := EvictionTimestamp
		cfg.WaitForPodsReady.RequeuingTimestamp = &defaultRequeuingTimestamp

		@@ -38,7 +39,10 @@ func TestFIFOClusterQueue(t *testing.T) {
		Spec: kueue.ClusterQueueSpec{

	DefaultJobFrameworkName = "batch/job"
	defaultJobFrameworkName = "batch/job"

	"defaulting waitForPodsReady.*": {
	"defaulting waitForPodsReady values": {

	defaultQueueOrdering = queueOrderingFunc(workload.Ordering{
	defaultQueueOrderingFunc = queueOrderingFunc(workload.Ordering{

	p1 := utilpriority.Priority(objA.Obj)
	p2 := utilpriority.Priority(objB.Obj)

	if p1 != p2 {
	return p1 > p2
	}

Make ordering timestamp configurable when Workload evicted due to Pods-ready #1542

Make ordering timestamp configurable when Workload evicted due to Pods-ready #1542

Conversation

nstogner commented Jan 3, 2024 • edited by k8s-ci-robot Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

netlify bot commented Jan 3, 2024 • edited Loading

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

k8s-ci-robot commented Jan 3, 2024

nstogner commented Jan 3, 2024

tenzen-y commented Jan 3, 2024

tenzen-y commented Jan 5, 2024

alculquicondor commented Jan 10, 2024

mimowo commented Jan 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nstogner Jan 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nstogner Jan 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y commented Jan 15, 2024

nstogner commented Jan 15, 2024

nstogner commented Jan 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nstogner Jan 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor commented Jan 15, 2024

tenzen-y left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mimowo Jan 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nstogner commented Jan 3, 2024 •

edited by k8s-ci-robot

Loading

netlify bot commented Jan 3, 2024 •

edited

Loading

nstogner Jan 11, 2024 •

edited

Loading

nstogner Jan 15, 2024 •

edited

Loading

nstogner Jan 16, 2024 •

edited

Loading

mimowo Jan 16, 2024 •

edited

Loading