Copy some labels from Pods/Jobs into the Workload object #1959

pajakd · 2024-04-09T09:39:16Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

When creating a Workload object from Pod/Job, copy labels of the Pod/Job into the Workload object. It will allow to easily identify and filter the Workload objects.

Which issue(s) this PR fixes:

Fixes #1834

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Added label copying from Pod/Job into the Kueue Workload.

k8s-ci-robot · 2024-04-09T09:39:26Z

Hi @pajakd. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

netlify · 2024-04-09T09:39:34Z

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	`4130abf`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/6619304a69a5e40008d6e42d

kannon92 · 2024-04-09T11:06:07Z

/ok-to-test

alculquicondor · 2024-04-09T15:43:33Z

apis/config/v1beta1/configuration_types.go

@@ -301,6 +301,10 @@ type Integrations struct {
 	Frameworks []string `json:"frameworks,omitempty"`
 	// PodOptions defines kueue controller behaviour for pod objects
 	PodOptions *PodIntegrationOptions `json:"podOptions,omitempty"`
+
+	// A list of label keys that should be copied from job
+	// into the workload object


Explain what happens if the job doesn't have the label.

And, in the case of composable jobs, what happens if there are different values in different objects.

When the label is not there it is not copied (no error, no warning). Added an explanation in the comment (also about the case of mismatched labels).

keps/1834-copy-labels-into-workload/README.md

alculquicondor · 2024-04-09T15:46:31Z

keps/1834-copy-labels-into-workload/kep.yaml

+  - "@alculquicondor"
+  - "@gabesaba"
+approvers:
+  - TBD


remove this line

alculquicondor · 2024-04-09T15:47:26Z

keps/1834-copy-labels-into-workload/kep.yaml

+
+
+# The target maturity stage in the current dev cycle for this KEP.
+stage: alpha


Since there is no feature gate, maybe we should call this as beta (to match the API version).

alculquicondor · 2024-04-09T15:47:28Z

keps/1834-copy-labels-into-workload/kep.yaml

+
+# The following PRR answers are required at alpha release
+# List the feature gate name and the components for which it must be enabled
+disable-supported: true


is it?
There is no feature gate (I wouldn't add one).

Oh, sorry. Removed the disable-supported and the comments.

alculquicondor · 2024-04-09T15:51:20Z

pkg/controller/jobframework/reconciler.go

+	workloadLabels := make(map[string]string)
+	if r.labelKeysToCopy == nil {
+		return workloadLabels
+	}


Reducing allocations:

Suggested change

workloadLabels := make(map[string]string)

if r.labelKeysToCopy == nil {

return workloadLabels

}

if len(r.labelKeysToCopy) == 0 {

return nil

}

workloadLabels := make(map[string]string, len(r.labelKeysToCopy))

Workload must have the Labels field initialized because later we use it (assigning JobUIDLabel) for every workload (that has a valid jobuid).

Since output of this function is used directly as Labels in the workload object we (probably) should not return nil.

I did apply the suggested change (but returning empty list instead of nil).

alculquicondor · 2024-04-09T15:58:43Z

pkg/controller/jobs/pod/pod_controller.go

+func (p *Pod) getWorkloadLabels(labelKeysToCopy []string) (map[string]string, error) {
+	jobLabels := p.Object().GetLabels()
+	workloadLabels := make(map[string]string)
+	if labelKeysToCopy == nil {


Similar recommendation as the job controller.

Here we can return nil, so I'm applying the suggestion entirely.

pkg/controller/jobs/pod/pod_controller.go

test/integration/controller/jobs/job/job_controller_test.go

test/integration/controller/jobs/pod/pod_controller_test.go

pajakd

Thank you for the comments. I think I addressed them. Please take a look if everything looks ok.

keps/1834-copy-labels-into-workload/README.md

pajakd · 2024-04-10T07:59:51Z

keps/1834-copy-labels-into-workload/kep.yaml

+  - "@alculquicondor"
+  - "@gabesaba"
+approvers:
+  - TBD


pajakd · 2024-04-10T08:00:21Z

keps/1834-copy-labels-into-workload/kep.yaml

+
+
+# The target maturity stage in the current dev cycle for this KEP.
+stage: alpha


pajakd · 2024-04-10T08:01:52Z

keps/1834-copy-labels-into-workload/kep.yaml

+
+# The following PRR answers are required at alpha release
+# List the feature gate name and the components for which it must be enabled
+disable-supported: true


Oh, sorry. Removed the disable-supported and the comments.

pajakd · 2024-04-10T08:23:25Z

pkg/controller/jobframework/reconciler.go

+	workloadLabels := make(map[string]string)
+	if r.labelKeysToCopy == nil {
+		return workloadLabels
+	}


Workload must have the Labels field initialized because later we use it (assigning JobUIDLabel) for every workload (that has a valid jobuid).

Since output of this function is used directly as Labels in the workload object we (probably) should not return nil.

I did apply the suggested change (but returning empty list instead of nil).

pkg/controller/jobs/pod/pod_controller.go

pajakd · 2024-04-10T08:25:26Z

pkg/controller/jobs/pod/pod_controller.go

+func (p *Pod) getWorkloadLabels(labelKeysToCopy []string) (map[string]string, error) {
+	jobLabels := p.Object().GetLabels()
+	workloadLabels := make(map[string]string)
+	if labelKeysToCopy == nil {


Here we can return nil, so I'm applying the suggestion entirely.

test/integration/controller/jobs/job/job_controller_test.go

test/integration/controller/jobs/pod/pod_controller_test.go

mimowo

Some questions to the KEP for now.

mimowo · 2024-04-10T11:22:24Z

keps/1834-copy-labels-into-workload/README.md

+
+A case that requires more attention is creating workloads from pod groups because in that case a single workload is based on multiple pods (each of which might have labels). We propose:
+ * When a label from the `labelKeysToCopy` list will be present at some of the pods from the group and the value of this label on all these pods will be identical then this label will be copied into the workload. Note that we do not require that all the pods have the label but all that do have must have the same value.
+ * When multiple pods from the group will have the label with different values, we will raise an exception.


Is the Workload object created only when we have all the pods in a group created? If it is created when the first pod is created, then we may not have a chance to raise the exception.
Also, related:

when is the exception raised, it is during workload creation, or later? If later, how is the error communicated to the user?

what happens when the workload already exists, and the user creates a new pod with a non-matching value?

As far as I know the workload object is created when all the pods are created.

The exception is raised during workload creation.

The new feature is only applied during the creation of a new workload so if the user creates another pod later it will have no effect on the labels on the running workload.

As far as I know the workload object is created when all the pods are created.

This would simplify this indeed. Could you please double check on a running system?

The exception is raised during workload creation.
The new feature is only applied during the creation of a new workload so if the user creates another pod later it will have no effect on the labels on the running workload.

sgtm, please add a sentence along the lines to clarify in KEP for future readers.

As far as I know the workload object is created when all the pods are created.

That is correct.

Done. Added a sentence to KEP clarifying this.

Thanks for clarifying and the update.

keps/1834-copy-labels-into-workload/README.md

pajakd

Thank you for the comments!

keps/1834-copy-labels-into-workload/README.md

pajakd · 2024-04-10T12:35:57Z

keps/1834-copy-labels-into-workload/README.md

+
+A case that requires more attention is creating workloads from pod groups because in that case a single workload is based on multiple pods (each of which might have labels). We propose:
+ * When a label from the `labelKeysToCopy` list will be present at some of the pods from the group and the value of this label on all these pods will be identical then this label will be copied into the workload. Note that we do not require that all the pods have the label but all that do have must have the same value.
+ * When multiple pods from the group will have the label with different values, we will raise an exception.


As far as I know the workload object is created when all the pods are created.

The exception is raised during workload creation.

The new feature is only applied during the creation of a new workload so if the user creates another pod later it will have no effect on the labels on the running workload.

tenzen-y

I reviewed only proposal.

tenzen-y · 2024-04-10T11:57:20Z

keps/1834-copy-labels-into-workload/kep.yaml

+kep-number: 1834
+authors:
+  - "@pajakd"
+status: draft


Suggested change

status: draft

status: implementable

Isn't this implementable?

keps/1834-copy-labels-into-workload/README.md

alculquicondor · 2024-04-10T17:38:03Z

keps/1834-copy-labels-into-workload/README.md

+
+## Design Details
+
+We believe that the KEP is rather straightforward and the [Proposal](#proposal) section contains sufficient details of the implementation.


You can add the configuration API detail here.

alculquicondor · 2024-04-10T17:38:55Z

apis/config/v1beta1/configuration_types.go

+	// constructed workload object will be created without this label. In the case
+	// of creating a workload from a composable job (pod group), if multiple objects
+	// have labels with some key form  the  list, the values of these labels must
+	// agree or otherwise the workload creation would fail.


Also mention that if the Job labels change after the first Workload object is created, the Workload will not be updated.

keps/1834-copy-labels-into-workload/README.md

alculquicondor · 2024-04-10T17:43:03Z

pkg/controller/jobframework/reconciler.go

@@ -784,16 +794,29 @@ func (r *JobReconciler) finalizeJob(ctx context.Context, job GenericJob) error {
 	return nil
 }

+func (r *JobReconciler) getWorkloadLabels(job GenericJob) map[string]string {
+	jobLabels := job.Object().GetLabels()


move this line after the check for len(r.labelKeysToCopy).

alculquicondor · 2024-04-10T17:46:40Z

test/integration/controller/jobs/pod/pod_controller_test.go

@@ -569,6 +573,9 @@ var _ = ginkgo.Describe("Pod controller", ginkgo.Ordered, ginkgo.ContinueOnFailu
 						},
 						wlConditionCmpOpts...,
 					))
+					ginkgo.By("checking the workload gets assigned the correct labels")


Suggested change

ginkgo.By("checking the workload gets assigned the correct labels")

// Checking the workload gets assigned the correct labels.

mimowo · 2024-04-11T07:00:37Z

pkg/controller/jobs/pod/pod_controller.go

@@ -898,7 +900,36 @@ func (p *Pod) ensureWorkloadOwnedByAllMembers(ctx context.Context, c client.Clie
 	return nil
 }

-func (p *Pod) ConstructComposableWorkload(ctx context.Context, c client.Client, r record.EventRecorder) (*kueue.Workload, error) {
+func (p *Pod) getWorkloadLabels(labelKeysToCopy []string) (map[string]string, error) {
+	jobLabels := p.Object().GetLabels()


nit: here also move below the len check

mimowo · 2024-04-11T07:06:14Z

pkg/controller/jobs/pod/pod_controller.go

+	for _, labelKey := range labelKeysToCopy {
+		if labelValue, found := jobLabels[labelKey]; found {
+			workloadLabels[labelKey] = labelValue
+		}
+	}


Could we move this pass under the !p.isGroup check? It seems the case when this is a group is handled below.

This is handling the case when the object is a single pod. We have to copy the labels from the single pod into the workload in this case.

Yes, my idea was:

workloadLabels := make(map[string]string, len(labelKeysToCopy)) if !p.isGroup { for _, labelKey := range labelKeysToCopy { if labelValue, found := jobLabels[labelKey]; found { workloadLabels[labelKey] = labelValue } } return workloadLabels, nil } // the code for group handling

would it work?

yes, I think its correct. Applied the suggested changes

mimowo · 2024-04-11T07:23:26Z

pkg/controller/jobs/pod/pod_controller_test.go

+				Obj()},
+			wantPods: nil,
+			reconcilerOptions: []jobframework.Option{
+				jobframework.WithLabelKeysToCopy([]string{"toCopyKey"}),


nit: add also a label key name that is absent in the pod.

pkg/controller/jobs/pod/pod_controller.go

alculquicondor

design, implementation and tests LGTM.
The API comment needs a bit of work.

@tenzen-y anything to add?

alculquicondor · 2024-04-11T18:06:36Z

apis/config/v1beta1/configuration_types.go

+	// a job does not have some label with the given key from this list, the
+	// constructed workload object will be created without this label. In the case
+	// of creating a workload from a composable job (pod group), if multiple objects
+	// have labels with some key form  the  list, the values of these labels must


Suggested change

// have labels with some key form the list, the values of these labels must

// have labels with some key from the list, the values of these labels must

alculquicondor · 2024-04-11T18:06:47Z

apis/config/v1beta1/configuration_types.go

+	// constructed workload object will be created without this label. In the case
+	// of creating a workload from a composable job (pod group), if multiple objects
+	// have labels with some key form  the  list, the values of these labels must
+	// agree or otherwise the workload creation would fail. The labels are copied only


Suggested change

// agree or otherwise the workload creation would fail. The labels are copied only

// match or otherwise the workload creation would fail. The labels are copied only

alculquicondor · 2024-04-11T18:07:14Z

apis/config/v1beta1/configuration_types.go

@@ -301,6 +301,17 @@ type Integrations struct {
 	Frameworks []string `json:"frameworks,omitempty"`
 	// PodOptions defines kueue controller behaviour for pod objects
 	PodOptions *PodIntegrationOptions `json:"podOptions,omitempty"`
+
+	// A list of label keys that should be copied from the job into the workload
+	// object. We don't require the job to have all the labels from this list. If


Suggested change

// object. We don't require the job to have all the labels from this list. If

// object. It is not required for the job to have all the labels from this list. If

keps/1834-copy-labels-into-workload/kep.yaml

alculquicondor · 2024-04-11T18:10:26Z

apis/config/v1beta1/configuration_types.go

@@ -301,6 +301,17 @@ type Integrations struct {
 	Frameworks []string `json:"frameworks,omitempty"`
 	// PodOptions defines kueue controller behaviour for pod objects
 	PodOptions *PodIntegrationOptions `json:"podOptions,omitempty"`
+
+	// A list of label keys that should be copied from the job into the workload


One general recommendation when writing documentation and API comments, is to use active voice https://kubernetes.io/docs/contribute/style/style-guide/#use-active-voice

Try to adhere.

Ok I'll read and follow the style guide. But here I don't see any good way of rephrasing "should be copied" into active voice...

Sounds good, there is the exception Use passive voice if active voice leads to an awkward construction :)

alculquicondor

/approve

Leaving LGTM to @tenzen-y

tenzen-y

Thank you for this update!
Basically, lgtm. I left a few minor comments.

tenzen-y · 2024-04-11T19:53:05Z

apis/config/v1beta1/configuration_types.go

@@ -301,6 +301,17 @@ type Integrations struct {
 	Frameworks []string `json:"frameworks,omitempty"`
 	// PodOptions defines kueue controller behaviour for pod objects
 	PodOptions *PodIntegrationOptions `json:"podOptions,omitempty"`
+
+	// A list of label keys that should be copied from the job into the workload


Suggested change

// A list of label keys that should be copied from the job into the workload

// labelKeysToCopy is a list of label keys that should be copied from the job into the workload

keps/1834-copy-labels-into-workload/README.md

tenzen-y · 2024-04-11T19:54:12Z

keps/1834-copy-labels-into-workload/README.md

+``` go
+type Integrations struct {
+    ...
+	// A list of label keys that should be copied from the job into the workload


Suggested change

// A list of label keys that should be copied from the job into the workload

// labelKeysToCopy is a list of label keys that should be copied from the job into the workload

tenzen-y · 2024-04-11T19:57:29Z

keps/1834-copy-labels-into-workload/kep.yaml

+  alpha: "v0.7"
+  beta: "v0.8"


Suggested change

alpha: "v0.7"

beta: "v0.8"

beta: "v0.7"

I'm wondering if we should declare this feature as a beta in v0.7, right?

I'm ok without a feature gate since the risk is minimal, but if there is no feature gate, should we declare "beta", or "stable" already?

I don't know (I'll let you decide). For the time being I applied the suggestion above.

tenzen-y · 2024-04-11T20:01:51Z

pkg/controller/jobs/job/job_controller_test.go

+						"toCopyKey":                  "toCopyValue"}).
+					Obj(),
+			},
+


Suggested change

tenzen-y · 2024-04-11T20:08:19Z

pkg/controller/jobframework/reconciler.go

@@ -784,16 +794,29 @@ func (r *JobReconciler) finalizeJob(ctx context.Context, job GenericJob) error {
 	return nil
 }

+func (r *JobReconciler) getWorkloadLabels(job GenericJob) map[string]string {
+	if len(r.labelKeysToCopy) == 0 {
+		return map[string]string{}


I'm curious about if we can return just nil. WDYT?

I reworked this logic (see the comment below).

tenzen-y · 2024-04-11T20:09:31Z

pkg/controller/jobs/pod/pod_controller.go

+		for _, labelKey := range labelKeysToCopy {
+			if labelValue, found := podLabels[labelKey]; found {
+				workloadLabels[labelKey] = labelValue
+			}
+		}


This is completely the same logic as the generic reconciler:
So, could we commonize them?

jobLabels := job.Object().GetLabels() workloadLabels := make(map[string]string, len(r.labelKeysToCopy)) for _, labelKey := range r.labelKeysToCopy { if labelValue, found := jobLabels[labelKey]; found { workloadLabels[labelKey] = labelValue } }

Ok, good idea. So this is in fact filtering a map with a certain list of keys. I extracted this into maps library (added a generic maps.FilterKeys function and a bunch of unit tests). And I'm using it here and in the generic reconciler. Please take a look if it looks better.

tenzen-y · 2024-04-11T20:14:58Z

test/integration/controller/jobs/pod/pod_controller_test.go

@@ -569,6 +573,9 @@ var _ = ginkgo.Describe("Pod controller", ginkgo.Ordered, ginkgo.ContinueOnFailu
 						},
 						wlConditionCmpOpts...,
 					))
+					// Checking the workload gets assigned the correct labels.


Suggested change

// Checking the workload gets assigned the correct labels.

ginkgo.By("Checking the workload gets assigned the correct labels")

This comment is helpful for the debug.

tenzen-y · 2024-04-11T20:18:46Z

pkg/controller/jobs/job/job_controller_test.go

+					Message:   "Created Workload: ns/" + GetWorkloadNameForJob(baseJobWrapper.Name, types.UID("test-uid")),
+				},
+			},
+		},


Could you add another case to verify if the labels are not populated to Workloads after the Workload is already created?

I'd like to prove this comment: "The labels are copied only during the workload creation and are not updated even if the labels of the underlying job are changed."

I added a check in the job_controller integration test. Please take a look.

alculquicondor

/lgtm
/approve

k8s-ci-robot · 2024-04-12T14:53:13Z

LGTM label has been added.

Git tree hash: eb17443205045edb36e8a055bd337161a9d89c32

k8s-ci-robot · 2024-04-12T14:53:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, pajakd

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [alculquicondor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…sigs#1959) * Copy labels for a single job into the workload. * Copying of the labels from pod (and pod groups) * KEP first draft * Unit tests for pod reconciler * Unit test for label copying from job to workload * Unit test for the case of label mismatch * Typos fix * Fix typos * Update the autogenerated file * Addressing PR comments * Addressing PR comments * Addressing PR comments * Addressing PR comments * Addressing PR comments * Addressing PR comments * Addressing PR comments * Addressing PR comments

pajakd and others added 12 commits April 4, 2024 06:32

Copy labels for a single job into the workload.

919a6b5

Merge branch 'kubernetes-sigs:main' into copy_labels

202c9ed

Copying of the labels from pod (and pod groups)

0fac577

KEP first draft

955045b

Merge branch 'kubernetes-sigs:main' into copy_labels

d4110ea

Unit tests for pod reconciler

ca7ffb1

Merge branch 'copy_labels' of github.com:pajakd/kueue into copy_labels

8e8660a

Unit test for label copying from job to workload

66a76ec

Unit test for the case of label mismatch

102da2b

Typos fix

c5471ca

Merge branch 'kubernetes-sigs:main' into copy_labels

170db10

Fix typos

097658f

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. labels Apr 9, 2024

k8s-ci-robot requested review from alculquicondor and denkensk April 9, 2024 09:39

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 9, 2024

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Apr 9, 2024

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 9, 2024

Update the autogenerated file

20b1e47

alculquicondor reviewed Apr 9, 2024

View reviewed changes

Addressing PR comments

04bd4e8

pajakd commented Apr 10, 2024

View reviewed changes

pajakd requested review from alculquicondor April 10, 2024 08:53

mimowo reviewed Apr 10, 2024

View reviewed changes

Addressing PR comments

b0faf25

pajakd commented Apr 10, 2024

View reviewed changes

tenzen-y reviewed Apr 10, 2024

View reviewed changes

alculquicondor reviewed Apr 10, 2024

View reviewed changes

mimowo reviewed Apr 11, 2024

View reviewed changes

pajakd added 2 commits April 11, 2024 09:33

Addressing PR comments

e7131e2

Addressing PR comments

6fb3487

alculquicondor reviewed Apr 11, 2024

View reviewed changes

pajakd added 2 commits April 11, 2024 19:06

Addressing PR comments

d0880c2

Addressing PR comments

e5fb23e

alculquicondor reviewed Apr 11, 2024

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 11, 2024

tenzen-y reviewed Apr 11, 2024

View reviewed changes

Addressing PR comments

bbb8874

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 12, 2024

Addressing PR comments

4130abf

alculquicondor reviewed Apr 12, 2024

View reviewed changes

k8s-ci-robot assigned alculquicondor Apr 12, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 12, 2024

k8s-ci-robot merged commit 85206ab into kubernetes-sigs:main Apr 12, 2024
14 checks passed

k8s-ci-robot added this to the v0.7 milestone Apr 12, 2024

pajakd deleted the copy_labels branch April 15, 2024 05:04

alculquicondor mentioned this pull request Jun 25, 2024

Copy some labels from Pods/Jobs into the Workload object #1834

Closed

3 tasks



		# The target maturity stage in the current dev cycle for this KEP.
		stage: alpha


		## Design Details

		We believe that the KEP is rather straightforward and the [Proposal](#proposal) section contains sufficient details of the implementation.

	ginkgo.By("checking the workload gets assigned the correct labels")
	// Checking the workload gets assigned the correct labels.

	// have labels with some key form the list, the values of these labels must
	// have labels with some key from the list, the values of these labels must

	// agree or otherwise the workload creation would fail. The labels are copied only
	// match or otherwise the workload creation would fail. The labels are copied only

	// object. We don't require the job to have all the labels from this list. If
	// object. It is not required for the job to have all the labels from this list. If

	// A list of label keys that should be copied from the job into the workload
	// labelKeysToCopy is a list of label keys that should be copied from the job into the workload

Copy some labels from Pods/Jobs into the Workload object #1959

Copy some labels from Pods/Jobs into the Workload object #1959

Conversation

pajakd commented Apr 9, 2024

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

k8s-ci-robot commented Apr 9, 2024

netlify bot commented Apr 9, 2024 • edited Loading

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

kannon92 commented Apr 9, 2024

Choose a reason for hiding this comment

pajakd Apr 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pajakd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mimowo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor Apr 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pajakd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mimowo Apr 11, 2024 • edited Loading

Choose a reason for hiding this comment

pajakd Apr 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor left a comment

Choose a reason for hiding this comment

tenzen-y left a comment

netlify bot commented Apr 9, 2024 •

edited

Loading

pajakd Apr 10, 2024 •

edited

Loading

alculquicondor Apr 10, 2024 •

edited

Loading

mimowo Apr 11, 2024 •

edited

Loading

pajakd Apr 11, 2024 •

edited

Loading