[jobframework] Use the jobs uid in workload's name hash. #1732

trasc · 2024-02-14T07:43:13Z

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

[jobframework] Use the jobs uuid in workload's name hash. With this we can avoid name collisions for workloads create for jobs having the same name and type created at different pints in time.

Which issue(s) this PR fixes:

Fixes #1726

Special notes for your reviewer:

Does this PR introduce a user-facing change?

The hash suffix of the workload's name are now influenced by the job's object UID. Recreated jobs with the same name and kind will use different workload names.

netlify · 2024-02-14T07:43:19Z

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	`791e737`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/65d6f74500977600080a03e9

trasc · 2024-02-14T07:43:30Z

/cc @alculquicondor

mimowo

Actually, I think it would be good to get to the bottom of 1726 before adding UID to the hash. At least I still don't fully understand it, so maybe just explaining would be enough if you already know. I'm worried that with this change with risk covering some issues, and complications on upgrade.

Concrete questions:

in case there is a conflict in a scenario when a job is recreated with different pod spec, so that the workload is not longer equivalent, should we not trigger recreation the workload?
what is the impact during upgrade, what will happen to user workloads which are pending, and admitted ones?

alculquicondor · 2024-02-14T19:00:38Z

I'm ok with this change, but before we merge it, I think we should make sure we are not depending on the predictability of the workload name.

tenzen-y · 2024-02-15T07:07:40Z

What happens if admins do cluster migration?
We often migrate the existing cluster to a new cluster to update the k8s version with backup and restore etcd.

I guess that workloads are just recreated with a new name since Job UID is changed by migration, right?
So, there is no impact, right?

mimowo · 2024-02-15T07:41:57Z

I still have concerns about this approach, summarized here: #1726 (comment).

EDIT: Having said that, I'm ok with this if we find it "least" evil, but I think it warrants a more descriptive release-note, describing the user-facing impact.

EDIT2: I checked that if I do in-place upgrade then the jobs are not actually stopped. In that case the concerns aren't that important, I'm ok with this change too.

trasc · 2024-02-15T07:43:38Z

@alculquicondor
Passed the workload creation the workload name is irrelevant for Kueue, job reconciliation is done based on ownership (with the exception of prebuilt workloads when there is no relationship between the wl and job name).

From a human user perspective the wl names will continue to look similar <job-prefix>-<job-name>-<hash-segment>, where hash-segment is an opaque string.

@tenzen-y
For etcd backup/restore I'd expect the workloads to be part of that process as well, if not, indeed the should just be recreated. (are the UIDs changed in migration?)

tenzen-y · 2024-02-15T08:20:52Z

For etcd backup/restore I'd expect the workloads to be part of that process as well, if not, indeed the should just be recreated. (are the UIDs changed in migration?)

If we directly backup and restore etcd, UID wouldn't be changed, but if we use cluster backup tools like the velero, UID will be changed. I'm ok with only recreating the workloads.

alculquicondor · 2024-02-15T14:29:00Z

My worry is this line:

kueue/pkg/controller/jobs/job/job_webhook.go

Line 80 in e10df68

if pwName, err := jobframework.GetWorkloadNameForOwnerRef(owner); err != nil {

trasc · 2024-02-15T15:16:58Z

GetWorkloadNameForOwnerRef

Yes , this could be problematic in case of kueue upgrades, it could keep the child job Suspended if the workload for the parent was created before the upgrade.

A similar problem we have with prebuilt workloads, but only for jobsets at this point, and we do this

kueue/pkg/controller/jobs/jobset/jobset_webhook.go

Lines 60 to 69 in 01bd532

    
           // If a prebuilt workload is used, propagate the name to it's replicated job templates. 
        
           if wlName, hasPrebuilt := jobframework.PrebuiltWorkloadFor(jobSet); hasPrebuilt { 
        
           	for i := range jobSet.Spec.ReplicatedJobs { 
        
           		rjTemplate := &jobSet.Spec.ReplicatedJobs[i].Template 
        
           		if rjTemplate.Annotations == nil { 
        
           			rjTemplate.Annotations = make(map[string]string) 
        
           		} 
        
           		rjTemplate.Annotations[constants.ParentWorkloadAnnotation] = wlName 
        
           	} 
        
           }

to overcome it.

Maybe we could droop the usage of the parentWlAnotation and cover the prebuilt scenario as well.

alculquicondor · 2024-02-15T16:24:50Z

Maybe we could drop the usage of the parentWlAnotation and cover the prebuilt scenario as well.

Can you expand on this?

trasc · 2024-02-16T11:12:35Z

Maybe we could drop the usage of the parentWlAnotation and cover the prebuilt scenario as well.

Can you expand on this?

We can use an ownership based approach, the child job and the worker have the same parent . 4dc9ba0

alculquicondor · 2024-02-16T14:13:22Z

I like that. Let's start with a PR just for that.

trasc · 2024-02-16T15:20:36Z

I like that. Let's start with a PR just for that.

Moved in #1747

With this we can avoid name collisions for workloads create for jobs having the same name and type created at different pints in time.

mimowo

Code LGTM, just some nits. Also, a suggestion for potential follow up.

I'm on a fence regarding the release note saying "to avoid potential name collisions". They are rare at rate 1/16mln they are still sort of possible to observe. Maybe we can rephrase a little not to be definitive.

pkg/controller/jobs/pod/pod_controller_test.go

pkg/controller/jobframework/workload_names.go

mimowo · 2024-02-20T11:10:28Z

Should we update the PR description to note that it fixes #1726? Or we believe there is more to do to close it, like the GC? In that case we may just mention "Part of #1726".

trasc · 2024-02-20T12:56:18Z

Should we update the PR description to note that it fixes #1726? Or we believe there is more to do to close it, like the GC? In that case we may just mention "Part of #1726".

Let's go with Fix, since the issue cannot be reproduce after adding this.

mimowo · 2024-02-22T07:56:54Z

/lgtm
IIUC the remaining issue is that when a running Job is deleted with --cascade orphan, then the workload remains admitted and continues to consume quota, which we want to tackle by GC (either removal or marking as finished). If this is correct, could you please open a follow up issue?

k8s-ci-robot · 2024-02-22T07:57:00Z

LGTM label has been added.

Git tree hash: fb96d8b5f91182d9714c4384522dfdba0f546a99

mimowo · 2024-02-22T08:48:46Z

For the release note, maybe something more user-oriented:
"The workload name is no longer the same for recreated jobs (with the same namespace and name),
because Job UID is used to determine the hash-like suffix in the workload name."

WDYT @tenzen-y

tenzen-y · 2024-02-22T09:41:48Z

For the release note, maybe something more user-oriented: "The workload name is no longer the same for recreated jobs (with the same namespace and name), because Job UID is used to determine the hash-like suffix in the workload name."

WDYT @tenzen-y

@mimowo Generally, sgtm, but it is better to emphasize that the creating workloads names lose consistency or obtain completely random names. WDYT?

mimowo · 2024-02-22T09:53:10Z

@mimowo Generally, sgtm, but it is better to emphasize that the creating workloads names lose consistency or obtain completely random names. WDYT?

Yeah, it is tricky, they are "completely random" from user perspective, but technically they are predictible, so I wanted to have something which captures the situation from user perspective (different names on recreation), yet don't say "random". However, I'm ok adding a sentence to clarify "The workload names are virtually random now." (or something like that).

trasc · 2024-02-22T10:36:19Z

"The workload names are virtually random now."

Cannot be added since they are not random.

mimowo · 2024-02-22T10:51:53Z

"The workload names are virtually random now."

Cannot be added since they are not random.

Well, I think saying "virtually" random, or "random from user's perspective" is fine (after saying that it is a hash-like in the previous sentence), but I don't have a strong opinion. Still, I think we should make set the release note to something understandable to users. WDYT @tenzen-y ?

trasc · 2024-02-22T12:37:10Z

Release note updated

alculquicondor · 2024-02-22T18:06:41Z

pkg/controller/jobs/job/job_controller_test.go

@@ -2164,7 +2164,7 @@ func TestReconciler(t *testing.T) {
 				Effect:   corev1.TaintEffectNoSchedule,
 			}).Obj(),
 			workloads: []kueue.Workload{
-				*utiltesting.MakeWorkload(GetWorkloadNameForJob(baseJobWrapper.Name), "ns").
+				*utiltesting.MakeWorkload(GetWorkloadNameForJob(baseJobWrapper.Name, baseJobWrapper.GetUID()), "ns").


Can we have another unit test where we put a completely different name, but the reconciler still recognizes it as the workload for the job?

It will be a bit of a stretch and has nothing to do with the current PR.

It does have to do. It's one minimal safety guard to ensure that existing jobs (before upgrading to the latest version) continue to run.

Why not add this when owner based workload section was added?

because of bad foresight, but it wasn't that important then.

for the vast majority of tests the workload has name a not the name that is usually generated.

ok, thanks for checking. That should be enough.

alculquicondor · 2024-02-23T14:47:22Z

/approve

k8s-ci-robot · 2024-02-23T14:47:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, trasc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/controller/OWNERS~~ [alculquicondor]
~~test/OWNERS~~ [alculquicondor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tenzen-y · 2024-02-23T15:01:27Z

"The workload names are virtually random now."

Cannot be added since they are not random.

Well, I think saying "virtually" random, or "random from user's perspective" is fine (after saying that it is a hash-like in the previous sentence), but I don't have a strong opinion. Still, I think we should make set the release note to something understandable to users. WDYT @tenzen-y ?

Sorry for the late. It sounds good to me.

…sigs#1732) * [jobframework] Use the jobs uuid in workload's name hash. With this we can avoid name collisions for workloads create for jobs having the same name and type created at different pints in time. * Review Remarks * Review Remarks

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 14, 2024

k8s-ci-robot requested review from denkensk and mimowo February 14, 2024 07:43

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 14, 2024

k8s-ci-robot requested a review from alculquicondor February 14, 2024 07:43

mimowo reviewed Feb 14, 2024

View reviewed changes

trasc mentioned this pull request Feb 14, 2024

Job gets admitted, but is never un-suspended. #1726

Closed

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 16, 2024

trasc force-pushed the job-uid-wl-hash branch from 4dc9ba0 to 7cc2922 Compare February 16, 2024 15:19

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Feb 16, 2024

tenzen-y mentioned this pull request Feb 16, 2024

Drop kueue.x-k8s.io/parent-workload in favor of a object ownership based approach. #1747

Merged

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 20, 2024

[jobframework] Use the jobs uuid in workload's name hash.

a3acdd5

With this we can avoid name collisions for workloads create for jobs having the same name and type created at different pints in time.

trasc force-pushed the job-uid-wl-hash branch from 7cc2922 to a3acdd5 Compare February 20, 2024 06:19

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 20, 2024

mimowo reviewed Feb 20, 2024

View reviewed changes

pkg/controller/jobs/pod/pod_controller_test.go Outdated Show resolved Hide resolved

pkg/controller/jobs/pod/pod_controller_test.go Outdated Show resolved Hide resolved

pkg/controller/jobframework/workload_names.go Show resolved Hide resolved

Review Remarks

6b718f6

Review Remarks

791e737

k8s-ci-robot assigned mimowo Feb 22, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 22, 2024

alculquicondor reviewed Feb 22, 2024

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 23, 2024

k8s-ci-robot merged commit 6244715 into kubernetes-sigs:main Feb 23, 2024
14 checks passed

k8s-ci-robot added this to the v0.7 milestone Feb 23, 2024

trasc deleted the job-uid-wl-hash branch March 12, 2024 07:28

[jobframework] Use the jobs uid in workload's name hash. #1732

[jobframework] Use the jobs uid in workload's name hash. #1732

Conversation

trasc commented Feb 14, 2024 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

netlify bot commented Feb 14, 2024 • edited Loading

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

trasc commented Feb 14, 2024

mimowo left a comment

Choose a reason for hiding this comment

alculquicondor commented Feb 14, 2024

tenzen-y commented Feb 15, 2024

mimowo commented Feb 15, 2024 • edited Loading

trasc commented Feb 15, 2024

tenzen-y commented Feb 15, 2024

alculquicondor commented Feb 15, 2024

trasc commented Feb 15, 2024

alculquicondor commented Feb 15, 2024

trasc commented Feb 16, 2024

alculquicondor commented Feb 16, 2024

trasc commented Feb 16, 2024

mimowo left a comment

Choose a reason for hiding this comment

mimowo commented Feb 20, 2024

trasc commented Feb 20, 2024 • edited Loading

mimowo commented Feb 22, 2024

k8s-ci-robot commented Feb 22, 2024

mimowo commented Feb 22, 2024

tenzen-y commented Feb 22, 2024

mimowo commented Feb 22, 2024

trasc commented Feb 22, 2024

mimowo commented Feb 22, 2024 • edited Loading

trasc commented Feb 22, 2024

alculquicondor Feb 22, 2024

Choose a reason for hiding this comment

trasc Feb 22, 2024

Choose a reason for hiding this comment

alculquicondor Feb 22, 2024

Choose a reason for hiding this comment

trasc Feb 22, 2024

Choose a reason for hiding this comment

alculquicondor Feb 22, 2024

Choose a reason for hiding this comment

trasc Feb 22, 2024 • edited Loading

Choose a reason for hiding this comment

alculquicondor Feb 23, 2024

Choose a reason for hiding this comment

alculquicondor commented Feb 23, 2024

k8s-ci-robot commented Feb 23, 2024

tenzen-y commented Feb 23, 2024

trasc commented Feb 14, 2024 •

edited

Loading

netlify bot commented Feb 14, 2024 •

edited

Loading

mimowo commented Feb 15, 2024 •

edited

Loading

trasc commented Feb 20, 2024 •

edited

Loading

mimowo commented Feb 22, 2024 •

edited

Loading

trasc Feb 22, 2024 •

edited

Loading