Limit the number of goroutines used when deleting jobs #123

tenzen-y · 2023-05-07T17:35:56Z

I limited the number of goroutines used when deleting Jobs.

tenzen-y · 2023-05-07T17:37:18Z

pkg/controllers/jobset_controller.go

-			// where the jobs are recreated.
-			backgroundPolicy := metav1.DeletePropagationBackground
-			if err := r.Delete(ctx, job, &client.DeleteOptions{PropagationPolicy: &backgroundPolicy}); client.IgnoreNotFound(err) != nil {
-				finalErr = err


As a potential issue, a race condition may occur.

tenzen-y · 2023-05-07T17:40:24Z

pkg/controllers/jobset_controller.go

@@ -364,31 +369,23 @@ func (r *JobSetReconciler) restartPolicyRecreateAll(ctx context.Context, js *job
 	return nil
 }

-func (r *JobSetReconciler) deleteJobs(ctx context.Context, js *jobset.JobSet, jobsForDeletion []*batchv1.Job) error {


This function doesn't use the JobSet.

pkg/controllers/jobset_controller.go

tenzen-y · 2023-05-07T17:53:22Z

pkg/controllers/jobset_controller.go

+		if err := r.Delete(ctx, targetJob, &client.DeleteOptions{PropagationPolicy: &backgroundPolicy}); client.IgnoreNotFound(err) != nil {
+			lock.Lock()
+			defer lock.Unlock()
+			finalErr = errors.Join(finalErr, err)


Oh, errors.Join() is introduced since Go 1.20, although this project uses Go 1.19.

@ahg-g @danielvegamyhre I'd like to update the Go version to 1.20 in another PR before merging this one into the main.

Does bumping the Go version to v1.20 sound good?

Sounds good to me, let's wait for Abdullah to confirm as well though.

I will say that I remember upgrading the e2e tests in test infra to 1.20

I meant that I think we are already using a go1.20 for the e2e tests

Oh, you're right :)
https://github.com/kubernetes/test-infra/blob/0e54866edb5d54816d2593bdfcd686183b4ea08e/config/jobs/kubernetes-sigs/jobset/jobset-presubmit.yaml#L13

Then, we just bump it in go.mod!

I'll open a PR bumping that and we can discuss.

tenzen-y · 2023-05-08T16:31:29Z

Squashed into one and rebased.

danielvegamyhre · 2023-05-08T17:10:37Z

/lgtm

Leaving approval for Abdullah's review

ahg-g · 2023-05-08T19:31:58Z

pkg/controllers/jobset_controller.go

-const RestartsKey string = "jobset.sigs.k8s.io/restart-attempt"
+const (
+	RestartsKey       string = "jobset.sigs.k8s.io/restart-attempt"
+	parallelDeletions int    = 8


isn't this too low? any idea what do we set this number to in other places like the Job API?

isn't this too low?

It makes sense. But I'm not sure appropriate value. How about 100 as a default value? Or, do you have any good ideas?

any idea what do we set this number to in other places like the Job API?

How about setting it via config: #55?

Or, does that mean like https://github.com/kubernetes/kubernetes/blob/5fd29596ef699a8658c941c1c7cf7a45e796c1c6/pkg/controller/job/job_controller.go#L1458-L1466C63?

I like the concept of a "slow start" in the linked code from the job controller. Maybe it would be simpler to start with some constant value like 100 then implement slow start separately?

I guess for deletions we can continue to use a constant (I would set it to 50); but adopt a slow start approach for parallel creation.

I guess for deletions we can continue to use a constant (I would set it to 50); but adopt a slow start approach for parallel creation.

Great suggestions! I agree.

ahg-g

Thanks Yuki, I think we need to parallelize job creation too, I created an issue: #130

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

tenzen-y · 2023-05-10T16:21:28Z

@ahg-g @danielvegamyhre I have updated.

tenzen-y · 2023-05-10T16:38:38Z

/test pull-jobset-test-e2e-main-1-24

danielvegamyhre · 2023-05-10T17:54:33Z

/lgtm

ahg-g · 2023-05-10T20:08:04Z

/test pull-jobset-test-e2e-main-1-24

any idea why this failed?

ahg-g · 2023-05-10T20:08:36Z

/approve

k8s-ci-robot · 2023-05-10T20:08:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahg-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tenzen-y · 2023-05-11T06:10:08Z

/test pull-jobset-test-e2e-main-1-24

any idea why this failed?

Actually, I created #135.

k8s-ci-robot requested review from ahg-g and danielvegamyhre May 7, 2023 17:36

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 7, 2023

tenzen-y commented May 7, 2023

View reviewed changes

tenzen-y force-pushed the use-workqueue-when-deleting-jobs branch 2 times, most recently from db70f73 to 0276be0 Compare May 7, 2023 17:38

tenzen-y commented May 7, 2023

View reviewed changes

danielvegamyhre reviewed May 7, 2023

View reviewed changes

pkg/controllers/jobset_controller.go Show resolved Hide resolved

tenzen-y changed the title ~~Limit the number of goroutines used when deleting jobs~~ WIP: Limit the number of goroutines used when deleting jobs May 7, 2023

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 7, 2023

tenzen-y commented May 7, 2023

View reviewed changes

tenzen-y force-pushed the use-workqueue-when-deleting-jobs branch 2 times, most recently from 72ba788 to f49cbc6 Compare May 8, 2023 16:30

tenzen-y changed the title ~~WIP: Limit the number of goroutines used when deleting jobs~~ Limit the number of goroutines used when deleting jobs May 8, 2023

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 8, 2023

k8s-ci-robot assigned danielvegamyhre May 8, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 8, 2023

ahg-g reviewed May 8, 2023

View reviewed changes

Limit the number of goroutines used when deleting jobs

b425f26

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

tenzen-y force-pushed the use-workqueue-when-deleting-jobs branch from f49cbc6 to b425f26 Compare May 10, 2023 16:20

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 10, 2023

tenzen-y mentioned this pull request May 10, 2023

Flaky test: should enable pods to ping each other via hostname #135

Closed

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 10, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 10, 2023

k8s-ci-robot merged commit 749a77c into kubernetes-sigs:main May 10, 2023

tenzen-y deleted the use-workqueue-when-deleting-jobs branch May 11, 2023 06:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Limit the number of goroutines used when deleting jobs #123

Limit the number of goroutines used when deleting jobs #123

tenzen-y commented May 7, 2023

tenzen-y May 7, 2023

tenzen-y May 7, 2023

tenzen-y May 7, 2023

tenzen-y May 7, 2023

danielvegamyhre May 7, 2023

tenzen-y May 7, 2023

kannon92 May 7, 2023

kannon92 May 8, 2023

tenzen-y May 8, 2023

kannon92 May 8, 2023

tenzen-y May 8, 2023

kannon92 May 8, 2023

tenzen-y commented May 8, 2023

danielvegamyhre commented May 8, 2023

ahg-g May 8, 2023

tenzen-y May 8, 2023 •

edited

Loading

tenzen-y May 8, 2023

danielvegamyhre May 8, 2023

ahg-g May 9, 2023

tenzen-y May 10, 2023

ahg-g left a comment

tenzen-y commented May 10, 2023

tenzen-y commented May 10, 2023

danielvegamyhre commented May 10, 2023

ahg-g commented May 10, 2023

ahg-g commented May 10, 2023

k8s-ci-robot commented May 10, 2023

tenzen-y commented May 11, 2023

Limit the number of goroutines used when deleting jobs #123

Limit the number of goroutines used when deleting jobs #123

Conversation

tenzen-y commented May 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y commented May 8, 2023

danielvegamyhre commented May 8, 2023

Choose a reason for hiding this comment

tenzen-y May 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahg-g left a comment

Choose a reason for hiding this comment

tenzen-y commented May 10, 2023

tenzen-y commented May 10, 2023

danielvegamyhre commented May 10, 2023

ahg-g commented May 10, 2023

ahg-g commented May 10, 2023

k8s-ci-robot commented May 10, 2023

tenzen-y commented May 11, 2023

tenzen-y May 8, 2023 •

edited

Loading