fix: tfjob with restartPolicy=ExitCode not work #1562

cheimu · 2022-03-23T19:52:12Z

What this PR does / why we need it:
As describe in issues linked below, the tfjob with restartPolicy=ExitCode won’t work. If workers/evaluators got exited, the whole job will be set to failed instead of restarting, which is not expected.

Need for more comments! Thank you in advance! :D

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #1560, kubeflow/common#186

Checklist:

Docs included if any changes are user facing

aws-kf-ci-bot · 2022-03-23T19:52:26Z

Hi @cheimu. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

coveralls · 2022-03-23T19:57:00Z

Pull Request Test Coverage Report for Build 2030402854

14 of 16 (87.5%) changed or added relevant lines in 1 file are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.2%) to 37.061%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/controller.v1/tensorflow/tfjob_controller.go	14	16	87.5%

Files with Coverage Reduction	New Missed Lines	%
pkg/controller.v1/tensorflow/tfjob_controller.go	1	71.04%

Totals
Change from base Build 2014597889:	0.2%
Covered Lines:	2290
Relevant Lines:	6179

💛 - Coveralls

richardsliu · 2022-03-23T21:42:13Z

/ok-to-test

richardsliu · 2022-03-23T22:08:29Z

pkg/controller.v1/tensorflow/tfjob_controller.go

+			// the restarting condition will be removed from jobStatus by commonv1.filterOutCondition(),
+			// so we need to append the restarting condition back to jobStatus.
+			if existingRestartingCondition != nil {
+				err := commonutil.UpdateJobConditions(jobStatus, commonv1.JobRestarting, existingRestartingCondition.Reason, existingRestartingCondition.Message)


Should we reset the value of existingRestartingCondition after this line?

Hi @richardsliu, I don't know if we could reset it. I record the existingRestartingCondition outside the loop, which is the condition and result for current reconcile(). Inside the loop, when iterate over all pods, if there are multiple workers or masters .etc, are running, and only one worker failed, by the logic inside of updateJobConditions(), restarting + running combination will make running condition overwrite restarting.

@richardsliu is this the right way to deal with it? If one of the pods is restarting, do we mark the whole job as restarting ?

If we do something along the lines of, would it achieve the same thing?

if condition.Type == commonv1.JobRunning || condition.Type == commonv1.JobRestarting { restart = true

Hi @pavanky, I would like to hear more advices: ) For now, I just follow the current convention. See https://github.com/kubeflow/training-operator/blob/master/pkg/controller.v1/tensorflow/tfjob_controller.go#L762 . Current strategy is that if one of the pods failed, then update the whole job status to failed : /

Wouldn't that technically mean we are updating the job status to restart and incrementing the count two times for the same failure ?

IMO, we could see it from 2 points of views.

If jobStatus is running, and one of roles got failed pods, then the whole job will be updated to failed, then in next reconcile, the pods will be deleted, so jobstatus should have a restarting (don't know if I understand the code base correctly, we don't have specific restarting logic but use general reconcile logic)

Technically we just set jobstatus's field twice without sending request to apiServer, so in fact, it didn't get really updated. But yeah, current one is not elegant. 🤔

cheimu · 2022-03-24T01:59:56Z

/retest

gaocegege · 2022-03-24T02:56:43Z

cc @rllin @pavanky

richardsliu · 2022-03-25T00:13:05Z

/lgtm

gaocegege

LGTM 👍
/lgtm

cheimu · 2022-05-05T06:47:17Z

Hi @richardsliu , what should we do next for this pr/issue?

emenendez · 2022-07-25T18:08:05Z

Hi all! Just wanted to ask if there is intent to merge this PR -- our users are definitely impacted by this issue and we'd really benefit from the fix. Thanks so much!

gaocegege · 2022-07-25T22:32:41Z

We get two LGTMs here. I think we should merge it.

Sorry for the late reply.

/approve.

gaocegege · 2022-07-25T22:33:17Z

/approve

google-oss-prow · 2022-07-25T22:33:35Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cheimu, gaocegege

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [gaocegege]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

emenendez · 2022-07-26T01:00:37Z

Thanks so much for the speedy merge!

(cherry picked from commit 9cc1cc9)

…le-cherrypick-exitcode-fix fix: tfjob with restartPolicy=ExitCode not work (kubeflow#1562)

fix tfjob with restartPolicy=exitCode not work

0e7bd9f

google-oss-prow bot added the size/S label Mar 23, 2022

aws-kf-ci-bot added the needs-ok-to-test label Mar 23, 2022

google-oss-prow bot requested review from jinchihe, johnugeorge and zw0610 March 23, 2022 19:52

cheimu changed the title ~~fix tfjob with restartPolicy=exitCode not work~~ fix: tfjob with restartPolicy=ExitCode not work Mar 23, 2022

aws-kf-ci-bot added ok-to-test and removed needs-ok-to-test labels Mar 23, 2022

richardsliu reviewed Mar 23, 2022

View reviewed changes

cheimu mentioned this pull request Mar 24, 2022

Add cheimu to Kubeflow Member List kubeflow/internal-acls#541

Merged

google-oss-prow bot assigned richardsliu Mar 25, 2022

google-oss-prow bot added the lgtm label Mar 25, 2022

gaocegege reviewed Mar 25, 2022

View reviewed changes

google-oss-prow bot assigned gaocegege Mar 25, 2022

google-oss-prow bot added the approved label Jul 25, 2022

google-oss-prow bot merged commit 9cc1cc9 into kubeflow:master Jul 25, 2022

djwhatle pushed a commit to djwhatle/training-operator that referenced this pull request Jul 26, 2022

fix: tfjob with restartPolicy=ExitCode not work (kubeflow#1562)

eee8d6b

(cherry picked from commit 9cc1cc9)

djwhatle added a commit to djwhatle/training-operator that referenced this pull request Jul 26, 2022

Merge pull request #1 from djwhatle/v1.3.0-enable-partial-gang-schedu…

b157f01

…le-cherrypick-exitcode-fix fix: tfjob with restartPolicy=ExitCode not work (kubeflow#1562)

johnugeorge mentioned this pull request Dec 22, 2022

When restartPolicy is ExitCode and a pod is deleted (137), the entire TFJob will still be marked as failed. kubeflow/common#186

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: tfjob with restartPolicy=ExitCode not work #1562

fix: tfjob with restartPolicy=ExitCode not work #1562

cheimu commented Mar 23, 2022

aws-kf-ci-bot commented Mar 23, 2022

coveralls commented Mar 23, 2022

richardsliu commented Mar 23, 2022

richardsliu Mar 23, 2022

cheimu Mar 24, 2022

pavanky Mar 25, 2022

pavanky Mar 25, 2022

cheimu Mar 25, 2022

pavanky Mar 25, 2022 •

edited

Loading

cheimu Mar 25, 2022 •

edited

Loading

cheimu commented Mar 24, 2022

gaocegege commented Mar 24, 2022

richardsliu commented Mar 25, 2022

gaocegege left a comment

cheimu commented May 5, 2022

emenendez commented Jul 25, 2022

gaocegege commented Jul 25, 2022

gaocegege commented Jul 25, 2022

google-oss-prow bot commented Jul 25, 2022

emenendez commented Jul 26, 2022

fix: tfjob with restartPolicy=ExitCode not work #1562

fix: tfjob with restartPolicy=ExitCode not work #1562

Conversation

cheimu commented Mar 23, 2022

aws-kf-ci-bot commented Mar 23, 2022

coveralls commented Mar 23, 2022

Pull Request Test Coverage Report for Build 2030402854

💛 - Coveralls

richardsliu commented Mar 23, 2022

richardsliu Mar 23, 2022

Choose a reason for hiding this comment

cheimu Mar 24, 2022

Choose a reason for hiding this comment

pavanky Mar 25, 2022

Choose a reason for hiding this comment

pavanky Mar 25, 2022

Choose a reason for hiding this comment

cheimu Mar 25, 2022

Choose a reason for hiding this comment

pavanky Mar 25, 2022 • edited Loading

Choose a reason for hiding this comment

cheimu Mar 25, 2022 • edited Loading

Choose a reason for hiding this comment

cheimu commented Mar 24, 2022

gaocegege commented Mar 24, 2022

richardsliu commented Mar 25, 2022

gaocegege left a comment

Choose a reason for hiding this comment

cheimu commented May 5, 2022

emenendez commented Jul 25, 2022

gaocegege commented Jul 25, 2022

gaocegege commented Jul 25, 2022

google-oss-prow bot commented Jul 25, 2022

emenendez commented Jul 26, 2022

pavanky Mar 25, 2022 •

edited

Loading

cheimu Mar 25, 2022 •

edited

Loading