-
Notifications
You must be signed in to change notification settings - Fork 438
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ignore tfjob/pytorch job if corresponding CRD not created #335
Conversation
/assign @richardsliu @YujiOshima |
/assign @johnugeorge |
@johnugeorge Do we still need this fix based on what we discussed today? |
pkg/controller/studyjob/utils.go
Outdated
func validateWorkerResource(wkind string) error { | ||
for _, crd := range invalidCRDResources { | ||
if crd == wkind { | ||
return fmt.Errorf("Cannot support %s; If CRD for it installed, please restart studyjob-controller to take effect", wkind) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe specifically say "Please install the CRD and restart studyjob-controller"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! updated
@@ -117,7 +121,8 @@ func add(mgr manager.Manager, r reconcile.Reconciler) error { | |||
OwnerType: &katibv1alpha1.StudyJob{}, | |||
}) | |||
if err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
invalidCRDResources should be updated only when CRD is not installed. We should explicitly check whether CRD exists or not. If watch fails for any other reason(even when CRD is installed), controller should fail by raising error.
You can use crd clientset to check.
https://github.com/kubeflow/tf-operator/blob/master/cmd/tf-operator.v1beta1/app/server.go#L192
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, I will handle it
pkg/controller/studyjob/utils.go
Outdated
return nil | ||
} | ||
|
||
func ignoreWatchError(err error, job string) bool { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The name of the function is a little confusing. Maybe change it to "isFatalWatchError" and return the opposite boolean values?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! Done!
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: richardsliu The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
As a short-term solution
Fixes: #317
This change is![Reviewable](https://camo.githubusercontent.com/23b05f5fb48215c989e92cc44cf6512512d083132bd3daf689867c8d9d386888/68747470733a2f2f72657669657761626c652e696f2f7265766965775f627574746f6e2e737667)