trainning-operator may need to monitor PodGroup

[qiankunli]

I create a pytorchjob, then pytorch-controller create a PodGroup. The PodGroup was pending state at the beginning, so no pod was created for the pytorchjob. After the PodGroup became the inqueue state, It seems that the controller is not listening to the change of Podgroup, so that the reconcile logic could not be triggered, and the job had no pod.

[cheimu]

Hi @qiankunli . I don't know if we are on the same page, but I think the base job controller has already tried to reconcile podGroup https://github.com/kubeflow/common/blob/master/pkg/controller.v1/common/job.go#L256

[qiankunli]

@cheimu I mean, if the state of podgroup changed from pending to inqueue, the reconcile should be triggered. when podgroup is pending, https://github.com/kubeflow/common/blob/master/pkg/controller.v1/common/job.go#L256 break the reconcile. when podgroup is inqueue, the reconcile can not be triggered, so the ReconcilePods and ReconcileServices can not be reached.

[zw0610]

It is true that PodGroup is not watched to trigger reconciliation but the controller does resync after certain period. You may change to a short time for the resynchronization.

[gaocegege]

I do not think the resync works here since podgroup is per job and should be watched in the controller.

[johnugeorge]

Closing this issue as feature is implemented as part of #1724 #1666
/close