Better explanation of why trials failed - Trial is missing K8s events and failure condition could use a better reason

[jlewi]

/kind feature

Describe the solution you'd like

I have a trial in the failed state.

apiVersion: kubeflow.org/v1alpha2
kind: Trial
metadata:
  ...
spec:
  ...
status:
  completionTime: 2019-10-02T14:24:36Z
  conditions:
  - lastTransitionTime: 2019-10-02T13:24:25Z
    lastUpdateTime: 2019-10-02T13:24:25Z
    message: Trial is created
    reason: TrialCreated
    status: "True"
    type: Created
  - lastTransitionTime: 2019-10-02T14:24:36Z
    lastUpdateTime: 2019-10-02T14:24:36Z
    message: Trial is running
    reason: TrialRunning
    status: "False"
    type: Running
  - lastTransitionTime: 2019-10-02T14:24:36Z
    lastUpdateTime: 2019-10-02T14:24:36Z
    message: Trial has failed
    reason: TrialFailed
    status: "True"
    type: Failed
  observation:
    metrics:
    - name: accuracy
      value: -0.08421557
  startTime: 2019-10-02T13:24:25Z

It is unclear why the trial is failed. This is especially confusing since an observation metric is reported which would suggest that the trial succeeded.

I looked for Kubernetes events associated with the trial and actually found none. I would have expected Kubernetes events to be associated with the trial indicating creation of the training jobs and metric collectors as well as reporting any failures.

It would also be good to include a more informative message in the TrialFailed condition about why the trial failed.

[gaocegege]

Can we aggregate the job/tfjob/pytorchjob 's event into trial's event? Does it work for you?

[johnugeorge]

@jlewi Did the job actually error out when you see the status of job?

[jlewi]

@johnugeorge the jobs were deleted presumably by the trial when it entered the failed state. The events for the Job show pods created but that's it. I looked at the pods and it looks like the container may not have started because of "failed to set up sandbox container".

@gaocegege My expectation is that a trial should emit events corresponding to the actions it takes. So when a trial creates a job it should emit an event indicating that the job was created.

Similarly when a Trial enters a failed state it should emit an event indicating that and there should be a reason. The reason should be indicative of why the Trial decided that it was failed. I wouldn't expect that to just be an aggregation of job events.