stdout metric collector failed

[chenwenjun-github]

/kind bug

What steps did you take and what happened:

I use tfjob as trial's job, my tfjob has one ps, one chief, one worker, and the metric collector is stdout, but I find that the metrics-logger-and-collector container sometimes will become error, and the print like above, this isn't must present.
But this will cause that the worker‘s metric can't be collected.

this error message in code like this:

can you give me some advice to aviod this problem?

What did you expect to happen:
don't appear this error

Environment:

• katib version (kfctl version): v0.10.1
• Kubernetes version: (use kubectl version): 1.13

[chenwenjun-github]

I made some change on katib code like this:

This reslove my problem, close this issue.

[andreyvelich]

Thank you for creating this @chenwenjun-github.
Yes, that should be an issue.
Maybe we should implement this change to the metrics collector source code?

WDYT @chenwenjun-github @gaocegege @johnugeorge ?

/priority p1
/help

[google-oss-robot]

@andreyvelich:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

Thank you for creating this @chenwenjun-github.
Yes, that should be an issue.
Maybe we should implement this change to the metrics collector source code?

WDYT @chenwenjun-github @gaocegege @johnugeorge ?

/priority p1
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[johnugeorge]

Sorry. I didn't understand the context very well.

when will metrics-logger-and-collector container error out?

[andreyvelich]

Sorry. I didn't understand the context very well.

when will metrics-logger-and-collector container error out?

This happens very rarely, when psutil.Pids() returns process, but when we try to create an object from this PID in psutil.NewProcess(pid) it can't be created.
My assumption that, this process might be stopped or finished before.

I think @chenwenjun-github solution above might work, since usually we wait only until mainPid is complete.
I would only verify that mainPid contains the PID after GetMainProcesses is finished.

We can discuss about it on our upcoming WG meeting tomorrow.