execDuration does not incrase when TUNER_NO_MORE_TRIAL

[prismformore]

Environment:

• NNI version: v1.7
• NNI mode (local|remote|pai): local
• Client OS: ubuntu 16.04
• Server OS (for remote mode only):
• Python version: .3.6
• PyTorch/TensorFlow version: Pytorch 1.6
• Is conda/virtualenv/venv used?: Yes
• Is running in Docker?: Yes

Log message:
`$ nnictl experiment show
return:
"execDuration":1,
......
'

What issue meet, what's expected?:
When using BOHB, execDuration does not increase when status=='TUNER_NO_MORE_TRIAL', because it only increases when status == 'RUNNING', as shown here:

nni/src/nni_manager/core/nnimanager.ts

Line 463 in 41312de

if (this.status.status === 'RUNNING') {

However, I think the time consumption of waiting for the trials to finish should also be considered. The trials ARE running.

As mentioned in this issue,

Simply put, it goes through the process of: [generate n1 parameters] -> [get n1 parameters' metrics to update BO model](indicates with TUNER_NO_MORE_TRIAL) -> [generate n2 parameters based on new model] -> [get n2 parameters] .....

The 'TUNER_NO_MORE_TRIAL' is supposed to be very long (most experiment time is spent on training and evaluating the trials, which means the status is 'TUNER_NO_MORE_TRIAL').

How to reproduce it?:
Use BOHB with limited computation resources.
$ nnictl experiment show

Additional information:

[prismformore]

And it is probably the same for the 'NO_MORE_TRIAL' status

[prismformore]

Anybody noticing this issue? I think this problem is quite critical.

[liuzhe-lz]

Ooops, don't know why I missed this issue. We will investigate this problem soon.

[J-shang]

Very grateful for your issue. The good news is that this problem will be fixed in version 2.0 ( #3043 ) and this issue will be closed. If there are any questions, please continue to communicate with us, we will confirm and resolve them as soon as possible.