fix no log from subprocess on trial #3653
Conversation
| @@ -414,13 +414,13 @@ class LocalTrainingService implements TrainingService { | |||
| if (process.platform === 'win32') { | |||
| script.push(`cd $env:NNI_CODE_DIR`); | |||
| script.push( | |||
| `cmd.exe /c ${this.config.trialCommand} 2>&1 | Out-File "${path.join(workingDirectory, 'stderr')}" -encoding utf8`, | |||
| `cmd.exe /c ${this.config.trialCommand} 1> "${path.join(workingDirectory, 'stdout')}" 2> "${path.join(workingDirectory, 'stderr')}" -encoding utf8`, | |||
There was a problem hiding this comment.
I'm not familiar with cmd. Can someone check this?
| `$NOW_DATE = [int64](([datetime]::UtcNow)-(get-date "1/1/1970")).TotalSeconds`, | ||
| `$NOW_DATE = "$NOW_DATE" + (Get-Date -Format fff).ToString()`, | ||
| `Write $LASTEXITCODE " " $NOW_DATE | Out-File "${path.join(workingDirectory, '.nni', 'state')}" -NoNewline -encoding utf8`); | ||
| } else { | ||
| script.push(`cd $NNI_CODE_DIR`); | ||
| script.push(`eval ${this.config.trialCommand} 2>"${path.join(workingDirectory, 'stderr')}"`); | ||
| script.push(`eval ${this.config.trialCommand} 1>"${path.join(workingDirectory, 'stdout')} 2>"${path.join(workingDirectory, 'stderr')}"`); |
There was a problem hiding this comment.
We still cannot view stdout from WebUI. That should be a problem?
There was a problem hiding this comment.
FIxed,
After some investigate about this issue, here are some result:
option1:
sys.stdout redirect for trial.py (->trial.log)
result:
only trial.py log show up, no subprocess log. (This is the issue linked on this PR)
option2:
sys.stdout redirect for trial.py (->trial.log)
launch trial.py with command "1 > stdout"
result:
trial.py log is in trial.log, subprocess log is in stdout.
option3:
sys.stdout redirect for trial.py (->trial.log)
launch trial.py with command "1 > trial.log"
result:
Thread save exceptions.
option4:
launch trial.py with command "1 > stdout"
result:
trial.py log and subprocess log are in stdout.
I recommend remove sys.stdout redirect which is only works for local platform.
(Somehow, It is no works for subprocess output if only setup "sys.stdout" on parent process, You have to change the code by "subprocess.Popen(your_command, stdout=sys.stdout)")
No description provided.