Determine when child process terminates unexpectedly (crashes)

[johnib]

Hello,

I'm using CliWrap with the following flavor:

            Command command = Cli.Wrap(filePath)
                .WithArguments(arguments)
                .WithValidation(CommandResultValidation.ZeroExitCode);

            _logger.LogInformation("Spawning process with command: {command}", command);
            await command.Observe(Encoding.UTF8, stoppingToken)
                .ForEachAsync(commandEvent =>

Is there any way I can determine the child process being spawned by CliWrap crashes? (which in that case I'd want to restart it..)
Initially I expected to it to be covered by the following case:

                        case ExitedCommandEvent exitedCommandEvent:
                            _logger.LogCritical("Process exited with code: {exitCode}", exitedCommandEvent.ExitCode);
                            break;

However, this only catches the case of graceful termination.

Hope my question is clear..

Thanks for helping!

[Tyrrrz]

Hi.

Hm. The exited event should definitely cover non-graceful termination as well. What do you experience instead? Does it hang? How exactly is the process getting terminated?

[johnib]

Hi

I've put a breakpoint on the 'exit' event, but it was never called and in fact the main process just hanged, waiting for more events from the Observable.

The way the process terminated was by me killing it via Windows' Task Manager.

[Tyrrrz]

Can you see if it also hangs when you use other execution models (i.e. ExecuteAsync()/ListenAsync())?

[johnib]

With ExecuteAsync() I get the same "hanging" behavior:

            Command command = Cli.Wrap(filePath)
                .WithArguments(arguments)
                .WithValidation(CommandResultValidation.ZeroExitCode);

            var result = await command.ExecuteAsync(stoppingToken);
            _logger.LogInformation("Result: {result}", result);

The last log line was never printed after killing the child process

[Tyrrrz]

I just tried the following code:

        [Fact]
        public async Task Test()
        {
            // Arrange
            var cmd = Cli.Wrap("dotnet")
                .WithArguments(a => a
                    .Add(Dummy.Program.FilePath)
                    .Add(Dummy.Program.Sleep)
                    .Add(100_000));

            // Act
            await cmd.ExecuteAsync();

            // Assert
            Assert.True(true);
        }

This is using the dummy program from tests and this command causes it to sleep for 100 seconds. While it was waiting, I killed it with task manager and await cmd.ExecuteAsync() completed with an exception (exit code not set to zero).

It seems I can't reproduce your issue. Could it be that there's something special in your setup?

[johnib]

This is weird.. are you running on Linux? or Windows?

Can you try setting up a simple console app, spawning some process, and kill it as well via Task Manager?
(just don't run it as a unit test via mstest/whatever)

[Tyrrrz]

Yes, I'm running it on Windows. Console or test shouldn't make a difference, but I'll try.

[Tyrrrz]

Tried it on a console, works the same way.

Can you try the same thing but with bare System.Diagnostics.Process? That's what CliWrap uses under the hood and relies on it to report when the process is terminated. Link

[vivainio]

I am having this same problem with one particular process. It's launched with "cmd /c mycommand.cmd". It stops yielding events when it has exited. It's a normal cmd file that exits with zero status.

My code:

            await foreach (var cmdEvent in cmd.ListenAsync())
            {
                switch (cmdEvent)
                {
                    case StartedCommandEvent start:
                        write(v, cname, "Start pid " + start.ProcessId);
                        RunningPid[index] = start.ProcessId;
                        break;
                    case StandardOutputCommandEvent stdOut:
                        write(v, cname, stdOut.Text); 
                        break;
                    case StandardErrorCommandEvent stdErr:
                        write(v, "red", stdErr.Text); 
                        break;
                    case ExitedCommandEvent ex:
                        return (ex.ExitCode, 0);
                    default:
                        write("UNK", "red", "Unknown event :" + cmdEvent.ToString());
                        break;
                }
            }

I also confirmed in debugger that process exited event was emitted:

// in ProcessEx.cs

    public void Start()
    {
        // Hook up events
        _nativeProcess.EnableRaisingEvents = true;
        _nativeProcess.Exited += (_, _) =>
        {
            ExitTime = DateTimeOffset.Now; // THIS was called
            _exitTcs.TrySetResult(null);
        };

... so it's some kind of race condition somewhere?

Further debugging indicated that TrySetResult returned true (success) and the following await statement blocked forever:

// Command.Execution.cs
            try
            {
                // Wait until piping is done and propagate exceptions
                await pipingTask.ConfigureAwait(false);
            }

@Tyrrrz do you think this is enough information to reopen the ticket? I can do further diagnostics if needed

[Tyrrrz]

@Tyrrrz do you think this is enough information to reopen the ticket? I can do further diagnostics if needed

Yes, please do.

[vivainio]

@Tyrrrz do you think this is enough information to reopen the ticket? I can do further diagnostics if needed

Yes, please do.

I think you need to do it, I don't have the rights

[vivainio]

@Tyrrrz I created a new ticket instead of bothering with this closed one, #185

[Tyrrrz]

@Tyrrrz I created a new ticket instead of bothering with this closed one, #185

Yes, that's what I meant. Thank you.