fixes a possible race condition in AutoRestartTrick

[ivg]

Just a long shot for a failure observed on #998. My hypothesis is that when we stop ProcessWatcher before we restart the process manually, we don't yield to it and immediately kill the process. Next, when the ProcessWatcher thread is woken up, we have to conditions ready - the popen_obj and stopped_event, see the corresponding code,

 while True: if self.popen_obj.poll() is not None: 
     break 
  if self.stopped_event.wait(timeout=0.1): 
     return 

And despite that stopped_event is set, we first check for popen_obj and trigger the process restart.

We can also make the ProcessWatcher logic more robust, by checking if we are stopped before calling the termination callback, e.g.,

        try:
            if not self.stopped_event.is_set():
                self.process_termination_callback()
        except Exception:
            logger.exception("Error calling process termination callback")

I am not 100% sure about that, as I don't really know what semantics is expected from ProcessWatcher by other users. But at least the AutoRestarter expects this semantics - i.e., a watcher shall not call any events after it was stopped.

[ivg]

Okay, the first option didn't work, so I tried the second and the tests passed, at least on Linux, not sure what's going on with macOS, but it looks like it is a known issue, correct me if I am wrong.

[BoboTiG]

Thanks @ivg 🍾