Side effects between exitcode of multiprocessing.Process and textual.work with thread = True

[kostrykin]

Spawning a multiprocessing.Process from within a worker thread seems to somehow disturb the exitcode of the process:

import multiprocessing

from textual import work
from textual.app import App


def run_process():
    pass


class ExampleApp(App):

    def on_mount(self):
        self.test()

    @work(exclusive = True, thread = True)
    def test(self):
        p = multiprocessing.Process(target = run_process)
        p.start()
        p.join()
        assert p.exitcode == 0, p.exitcode  # this assertion fails, p.exitcode is 1


if __name__ == "__main__":

    app = ExampleApp()
    app.run()

Importantly: The same code works nicely, if the process is spawned from within the main thread (just remove the @work(exclusive = True, thread = True) line).

---

Work-around: The obvious work-around is to use a Pipe for the exitcode, but it is a workaround:

def run_process(pipe):
    pipe[1].send(0)


class ExampleApp(App):

    def on_mount(self):
        self.test()

    @work(exclusive = True, thread = True)
    def test(self):
        pipe = multiprocessing.Pipe(duplex = False)
        p = multiprocessing.Process(target = run_process, args = (pipe,))
        p.start()
        p.join()
        exitcode = pipe[0].recv()
        assert exitcode == 0, exitcode

---

Update: I can't reproduce the issue with the regular threading module. The MWE above works nicely if the process is spawned from a thread created using the threading module. This issue must be somehow related to Textual, indeed?

[willmcgugan]

I think you are getting an error, and likely a traceback. However when Textual runs, it captures stdout / stderr so that print statements don't write over the UI.

Try setting headless=True on the run() call. That might allow you to see the the traceback. You could also try running the multiprocessing code without Textual to see the error.

In general though, I'd recommend asyncio subprocesses over multiprocessing. It's slightly lower level, bit gives you more control, and is known to work with with Textual.

[kostrykin]

Thanks for the suggestions!

---

I think you are getting an error, and likely a traceback. However when Textual runs, it captures stdout / stderr so that print statements don't write over the UI.

Try setting headless=True on the run() call. That might allow you to see the the traceback.

I have added headless = True to run() as you suggested,

if __name__ == "__main__":

     app = ExampleApp()
     app.run(headless = True)

but there is no error/traceback besides of the one from the failing assertion:

$ python test.py
╭──────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────╮
│ /home/void/.../python3.11/site-packages/textual/worker.py:368 in _run                                     │
│                                                                                                           │
│   365 │   │   self.state = WorkerState.RUNNING                                                            │
│   366 │   │   app.log.worker(self)                                                                        │
│   367 │   │   try:                                                                                        │
│ ❱ 368 │   │   │   self._result = await self.run()                                                         │
│   369 │   │   except asyncio.CancelledError as error:                                                     │
│   370 │   │   │   self.state = WorkerState.CANCELLED                                                      │
│   371 │   │   │   self._error = error                                                                     │
│                                                                                                           │
│ ╭────────────────────────────────── locals ──────────────────────────────────╮                            │
│ │           app = ExampleApp(title='ExampleApp', classes={'-dark-mode'})     │                            │
│ │         error = AssertionError(1)                                          │                            │
│ │          self = <Worker ERROR name='test' description='test()'>            │                            │
│ │     Traceback = <class 'rich.traceback.Traceback'>                         │                            │
│ │ worker_failed = WorkerFailed('Worker raised exception: AssertionError(1)') │                            │
│ ╰────────────────────────────────────────────────────────────────────────────╯                            │
│                                                                                                           │
│ /home/void/.../python3.11/site-packages/textual/worker.py:352 in run                                      │
│                                                                                                           │
│   349 │   │   Returns:                                                                                    │
│   350 │   │   │   Return value of the work.                                                               │
│   351 │   │   """                                                                                         │
│ ❱ 352 │   │   return await (                                                                              │
│   353 │   │   │   self._run_threaded() if self._thread_worker else self._run_async()                      │
│   354 │   │   )                                                                                           │
│   355                                                                                                     │
│                                                                                                           │
│ ╭──────────────────────── locals ────────────────────────╮                                                │
│ │ self = <Worker ERROR name='test' description='test()'> │                                                │
│ ╰────────────────────────────────────────────────────────╯                                                │
│                                                                                                           │
│ /home/void/.../python3.11/site-packages/textual/worker.py:324 in _run_threaded                            │
│                                                                                                           │
│   321 │   │                                                                                               │
│   322 │   │   loop = asyncio.get_running_loop()                                                           │
│   323 │   │   assert loop is not None                                                                     │
│ ❱ 324 │   │   return await loop.run_in_executor(None, runner, self._work)                                 │
│   325 │                                                                                                   │
│   326 │   async def _run_async(self) -> ResultType:                                                       │
│   327 │   │   """Run an async worker.                                                                     │
│                                                                                                           │
│ ╭───────────────────────────────────────── locals ─────────────────────────────────────────╮              │
│ │          loop = <_UnixSelectorEventLoop running=True closed=False debug=False>           │              │
│ │ run_awaitable = <function Worker._run_threaded.<locals>.run_awaitable at 0x761417bb5c60> │              │
│ │  run_callable = <function Worker._run_threaded.<locals>.run_callable at 0x761417bb5da0>  │              │
│ │ run_coroutine = <function Worker._run_threaded.<locals>.run_coroutine at 0x761417bb5d00> │              │
│ │        runner = <function Worker._run_threaded.<locals>.run_callable at 0x761417bb5da0>  │              │
│ │          self = <Worker ERROR name='test' description='test()'>                          │              │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────╯              │
│                                                                                                           │
│ /home/void/.../python3.11/concurrent/futures/thread.py:58 in run                                          │
│                                                                                                           │
│    55 │   │   │   return                                                                                  │
│    56 │   │                                                                                               │
│    57 │   │   try:                                                                                        │
│ ❱  58 │   │   │   result = self.fn(*self.args, **self.kwargs)                                             │
│    59 │   │   except BaseException as exc:                                                                │
│    60 │   │   │   self.future.set_exception(exc)                                                          │
│    61 │   │   │   # Break a reference cycle with the exception 'exc'                                      │
│                                                                                                           │
│ ╭── locals ───╮                                                                                           │
│ │ self = None │                                                                                           │
│ ╰─────────────╯                                                                                           │
│                                                                                                           │
│ /home/void/.../python3.11/site-packages/textual/worker.py:307 in run_callable                             │
│                                                                                                           │
│   304 │   │   def run_callable(work: Callable[[], ResultType]) -> ResultType:                             │
│   305 │   │   │   """Set the active worker, and call the callable."""                                     │
│   306 │   │   │   active_worker.set(self)                                                                 │
│ ❱ 307 │   │   │   return work()                                                                           │
│   308 │   │                                                                                               │
│   309 │   │   if (                                                                                        │
│   310 │   │   │   inspect.iscoroutinefunction(self._work)                                                 │
│                                                                                                           │
│ ╭─────────────────────────────────────────────── locals ────────────────────────────────────────────────╮ │
│ │ self = <Worker ERROR name='test' description='test()'>                                                │ │
│ │ work = functools.partial(<function ExampleApp.test at 0x761417b563e0>, ExampleApp(title='ExampleApp', │ │
│ │        classes={'-dark-mode'}))                                                                       │ │
│ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                           │
│ /home/void/.../test.py:21 in test                                                                         │
│                                                                                                           │
│   18 │   │   p = multiprocessing.Process(target = run_process)                                            │
│   19 │   │   p.start()                                                                                    │
│   20 │   │   p.join()                                                                                     │
│ ❱ 21 │   │   assert p.exitcode == 0, p.exitcode  # this assertion fails, p.exitcode is 1                  │
│   22                                                                                                      │
│   23                                                                                                      │
│   24 if __name__ == "__main__":                                                                           │
│                                                                                                           │
│ ╭─────────────────────────────────── locals ────────────────────────────────────╮                         │
│ │    p = <Process name='Process-1' pid=104984 parent=104982 stopped exitcode=1> │                         │
│ │ self = ExampleApp(title='ExampleApp', classes={'-dark-mode'})                 │                         │
│ ╰───────────────────────────────────────────────────────────────────────────────╯                         │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯
AssertionError: 1

---

You could also try running the multiprocessing code without Textual to see the error.

I did that, I tried replicating the error by using a thread created with the threading module, but this works nicely:

import multiprocessing
import threading


def run_process():
    pass

def test():
    p = multiprocessing.Process(target = run_process)
    p.start()
    p.join()
    assert p.exitcode == 0, p.exitcode  # the assertion does not fail

t = threading.Thread(target = test)
t.start()
t.join()

---

In general though, I'd recommend asyncio subprocesses over multiprocessing

Thanks for pointing this out. I will consider this, however, at the moment I need to deal with another library which is written around multiprocessing and this is why I'm stuck with the multiprocessing.Process requirement at the moment.