multiprocessing.Pool in spawn mode breaks logging.handlers.QueueHandler configuration

[noreentry]

Bug report

Bug description:

multiprocessing.Pool with spawn method breaks logging.handlers.QueueHandler configuration

import logging
import logging.config
import multiprocessing
import time


def _init(q):
    logging.config.dictConfig({
        'version': 1,
        'disable_existing_loggers': True,
        'handlers': {'log_to_parent': {'class': 'logging.handlers.QueueHandler', 'queue': q}},
        'root': {'handlers': ['log_to_parent'], 'level': 'DEBUG'}
    })
    logging.getLogger().info('log from child')


if __name__ == '__main__':    
    multiprocessing.set_start_method('spawn')
    with multiprocessing.Manager() as manager:
        # q = manager.Queue()
        q = multiprocessing.Queue()
        
        # listen for log messages from child processes
        # ...

        with multiprocessing.Pool(processes=1, maxtasksperchild=1, initializer=_init, initargs=(q,)) as pool:
            time.sleep(1)

got exception (when q = manager.Queue() or q = multiprocessing.Queue())

Process SpawnPoolWorker-2:
Traceback (most recent call last):
  File "/home/foo/miniconda3/envs/py312forge/lib/python3.12/logging/config.py", line 581, in configure
    handler = self.configure_handler(handlers[name])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/foo/miniconda3/envs/py312forge/lib/python3.12/logging/config.py", line 792, in configure_handler
    proxy_queue = MM().Queue()
                  ^^^^
  File "/home/foo/miniconda3/envs/py312forge/lib/python3.12/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/home/foo/miniconda3/envs/py312forge/lib/python3.12/multiprocessing/managers.py", line 562, in start
    self._process.start()
  File "/home/foo/miniconda3/envs/py312forge/lib/python3.12/multiprocessing/process.py", line 118, in start
    assert not _current_process._config.get('daemon'), \
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: daemonic processes are not allowed to have children

The above exception was the direct cause of the following exception:
...

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

• gh-121723: Relax constraints on queue objects for logging.handlers.QueueHandler. #122154
• [3.13] gh-121723: Relax constraints on queue objects for logging.handlers.QueueHandler. (GH-122154) #122603
• [3.12] gh-121723: Relax constraints on queue objects for logging.handlers.QueueHandler. (GH-122154) #122604
• GH-121723: Skip test_config_queue_handler_multiprocessing_context in emulated JIT CI #122969
• [3.13] GH-121723: Skip test_config_queue_handler_multiprocessing_context in emulated JIT CI (GH-122969) #122991

[picnixz]

@vsajip I suspect this is caused by the fact that you instantiate a queue to get its runtime type. Maybe a protocol approach for a queue would be better? I don't know if the queue you expect should be any queue or if whatever is a queue-like API is fine.

[vsajip]

A queue-like API is all that's needed, but by protocol approach you mean just checking for the existence of queue methods used by logging? The fundamental problem here is that a proxy is used from which you can't easily infer what type the proxy is to. Another, perhaps more generally useful way might be to expose the underlying runtime type on the proxy so that it can be readily queried.

[picnixz]

A queue-like API is all that's needed, but by protocol approach you mean just checking for the existence of queue methods used by logging

Yes. Don't really care about the type itself, just checking that methods of certain names exist, not even their signatures. And if something goes wrong, there would be a TypeError when you want to use the queue I think. You could reduce issues by having a well-documented section on what kind of queue API you expect, and that should be enough I think.

[picnixz]

Another, perhaps more generally useful way might be to expose the underlying runtime type on the proxy so that it can be readily queried

IIRC, the proxy is a local class and it does not necessarily require to have a class. It's a bit hard to explain here but if you look at multiprocessing.managers.BaseManager.register and how queue classes are registered in managers, you'll see that keeping track of the base type might be more tricky.

[vsajip]

Right, I had a quick look when this issue came up before and it didn't seem that straightforward, which is why I went with the approach of instantiating the queue to get its type 😞

[picnixz]

So... protocol approach it will be? (I can make a PR tomorrow or in two days)