Skip to content

Commit

Permalink
Update logging cookbook to show multiple worker processes using the c…
Browse files Browse the repository at this point in the history
…oncurrent.futures module. (GH-14905) (GH-14906)

(cherry picked from commit d309352)
  • Loading branch information
miss-islington authored and vsajip committed Jul 22, 2019
1 parent 22fd679 commit 5b39852
Showing 1 changed file with 35 additions and 0 deletions.
35 changes: 35 additions & 0 deletions Doc/howto/logging-cookbook.rst
Original file line number Diff line number Diff line change
Expand Up @@ -948,6 +948,41 @@ This variant shows how you can e.g. apply configuration for particular loggers
machinery in the main process (even though the logging events are generated in
the worker processes) to direct the messages to the appropriate destinations.

Using concurrent.futures.ProcessPoolExecutor
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If you want to use :class:`concurrent.futures.ProcessPoolExecutor` to start
your worker processes, you need to create the queue slightly differently.
Instead of

.. code-block:: python
queue = multiprocessing.Queue(-1)
you should use

.. code-block:: python
queue = multiprocessing.Manager().Queue(-1) # also works with the examples above
and you can then replace the worker creation from this::

workers = []
for i in range(10):
worker = multiprocessing.Process(target=worker_process,
args=(queue, worker_configurer))
workers.append(worker)
worker.start()
for w in workers:
w.join()

to this (remembering to first import :mod:`concurrent.futures`)::

with concurrent.futures.ProcessPoolExecutor(max_workers=10) as executor:
for i in range(10):
executor.submit(worker_process, queue, worker_configurer)


Using file rotation
-------------------

Expand Down

0 comments on commit 5b39852

Please sign in to comment.