Repository navigation
multiprocessing.Pool in spawn mode breaks logging.handlers.QueueHandler configuration #121723
Description
Bug report
Bug description:
multiprocessing.Pool with spawn method breaks logging.handlers.QueueHandler configuration
import logging
import logging.config
import multiprocessing
import time
def _init(q):
logging.config.dictConfig({
'version': 1,
'disable_existing_loggers': True,
'handlers': {'log_to_parent': {'class': 'logging.handlers.QueueHandler', 'queue': q}},
'root': {'handlers': ['log_to_parent'], 'level': 'DEBUG'}
})
logging.getLogger().info('log from child')
if __name__ == '__main__':
multiprocessing.set_start_method('spawn')
with multiprocessing.Manager() as manager:
# q = manager.Queue()
q = multiprocessing.Queue()
# listen for log messages from child processes
# ...
with multiprocessing.Pool(processes=1, maxtasksperchild=1, initializer=_init, initargs=(q,)) as pool:
time.sleep(1)got exception (when q = manager.Queue() or q = multiprocessing.Queue())
Process SpawnPoolWorker-2:
Traceback (most recent call last):
File "/home/foo/miniconda3/envs/py312forge/lib/python3.12/logging/config.py", line 581, in configure
handler = self.configure_handler(handlers[name])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/foo/miniconda3/envs/py312forge/lib/python3.12/logging/config.py", line 792, in configure_handler
proxy_queue = MM().Queue()
^^^^
File "/home/foo/miniconda3/envs/py312forge/lib/python3.12/multiprocessing/context.py", line 57, in Manager
m.start()
File "/home/foo/miniconda3/envs/py312forge/lib/python3.12/multiprocessing/managers.py", line 562, in start
self._process.start()
File "/home/foo/miniconda3/envs/py312forge/lib/python3.12/multiprocessing/process.py", line 118, in start
assert not _current_process._config.get('daemon'), \
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: daemonic processes are not allowed to have children
The above exception was the direct cause of the following exception:
...
CPython versions tested on:
3.12
Operating systems tested on:
Linux
Linked PRs
- gh-121723: Relax constraints on queue objects for
logging.handlers.QueueHandler. #122154 - [3.13] gh-121723: Relax constraints on queue objects for
logging.handlers.QueueHandler. (GH-122154) #122603 - [3.12] gh-121723: Relax constraints on queue objects for
logging.handlers.QueueHandler. (GH-122154) #122604 - GH-121723: Skip
test_config_queue_handler_multiprocessing_contextin emulated JIT CI #122969 - [3.13] GH-121723: Skip test_config_queue_handler_multiprocessing_context in emulated JIT CI (GH-122969) #122991
Metadata
Metadata
Assignees
Projects
- Status
Activity
[-]multirpocessing.Pool in spawn mode breaks logging.handlers.QueueHandler configuration[/-][+]multiprocessing.Pool in spawn mode breaks logging.handlers.QueueHandler configuration[/+]picnixz commentedon Jul 22, 2024
@vsajip I suspect this is caused by the fact that you instantiate a queue to get its runtime type. Maybe a protocol approach for a queue would be better? I don't know if the queue you expect should be any queue or if whatever is a queue-like API is fine.
vsajip commentedon Jul 22, 2024
A queue-like API is all that's needed, but by protocol approach you mean just checking for the existence of queue methods used by logging? The fundamental problem here is that a proxy is used from which you can't easily infer what type the proxy is to. Another, perhaps more generally useful way might be to expose the underlying runtime type on the proxy so that it can be readily queried.
picnixz commentedon Jul 22, 2024
Yes. Don't really care about the type itself, just checking that methods of certain names exist, not even their signatures. And if something goes wrong, there would be a TypeError when you want to use the queue I think. You could reduce issues by having a well-documented section on what kind of queue API you expect, and that should be enough I think.
picnixz commentedon Jul 22, 2024
IIRC, the proxy is a local class and it does not necessarily require to have a class. It's a bit hard to explain here but if you look at
multiprocessing.managers.BaseManager.registerand how queue classes are registered in managers, you'll see that keeping track of the base type might be more tricky.vsajip commentedon Jul 22, 2024
Right, I had a quick look when this issue came up before and it didn't seem that straightforward, which is why I went with the approach of instantiating the queue to get its type 😞
picnixz commentedon Jul 22, 2024
So... protocol approach it will be? (I can make a PR tomorrow or in two days)
gh-121723: Relax constraints on queue objects for `logging.handlers.Q…
pythongh-121723: Relax constraints on queue objects for `logging.hand…
pythongh-121723: Relax constraints on queue objects for `logging.hand…
[3.12] gh-121723: Relax constraints on queue objects for `logging.han…
[3.13] gh-121723: Relax constraints on queue objects for `logging.han…
pythongh-121723: Relax constraints on queue objects for `logging.hand…
GH-121723: Skip test_config_queue_handler_multiprocessing_context in …
pythonGH-121723: Skip test_config_queue_handler_multiprocessing_conte…
pythongh-121723: Relax constraints on queue objects for `logging.hand…
pythonGH-121723: Skip test_config_queue_handler_multiprocessing_conte…
[3.13] GH-121723: Skip test_config_queue_handler_multiprocessing_cont…