ProcessPoolExecutor: a worker spawned on demand by submit() can die unnoticed (no BrokenProcessPool, hang) with forkserver/spawn #161931

Bug report

Bug description

With the forkserver or spawn start method, ProcessPoolExecutor starts workers on demand in submit().
submit() wakes the executor manager thread before it spawns the new worker:

            # Wake up queue management thread
            self._executor_manager_thread_wakeup.wakeup()

            if self._safe_to_dynamically_spawn_children:
                self._adjust_process_count()

The manager thread can wake, build its list of worker sentinels, and go back to waiting before
_adjust_process_count() adds the new process. If no further result or submit wakes it, the new worker's sentinel
is never waited on: when that worker is killed (SIGKILL, the OOM killer) while the other workers are busy, the pool
is not marked broken, no future gets BrokenProcessPool, and the caller waits until another worker happens to
finish. With long tasks, or tasks that wait on something, that is a hang. With fork all workers are started up
front and the problem does not occur.

Reproducer

import concurrent.futures as cf
import multiprocessing as mp
import os
import signal
import sys
import time
                                                                                                 
def once(method):
    with cf.ProcessPoolExecutor(max_workers=2, mp_context=mp.get_context(method)) as ex:
        futs = [ex.submit(time.sleep, 30) for _ in range(2)]     # every worker busy: no result, no wakeup
        while len(ex._processes) < 2:
            time.sleep(0.01)
        time.sleep(1)
        newest = list(ex._processes.values())[-1]                 # the worker the second submit() spawned
        os.kill(newest.pid, signal.SIGKILL)
        done, _ = cf.wait(futs, timeout=10)
        hung = not done
        for p in ex._processes.values():                          # end the other worker, so the pool can close
            try:
                os.kill(p.pid, signal.SIGKILL)
            except ProcessLookupError:
                pass
    return hung


if __name__ == "__main__":
    for method in sys.argv[1:] or ["forkserver", "spawn", "fork"]:
        n = 5
        print(sys.version.split()[0], method, "no future completed within 10 s:",
              sum(once(method) for _ in range(n)), "of", n, flush=True)

Note: the reproducer reads a private attribute, ex._processes, only to find the worker the second submit()
spawned (the one whose sentinel is missed) and to end the other worker afterwards.

Expected: within 10 s of the kill both futures fail with BrokenProcessPool (the documented behaviour for a
worker "killed from the outside"). Observed: neither completes until the surviving worker's 30 s task ends.

Results

(An earlier version of the reproducer, with a shorter wait before the kill, hung less often under spawn (2 of 3),
consistent with a race.) Not tried on main.

Possible fix

Spawn before waking the manager (move the wakeup() after _adjust_process_count()), or have the manager
rebuild its sentinel list when _adjust_process_count() adds a process (e.g. a second wakeup() after the spawn).

Workaround

Poll p.exitcode of executor._processes on a timeout while waiting for futures, and treat a worker with an exit
code as a broken pool (valid when max_tasks_per_child is not used, since workers then never exit on their own).

Related

#159689 (and its PR #159697) is a similar hang with a different window: the worker is killed while its result is
in the pipe. Here no result is in flight; the manager never waits on the new worker's sentinel at all, so #159697
does not cover it.

CPython versions tested on:

3.12, 3.14

Operating systems tested on:

Linux

Linked PRs

• gh-161931: Wake up process manager after spawning workers #161953