Skip to content

gh-115634: Fix ProcessPoolExecutor deadlock with max_tasks_per_child #140900

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
gpshead wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

gh-115634: Fix ProcessPoolExecutor deadlock with max_tasks_per_child #140900

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
47959a4
gh-115634: Fix ProcessPoolExecutor deadlock with max_tasks_per_child
gpshead Nov 2, 2025
c8dd63c
gh-115634: Improve test robustness for max_tasks_per_child tests
gpshead Nov 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
gh-115634: Improve test robustness for max_tasks_per_child tests 
Make the gh-115634 regression tests more robust:

1. Add try/finally blocks to ensure executor.shutdown() is always
   called, preventing resource leaks on test failures or timeouts

2. Replace loops with individual assertions with single list
   comparisons for clearer failure messages - when a test fails,
   the assertion will show exactly which results differ

3. Add descriptive assertion messages that explain the issue context
   without requiring reading the test code

These improvements ensure tests clean up properly even when run
against unpatched CPython where the deadlock would occur, and make
assertion failures more actionable for debugging.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
  • Loading branch information
@gpshead @claude
gpshead and claude committed Nov 2, 2025
commit c8dd63c2e72df3e5463b6f0f160e6761fdb4229d
56 changes: 33 additions & 23 deletions Lib/test/test_concurrent_futures/test_process_pool.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,14 +210,17 @@ def test_max_tasks_per_child_rapid_task_submission_gh115634(self):

# Test the exact bug scenario: 1 worker, max_tasks=2, 10 tasks
executor = self.executor_type(1, mp_context=context, max_tasks_per_child=2)
futures = [executor.submit(mul, i, i) for i in range(10)]

# All futures should complete without hanging
for i, future in enumerate(futures):
result = future.result(timeout=support.SHORT_TIMEOUT)
self.assertEqual(result, i * i)

executor.shutdown()
try:
futures = [executor.submit(mul, i, i) for i in range(10)]

# All futures should complete without hanging
results = [future.result(timeout=support.SHORT_TIMEOUT) for future in futures]
expected = [i * i for i in range(10)]
self.assertEqual(results, expected,
"ProcessPoolExecutor with max_tasks_per_child=2 failed to "
"complete all 10 tasks (gh-115634 deadlock regression)")
finally:
executor.shutdown(wait=True, cancel_futures=True)

def test_max_tasks_per_child_multiple_worker_restarts_gh115634(self):
# gh-115634: Test multiple restart cycles with various max_tasks values
Expand All @@ -227,14 +230,18 @@ def test_max_tasks_per_child_multiple_worker_restarts_gh115634(self):

# Test with max_tasks=3, forcing multiple restarts
executor = self.executor_type(1, mp_context=context, max_tasks_per_child=3)
futures = [executor.submit(mul, i, 2) for i in range(15)]

# All 15 tasks should complete (requires 5 worker restarts)
for i, future in enumerate(futures):
result = future.result(timeout=support.SHORT_TIMEOUT)
self.assertEqual(result, i * 2)

executor.shutdown()
try:
futures = [executor.submit(mul, i, 2) for i in range(15)]

# All 15 tasks should complete (requires 5 worker restarts)
results = [future.result(timeout=support.SHORT_TIMEOUT) for future in futures]
expected = [i * 2 for i in range(15)]
self.assertEqual(results, expected,
"ProcessPoolExecutor with max_tasks_per_child=3 failed to "
"complete all 15 tasks across 5 worker restart cycles "
"(gh-115634 deadlock regression)")
finally:
executor.shutdown(wait=True, cancel_futures=True)

def test_max_tasks_per_child_multiple_workers_gh115634(self):
# gh-115634: Test with multiple workers to ensure fix works
Expand All @@ -246,13 +253,16 @@ def test_max_tasks_per_child_multiple_workers_gh115634(self):
# 2 workers, max_tasks=2 each, 12 tasks total
# Should require 3 restart cycles per worker
executor = self.executor_type(2, mp_context=context, max_tasks_per_child=2)
futures = [executor.submit(mul, i, 3) for i in range(12)]

results = [future.result(timeout=support.SHORT_TIMEOUT) for future in futures]
expected = [i * 3 for i in range(12)]
self.assertEqual(results, expected)

executor.shutdown()
try:
futures = [executor.submit(mul, i, 3) for i in range(12)]

results = [future.result(timeout=support.SHORT_TIMEOUT) for future in futures]
expected = [i * 3 for i in range(12)]
self.assertEqual(results, expected,
"ProcessPoolExecutor with 2 workers and max_tasks_per_child=2 "
"failed to complete all 12 tasks (gh-115634 deadlock regression)")
finally:
executor.shutdown(wait=True, cancel_futures=True)

def test_max_tasks_early_shutdown(self):
context = self.get_context()
Expand Down
Loading