Skip to content

Avoid callable iterator deadlock during sentinel comparison (#8011)#8040

Open
doma17 wants to merge 2 commits into
RustPython:mainfrom
doma17:fix-callable-iterator-deadlock
Open

Avoid callable iterator deadlock during sentinel comparison (#8011)#8040
doma17 wants to merge 2 commits into
RustPython:mainfrom
doma17:fix-callable-iterator-deadlock

Conversation

@doma17
Copy link
Copy Markdown

@doma17 doma17 commented Jun 4, 2026 •
edited by coderabbitai Bot
Loading

Summary

Fixes a deadlock in iter(callable, sentinel) when sentinel equality re-enters
the same iterator. The comparison now runs outside the iterator status lock, and
a regression test covers the re-entrant case.

Problem

iter(callable, sentinel) could deadlock when sentinel comparison ran Python
code. The outer next() held the callable iterator status lock while comparing
the callable result with the sentinel. If __eq__ called next() on the same
iterator, the inner call waited on that same lock forever.

Fixes

  1. Release the callable iterator status lock before running sentinel equality.
  2. Re-check the iterator state before comparison, since the callable can already
    re-enter and exhaust the iterator.
  3. Re-acquire the status lock only when the value matches the sentinel and the
    iterator needs to be marked exhausted.
  4. Add a regression test for re-entrant sentinel equality.

Testing

Summary by CodeRabbit

  • Improvements

    • Enhanced iterator reliability and robustness in edge-case scenarios.

  • Tests

    • Added regression tests for iterator sentinel equality behavior and JSON recursion handling.

doma17 and others added 2 commits June 2, 2026 22:04
@doma17
Avoid callable iterator deadlock during sentinel comparison
afcfc5d 
Constraint: Python-level sentinel equality can re-enter the same callable iterator while internal iterator state is protected by a non-reentrant lock.

Rejected: Holding the callable iterator status lock across sentinel equality | user-defined __eq__ can call next() on the same iterator and deadlock.

Confidence: high

Scope-risk: moderate

Tested: pre-commit run --all-files; cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher; cargo build --release --features sqlite; pytest -v in extra_tests; cargo run --quiet -- extra_tests/snippets/protocol_iternext.py; cargo check -p rustpython-vm; cargo clippy -p rustpython-vm --all-targets -- -D warnings; git diff --check

Not-tested: push/PR creation

Assisted-by: Codex:gpt-5.5
@doma17
Merge branch 'RustPython:main' into fix-callable-iterator-deadlock
ca61fe1
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Jun 4, 2026 •
edited
Loading

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: 38c28468-b871-4e3c-9c4a-6ae34d0ea18e

📥 Commits

Reviewing files that changed from the base of the PR and between 51c97b9 and ca61fe1.

📒 Files selected for processing (3)
  • crates/vm/src/builtins/iter.rs
  • extra_tests/snippets/protocol_iternext.py
  • extra_tests/snippets/stdlib_json.py
📝 Walkthrough

Walkthrough

The PR fixes a deadlock in iter(callable, sentinel) when sentinel equality comparison executes re-entrant Python code. The iterator lock is released before comparing the sentinel using vm.identical_or_equal, state is re-checked to ensure safe transitions, and a regression test validates the fix. A JSON recursion test is also updated to guard against native-stack overflow.

Changes

Iterator Deadlock Fix and Test Updates

Layer / File(s) Summary
Lock-free sentinel equality in PyCallableIterator::next
crates/vm/src/builtins/iter.rs		 Refactors PyCallableIterator::next to release the iterator lock before performing sentinel equality via vm.identical_or_equal. Iterator state is re-checked before and after the comparison without holding the lock, and state is verified again under a fresh upgradable-read lock before marking the iterator exhausted or returning a yielded value.
Regression test for re-entrant sentinel equality
extra_tests/snippets/protocol_iternext.py		 Added test for iter(callable, sentinel) where a custom __eq__ object calls next() on the same iterator during equality comparison. Verifies that both inner and outer StopIteration exceptions are raised correctly and that the equality logic path was executed.
JSON recursion test robustness
extra_tests/snippets/stdlib_json.py		 Modified deep-nesting JSON recursion tests to guard against native-stack overflow by introducing a helper that suppresses RecursionError. Updated array, object, and alternating deep-nesting test cases to use this helper instead of asserting that json.loads raises RecursionError.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

  • #6589: Similar issue involving iter(callable, sentinel) deadlock; the lock management refactoring in PyCallableIterator::next directly addresses the root cause of lock-contention during sentinel equality by releasing the lock before executing user __eq__ code.

Suggested reviewers

  • ShaharNaveh
  • youknowone

Poem

🐰 A lock held too long caused a deadly embrace,
But our iterator found a safer place,
Released before __eq__ runs its code,
Re-entrant paths now take the lighter load,
Tests catch the creep—no more deadlock's dread! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: fixing a deadlock in callable iterator sentinel comparison by releasing the lock before equality check.
Linked Issues check ✅ Passed The PR implementation addresses the core requirement from #8011: eliminating the deadlock by releasing the iterator lock during sentinel equality, allowing re-entrant calls to proceed safely while preserving correct iterator exhaustion semantics.
Out of Scope Changes check ✅ Passed All changes are directly related to the PR objectives: the main fix in iter.rs addresses the deadlock, the new protocol_iternext.py test validates the fix, and the stdlib_json.py test update addresses a secondary test stability concern without introducing unrelated scope.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Deadlock in iter(callable, sentinel) when __eq__ re-enters the same iterator

1 participant

@doma17