Align compiler and VM bytecode contracts with CPython 3.15 by youknowone · Pull Request #9105 · RustPython/RustPython

youknowone · 2026-10-14T08:30:35Z

This PR follows our AI policy.

Replaces #9088, rebased onto main after #8951 and #9094 were merged. The PR author's commits are kept as they were; the commits from #9094 and the Windows manager CI diagnostic commits (which cancel each other out) are dropped.

Changes on top of #9088

Store code-object fields and marshal them the way CPython 3.15 does. Emit the generator prefix in enter_scope and expose _co_firsttraceable. Line up monitoring DISABLE branches, check_matched, % format errors, and IMPORT_NAME flags with CPython (_PyCompile_InExceptionHandler).
Emit RETURN_GENERATOR; POP_TOP for modules compiled with PyCF_ALLOW_TOP_LEVEL_AWAIT, matching codegen_enter_scope. This fixes the 3.15 test_builtin.test_compile_top_level_await that came in with Implement PEP 814 frozendict #8951.
Replace hand-written bytecode assertions with insta dis snapshots.
Bring Lib/test files to v3.15.0 using update_lib. The only edits are TODO: RUSTPYTHON markers.
Remove extra_tests/snippets/generator_iterator_prefix.py, which mostly duplicated test_generators, test_coroutines, test_asyncgen, test_yield_from and test_monitoring. The checks those modules don't cover (tracing and monitoring of the genexpr eager-iterator prefix, and the caller frame seen from it) moved into stdlib_sys.py and stdlib_sys_getframe.py.

Validation (local, macOS)

cargo fmt, clippy (workspace and capi), cargo test (workspace and capi) all pass.
Snippets: 494 passed. The 2 failures (stdlib_mmap under host CPython, and stdlib_sqlite because _sqlite3 is missing) also fail without this PR.
Full regrtest (--dont-add-python-opts): only test_pyrepl fails, and it fails the same way without this PR. test_builtin passes after the top-level await fix.

🤖 Generated with Claude Code

Summary by CodeRabbit

New Features
- Expanded Python 3.15 bytecode support, including generator expressions, iteration, constants, and monitoring.
- Added the read-only gi_state property to report a generator’s current state.
- Improved code-object serialization and preservation of metadata.
- Added source-position information to disassembly output.
Bug Fixes
- Improved formatting errors, warning-filter matching, frame inspection, and coroutine lifecycle handling.
- Corrected opcode stack effects and validation of stack-effect arguments.

Reconstruct the compiler-only baseline from validated487ae47 on the current main and frozen mapping prerequisitec78d811. Preserve the full source delta while dropping obsolete prerequisite ancestry; detailed old-to-new provenance is retained in the recovery artifact. Assisted-by: OpenAI Codex: model identifier unavailable

…racts Replay reviewed compiler CI repairs 2191a9d, 1f1fe04, a2dacaf on the frozen current prerequisite. Intermediate compile/diagnostic corrections are combined where they address the same failure. Assisted-by: OpenAI Codex: model identifier unavailable

Replay reviewed compiler CI repairs 476bb74 on the frozen current prerequisite. Intermediate compile/diagnostic corrections are combined where they address the same failure. Assisted-by: OpenAI Codex: model identifier unavailable

Replay reviewed compiler CI repairs 6482adf on the frozen current prerequisite. Intermediate compile/diagnostic corrections are combined where they address the same failure. Assisted-by: OpenAI Codex: model identifier unavailable

Replay reviewed compiler CI repairs 246c309 on the frozen current prerequisite. Intermediate compile/diagnostic corrections are combined where they address the same failure. Assisted-by: OpenAI Codex: model identifier unavailable

Replay reviewed compiler CI repairs 7ecedcf on the frozen current prerequisite. Intermediate compile/diagnostic corrections are combined where they address the same failure. Assisted-by: OpenAI Codex: model identifier unavailable

…l 3.15 Replay reviewed compiler CI repairs ec206b1, 191fe58 on the frozen current prerequisite. Intermediate compile/diagnostic corrections are combined where they address the same failure. Assisted-by: OpenAI Codex: model identifier unavailable

- Store co_consts, co_names, co_localsplusnames, co_localspluskinds, co_linetable, co_exceptiontable and co_name as objects on PyCode; build co_code and the varnames/cellvars/freevars tuples lazily in _co_cached, with co_code deoptimized through deopt_code. - Move the compiler constant cache (_PyCode_ConstantKey, const_cache_insert) to builtins/code/const_cache.rs. - Write every code field with w_object in marshal and pass the read field objects to MarshalBag::make_code through CodeFields. - Emit RETURN_GENERATOR/POP_TOP in enter_scope and place SETUP_CLEANUP after them; drop the generator part of insert_prefix_instructions. - Add PyCode::first_traceable (_co_firsttraceable) for _PyFrame_IsIncomplete and sys.monitoring. - Follow the CPython DISABLE branches in sys.monitoring, the check_matched signature in warnings, the 3.15 % format error messages for str and bytes, code_returns_only_none, and the FOR_ITER stack analysis. - Compute IMPORT_NAME flags with in_exception_handler. - Replace the linetable/exceptiontable byte tests in codegen with dis(show_positions=True) insta snapshots in _opcode. - Remove the opcode metadata verification tooling, byte-exact scripts, private cache tests and redundant snippets. - Restore test_code.py and test_monitoring.py to v3.15.0 plus RUSTPYTHON markers. Assisted-by: Claude Code:claude-opus-5-5

Drop extra_tests/snippets/generator_iterator_prefix.py. Its loop, delegation, async, f_back and never-awaited checks are covered by test_generators, test_coroutines, test_asyncgen, test_yield_from and test_monitoring. Move the remaining checks into existing files: genexpr prefix tracing and monitoring into stdlib_sys.py, and the caller frame seen from the eager iterator into stdlib_sys_getframe.py. Assisted-by: Claude Code:claude-opus-5-5

Module code compiled with PyCF_ALLOW_TOP_LEVEL_AWAIT and marked ste_coroutine now starts with RETURN_GENERATOR; POP_TOP. The prefix and RESUME emission move into emit_scope_prefix, shared by enter_scope and the module entry points, as in codegen_enter_scope. Fixes test_builtin.test_compile_top_level_await. Assisted-by: Claude Code:claude-opus-5-5

chatgpt-codex-connector · 2026-10-14T08:30:54Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-14T08:37:54.029456Z	`9ad52b3`	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

github-actions · 2026-10-14T08:31:33Z

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
[x] lib: cpython/Lib/opcode.py
[ ] lib: cpython/Lib/_opcode_metadata.py
[x] test: cpython/Lib/test/test__opcode.py
[x] test: cpython/Lib/test/test_opcodes.py

dependencies:

opcode (native: _opcode, builtins)
- _opcode_metadata
- _opcode_metadata

dependent tests: (50 tests)

opcode: test__opcode test_code test_compile test_dis test_peepholer
- dis: test_ast test_compiler_assemble test_dtrace test_fstring test_inspect test_monitoring test_opcache test_patma test_positional_only_arg test_type_cache
  - bdb: test_bdb test_pdb
  - inspect: test_abc test_argparse test_asyncgen test_buffer test_builtin test_clinic test_collections test_coroutines test_decimal test_enum test_functools test_generators test_grammar test_ntpath test_operator test_posixpath test_pydoc test_signal test_sqlite3 test_traceback test_turtle test_type_annotations test_type_params test_types test_typing test_unittest test_yield_from test_zipimport test_zipimport_support test_zoneinfo
  - modulefinder: test_importlib test_modulefinder
  - trace: test_trace

[x] test: cpython/Lib/test/test_generators.py
[ ] test: cpython/Lib/test/test_genexps.py (TODO: 2)
[x] test: cpython/Lib/test/test_generator_stop.py
[x] test: cpython/Lib/test/test_yield_from.py

dependencies:

dependent tests: (no tests depend on generator)

[ ] lib: cpython/Lib/code.py
[x] test: cpython/Lib/test/test_code_module.py

dependencies:

code (native: builtins, readline, sys)
- argparse (native: sys)
- codeop
- traceback (native: _missing_stdlib_info, _suggestions, collections.abc, importlib.machinery, importlib.resources.readers, importlib.util, itertools, sys, unicodedata)

dependent tests: (2 tests)
- [ ] pdb: test_pdb
- [ ] sqlite3.main: test_sqlite3

[x] test: cpython/Lib/test/test_compile.py
[x] test: cpython/Lib/test/test_compiler_assemble.py
[x] test: cpython/Lib/test/test_compiler_codegen.py
[x] test: cpython/Lib/test/test_peepholer.py

dependencies:

dependent tests: (no tests depend on compile)

[x] lib: cpython/Lib/warnings.py
[x] lib: cpython/Lib/_py_warnings.py
[ ] test: cpython/Lib/test/test_warnings (TODO: 1)

dependencies:

warnings

dependent tests: (55 tests)

warnings: test_argparse test_array test_asyncio test_buffer test_builtin test_codecs test_codeop test_compile test_coroutines test_ctypes test_decimal test_descr test_fnmatch test_fstring test_funcattrs test_genericpath test_glob test_global test_grammar test_gzip test_hashlib test_hmac test_importlib test_inspect test_io test_logging test_ntpath test_os test_pickle test_posix test_pty test_pyclbr test_pyrepl test_random test_re test_runpy test_set test_socket test_sqlite3 test_str test_string_literals test_support test_sys test_sys_settrace test_tarfile test_tempfile test_thread test_threading test_type_cache test_typing test_unicode_file_functions test_unittest test_unparse test_xml_etree test_zipimport

[x] test: cpython/Lib/test/test_monitoring.py (TODO: 4)

dependencies:

dependent tests: (no tests depend on monitoring)

[x] lib: cpython/Lib/dis.py
[x] test: cpython/Lib/test/test_dis.py (TODO: 1)

dependencies:

dis

dependent tests: (80 tests)

[ ] test: cpython/Lib/test/test_format.py

dependencies:

dependent tests: (no tests depend on format)

Legend:

[+] path exists in CPython
[x] up-to-date, [ ] outdated

coderabbitai · 2026-10-14T08:31:55Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: RustPython/RustPython/.coderabbit.yml
Review profile: CHILL
Plan: Advanced
Run ID: cc7a19a9-9598-4d02-af76-93e6b18f2be1

📥 Commits

Reviewing files that changed from the base of the PR and between 9ad52b3 and 7986696.

⛔ Files ignored due to path filters: 1

Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing: 2

Cargo.toml
crates/stdlib/src/_opcode.rs

🚧 Files skipped from review as they are similar to previous changes: 1

crates/stdlib/src/_opcode.rs

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

📝 Walkthrough

This pull request updates RustPython’s bytecode, compiler, and runtime paths for CPython 3.15. It revises code-object and marshal handling, generator startup, frame visibility, and monitoring. It also changes percent-format error reporting and warning-filter matching.

Changes: CPython 3.15 Bytecode and Runtime

Layer / File(s)	Summary
Bytecode definitions and metadata `.github/workflows/ci.yaml`, `crates/compiler-core/src/bytecode/*`, `crates/compiler-core/src/bytecode.rs`, `crates/vm/src/version.rs`, `tools/opcode_metadata/generate_py_opcode_metadata.py`, `crates/stdlib/src/_opcode.rs`	Opcode IDs, variants, arguments, cache entries, deoptimization mappings, and stack effects are updated. The bytecode magic number changes to 3666, and the CI lint checkout uses CPython 3.15.0. The disassembly tests add source-position snapshots.
Compiler constants and stack analysis `crates/codegen/src/ir.rs`, `crates/jit/src/instructions.rs`, `crates/jit/tests/common.rs`	The compiler recognizes common constants and folds constant list and set constructions. Instruction analysis and JIT handling are updated for common constants and revised stack effects.
Code-object fields and marshal data `crates/compiler-core/src/marshal.rs`, `crates/vm/src/builtins/code.rs`, `crates/vm/src/builtins/code/const_cache.rs`, `crates/vm/src/stdlib/marshal.rs`, `crates/vm/src/vm/vm_new.rs`	Code objects retain Python-facing fields and lazily generate deoptimized bytecode. Compilation merges constants, and marshal handling passes retained code fields through serialization and deserialization.
Generator startup and iterator stack handling `crates/codegen/src/ir.rs`, `crates/compiler-core/src/bytecode.rs`, `crates/compiler-core/src/bytecode/oparg.rs`, `crates/vm/src/builtins/frame.rs`, `crates/vm/src/builtins/function.rs`, `crates/vm/src/builtins/generator.rs`, `crates/vm/src/builtins/coroutine.rs`, `crates/vm/src/coroutine.rs`, `crates/vm/src/frame.rs`	Generator expressions receive a startup prefix. Frame execution and iterator operations handle the revised stack layout. Coroutine startup and frame clearing change, and `gi_state` is added.
Frame visibility and monitoring `crates/vm/src/frame.rs`, `crates/vm/src/protocol/callable.rs`, `crates/vm/src/vm/mod.rs`, `crates/vm/src/vm/thread.rs`, `crates/vm/src/vm/crossinterp.rs`, `crates/vm/src/stdlib/sys/monitoring.rs`, `extra_tests/snippets/stdlib_sys.py`, `extra_tests/snippets/stdlib_sys_getframe.py`	Frame traversal and trace dispatch skip incomplete generator-creation frames. Monitoring behavior changes, and tests cover generator-expression tracing, monitoring, and frame links.

Formatting and Warning Filters

Layer / File(s)	Summary
Percent-format parsing and conversion errors `crates/common/src/cformat.rs`, `crates/vm/src/cformat.rs`	Stray percent errors point to the percent character. Formatting reports argument-specific type errors and distinguishes alphabetic unsupported conversions from other unsupported characters.
Optional-module warning matching `crates/vm/src/warn.rs`	Warning filters receive the optional module and original filename separately. Filename-to-module normalization is removed.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant CodeGenerator
  participant InterpreterFrame
  participant Iterable
  participant TraceMonitor
  CodeGenerator->>InterpreterFrame: Add generator-expression prefix
  InterpreterFrame->>Iterable: Acquire outer iterator during preparation
  InterpreterFrame->>TraceMonitor: Dispatch events when eager iteration raises

Suggested reviewers: joshuamegnauth54

Merge Risk: ⚪ Minimal · up to 79866

No actionable regression is established in the reviewed changes; mergeability risk is minimal.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 59.24% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 238 functions across 30 files. (1 skipped…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly summarizes the main changes: aligning compiler and VM bytecode behavior with CPython 3.15, including opcode, code-object, marshal, generator, and monitoring updates.

Full details: Docstring Coverage

Explanation

Docstring coverage is 59.24% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 238 functions across 30 files. (1 skipped: 1 unsupported.)

Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

c_long is i32 on Windows, where clippy reports the cast to i32 as unnecessary. Assisted-by: Claude Code:claude-opus-5-5

pymath requires malachite-bigint ^0, so the example projects, which have no lock file, resolve it to 0.14.0 while the workspace pinned 0.13.0, and fail to build. Assisted-by: Claude Code:claude-opus-5-5

youknowdot and others added 10 commits October 14, 2026 16:45

Assert exact final constant-list iterable folding

c659c27

Replay reviewed compiler CI repairs 7ecedcf on the frozen current prerequisite. Intermediate compile/diagnostic corrections are combined where they address the same failure. Assisted-by: OpenAI Codex: model identifier unavailable

youknowone added 2 commits October 14, 2026 19:28

Allow the c_long cast in _opcode.stack_effect

32a7e66

c_long is i32 on Windows, where clippy reports the cast to i32 as unnecessary. Assisted-by: Claude Code:claude-opus-5-5

Update malachite crates to 0.14.0

7986696

pymath requires malachite-bigint ^0, so the example projects, which have no lock file, resolve it to 0.14.0 while the workspace pinned 0.13.0, and fail to build. Assisted-by: Claude Code:claude-opus-5-5

youknowdot mentioned this pull request Oct 14, 2026

Implement Python 3.15 type alias and variadic metadata #9113

Merged

youknowone merged commit 2a09ee0 into RustPython:main Oct 14, 2026
30 checks passed

youknowone deleted the compiler-315-current-integration branch October 14, 2026 13:38

youknowdot mentioned this pull request Oct 14, 2026

Implement bytearray.take_bytes with stable export snapshots #9119

Merged

1 task done

codspeed Bot mentioned this pull request Oct 14, 2026

build(deps): bump libc from 0.2.189 to 0.2.190 #9010

Open

Navigation Menu

Repository navigation

Align compiler and VM bytecode contracts with CPython 3.15 - #9105

Align compiler and VM bytecode contracts with CPython 3.15#9105
youknowone merged 12 commits into
RustPython:mainfrom
youknowone:compiler-315-current-integration

youknowone commented Oct 14, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 14, 2026 •

edited

Loading

Sequence Diagram(s)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

youknowone commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes on top of #9088

Validation (local, macOS)

Summary by CodeRabbit

Uh oh!

chatgpt-codex-connector Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 Library Dependencies

Uh oh!

coderabbitai Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sequence Diagram(s)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

youknowone commented Oct 14, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading

coderabbitai Bot commented Oct 14, 2026 •

edited

Loading