Skip to content

marshal: round-trip code constants through the runtime bag - #8516

Merged
youknowone merged 3 commits into
RustPython:mainfrom
youknowone:fix/marshal-recursive-references
Aug 13, 2026
Merged

marshal: round-trip code constants through the runtime bag#8516
youknowone merged 3 commits into
RustPython:mainfrom
youknowone:fix/marshal-recursive-references

Conversation

@youknowone

@youknowone youknowone commented Aug 13, 2026

Copy link
Copy Markdown
Member

marshal.loads(marshal.dumps(code)) loses or rejects any co_consts entry that the compiler constant enum cannot describe, because a code object's constants are read and written through ConstantBag/BorrowedConstant rather than through the runtime object bag.

>>> import marshal
>>> c = compile("1 + 1", "", "eval").replace(co_consts=([],))
>>> marshal.loads(marshal.dumps(c)).co_consts
# CPython 3.14: ([],)
# before:       thread 'main' panicked at crates/vm/src/builtins/code.rs:254:14:
#               unexpected payload for constant python value

Reader

deserialize_code_value_inner decodes the code fields through the runtime MarshalBag instead of deserialize_code_inner, so co_consts values survive as the objects they were, and the code fields share the single reference index space CPython's r_object() uses. MarshalBag gains make_code_with_constants, code_constant_from_value, bytes_from_value, str_from_value and tuple_elements_from_value; the compiler bags keep the previous behaviour through the defaults and the blanket impl.

PyMarshalBag implements those hooks. Literal wraps any object, so a decoded constant is already its own compiler-side constant and make_code_with_constants keeps the default.

Writer

serialize_code gains a serialize_code_with variant that writes each co_consts entry through a caller-supplied writer; serialize_code passes the BorrowedConstant writer, so compiler callers are unchanged. The VM writer passes its own write_object_depth, which

  • reaches the values BorrowedConstant cannot describe, instead of panicking in borrow_obj_constant, and
  • gives a constant shared with the enclosing object an entry in the writer's reference table, which the reader now indexes against.

Tests

extra_tests/snippets/stdlib_marshal.py gains a round trip over list/dict/set constants and an identity check for a constant shared with the enclosing tuple. Both pass on CPython 3.14 and on this branch (13/13).

Lib/test/test_marshal.py passes in a release build (75 run, 18 skipped). A debug build segfaults in test_loads_recursion, which nests 2**20 tuples and exhausts the native stack before the 2000 depth limit — unrelated to this change and present on both sides.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Code objects can now preserve runtime constant values during serialization and deserialization.
    • Support added for bytes, strings, tuples, and other literal values in code constants.
    • Shared constants and references are retained across marshal round trips.
  • Bug Fixes

    • Improved handling of code objects containing non-constant values, preventing loss or incorrect conversion of runtime data.
  • Tests

    • Added coverage for round-trip preservation and shared-reference identity.

Assisted-by: OpenAI Codex:gpt-5.5
Decode code fields through the runtime MarshalBag so co_consts values that do not fit the compiler constant enum remain available to VM code wrappers while the compiler table receives shape placeholders.

Assisted-by: OpenAI Codex:GPT-5.4
serialize_code gains a serialize_code_with variant that writes each
co_consts entry through a caller-supplied writer; serialize_code keeps
the BorrowedConstant writer as its default.  The VM writer passes its own
write_object_depth, so a code constant that Literal holds but
BorrowedConstant cannot describe reaches the stream instead of panicking
in borrow_obj_constant, and a constant shared with the enclosing object
takes an entry in the writer's reference table.

PyMarshalBag implements constant_ref_from_value, bytes_from_value,
str_from_value and tuple_elements_from_value, which
deserialize_code_value_inner requires to decode code fields through the
runtime bag.

Assisted-by: Claude
@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: d7dfbc21-72bf-4742-bcf9-e38923046aed

📥 Commits

Reviewing files that changed from the base of the PR and between 2ed082a and ec7f273.

⛔ Files ignored due to path filters (1)
  • Lib/test/test_marshal.py is excluded by !Lib/**
📒 Files selected for processing (3)
  • crates/compiler-core/src/marshal.rs
  • crates/vm/src/stdlib/marshal.rs
  • extra_tests/snippets/stdlib_marshal.py

📝 Walkthrough

Walkthrough

Marshal now preserves runtime values in code constants during serialization and deserialization. Compiler and VM marshal layers share reference-table handling. Tests cover arbitrary constants and shared object identity.

Changes

Runtime Constant Marshal

Layer / File(s) Summary
Runtime code deserialization
crates/compiler-core/src/marshal.rs
MarshalBag adds runtime conversion hooks and preserves original runtime constants when constructing code objects.
Custom code constant serialization
crates/compiler-core/src/marshal.rs, crates/vm/src/stdlib/marshal.rs
serialize_code_with accepts a constant writer. The VM writes code constants through its object marshal path and shared reference table.
Marshal round-trip validation
extra_tests/snippets/stdlib_marshal.py
Tests cover arbitrary code constants and shared identity between code objects and enclosing tuples.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PythonMarshal
  participant VMMarshal
  participant CompilerMarshal
  participant ReferenceTable
  PythonMarshal->>VMMarshal: serialize code object
  VMMarshal->>CompilerMarshal: call serialize_code_with
  CompilerMarshal->>VMMarshal: write each runtime constant
  VMMarshal->>ReferenceTable: marshal constant with shared references
  ReferenceTable-->>PythonMarshal: return serialized code data
Loading
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

[x] test: cpython/Lib/test/test_itertools.py (TODO: 5)

dependencies:

dependent tests: (56 tests)

  • itertools: test_annotationlib test_ast test_asyncio test_bdb test_buffer test_builtin test_call test_codeccallbacks test_collections test_compile test_concurrent_futures test_csv test_ctypes test_descr test_dis test_email test_exceptions test_functools test_genericalias test_hashlib test_heapq test_httplib test_importlib test_inspect test_io test_iterlen test_itertools test_launcher test_logging test_math test_memoryview test_mmap test_os test_peepholer test_platform test_pprint test_pyrepl test_queue test_range test_set test_shlex test_slice test_socket test_sort test_statistics test_str test_struct test_subprocess test_tokenize test_tuple test_typing test_unittest test_uuid test_winreg test_xml_etree test_zipfile

[x] lib: cpython/Lib/code.py
[x] test: cpython/Lib/test/test_code_module.py (TODO: 3)

dependencies:

  • code

dependent tests: (2 tests)
- [x] pdb: test_pdb
- [ ] sqlite3.main: test_sqlite3

[x] test: cpython/Lib/test/test_structseq.py (TODO: 7)

dependencies:

dependent tests: (no tests depend on structseq)

[x] test: cpython/Lib/test/test_marshal.py (TODO: 8)

dependencies:

dependent tests: (25 tests)

  • marshal: test_bool test_exceptions test_importlib test_inspect test_marshal test_zipimport
    • importlib._bootstrap_external: test_importlib test_unittest
      • modulefinder: test_importlib test_modulefinder
      • py_compile: test_argparse test_cmd_line_script test_compileall test_importlib test_multiprocessing_main_handling test_py_compile test_pydoc test_runpy
      • pydoc: test_enum
    • pkgutil: test_pkgutil test_pyrepl
    • profile: test_profile
    • pstats: test_pstats
    • zipimport: test_importlib test_zipimport_support

[ ] test: cpython/Lib/test/test_set.py (TODO: 4)

dependencies:

dependent tests: (no tests depend on set)

Legend:

  • [+] path exists in CPython
  • [x] up-to-date, [ ] outdated

Comment thread Lib/test/test_marshal.py
self.assertIs(a[0], a)
# Direct self-reference which cannot be created in Python.
# This creates a reference loop which cannot be collected.
if False:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

illegal patch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant