Skip to content

Commit 1a959cf

Browse files
youknowoneyouknowone
authored
Align codegen passes and opcode metadata with CPython (RustPython#7987)
* Share marshal ref table between code object and its internals read_marshal_bytes, _str, _str_vec, _name_tuple, and _const_tuple now take a shared ref table and resolve TYPE_REF / register FLAG_REF entries. deserialize_code is split into a public wrapper and an inner function that receives the ref table; deserialize_value_depth opens a fresh inner ref space when it hits Type::Code, mirroring CPython's behaviour of putting the code object itself at ref slot 0. Nested code objects inside const tuples reuse the surrounding code's ref space via the new read_const_value helper. * Align PYC magic number, FORMAT_VERSION, and header check with CPython 3.14 PYC_MAGIC_NUMBER changes from 2994 to 3627, matching CPython 3.14's pyc_magic_number_token (0x0a0d0e2b). marshal FORMAT_VERSION drops from 5 to 4 (the encoder/marshal.version value; the decoder already accepts both). check_pyc_magic_number_bytes now compares all four magic bytes instead of the first two. * Accept CPython-tagged .pyc as read-only bytecode source SourceFileLoader.get_code now also looks for .pyc files using _RP_FALLBACK_CACHE_TAGS (currently ('cpython-314',)) in addition to sys.implementation.cache_tag. The matched .pyc is only used for reading; recompilation still writes to the RustPython-tagged path, so CPython's .pyc is never overwritten. Source-stat / hash / timestamp validation logic is unchanged. * Apply rustfmt to marshal helpers * Marshal PySlice from format version 4 instead of 5 CPython's marshal supports TYPE_SLICE from format version 4 onwards and that is the default version. Rejecting slice dumps below version 5 made marshal.dumps(slice(...)) fail with the default version and broke test.test_marshal.SliceTestCase.test_slice. * Revert "Accept CPython-tagged .pyc as read-only bytecode source" Lib/importlib/_bootstrap_external.py is CPython's own code copied verbatim; local patches here defeat compatibility tracking. The cpython-XX cache_tag fallback needs to live on the RustPython side (Rust code or sys.implementation.cache_tag policy), not as edits to the imported standard library. This reverts commit 1fc426d0fb5fcdb50d35cad13bbb43e8f6ce1c7f. * Set marshal FORMAT_VERSION to 5 to match CPython 3.14.5 Py_MARSHAL_VERSION is 5 in CPython 3.14.5 (Include/marshal.h:16) and TYPE_SLICE serialization rejects version < 5 (Python/marshal.c:720). Restore the same threshold and constant so marshal.version and the slice-marshal gate match CPython. * Thread marshal recursion depth through nested code objects Code objects embedded in const-tuples reset the depth budget on each recursion, so a hostile or pathological marshal stream of code-in-tuple- in-code can blow the stack despite MAX_MARSHAL_STACK_DEPTH. Pass the current depth through deserialize_code_inner and read_marshal_const_tuple and decrement at each code-object/tuple boundary. Also route dict keys through deserialize_value_after_header so TYPE_CODE keys decode instead of failing with BadType. * align compiler to CPython * Align codegen with CPython compile.c Rename CFG helpers and accessors to the names used in CPython's compile.c (basicblock_next_instr, basicblock_last_instr, basicblock_append_instructions, bb_has_fallthrough, is_jump, make_cfg_traversal_stack, mark_warm/mark_cold, etc.). Drop the unused boolop-folding gate, mark_cpython_cfg_label_block helper, and ComprehensionLoopControl::iter_range field. Track an is_coroutine flag on SymbolTable, set in async def, await, and async comprehensions, and propagate it through non-generator comprehensions per symtable_handle_comprehension(). Mark SetupCleanup/SetupFinally/SetupWith as has_arg pseudo-ops, mark ForIter as a terminator, and add has_arg/has_const on AnyInstruction. Fix Instruction::stack_effect_jump to delegate to the opcode's stack_effect_jump rather than stack_effect. * Align codegen IR with CPython CFG structures * Match CPython CFG annotation offset arithmetic * Propagate CPython CFG label translation errors * Align CPython exception target labeling flow * Propagate CPython CFG traversal stack allocation errors * Match CPython optimize_load_fast allocation flow * Propagate CPython basicblock allocation errors * Propagate CPython redundant NOP cleanup errors * Propagate CPython unused const cleanup errors * Propagate CPython const folding errors * Propagate CPython swaptimize allocation errors * Match CPython list-to-tuple fold allocation * Skip const folding on CPython allocation failures * Skip subscript folding on CPython allocation failures * Propagate CPython assembler allocation errors * Propagate CPython localsplus allocation errors * Propagate CPython localsplus setup allocation errors * Propagate CPython jump label map allocation errors * Propagate CPython instruction sequence allocation errors * Propagate CPython instruction label allocation errors * Guard CPython codegen block allocation * Guard CPython label shadow allocation * Propagate CPython label shadow allocation errors * Align CPython c-array allocation updates * Align CPython ref stack growth * Match CPython CFG builder debug check * Avoid Rust-only CFG append clone * Reuse CPython cleared block slots * Use CPython block append in copy_basicblock * Match CPython cfg builder creation order * Clear label map after CPython apply pass * Match CPython cfg builder allocation check * Propagate CPython c-array size errors * Drop Rust-only label uniqueness check * Drop Rust-only label shadow debug checks * Propagate CFG block index overflow * Propagate CFG label oparg overflow * Model CPython basicblock instruction storage * Drop Rust-only recorded CFG precheck * Model CPython instruction sequence storage * Propagate instruction sequence offset overflow * Model CPython instruction sequence labels * Match CPython jump offset arithmetic * Match CPython exception table arithmetic * Match CPython label index arithmetic * Match CPython instruction offset casts * Match CPython jump offset indexing * Match CPython oparg locals casts * Match CPython localsplus offset arithmetic * Match CPython cell prefix indexing * Match CPython C array growth arithmetic * Match CPython label map allocation arithmetic * Match CPython label map size tracking * Use CPython label map size in sequence passes * Assert CPython label map clearing invariants * Match CPython label oparg assignment * Match CPython compiler direct arithmetic * Match CPython load fast local casts * Match CPython load fast depth assert * Match CPython resume depth flagging * Match CPython stack depth arithmetic * Drop Rust-only stack overflow error * Return CPython stack depth directly * Match CPython C array growth errors * Match CPython instruction insert asserts * Match CPython unreachable pseudo jump * Match CPython CFG size guard * Match CPython superinstruction assert * Match CPython redundant jump assert * Match CPython stackdepth errors * Match CPython jump offset flow * Match CPython assembler buffer defaults * Match CPython bytecode emit growth * Match CPython assembler entry growth * Match CPython assembler growth overflow check * Match CPython remove_unreachable structure * Match CPython static swap flow * Inline CPython code unit preprocessing * Match CPython C array growth checks * Match CPython label map size guard * Match CPython load-fast flow * Simplify CPython CFG condition flow * Align exception fallthrough propagation * Match CPython pseudo target table * Match CPython annotations CFG assert * Match CPython inverted op assert * Match CPython many-locals guard * Reject deopt opcodes in CFG stack effects * Match CPython invalid stack effect error * Test CPython deopt stack effect guard * Match CPython load-fast extended-arg assert * Match CPython instruction allocation asserts * Match CPython basicblock last-instr asserts * Match CPython opcode range asserts * Assert CPython fallthrough line propagation invariant * Assert CPython CFG target offset sign * Assert CPython exception fallthrough invariant * Assert CPython exception stack bounds * Assert CPython traversal stack allocation * Match CPython label-map allocation in shadow * Mirror CPython label-map sentinel fill * Match CPython CFG builder allocation asserts * Match CPython exception stack structure * Match CPython ref stack structure * Match CPython CFG traversal stack structure * Mirror CPython CFG traversal stack pointer * Use CPython fixed exception handler stack * Mirror CPython ref stack capacity field * Match CPython swap optimizer scratch stack * Align static swap helpers with CPython blocks * Align swaptimize signature with CPython * Match CPython redundant pair pass result * Match CPython inline pass result * Match CPython redundant NOP pass results * Fix bytecode metadata after upstream rebase * Match CPython opcode stack metadata * Add CPython identifiers to cspell dictionary Add CNOTAB, LNOTAB, ialloc, ioffset, iused, nblocks, ncellsused, ncellvars, nextop, noffsets, nvars, swaptimize, untargeted to .cspell.dict/cpython.txt for the new CFG/assembler code in crates/codegen/src/ir.rs. * Fix CI failures from bytecode-parity work - clippy: drop redundant `test_` prefix on three test functions and remove an unnecessary `u32` cast in basicblock_clear_reuses_cpython_spare_slots_in_offset_order - insta: regenerate nested_double_async_with snapshot to match the new CFG output that drops unreferenced labels after the redundant-NOP pass - regrtest: drop `@expectedFailure` markers from test_func_args, test_meth_args (test_compile), test_disassemble_with, test_disassemble_try_finally (test_dis), and test_except_star (test_monitoring) which now pass * Resync generated opcode metadata Empty conf.toml since WithExceptStart and Setup{Cleanup,Finally,With} stack effects already match CPython, so the TODO override entries are stale and only cause CI hook diffs. Regenerate opcode_metadata.rs and drop the matching SetupCleanup/ SetupFinally/SetupWith assertions on PseudoOpcode::has_arg(); their `HAS_ARG` flag comes from pseudo definitions in bytecodes.c that the upstream analyzer does not propagate through PseudoInstruction.properties, so the generated has_arg() excludes them. has_target() still covers these block-push pseudos via is_block_push(). * Drop is_block_push has_arg invariant The CPython invariant `assert(OPCODE_HAS_ARG(op) || !IS_BLOCK_PUSH(op))` relies on SETUP_{FINALLY,CLEANUP,WITH} carrying `HAS_ARG_FLAG` in CPython's metadata. The autogen tool reads pseudo-opcode properties from target instructions and does not propagate the pseudo's own HAS_ARG flag, so PseudoOpcode::has_arg() omits these three opcodes. Drop the debug_assert that fired inside py_freeze proc-macro expansion. * Auto-generate has_eval_break and route AnyInstruction has_arg/has_const via macro Add fn_has_eval_break to generate_rs_opcode_metadata.py using CPython's Properties.eval_breaker, removing the hand-written matches! body for Opcode::has_eval_break and PseudoOpcode::has_eval_break. Forward has_arg/has_const from Instruction and PseudoInstruction to their opcode, so AnyInstruction can use either_real_pseudo! like the other has_* accessors instead of an open-coded match.
1 parent ca412fc commit 1a959cf

16 files changed

Lines changed: 8392 additions & 15672 deletions

File tree

‎.cspell.dict/cpython.txt‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ CFWS
3737
CLASSDEREF
3838
classdict
3939
cmpop
40+
CNOTAB
4041
codedepth
4142
CODEUNIT
4243
CONIN
@@ -98,19 +99,22 @@ HASUNION
9899
heaptype
99100
hexdigit
100101
HIGHRES
102+
ialloc
101103
IFUNC
102104
IMMUTABLETYPE
103105
INCREF
104106
inlinedepth
105107
inplace
106108
inpos
109+
ioffset
107110
isbytecode
108111
ishidden
109112
ismine
110113
ISPOINTER
111114
isoctal
112115
iteminfo
113116
Itertool
117+
iused
114118
keeped
115119
kwnames
116120
kwonlyarg
@@ -122,6 +126,7 @@ linearise
122126
lineful
123127
lineiterator
124128
linetable
129+
LNOTAB
125130
loadfast
126131
localsplus
127132
localspluskinds
@@ -137,24 +142,30 @@ multibytecodec
137142
nameobj
138143
nameop
139144
nargsf
145+
nblocks
140146
ncells
147+
ncellsused
148+
ncellvars
141149
nconsts
142150
newargs
143151
newfree
144152
NEWLOCALS
145153
newsemlockobject
154+
nextop
146155
nfrees
147156
nkwargs
148157
nkwelts
149158
nlocalsplus
150159
nointerrupt
160+
noffsets
151161
Nondescriptor
152162
noninteger
153163
nops
154164
noraise
155165
nseen
156166
NSIGNALS
157167
numer
168+
nvars
158169
opname
159170
opnames
160171
orelse
@@ -220,6 +231,7 @@ subparams
220231
subscr
221232
sval
222233
swappedbytes
234+
swaptimize
223235
sysdict
224236
tbstderr
225237
templatelib
@@ -240,6 +252,7 @@ uncollectable
240252
Unhandle
241253
unparse
242254
unparser
255+
untargeted
243256
untracking
244257
VARKEYWORDS
245258
varkwarg

‎Lib/test/test_compile.py‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2580,15 +2580,13 @@ def test_set(self):
25802580
def test_dict(self):
25812581
self.check_stack_size("{" + "x:x, " * self.N + "x:x}")
25822582

2583-
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 102 not less than or equal to 6
25842583
def test_func_args(self):
25852584
self.check_stack_size("f(" + "x, " * self.N + ")")
25862585

25872586
def test_func_kwargs(self):
25882587
kwargs = (f'a{i}=x' for i in range(self.N))
25892588
self.check_stack_size("f(" + ", ".join(kwargs) + ")")
25902589

2591-
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 102 not less than or equal to 6
25922590
def test_meth_args(self):
25932591
self.check_stack_size("o.m(" + "x, " * self.N + ")")
25942592

‎Lib/test/test_dis.py‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1211,14 +1211,12 @@ def test_disassemble_coroutine(self):
12111211
def test_disassemble_fstring(self):
12121212
self.do_disassembly_test(_fstring, dis_fstring)
12131213

1214-
@unittest.expectedFailure # TODO: RUSTPYTHON
12151214
def test_disassemble_with(self):
12161215
self.do_disassembly_test(_with, dis_with)
12171216

12181217
def test_disassemble_asyncwith(self):
12191218
self.do_disassembly_test(_asyncwith, dis_asyncwith)
12201219

1221-
@unittest.expectedFailure # TODO: RUSTPYTHON
12221220
def test_disassemble_try_finally(self):
12231221
self.do_disassembly_test(_tryfinally, dis_tryfinally)
12241222
self.do_disassembly_test(_tryfinallyconst, dis_tryfinallyconst)

‎Lib/test/test_monitoring.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1624,7 +1624,6 @@ def whilefunc(n=0):
16241624
('branch left', 'func', 44, 50),
16251625
('branch right', 'func', 28, 70)])
16261626

1627-
@unittest.expectedFailure # TODO: RUSTPYTHON; - bytecode layout differs from CPython
16281627
def test_except_star(self):
16291628

16301629
class Foo:

0 commit comments

Comments
 (0)