Skip to content

Commit a1ba6a6

Browse files
youknowoneyouknowone
authored
Align compiler and AST behavior with CPython (#8138)
* Align compiler and AST behavior with CPython Update the compiler pipeline, AST module support, and related VM compile/eval paths toward CPython 3.14 behavior. Use upstream Ruff parser crates directly and keep RustPython-specific syntax preflight handling in the compile path. Refresh opcode metadata, snapshots, and targeted tests for the aligned bytecode and AST behavior. * fix * Fix PR 8138 CI failures * Fix match mapping wildcard scan complexity * fixes * Use Ruff runtime AST fields * Minimize residual patch diff * refactor error handling * Remove _ast conversion context wrappers * Clean up _ast conversion glue and compiler warning escalation Rename the Node conversion parameter `ctx` to `vm` throughout the _ast module. Remove the unused compile_program, compile_program_single, compile_block_expression, and compile_expression forwarders; compile_top stays as the entry point. Stash an escalated compiler SyntaxWarning so a non-SyntaxWarning category propagates unchanged instead of being rewritten to SyntaxError, matching PyErr_ExceptionMatches(SyntaxWarning) in compiler_warn. Drop the now-unused CompileWarningError::into_codegen_error. Replace the PositionalArguments two-variant enum with a struct holding a shared range and a PositionalArgumentsKind enum. Assisted-by: Claude * Represent compiler flags as a CompilerFlags bitflags Replace the loose PY_CF_*/CO_* i32 constants in compile_mode with a CompilerFlags bitflags covering the PyCF_* flags and __future__ CO_FUTURE_* bits. The Python-visible ast.PyCF_* attribute values are derived from it via bits(); parser input modes stay plain consts. builtins::compile and the compile_string flag handling test flags through CompilerFlags::contains instead of raw bit math. Assisted-by: Claude * Fix PR 8138 test regressions * Fix PR 8138 CI failures * Avoid shell interpolation in Linux deps action * Fix review findings in compiler/AST changes - frame.rs: compute the MATCH_CLASS class name lazily so a successful match no longer takes the type name lock or allocates; the __match_args__-not-a-tuple branch reuses it instead of recomputing. - _ast/constant.rs: add the missing leading space to the with_recursion label ("maximum recursion depth exceeded during compilation"). - sys.set_int_max_str_digits: restore the "maxdigits must be 0 or larger than {n}" error text. - codegen: set CO_MAXBLOCKS to 20 in both ir.rs and compile.rs. - _ast/exception.rs: extract the duplicated excepthandler node guard into ensure_excepthandler_node. - codegen: correct the validate_keywords doc comment. - Drop redundant "CPython" qualifiers from comments added in this branch. Assisted-by: Claude
1 parent 473ba73 commit a1ba6a6

105 files changed

Lines changed: 23195 additions & 6976 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.cspell.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,8 +66,11 @@
6666
"deoptimize",
6767
"emscripten",
6868
"excs",
69+
"flufl",
6970
"fnfe",
71+
"fsdefault",
7072
"ifexp",
73+
"implicits",
7174
"interps",
7275
"jitted",
7376
"jitting",

‎.github/actions/install-linux-deps/action.yml‎

Lines changed: 32 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -39,11 +39,36 @@ runs:
3939
- name: Install Linux dependencies
4040
shell: bash
4141
if: ${{ runner.os == 'Linux' }}
42-
run: >
43-
sudo apt-get update
42+
env:
43+
GCC_MULTILIB: ${{ inputs.gcc-multilib }}
44+
MUSL_TOOLS: ${{ inputs.musl-tools }}
45+
CLANG: ${{ inputs.clang }}
46+
GCC_AARCH64_LINUX_GNU: ${{ inputs.gcc-aarch64-linux-gnu }}
47+
run: |
48+
if ! sudo apt-get update; then
49+
echo "::warning::apt-get update failed; disabling nonessential Microsoft apt sources and retrying"
50+
for source in /etc/apt/sources.list.d/*microsoft* /etc/apt/sources.list.d/*azure-cli*; do
51+
if [ -e "$source" ]; then
52+
sudo mv "$source" "$source.disabled"
53+
fi
54+
done
55+
sudo apt-get update
56+
fi
4457
45-
sudo apt-get install --no-install-recommends
46-
${{ fromJSON(inputs.gcc-multilib) && 'gcc-multilib' || '' }}
47-
${{ fromJSON(inputs.musl-tools) && 'musl-tools' || '' }}
48-
${{ fromJSON(inputs.clang) && 'clang' || '' }}
49-
${{ fromJSON(inputs.gcc-aarch64-linux-gnu) && 'gcc-aarch64-linux-gnu linux-libc-dev-arm64-cross libc6-dev-arm64-cross' || '' }}
58+
packages=()
59+
if [[ "$GCC_MULTILIB" == "true" ]]; then
60+
packages+=(gcc-multilib)
61+
fi
62+
if [[ "$MUSL_TOOLS" == "true" ]]; then
63+
packages+=(musl-tools)
64+
fi
65+
if [[ "$CLANG" == "true" ]]; then
66+
packages+=(clang)
67+
fi
68+
if [[ "$GCC_AARCH64_LINUX_GNU" == "true" ]]; then
69+
packages+=(gcc-aarch64-linux-gnu linux-libc-dev-arm64-cross libc6-dev-arm64-cross)
70+
fi
71+
72+
if ((${#packages[@]})); then
73+
sudo apt-get install --no-install-recommends "${packages[@]}"
74+
fi

‎Cargo.lock‎

Lines changed: 18 additions & 15 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 6 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -183,19 +183,11 @@ rustpython-sre_engine = { path = "crates/sre_engine", version = "0.5.0" }
183183
rustpython-wtf8 = { path = "crates/wtf8", version = "0.5.0" }
184184
rustpython-doc = { path = "crates/doc", version = "0.5.0" }
185185

186-
# Use RustPython-packaged Ruff crates from the published fork while keeping
187-
# existing crate names in the codebase.
188-
ruff_python_parser = { package = "rustpython-ruff_python_parser", version = "0.15.8" }
189-
ruff_python_ast = { package = "rustpython-ruff_python_ast", version = "0.15.8" }
190-
ruff_text_size = { package = "rustpython-ruff_text_size", version = "0.15.8" }
191-
ruff_source_file = { package = "rustpython-ruff_source_file", version = "0.15.8" }
192-
# To update ruff crates, comment out the above lines and uncomment the following lines to pull directly from the Ruff repository at the specified commit hash.
193-
# Ruff tag 0.15.8 is based on commit c2a8815842f9dc5d24ec19385eae0f1a7188b0d9
194-
# at the time of this capture. We use the commit hash to ensure reproducible builds.
195-
# ruff_python_parser = { git = "https://github.com/astral-sh/ruff.git", rev = "c2a8815842f9dc5d24ec19385eae0f1a7188b0d9" }
196-
# ruff_python_ast = { git = "https://github.com/astral-sh/ruff.git", rev = "c2a8815842f9dc5d24ec19385eae0f1a7188b0d9" }
197-
# ruff_text_size = { git = "https://github.com/astral-sh/ruff.git", rev = "c2a8815842f9dc5d24ec19385eae0f1a7188b0d9" }
198-
# ruff_source_file = { git = "https://github.com/astral-sh/ruff.git", rev = "c2a8815842f9dc5d24ec19385eae0f1a7188b0d9" }
186+
# Use the RustPython Ruff fork for RustPython public `_ast` metadata.
187+
ruff_python_parser = { package = "rustpython-ruff_python_parser", git = "https://github.com/RustPython/ruff.git", tag = "0.15.19-rustpython" }
188+
ruff_python_ast = { package = "rustpython-ruff_python_ast", git = "https://github.com/RustPython/ruff.git", tag = "0.15.19-rustpython" }
189+
ruff_text_size = { package = "rustpython-ruff_text_size", git = "https://github.com/RustPython/ruff.git", tag = "0.15.19-rustpython" }
190+
ruff_source_file = { package = "rustpython-ruff_source_file", git = "https://github.com/RustPython/ruff.git", tag = "0.15.19-rustpython" }
199191

200192
der = { version = "0.8", features = ["alloc", "oid", "pem", "zeroize"] }
201193
phf = { version = "0.13.1", default-features = false, features = ["macros"]}
@@ -311,6 +303,7 @@ tcl-sys = { git = "https://github.com/arihant2math/tkinter.git", tag = "v0.2.0"
311303
textwrap = { version = "0.16.2", default-features = false }
312304
termios = "0.3.3"
313305
thiserror = "2.0"
306+
thin-vec = "0.2.14"
314307
timsort = "0.1.2"
315308
tk-sys = { git = "https://github.com/arihant2math/tkinter.git", tag = "v0.2.0" }
316309
icu_casemap = "2"

‎Lib/test/test_ast/test_ast.py‎

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -150,7 +150,6 @@ def test_parse_invalid_ast(self):
150150
self.assertRaises(TypeError, ast.parse, ast.Constant(42),
151151
optimize=optval)
152152

153-
@unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags
154153
def test_optimization_levels__debug__(self):
155154
cases = [(-1, '__debug__'), (0, '__debug__'), (1, False), (2, False)]
156155
for (optval, expected) in cases:
@@ -586,7 +585,6 @@ def test_invalid_sum(self):
586585
compile(m, "<test>", "exec")
587586
self.assertIn("but got expr()", str(cm.exception))
588587

589-
@unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: expected str for name
590588
def test_invalid_identifier(self):
591589
m = ast.Module([ast.Expr(ast.Name(42, ast.Load()))], [])
592590
ast.fix_missing_locations(m)
@@ -1365,7 +1363,6 @@ def test_replace_ignore_known_custom_instance_fields(self):
13651363
self.assertIs(repl.ctx, context)
13661364
self.assertRaises(AttributeError, getattr, repl, 'extra')
13671365

1368-
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: "Name\.__replace__\ missing\ 1\ keyword\ argument:\ 'id'\." does not match "replace() does not support Name objects"
13691366
def test_replace_reject_missing_field(self):
13701367
# case: warn if deleted field is not replaced
13711368
node = ast.parse('x').body[0].value
@@ -1700,7 +1697,6 @@ def check_text(code, empty, full, **kwargs):
17001697
full="Module(body=[Import(names=[alias(name='_ast', asname='ast')]), ImportFrom(module='module', names=[alias(name='sub')], level=0)], type_ignores=[])",
17011698
)
17021699

1703-
@unittest.expectedFailure # TODO: RUSTPYTHON; ? ^^^^^^^^^ ^^^^^^^^^
17041700
def test_copy_location(self):
17051701
src = ast.parse('1 + 1', mode='eval')
17061702
src.body.right = ast.copy_location(ast.Constant(2), src.body.right)
@@ -1737,7 +1733,6 @@ def test_fix_missing_locations(self):
17371733
"end_col_offset=0), lineno=1, col_offset=0, end_lineno=1, end_col_offset=0)])"
17381734
)
17391735

1740-
@unittest.expectedFailure # TODO: RUSTPYTHON; ? ^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
17411736
def test_increment_lineno(self):
17421737
src = ast.parse('1 + 1', mode='eval')
17431738
self.assertEqual(ast.increment_lineno(src, n=3), src)
@@ -1959,7 +1954,6 @@ def test_literal_eval_syntax_errors(self):
19591954
(\
19601955
\ ''')
19611956

1962-
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: required field "lineno" missing from alias
19631957
def test_bad_integer(self):
19641958
# issue13436: Bad error message with invalid numeric values
19651959
body = [ast.ImportFrom(module='time',
@@ -3259,7 +3253,6 @@ class MyAttrs(ast.AST):
32593253
r"MyAttrs.__init__ got an unexpected keyword argument 'c'."):
32603254
obj = MyAttrs(c=3)
32613255

3262-
@unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered
32633256
def test_fields_and_types_no_default(self):
32643257
class FieldsAndTypesNoDefault(ast.AST):
32653258
_fields = ('a',)
@@ -3273,7 +3266,6 @@ class FieldsAndTypesNoDefault(ast.AST):
32733266
obj = FieldsAndTypesNoDefault(a=1)
32743267
self.assertEqual(obj.a, 1)
32753268

3276-
@unittest.expectedFailure # TODO: RUSTPYTHON; DeprecationWarning not triggered
32773269
def test_incomplete_field_types(self):
32783270
class MoreFieldsThanTypes(ast.AST):
32793271
_fields = ('a', 'b')
@@ -3293,7 +3285,6 @@ class MoreFieldsThanTypes(ast.AST):
32933285
self.assertEqual(obj.a, 1)
32943286
self.assertEqual(obj.b, 2)
32953287

3296-
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: Expected type 'str' but 'bytes' found.
32973288
def test_malformed_fields_with_bytes(self):
32983289
class BadFields(ast.AST):
32993290
_fields = (b'\xff'*64,)
@@ -3713,7 +3704,6 @@ def assert_ast(self, code, non_optimized_target, optimized_target):
37133704
f"{ast.dump(optimized_tree)}",
37143705
)
37153706

3716-
@unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: compile() unrecognized flags
37173707
def test_folding_format(self):
37183708
code = "'%s' % (a,)"
37193709

‎Lib/test/test_audit.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,6 @@ def test_monkeypatch(self):
7777
def test_open(self):
7878
self.do_test("test_open", os_helper.TESTFN)
7979

80-
@unittest.expectedFailure # TODO: RUSTPYTHON
8180
def test_cantrace(self):
8281
self.do_test("test_cantrace")
8382

‎Lib/test/test_builtin.py‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -486,7 +486,6 @@ def test_compile_top_level_await_no_coro(self):
486486
msg=f"source={source} mode={mode}")
487487

488488

489-
@unittest.expectedFailure # TODO: RUSTPYTHON
490489
def test_compile_top_level_await(self):
491490
"""Test whether code with top level await can be compiled.
492491
@@ -627,7 +626,6 @@ def test_compile_async_generator(self):
627626
exec(co, glob)
628627
self.assertEqual(type(glob['ticker']()), AsyncGeneratorType)
629628

630-
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: <_ast.Name object at 0xb40000731e3d1360> is not an instance of <class '_ast.Constant'>
631629
def test_compile_ast(self):
632630
args = ("a*__debug__", "f.py", "exec")
633631
raw = compile(*args, flags = ast.PyCF_ONLY_AST).body[0]
@@ -1020,7 +1018,6 @@ def test_exec_redirected(self):
10201018
finally:
10211019
sys.stdout = savestdout
10221020

1023-
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: Unexpected keyword argument closure
10241021
def test_exec_closure(self):
10251022
def function_without_closures():
10261023
return 3 * 5

‎Lib/test/test_cmd_line_script.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -645,7 +645,6 @@ def test_syntaxerror_indented_caret_position(self):
645645
self.assertNotIn("\f", text)
646646
self.assertIn("\n 1 + 1 = 2\n ^^^^^\n", text)
647647

648-
@unittest.expectedFailure # TODO: RUSTPYTHON
649648
def test_syntaxerror_multi_line_fstring(self):
650649
script = 'foo = f"""{}\nfoo"""\n'
651650
with os_helper.temp_dir() as script_dir:

‎Lib/test/test_codeop.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -279,7 +279,6 @@ def test_filename(self):
279279
self.assertNotEqual(compile_command("a = 1\n", "abc").co_filename,
280280
compile("a = 1\n", "def", 'single').co_filename)
281281

282-
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 0 != 2
283282
def test_warning(self):
284283
# Test that the warning is only returned once.
285284
with warnings_helper.check_warnings(

0 commit comments

Comments
 (0)