Fix SymbolUsge::Iter, os.rename, sre_engine by youknowone · Pull Request #8390 · RustPython/RustPython

youknowone · 2026-07-27T12:17:41Z

Closes #xxxx
This PR follows our AI policy

Summary

Summary by CodeRabbit

Bug Fixes
- Improved regular-expression possessive repetition so final captures are preserved reliably.
- Updated \B behavior for empty input to align with Python 3.14.
- Fixed case-insensitive ASCII matching so non-ASCII range literals remain intact.
- Corrected token classification for unknown tokens when extra token information is enabled.
- Improved local-variable handling for comprehension iterator targets.
Platform Support
- Added consistent file-renaming behavior across supported platforms, with clear errors where descriptor-based renaming is unavailable.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

coderabbitai · 2026-07-27T12:17:50Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: 9130e44f-df8d-4104-8db8-483ecbde13a2

📥 Commits

Reviewing files that changed from the base of the PR and between 59e903d and 7cb980c.

⛔ Files ignored due to path filters (5)

Lib/test/test_inspect/test_inspect.py is excluded by !Lib/**
Lib/test/test_pdb.py is excluded by !Lib/**
Lib/test/test_pydoc/test_pydoc.py is excluded by !Lib/**
Lib/test/test_re.py is excluded by !Lib/**
Lib/test/test_unittest/testmock/testhelpers.py is excluded by !Lib/**

📒 Files selected for processing (5)

crates/codegen/src/symboltable.rs
crates/host_env/src/os.rs
crates/sre_engine/src/engine.rs
crates/sre_engine/tests/tests.rs
crates/stdlib/src/_tokenize.rs

📝 Walkthrough

Walkthrough

The changes update comprehension iterator symbol flags, add platform-specific rename handling, adjust regex possessive-repeat and boundary semantics with tests, and classify unknown tokenizer tokens as operators when extra tokens are enabled.

Changes

Comprehension symbol classification

Layer / File(s)	Summary
Iterator local-definition flags `crates/codegen/src/symboltable.rs`	Comprehension iterator targets now receive `DEF_LOCAL` alongside `ITER` and `DEF_COMP_ITER`.

Platform-specific rename handling

Layer / File(s)	Summary
Rename platform wiring and fallback `crates/host_env/src/os.rs`	`rename` is re-exported for Windows, Unix, and WASI, while other platforms use `std::fs::rename` without file descriptors and return `Unsupported` for descriptor-based calls.

Regex matching state and boundary behavior

Layer / File(s)	Summary
Marks checkpointing and possessive-repeat restoration `crates/sre_engine/src/engine.rs`, `crates/sre_engine/tests/tests.rs`	Possessive-repeat handling checkpoints and restores marks-stack depth, synchronizes the cursor, initializes the checkpoint field, and tests retained captures.
Boundary and literal matching behavior `crates/sre_engine/src/engine.rs`, `crates/sre_engine/tests/tests.rs`	Empty-input `\B` handling returns true, with updated boundary coverage and tests for non-ASCII range literals under ASCII-ignore matching.

Unknown token classification

Layer / File(s)	Summary
Unknown token operator classification `crates/stdlib/src/_tokenize.rs`	Unknown tokens map to `TOKEN_OP` when `extra_tokens` is enabled.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

RustPython/RustPython#8118: Also changes the host environment rename API to accept optional directory file descriptors.
RustPython/RustPython#8212: Overlaps with platform-specific rename wiring and non-Unix behavior.

Suggested reviewers: shaharnaveh

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is specific and matches the main changes in symbol iteration, os.rename, and sre_engine behavior.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

posix_windows.rs is mounted as the `posix` module on Windows, so `crate::posix_windows` does not resolve. Assisted-by: Claude

Assisted-by: Claude

- test_re: drop expectedFailure from test_word_boundaries, test_possessive_quantifiers and test_bug_gh101955 - test_inspect, test_pydoc: drop expectedFailure from the signature tests that now pass - test_pdb: drop expectedFailure from the two file-modification tests and +EXPECTED_FAILURE from the test_post_mortem_chained and test_pdb_asynctask doctests - test_unittest: mark test_autospec_on_bound_builtin_function as expectedFailure; inspect.signature() now succeeds on time.ctime because of its auto-generated __text_signature__ Assisted-by: Claude

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

github-actions · 2026-07-27T17:25:33Z

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

[x] lib: cpython/Lib/inspect.py
[ ] test: cpython/Lib/test/test_inspect (TODO: 30)

dependencies:

inspect

dependent tests: (96 tests)

[x] lib: cpython/Lib/re
[x] lib: cpython/Lib/sre_compile.py
[x] lib: cpython/Lib/sre_constants.py
[x] lib: cpython/Lib/sre_parse.py
[ ] test: cpython/Lib/test/test_re.py (TODO: 8)
[x] test: cpython/Lib/test/re_tests.py

dependencies:

re

dependent tests: (81 tests)

re: test_android test_ast test_asyncio test_binascii test_builtin test_bytes test_clinic test_cmd_line test_ctypes test_dict test_dis test_docxmlrpc test_dtrace test_email test_embed test_faulthandler test_filecmp test_fileinput test_fnmatch test_format test_frame test_fstring test_functools test_future_stmt test_generated_cases test_genericalias test_glob test_hashlib test_http_cookiejar test_httplib test_httpservers test_imaplib test_importlib test_inspect test_ipaddress test_launcher test_logging test_mailbox test_mmap test_optparse test_ordered_dict test_platform test_pprint test_pydoc test_pyexpat test_pyrepl test_re test_regrtest test_runpy test_set test_site test_smtplib test_socket test_ssl test_strftime test_strptime test_strtod test_structseq test_symtable test_syntax test_sys test_sysconfig test_tarfile test_tempfile test_tokenize test_tools test_traceback test_turtle test_typing test_unittest test_unparse test_venv test_webbrowser test_winapi test_with test_wsgiref test_xmlrpc test_zipfile test_zipimport test_zoneinfo test_zstd

[x] lib: cpython/Lib/pydoc.py
[x] lib: cpython/Lib/pydoc_data
[ ] test: cpython/Lib/test/test_pydoc (TODO: 32)

dependencies:

pydoc

dependent tests: (5 tests)

pydoc: test_enum test_pydoc
- pdb: test_pdb
- xmlrpc.server: test_docxmlrpc test_xmlrpc

[x] lib: cpython/Lib/pdb.py
[ ] test: cpython/Lib/test/test_pdb.py (TODO: 43)

dependencies:

pdb

dependent tests: (1 tests)

pdb: test_pdb

[ ] lib: cpython/Lib/unittest
[x] test: cpython/Lib/test/test_unittest (TODO: 12)

dependencies:

unittest (native: _io, _log, async_case, builtins, case, loader, main, os.path, result, runner, signals, suite, sys, time, unittest.util, util)
- warnings (native: _contextvars, _thread, _warnings, builtins, sys)
- _colorize, annotationlib, argparse, asyncio, collections, contextlib, contextvars, dataclasses, difflib, fnmatch, functools, inspect, io, logging, os, pkgutil, pprint, re, signal, threading, traceback, types, weakref

dependent tests: (408 tests)

unittest: regrtestdata test___all__ test__colorize test__locale test__opcode test__osx_support test_abc test_abstract_numbers test_android test_annotationlib test_apple test_argparse test_array test_asdl_parser test_ast test_asyncgen test_asyncio test_atexit test_audit test_augassign test_base64 test_baseexception test_bdb test_bigaddrspace test_bigmem test_binascii test_binop test_bisect test_bool test_buffer test_bufio test_builtin test_bytes test_bz2 test_c_locale_coercion test_calendar test_call test_charmapcodec test_class test_clinic test_cmath test_cmd test_cmd_line test_cmd_line_script test_code test_code_module test_codeccallbacks test_codecencodings_cn test_codecencodings_hk test_codecencodings_iso2022 test_codecencodings_jp test_codecencodings_kr test_codecencodings_tw test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_codecs test_codeop test_collections test_colorsys test_compare test_compile test_compileall test_complex test_concurrent_futures test_configparser test_contains test_context test_contextlib test_contextlib_async test_copy test_copyreg test_coroutines test_cprofile test_csv test_ctypes test_datetime test_dbm test_dbm_dumb test_dbm_sqlite3 test_decimal test_decorators test_defaultdict test_deque test_descr test_descrtut test_devpoll test_dict test_dictcomps test_dictviews test_difflib test_dis test_doctest test_doctest2 test_docxmlrpc test_dtrace test_dummy_thread test_dummy_threading test_dynamic test_dynamicclassattribute test_eintr test_email test_embed test_ensurepip test_enum test_enumerate test_eof test_epoll test_errno test_except_star test_exception_group test_exception_hierarchy test_exception_variations test_exceptions test_extcall test_external_inspection test_faulthandler test_fcntl test_file test_file_eintr test_filecmp test_fileinput test_fileio test_fileutils test_finalization test_float test_flufl test_fnmatch test_fork1 test_format test_fractions test_frame test_frozen test_fstring test_ftplib test_funcattrs test_functools test_future_stmt test_gc test_generated_cases test_generator_stop test_generators test_genericalias test_genericclass test_genericpath test_genexps test_getopt test_getpass test_gettext test_glob test_global test_grammar test_graphlib test_grp test_gzip test_hash test_hashlib test_heapq test_hmac test_html test_htmlparser test_http_cookiejar test_http_cookies test_httplib test_httpservers test_idle test_imaplib test_importlib test_index test_inspect test_int test_int_literal test_io test_ioctl test_ipaddress test_isinstance test_iter test_iterlen test_itertools test_json test_keyword test_keywordonlyarg test_kqueue test_largefile test_launcher test_linecache test_list test_listcomps test_locale test_logging test_long test_longexp test_lzma test_mailbox test_marshal test_math test_math_property test_memoryio test_memoryview test_metaclass test_mimetypes test_minidom test_mmap test_modulefinder test_monitoring test_msvcrt test_multibytecodec test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_main_handling test_multiprocessing_spawn test_named_expressions test_netrc test_ntpath test_nturl2path test_numeric_tower test_opcache test_opcodes test_openpty test_operator test_optimizer test_optparse test_ordered_dict test_os test_osx_env test_pathlib test_patma test_pdb test_peepholer test_peg_generator test_pep646_syntax test_perfmaps test_pickle test_picklebuffer test_pickletools test_pkg test_pkgutil test_platform test_plistlib test_poll test_popen test_poplib test_positional_only_arg test_posix test_posixpath test_pow test_pprint test_print test_profile test_property test_pstats test_pty test_pulldom test_pwd test_py_compile test_pyclbr test_pydoc test_pyexpat test_pyrepl test_queue test_quopri test_raise test_random test_range test_re test_regrtest test_repl test_reprlib test_resource test_richcmp test_rlcompleter test_robotparser test_runpy test_sax test_sched test_scope test_script_helper test_secrets test_select test_selectors test_set test_setcomps test_shelve test_shlex test_shutil test_signal test_site test_slice test_smtplib test_smtpnet test_socket test_socketserver test_sort test_sqlite3 test_ssl test_stable_abi_ctypes test_stat test_statistics test_str test_strftime test_string test_string_literals test_stringprep test_strptime test_strtod test_struct test_structseq test_subclassinit test_subprocess test_sundry test_super test_support test_symtable test_syntax test_sys test_sys_setprofile test_sys_settrace test_sysconfig test_syslog test_tabnanny test_tarfile test_tempfile test_termios test_textwrap test_thread test_thread_local_bytecode test_threadedtempfile test_threading test_threading_local test_threadsignals test_time test_timeit test_timeout test_tokenize test_tomllib test_tools test_trace test_traceback test_tracemalloc test_tstring test_tty test_tuple test_turtle test_type_aliases test_type_annotations test_type_cache test_type_comments test_type_params test_typechecks test_types test_typing test_ucn test_unary test_unicode_file test_unicode_file_functions test_unicode_identifiers test_unicodedata test_unittest test_univnewlines test_unpack test_unpack_ex test_unparse test_urllib test_urllib2 test_urllib2_localnet test_urllib2net test_urllib_response test_urllibnet test_urlparse test_userdict test_userlist test_userstring test_utf8_mode test_utf8source test_uuid test_venv test_wait3 test_wait4 test_wave test_weakref test_weakset test_webbrowser test_winapi test_winconsoleio test_winreg test_winsound test_with test_wmi test_wsgiref test_xml_dom_minicompat test_xml_dom_xmlbuilder test_xml_etree test_xml_etree_c test_xmlrpc test_xpickle test_xxlimited test_xxtestfuzz test_yield_from test_zipapp test_zipfile test_zipfile64 test_zipimport test_zipimport_support test_zlib test_zoneinfo test_zstd

Legend:

[+] path exists in CPython
[x] up-to-date, [ ] outdated

* Fix SymbolUsge::Iter * fix os.rename * fix sre * Re-export rename from crate::posix on Windows posix_windows.rs is mounted as the `posix` module on Windows, so `crate::posix_windows` does not resolve. Assisted-by: Claude * Apply rustfmt to sre_engine tests Assisted-by: Claude * Update test expectations for the sre and tokenize fixes - test_re: drop expectedFailure from test_word_boundaries, test_possessive_quantifiers and test_bug_gh101955 - test_inspect, test_pydoc: drop expectedFailure from the signature tests that now pass - test_pdb: drop expectedFailure from the two file-modification tests and +EXPECTED_FAILURE from the test_post_mortem_chained and test_pdb_asynctask doctests - test_unittest: mark test_autospec_on_bound_builtin_function as expectedFailure; inspect.signature() now succeeds on time.ctime because of its auto-generated __text_signature__ Assisted-by: Claude

Copilot AI review requested due to automatic review settings July 27, 2026 12:17

Copilot AI reviewed Jul 27, 2026

youknowone added 6 commits July 27, 2026 23:17

Fix SymbolUsge::Iter

f6c1e56

fix os.rename

d11ad22

fix sre

8127f98

Re-export rename from crate::posix on Windows

b725346

posix_windows.rs is mounted as the `posix` module on Windows, so `crate::posix_windows` does not resolve. Assisted-by: Claude

Apply rustfmt to sre_engine tests

5b89de9

Assisted-by: Claude

youknowone force-pushed the fix branch from 9b50bd2 to 7cb980c Compare July 27, 2026 14:41

Copilot AI review requested due to automatic review settings July 27, 2026 14:41

Copilot AI reviewed Jul 27, 2026

youknowone marked this pull request as ready for review July 28, 2026 01:18

youknowone merged commit 1613741 into RustPython:main Jul 28, 2026
25 of 27 checks passed

youknowone deleted the fix branch July 28, 2026 01:20

leehanjeong mentioned this pull request Aug 2, 2026

Malformed builtin __text_signature__ makes inspect.signature(len/sorted/open) raise ValueError #8383

Open

coderabbitai Bot mentioned this pull request Aug 15, 2026

Remove private comprehension flags #8537

Open

1 task

Navigation Menu

Uh oh!

Fix SymbolUsge::Iter, os.rename, sre_engine - #8390

Fix SymbolUsge::Iter, os.rename, sre_engine#8390
youknowone merged 6 commits into
RustPython:mainfrom
youknowone:fix

youknowone commented Jul 27, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

coderabbitai Bot commented Jul 27, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

youknowone commented Jul 27, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Summary by CodeRabbit

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 27, 2026

📦 Library Dependencies

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

youknowone commented Jul 27, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 27, 2026 •

edited

Loading