Implement Unicode 17 grapheme iteration and properties by youknowdot · Pull Request #9145 · RustPython/RustPython

youknowdot · 2026-10-15T09:04:11Z

Summary

Implement the Unicode 17 grapheme prerequisite for #8954 as a standalone native feature with its canonical tests.

Add unicodedata.iter_graphemes(), grapheme_cluster_break(), indic_conjunct_break(), and extended_pictographic().
Port the exact CPython 3.15 final grapheme state machine, preserving Python code-point indices, __index__ conversion/overflow, lone surrogates, subclass ownership, and streaming boundary state.
Add non-exported immutable Segment and iterator types with traced references to the original string.
Use the existing pinned ICU properties data. Add a full-domain differential test against exact-final CPython Unicode 17 properties, stored as a 59,332-byte range fixture rather than a 3.34 MB raw dump.
Add an explicit opt-in grapheme oracle generation mode without rewriting existing Unicode predicate fixtures.
Synchronize test_unicodedata.py through the actual update_lib tool and remove seven stale expected-failure markers only after validation. Preserve unrelated markers.

Reference: CPython final d220409a13b77de7a6c30abe16ea90db01528edc.

This does not import the remaining traceback/theme/lazy-import consumers or expand legacy Unicode 3.2 behavior. No updater implementation, validation receipts, or new snippets are included.

Validation

Local Linux validation with Rust 1.99.0:

Release compile of rustpython-unicode's differential target, then exactly unicode17_grapheme_properties_exact_final: 1 passed. All 1,114,112 code points, including surrogates, matched all three reference properties without a version-skew allowance.
Narrow release CLI build with threading,stdlib,stdio,importlib,host_env.
Strict release Clippy for VM/stdlib, all targets, matching feature closure, with -D warnings.
Five unchanged exact-final canonical grapheme/property/Segment methods and five supplemental index, overflow, surrogate, subclass, and type-contract checks: 10 passed.
Complete official Unicode 17 GraphemeBreakTest: 766 rows, plus all 625 cluster-boundary suffix checks from the unchanged canonical helper. Data SHA-256: e2d134d2c52919bace503ebb6a551c1855fe1a1faec18478c78fff254a1793ec.
Unchanged canonical GraphemeBreakTest.test_reference_loops: passed, covering the iterator/string and Segment/string cycles.
After removing the seven markers through updater metadata, reran the six direct canonical methods and complete corpus helper from the actual migrated test file: 7 passed, zero skips/failures.

Python checks used fresh private processes, verified cache-free library inventories and native origins, 1 GiB address-space limits, 20-second CPU/30-second wall limits, and 64 KiB combined output caps. The corpus was supplied locally; no test helper fetched data during execution.

The native binary was built at df5d20c073cc817ac8dfaa447e33e4883f6096e8; final f50958f43366a9f6ec2173f29410a83503acdfc2 changes only the seven test-marker lines. Native sources and Cargo inputs are identical, and post-marker validation records that library-only delta explicitly.

These are targeted results, not a full workspace test run, general memory-safety proof, or platform-complete validation. Normal commit hooks passed.

AI assistance is disclosed in the commit trailers.

coderabbitai · 2026-10-15T09:04:19Z

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: RustPython/RustPython/.coderabbit.yml
Review profile: CHILL
Plan: Advanced
Run ID: 9113cc0b-d9e9-45a4-96a1-1735dbd2c338

📥 Commits

Reviewing files that changed from the base of the PR and between fd5b30e and 07e76cd.

⛔ Files ignored due to path filters: 1

Lib/test/test_unicodedata.py is excluded by !Lib/**

📒 Files selected for processing: 7

benches/execution.rs
crates/stdlib/src/unicodedata.rs
crates/unicode/src/grapheme.rs
crates/unicode/src/lib.rs
crates/unicode/tests/data/cpython3.15_graphemes.txt
crates/unicode/tests/differential.rs
crates/unicode/tests/generate_reference.py

+----------------------------------------------------------------------------------------------------------------------+
| You can't write perfect software. Software can't be perfect. Protect your code and users from the inevitable errors. |
+----------------------------------------------------------------------------------------------------------------------+
  \
   \     /)/)
        ( '')
        /  |
      o(___)

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-10-15T10:09:02Z

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
[ ] test: cpython/Lib/test/test_unicodedata.py (TODO: 17)
[x] test: cpython/Lib/test/test_unicode_file.py
[x] test: cpython/Lib/test/test_unicode_file_functions.py
[x] test: cpython/Lib/test/test_unicode_identifiers.py
[ ] test: cpython/Lib/test/test_ucn.py (TODO: 1)

dependencies:

dependent tests: (no tests depend on unicode)

Legend:

[+] path exists in CPython
[x] up-to-date, [ ] outdated

Add the CPython extended grapheme state machine, full-codepoint properties, and traced Segment/iterator types with positional index handling. Refresh canonical unicodedata tests through update_lib while retaining pending markers. Include a losslessly range-compressed exact-final property oracle and differential test. Source-reviewed implementation; candidate runtime validation remains pending. Reference property generation completed against the pinned CPython Unicode extension. Assisted-by: OpenAI Codex:model version unavailable

Use the required Py<Segment> receiver for native descriptors; preserve payload ownership and representation behavior. Assisted-by: OpenAI Codex:model version unavailable

Keep the inherent impl on the local payload and receive &Py<Self>, as required for stdlib classes outside the VM crate. Assisted-by: OpenAI Codex:model version unavailable

Satisfy strict Clippy for the three pure property functions without changing their behavior. Assisted-by: OpenAI Codex:model version unavailable

Remove seven stale expected-failure markers through update_lib metadata after native property parity, canonical methods, full Unicode 17 grapheme corpus, and reference-cycle GC validation. Assisted-by: OpenAI Codex:model version unavailable

Compare the complete provenance line without its line ending so the same exact Unicode 17 reference is accepted after Windows checkout conversion. Keep the complete code-point property sweep unchanged. Assisted-by: OpenAI Codex:model version unavailable

Move the existing complete Unicode 17 property regression into the existing cfg(test) module to satisfy strict Clippy. Preserve the fixture, assertions, code-point coverage, and platform-independent provenance line check unchanged. Assisted-by: OpenAI Codex:model version unavailable

Assisted-by: OpenAI Codex:model version unavailable

Use the canonical code module path introduced in PyO3 0.30 without suppressing deprecation warnings. Assisted-by: OpenAI Codex:model version unavailable

Reuse the exact eight-file RustPython#9145 head 07e76cd delta, including the shared benchmark PyCodeInput repair once. Preserve optional Unicode 3.2 support. Replay canonical final unicodedata tests through actual update_lib with the reviewed seven retired guard targets; no manual Lib patch. The complete delta is byte-identical to the reviewed source branch; its recorded native/full-domain validation is retained. This checkpoint still requires integrated RustPython behavior validation. The source branch and PR remain unchanged. Assisted-by: OpenAI Codex:model version unavailable

chatgpt-codex-connector · 2026-10-16T14:40:32Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-16T14:47:55.357212Z	`07e76cd`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 07e76cd2ee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-16T14:47:52Z

+    #[pyfunction]
+    fn grapheme_cluster_break(chr: PyStrRef, vm: &VirtualMachine) -> PyResult<&'static str> {


Expose the grapheme APIs on UCD instances

With the default ucd_3_2_0 feature enabled, these APIs are registered only as module-level #[pyfunction]s; no corresponding methods are added to Ucd. Unicode_3_2_0_FunctionsTest sets db = unicodedata.ucd_3_2_0 and inherits the four newly unmarked tests from BaseUnicodeFunctionsTest, so those tests now raise AttributeError when they access self.db.grapheme_cluster_break, the other properties, or iter_graphemes, making the standard test_unicodedata suite fail and leaving the CPython UCD API incomplete. Add the methods to Ucd or retain expected-failure markers until they work.

AGENTS.md reference: AGENTS.md:L279-L282

Useful? React with 👍 / 👎.

youknowone approved these changes Oct 15, 2026

View reviewed changes

youknowdot added 9 commits October 16, 2026 12:59

Bind grapheme Segment methods to the Python object

695e0d4

Use the required Py<Segment> receiver for native descriptors; preserve payload ownership and representation behavior. Assisted-by: OpenAI Codex:model version unavailable

Use explicit object receivers for Segment descriptors

c16f5e1

Keep the inherent impl on the local payload and receive &Py<Self>, as required for stdlib classes outside the VM crate. Assisted-by: OpenAI Codex:model version unavailable

Mark grapheme property lookups as must-use

1f0116d

Satisfy strict Clippy for the three pure property functions without changing their behavior. Assisted-by: OpenAI Codex:model version unavailable

Qualify grapheme iterator return type for generated bindings

52a10cf

Assisted-by: OpenAI Codex:model version unavailable

Fix deprecated PyO3 code input import in benchmarks

07e76cd

Use the canonical code module path introduced in PyO3 0.30 without suppressing deprecation warnings. Assisted-by: OpenAI Codex:model version unavailable

youknowdot force-pushed the split/unicode-graphemes-final branch from 5582424 to 07e76cd Compare October 16, 2026 04:03

youknowone approved these changes Oct 16, 2026

View reviewed changes

youknowone marked this pull request as ready for review October 16, 2026 14:40

youknowone merged commit 64330b8 into RustPython:main Oct 16, 2026
20 of 21 checks passed

chatgpt-codex-connector Bot reviewed Oct 16, 2026

View reviewed changes

Navigation Menu

Repository navigation

Implement Unicode 17 grapheme iteration and properties - #9145

Implement Unicode 17 grapheme iteration and properties#9145
youknowone merged 9 commits into
RustPython:mainfrom
youknowdot:split/unicode-graphemes-final

youknowdot commented Oct 15, 2026

Uh oh!

coderabbitai Bot commented Oct 15, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 15, 2026 •

edited

Loading

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 16, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Oct 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		#[pyfunction]
		fn grapheme_cluster_break(chr: PyStrRef, vm: &VirtualMachine) -> PyResult<&'static str> {

Conversation

youknowdot commented Oct 15, 2026

Summary

Validation

Uh oh!

coderabbitai Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 Library Dependencies

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

coderabbitai Bot commented Oct 15, 2026 •

edited

Loading

github-actions Bot commented Oct 15, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 16, 2026 •

edited

Loading