Fix int unicode decimal digits - #8521
Conversation
CPython runs a string argument through
_PyUnicode_TransformDecimalAndSpaceToASCII before parsing it, so decimal
digits from any script are accepted:
int('١٢٣') # 123
int('0x١f', 16) # 31
Decimal('١٢٣') # Decimal('123')
complex('１+２j') # (1+2j)
RustPython only did this for float(), which had the transform inlined.
int() handed the raw UTF-8 bytes to bytes_to_int(), whose digit check is
is_ascii_alphanumeric(), so every non-ASCII digit was rejected — even
though float() accepted the same string.
Lift the inlined transform out of float_from_string() into
common::str::transform_decimal_and_space_to_ascii() and apply it to the
str paths of int() and complex() too. The result is always ASCII: as in
CPython, a character that is neither ASCII, whitespace nor a decimal
digit becomes '?' and truncates the string, which no parser accepts at
any base, leaving the caller to raise the error from the original string.
Bytes-like input keeps going straight to the parser, matching CPython's
split between PyLong_FromUnicodeObject and PyLong_FromString.
This unmarks two expectedFailure tests: test_int.test_unicode and
test_decimal.test_unicode_digits.
All three constructors need the same thing from a str argument: trim it, fold Unicode decimal digits and whitespace to ASCII, and give up on a string holding surrogates. Each expressed that last part differently — float matched PyKindStr and returned b"", complex leaned on to_str() returning None, int returned an empty Cow — so the rule lived in three places at once. Move it into protocol::numeric_literal_from_str() and have all three call it. CPython repeats this per type because its wrapper is three lines over a single PyUnicode representation; ours has to match over Ascii/Utf8/Wtf8, which is worth writing once. Only the shared step moves. int keeps its base handling, int and float keep accepting bytes-like input, complex keeps rejecting it, and each keeps raising its own error, because none of that is shared. No behavior change: the CPython differential suite is byte-identical before and after.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yml Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (6)
📝 WalkthroughWalkthroughThe change adds shared normalization for Unicode decimal digits and whitespace. ChangesNumeric string normalization
Estimated code review effort: 3 (Moderate) | ~20 minutes Merge Risk: 🔵 Low · up to The PR broadens numeric parsing to Unicode decimal digits and whitespace across int(), float(), complex(), and Decimal(); it is mergeable with explicit follow-up to run the required Rust checks and verify constructor-level compatibility, since localized parser integration or build-quality regressions could otherwise go unnoticed. Sequence Diagram(s)sequenceDiagram
participant NumericBuiltin
participant numeric_literal_from_str
participant transform_decimal_and_space_to_ascii
participant LiteralParser
NumericBuiltin->>numeric_literal_from_str: normalize string input
numeric_literal_from_str->>transform_decimal_and_space_to_ascii: convert digits and whitespace
transform_decimal_and_space_to_ascii-->>numeric_literal_from_str: normalized text
numeric_literal_from_str-->>NumericBuiltin: trimmed numeric literal
NumericBuiltin->>LiteralParser: parse normalized literal
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
📦 Library DependenciesThe following Lib/ modules were modified. Here are their dependencies: [x] lib: cpython/Lib/code.py dependencies:
dependent tests: (2 tests) [x] test: cpython/Lib/test/test_structseq.py (TODO: 7) dependencies: dependent tests: (no tests depend on structseq) [x] lib: cpython/Lib/decimal.py dependencies:
dependent tests: (75 tests)
[ ] test: cpython/Lib/test/test_set.py (TODO: 4) dependencies: dependent tests: (no tests depend on set) [x] lib: cpython/Lib/ssl.py dependencies:
dependent tests: (53 tests)
[ ] lib: cpython/Lib/typing.py dependencies:
dependent tests: (19 tests)
[x] lib: cpython/Lib/_pylong.py dependencies:
dependent tests: (no tests depend on int) [x] test: cpython/Lib/test/test_itertools.py (TODO: 6) dependencies: dependent tests: (56 tests)
[x] test: cpython/Lib/test/test_marshal.py (TODO: 15) dependencies: dependent tests: (25 tests)
Legend:
|
Summary
CPython runs the string argument of every numeric constructor through
_PyUnicode_TransformDecimalAndSpaceToASCIIbefore parsing, so digits from anyscript — and any Unicode whitespace — are accepted:
Changes
rustpython_common::str::transform_decimal_and_space_to_ascii, a port of CPython's transform. Unicode decimal digits fold to ASCII, Unicode whitespace folds to a plain space, and ASCII input is returned borrowed without allocating. Any other non-ASCII character can never appear in a numeric literal, so it is replaced with?and the rest of the string is dropped —?is rejected by every parser at every base, which leaves the error message to the caller that knows the base and owns the original string.protocol::numeric_literal_from_str, the shared trim + transform step, and routeint()(bothtry_int_radixandtry_int),float()andcomplex()through it. This is the only step the three constructors share — onlyinttakes a base, and onlyintandfloataccept bytes-like input — so each keeps its own entry point around it. Strings holding surrogates fold to an empty (and therefore invalid) literal, as before.float()'s inline mapping is replaced by the shared helper; its previous version left non-digit non-ASCII characters in place, which the new one rejects up front.Decimal()inherits the fix through itsint()call in_pydecimal.Summary by CodeRabbit