Add Python 3.15 encoding normalization helper by youknowdot · Pull Request #9115 · RustPython/RustPython

youknowdot · 2026-10-14T11:09:21Z

I did not use AI to write the code of this patch.
This PR follows our AI policy, using the maintainer exemption.

Summary

Extract the native _codecs._normalize_encoding helper from #8954, alongside the final Python 3.15 encodings.normalize_encoding delegation. Preserve ASCII case and dots, collapse other UTF-8 bytes to interior underscores, and validate the whole string before stopping normalization at NUL. Accept string subclasses and the encoding= keyword.

The Python wrapper retains its existing non-ASCII deprecation warning and ASCII bytes support. Main already has that warning and the cache bound; this is not presented as a fix for an email warning. Registry lookup normalization/error messages are unchanged. No IDNA, punycode, bytearray, or other encoding module changes.

CPython provenance

Only Lib/encodings/__init__.py was updated, using the repository's actual exact-file workflow:

python3 scripts/update_lib migrate /workspace/shared/cpython-315-final/Lib/encodings/__init__.py

Source: CPython d220409a13b77de7a6c30abe16ea90db01528edc. Input and generated output are byte-identical:

SHA256: 849a5e31effe6c647b6d526b92f2f063a67fac36e7750267f721f15e3693de08
Git blob: 169c48324f227bf51b88d8b2e1a6a9df39067533

The supported migrate command was chosen because copy-lib resolves the top-level package and would copy all of encodings. No manual Lib edits or new test skips were used.

Native behavior was source-reviewed against final CPython _codecsmodule.c:1025–1062 and unicodeobject.c:3465–3513.

Validation

Passed: narrow release build, cargo build --release --locked --no-default-features --features threading,stdlib,stdio,importlib,host_env --bin rustpython -j 1.
Passed: the new assertions appended to stdlib_codecs_wide.py, extracted by AST into a bounded script and run on the candidate RustPython binary with a 30-second timeout. The candidate encodings.__file__ was asserted. Covers positional/keyword/subclass binding, argument errors, case/dot preservation, punctuation collapse, Unicode, NUL, surrogate-after-NUL validation, native warning absence, and wrapper warning/bytes behavior.
Passed: strict VM Clippy, cargo clippy --release --locked -p rustpython-vm --no-default-features --features compiler,threading,host_env -j 1 -- -D warnings.
Passed: normal configured pre-commit hooks during the local commit (merge conflicts, ruff format/check, rustfmt, cspell).
Independently source-reviewed the exact patch and update_lib provenance.

Not run: the pre-existing whole codec snippet, pytest/full snippet suite, full codecs test suites, workspace/C-API tests, all-target Clippy, WASM, or a CPython 3.15 runtime oracle. These are focused automated results, not full-suite or platform validation.

AI assistance

OpenAI Codex extracted and source-reviewed the implementation, ran the library update tool, wrote bounded regression assertions, and performed the automated checks listed above. This contribution is made under the maintainer exemption. No human manual testing is claimed.

coderabbitai · 2026-10-14T11:09:32Z

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: RustPython/RustPython/.coderabbit.yml
Review profile: CHILL
Plan: Advanced
Run ID: 29aa26fe-cf23-4046-ad69-65421f0c05a8

📥 Commits

Reviewing files that changed from the base of the PR and between 016ed92 and a93f8f8.

⛔ Files ignored due to path filters: 1

Lib/encodings/__init__.py is excluded by !Lib/**

📒 Files selected for processing: 2

crates/vm/src/stdlib/_codecs.rs
extra_tests/snippets/stdlib_codecs_wide.py

 _______________________________________________________________________________________________________
< Start when you're ready. You've been building experience all your life. Don't ignore niggling doubts. >
 -------------------------------------------------------------------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-10-14T11:21:06Z

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
[ ] lib: cpython/Lib/encodings

dependencies:

encodings (native: _codecs, _codecs_cn, _codecs_hk, _codecs_iso2022, _codecs_jp, _codecs_kr, _codecs_tw, _multibytecodec, _win_cp_codecs, binascii, sys, unicodedata, zlib)
- base64 (native: binascii, sys)
- io (native: _io, _thread, errno, msvcrt, sys)
- re (native: _casefix, _constants, _sre, sys, unicodedata)
- stringprep (native: unicodedata)
- warnings (native: _contextvars, _thread, _warnings, builtins, sys)
- bz2, codecs, quopri

dependent tests: (164 tests)

Legend:

[+] path exists in CPython
[x] up-to-date, [ ] outdated

Extract _codecs._normalize_encoding and delegate encodings.normalize_encoding to it using the final CPython 3.15 library source. Preserve case and dots, validate UTF-8 before NUL truncation, and retain wrapper warnings and bytes conversion. Add bounded regression assertions without changing registry keys. Lib/encodings/__init__.py was updated via scripts/update_lib migrate from CPython d220409a13b77de7a6c30abe16ea90db01528edc, with byte-identical output. AI assistance: OpenAI Codex implemented and source-reviewed this split, ran update_lib, prepared bounded regressions and performed focused automated validation. No human manual testing is claimed. Assisted-by: OpenAI Codex:model version unavailable

chatgpt-codex-connector · 2026-10-14T15:54:04Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-14T16:01:30.870956Z	`a93f8f8`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a93f8f85fe

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-14T16:01:28Z

+        assert _codecs._normalize_encoding(encoding) == expected
+        assert _codecs._normalize_encoding(encoding=EncodingName(encoding)) == expected


Remove prohibited test-file changes

This newly added regression block changes test data, control flow, and assertions in extra_tests/snippets/stdlib_codecs_wide.py, but the repository's critical test policy permits test-file changes only for adding or removing expected-failure markers. Because none of this block is such a marker change, the test-file hunk needs to be removed before this commit complies.

AGENTS.md reference: AGENTS.md:L273-L279

Useful? React with 👍 / 👎.

youknowdot force-pushed the split-codec-normalization-315 branch from 412ddea to a93f8f8 Compare October 14, 2026 13:00

youknowone approved these changes Oct 14, 2026

View reviewed changes

youknowone marked this pull request as ready for review October 14, 2026 15:53

youknowone merged commit 6e82636 into RustPython:main Oct 14, 2026
30 checks passed

chatgpt-codex-connector Bot reviewed Oct 14, 2026

View reviewed changes

youknowdot mentioned this pull request Oct 14, 2026

Update gzip, tarfile, and shutil from CPython 3.15 #9112

Merged

1 task done

Navigation Menu

Repository navigation

Add Python 3.15 encoding normalization helper - #9115

Add Python 3.15 encoding normalization helper#9115
youknowone merged 1 commit into
RustPython:mainfrom
youknowdot:split-codec-normalization-315

youknowdot commented Oct 14, 2026

Uh oh!

coderabbitai Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Oct 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		assert _codecs._normalize_encoding(encoding) == expected
		assert _codecs._normalize_encoding(encoding=EncodingName(encoding)) == expected

Conversation

youknowdot commented Oct 14, 2026

Summary

CPython provenance

Validation

AI assistance

Uh oh!

coderabbitai Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 Library Dependencies

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

coderabbitai Bot commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 14, 2026 •

edited

Loading