Align JSON array hooks and libraries with Python 3.15 by youknowdot · Pull Request #9135 · RustPython/RustPython

youknowdot · 2026-10-15T05:51:05Z

Summary

Add native JSON array_hook support, including empty and nested arrays, callback error propagation, and the scanner's read-only hook member.
Generate the related JSON library and selected tests from exact-final CPython d220409a13b77de7a6c30abe16ea90db01528edc using scripts/update_lib. This also brings the matching immutable-container support into the Python implementation.
Preserve RustPython's native JSON import/exception-cache workaround and existing test markers. Remove the obsolete, unused _from_serde adapter.
Keep generated __pycache__ directories out of update_lib copy-lib while preserving canonical source and asset files, with a tempfile-only regression test.

Split from #8954. No dependency, ABI-export, workflow, or unrelated native changes.

Validation

At b9d2e73:

Normal commit hooks passed.
Narrow threaded release build passed; strict scoped stdlib/VM all-target Clippy passed with -D warnings.
21 individually selected bounded candidate cases passed: four callback-contract checks, sixteen canonical Python/native JSON cases, and the new tiny JSONDecodeError pickle roundtrip. No skips or expected/unexpected failures were accepted.
Four callback-contract cases also passed on the verified exact-final CPython reference interpreter.
Corrected update_lib regeneration produced the reviewed source hashes; native import/cache guards and existing markers remain intact.
Copy-boundary regression test passed.

Each interpreter case ran in a separate 30-second/1-GiB process with exact library origins and native backend provenance checked. These are focused checks, not a full JSON/GC/UAF/large-input/recursion/CAPI suite. The reference interpreter's existing dirty build banner remains a metadata caveat despite the verified clean exact-final source checkout.

AI assistance

Prepared with OpenAI Codex under the requesting maintainer's direction. The runtime does not expose an exact model identifier; the commit records that limitation.

Summary by CodeRabbit

New Features
- JSON parsing can now apply a configured hook to parsed arrays, allowing their values to be customized. Errors from the hook are passed through.
Bug Fixes
- Library directory copies now exclude __pycache__ directories while preserving files with that name and other files.

Generate the JSON library and selected tests with update_lib from CPython d220409a13b77de7a6c30abe16ea90db01528edc, preserving native bootstrap workarounds and test markers. Exclude generated cache directories at the library copy boundary. Assisted-by: OpenAI Codex:model version unavailable

coderabbitai · 2026-10-15T05:51:14Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: RustPython/RustPython/.coderabbit.yml
Review profile: CHILL
Plan: Advanced
Run ID: 0a4b8ef0-6709-4352-b681-864c4e3c1a4f

📥 Commits

Reviewing files that changed from the base of the PR and between 92e20da and b9d2e73.

⛔ Files ignored due to path filters: 11

Lib/json/__init__.py is excluded by !Lib/**
Lib/json/decoder.py is excluded by !Lib/**
Lib/json/encoder.py is excluded by !Lib/**
Lib/json/scanner.py is excluded by !Lib/**
Lib/json/tool.py is excluded by !Lib/**
Lib/test/test_json/test_decode.py is excluded by !Lib/**
Lib/test/test_json/test_dump.py is excluded by !Lib/**
Lib/test/test_json/test_encode_basestring_ascii.py is excluded by !Lib/**
Lib/test/test_json/test_enum.py is excluded by !Lib/**
Lib/test/test_json/test_pickleable.py is excluded by !Lib/**
Lib/test/test_json/test_unicode.py is excluded by !Lib/**

📒 Files selected for processing: 3

crates/stdlib/src/json.rs
scripts/update_lib/cmd_copy_lib.py
scripts/update_lib/tests/test_copy_lib.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 7 remain after this review.

📝 Walkthrough

The changes add an optional hook for finalizing parsed JSON arrays and exclude __pycache__ directories from copied libraries. Tests verify the directory-copy filtering and preservation of source contents.

Changes: JSON Array Hook

Layer / File(s)	Summary
Store and expose the array hook `crates/stdlib/src/json.rs`	`JsonScanner` reads an optional `array_hook` from its construction context, stores it, and exposes it through a getter.
Finalize parsed arrays `crates/stdlib/src/json.rs`	Empty and nonempty arrays pass through `finalize_array`. When a hook is configured, its result becomes the parsed value, and hook errors propagate.

Library Copy Cache Filtering

Layer / File(s)	Summary
Filter bytecode cache directories during copies `scripts/update_lib/cmd_copy_lib.py`, `scripts/update_lib/tests/test_copy_lib.py`	Directory copies exclude `__pycache__` directories. The test checks destination contents and confirms that source files remain unchanged.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Feature

Suggested reviewers: youknowone

Merge Risk: ⚪ Minimal · up to b9d2e

No merge-blocking issue is identified; merge after normal checks.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 44.44% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 9 functions across 3 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly summarizes the main changes: aligning JSON array hooks and library handling with Python 3.15.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-10-15T06:26:10Z

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
[x] lib: cpython/Lib/json
[ ] test: cpython/Lib/test/test_json (TODO: 2)

dependencies:

json

dependent tests: (14 tests)

json: test_embed test_json test_logging test_plistlib test_pyrepl test_subprocess test_sysconfig test_tomllib test_tools test_traceback test_zoneinfo
- importlib.metadata: test_importlib
- multiprocessing.resource_tracker: test_concurrent_futures
- pdb: test_pdb

Legend:

[+] path exists in CPython
[x] up-to-date, [ ] outdated

chatgpt-codex-connector · 2026-10-15T06:57:42Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-15T07:04:32.971400Z	`b9d2e73`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b9d2e73d39

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-15T07:04:29Z

        self.assertEqual(self.loads('[]'), [])
        self.assertEqual(self.loads('""'), "")

+    def test_object_hook(self):


Remove direct test logic additions

This new test method, together with the added or rewritten methods in the other Lib/test/test_json files, changes assertions, test logic, and test data; repository policy restricts test-file edits to adding or removing expectedFailure markers. Please leave those test bodies unchanged rather than landing them as part of the JSON implementation.

AGENTS.md reference: AGENTS.md:L273-L279

Useful? React with 👍 / 👎.

youknowone approved these changes Oct 15, 2026

View reviewed changes

youknowone marked this pull request as ready for review October 15, 2026 06:57

youknowone enabled auto-merge (squash) October 15, 2026 06:57

chatgpt-codex-connector Bot reviewed Oct 15, 2026

View reviewed changes

youknowone merged commit 498adff into RustPython:main Oct 15, 2026
22 checks passed

Navigation Menu

Repository navigation

Align JSON array hooks and libraries with Python 3.15 - #9135

Align JSON array hooks and libraries with Python 3.15#9135
youknowone merged 1 commit into
RustPython:mainfrom
youknowdot:split-json-array-hook-final

youknowdot commented Oct 15, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Oct 15, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 15, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 15, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

youknowdot commented Oct 15, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

AI assistance

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 15, 2026

📦 Library Dependencies

Uh oh!

chatgpt-codex-connector Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

youknowdot commented Oct 15, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 15, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 15, 2026 •

edited

Loading