Skip to content

Preserve regression collector outcomes and canonical process context - #9234

Merged
youknowone merged 4 commits into
RustPython:mainfrom
youknowdot:fix/regression-runner-observability-315
Oct 20, 2026
Merged

youknowone merged 4 commits into
RustPython:mainfrom
youknowdot:fix/regression-runner-observability-315

Conversation

@youknowdot

@youknowdot youknowdot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Make the existing Python 3.15 regression-data collector diagnosable and give it the canonical test-process context:

  • Preserve full test IDs and every subtest callback reported by unittest, including successful/skipped outcomes and failure/error tracebacks. Print error details at the existing verbosity.
  • Record module/class fixture failures and skips without inventing unittest test executions. Count each completed parent/fixture record once, while retaining unittest's separate event-based summary.
  • Reuse test.libregrtest.setup.setup_process() before discovery. Remove the collector's redundant RUSTPYTHONPATH override; the same-checkout interpreter discovers its compiled-in Lib normally.

No Lib changes, new workflow, matrix expansion, test skip, or assertion change to canonical tests. A subsequent coverage repair adds the established dev-profile VM optimization setting described below.

Validation

  • Five focused infrastructure tests pass on pinned CPython 3.15.0 and a source-bound RustPython release: mixed/nested subtests, repeated IDs, interleaved suites, parent/cleanup errors, expected/unexpected successes, 12 module/class fixture skip/error variants, and interruption propagation.
  • Canonical test_unicode_guard_env and test_dump_path_config pass in bounded owned children after canonical setup, on both runtimes without a path override. Normal/-E/-I RustPython provenance identifies its compiled-in checkout Lib.
  • A paired startup observation confirms both runtimes can legitimately recover from an invalid PYTHONHOME when their respective path variable supplies encodings. This change does not alter runtime path precedence.
  • Normal commit hooks pass. The initial runner/workflow-only changes required no new Rust build. The later dev-profile coverage repair has the fresh, narrowly scoped native validation described below; broader native verification remains historical.

The completed mega collector run still reported 42,754 cases, 15 failures, 180 errors, 4,716 skips and 291 expected failures. This PR does not make that suite green or rewrite historical results. Corrected JSON parent/fixture totals are not directly comparable to previously incomplete/overcounted totals; the raw unittest event summary remains distinct. The existing interleaved-suite timing limitation is unchanged.

The current regression website tolerates the additive JSON fields and renders status strings generically. Its pre-existing summary omits failure counts; that separate website behavior is unchanged.

Hosted collector follow-up

The first completed collector on this PR's main-based source reported 43,315 unittest executions, 12 failure events and 212 error events. Full IDs and tracebacks exposed two runner problems now repaired: successful subtest ID formatting could raise on huge integers or failing repr methods (three recorded reporter errors), and already-qualified split-package names were prefixed twice (64 import errors). The repair preserves normal full IDs, uses an explicit parent-ID fallback when formatting raises, and qualifies only bare discovery names. It does not change the integer conversion limit in production or suppress genuine import/test errors.

All five focused infrastructure tests pass on pinned CPython and the existing RustPython release, including huge-integer/raising-repr parameters across success/failure/error/skip and bare/qualified discovery with a genuine import failure. The test temporarily pins and restores its own integer limit. A subsequent Linux infrastructure assertion exposed forced-color output in CI; its two traceback text assertions now reuse the standard color-removal helper. All five tests also pass in paired plain, PYTHON_COLORS=1 and FORCE_COLOR=1 runs. Production output is unchanged. The full hosted collector has not yet run on this repair; its historical counts remain unchanged. The previous hosted run confirms unicode_guard_env passes; dump_path_config remains expected-failure on this main-based source because the separate startup fix has not landed.

Dev-profile coverage repair

The d9eed4a coverage job stopped progressing after an XML-RPC import raised RecursionError in the multiprocessing manager accepter thread. This follow-up carries the existing integration-tree [profile.dev.package.rustpython-vm] opt-level = 3 setting into this branch. It does not raise Python recursion limits, change native stack guards or thread stack sizes, disable debug assertions, or change test/release profiles.

Fresh bounded local validation: the same uninstrumented dev source failed cold xmlrpc.client imports on both the main thread and an owned helper thread before the override, and passed both after it, retaining recursion limit 256 and debug assertions. The matched native evaluator frame allocation fell from 237,544 bytes to 776 bytes plus saved registers. A build using the committed manifest produced the same binary as the configuration-only comparison; both import probes passed again. Two exact dev-profile native tests also passed: default_python_thread_stack_size_debug and explicit_python_thread_stack_size_is_a_floor_debug. Normal commit hooks passed.

Full workspace/CAPI tests, strict workspace/CAPI Clippy, multiprocessing/forkserver suites, and instrumented coverage were not rerun locally; the broader build gates exceeded the existing disk preflight. Prior broader native results remain historical evidence only. The matched imports establish the local dev-profile failure and repair, not that the complete hosted stall is resolved. Hosted instrumented coverage on this commit is still pending.

AI assistance

Prepared at the maintainer's request as part of the ongoing Python 3.15 upgrade. OpenAI Codex assisted with implementation, source review, diagnostics and automated validation. No claim is made that the bot account is a maintainer or that these checks were manually performed by a human. Human review and platform CI remain necessary.

Record complete test identities, subtest diagnostics and fixture outcomes while keeping unittest event counts separate from once-per-case JSON totals. Reuse canonical process setup and avoid an unnecessary library-path override.

Assisted-by: OpenAI Codex: model version unavailable
@coderabbitai

coderabbitai Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration
  • Configuration used: Repository: RustPython/RustPython/.coderabbit.yml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 41c7f5b3-951f-4ed0-8b5b-1fb0f0466b3b
📥 Commits

Reviewing files that changed from the base of the PR and between c08375b and f3524b8.

📒 Files selected for processing: 6
  • .github/workflows/cron-ci.yaml
  • Cargo.toml
  • extra_tests/README.md
  • extra_tests/custom_text_test_runner.py
  • extra_tests/jsontests.py
  • extra_tests/test_custom_text_test_runner.py
+------------------------------------------------------------------------+
| I'm CodeRabbit. I review the code you never knew you needed reviewing. |
+------------------------------------------------------------------------+
  \
   \   (\__/)
       (•ㅅ•)
       /   づ
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Assisted-by: OpenAI Codex: model version unavailable
Assisted-by: OpenAI Codex: model version unavailable
@codspeed

codspeed Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will degrade performance by 13.49%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 1 regressed benchmark
✅ 61 untouched benchmarks
⏩ 4 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ rustpython[frozenset.py] 357.5 µs 413.3 µs -13.49%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing youknowdot:fix/regression-runner-observability-315 (f3524b8) with main (398dd70)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Carry the established integration VM optimization override into the regression runner split. Retain debug assertions, stack guards and Python recursion limits.

Assisted-by: OpenAI Codex: model version unavailable
@youknowone
youknowone marked this pull request as ready for review October 20, 2026 09:07
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-20T09:18:40.016183Z f3524b8 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@youknowone
youknowone merged commit c3b8ce7 into RustPython:main Oct 20, 2026
23 of 25 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f3524b81e3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extra_tests/README.md
release mode, which should take less time than it would to run every test
snippet with RustPython compiled in debug mode.

## Regression result collector

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add the required AI-assistance trailer

This commit identifies Codex assistance in prose, but git reports no Assisted-by: AGENT_NAME:MODEL_VERSION trailer. Add the required trailer before accepting the commit so the disclosure is machine-readable and complies with the repository policy.

AGENTS.md reference: AGENTS.md:L20-L23

Useful? React with 👍 / 👎.

Comment thread Cargo.toml
Comment on lines +106 to +107
[profile.dev.package.rustpython-vm]
opt-level = 3

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Run the mandatory workspace and CAPI tests

The commit message explicitly says the full workspace and CAPI suites were not rerun for this VM-wide dev-profile override, while repository completion policy requires both command sets for every change. Run the mandated workspace and crates/capi tests before accepting this profile change.

AGENTS.md reference: AGENTS.md:L31-L35

Useful? React with 👍 / 👎.

def test_pass(self):
for x in range(2):
with self.subTest(x=x):
self.assertTrue(True)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Remove forbidden test-logic additions

This newly added module introduces executable test bodies, assertions, and test data, but repository policy limits test-file edits to adding or removing expected-failure markers and explicitly forbids assertion, logic, or data changes. Remove or relocate this new test logic rather than landing it in a test file.

AGENTS.md reference: AGENTS.md:L278-L282

Useful? React with 👍 / 👎.

traceback_text = None
if err is not None:
status = "failed" if issubclass(err[0], test.failureException) else "error"
traceback_text = self._exc_info_to_string(err, subtest)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Strip ANSI escapes from persisted tracebacks

When the runner is used with a TTY or with FORCE_COLOR=1/PYTHON_COLORS=1, the newly used _exc_info_to_string colorizes its return value, and that same string is persisted in both subtests[*].traceback and the case-level failure/error fields. I reproduced terminal escape sequences in the generated JSON under FORCE_COLOR=1; unlike the previous plain traceback.format_exception output, this makes result data environment-dependent and renders terminal control sequences in non-terminal consumers, so decolor the value before storing it.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants