Repository navigation
Preserve regression collector outcomes and canonical process context - #9234
youknowone merged 4 commits into
Conversation
Record complete test identities, subtest diagnostics and fixture outcomes while keeping unittest event counts separate from once-per-case JSON totals. Reuse canonical process setup and avoid an unnecessary library-path override. Assisted-by: OpenAI Codex: model version unavailable
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configuration
📒 Files selected for processing: 6
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Assisted-by: OpenAI Codex: model version unavailable
Assisted-by: OpenAI Codex: model version unavailable
Merging this PR will degrade performance by 13.49%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | rustpython[frozenset.py] |
357.5 µs | 413.3 µs | -13.49% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing youknowdot:fix/regression-runner-observability-315 (f3524b8) with main (398dd70)
Footnotes
-
4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Carry the established integration VM optimization override into the regression runner split. Retain debug assertions, stack guards and Python recursion limits. Assisted-by: OpenAI Codex: model version unavailable
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f3524b81e3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| release mode, which should take less time than it would to run every test | ||
| snippet with RustPython compiled in debug mode. | ||
|
|
||
| ## Regression result collector |
There was a problem hiding this comment.
Add the required AI-assistance trailer
This commit identifies Codex assistance in prose, but git reports no Assisted-by: AGENT_NAME:MODEL_VERSION trailer. Add the required trailer before accepting the commit so the disclosure is machine-readable and complies with the repository policy.
AGENTS.md reference: AGENTS.md:L20-L23
Useful? React with 👍 / 👎.
| [profile.dev.package.rustpython-vm] | ||
| opt-level = 3 |
There was a problem hiding this comment.
Run the mandatory workspace and CAPI tests
The commit message explicitly says the full workspace and CAPI suites were not rerun for this VM-wide dev-profile override, while repository completion policy requires both command sets for every change. Run the mandated workspace and crates/capi tests before accepting this profile change.
AGENTS.md reference: AGENTS.md:L31-L35
Useful? React with 👍 / 👎.
| def test_pass(self): | ||
| for x in range(2): | ||
| with self.subTest(x=x): | ||
| self.assertTrue(True) |
There was a problem hiding this comment.
Remove forbidden test-logic additions
This newly added module introduces executable test bodies, assertions, and test data, but repository policy limits test-file edits to adding or removing expected-failure markers and explicitly forbids assertion, logic, or data changes. Remove or relocate this new test logic rather than landing it in a test file.
AGENTS.md reference: AGENTS.md:L278-L282
Useful? React with 👍 / 👎.
| traceback_text = None | ||
| if err is not None: | ||
| status = "failed" if issubclass(err[0], test.failureException) else "error" | ||
| traceback_text = self._exc_info_to_string(err, subtest) |
There was a problem hiding this comment.
Strip ANSI escapes from persisted tracebacks
When the runner is used with a TTY or with FORCE_COLOR=1/PYTHON_COLORS=1, the newly used _exc_info_to_string colorizes its return value, and that same string is persisted in both subtests[*].traceback and the case-level failure/error fields. I reproduced terminal escape sequences in the generated JSON under FORCE_COLOR=1; unlike the previous plain traceback.format_exception output, this makes result data environment-dependent and renders terminal control sequences in non-terminal consumers, so decolor the value before storing it.
Useful? React with 👍 / 👎.
Summary
Make the existing Python 3.15 regression-data collector diagnosable and give it the canonical test-process context:
test.libregrtest.setup.setup_process()before discovery. Remove the collector's redundantRUSTPYTHONPATHoverride; the same-checkout interpreter discovers its compiled-in Lib normally.No Lib changes, new workflow, matrix expansion, test skip, or assertion change to canonical tests. A subsequent coverage repair adds the established dev-profile VM optimization setting described below.
Validation
test_unicode_guard_envandtest_dump_path_configpass in bounded owned children after canonical setup, on both runtimes without a path override. Normal/-E/-I RustPython provenance identifies its compiled-in checkout Lib.The completed mega collector run still reported 42,754 cases, 15 failures, 180 errors, 4,716 skips and 291 expected failures. This PR does not make that suite green or rewrite historical results. Corrected JSON parent/fixture totals are not directly comparable to previously incomplete/overcounted totals; the raw unittest event summary remains distinct. The existing interleaved-suite timing limitation is unchanged.
The current regression website tolerates the additive JSON fields and renders status strings generically. Its pre-existing summary omits failure counts; that separate website behavior is unchanged.
Hosted collector follow-up
The first completed collector on this PR's main-based source reported 43,315 unittest executions, 12 failure events and 212 error events. Full IDs and tracebacks exposed two runner problems now repaired: successful subtest ID formatting could raise on huge integers or failing repr methods (three recorded reporter errors), and already-qualified split-package names were prefixed twice (64 import errors). The repair preserves normal full IDs, uses an explicit parent-ID fallback when formatting raises, and qualifies only bare discovery names. It does not change the integer conversion limit in production or suppress genuine import/test errors.
All five focused infrastructure tests pass on pinned CPython and the existing RustPython release, including huge-integer/raising-repr parameters across success/failure/error/skip and bare/qualified discovery with a genuine import failure. The test temporarily pins and restores its own integer limit. A subsequent Linux infrastructure assertion exposed forced-color output in CI; its two traceback text assertions now reuse the standard color-removal helper. All five tests also pass in paired plain, PYTHON_COLORS=1 and FORCE_COLOR=1 runs. Production output is unchanged. The full hosted collector has not yet run on this repair; its historical counts remain unchanged. The previous hosted run confirms unicode_guard_env passes; dump_path_config remains expected-failure on this main-based source because the separate startup fix has not landed.
Dev-profile coverage repair
The d9eed4a coverage job stopped progressing after an XML-RPC import raised RecursionError in the multiprocessing manager accepter thread. This follow-up carries the existing integration-tree
[profile.dev.package.rustpython-vm] opt-level = 3setting into this branch. It does not raise Python recursion limits, change native stack guards or thread stack sizes, disable debug assertions, or change test/release profiles.Fresh bounded local validation: the same uninstrumented dev source failed cold
xmlrpc.clientimports on both the main thread and an owned helper thread before the override, and passed both after it, retaining recursion limit 256 and debug assertions. The matched native evaluator frame allocation fell from 237,544 bytes to 776 bytes plus saved registers. A build using the committed manifest produced the same binary as the configuration-only comparison; both import probes passed again. Two exact dev-profile native tests also passed:default_python_thread_stack_size_debugandexplicit_python_thread_stack_size_is_a_floor_debug. Normal commit hooks passed.Full workspace/CAPI tests, strict workspace/CAPI Clippy, multiprocessing/forkserver suites, and instrumented coverage were not rerun locally; the broader build gates exceeded the existing disk preflight. Prior broader native results remain historical evidence only. The matched imports establish the local dev-profile failure and repair, not that the complete hosted stall is resolved. Hosted instrumented coverage on this commit is still pending.
AI assistance
Prepared at the maintainer's request as part of the ongoing Python 3.15 upgrade. OpenAI Codex assisted with implementation, source review, diagnostics and automated validation. No claim is made that the bot account is a maintainer or that these checks were manually performed by a human. Human review and platform CI remain necessary.