Skip to content

Preserve regression collector outcomes and canonical process context #9234

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
youknowone merged 4 commits into from
Oct 20, 2026
+449 −60
Merged

Preserve regression collector outcomes and canonical process context #9234

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3a2331c
Preserve regression collector outcomes and canonical process context
youknowdot Oct 19, 2026
8a872d9
Fix regression runner subtest formatting and qualified discovery
youknowdot Oct 19, 2026
d9eed4a
Make runner traceback assertions independent of forced colors
youknowdot Oct 19, 2026
f3524b8
Keep dev VM frames compact for stdlib imports
youknowdot Oct 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .github/workflows/cron-ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,8 +78,6 @@ jobs:

- name: collect tests data
run: target/release/rustpython extra_tests/jsontests.py
env:
RUSTPYTHONPATH: ${{ github.workspace }}/Lib

- name: upload tests data to the website
if: ${{ github.event_name != 'pull_request' }}
Expand Down
5 changes: 5 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,11 @@ required-features = [
[profile.dev.package."*"]
opt-level = 3

# Unoptimized opcode dispatch leaves oversized native frames active during
# imports and callbacks. Keep the evaluator compact without changing dev guards.
[profile.dev.package.rustpython-vm]
opt-level = 3
Comment on lines +106 to +107

@chatgpt-codex-connector chatgpt-codex-connector Bot Oct 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Run the mandatory workspace and CAPI tests

The commit message explicitly says the full workspace and CAPI suites were not rerun for this VM-wide dev-profile override, while repository completion policy requires both command sets for every change. Run the mandated workspace and crates/capi tests before accepting this profile change.

AGENTS.md reference: AGENTS.md:L31-L35

Useful? React with 👍 / 👎.


[profile.test]
opt-level = 3
# https://github.com/rust-lang/rust/issues/92869
Expand Down
9 changes: 9 additions & 0 deletions extra_tests/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,12 @@ Simply run `pytest -v` in this directory, and the tests should run (and hopefull
pass). If it hangs for a long time, that's because it's building RustPython in
release mode, which should take less time than it would to run every test
snippet with RustPython compiled in debug mode.

## Regression result collector

@chatgpt-codex-connector chatgpt-codex-connector Bot Oct 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add the required AI-assistance trailer

This commit identifies Codex assistance in prose, but git reports no Assisted-by: AGENT_NAME:MODEL_VERSION trailer. Add the required trailer before accepting the commit so the disclosure is machine-readable and complies with the repository policy.

AGENTS.md reference: AGENTS.md:L20-L23

Useful? React with 👍 / 👎.


`jsontests.py` uses the canonical regrtest process setup before discovery. Its
JSON and tables count each parent test once; failed subtests make their parent
fail. The standard unittest summary counts individual failure/error/skip events
and can therefore differ. Each case retains its full `id` and all subtest
callbacks reported by unittest, including their IDs, outcomes and tracebacks.
Historical result files are not rewritten.
186 changes: 131 additions & 55 deletions extra_tests/custom_text_test_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,20 +32,27 @@
import re
import sys
import time
import traceback
import unittest
from functools import reduce
from unittest.runner import registerResult, result


def _subtest_id(parent, subtest):
try:
return subtest.id()
except Exception as exc:
# Parameter formatting must not change the test's outcome.
return f"{parent.id()} (<subtest id unavailable: {type(exc).__name__}>)"


def _get_method_dict(test):
"""Get the __dict__ of the underlying function for a test method.

Works for both bound methods (__func__.__dict__) and plain functions.
"""
method = getattr(test, test._testMethodName)
method = getattr(test, getattr(test, "_testMethodName", ""), None)
func = getattr(method, "__func__", method)
return func.__dict__
return getattr(func, "__dict__", {})


class TablePrinter(object):
Expand Down Expand Up @@ -228,9 +235,10 @@ def __init__(
self.show_test_info = verbosity > 1
self.show_individual_suite_results = verbosity > 2
self.show_errors = verbosity > 3
self.show_errors_detail = verbosity > 4
self.show_errors_detail = verbosity > 1
self.show_all = verbosity > 4
self.suite = None
self.current_test = None
self.total_execution_time = 0
self.separator1 = "=" * CustomTextTestResult._num_formatting_chars
self.separator2 = "-" * CustomTextTestResult._num_formatting_chars
Expand Down Expand Up @@ -306,6 +314,10 @@ def stopTestRun(self):
json.dump(self.results, f)

def startTest(self, test):
super().startTest(test)
self._start_case(test)

def _start_case(self, test):
suite_base_category = (
test.__class__.base_test_category
if hasattr(test.__class__, "base_test_category")
Expand All @@ -317,8 +329,8 @@ def startTest(self, test):
if hasattr(test.__class__, "name")
else test.__class__.__name__,
)
self.case = test._testMethodName
super(CustomTextTestResult, self).startTest(test)
self.current_test = test
self.case = getattr(test, "_testMethodName", test.id())
if not self.suite or self.suite != self.next_suite:
if self.suite:
self.results["suites"][self.suite_map[self.suite]]["execution_time"] = (
Expand All @@ -341,7 +353,7 @@ def startTest(self, test):
self.case = name_override if name_override else self.case
if self.show_test_info:
# self.stream.writeln(self.separator2)
self.stream.write("CASE: %s" % self.case)
self.stream.write("CASE: %s" % test.id())
if desc := test.shortDescription():
self.stream.write(" (Description: %s)" % desc)
self.stream.write("... ")
Expand All @@ -368,16 +380,13 @@ def startTest(self, test):
"execution_time": None,
}
self.suite_number += 1
self.num_cases = 0
self.num_passed = 0
self.num_failed = 0
self.num_skipped = 0
self.num_expected_failures = 0
self.results["suites"][self.suite_map[self.suite]]["cases"][
self.case_number
] = {
"name": self.case,
"method": test._testMethodName,
"method": getattr(test, "_testMethodName", test.id()),
"id": test.id(),
"subtests": [],
"result": None,
"description": test.shortDescription(),
"note": None,
Expand Down Expand Up @@ -409,42 +418,49 @@ def startTest(self, test):
break

def stopTest(self, test):
super().stopTest(test)
self._finish_case()

def _finish_case(self):
case = self.results["suites"][self.suite_map[self.suite]]["cases"][
self.current_case_number
]
if case["result"] is None and any(
subtest["result"] == "skipped" for subtest in case["subtests"]
):
# A skipped subtest prevents unittest from calling addSuccess.
case["result"] = "skipped"
if self.show_test_info:
self.stream.writeln("SKIPPED (subtest)")
self.end_time = time.time()
self.execution_time = self.end_time - self.start_time
self.suite_execution_time += self.execution_time
self.total_execution_time += self.execution_time
super(CustomTextTestResult, self).stopTest(test)
self.num_cases += 1
self.results["suites"][self.suite_map[self.suite]]["num_passed"] = (
self.num_passed
)
self.results["suites"][self.suite_map[self.suite]]["num_failed"] = (
self.num_failed
)
self.results["suites"][self.suite_map[self.suite]]["num_skipped"] = (
self.num_skipped
)
self.results["suites"][self.suite_map[self.suite]]["num_expected_failures"] = (
self.num_expected_failures
)
# JSON/table totals count parent cases once. The standard unittest
# summary still counts individual failure/error/skip events, including
# subtests; keep those two metrics distinct.
suite = self.results["suites"][self.suite_map[self.suite]]
status = case["result"]
outcome = {
"passed": "passed",
"failed": "failed",
"error": "failed",
"unexpected_success": "failed",
"skipped": "skipped",
"expected_failure": "expected_failures",
}.get(status)
if outcome is not None:
key = "num_" + outcome
suite[key] += 1
self.results[key] += 1
self.results["suites"][self.suite_map[self.suite]]["cases"][
self.current_case_number
]["execution_time"] = format(
self.execution_time,
".%sf" % CustomTextTestResult._execution_time_significant_digits,
)
self.results["num_passed"] += self.num_passed
self.results["num_failed"] += self.num_failed
self.results["num_skipped"] += self.num_skipped
self.results["num_expected_failures"] += self.num_expected_failures
self.case_number += 1

def print_error_string(self, err):
error_string = "".join(traceback.format_exception(err[0], err[1], err[2]))
if self.show_errors:
self.stream.writeln(self.separator_failure)
self.stream.write(error_string)
return error_string
self.current_test = None

def addScreenshots(self, test):
for root, dirs, files in os.walk(self.result_screenshots_dir):
Expand All @@ -462,11 +478,38 @@ def addSuccess(self, test):
self.results["suites"][self.suite_map[self.suite]]["cases"][
self.current_case_number
]["result"] = "passed"
self.num_passed += 1
self.addScreenshots(test)

def addSubTest(self, test, subtest, err):
super().addSubTest(test, subtest, err)
case = self.results["suites"][self.suite_map[self.suite]]["cases"][
self.current_case_number
]
subtest_id = _subtest_id(test, subtest)
status = "passed"
traceback_text = None
if err is not None:
status = "failed" if issubclass(err[0], test.failureException) else "error"
traceback_text = self._exc_info_to_string(err, subtest)

@chatgpt-codex-connector chatgpt-codex-connector Bot Oct 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Strip ANSI escapes from persisted tracebacks

When the runner is used with a TTY or with FORCE_COLOR=1/PYTHON_COLORS=1, the newly used _exc_info_to_string colorizes its return value, and that same string is persisted in both subtests[*].traceback and the case-level failure/error fields. I reproduced terminal escape sequences in the generated JSON under FORCE_COLOR=1; unlike the previous plain traceback.format_exception output, this makes result data environment-dependent and renders terminal control sequences in non-terminal consumers, so decolor the value before storing it.

Useful? React with 👍 / 👎.

if case["result"] != "error":
case["result"] = status
key = "failures" if status == "failed" else "errors"
case[key] = (case[key] or "") + subtest_id + "\n" + traceback_text
if self.show_test_info:
self.stream.writeln("%s: %s" % (status.upper(), subtest_id))
case["subtests"].append(
{
"id": subtest_id,
"result": status,
"traceback": traceback_text,
}
)

def addError(self, test, err):
error_string = self.print_error_string(err)
fixture = self.current_test is None
if fixture:
self._start_case(test)
error_string = self._exc_info_to_string(err, test)
super(CustomTextTestResult, self).addError(test, err)
if self.show_test_info:
# self.stream.writeln(self.separator_pre_result)
Expand All @@ -477,28 +520,64 @@ def addError(self, test, err):
]["result"] = "error"
self.results["suites"][self.suite_map[self.suite]]["cases"][
self.current_case_number
]["errors"] = error_string
self.num_failed += 1
]["errors"] = (
self.results["suites"][self.suite_map[self.suite]]["cases"][
self.current_case_number
]["errors"]
or ""
) + error_string
self.addScreenshots(test)
if fixture:
self._finish_case()

def addFailure(self, test, err):
error_string = self.print_error_string(err)
error_string = self._exc_info_to_string(err, test)
super(CustomTextTestResult, self).addFailure(test, err)
if self.show_test_info:
# self.stream.writeln(self.separator_pre_result)
self.stream.writeln("FAIL")
self.stream.flush()
self.results["suites"][self.suite_map[self.suite]]["cases"][
self.current_case_number
]["result"] = "failed"
]["result"] = (
"error"
if self.results["suites"][self.suite_map[self.suite]]["cases"][
self.current_case_number
]["errors"]
else "failed"
)
self.results["suites"][self.suite_map[self.suite]]["cases"][
self.current_case_number
]["failures"] = error_string
self.num_failed += 1
]["failures"] = (
self.results["suites"][self.suite_map[self.suite]]["cases"][
self.current_case_number
]["failures"]
or ""
) + error_string
self.addScreenshots(test)

def addSkip(self, test, reason):
fixture = self.current_test is None
if fixture:
self._start_case(test)
super(CustomTextTestResult, self).addSkip(test, reason)
if test is not self.current_test:
self.results["suites"][self.suite_map[self.suite]]["cases"][
self.current_case_number
]["subtests"].append(
{
"id": _subtest_id(self.current_test, test),
"result": "skipped",
"reason": reason,
}
)
return
case = self.results["suites"][self.suite_map[self.suite]]["cases"][
self.current_case_number
]
if case["result"] in ("failed", "error"):
case["note"] = reason
return
if self.show_test_info:
# self.stream.writeln(self.separator_pre_result)
self.stream.writeln("SKIPPED {0!r}".format(reason))
Expand All @@ -509,7 +588,8 @@ def addSkip(self, test, reason):
self.results["suites"][self.suite_map[self.suite]]["cases"][
self.current_case_number
]["note"] = reason
self.num_skipped += 1
if fixture:
self._finish_case()

def addExpectedFailure(self, test, err):
super(CustomTextTestResult, self).addExpectedFailure(test, err)
Expand All @@ -520,7 +600,6 @@ def addExpectedFailure(self, test, err):
self.results["suites"][self.suite_map[self.suite]]["cases"][
self.current_case_number
]["result"] = "expected_failure"
self.num_expected_failures += 1
self.addScreenshots(test)

def addUnexpectedSuccess(self, test):
Expand All @@ -529,7 +608,9 @@ def addUnexpectedSuccess(self, test):
# self.stream.writeln(self.separator_pre_result)
self.stream.writeln("UNEXPECTED SUCCESS")
self.stream.flush()
self.num_failed += 1
self.results["suites"][self.suite_map[self.suite]]["cases"][
self.current_case_number
]["result"] = "unexpected_success"
self.addScreenshots(test)

def printOverallSuiteResults(self, r):
Expand Down Expand Up @@ -703,12 +784,7 @@ def printErrorsDetail(self, r):
)
for case in failures_and_errors:
case_ref = r[suite]["cases"][case]
case_str = "%s%s" % (
case_ref["name"],
" (%s)" % case_ref["method"]
if case_ref["name"] != case_ref["method"]
else "",
)
case_str = case_ref.get("id", case_ref["method"])
errors = case_ref["errors"]
failures = case_ref["failures"]
self.stream.writeln(self.separator1)
Expand Down
8 changes: 5 additions & 3 deletions extra_tests/jsontests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,12 @@

from custom_text_test_runner import CustomTextTestRunner as Runner
from test.libregrtest.findtests import findtests
from test.libregrtest.setup import setup_process


def loadTestsOrSkip(loader, name):
if not name.startswith("test."):
name = "test." + name
try:
return loader.loadTestsFromName(name)
except unittest.SkipTest as exc:
Expand All @@ -20,15 +23,14 @@ def testSkipped(self):


def main():
setup_process()
testnames = findtests()
# idk why this fixes the hanging, if it does
testnames.remove("test_importlib")
testnames.insert(0, "test_importlib")

loader = unittest.defaultTestLoader
suite = loader.suiteClass(
[loadTestsOrSkip(loader, "test." + name) for name in testnames]
)
suite = loader.suiteClass([loadTestsOrSkip(loader, name) for name in testnames])

resultsfile = os.path.join(os.path.dirname(__file__), "cpython_tests_results.json")
if os.path.exists(resultsfile):
Expand Down
Loading
Loading