Skip to content

[3.12] gh-108834: Sync libregrtest with the main branch #108966

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Yhg1s merged 4 commits into from
Sep 8, 2023
Merged

[3.12] gh-108834: Sync libregrtest with the main branch #108966

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3f81740
gh-108834: regrtest reruns failed tests in subprocesses (#108839)
vstinner Sep 3, 2023
4f7e7b9
gh-108834: regrtest --fail-rerun exits with code 5 (#108896)
vstinner Sep 5, 2023
2cf86ef
gh-108416: Mark slow but not CPU bound test methods with requires_res…
serhiy-storchaka Sep 5, 2023
e26b17f
Manually sync Lib/test/libregrtest/ from main
vstinner Sep 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
gh-108834: regrtest reruns failed tests in subprocesses (#108839) 
When using --rerun option, regrtest now re-runs failed tests
in verbose mode in fresh worker processes to have more
deterministic behavior. So it can write its final report even
if a test killed a worker progress.

Add --fail-rerun option to regrtest: exit with non-zero exit code
if a test failed pass passed when re-run in verbose mode (in a
fresh process). That's now more useful since tests can pass
when re-run in a fresh worker progress, whereas they failed
when run after other tests when tests are run sequentially.

Rename --verbose2 option (-w) to --rerun. Keep --verbose2 as a
deprecated alias.

Changes:

* Fix and enhance statistics in regrtest summary. Add "(filtered)"
  when --match and/or --ignore options are used.
* Add RunTests class.
* Add TestResult.get_rerun_match_tests() method
* Rewrite code to serialize/deserialize worker arguments as JSON
  using a new WorkerJob class.
* Fix stats when a test is run with --forever --rerun.
* If failed test names cannot be parsed, log a warning and don't
  filter tests.
* test_regrtest.test_rerun_success() now uses a marker file, since
  the test is re-run in a separated process.
* Add tests on normalize_test_name() function.
* Add test_success() and test_skip() tests to test_regrtest.

(cherry picked from commit 31c2945)
  • Loading branch information
@vstinner
vstinner committed Sep 5, 2023
commit 3f81740be7901f6fdb18caefb2dcae6a3a5140fa
7 changes: 4 additions & 3 deletions Lib/test/bisect_cmd.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,9 +109,10 @@ def parse_args():

def main():
args = parse_args()
if '-w' in args.test_args or '--verbose2' in args.test_args:
print("WARNING: -w/--verbose2 option should not be used to bisect!")
print()
for opt in ('-w', '--rerun', '--verbose2'):
if opt in args.test_args:
print(f"WARNING: {opt} option should not be used to bisect!")
print()

if args.input:
with open(args.input) as fp:
Expand Down
11 changes: 8 additions & 3 deletions Lib/test/libregrtest/cmdline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ def __init__(self, **kwargs) -> None:
self.coverdir = 'coverage'
self.runleaks = False
self.huntrleaks = False
self.verbose2 = False
self.rerun = False
self.verbose3 = False
self.print_slow = False
self.random_seed = None
Expand Down Expand Up @@ -213,8 +213,10 @@ def _create_parser():
group = parser.add_argument_group('Verbosity')
group.add_argument('-v', '--verbose', action='count',
help='run tests in verbose mode with output to stdout')
group.add_argument('-w', '--verbose2', action='store_true',
group.add_argument('-w', '--rerun', action='store_true',
help='re-run failed tests in verbose mode')
group.add_argument('--verbose2', action='store_true', dest='rerun',
help='deprecated alias to --rerun')
group.add_argument('-W', '--verbose3', action='store_true',
help='display test output on failure')
group.add_argument('-q', '--quiet', action='store_true',
Expand Down Expand Up @@ -309,6 +311,9 @@ def _create_parser():
group.add_argument('--fail-env-changed', action='store_true',
help='if a test file alters the environment, mark '
'the test as failed')
group.add_argument('--fail-rerun', action='store_true',
help='if a test failed and then passed when re-run, '
'mark the tests as failed')

group.add_argument('--junit-xml', dest='xmlpath', metavar='FILENAME',
help='writes JUnit-style XML results to the specified '
Expand Down Expand Up @@ -380,7 +385,7 @@ def _parse_args(args, **kwargs):
ns.python = shlex.split(ns.python)
if ns.failfast and not (ns.verbose or ns.verbose3):
parser.error("-G/--failfast needs either -v or -W")
if ns.pgo and (ns.verbose or ns.verbose2 or ns.verbose3):
if ns.pgo and (ns.verbose or ns.rerun or ns.verbose3):
parser.error("--pgo/-v don't go together!")
if ns.pgo_extended:
ns.pgo = True # pgo_extended implies pgo
Expand Down
Loading