Skip to content

gh-108828: Support selecting tests by labels #108829

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
serhiy-storchaka wants to merge 7 commits into
base: main
Choose a base branch
from
Open

gh-108828: Support selecting tests by labels #108829

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
6ea2f47
gh-108828: Support selecting tests by labels
serhiy-storchaka Sep 2, 2023
16ee29f
Support marking modules.
serhiy-storchaka Sep 3, 2023
fe7fcd0
Add documentation.
serhiy-storchaka Sep 3, 2023
5ae7161
Fix Sphinx warnings.
serhiy-storchaka Sep 3, 2023
4f41c75
Merge branch 'main' into libregrtest-labels
serhiy-storchaka Oct 29, 2023
c884b55
Make the order of options matter.
serhiy-storchaka Oct 29, 2023
22d68c6
Merge branch 'main' into libregrtest-labels
serhiy-storchaka Oct 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 35 additions & 2 deletions Doc/library/test.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -482,6 +482,8 @@ The :mod:`test.support` module defines the following functions:
``True`` if called by a function whose ``__name__`` is ``'__main__'``.
Used when tests are executed by :mod:`test.regrtest`.

If called at the top level, sets label "requires\_\ *resource*" on the module.


.. function:: sortdict(dict)

Expand All @@ -498,16 +500,34 @@ The :mod:`test.support` module defines the following functions:
rather than looking directly in the path directories.


.. function:: mark(label, *, globals=None)
Copy link
Copy Markdown
Member

@sobolevn sobolevn Oct 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to always call them "marks". The main reason is that pytest uses the same idea and pytest's marks are well-known.

Suggested change
.. function:: mark(label, *, globals=None)
.. function:: mark(name, *, globals=None)

Right now mark function adds a "label". It does not sound right.

Copy link
Copy Markdown
Member

@sobolevn sobolevn Oct 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or we can always call it "label".

Copy link
Copy Markdown
Member Author

@serhiy-storchaka serhiy-storchaka Oct 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is similar to pytest's markers, but there are enough differences in applying them and filtering by them. I do not have preference, "label", "marker" and "tag" are all look like synonyms in this context to me.


Add a label to tests.
Copy link
Copy Markdown
Member

@sobolevn sobolevn Oct 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Add a label to tests.
Add a mark to tests.

The ``@mark('label')`` decorator adds a label to method or class.
``test.support.mark('label', globals=globals())`` adds a label to the whole
module.

Many :mod:`test.support` decorators like :func:`requires_resource`,
:func:`~test.support.cpython_only` or :func:`bigmemtest` add labels
automatically.


.. function:: match_test(test)

Determine whether *test* matches the patterns set in :func:`set_match_tests`.
Determine whether *test* matches the patterns set in :func:`set_match_tests`
and labels set in :func:`set_match_tests2`.


.. function:: set_match_tests(accept_patterns=None, ignore_patterns=None)

Define match patterns on test filenames and test method names for filtering tests.


.. function:: set_match_tests2(accept_labels=None, ignore_labels=None)

Define labels on tests for filtering.


.. function:: run_unittest(*classes)

Execute :class:`unittest.TestCase` subclasses passed to the function. The
Expand Down Expand Up @@ -774,26 +794,31 @@ The :mod:`test.support` module defines the following functions:
.. decorator:: requires_zlib

Decorator for skipping tests if :mod:`zlib` doesn't exist.
Adds label "requires_zlib".


.. decorator:: requires_gzip

Decorator for skipping tests if :mod:`gzip` doesn't exist.
Adds label "requires_gzip".


.. decorator:: requires_bz2

Decorator for skipping tests if :mod:`bz2` doesn't exist.
Adds label "requires_bz2".


.. decorator:: requires_lzma

Decorator for skipping tests if :mod:`lzma` doesn't exist.
Adds label "requires_lzma".


.. decorator:: requires_resource(resource)

Decorator for skipping tests if *resource* is not available.
Adds label "requires\_\ *resource*".


.. decorator:: requires_docstrings
Expand All @@ -810,13 +835,16 @@ The :mod:`test.support` module defines the following functions:
.. decorator:: cpython_only

Decorator for tests only applicable to CPython.
Adds label "impl_detail_cpython".


.. decorator:: impl_detail(msg=None, **guards)

Decorator for invoking :func:`check_impl_detail` on *guards*. If that
returns ``False``, then uses *msg* as the reason for skipping the test.

For every keyword argument *name* adds a label
"impl_detail\_\ *name*" if its value is true or
"impl_detail_no\_\ *name*" otherwise.

.. decorator:: no_tracing

Expand Down Expand Up @@ -845,10 +873,13 @@ The :mod:`test.support` module defines the following functions:
method may be less than the requested value. If *dry_run* is ``False``, it
means the test doesn't support dummy runs when ``-M`` is not specified.

Adds label "bigmemtest".


.. decorator:: bigaddrspacetest

Decorator for tests that fill the address space.
Adds label "bigaddrspacetest".


.. function:: check_syntax_error(testcase, statement, errtext='', *, lineno=None, offset=None)
Expand Down Expand Up @@ -1630,6 +1661,8 @@ The :mod:`test.support.import_helper` module provides support for import tests.
optional for others, set *required_on* to an iterable of platform prefixes
which will be compared against :data:`sys.platform`.

If called at the top level, sets label "requires\_\ *name*" on the module.

.. versionadded:: 3.1


Expand Down
8 changes: 8 additions & 0 deletions Lib/test/libregrtest/cmdline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,8 @@ def __init__(self, **kwargs) -> None:
self.failfast = False
self.match_tests = None
self.ignore_tests = None
self.accept_labels = None
self.ignore_labels = None
self.pgo = False
self.pgo_extended = False
self.worker_json = None
Expand Down Expand Up @@ -259,6 +261,12 @@ def _create_parser():
group.add_argument('-i', '--ignore', metavar='PAT',
dest='ignore_tests', action='append',
help='ignore test cases and methods with glob pattern PAT')
group.add_argument('--label', metavar='NAME',
dest='accept_labels', action='append',
help='match test cases and methods with label NAME')
group.add_argument('--no-label', metavar='NAME',
dest='ignore_labels', action='append',
help='ignore test cases and methods with label NAME')
group.add_argument('--matchfile', metavar='FILENAME',
dest='match_filename',
help='similar to --match but get patterns from a '
Expand Down
3 changes: 3 additions & 0 deletions Lib/test/libregrtest/findtests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,9 +77,12 @@ def _list_cases(suite):
def list_cases(tests: TestTuple, *,
match_tests: FilterTuple | None = None,
ignore_tests: FilterTuple | None = None,
accept_labels: tuple[str, ...] | None = None,
ignore_labels: tuple[str, ...] | None = None,
test_dir: StrPath | None = None):
support.verbose = False
support.set_match_tests(match_tests, ignore_tests)
support.set_match_tests2(accept_labels, ignore_labels)

skipped = []
for test_name in tests:
Expand Down
10 changes: 10 additions & 0 deletions Lib/test/libregrtest/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,14 @@ def __init__(self, ns: Namespace):
self.ignore_tests: FilterTuple = tuple(ns.ignore_tests)
else:
self.ignore_tests = None
if ns.accept_labels:
self.accept_labels: tuple[str, ...] = tuple(ns.accept_labels)
Copy link
Copy Markdown
Member

@sobolevn sobolevn Oct 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe set[str]?

Copy link
Copy Markdown
Member Author

@serhiy-storchaka serhiy-storchaka Oct 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some reasons they should be immutable, as well as match_tests and ignore_tests.

In any case, I am planning to rewrite filtering by names. Instead of two tuples there will be more complex structure where order matters. And the same can be used for labels.

else:
self.accept_labels = None
if ns.ignore_labels:
self.ignore_labels: tuple[str, ...] = tuple(ns.ignore_labels)
else:
self.ignore_labels = None
self.exclude: bool = ns.exclude
self.fromfile: StrPath | None = ns.fromfile
self.starting_test: TestName | None = ns.start
Expand Down Expand Up @@ -484,6 +492,8 @@ def main(self, tests: TestList | None = None):
list_cases(selected,
match_tests=self.match_tests,
ignore_tests=self.ignore_tests,
accept_labels=self.accept_labels,
ignore_labels=self.ignore_labels,
test_dir=self.test_dir)
else:
exitcode = self.run_tests(selected, tests)
Expand Down
2 changes: 2 additions & 0 deletions Lib/test/libregrtest/runtests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ class RunTests:
fail_env_changed: bool = False
match_tests: FilterTuple | None = None
ignore_tests: FilterTuple | None = None
accept_labels: tuple[str, ...] | None = None
ignore_labels: tuple[str, ...] | None = None
match_tests_dict: FilterDict | None = None
rerun: bool = False
forever: bool = False
Expand Down
1 change: 1 addition & 0 deletions Lib/test/libregrtest/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ def setup_tests(runtests: RunTests):
support.PGO_EXTENDED = runtests.pgo_extended

support.set_match_tests(runtests.match_tests, runtests.ignore_tests)
support.set_match_tests2(runtests.accept_labels, runtests.ignore_labels)

if runtests.use_junit:
support.junit_xml_list = []
Expand Down
Loading