Skip to content

GH-102613: Fast recursive globbing in pathlib.Path.glob() #104512

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
barneygale merged 17 commits into from
Jun 6, 2023
Merged

GH-102613: Fast recursive globbing in pathlib.Path.glob() #104512

Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
7a58251
GH-102613: Simplify implementation of `pathlib.Path.glob()`
barneygale May 15, 2023
d5b1836
Speed up matching `*`
barneygale May 15, 2023
d5c86c6
Add comments, docstrings.
barneygale May 15, 2023
53dcb79
Merge branch 'main' into gh-102613-remove-selector-classes
barneygale May 18, 2023
6da6a83
Merge branch 'main' into gh-102613-remove-selector-classes
barneygale May 29, 2023
73ed81d
Merge branch 'main' into gh-102613-remove-selector-classes
barneygale May 30, 2023
4005619
Add support for matching files recursively.
barneygale May 31, 2023
9401d36
Implement walk-and-match algorithm.
barneygale May 31, 2023
b217587
Fix up docs, news blurb.
barneygale May 31, 2023
de7e857
Fix comment
barneygale May 31, 2023
d1023c7
Fix handling of newlines in filenames.
barneygale May 31, 2023
ad33eec
Speed up recursive selection
barneygale Jun 1, 2023
064efdb
Exclude `self` from walk-and-match matching.
barneygale Jun 1, 2023
14c6a58
Optimize walk-and-match logic.
barneygale Jun 1, 2023
04720bd
Consume adjacent '**' segments before considering use of matching.
barneygale Jun 1, 2023
9c6b44f
Add some more tests for complex patterns.
barneygale Jun 1, 2023
4cfb836
Drop test case that doesn't work on Windows.
barneygale Jun 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add comments, docstrings.
  • Loading branch information
@barneygale
barneygale committed May 15, 2023
commit d5c86c6e003b529d9c367815615c7644d2f7d5f1
8 changes: 8 additions & 0 deletions Lib/pathlib.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,13 +64,16 @@ def _is_case_sensitive(flavour):

@functools.lru_cache()
def _compile_pattern_part(pattern_part, case_sensitive):
"""Compile given glob pattern to a re.Pattern object (observing case
sensitivity), or None if the pattern should match everything."""
if pattern_part == '*':
return None
flags = re.NOFLAG if case_sensitive else re.IGNORECASE
return re.compile(fnmatch.translate(pattern_part), flags=flags).fullmatch


def _select_children(paths, dir_only, match):
"""Yield direct children of given paths, filtering by name and type."""
for path in paths:
try:
# We must close the scandir() object before proceeding to
Expand All @@ -93,6 +96,7 @@ def _select_children(paths, dir_only, match):


def _select_recursive(paths):
"""Yield given paths and all their subdirectories, recursively."""
for path in paths:
yield path
for dirpath, dirnames, _ in path.walk():
Expand All @@ -101,6 +105,7 @@ def _select_recursive(paths):


def _select_unique(paths):
"""Yields the given paths, filtering out duplicates."""
yielded = set()
try:
for path in paths:
Expand Down Expand Up @@ -939,13 +944,16 @@ def _glob(self, pattern_parts, case_sensitive):
part = pattern_parts[part_idx]
part_idx += 1
if part == '':
# Trailing slash.
pass
elif part == '..':
paths = (path._make_child_relpath('..') for path in paths)
elif part == '**':
# Collapse adjacent '**' segments.
while part_idx < len(pattern_parts) and pattern_parts[part_idx] == '**':
part_idx += 1
paths = _select_recursive(paths)
# De-duplicate if we've already seen a '**' segment.
if deduplicate:
paths = _select_unique(paths)
deduplicate = True
Expand Down