Skip to content

gh-89727: Fix os.walk RecursionError on deep trees #99803

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JelleZijlstra merged 16 commits into from
Dec 19, 2022
Merged

gh-89727: Fix os.walk RecursionError on deep trees #99803

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
99f4691
gh-89727: Fix os.walk RecursionError on deep trees
jonburdo Nov 26, 2022
3552d49
add unit test to ensure os.walk is not affect by recursion limit
jonburdo Dec 16, 2022
3078ea6
move os._walk code to os.walk
jonburdo Dec 16, 2022
f37bbe8
fix blurb to mention os.walk
jonburdo Dec 16, 2022
f26a5b8
use deque in os.walk
jonburdo Dec 16, 2022
955d21b
Merge branch 'main' into iterative-os-walk
jonburdo Dec 16, 2022
8fbfb2e
Merge branch 'main' into iterative-os-walk
jonburdo Dec 16, 2022
ef46eda
move deque import inside os.walk
jonburdo Dec 16, 2022
e261b9f
rename temp_recursion_limit to set_recursion_limit
jonburdo Dec 16, 2022
5ec53f7
Merge branch 'main' into iterative-os-walk
jonburdo Dec 16, 2022
507b650
switch back to list instead of deque for os.walk stack
jonburdo Dec 16, 2022
1c35610
Apply suggestions from code review
jonburdo Dec 18, 2022
73138a6
clean up os.walk comments
jonburdo Dec 18, 2022
2814cc5
test correct value in test_walk_above_recursion_limit
jonburdo Dec 18, 2022
37f3cc7
set islink and join before loop in os.walk
jonburdo Dec 18, 2022
d9c766c
Merge branch 'main' into iterative-os-walk
jonburdo Dec 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
clean up os.walk comments
  • Loading branch information
@jonburdo
jonburdo committed Dec 18, 2022
commit 73138a6363e5b8276e10d4639e66788bcd7323f4
18 changes: 9 additions & 9 deletions Lib/os.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -354,7 +354,7 @@ def walk(top, topdown=True, onerror=None, followlinks=False):

# We may not have read permission for top, in which case we can't
# get a list of the files the directory contains.
# We suppressed the exception here, rather than blow up for a
# We suppress the exception here, rather than blow up for a
# minor reason when (say) a thousand readable directories are still
# left to visit.
try:
Expand All @@ -381,8 +381,8 @@ def walk(top, topdown=True, onerror=None, followlinks=False):
try:
is_dir = entry.is_dir()
except OSError:
# If is_dir() raises an OSError, consider that the entry is not
# a directory, same behaviour as os.path.isdir().
# If is_dir() raises an OSError, consider the entry not to
# be a directory, same behaviour as os.path.isdir().
is_dir = False

if is_dir:
Expand All @@ -391,17 +391,17 @@ def walk(top, topdown=True, onerror=None, followlinks=False):
nondirs.append(entry.name)

if not topdown and is_dir:
# Bottom-up: traverse into sub-directory, but exclude symlinks to
# directories if followlinks is False
# Bottom-up: traverse into sub-directory, but exclude
# symlinks to directories if followlinks is False
if followlinks:
walk_into = True
else:
try:
is_symlink = entry.is_symlink()
except OSError:
# If is_symlink() raises an OSError, consider that the
# entry is not a symbolic link, same behaviour than
# os.path.islink().
# If is_symlink() raises an OSError, consider the
# entry not to be a symbolic link, same behaviour
# as os.path.islink().
is_symlink = False
walk_into = not is_symlink

Expand All @@ -410,8 +410,8 @@ def walk(top, topdown=True, onerror=None, followlinks=False):
if cont:
continue

# Yield before sub-directory traversal if going top down
if topdown:
# Yield before sub-directory traversal if going top down
yield top, dirs, nondirs
# Traverse into sub-directories
islink, join = path.islink, path.join
Copy link
Copy Markdown
Member

@JelleZijlstra JelleZijlstra Dec 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're caching these functions in locals, might as well do it outside the main while loop to micro-optimize a bit more.

Expand Down