Skip to content

gh-89727: Fix os.walk RecursionError on deep trees #99803

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JelleZijlstra merged 16 commits into from
Dec 19, 2022
Merged

gh-89727: Fix os.walk RecursionError on deep trees #99803

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
99f4691
gh-89727: Fix os.walk RecursionError on deep trees
jonburdo Nov 26, 2022
3552d49
add unit test to ensure os.walk is not affect by recursion limit
jonburdo Dec 16, 2022
3078ea6
move os._walk code to os.walk
jonburdo Dec 16, 2022
f37bbe8
fix blurb to mention os.walk
jonburdo Dec 16, 2022
f26a5b8
use deque in os.walk
jonburdo Dec 16, 2022
955d21b
Merge branch 'main' into iterative-os-walk
jonburdo Dec 16, 2022
8fbfb2e
Merge branch 'main' into iterative-os-walk
jonburdo Dec 16, 2022
ef46eda
move deque import inside os.walk
jonburdo Dec 16, 2022
e261b9f
rename temp_recursion_limit to set_recursion_limit
jonburdo Dec 16, 2022
5ec53f7
Merge branch 'main' into iterative-os-walk
jonburdo Dec 16, 2022
507b650
switch back to list instead of deque for os.walk stack
jonburdo Dec 16, 2022
1c35610
Apply suggestions from code review
jonburdo Dec 18, 2022
73138a6
clean up os.walk comments
jonburdo Dec 18, 2022
2814cc5
test correct value in test_walk_above_recursion_limit
jonburdo Dec 18, 2022
37f3cc7
set islink and join before loop in os.walk
jonburdo Dec 18, 2022
d9c766c
Merge branch 'main' into iterative-os-walk
jonburdo Dec 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
test correct value in test_walk_above_recursion_limit
  • Loading branch information
@jonburdo
jonburdo committed Dec 18, 2022
commit 2814cc5f728a054fe463dd8aa65ccfd0b14c6f74
40 changes: 37 additions & 3 deletions Lib/test/test_os.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1473,10 +1473,44 @@ def test_walk_many_open_files(self):
p = os.path.join(p, 'd')

def test_walk_above_recursion_limit(self):
os.makedirs(os.path.join(self.walk_path, *(['d'] * 50)))
with set_recursion_limit(50):
depth = 50
os.makedirs(os.path.join(self.walk_path, *(['d'] * depth)))
with set_recursion_limit(depth - 5):
all = list(self.walk(self.walk_path))
self.assertEqual(len(all), 54)

sub2_path = self.sub2_tree[0]
for root, dirs, files in all:
if root == sub2_path:
dirs.sort()
files.sort()

d_entries = []
d_path = self.walk_path
for _ in range(depth):
d_path = os.path.join(d_path, "d")
d_entries.append((d_path, ["d"], []))
d_entries[-1][1].clear()

# Sub-sequences where the order is known
sections = {
"SUB1": [
(self.sub1_path, ["SUB11"], ["tmp2"]),
(self.sub11_path, [], []),
],
"SUB2": [self.sub2_tree],
"d": d_entries,
}

# The ordering of sub-dirs is arbitrary but determines the order in
# which sub-sequences appear
dirs = all[0][1]
expected = [(self.walk_path, dirs, ["tmp1"])]
for d in dirs:
expected.extend(sections[d])

self.assertEqual(len(all), depth + 4)
self.assertEqual(sorted(dirs), ["SUB1", "SUB2", "d"])
self.assertEqual(all, expected)


@unittest.skipUnless(hasattr(os, 'fwalk'), "Test needs os.fwalk()")
Expand Down