fix(core): credit an entry that vanishes at unlink time during cache eviction - #2580
Open
LeSingh1 wants to merge 1 commit into
Open
fix(core): credit an entry that vanishes at unlink time during cache eviction#2580LeSingh1 wants to merge 1 commit into
LeSingh1 wants to merge 1 commit into
Conversation
`_enforce_size_cap` walks its snapshot and subtracts each evicted entry's
size from `total` until it is back under the cap. It handles "the file is
already gone" twice, inconsistently:
try:
stat_now = path.stat()
except FileNotFoundError:
total -= size # credited
continue
...
try:
_unlink_with_sharing_retry(path)
total -= size
except FileNotFoundError:
pass # NOT credited
Both are the same condition -- another process removed the entry -- observed a
few microseconds apart, and the bytes are off disk either way. Not crediting
the second one leaves `total` above the cap, so the pass evicts a second,
live entry that did not need to go, and then reseeds
`self._tracked_size_bytes = total` with that same overcount, which makes the
next write trip the cap early and over-evict again.
This is a documented, expected race for this backend: the class is designed
for multi-process use, and a concurrent `__delitem__` or another process's
eviction pass hits exactly this window.
The `PermissionError` branch is now an explicit `continue`: an exhausted
Windows sharing-violation retry leaves the file on disk, so its bytes must
stay in `total`. That was already the behaviour; making it explicit keeps
the single `total -= size` at the end honest.
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
FileStreamProgramCache._enforce_size_capwalks its snapshot oldest-atime-first and subtracts each evicted entry's size fromtotaluntil it is back under the cap. It handles "the file is already gone" twice, inconsistently (_file_stream.py:740-761):Both branches are the same condition — another process removed the entry — observed a few microseconds apart, and the bytes are off disk either way. This backend is explicitly designed for multi-process use, and a concurrent
__delitem__or another process's eviction pass lands in exactly this window.Not crediting the second one has two compounding effects:
totalstays above the cap, so the pass evicts a second, live entry that did not need to go — a needless cache miss and recompile.self._tracked_size_bytes = totalwith that same overcount, so the next write trips the cap early and over-evicts again.Measured on the real module (three 30-byte entries, a 100-byte cap, one racing unlink): 2 entries survive instead of 3, and the tracker reports 90 bytes against 60 on disk.
Fix
Move the single
total -= sizepast the handler so theFileNotFoundErrorbranch reaches it, matching the stat-miss branch above. ThePermissionErrorbranch becomes an explicitcontinue: an exhausted Windows sharing-violation retry leaves the file on disk, so its bytes must stay intotal. That was already the behaviour — making it explicit is what keeps the singletotal -= sizeat the end honest.No behaviour change on the non-racing path.
Tests
test_filestream_size_cap_credits_an_entry_that_vanished_at_unlinkmonkeypatches_file_stream._unlink_with_sharing_retryso the pass's first victim is unlinked by "another process" in the window between our stat-guard and our own unlink — the same injection style as the existingtest_filestream_cache_tracker_clamps_at_zero_under_delete_race. It asserts all three remaining entries survive and that the tracker agrees with_compute_total_size().What I ran
Environment: macOS, no CUDA driver and no CUDA toolkit, so
cuda.corecannot be built or imported here.cuda_core/tests/test_program_cache.pyitself — it importscuda.core.utils._file_stream.py's onlycuda.coredependency isObjectCode, used for anisinstancecheck in_extract_bytes, so I loaded the real module withcuda.core._modulestubbed and ran the new test's exact assertions against it:main:FAIL: len(cache) == 2, expected 3PASS_tracked_size_bytesto_compute_total_size()after each trial — no drift before or after the change, confirming the single-process accounting is untouched.ruff checkandruff format --checkon both changed files — clean, no new findings against amainbaseline._enforce_size_capoutcomes are unaffected — the stat-guard mismatch branch still does its owntotal += stat_now.st_size - sizeandcontinue, and the Windows sharing-violation branch still leavestotalalone.Overlap note: #2553 also touches
_file_stream.py, but only__init__(argument validation); the two changes are in different methods.