"cuda.core tests: reserve driver-managed mempools up front" - #2522
Open
juenglin wants to merge 2 commits into
Open
"cuda.core tests: reserve driver-managed mempools up front"#2522juenglin wants to merge 2 commits into
juenglin wants to merge 2 commits into
Conversation
Contributor
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Contributor
Author
|
/ok to test |
Andy-Jost
approved these changes
Aug 6, 2026
juenglin
marked this pull request as ready for review
August 6, 2026 20:07
…sion start The driver's default memory pool and graph memory pool each permanently reserve roughly twice the installed device memory of virtual address space, and neither can be capped or released. On a large-memory GPU with a bounded per-process address space, whether the second reservation still finds a contiguous range depends on how fragmented the space has become by the time some test needs it, which is what makes the failures in NVIDIA#2381 intermittent. Measurements and documentation references: NVIDIA#2381 (comment) Take both reservations up front in a session-scoped autouse fixture so they land back to back in a nearly empty address space. This does not reduce the footprint; it makes the outcome deterministic. If either is refused the session aborts with an explanation, since every later test needing that pool would fail the same way and a cascade of identical OOM errors says nothing about the cause. Available address space is reported before and after via cuMemAddressReserve rather than an OS query, so the measurement behaves the same on Windows and Linux. CUDA_CORE_TEST_SKIP_EARLY_RESERVATION=1 disables the fixture; CUDA_CORE_TEST_FORCE_RESERVATION_FAILURE=1 exercises the abort path.
…orted Both driver-managed pools require mempool support, so on a device without it there is nothing to reserve and nothing to pre-empt. Reporting a failed reservation there would abort the session, when the correct behaviour is to carry on: the tests that need pools already skip themselves via Device.properties.memory_pools_supported, and the rest still run.
juenglin
force-pushed
the
oom-early-reservation-workaround
branch
from
August 6, 2026 20:09
6971e4e to
51f6ff4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #2381
Root cause and measurements are in this comment; in short, the driver's default memory pool and graph memory pool each permanently reserve ~2x installed device memory of virtual address space, and neither can be capped or released.
A session-scoped autouse fixture now takes both reservations up front, so they land back to back in a nearly empty address space instead of whenever test order happens to trigger them. This does not reduce the footprint; it just makes the big reservations deterministic. If either reservation is refused the session aborts with an explanation, rather than cascading ~194 identical OOM failures that say nothing about the cause.
Available address space is reported before and after via
cuMemAddressReserverather than an OS query, so it behaves the same on Windows and Linux. Devices without mempool support are skipped, not failed. Escape hatches:CUDA_CORE_TEST_SKIP_EARLY_RESERVATION=1disables the fixture (use as the control when measuring whether it helps or is still needed).The fixture can be removed once either driver-side improvement lands:
(
cuGraphAddMemAllocNodecurrently acceptspoolProps.maxSizebut ignores it); orfixture's own
remaining headroom: N more pool-sized reservationsline reports —an N comfortably above 1 means ordering can no longer matter on that machine.
Checklist