Skip to content

fix(core): make cuda.core.system actually fall back when NVML is unimportable - #2574

Open
LeSingh1 wants to merge 1 commit into
NVIDIA:mainfrom
LeSingh1:system-nvml-fallback
Open

fix(core): make cuda.core.system actually fall back when NVML is unimportable#2574
LeSingh1 wants to merge 1 commit into
NVIDIA:mainfrom
LeSingh1:system-nvml-fallback

Conversation

@LeSingh1

@LeSingh1 LeSingh1 commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Problem

cuda_core/cuda/core/system/_system.pyx opens with the contract it is meant to satisfy:

# This file needs to either use NVML exclusively, or when `cuda.bindings.nvml`
# isn't available, fall back to non-NVML-based methods for backward
# compatibility.

The code that is supposed to implement that does neither:

if CUDA_BINDINGS_NVML_IS_COMPATIBLE:
    try:
        from cuda.bindings import nvml
    except ImportError:
        CUDA_BINDINGS_NVML_IS_COMPATIBLE = False

    from cuda.core.system._nvml_context import initialize
else:
    from cuda.core._utils.cuda_utils import driver, handle_return, runtime

The except clears CUDA_BINDINGS_NVML_IS_COMPATIBLE, but the else hangs off the outer if, which has already been evaluated. So on the one path that clears the flag, the non-NVML names are never bound — while every consumer in the file keys off the now-False flag and reaches for exactly those names:

consumer line falls back to
get_user_mode_driver_version() 74-75 handle_return(driver.cuDriverGetVersion())
get_num_devices() 139 handle_return(runtime.cudaGetDeviceCount())

Both would raise NameError. In practice the module never reaches them: from cuda.core.system._nvml_context import initialize on the next line runs unconditionally, and _nvml_context.pyx:7 is itself from cuda.bindings import nvml — so the very ImportError the try/except exists to absorb is re-raised one statement later and import cuda.core.system fails outright.

cuda.core.system.__init__ is built around the flag being usable in this state (elif CUDA_BINDINGS_NVML_IS_COMPATIBLE: gates the NVML-only submodules), and cuda.core._device reads it at _device.pyx:1053, so the degraded mode is a supported configuration — it just cannot be reached.

Fix

Move the _nvml_context import into the same try (it depends on nvml, so it belongs there), and replace the else with a second if not CUDA_BINDINGS_NVML_IS_COMPATIBLE: so a flag cleared by the failed import selects the fallback.

Tests

The branch is import-time, so it can only be exercised in a fresh interpreter. test_system_falls_back_when_nvml_is_unimportable runs a subprocess that installs a sys.meta_path finder raising ImportError for cuda.bindings.nvml, then imports cuda.core.system and asserts:

  • the import succeeds (it used to raise),
  • CUDA_BINDINGS_NVML_IS_COMPATIBLE is False,
  • driver, handle_return, runtime are bound on cuda.core.system._system.

It does not call into the driver, so it needs no GPU.

What I ran

Environment: macOS, no CUDA driver and no CUDA toolkit, so cuda.core cannot be built or imported here.

  • Did not run: the new test, or anything else under cuda_core/tests/ — they need a built cuda.core.
  • Ran: a reduction of the import block with two stand-in modules (repro_nvml, unimportable; repro_nvml_context, which imports it), keeping the control flow verbatim:
before: import of the module FAILED -> ImportError: libnvidia-ml not present in this build
after:  CUDA_BINDINGS_NVML_IS_COMPATIBLE=False, non-NVML fallback bound=True
  • Ran: the sys.meta_path blocker used by the new test, standalone against a stdlib submodule, to confirm it turns both from X import Y and import X.Y into ImportError.
  • Ran: python -m py_compile on the changed test file and compile() on the embedded subprocess script (both parse), plus ruff check / ruff format --check. ruff check reports the same single pre-existing I001 on this file as it does on main; no new findings.
  • Checked: every reader of CUDA_BINDINGS_NVML_IS_COMPATIBLE in the tree (_system.pyx, system/__init__.py, _device.pyx:1053, and four test modules) — none of them depend on the else form.

…portable

The file's own header states the contract: use NVML exclusively, or when
`cuda.bindings.nvml` is not available fall back to non-NVML methods. The code
does neither:

    if CUDA_BINDINGS_NVML_IS_COMPATIBLE:
        try:
            from cuda.bindings import nvml
        except ImportError:
            CUDA_BINDINGS_NVML_IS_COMPATIBLE = False

        from cuda.core.system._nvml_context import initialize
    else:
        from cuda.core._utils.cuda_utils import driver, handle_return, runtime

The `except` clears the flag, but the `else` belongs to the outer `if`,
which has already been evaluated -- so the fallback names are never bound on
the path that clears the flag. Every consumer keys off the now-False flag and
reaches for exactly those names:

    get_user_mode_driver_version()  ->  handle_return(driver.cuDriverGetVersion())
    get_num_devices()               ->  handle_return(runtime.cudaGetDeviceCount())

both of which would raise `NameError`. In practice the module never gets that
far: `_nvml_context` is imported unconditionally on the next line and
`_nvml_context.pyx` starts with `from cuda.bindings import nvml`, so the
ImportError the try/except exists to absorb is re-raised out of
`import cuda.core.system` a statement later.

Move the `_nvml_context` import inside the same `try` and turn the `else`
into a second `if not ...` so a cleared flag selects the fallback.
@copy-pr-bot

copy-pr-bot Bot commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the cuda.core Everything related to the cuda.core module label Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cuda.core Everything related to the cuda.core module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant