vmm: sanitize GPUs via VFIO hot reset instead of sysfs SBR - #1058
vmm: sanitize GPUs via VFIO hot reset instead of sysfs SBR#1058Leechael wants to merge 2 commits into
Conversation
The sanitize-on-attach path issued the Secondary Bus Reset by writing Bridge Control in the upstream bridge sysfs config space and re-probed devices through /sys/bus/pci/drivers_probe. Both files are writable by root only, so the feature could not be enabled in production where dstack-vmm runs as an unprivileged user with no sudo. Switch to the VFIO_DEVICE_PCI_HOT_RESET ioctl, which makes the kernel perform the same bus reset. The ioctl is authorized by device ownership rather than privilege: the caller presents fds for every VFIO group affected by the reset, and the /dev/vfio group nodes are the same ones QEMU opens to attach the GPU, so the VMM user already has access. A single group fd suffices because every sanitized GPU sits alone behind a dedicated PCIe bridge and alone in its IOMMU group. The bridge topology check is kept as defense, and the kernel-reported set of affected devices must all belong to the GPU own group or the launch is aborted. Devices stay bound to vfio-pci across the reset, so the drivers_probe re-probe logic is no longer needed and is removed. Not yet validated on GPU hardware; see plans/2026-08-14-vfio-gpu-hot-reset.md for the pending experiment.
Expose the sanitize path as "dstack-vmm sanitize-gpu <slot>..." so operators can reset GPUs by hand and the pending hardware experiment can exercise exactly the code path used at VM launch, running as the unprivileged VMM user. The subcommand needs no server configuration, only /dev/vfio access, and is handled before config loading like the other special modes.
|
Overall, switching from privileged sysfs writes to the VFIO hot-reset API is the right direction. I have two implementation suggestions:
We have created a temporary fork that adds the missing APIs: I suggest using this fork temporarily, validating the complete reset flow on the target GPU hardware, and then submitting the APIs upstream to This should substantially reduce the amount of unsafe VFIO ABI code maintained in dstack and give the hot-reset implementation a more appropriate reusable home. |
Summary
VFIO_DEVICE_PCI_HOT_RESETioctl so it works with the VMM running as an unprivileged user/sys/bus/pci/drivers_probere-probe; devices stay bound to vfio-pci across the resetdstack-vmm sanitize-gpu <slot>...subcommand for manually resetting GPUs through the exact launch-time code pathMotivation
#1048 cannot be enabled in a common operations setup:
dstack-vmmrunning as a dedicated unprivileged user without sudo. It issues the Secondary Bus Reset by writing Bridge Control in the upstream bridge's sysfs config space and re-probes devices through/sys/bus/pci/drivers_probe, and both files are writable by root only.VFIO_DEVICE_PCI_HOT_RESETmakes the kernel perform the same Secondary Bus Reset on the parent bridge, but it is authorized by device ownership instead of privilege: the caller presents an fd for every VFIO group affected by the reset. The/dev/vfiogroup nodes are the same ones QEMU opens to attach the GPU, so the VMM user already has access to them. This is also the mechanism QEMU itself uses to bus-reset devices without a usable FLR.A single group fd suffices because every sanitized GPU sits alone behind a dedicated PCIe bridge and alone in its IOMMU group. This matches the typical passthrough host setup: H200-class GPUs behind dedicated downstream ports, one device per IOMMU group, vfio-pci bound at boot, and the VMM user granted
/dev/vfioaccess via the same group membership QEMU requires. A read-only survey of production H200 hosts confirmed this topology and access model.The privilege constraints and topology preconditions are documented at the top of
gpu_reset.rs.Safety
VFIO_DEVICE_PCI_GET_HOT_RESET_INFOoutput is validated before resetting: any affected device outside the GPU's own IOMMU group aborts the launch, and the affected set is loggedTests
cargo check -p dstack-vmm;cargo clippycleancargo test -p dstack-vmm(117 passed)pci_bridge_secondary_bus_resetto confirm the kernel executes the SBR on the expected bridge, config-space polling for the link-down window during reset, an A/B comparison against the sysfs method, and a reproduction of the original SPDM-timeout incident.sanitize_on_attachstays config-gated until then, anddstack-vmm sanitize-gpuruns the same path standalone for the experiment.