Fix block device resize disk by vincent-thomas · Pull Request #7948 · cloud-hypervisor/cloud-hypervisor

vincent-thomas · 2026-04-01T19:08:01Z

As it currently is, CH doesn't support resizing disks of VMs whose storage is backed by a host block device. This PR addresses this and prevents (but doesn't fix) a deadlock of the CH process (which in turn causes VM cpu to freeze) from happening when doing so.

Previous discussion and further details: #7923

Fixes #7923

tpressure · 2026-04-01T20:32:38Z


-        let nsectors = new_size / SECTOR_SIZE;
-
-        self.common.pause().map_err(Error::PauseVcpus)?;


don't we need this for proper synchronization during disk-resize?

Also, PauseVcpus is wrong here (a mistake I made). We should correct this and make it PauseVirtioThreads or similar.

pause and resume actually caused a deadlock of the cloud-hypervisor process and froze the whole VM indefinitely.

this should not happen. Does it only deadlock when you use block devices? For me, using a raw file with the original code works perfectly fine.

It does not deadlock when using a block device on the host with "fast" I/O (for example losetup).
It does deadlock when using a host block device on the host with "slow", network-bound I/O (for example ceph; rbd mapped block device). Both scenarios happen 100% of the time when manually testing.

The removal of the pause/resume came from experimentation, which turned out to fix this issue and work well.

If the VM was paused, and we can ensure that it says paused during the resize call, I would suggest that we only then skip pausing/resuming the virtio-queues. @phip1611 @vincent-thomas

@phip1611 you mix up vcpu pausing and virtqueu pausing ;)

@tpressure @phip1611 Have a look at my recently updated diff, It is changed to fix the root cause.

only then skip pausing/resuming the virtio-queues.

I agree - looks like @vincent-thomas did the right thing now. LGTM on a first glance

I'll ask for another review since it sounds like this conversation has cleared up maintainers worries.

phip1611 · 2026-04-09T10:03:04Z

please fix CI

phip1611

The changes are good, thanks!

It looks to me that your second commit changes lines you introduced in the first commit. We value clean git histories.

Please bring your commits into clean structure, then we can move this forward.

Also see https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/CONTRIBUTING.md#patch-format--git-commit-hygiene

vincent-thomas · 2026-04-09T15:59:57Z

Please bring your commits into clean structure

Sorry about that, should be okay now.

phip1611 · 2026-04-09T18:38:25Z

Please run cargo +nightly fmt --all to fix your formatting.

The commit message check can be ignored (optional CI step). You could properly fix it by using the scheme:

See [0].

[0] https://...

phip1611 · 2026-04-10T09:23:02Z

        );
-        self.paused.store(true, Ordering::SeqCst);
+
+        // If already paused, return early to avoid deadlock waiting on barrier


This situation occurs when the VMM thread holds a device mutex while
calling an operation that triggers pause(), and a vCPU thread
simultaneously needs that same mutex for MMIO access. With slow I/O
backends (like RBD/Ceph), the timing window for this race is larger,
making the deadlock more likely to occur.

Ah, wait. Doesn't this open another race window? Another thread may unpause this too early? Or are we protected by mutexes?

Maybe this is another place where we need a three-way handshake with atomic

RUNNING -> PAUSING -> PAUSED

It might not strictly needed in this case since we don't really do anything after.

rbradford

Your race commit fix deserves its own PR as I think it is independent of the resize fix?

vincent-thomas · 2026-04-16T14:01:55Z

Your race commit fix deserves its own PR as I think it is independent of the resize fix?

Yeah sure but if the resize fix lands and the race commit doesn't, then CH will deadlock on any resize-disk API call where:

VM has been paused before the resize disk call.
The VM is backed by a slow host block device.

rbradford · 2026-04-16T17:13:04Z

The ordering of the commits in the PR is probably wrong then - but I asked you to split the race fix out as it's an obviously good fix.

Previously, calling pause() when already paused would wait on a barrier for worker threads that were already parked, causing a deadlock. This situation occurs when the VMM thread holds a device mutex while calling an operation that triggers pause(), and a vCPU thread simultaneously needs that same mutex for MMIO access. With slow I/O backends (like RBD/Ceph), the timing window for this race is larger, making the deadlock more likely to occur, see [0]. Make pause() idempotent by checking the paused state atomically and returning early if already paused, avoiding the barrier wait. [0] #7948 (comment) Signed-off-by: Vincent Thomas <vincent@v-thomas.com>

rbradford · 2026-04-16T17:16:58Z

I flipped the order of your commits.

PR was cleaned up

rbradford

Still need cargo fmt

rbradford · 2026-04-16T17:21:28Z

        );
-        self.paused.store(true, Ordering::SeqCst);
+
+        // If already paused, return early to avoid deadlock waiting on barrier


Maybe this is another place where we need a three-way handshake with atomic

RUNNING -> PAUSING -> PAUSED

It might not strictly needed in this case since we don't really do anything after.

Block devices (LVM volumes, loop devices, RBD, etc.) cannot be resized via ftruncate - they are resized externally. When vm.resize-disk is called for a block device backend, verify the device size matches the requested size instead of attempting ftruncate. This enables the resize-disk API to work with block device backends by validating the externally-resized device matches the expected size. Signed-off-by: Vincent Thomas <vincent@v-thomas.com>

vincent-thomas · 2026-04-17T07:02:37Z

Still need cargo fmt

Sorry about this, ran it a couple of times I thought.... Should be fixed now.

Previously, calling pause() when already paused would wait on a barrier for worker threads that were already parked, causing a deadlock. This situation occurs when the VMM thread holds a device mutex while calling an operation that triggers pause(), and a vCPU thread simultaneously needs that same mutex for MMIO access. With slow I/O backends (like RBD/Ceph), the timing window for this race is larger, making the deadlock more likely to occur, see [0]. Make pause() idempotent by checking the paused state atomically and returning early if already paused, avoiding the barrier wait. [0] #7948 (comment) Signed-off-by: Vincent Thomas <vincent@v-thomas.com>

vincent-thomas requested a review from a team as a code owner April 1, 2026 19:08

vincent-thomas force-pushed the fix-block-device-resize-disk branch from ecbc12e to 51a33af Compare April 1, 2026 19:22

tpressure reviewed Apr 1, 2026

View reviewed changes

Comment thread block/src/raw_async.rs Outdated

tpressure reviewed Apr 1, 2026

View reviewed changes

weltling reviewed Apr 1, 2026

View reviewed changes

Comment thread block/src/raw_async.rs Outdated

weltling reviewed Apr 1, 2026

View reviewed changes

Comment thread block/src/raw_async.rs Outdated

weltling reviewed Apr 1, 2026

View reviewed changes

Comment thread block/src/raw_async.rs Outdated

vincent-thomas force-pushed the fix-block-device-resize-disk branch 3 times, most recently from bbc9d98 to 603b73b Compare April 8, 2026 10:40

vincent-thomas requested a review from phip1611 April 9, 2026 09:46

phip1611 previously requested changes Apr 9, 2026

View reviewed changes

Comment thread block/src/raw_async.rs

tpressure approved these changes Apr 9, 2026

View reviewed changes

vincent-thomas force-pushed the fix-block-device-resize-disk branch from 603b73b to 75256f5 Compare April 9, 2026 14:46

vincent-thomas requested a review from phip1611 April 9, 2026 16:00

phip1611 reviewed Apr 10, 2026

View reviewed changes

vincent-thomas force-pushed the fix-block-device-resize-disk branch 2 times, most recently from 1bd5f9d to 5bace4c Compare April 16, 2026 10:03

rbradford requested changes Apr 16, 2026

View reviewed changes

rbradford force-pushed the fix-block-device-resize-disk branch from 5bace4c to c2ec811 Compare April 16, 2026 17:16

rbradford approved these changes Apr 16, 2026

View reviewed changes

rbradford enabled auto-merge April 16, 2026 17:17

rbradford requested a review from phip1611 April 16, 2026 17:17

rbradford disabled auto-merge April 16, 2026 17:21

rbradford requested changes Apr 16, 2026

View reviewed changes

rbradford mentioned this pull request Apr 16, 2026

Add three stage atomic pause flow #8047

Open

vincent-thomas force-pushed the fix-block-device-resize-disk branch from c2ec811 to e91273d Compare April 17, 2026 07:01

rbradford approved these changes Apr 17, 2026

View reviewed changes

rbradford enabled auto-merge April 17, 2026 07:09

rbradford added this pull request to the merge queue Apr 17, 2026

Merged via the queue into cloud-hypervisor:main with commit fd8ded9 Apr 17, 2026
38 checks passed

vincent-thomas deleted the fix-block-device-resize-disk branch April 17, 2026 12:56


		let nsectors = new_size / SECTOR_SIZE;

		self.common.pause().map_err(Error::PauseVcpus)?;

Conversation

vincent-thomas commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vincent-thomas Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vincent-thomas Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phip1611 commented Apr 9, 2026

Uh oh!

phip1611 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vincent-thomas commented Apr 9, 2026

Uh oh!

phip1611 commented Apr 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

vincent-thomas commented Apr 16, 2026

Uh oh!

rbradford commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rbradford commented Apr 16, 2026

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vincent-thomas commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vincent-thomas commented Apr 1, 2026 •

edited

Loading

vincent-thomas Apr 2, 2026 •

edited

Loading

vincent-thomas Apr 8, 2026 •

edited

Loading

phip1611 left a comment •

edited

Loading

rbradford commented Apr 16, 2026 •

edited

Loading

vincent-thomas commented Apr 17, 2026 •

edited

Loading