feat(computer): the disk form on Modal VM sandboxes - #468

FD-4371 (M4.2): a computer's Linux disk form boots on a Modal VM sandbox under nested virtualization, carried from this machine's local stores, as M4.5's exit (FD-4379) needs. It implements the seven decisions in the design doc.

What changes

• D1, the sandbox. With a disk as image, modal_vm.create_vm makes a VM sandbox (runtime="vm", enable_nested_virt) whose own image is the QEMU launcher, ENTRYPOINT cleared, with no Docker in it. It's sized at the VM's CPUs and its memory plus 2 GiB, so the default 2 CPUs and 4096 MB give a 2-CPU, 6 GiB sandbox.
  • The launcher image is the published one when the sources are published (pulled by digest), and otherwise Modal builds it from its folder.
  • disk_image_platforms() is {linux/amd64}.
  • The launcher gets VM_REQUIRE_KVM=1, so it fails clearly without /dev/kvm rather than emulate. Local Docker sets it too. The release smoke leaves it unset, so arm64 still boots there under TCG.
• D2, carrying the disk. local_docker.disk.exported_disk exports disk.img from the image store with this machine's Docker, once per composed disk ref (whose tag is the composer's key). It's zstd-compressed with the launcher's zstd and kept under config.paths.cache_root()/disks.
  • The provider mounts the export on the launcher image with add_local_file(..., copy=False). Modal uploads each content once.
  • The launcher (VM_DISK=/disk/disk.img.zst) expands it onto the sandbox's disk.
  • No grant and no write_file_from_object are involved.
• D3, networking. The launcher serves each VM_PORTS port with socat TCP-LISTEN:<port>,fork,reuseaddr,backlog=1024 in front of slirp's hostfwd on 127.0.0.1:<port+10000>. socat is in the launcher image.
  • socat's own connection to that forward times out at 0.5 s and retries every 50 ms, up to 600 times, so a connection the forward drops doesn't wait out the kernel's SYN backoff (8ea287c; see the larger bursts below).
  • The launcher refuses a port whose forward would land above 65535 (that is, a port above 55535) or on another of its ports.
  • It stops the computer and exits 1 when a proxy exits, whenever that is. It watches QEMU and the proxies every 0.2 s, since bash's wait -n skips a job that ended before it was called (found in review).
  • This is the same on every provider.
• D4, region. [sandbox.providers.modal_vm.config] region (unset by default) applies to every sandbox the provider makes: the computer's, the agent's and the gateway's.
• D5, reset in place. The launcher, now bash, runs QEMU in a loop.
  • launch reset (the new environment as VM_ENV_* lines on stdin) stops QEMU and returns once its successor has started on a fresh overlay. The ports stay up, so the URL is the same.
  • reset_vm runs it in the sandbox hosting the launcher: Modal's exec, and docker exec -i on local Docker, which no longer recreates its container.
  • A QEMU exit that no reset asked for still ends the launcher with QEMU's status.
  • The launcher's work folder moved from RAM-backed /run/vm to /var/lib/agentenv-vm.
• D6, gate 3. A probe that ends on a transport error (httpx's: dropped, refused or timed out, alone or in a task group) runs again on a fresh boot, up to 3 in all. The record keeps every attempt (attempts). A failed check, a boot that never got ready, and the probe's own timeout are recorded at once.
• D7, where it composes. Composing.for_run(provider) picks the forms on the run's sandbox (deploys_on) and builds them on the env's default provider, which is this machine unless the config names another.
  • ComputerEnv.compose_unstarted builds there too, never on the deploy's sandbox, and says where ("composing disk/amd64 before this deploy on LocalSandboxProvider (no compose built it for ModalVmSandboxProvider yet)").
  • The M1.9 compose provider (put / env build --sandbox) is unchanged.

Three additions beyond the seven decisions, for review

1. The form pick knows where composed containers live (boot.native_links / first_native take the env's store id; boot.pulls_containers).
   • Without this, --sandbox modal_vm picks the container form for a bench task, because modal_vm runs amd64 containers natively and the container comes first in the base's order. A Modal VM then can't pull the container from this machine's localhost:5000.
   • Now a sandbox elsewhere boots a container only when the env's image store isn't on this machine's loopback, and a disk wherever it boots one, since it carries the disk itself.
   • So with the default local stores a run on modal_vm composes and boots the disk, and with registered stores the container is still picked first.
   • This is also FD-4379's preflight blocker: deploy_refusal allows the disk form on such a sandbox. For a container-only base it now says why: "…can't pull its containers from the image store's registry on this machine".
2. [sandbox.providers.modal_vm.config] idle_timeout (seconds, unset by default) for every sandbox the provider makes.
   • A disk's sandbox always gets one: the configured value, or 30 minutes, since a running QEMU isn't activity, so without one a computer nobody uses lasts its whole timeout. agent-env run --keep's help now says a provider's idle timeout still ends a sandbox nothing is connected to (e45cf46).
   • The e2e runs below set 1200 so the agent's and grader's sandboxes met Scale's Modal Budgets rule too. Without it they get no idle timeout, as on main.
3. A run preflights each step on its --sandbox (33507b5, bundle/materialize.py).
   • Writing a bundle's tasks checked each deploy step's own refusal on the provider the step names, the config's default, though the run deploys it on --sandbox. The run-wide preflight (preflight_run) already used --sandbox.
   • So on a host whose Docker boots no disk (no KVM), a run of a disk-form task with --sandbox modal_vm was refused before it started: "LocalSandboxProvider runs none of its arches (amd64, arm64) natively as disk". The amd64 EC2 boxes have KVM, which hid it. The arm64 check below found it.
   • Now each step's sandbox_type and judge_sandbox_type are checked as the run's sandbox, as sandbox_overrides deploys them.

Smaller changes:

• composer.REPOSITORY moved to composed.py, so boot can read it without an import cycle.
• The release's launcher smoke also checks socat and zstd.
• computers/README.md, sandbox_providers/AGENTS.md, agent-env run --sandbox's help and run_bundle's docstring say all of the above.
• region and idle_timeout sit in the provider's free-form config table, which config/describe.py doesn't descend into (as for vercel's region), so it needs no new row.
• The plugin surface is unchanged: SandboxProvider gains nothing, and modal_vm's constructor only gains optional keys.

The end-to-end bar (Edgar's condition for D3)

Run on EC2 (c8i.4xlarge with nested KVM, earakelyan-m42-e2e), which acted as the laptop: default local stores, this branch merged with main at 0.9.1426 (with #464), [sandbox.providers.modal_vm.config] region = "us-west", and every deploy through the framework (ComputerEnv.deploy(sandbox_type=…, env_params={"form": "disk"})).

Probes at full, 20 boots each. Every boot was a fresh deploy and teardown.

There were no transport errors (dropped or refused connections, timeouts) and no failed checks, P4/P6 included. Before merging #464 the same 80 boots passed too.

Latency in ms (p50 / p95 / p99). New connections are each on a fresh HTTPS (Modal) or HTTP (local) connection to the driver's card, 40 a boot one at a time and 40 a boot in bursts of 8; tool calls are 20 a boot over an MCP session. n is 800 a cell for connections and 400 for tool calls.

Larger bursts: one desktop boot each, 3 rounds, every connection with a 150 s timeout. Each cell gives p50 / p95 / p99 / max in ms for the worst round.

No connection was dropped anywhere. But on a GitHub runner (4 vCPUs), one of 64 at once waited past 30 s in launcher_test.py.

The cause: socat takes the burst, but each of its forked children then connects to slirp's forward inside the container, whose listen backlog is 1. A connect that overflows it waited for the kernel's SYN retry, at 1, 3, 7, 15 and 31 s, so the wait grew with the burst and the host's load. That was the 1 s tail on local Docker in the bursts of 8 above (about 1 in 20 connections), and how a starved runner reached 30 s. Modal's tunnel spreads the arrivals, so its tail stayed under 0.7 s at 8 at once.

The fix, 8ea287c: socat's onward connect is TCP:127.0.0.1:<port+10000>,connect-timeout=0.5,retry=600,interval=0.05, so socat retries a dropped connect itself every 50 ms. Measured on earakelyan-m42-e2e with the desktop disk on local Docker: 2 rounds of each burst, idle and with all 16 CPUs busy, a 60 s client timeout. Each cell gives p50 / p95 / max in seconds for the worst round.

Every connection passed after the fix. launcher_test.py is back at a burst of 64. On the same VM it passed in 88 s idle and in 87 s with 16 CPU hogs. The bursts-of-8 and 16/64 tables above were measured before 8ea287c. Raising libslirp's backlog upstream would still remove the retries.

A real G02 task end to end: agent-env run ~/bench --task cdb-g02-foreground --sandbox modal_vm --model anthropic/claude-haiku-5-5, in 321 s.

• Compose: the run composed the three computers on this machine (D7) and picked the disk on modal_vm.
• Deploy: the computer (disk form), the grader VM and the agent VM all ran on modal_vm. The computer deployed in 15.3 s, including the export and the first upload of the 1.7 GB disk.
• Play: Haiku 5.5 made 60 tool calls and stopped at the task's 45-turn cap, which the task tolerates (fail_task_on_error: false).
• Grade: trycua's evaluator gave 0.8 (6 of 7 checks; workbook_agreement failed), the sentinel check passed, and GUI-only scored 1.0.
• Teardown: all 3 sandboxes.
• Recording: refused, as expected with local stores ("the object store issues no upload grant that reaches … modal_vm"). That step is non-fatal.

The agent's own tool calls, timed from its trajectory (tool_use event to tool_result event), include each tool's work. run_steps batches actions and waits, and get_window_state reads the accessibility tree.

After the review fixes (9c61d1d, the launcher's watch loop), the recheck on Modal passed: tst/integration/computer/disk --task probe (probe, a reset in 8.3 s, probe), and 5 of 5 G02 probes at full, with the same latency.

Reset in place, in a run:

• tst/integration/computer/disk --task probe on modal_vm: probe passed, then reset_env took 7.6 s, then the probe passed again at the same URL.
• A G02 task (deploy desktop-s02, probe, reset_env, probe, then the bench's check_app): the reset took 12.5 s and both probes passed. check_app found every app pristine, placed and relaunched after the reset.

Rechecked on Modal after the socat fix (8ea287c), on a fresh earakelyan-m42-recheck (c8i.4xlarge) with this machine's stores, us-west:

• disk_modal_vm_test.py: passed (286 s). It exports once, boots, probes, resets in place and probes, and a second run exports nothing.
• agent-env run ~/bench --task app-s02 --sandbox modal_vm: it composed the three computers here, deployed desktop-s02's disk on modal_vm in 44 s (the export and first upload included), and check_app passed.
• 5 G02 probes at full: 5/5 passed. Latency matched the 20-boot run above:

• Bursts of 64 new connections to G02's driver, through Modal's tunnel, before, during and after two resets in place. The "during" burst was fired a second into the reset, while QEMU was down. Every one of the 5 × 64 connections answered 200, and none failed.

A connection that arrives while QEMU's forward is down now waits in socat, retrying every 50 ms for up to 30 s, and is served once the computer is back. Before 8ea287c it was refused at once.

From an arm64 host, as an Apple-silicon laptop runs it: a fresh earakelyan-m42-arm (Graviton c7g.4xlarge, Docker 29.9.3 with the containerd image store, no KVM, amd64 under binfmt emulation).

• The first run was refused before it started: that's addition 3's bug.
• With 33507b5, agent-env run tst/integration/computer/disk --task probe --sandbox modal_vm passed in 120 s.
  • It used the desktop composed here: container/amd64 emulated in 6 min, disk/amd64 left unverified, since no link here boots it.
  • The export from this arm64 host's Docker needed no --platform.
  • The deploy on modal_vm took 83.5 s, then probe, a reset in 7.8 s, and probe again.

Tests

• Unit, with a fake Modal and the launcher's tools stood in for:
  • a run preflighting each step's own refusal on its --sandbox (preflight_test.py);
  • the launcher's front proxy and its bounded connect, its two refusals, a proxy that exits, KVM required, a compressed disk, and reset in place (the real launch script);
  • the local reset through launch reset, and the export once for each disk;
  • modal_vm's disk create (launcher image, mount, seed and ports, size, idle timeout), region and idle timeout on every sandbox, reset, disk_image_platforms, the launcher image, and every create argument bound to the installed Modal's signature;
  • the config keys;
  • gate 3 retrying a dropped probe but not a failed check;
  • a run on a VM provider composing with this machine's Docker;
  • a deploy's compose on this machine;
  • a sandbox elsewhere picking the disk, and the refusal for a container-only base.
• Integration:
  • tst/integration/computer/disk_modal_vm_test.py, gated on remote_sandbox: the desktop disk carried, booted, probed, reset and probed on Modal, and a second run that exports nothing.
  • launcher_test.py, gated on kvm: a burst of 64 new connections through the proxy, and a reset in place on local Docker.
  • disk_boot_test.py, updated for the deploy's compose line.
• Runs on fresh agent-environments-admin EC2 VMs with nested KVM:
  • On earakelyan-m42-tests (c8i.2xlarge), without Modal keys, as CI runs:
    • make unit-test at the head: 12,796 passed, 7 skipped.
    • make int-test-fast at 9c61d1d: 180 passed.
    • make int-test-slow, serially at the merge with main (34b02c7): 172 passed and 825 skipped, the skips being capability gaps (remote providers, model endpoint). One failed: disk_boot_test expected "making the disk", while an earlier module in the same serial run had already made that disk under its key. Its run itself passed (probe, a reset in 6.3 s, probe). 97a18c6 accepts a reused disk.
    • disk_boot_test.py and launcher_test.py at 9c61d1d: 2 passed.
  • On earakelyan-m42-e2e (c8i.4xlarge), with the CI project's Modal keys: disk_modal_vm_test.py passed (2 runs, 142 s).

Not in this PR

• A Modal Volume for disks (FD-5118).
• Pruning disk exports and the image store's composed images (FD-5163).
• Composing in a Modal VM (M4.9, FD-5051).
• noVNC's port.
• Recordings and other grants from local stores to Modal.
• The idle heartbeat (Can Ersoz).
• arm64 disks on Modal.

For the website docs: --sandbox modal_vm for computer envs; the disk export in ~/.cache/agentenv/disks (prunable by hand until FD-5163); region and idle_timeout; and that the idle timeout ends a computer with no connection open.

🤖 Generated with Claude Code

Confidence Score: 5/5

Summary

Adds disk boots on Modal VM sandboxes, compressed disk exports, and resets that keep the same URLs.

• Modal VM sandboxes boot computer disks and reset them in place.
• Runs choose computer forms the destination can access and build them on the environment's provider.
• Computer build probes retry when the driver's connection drops.
• Modal VM sandboxes can use a configured region and idle timeout.

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[Compose disk on default provider] --> B[Export and compress disk locally]
    B --> C[Mount export on Modal VM sandbox]
    C --> D[Launcher expands disk]
    D --> E[QEMU boots fresh overlay]
    F[Client] --> G[socat proxy]
    G --> E
    H[Reset with new environment] --> I[Replace QEMU and overlay]
    I --> E

_{Reviews (6) · Last reviewed commit: "fix(bundle): a run preflights each step ..." · Reviewed by Greptile}