feat(isolation): add UID and GID isolation for evaluated agents - #87
feat(isolation): add UID and GID isolation for evaluated agents#87dmorosanu wants to merge 4 commits into
Conversation
Real-container validation: claude and codex armsRan this branch on a native-Linux Docker host (Azure VM) against tasks drawn from runs where the deployed harness had read grading material. Both agent backends, same tasks, same probes. Evidence below is from live containers, not code reading. The launch seams both drop privileges
This matters because the two backends reach the barrier by different routes: claude via Grading material is unreachableProbed inside each live container as the agent identity (
A The two agents behaved very differently, and both results are informativeClaude did go looking, and was stopped. It ran Codex never tried at all. Across 7 tasks and 99 tool calls, zero commands touched any grading-material path. It read Neither result alone is proof. Claude's denials show the barrier holds under pressure; codex's silence shows the layout removes the motive. The deterministic probes above are what actually establish reachability, independent of what any model chose to do. What this does not coverContent staged into the agent's own workspace via Also: Fixes pushed while validating
Suite on Linux after these: 4214 passed, 8 skipped. Remaining failures are environmental on that host (no |
|
Claude finished @dmorosanu's task in 1m 31s —— View job Code Review in Progress
|
There was a problem hiding this comment.
Pull request overview
Introduces a hardened Docker execution boundary for evaluated agents by running them under a dedicated unprivileged UID/GID, projecting plugins through manifest-verified bundles, and adding a protected Unix-socket mock service whose fixtures are not readable by the agent.
Changes:
- Add Linux UID/GID isolation primitives (drop-privilege launchers, workspace ownership transfer, residual process reaping) and enforce compatible images via a Docker label preflight.
- Add protected mock subsystem (fixture-only RPC server + thin client wrappers) and wire it through DockerRunner/Sandbox + task loading.
- Add plugin bundle staging with manifest verification to prevent mounting raw plugin repositories into agent-readable paths.
Reviewed changes
Copilot reviewed 33 out of 33 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_protected_mock.py | Covers protected mock fixture loading, match modes, passthrough caching, and Docker staging behavior. |
| tests/test_plugin_bundle.py | Verifies plugin bundle sanitization, symlink hardening, and DockerRunner path rewriting/mount behavior. |
| tests/test_docker_runner_mounts.py | Updates mount/layout assertions for protected grader roots and agent home mounts. |
| tests/test_docker_identity_isolation.py | Drift guards for UID/GID constants, labels, capability-clearing launchers, and env scrubbing. |
| tests/test_docker_build_failure.py | Adjusts build-failure test to pass isolation compatibility gating. |
| tests/test_codex_agent.py | Stabilizes Codex env assertions under new env-scrubbing behavior. |
| src/coder_eval/utils.py | Adds agent env scrub allow/deny lists and a helper to generate safe env overrides. |
| src/coder_eval/sandbox.py | Generates protected-mock client wrappers and ensures generated mock dir is added to PATH precedence. |
| src/coder_eval/protected_mock/server.py | Implements mockd server with exact/normalized/subset match modes, budgets, and limited passthrough. |
| src/coder_eval/protected_mock/runtime.py | Adds lifecycle management for mockd subprocess startup/teardown with stderr tailing. |
| src/coder_eval/protected_mock/protocol.py | Defines shared protocol constants and size/time limits. |
| src/coder_eval/protected_mock/client.py | Implements thin agent-visible client that logs invocations and enforces protocol limits. |
| src/coder_eval/protected_mock/init.py | Adds package marker for protected mock subsystem. |
| src/coder_eval/plugin_bundle.py | Adds manifest-verified staging of agent-visible plugin projections with symlink validation. |
| src/coder_eval/orchestrator.py | Re-grants workspace ownership post pre-run and terminates isolated agent processes before finalization. |
| src/coder_eval/orchestration/task_loader.py | Resolves protected mock fixture paths relative to task YAML/experiment directories. |
| src/coder_eval/orchestration/experiment.py | Ensures protected mock fixtures are resolved for experiment-driven tasks. |
| src/coder_eval/models/sandbox.py | Adds agent_isolation default-on and the ProtectedMockConfig model + validations. |
| src/coder_eval/models/container_paths.py | Introduces protected grader root layout, agent/mock identities, and reserved container directories. |
| src/coder_eval/models/init.py | Re-exports new container path and identity constants via coder_eval.models. |
| src/coder_eval/isolation/docker_runner.py | Enforces image capability labels, stages sanitized sources/bundles, and updates mounts/argv for protected layout. |
| src/coder_eval/isolation/agent_identity.py | Adds runtime checks, workspace chowning, and UID-based process termination verification. |
| src/coder_eval/cli/run_task_internal_command.py | Validates protected grader root, grants Claude state to agent UID, and runs mockd during in-container orchestration. |
| src/coder_eval/agents/codex_agent.py | Routes Codex app-server through drop-privilege shim and adjusts HOME/CODEX_HOME behavior under isolation. |
| src/coder_eval/agents/claude_code_agent.py | Scrubs harness-only env vars and routes Claude CLI execution via the isolation shim. |
| src/coder_eval/agents/antigravity_agent.py | Stages a localharness wrapper under drop-privilege policy and scrubs env during serialized spawn. |
| docs/TASK_DEFINITION_GUIDE.md | Documents protected_mocks schema, fixture format, match modes, and passthrough constraints. |
| docs/DOCKER_ISOLATION.md | Documents the UID/GID boundary, protected mount layout, compatibility limits, and runtime-kit constraints. |
| docker/Dockerfile | Adds identities/groups, protected directory layout, setpriv dependency, launch scripts, and capability label. |
| docker/coder_eval_mockd.sh | Launches mockd under the mockd identity with cleared caps and no_new_privs. |
| docker/coder_eval_mock_client | Provides agent-visible client executable entrypoint. |
| docker/coder_eval_drop_privilege.sh | Drops evaluated agent processes to the agent identity with cleared caps/no_new_privs and optional RPC group. |
| docker/coder_eval_claude_agent.sh | Wraps Claude CLI execution through the generic drop-privilege launcher. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| replacements = sorted(self._host_to_private_paths.items(), key=lambda item: len(item[0]), reverse=True) | ||
|
|
||
| def rewrite(value: object) -> object: | ||
| if isinstance(value, str): | ||
| for source, target in replacements: | ||
| value = value.replace(source, target) | ||
| return value | ||
| if isinstance(value, list): | ||
| return [rewrite(item) for item in value] | ||
| if isinstance(value, dict): | ||
| return {key: rewrite(item) for key, item in value.items()} | ||
| return value |
| def _generate_protected_mock_clients(self) -> None: | ||
| """Generate data-free wrappers for fixture-backed mockd tools.""" | ||
|
|
||
| if not self.config.protected_mocks: | ||
| return | ||
| assert self.sandbox_dir is not None, "Sandbox directory not initialized" | ||
|
|
||
| from coder_eval.protected_mock.protocol import CLIENT_EXECUTABLE | ||
|
|
||
| client_dir = self._resolve_within_sandbox(RECORD_CLI_DIR, field="protected mock client directory") | ||
| client_dir.mkdir(parents=True, exist_ok=True) | ||
| log_path = self.sandbox_dir / RECORD_CLI_LOG | ||
| log_path.touch(exist_ok=True) | ||
|
|
||
| for spec in self.config.protected_mocks: | ||
| wrapper = client_dir / spec.tool | ||
| if wrapper.exists(): | ||
| raise RuntimeError( | ||
| f"protected mock client would overwrite {wrapper}; remove the colliding record_cli or mock" | ||
| ) | ||
| wrapper.write_text( | ||
| "#!/bin/sh\n" | ||
| + f"CODER_EVAL_MOCK_CALL_LOG={shlex.quote(str(log_path))} " | ||
| + f'exec {shlex.quote(CLIENT_EXECUTABLE)} {shlex.quote(spec.tool)} "$@"\n', | ||
| encoding="utf-8", | ||
| newline="\n", | ||
| ) |
| if self.rt.task_file: | ||
| host_task_dir = self.rt.task_file.parent.resolve() | ||
| argv += ["-v", f"{host_task_dir}:{host_task_dir}:ro"] | ||
| argv += ["-v", f"{host_task_dir}:{CONTAINER_TASK_DIR}:ro"] |
|
Thanks for the PR. Moving the protected-mock service and plugin-bundle projection to follow-ups makes the UID/GID core much easier to review, and the fail-closed discipline (refuse rather than silently run unprotected) is the right instinct. Before we push this, I'd like to sync, because this PR and #88 close the same leak and I want to make sure we don't lock ourselves out of the harness×model leaderboard. You already document most of the compatibility limits in the body (criteria / Isolation is gated on a closed, per-SDK allowlist, which conflicts with the open Agent SPI. In supported_agents = {AgentKind.CLAUDE_CODE.value, AgentKind.CODEX.value, AgentKind.ANTIGRAVITY.value, AgentKind.NONE.value}
if agent_type not in supported_agents:
raise DockerRunError("… no verified UID-drop launch seam for agent type …")
With
Why I'd like to bring in #88: grade-outside closes the same leak by absence — graders/criteria/reference never enter the container; grading runs host-side over the copied-out workspace. That's harness- and plugin-agnostic — the Delegate SDK works today with zero per-SDK wiring, and any harness we add later inherits leak-closure for free — and it keeps Could we grab 30 min to align the two before either merges? Happy to walk through the grade-outside side. |
1dd19ec to
7a2c59a
Compare
… to normal mode when unsupported
…, harden downgrade and reap paths
…are reference.file cannot corrupt mount_point
What this changes
Run the evaluated agent under a dedicated unprivileged Linux UID/GID inside the
driver: dockercontainer, so the agent identity cannot read the material that grades it (task sources, references, templates, staged inputs, output). The harness and the grader stay root; the agent process is dropped toagent:agent(2000:2000).Isolation is best-effort and on by default (
sandbox.docker.agent_isolation: true). When a prerequisite is missing, the run downgrades to the normal single-identity container with one warning instead of failing.Why
The evaluated agent shares the container with the harness that grades it. Without a second identity, the agent process can read the task sources, references, and expected answers that decide its own score. The boundary keeps grading material out of reach of the agent identity while leaving the agent workspace fully writable.
How it works
Two identities inside one container:
/opt/coder-eval/grader/**at mode0700. Task sources, references, templates, staged input, and the run output directory are all mounted below it./work/agentand/home/agent.The grader is root, so it traverses the
0700tree; the agent UID cannot. The agent launcher drops privileges withsetpriv --reuid=agent --regid=agent --clear-groups --inh-caps=-all --ambient-caps=-all --bounding-set=-all --no-new-privs. The agent's SDK subprocess environment is scrubbed of harness-only variables (scrub_agent_env_overrides): the evaluator credentials, including the Bedrock token, are masked, andHOMEis set to/home/agent.Path rewriting
Grading material is relocated below the root-only grader tree, so the staged
task.yamlcannot keep pointing at agent-readable host paths._prepare_isolated_sourcesbuilds a host-path to container-path map (task dir, template sources, reference file/dir), and_rewrite_task_pathsapplies it to the staged task payload. The map keys on both the resolved and the raw textual form of each source, so a symlinked prefix (macOS/tmpto/private/tmp) still matches; the raw alias is added only when it is itself an absolute path, so a relative form such as.never becomes a rewrite key.SKILLS_REPO_PATH
SKILLS_REPO_PATHforwards into the container like any allowlisted variable: name-only host value in downgraded and plugin-path runs, rewritten to the staged container path when isolation staged the checkout as a template source. The evaluated agent's own process never sees the variable (it is scrubbed to an empty string), so it cannot use it to reach grading material.Grader phase
Dynamic criteria (
run_command,uipath_eval,agent_judge) do not gate isolation. They run in the grader phase, as root, after the agent identity is stopped and reaped.Orchestrator._stop_isolated_agent_processessends SIGTERM then SIGKILL to every UID-2000 process and verifies the UID is empty (zombies are skipped, since a reaped-pending zombie cannot act and SIGKILL cannot remove it). Theagent_judgesub-agent is a trusted grader, not an evaluated agent: it is exempt from the UID drop and keeps evaluator privileges over its root-owned sandbox copy.Best-effort downgrade
A missing prerequisite downgrades the run to the pre-isolation single-identity container and logs one WARNING ("Running WITHOUT agent isolation") naming the reason. The downgrade conditions are:
org.coder-eval.agent-isolation=uid-gid-v1capability label,docker.working_dir,docker.extra_mounts,agent.system_prompt_filethat survived resolution.A downgraded run never sets
CODER_EVAL_AGENT_ISOLATION=1, so every in-container seam stays on the pre-isolation path.agent_isolation: falseturns the boundary off without a warning; it is not a secure evaluation boundary. Behavior is byte-for-byte unchanged from main when no isolation applies.How it starts
Algorithm (one grading turn)
/work/agent; the grader tree is0700root and unreadable to it._stop_isolated_agent_processesreaps residual UID-2000 processes and verifies the UID is empty.Diagrams
Identity boundary:
flowchart TB host["host runner<br/>DockerRunner"] --> container subgraph container["eval container"] direction TB subgraph grader["grader (root)"] gdir["/opt/coder-eval/grader (0700)<br/>task_dir, input, output,<br/>references, templates"] checker["criteria checker<br/>(static + dynamic, after agent reap)"] end subgraph agentzone["agent (UID 2000)"] work["/work/agent"] home["/home/agent"] end grader -- "coder_eval_drop_privilege.sh<br/>setpriv --reuid/--regid 2000" --> agentzone agentzone -. "denied (0700 root-only)" .-> gdir endOne-turn sequence:
sequenceDiagram participant H as Host DockerRunner participant R as In-container root (grader) participant A as Agent UID 2000 H->>R: docker run (CODER_EVAL_AGENT_ISOLATION=1) R->>R: verify 0700 grader root, grant /work/agent R->>A: setpriv drop to agent:agent A->>A: work in /work/agent A--xR: read /opt/coder-eval/grader -> Permission denied A-->>R: turn ends R->>R: reap UID 2000, verify empty R->>R: run criteria (static + dynamic) as root R-->>H: task.json via output mountWhat was tested and how it behaved
Unit and suite (Windows):
ruff format --check,ruff check,pyright0 errors, custom architectural lint, and the full non-live suite pass, coverage above the 80% gate. Only the two known pre-existing Windows symlink-privilege failures remain (a file this branch does not touch).tests/test_docker_identity_isolation.pycovers the downgrade resolution, the image capability check, SKILLS_REPO_PATH forwarding (name-only, unstaged, and rewritten), the POSIX.claudemount target, the symmetric-versus-relocated task_dir mount, the agent_judge isolation exemption, zombie-skip reaping, and the relative-mount_point rewrite guard.End-to-end in a local container engine (podman, on native Linux and via WSL2; image built from this branch, label verified
uid-gid-v1):CODER_EVAL_AGENT_ISOLATION=1and-w /work/agentin argv; arun_commandcriterionid -ureturned0(grader phase is root); no downgrade warningextra_mountspresentCODER_EVAL_AGENT_ISOLATIONin argv; the extra mount was visible; criteria passeduid=2000(agent);cat/ls/findon/opt/coder-eval/graderand the answer-key file returnedPermission denied; the secret string was absent from the agent workspacecatprinted the secret; the same read through the drop-privilege launcher (UID 2000) returnedPermission denied, so the denial is a real boundary, not a missing filerun_commandcriterion runs as root in the grader phase and reads$TASK_DIR/answer_key.txt. This is the trust model, not an agent bypass: a task author who writes a grading command is trusted and their command has grader privileges. No agent-side path reads the answer key.Trust-model note
The boundary is between the evaluated agent and the grading material, not between a task-authored grading command and the grading material. A
run_command/uipath_eval/agent_judgecriterion is authored by the trusted task author and runs with grader (root) privileges. Prefer static built-in criteria where a task allows it.Known limitations
/work/agentand is captured to the host after the container exits, so an abrupt kill (timeout, OOM, host crash) leaves no partial artifacts on the host. Tracked as a follow-up.agent.plugins[].path) renders ahost:containervolume spec whose Windows drive-letter colon collides with docker volume syntax. This is a pre-existing pattern (present on main for the symmetric task_dir mount) and does not affect the Linux deployment target.os.environfor the spawn window; a concurrent task in the same batch process could observe that window. Needs an SDK env seam to fix cleanly. Tracked as a follow-up.Compatibility
extra_mounts: ~/.uipath) therefore keeps running in normal mode until its migration lands.FROM coder-eval-agent:<version>inherit the capability label. Runtime-kit injection into an unrelated base is not yet compatible and downgrades.protected_mocksand the mock service are not part of this PR; they move to a follow-up.