A/B Coder Agents between a branch and main using Harbor and Terminal-Bench.

• harbor_coder.environment:CoderEnvironment: Harbor environment provider that runs each trial in a Coder workspace (task image plus the coder binary, created from template/, exec over coder ssh).
• harbor_coder.agent:CoderAgentsAgent: Harbor agent that binds a chat to that workspace, waits for the turn to finish, and records tokens and cost.
• harbor_coder.bootstrap: first user, AI provider (dogfood's gateway), chat model, template push on a fresh coderd.
• harbor_coder.compare: pass rate per task and cost for two job directories.

Comment /coder-agents-eval on a PR to run .github/workflows/eval-coder-agents.yaml: each arm builds its own coderd (main vs the PR), runs terminal-bench@2.0 (74 of 89 tasks: 3 are broken upstream, 9 are 20 to 60 minute ML and emulation tasks the prompt cannot affect, and 3 security tasks are always refused by Anthropic's content policy) with Coder Agents on claude-sonnet-5-5, and the comparison is posted on the PR. The baseline is the commit the PR branched from, not the main tip, so later main changes are not attributed to the PR. The system prompt is Go code, so a prompt change is a branch and the unit of comparison is a build. The verdict calls a change better or worse only when a paired bootstrap 95% interval over tasks excludes zero.

Measured on the 77-task set (before the 3 refused tasks were dropped) at concurrency 5 (chatd's unlicensed cap on concurrent chats): about $23 and 90 minutes per arm (run). One /coder-agents-eval runs two arms in parallel, so about $47 of gateway spend and two 8-core runners for roughly 1.5h. The full 86 tasks on main scored 73% at two attempts, $31.54 and 3.4h for one arm. Dataset, model, attempts, concurrency, and the Harbor version are the env block at the top of the workflow.

# Installs Harbor into .harbor-coder/venv from requirements.lock with
# --require-hashes, so every dependency (litellm included) is pinned by sha256.
./scripts/harbor-coder/install.sh && export PATH="$PWD/.harbor-coder/venv/bin:$PATH"
# dogfood gateway auth; the gateway ignores token scopes, so keep it narrow
export HARBOR_CODER_UPSTREAM_TOKEN=$(coder tokens create --lifetime 8h --scope user:read_personal)
export HARBOR_CODER_MODEL=claude-sonnet-5-5
./scripts/harbor-coder/serve-coderd.sh .harbor-coder/env
set -a; . .harbor-coder/env; set +a; export PATH="$PWD/.harbor-coder:$PATH"
# add the -x list from HARBOR_EXCLUDE in the workflow to match CI
harbor run -d terminal-bench@2.0 -x 'qemu-*' -x build-cython-ext \
  --agent harbor_coder.agent:CoderAgentsAgent --env harbor_coder.environment:CoderEnvironment \
  -k 2 -n 4 -o jobs --job-name main

Stop the first coderd (kill $(cat .harbor-coder/coderd.pid)), check out the other branch, rerun serve-coderd.sh and Harbor under another --job-name, then python -m harbor_coder.compare --baseline jobs/main --candidate jobs/branch. serve-coderd.sh refuses to start while something answers on its port, and task images are tagged with the coder binary's hash, so the second run cannot silently reuse the first build. --agent oracle validates tasks without spending tokens. Any registry dataset works if its image has bash and tar, runs as root, and does not need no-network or a GPU.