docs(agent-eval): evals must run non-nested + add ab-new-vs-baseline.sh - #734
Merged
Conversation
Running scripts/agent-eval against a `claude -p` spawned from within a Claude Code session (nested, e.g. from a Bash tool call) makes the codegraph MCP attach unreliable: the server is healthy (full handshake ~165ms) but the nested client marks it status:"pending"/0-tools under CPU/timing contention, so the agent silently runs with no codegraph. NO_DAEMON + `< /dev/null` don't fix it — it's the nested client, not the server. Documented in CLAUDE.md's validation methodology. Adds ab-new-vs-baseline.sh: A/Bs a retrieval/steering change as new-build vs baseline-build (both codegraph-on, isolating the change — vs run-all.sh's with-vs-without), on a throwaway copy of an indexed repo. Run it in a real terminal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
colbymchenry
added a commit
that referenced
this pull request
Jun 8, 2026
…ock (#735) Corrects the "run non-nested only" conclusion from #734. The codegraph server is healthy (handshake ~165ms); the flakiness is that on a multi-step implementation task the agent dives into Read/grep before codegraph finishes its ~2-3s startup (worse under nested CPU contention), so it runs with no codegraph. Fix: pre-warm a persistent daemon (high idle timeout) + skip the startup re-exec (CODEGRAPH_WASM_RELAUNCHED=1) so claude connects before the agent's first turn. claude's init snapshot can show status:"pending" even when it then connects — judge by actual codegraph usage, not the init line. ab-new-vs-baseline.sh now bakes in the pre-warm + skip-re-exec. Validated: a clean A/B showed the new build's agent used codegraph 2x / 5 Reads vs the baseline's 0 / 8 on the same fully-implemented task. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents the nested-
claude -pMCP-attach flakiness (the codegraph server is healthy ~165ms; a nested client marks it pending/0-tools under load) and addsab-new-vs-baseline.shto isolate a change (new-build vs baseline-build, both codegraph-on) in a real terminal.🤖 Generated with Claude Code