feat(coderd/x/chatd/chatloop): retain user constraints in compaction summaries - #27230
Merged
Conversation
…summaries Compaction summaries drop or soften user-stated constraints, corrections, and prohibitions, so post-compaction assistants repeat behavior the user already corrected. Add a summary prompt bullet that instructs the summarizer to quote them, treat them as standing until revoked, and attribute rules to their true source instead of defaulting to the user. Validated offline on unseen human chats: holdout P1 delta +0.175 (arbitrated), 13/18 cases improve. This improves per-compaction retention only; it does not address deep-chain correction loss.
DanielleMaywood
approved these changes
Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a chat grows past its context budget, chatd compacts older history into a summary produced by
defaultCompactionSummaryPrompt(coderd/x/chatd/chatloop). In production transcripts these summaries often drop or soften constraints the user stated ("do not push and do not create a PR" surviving only as "no push"), so after compaction the assistant repeats behavior the user already corrected. This PR adds one bullet to the prompt telling the summarizer to keep user constraints, corrections, and prohibitions in the user's wording, treat them as standing until revoked, record corrections to the assistant's behavior rather than only the corrected outcome, and label rules by their true source instead of attributing config-file rules to the user.Improves per-compaction retention, validated on unseen human chats (holdout P1 +0.175, arbitrated); does not and cannot fix deep-chain correction loss.
How the result was measured
37 real compaction events from production chats were replayed offline: for each event, the recorded history up to the compaction point was summarized twice, once with the new prompt and once with the current one, and both summaries were scored 0 to 1 on a pre-registered checklist of retention properties. Scoring is LLM-based with controls: a fresh judge per event, a second cross-family judge on a sample, a third model arbitrating disagreements (arbitrated numbers are primary), and an audit pass that excludes claims a summary invented. The pass/fail rules were fixed before the run.
Two metrics appear below:
Two test sets: 18 events from human chats never used while developing the prompt (holdout), and 19 events from one long chat, cfeca7ba, that was compacted many times and is the known worst case for corrections getting lost across compaction rounds.
Unarbitrated scores agree in direction: holdout P1 +0.1599 (PASS), cfeca7ba 0.3638 vs 0.3808 (FAIL).
Impact and scope
For human chats, a single compaction now retains materially more of what the user constrained and corrected. The deep-chain criterion failed: on the repeatedly compacted chat, the new prompt did not improve correction survival. The operator explicitly waived that criterion and scoped the claim accordingly. The limitation is structural: each compaction summarizes the previous summary, so anything one round drops is unrecoverable in later rounds; no wording change to a single-round summary prompt can fix that. Deep-chain loss stays an open problem.
The retention gain costs summary length: holdout summaries average 11% more tokens (ratio 1.106).
Supporting context for reviewers
Judging pipeline
Primary judge: one fresh single-event agent per (event, summary), because a judge scoring events sequentially accumulates context and measurably degrades (55.8% vs 77.9% weakened-constraint detection on identical events). Secondary judge from a different model family on a 20% sample plus all high-stakes runs; a third model arbitrates disagreement cells. A separate assertion audit filters phantom claims (content the summary invented) before scoring. A control smoke test passed 10/10. The regression clause is binomial because roughly 16% of paired comparisons exceed 1 sd downward under pure noise, so a "no case regresses" clause can never pass; the gate instead requires the regression count not to significantly exceed chance.
Generalization: the holdout is all human-user chats with no sub-agent events, so the validation supports human chats only.
Holdout per-case results (arbitrated)
New prompt is internally called variant h001b; "default" is the current prompt.
P3 is attribution quality: rules labeled by origin (user-stated, agent-decided, agent-inferred), guarded so the constraint-retention gain does not come from attributing everything to the user.
cfeca7ba deep-chain results (arbitrated)
Per-compaction correction-capture (P10) mean delta over the 19 chain events is -0.031 (0.4365 vs 0.4675), so the new prompt does not improve retention on this chat and the criterion fails. Arbitration widened the primary-only gap (-0.017 to -0.031), so the earlier apparent flip was judge noise. The only chain-level evidence available is a one-hop substitution result (candidate config 0.450 vs production 0.356 pooled constraint survival), which cannot speak to full chain dynamics; those are only measurable live.
Process note: pre-push hook bypass
The local pre-push hook was bypassed once (
git push --no-verify) with explicit operator authorization. Grounds:make test-storybookfails on the base commit 55b06f1 itself in the dev environment (three AgentsPage stories, verified on a clean detached checkout), the diff is 14 added lines in a Go string constant that does not touch site/, and GitHub check runs on the base sha report no failures (7 success, 23 skipped).