Skip to content

fix(coderd/x/chatd): stream compaction summaries - #28886

Merged
ibetitsmike merged 8 commits into
mainfrom
mike/chatd-stream-compaction-summary
Sep 21, 2026
Merged

ibetitsmike merged 8 commits into
mainfrom
mike/chatd-stream-compaction-summary

Conversation

@ibetitsmike

Copy link
Copy Markdown
Collaborator

Chats could get permanently stuck once their context crossed the compaction threshold on adaptive-thinking Claude models: every compaction attempt failed with compaction produced an empty summary, and both automatic and manual compaction take the same path, so the chat could never recover.

Problem

compactionSummaryCall deliberately set MaxOutputTokens = nil because the summary request was non-streaming and the Anthropic SDK rejects non-streaming requests whose max_tokens implies a completion longer than ten minutes. But a nil cap makes fantasy's Anthropic provider silently default to 4096 output tokens, and adaptive-thinking Claude models can spend that entire budget on a thinking block, leaving zero summary text.

Fix

  • Stream the compaction summary request. Streaming lifts the Anthropic SDK's non-streaming duration restriction, which was the only reason the cap was nil'd.
  • Keep the resolved output cap (32k default) on the summary call instead of nil'ing it.
  • When the summary comes back empty with truncation evidence (length finish reason, or reasoning with no text), fail with compaction summary was truncated at the output token cap instead of the generic empty-summary error, so the next occurrence is attributable at a glance.

Unit tests cover text-block joining and the truncation classification; the compaction e2e fixtures now serve streamed summaries and assert the summary request streams with max_tokens: 32000. Dogfood UAT on a deployment built from this branch verified manual compaction with Claude Opus 5 (adaptive thinking, twice, with reload persistence) and an OpenAI regression chat.

🤖 Opened by Xum on behalf of @ibetitsmike.

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 1, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-20T12:32:36.279284Z a7e875d Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ae69401255

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chatloop/compaction.go Outdated
Comment thread coderd/x/chatd/chatloop/compaction.go Outdated
Comment thread coderd/x/chatd/chatloop/compaction.go Outdated

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: bcfe45dd52

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Require a finish part before accepting a streamed compaction summary and route the summary stream through the stream-silence guard so a silent provider fails fast with a classified retryable error.
@ibetitsmike
ibetitsmike force-pushed the mike/chatd-stream-compaction-summary branch from bcfe45d to f6496b1 Compare September 8, 2026 18:59
@ibetitsmike
ibetitsmike marked this pull request as ready for review September 9, 2026 14:43

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f6496b12f8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chatloop/compaction.go Outdated
Comment thread coderd/x/chatd/chatloop/compaction.go Outdated
…ary cap

Wrap compaction summary stream errors with the same provider stream
classification GenerateAssistant uses, so remote stream resets surfaced
as bare context.Canceled retry instead of terminally erroring the chat.
Double the summary call's output cap so a low configured chat cap does
not truncate the summary that replaces pruned history.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 30ae3fc134

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/modelcall.go Outdated
Comment thread coderd/x/chatd/chatloop/compaction.go Outdated
@chrisbalmer

Copy link
Copy Markdown

Looks like this resolves an issue I hit as well with local models. The 600s fixed timeout in model.Generate when compacting a large chat with a dense model would trigger a hard failure. Switching to Stream should fix my issue with compaction as it allows slow responses to continue without timing out as chunks come in.

…t window

Sum-enforcing providers reject requests whose input plus max_tokens
exceeds the context window, so the doubled summary cap could make
compaction itself fail on large windows at default thresholds. Clamp
the cap to the remaining window at trigger time; degenerate cases
(unknown limit, usage at or over the limit) leave the cap unchanged.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d52f350a1a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chatloop/compaction.go Outdated
Comment thread coderd/x/chatd/modelcall.go Outdated
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 726961fce4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chatloop/compaction.go Outdated
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: b606d66b31

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…paction-summary

# Conflicts:
#	coderd/x/chatd/chatd_test.go
#	coderd/x/chatd/chatloop/compaction.go
#	coderd/x/chatd/chatloop/compaction_internal_test.go
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: a7e875d601

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike merged commit e35cbe2 into main Sep 21, 2026
29 checks passed
@ibetitsmike
ibetitsmike deleted the mike/chatd-stream-compaction-summary branch September 21, 2026 19:28
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 21, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants