fix: keep chat token usage for streams ending with usage-less chunks - #28068
Open
ibetitsmike wants to merge 1 commit into
Open
fix: keep chat token usage for streams ending with usage-less chunks#28068ibetitsmike wants to merge 1 commit into
ibetitsmike wants to merge 1 commit into
Conversation
…chunks Picks up coder/fantasy#52: OpenAI-compatible backends that report cumulative usage on delta chunks and end with a usage-less finish chunk (observed with Poolside laguna-xs when tools are declared) no longer have their usage zeroed at Finish. Restores per-message token accounting, the context gauge, and automatic compaction for such models.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumps the coder/fantasy fork pin to pick up coder/fantasy#52.
Problem
Chats on
poolside/laguna-xs-2.1showed no context usage: every assistant message persisted NULL token columns, and automatic compaction never triggered, so chats ran to context overflow. AIBridge recorded correct usage for the same requests, so the loss was client-side in fantasy.When tools are declared, laguna-xs reports cumulative usage on every delta chunk and ends the stream with a
finish_reasonchunk whoseusageis null, with no trailing usage-only chunk. Fantasy's chat-completions stream loops reassigned usage from every chunk, and the default stream usage hook returns zero usage for usage-less chunks, so the trailing finish chunk wiped the real usage one chunk earlier. The Finish part then reportedUsage{0,0,0}, which chatd persists as NULL (nullInt64IfNonZero).Fix
coder/fantasy#52 adopts the stream usage hook's result only when the chunk actually carries usage, in both the chat-completions stream loop and the JSON-mode object stream loop. This mirrors the aibridge fix in #27967, which is why the gateway recorded usage correctly while fantasy lost it. Spec-compliant backends that emit usage once on the final chunk are unaffected.
This PR pins the fork at the merged commit and documents the fork-only patch in the go.mod comment block.
Validation
go build ./...,go vet ./coderd/x/chatd/..., andgo test ./coderd/x/chatd/...(including the chatdebug field-coverage guard) all pass with the bumped pin.