fix(providers): final regenerated stream must not re-call tools (empty chat answers)#5915
Conversation
…ls and clobbering the answer
After the silent tool loop settles, OpenAI Responses and Gemini re-issue a
streaming request purely to stream the answer as prose — but with tools still
attached and auto tool choice, a reasoning model can re-decide to call a tool
there. Streamed calls are never executed on this path, so the run ends with a
dead function call, an empty streamed answer, and the stream callback
clobbering the tool loop's settled text with '' (deployed chat rendered
{"content": ""}).
Force tool_choice 'none' / functionCallingConfig NONE on the regeneration and
keep the tool loop's settled answer whenever the stream ends without text.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
PR SummaryMedium Risk Overview Across Anthropic, Azure OpenAI chat, Bedrock, Cerebras, DeepSeek, Groq, Mistral, OpenRouter, xAI, and OpenAI Responses (shared core), final regeneration now pins OpenAI Responses additionally records executed tools and prepends settled Regression test covers OpenAI: third request uses Reviewed by Cursor Bugbot for commit b6fb82c. Configure here. |
Greptile SummaryThis PR prevents regenerated answer streams from issuing unhandled tool calls.
Confidence Score: 5/5The PR appears safe to merge, with no blocking failures remaining from the eligible follow-up review scope. No blocking failure remains. Important Files Changed
Sequence DiagramsequenceDiagram
participant Client
participant Provider as Provider adapter
participant Model
participant Tool
Client->>Provider: Streamed request with tools
Provider->>Model: Non-streaming tool-loop request
Model-->>Provider: Tool call
Provider->>Tool: Execute call
Tool-->>Provider: Tool result
Provider->>Model: Continue tool loop
Model-->>Provider: Settled answer
Provider-->>Client: Settled tool start/end events
Provider->>Model: Regenerate stream with tool calls disabled
Model-->>Provider: Streamed prose
Provider-->>Client: Answer events
Note over Provider,Client: If regeneration has no text, retain the settled answer
Reviews (2): Last reviewed commit: "fix(providers): apply the regeneration g..." | Re-trigger Greptile |
The silent Responses tool loop has no live stream while tools run, so opted-in consumers saw no tool chips at all for OpenAI. The loop now records each executed call and prepends settled tool_call_start/end pairs (name + status only) to the agent-events stream ahead of the regenerated answer. Runs without a sink never see these events, so legacy output is unchanged.
Audit of every silent tool loop for the same race fixed for OpenAI/Gemini
(final regenerated stream re-calls a tool that is never executed, ending with
an empty answer that clobbers the settled text):
- anthropic (both implementations): tool_choice {type:'none'} on the
regeneration (tools must stay — history carries tool_use blocks) + keep the
settled answer when the stream ends without text
- groq: was re-applying the ORIGINAL tool_choice, so forced-tool runs
re-forced the tool on the regeneration — guaranteed dead call; now 'none'
+ fallback
- deepseek, mistral, cerebras, azure-openai (legacy chat path), openrouter,
xai: 'auto' -> 'none' + fallback
- bedrock: fallback only — Bedrock's ToolChoice has no 'none' and toolConfig
is required when history carries toolUse blocks
Already guarded (no change): meta, sakana, nvidia, vllm, litellm, baseten,
together, fireworks, kimi, zai, ollama.
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit b6fb82c. Configure here.
Summary
{"content": ""}) while thinking streamed fine. Root cause: after the silent tool loop settles, the final regenerated streaming request still carried tools withtool_choice: 'auto'— a reasoning model can re-decide to call a tool there, but streamed calls are never executed on this path, so the stream ends with a deadfunction_call, no text, and the callback clobbers the tool loop's settled answer with''.tool_choice: 'none', and an empty stream keeps the tool loop's settled answer instead of clobbering it (with a warn log).functionCallingConfig: NONE(unless the responseFormat branch already stripped tools), same settled-answer fallback.tool_choice: 'auto'regeneration exists on old staging); reasoning models re-reasoning during the regeneration made it likely enough to hit in practice.tool_call_start/tool_call_endpairs (name + status only) to the agent-events stream ahead of the regenerated answer, so opted-in chat/canvas render chips for OpenAI runs. No live in-progress chips (tools run before any stream exists); legacy runs without a sink are untouched. Docs updated.Test plan
tool_choice: 'none', that settled chip events precede the answer, and that an empty stream preserves the settled answer inexecution.output.contentvitest run providers/openai providers/gemini providers/googlepassingtsc --noEmitclean;agent-stream-docs:checkup to date