Repository navigation
fix(models): stop running Claude tool calls cut off by max_tokens - #7558
donggyun112 wants to merge 2 commits into
Conversation
When max_tokens cut a response off inside a tool_use block, the non-streaming path returned the partial input as a function call, so the tool ran with arguments Claude never finished, and the streaming path raised ValueError parsing the incomplete JSON. Report such a response as a MAX_TOKENS error with no tool call, on both paths, as LiteLlm does for a truncated tool call.
There was a problem hiding this comment.
Code Review
This pull request introduces handling for scenarios where an Anthropic LLM tool call is truncated due to reaching the max_tokens limit. It adds a helper function _truncated_tool_use_response to generate an error response with a MAX_TOKENS finish reason and an informative error message. This logic is integrated into both the standard response mapping (message_to_generate_content_response) and the streaming response generator (_generate_content_streaming). Additionally, corresponding unit tests have been added to verify this behavior for both streaming and non-streaming responses. I have no further feedback to provide as the implementation is clean and well-tested.
Somay-kousis
left a comment
There was a problem hiding this comment.
max_tokens isn't the only stop reason that truncates a tool_use. Claude can also stop with model_context_window_exceeded, which the anthropic SDK (1.14.0) lists in StopReason and documents as "we exceeded the model's context window". On this branch that case still runs the tool with the cut-off input (non-streaming) and still raises the JSON ValueError (streaming). Both new checks compare stop_reason == "max_tokens" only.
Runner + LlmAgent(model=AnthropicLlm(...)) with a write_file tool and a scripted client returning a cut-off tool_use, on this branch (a7a7992):
stop_reason=max_tokens tool ran with=[] error_codes=['MAX_TOKENS']
stop_reason=model_context_window_exceeded tool ran with=['def main():\n retu'] error_codes=[]
Streaming, same events as test_streaming_tool_use_cut_off_by_max_tokens_is_reported with only the stop reason changed:
stream stop_reason=max_tokens last error_code=MAX_TOKENS
stream stop_reason=model_context_window_exceeded raised ValueError: Invalid JSON: Unterminated string starting at: line 1 column 31 (char 30)
Suggestion: check against a small set, e.g. _TRUNCATING_STOP_REASONS = frozenset({"max_tokens", "model_context_window_exceeded"}), in both message_to_generate_content_response and _generate_content_streaming, and parametrize the two new tests over it. #7584 maps the same stop reason to MAX_TOKENS, so the two PRs line up.
Checked: the two new tests fail on the base's anthropic_llm.py (2 failed, 183 passed) and pass on the branch (185 passed).
…_use Claude can also stop with model_context_window_exceeded mid tool_use, which left the same incomplete input. Check both stop reasons on both paths.
|
@Somay-kousis Good catch, thanks. Both paths now check |
When Claude's response is cut off by
max_tokensormodel_context_window_exceededinside atool_useblock,message_to_generate_content_responsereturns the incomplete input as a function call, so the tool runs with arguments Claude never finished. The streaming path instead raisesValueErrorparsing the incomplete JSON. Anthropic's docs say to retry with a highermax_tokensin this case.When the last block is a
tool_useandstop_reasonis either of these, report aMAX_TOKENSerror with no tool call on both paths, asLiteLlmdoes for a truncated tool call.Testing Plan
Unit tests
One non-streaming and one streaming test check that a truncated
tool_useends in aMAX_TOKENSerror with no content. On main they fail with a returned function call and aValueError.Manual E2E
A Runner with a
write_filetool and a scripted client that returns atool_usecut off bymax_tokens. On mainwrite_fileruns withcontent="def main():\n retu"; with this change it doesn't run and the event carrieserror_code=MAX_TOKENS.