fix: make _EvalMetricResultWithInvocation.expected_invocation Optional for conversation_scenario support#5215
Open
ASRagab wants to merge 1 commit intogoogle:mainfrom
Open
Conversation
When using conversation_scenario for user simulation, expected_invocation is None because conversations are dynamically generated. The public model EvalMetricResultPerInvocation already types this as Optional[Invocation], but the private _EvalMetricResultWithInvocation requires non-None, causing a pydantic ValidationError during post-processing. - Make expected_invocation Optional[Invocation] = None - Guard attribute accesses in _print_details to handle None - Fall back to actual_invocation.user_content for the prompt column Fixes google#5214
Collaborator
|
Response from ADK Triaging Agent Hello @ASRagab, thank you for submitting this pull request! To help the reviewers, could you please add a Including the logs or a screenshot showing that the You can find more details in our contribution guidelines. Thanks! |
kylegallatin
reviewed
Apr 8, 2026
|
|
||
| actual_invocation: Invocation | ||
| expected_invocation: Invocation | ||
| expected_invocation: Optional[Invocation] = None |
There was a problem hiding this comment.
Suggested change
| expected_invocation: Optional[Invocation] = None | |
| expected_invocation: Invocation | None = None |
PEP!
Author
There was a problem hiding this comment.
Google does not use this convention, in this repo.
kylegallatin
approved these changes
Apr 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_EvalMetricResultWithInvocation.expected_invocationis typed asInvocation(required), butlocal_eval_service.py:285-287intentionally sets it toNonewheneval_case.conversationisNone(i.e.,conversation_scenariouser-simulation cases)EvalMetricResultPerInvocationineval_metrics.py:323already types this field asOptional[Invocation] = NoneValidationErrorduring post-processing in_get_eval_metric_results_with_invocation, after all metrics have been computedChanges
expected_invocationOptional[Invocation] = Nonein_EvalMetricResultWithInvocation_print_detailsto handleNone(fall back toactual_invocation.user_contentfor the prompt column,Nonefor expected response/tool calls)_convert_content_to_textand_convert_tool_calls_to_textalready acceptOptionalparametersTesting Plan
Verified with a pytest-based evaluation using
AgentEvaluator.evaluate()against an evalset containingconversation_scenariocases (LLM-backed user simulation, no explicitconversationarrays).Before fix — crashes after ~33 minutes of metric computation during post-processing:
After fix — the
ValidationErroris eliminated. TheNoneexpected_invocation flows through correctly because:Optional[Invocation], matching the upstreamEvalMetricResultPerInvocationmodel_print_detailsgracefully handlesNoneby falling back toactual_invocation.user_contentfor the prompt column and passingNoneto_convert_content_to_text/_convert_tool_calls_to_text(both already acceptOptionalinputs)Reproduction evalset (any evalset with
conversation_scenariotriggers this):{ "eval_set_id": "test", "eval_cases": [{ "eval_id": "scenario_1", "conversation_scenario": { "starting_prompt": "Hello", "conversation_plan": "Ask the agent a question and accept the answer." }, "session_input": {"app_name": "my_agent", "user_id": "user1", "state": {}} }] }Fixes #5214