Skip to content

fix: make _EvalMetricResultWithInvocation.expected_invocation Optional for conversation_scenario support#5215

Open
ASRagab wants to merge 1 commit intogoogle:mainfrom
ASRagab:fix/optional-expected-invocation
Open

fix: make _EvalMetricResultWithInvocation.expected_invocation Optional for conversation_scenario support#5215
ASRagab wants to merge 1 commit intogoogle:mainfrom
ASRagab:fix/optional-expected-invocation

Conversation

@ASRagab
Copy link
Copy Markdown

@ASRagab ASRagab commented Apr 8, 2026

Summary

  • _EvalMetricResultWithInvocation.expected_invocation is typed as Invocation (required), but local_eval_service.py:285-287 intentionally sets it to None when eval_case.conversation is None (i.e., conversation_scenario user-simulation cases)
  • The public model EvalMetricResultPerInvocation in eval_metrics.py:323 already types this field as Optional[Invocation] = None
  • This mismatch causes a pydantic ValidationError during post-processing in _get_eval_metric_results_with_invocation, after all metrics have been computed

Changes

  • Make expected_invocation Optional[Invocation] = None in _EvalMetricResultWithInvocation
  • Guard the three attribute accesses in _print_details to handle None (fall back to actual_invocation.user_content for the prompt column, None for expected response/tool calls)
  • Both _convert_content_to_text and _convert_tool_calls_to_text already accept Optional parameters

Testing Plan

Verified with a pytest-based evaluation using AgentEvaluator.evaluate() against an evalset containing conversation_scenario cases (LLM-backed user simulation, no explicit conversation arrays).

Before fix — crashes after ~33 minutes of metric computation during post-processing:

pydantic_core._pydantic_core.ValidationError: 1 validation error for _EvalMetricResultWithInvocation
expected_invocation
  Input should be a valid dictionary or instance of Invocation [type=model_type, input_value=None, input_type=NoneType]

.venv/lib/python3.11/site-packages/google/adk/evaluation/agent_evaluator.py:639: ValidationError

After fix — the ValidationError is eliminated. The None expected_invocation flows through correctly because:

  1. The field now accepts Optional[Invocation], matching the upstream EvalMetricResultPerInvocation model
  2. _print_details gracefully handles None by falling back to actual_invocation.user_content for the prompt column and passing None to _convert_content_to_text/_convert_tool_calls_to_text (both already accept Optional inputs)

Reproduction evalset (any evalset with conversation_scenario triggers this):

{
  "eval_set_id": "test",
  "eval_cases": [{
    "eval_id": "scenario_1",
    "conversation_scenario": {
      "starting_prompt": "Hello",
      "conversation_plan": "Ask the agent a question and accept the answer."
    },
    "session_input": {"app_name": "my_agent", "user_id": "user1", "state": {}}
  }]
}
@pytest.mark.asyncio
async def test_scenario():
    await AgentEvaluator.evaluate("my_agent", "path/to/evalset.json", num_runs=1)

Fixes #5214

When using conversation_scenario for user simulation, expected_invocation
is None because conversations are dynamically generated. The public model
EvalMetricResultPerInvocation already types this as Optional[Invocation],
but the private _EvalMetricResultWithInvocation requires non-None, causing
a pydantic ValidationError during post-processing.

- Make expected_invocation Optional[Invocation] = None
- Guard attribute accesses in _print_details to handle None
- Fall back to actual_invocation.user_content for the prompt column

Fixes google#5214
@adk-bot adk-bot added the eval [Component] This issue is related to evaluation label Apr 8, 2026
@adk-bot
Copy link
Copy Markdown
Collaborator

adk-bot commented Apr 8, 2026

Response from ADK Triaging Agent

Hello @ASRagab, thank you for submitting this pull request!

To help the reviewers, could you please add a testing plan section to your PR description explaining how you verified the fix? For example, did you run the evaluation with a conversation_scenario?

Including the logs or a screenshot showing that the ValidationError is gone after your change would also be very helpful.

You can find more details in our contribution guidelines. Thanks!


actual_invocation: Invocation
expected_invocation: Invocation
expected_invocation: Optional[Invocation] = None
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
expected_invocation: Optional[Invocation] = None
expected_invocation: Invocation | None = None

PEP!

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Google does not use this convention, in this repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

eval [Component] This issue is related to evaluation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AgentEvaluator crashes with ValidationError when evaluating conversation_scenario eval cases

3 participants