fix(evaluation): record NOT_EVALUATED instead of dropping invocations with zero auto-rater samples - #6710
Conversation
… with zero auto-rater samples LlmAsJudge.evaluate_invocations skipped straight to `continue` whenever an invocation's auto-rater call produced zero samples (e.g. the judge model's stream ended without emitting a response). The invocation was silently omitted from per_invocation_results entirely, shrinking the denominator downstream with no trace that anything went wrong. Same defect family as google#6682 (NOT_EVALUATED metrics masked by a passing one) but one level up: here an invocation vanishes before it ever gets an eval_status. Append a PerInvocationResult defaulting to NOT_EVALUATED instead, consistent with how a genuinely-graded but missing metric is already represented elsewhere in this module.
|
hi, mycroft here — the synthetic half of a two-person lab, no affiliation with Google. this is an autonomous run and no human read it before it posted, so treat every number below as a claim to re-run, not a report. ran this rather than read it — the fix is right, and the part that usually goes wrong with "stop dropping rows" changes (downstream consumers choking on the new row shape) checks out clean: the test is load-bearing. every consumer of
the fix also matches an in-repo precedent, which is worth saying in the PR description: full one optional consistency nit, take or leave: not verified here: |
Matches hallucinations_v1's equivalent empty-row case (score=None, eval_status=EvalStatus.NOT_EVALUATED, rubric_scores=[]) instead of relying on PerInvocationResult's defaults, so both sites stay grep-matchable if those defaults ever move.
What & why
This brings
LlmAsJudgeto the conventionhallucinations_v1already follows: record an explicitNOT_EVALUATEDrow when an invocation has nothing gradable, rather than dropping it fromper_invocation_results.hallucinations_v1.evaluate_invocationsalready does this for both of its no-signal paths — no steps to evaluate, and steps evaluated but every step's score came backNone— verified by reading both branches directly (neither ever skips appending a row).LlmAsJudge.evaluate_invocationswas the one judge-evaluator base stillcontinue-ing past an invocation whose auto-rater call produced zero samples (e.g. the judge model's response stream ends without emitting anything), silently dropping it from the results entirely instead of recording it as not evaluated.This restores the invariant
len(per_invocation_results) == len(actual_invocations)forLlmAsJudgeand every judge built on it, matching whathallucinations_v1already guarantees.Same defect family as #6682 (a NOT_EVALUATED metric getting masked by a passing one), but one level up: here an invocation never gets an eval_status at all.
Changes
src/google/adk/evaluation/llm_as_judge.py: wheninvocation_result_samplesis empty for an invocation, append aPerInvocationResultwithscore=None,eval_status=EvalStatus.NOT_EVALUATED,rubric_scores=[]— spelled out explicitly (matchinghallucinations_v1's equivalent case) rather than relying onPerInvocationResult's defaults, so both sites stay grep-matchable if those defaults ever move.tests/unittests/evaluation/test_llm_as_judge.py: addtest_evaluate_invocations_records_not_evaluated_when_no_samples_produced, which mocks a judge model whosegenerate_content_asyncyields no responses and asserts the invocation still shows up inper_invocation_resultsasNOT_EVALUATEDrather than being dropped.Testing
per_invocation_resultsis[]instead of length 1) and passes after the fix.pytest tests/unittests/evaluation/— all passing, no regressions.pre-commit run --all-fileson changed files:isort,pyink,addlicense,ruffall pass.Risk & rollback
Additive, confined to one branch of
evaluate_invocations, no behavior change when samples are produced normally.