Repository navigation
feat(cli): replay selected tool responses in conformance tests - #7593
Open
SHAI-shivansh-sharma wants to merge 1 commit into
Open
SHAI-shivansh-sharma wants to merge 1 commit into
SHAI-shivansh-sharma wants to merge 1 commit into
Conversation
Allow test specs to select agent tools whose recorded responses are returned without executing their bodies. Preserve verification of call order and arguments, reject missing recordings and missing ReplayPlugin before execution, and leave default replay behavior unchanged. Closes google#7592
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to Issue
Closes: #7592
Problem
Conformance replay executes every recorded tool body before substituting its response. Tools that call paid APIs, mutate external state, or need unavailable credentials make otherwise deterministic replay unsafe or impossible.
Solution
Add an opt-in
replay_tool_responsesmapping to test specs, keyed by agent name. For selected tools, ReplayPlugin still verifies the recorded call order, name, and arguments, but returns a copy of the recorded response without executing the tool. Unselected tools retain existing behavior. Invalid selections, missing responses, unsupported AgentTool selection, and a missing ReplayPlugin fail rather than falling back to live execution.Example test-spec.yaml:
Testing Plan
ruff,isort,pyink, license, compliance, and other applicable pre-commit hooks passed.adk api_serverwith--extra_plugins google.adk.cli.plugins.replay_plugin.ReplayPlugin, add the mapping above to a conformance test spec with a recordedget_forecastcall, run replay, and confirm the tool body is not invoked. Without the plugin,/runand/run_ssereject the request before execution.The Windows-only pre-commit run could not execute the repository's
check-new-py-prefixhook because it invokes/bin/bash; this PR creates no new Python files. A full local API suite also needs unrelated optional Agent Platform, eval, and agent-identity dependencies.