Skip to content

feat(cli): replay selected tool responses in conformance tests - #7593

Open
SHAI-shivansh-sharma wants to merge 1 commit into
google:mainfrom
SHAI-shivansh-sharma:feat/conformance-recorded-tool-responses
Open

SHAI-shivansh-sharma wants to merge 1 commit into
google:mainfrom
SHAI-shivansh-sharma:feat/conformance-recorded-tool-responses

Conversation

@SHAI-shivansh-sharma

@SHAI-shivansh-sharma SHAI-shivansh-sharma commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Link to Issue

Closes: #7592

Problem

Conformance replay executes every recorded tool body before substituting its response. Tools that call paid APIs, mutate external state, or need unavailable credentials make otherwise deterministic replay unsafe or impossible.

Solution

Add an opt-in replay_tool_responses mapping to test specs, keyed by agent name. For selected tools, ReplayPlugin still verifies the recorded call order, name, and arguments, but returns a copy of the recorded response without executing the tool. Unselected tools retain existing behavior. Invalid selections, missing responses, unsupported AgentTool selection, and a missing ReplayPlugin fail rather than falling back to live execution.

Example test-spec.yaml:

replay_tool_responses:
  weather_agent:
    - get_forecast

Testing Plan

  • Added or updated unit tests for selected/default replay, SSE and non-SSE, repeated calls, agent scoping, mismatch/error cases, spec parsing, client request propagation, and API fail-fast.
  • Scoped local suite: 293 passed, 5 skipped (38 unrelated optional-integration cases deselected).
  • Final feature/API smoke: 18 passed.
  • ruff, isort, pyink, license, compliance, and other applicable pre-commit hooks passed.
  • Manual end-to-end server run not performed locally. To verify: start adk api_server with --extra_plugins google.adk.cli.plugins.replay_plugin.ReplayPlugin, add the mapping above to a conformance test spec with a recorded get_forecast call, run replay, and confirm the tool body is not invoked. Without the plugin, /run and /run_sse reject the request before execution.

The Windows-only pre-commit run could not execute the repository's check-new-py-prefix hook because it invokes /bin/bash; this PR creates no new Python files. A full local API suite also needs unrelated optional Agent Platform, eval, and agent-identity dependencies.

Allow test specs to select agent tools whose recorded responses are returned without executing their bodies. Preserve verification of call order and arguments, reject missing recordings and missing ReplayPlugin before execution, and leave default replay behavior unchanged.

Closes google#7592
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Allow conformance replay to use recorded responses without executing selected tools

1 participant