Skip to content

feat(plugins): Support opt-in session retention in MultimodalToolResultsPlugin - #6698

Open
chelsealong wants to merge 2 commits into
google:mainfrom
chelsealong:fix-6695-multimodal-session-retention
Open

feat(plugins): Support opt-in session retention in MultimodalToolResultsPlugin#6698
chelsealong wants to merge 2 commits into
google:mainfrom
chelsealong:fix-6695-multimodal-session-retention

Conversation

@chelsealong

Copy link
Copy Markdown

Fixes #6695

Problem

MultimodalToolResultsPlugin attaches google.genai.types.Part objects
returned by a tool to the immediately following model request, then
clears its saved state:

https://github.com/google/adk-python/blob/main/src/google/adk/plugins/multimodal_tool_results_plugin.py#L86-L92

For URI-based multimodal content (e.g. a PDF fetched with
Part.from_uri()), this means the document is available for the first
model response but silently disappears from every subsequent turn. As
described in the issue, a user can ask a follow-up question like "What is
the table of contents?" and the model will answer from its earlier
summary — or hallucinate — because the actual file_data part is no
longer in the request, even though the conversation looks well-grounded.

Fix

This adds an opt-in retention constructor argument to
MultimodalToolResultsPlugin:

  • "next_model_call" (default): unchanged behavior — parts are attached
    once and cleared.
  • "session": the plugin keeps re-attaching the latest saved parts to
    every subsequent model request for the rest of the session, so
    multi-turn document Q&A keeps the source material in context.
from google.adk.plugins.multimodal_tool_results_plugin import (
    MultimodalToolResultsPlugin,
)

app = App(
    name="multimodal_document_agent",
    root_agent=root_agent,
    plugins=[
        MultimodalToolResultsPlugin(retention="session"),
    ],
)

An earlier version of this PR only changed whether the saved parts were
cleared after one use, while still storing them under the existing
"temp:PARTS_RETURNED_BY_TOOLS_ID" key. That did not actually fix the
reported bug: "temp:"-prefixed state is invocation-scoped by the
session layer (BaseSessionService._trim_temp_delta_state strips it
before an event is persisted — see
src/google/adk/sessions/base_session_service.py), so it never survives
past the end of the current invocation regardless of what the plugin
does with it. Since one invocation is one conversational turn, the parts
were still gone by the very next turn — exactly the multi-turn scenario
the issue reports.

The fix now stores retention="session" parts under a new,
session-scoped (non-"temp:"-prefixed) key,
"multimodal_tool_results_plugin:PARTS_RETURNED_BY_TOOLS_ID", so the
value is written through to persisted session state and is still present
when a later invocation's before_model_callback runs. The default
"next_model_call" retention is unaffected and still uses the original
"temp:"-prefixed key with unchanged semantics.

This covers the session retention policy proposed in the issue. The
issue also sketches max_turns=N bounding and a custom
predicate/callback; those are left out of this PR to keep the change
minimal and reviewable — happy to follow up if maintainers want the
bounded/custom variants too.

Testing plan

Rewrote test_session_retention_reattaches_parts_across_turns to go
through two separate runner.run_async() calls sharing one
InMemorySessionService-backed session (turn 1 triggers a tool call
that returns a file_data part; turn 2 is a follow-up question), instead
of calling before_model_callback twice on the same in-memory State
object. The previous version of the test could not detect the bug above
because it never went through SessionService.append_event(), which is
where "temp:" state actually gets stripped between turns.

Confirmed the new test fails without the source fix
(git checkout HEAD~1 -- src/google/adk/plugins/multimodal_tool_results_plugin.py)
with the saved file_data part missing from turn 2's request, and passes
once the fix is restored:

$ pytest tests/unittests/plugins/test_multimodal_tool_results_plugin.py -v
...
tests/unittests/plugins/test_multimodal_tool_results_plugin.py::test_tool_returning_parts_are_added_to_llm_request PASSED
tests/unittests/plugins/test_multimodal_tool_results_plugin.py::test_tool_returning_non_list_of_parts_is_unchanged PASSED
tests/unittests/plugins/test_multimodal_tool_results_plugin.py::test_empty_contents_leaves_saved_parts_pending PASSED
tests/unittests/plugins/test_multimodal_tool_results_plugin.py::test_session_retention_reattaches_parts_across_turns PASSED
tests/unittests/plugins/test_multimodal_tool_results_plugin.py::test_multiple_tools_returning_parts_are_accumulated PASSED
5 passed, 3 warnings in 3.10s

Also ran the full plugins unit test suite to check for regressions:

$ pytest tests/unittests/plugins/ -q
718 passed, 20 warnings in 43.30s

pre-commit run (ruff, isort, pyink, addlicense, ADK compliance checks)
passes on both changed files.

AI-assistance disclosure

This PR was prepared with the assistance of an AI coding agent (Claude
Code), which read the issue, implemented the fix, wrote and verified the
regression test, and ran the project's lint/test tooling. All changes
were reviewed before submission. An independent review pass identified
that an earlier version of this fix did not actually solve the
cross-turn persistence problem (see "Fix" section above); this version
addresses that finding directly.

…ltsPlugin

MultimodalToolResultsPlugin attached tool-returned parts (e.g. a PDF
fetched via Part.from_uri()) to the immediately following model request
only, then cleared them. Follow-up turns lost access to the original
multimodal content, so the model could answer from a stale summary
instead of the source document.

Add a retention constructor argument: "next_model_call" (default) keeps
the existing one-shot behavior; "session" keeps re-attaching the latest
saved parts to every subsequent model request for the rest of the
session, so multi-turn document Q&A keeps the source material in context.

Fixes google#6695
The prior implementation kept parts under the existing
"temp:PARTS_RETURNED_BY_TOOLS_ID" state key and only changed whether that
key got cleared after one use. "temp:"-prefixed state is invocation-scoped
by the session layer (stripped before an event is persisted), so it never
reached a later invocation regardless of the plugin's own clearing logic -
the fix was a no-op for the actual cross-turn scenario in google#6695.

Store retention="session" parts under a new, non-"temp:"-prefixed session
state key instead, so the value is persisted and still present when a
later turn's before_model_callback runs. Also rewrite the regression test
to drive two separate runner.run_async() invocations against a real
session service, since the previous version (two direct
before_model_callback calls sharing one in-memory State object) could not
detect this class of bug.
@adk-bot adk-bot added the tools [Component] This issue is related to tools label Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tools [Component] This issue is related to tools

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Support opt-in multi-turn retention of URI-based multimodal tool results

3 participants