feat(plugins): Support opt-in session retention in MultimodalToolResultsPlugin - #6698
Open
chelsealong wants to merge 2 commits into
Open
feat(plugins): Support opt-in session retention in MultimodalToolResultsPlugin#6698chelsealong wants to merge 2 commits into
chelsealong wants to merge 2 commits into
Conversation
…ltsPlugin MultimodalToolResultsPlugin attached tool-returned parts (e.g. a PDF fetched via Part.from_uri()) to the immediately following model request only, then cleared them. Follow-up turns lost access to the original multimodal content, so the model could answer from a stale summary instead of the source document. Add a retention constructor argument: "next_model_call" (default) keeps the existing one-shot behavior; "session" keeps re-attaching the latest saved parts to every subsequent model request for the rest of the session, so multi-turn document Q&A keeps the source material in context. Fixes google#6695
The prior implementation kept parts under the existing "temp:PARTS_RETURNED_BY_TOOLS_ID" state key and only changed whether that key got cleared after one use. "temp:"-prefixed state is invocation-scoped by the session layer (stripped before an event is persisted), so it never reached a later invocation regardless of the plugin's own clearing logic - the fix was a no-op for the actual cross-turn scenario in google#6695. Store retention="session" parts under a new, non-"temp:"-prefixed session state key instead, so the value is persisted and still present when a later turn's before_model_callback runs. Also rewrite the regression test to drive two separate runner.run_async() invocations against a real session service, since the previous version (two direct before_model_callback calls sharing one in-memory State object) could not detect this class of bug.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6695
Problem
MultimodalToolResultsPluginattachesgoogle.genai.types.Partobjectsreturned by a tool to the immediately following model request, then
clears its saved state:
https://github.com/google/adk-python/blob/main/src/google/adk/plugins/multimodal_tool_results_plugin.py#L86-L92
For URI-based multimodal content (e.g. a PDF fetched with
Part.from_uri()), this means the document is available for the firstmodel response but silently disappears from every subsequent turn. As
described in the issue, a user can ask a follow-up question like "What is
the table of contents?" and the model will answer from its earlier
summary — or hallucinate — because the actual
file_datapart is nolonger in the request, even though the conversation looks well-grounded.
Fix
This adds an opt-in
retentionconstructor argument toMultimodalToolResultsPlugin:"next_model_call"(default): unchanged behavior — parts are attachedonce and cleared.
"session": the plugin keeps re-attaching the latest saved parts toevery subsequent model request for the rest of the session, so
multi-turn document Q&A keeps the source material in context.
An earlier version of this PR only changed whether the saved parts were
cleared after one use, while still storing them under the existing
"temp:PARTS_RETURNED_BY_TOOLS_ID"key. That did not actually fix thereported bug:
"temp:"-prefixed state is invocation-scoped by thesession layer (
BaseSessionService._trim_temp_delta_statestrips itbefore an event is persisted — see
src/google/adk/sessions/base_session_service.py), so it never survivespast the end of the current invocation regardless of what the plugin
does with it. Since one invocation is one conversational turn, the parts
were still gone by the very next turn — exactly the multi-turn scenario
the issue reports.
The fix now stores
retention="session"parts under a new,session-scoped (non-
"temp:"-prefixed) key,"multimodal_tool_results_plugin:PARTS_RETURNED_BY_TOOLS_ID", so thevalue is written through to persisted session state and is still present
when a later invocation's
before_model_callbackruns. The default"next_model_call"retention is unaffected and still uses the original"temp:"-prefixed key with unchanged semantics.This covers the
sessionretention policy proposed in the issue. Theissue also sketches
max_turns=Nbounding and a custompredicate/callback; those are left out of this PR to keep the change
minimal and reviewable — happy to follow up if maintainers want the
bounded/custom variants too.
Testing plan
Rewrote
test_session_retention_reattaches_parts_across_turnsto gothrough two separate
runner.run_async()calls sharing oneInMemorySessionService-backed session (turn 1 triggers a tool callthat returns a
file_datapart; turn 2 is a follow-up question), insteadof calling
before_model_callbacktwice on the same in-memoryStateobject. The previous version of the test could not detect the bug above
because it never went through
SessionService.append_event(), which iswhere
"temp:"state actually gets stripped between turns.Confirmed the new test fails without the source fix
(
git checkout HEAD~1 -- src/google/adk/plugins/multimodal_tool_results_plugin.py)with the saved
file_datapart missing from turn 2's request, and passesonce the fix is restored:
Also ran the full plugins unit test suite to check for regressions:
pre-commit run(ruff, isort, pyink, addlicense, ADK compliance checks)passes on both changed files.
AI-assistance disclosure
This PR was prepared with the assistance of an AI coding agent (Claude
Code), which read the issue, implemented the fix, wrote and verified the
regression test, and ran the project's lint/test tooling. All changes
were reviewed before submission. An independent review pass identified
that an earlier version of this fix did not actually solve the
cross-turn persistence problem (see "Fix" section above); this version
addresses that finding directly.