Repository navigation
perf(telemetry): skip serializing the request history when content capture is off (20% faster at 300 turns) - #7579
Conversation
…e is off With the experimental GenAI semconv (default on schema v2) every LLM call converted the full request contents and system instruction into OTel AnyValues, even in the default NO_CONTENT mode where only the tool definitions survive redaction. That made each step O(history), so a session paid O(n^2). Build only the tool definitions when neither logs nor spans capture content; emitted telemetry is unchanged.
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
There was a problem hiding this comment.
Code Review
This pull request introduces an include_content parameter to set_operation_details_attributes_from_request to avoid serializing the full conversation history when content capture is disabled, while still retaining tool definitions. This is integrated into the tracing logic and verified with a new unit test. Feedback suggests adding a defensive check to prevent a potential AttributeError if llm_request.config is None.
| # Without content capture only the tool definitions survive redaction | ||
| # (see _operation_details_attributes_no_content), so skip serializing the | ||
| # whole conversation history on every request. | ||
| tool_definitions = _resolve_tool_definitions(llm_request.config.tools or []) |
There was a problem hiding this comment.
To prevent potential AttributeError if llm_request.config is None (e.g., in unit tests or dynamic configurations), it is safer to use defensive programming and check if config is present before accessing its tools attribute.
tools = llm_request.config.tools if llm_request.config else None
tool_definitions = _resolve_tool_definitions(tools or [])There was a problem hiding this comment.
LlmRequest.config is a required field with default_factory=types.GenerateContentConfig, so it is never None here, and the existing content-capture path a few lines up reads llm_request.config.tools the same way. I kept the two consistent.
Link to Issue or Description of Change
2. Or, if no issue exists, describe the change:
Problem:
The experimental GenAI semconv is the default with the v2 schema. Under it, every LLM call runs
set_operation_details_attributes_from_request, which converts the full request history (contents) and the system instructions into OTel values. In the defaultNO_CONTENTmode, both the completion log and the span go through_operation_details_attributes_no_content, which keeps onlyGEN_AI_TOOL_DEFINITIONS. Everything else is thrown away. Each step therefore costs O(history), and a session costs O(n²) in its number of turns.Solution:
When neither logs nor experimental spans capture content, only the tool definitions are built (the same
_resolve_tool_definitions+_to_any_valueas the full path). The full path is unchanged when content capture is on, and the emitted telemetry is identical in both modes.Testing Plan
Unit Tests:
I added a test that checks that, with content capture off, the attributes contain only the tool definitions and match the no-content view of the full path. The telemetry suite has the same results with and without this change (886 passed). Its 100 pre-existing local failures also happen on
main.Manual End-to-End (E2E) Tests:
InMemoryRunnerwith a fakeBaseLlm, one tool call per turn, median of 5 runs:The time saved grows with the session length. A dump of all events of a 100-turn session hashes identically before and after.
Checklist