feat(observability): add base OpenTelemetry span enricher interceptor#17528
feat(observability): add base OpenTelemetry span enricher interceptor#17528chalmerlowe wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces OpenTelemetry tracing observability features to google-api-core, including environment variable and client options resolution helpers, and a gRPC client interceptor (OtelSpanEnricher) to enrich active spans with static and dynamic attributes. The feedback suggests using getattr instead of direct __dict__ access on client_options to properly support properties and slotted attributes, and checking static attributes for None values before setting them on the span to prevent potential OpenTelemetry issues.
| if client_options is not None: | ||
| options_dict = ( | ||
| client_options | ||
| if isinstance(client_options, dict) | ||
| else getattr(client_options, "__dict__", {}) | ||
| ) | ||
| option_key = f"enable_{signal_type.lower()}" | ||
| provider_key = f"{signal_type.rstrip('s').lower()}_provider" | ||
|
|
||
| if options_dict.get(option_key) is not None: | ||
| return bool(options_dict.get(option_key)) | ||
| if options_dict.get(provider_key) is not None: | ||
| return True |
There was a problem hiding this comment.
Accessing __dict__ directly on client_options to retrieve configuration options will fail to detect attributes that are defined as properties, inherited from parent classes, or defined via __slots__. This is a common pattern for client options classes (such as google.api_core.client_options.ClientOptions).
Using getattr instead of __dict__.get ensures that these attributes are correctly resolved regardless of how they are defined on the object.
| if client_options is not None: | |
| options_dict = ( | |
| client_options | |
| if isinstance(client_options, dict) | |
| else getattr(client_options, "__dict__", {}) | |
| ) | |
| option_key = f"enable_{signal_type.lower()}" | |
| provider_key = f"{signal_type.rstrip('s').lower()}_provider" | |
| if options_dict.get(option_key) is not None: | |
| return bool(options_dict.get(option_key)) | |
| if options_dict.get(provider_key) is not None: | |
| return True | |
| if client_options is not None: | |
| option_key = f"enable_{signal_type.lower()}" | |
| provider_key = f"{signal_type.rstrip('s').lower()}_provider" | |
| if isinstance(client_options, dict): | |
| opt_val = client_options.get(option_key) | |
| prov_val = client_options.get(provider_key) | |
| else: | |
| opt_val = getattr(client_options, option_key, None) | |
| prov_val = getattr(client_options, provider_key, None) | |
| if opt_val is not None: | |
| return bool(opt_val) | |
| if prov_val is not None: | |
| return True |
| # Inject static attributes | ||
| for key, val in self._static_attributes.items(): | ||
| span.set_attribute(key, val) |
There was a problem hiding this comment.
For consistency and to prevent potential issues with OpenTelemetry's set_attribute (which expects non-None values), we should check if the static attribute value is not None before setting it, similar to how dynamic attributes are handled.
| # Inject static attributes | |
| for key, val in self._static_attributes.items(): | |
| span.set_attribute(key, val) | |
| # Inject static attributes | |
| for key, val in self._static_attributes.items(): | |
| if val is not None: | |
| span.set_attribute(key, val) |
This PR introduces the base OpenTelemetry (OTel) gRPC client interceptor (
OtelSpanEnricher) under thegoogle.api_core.observabilitymodule. This interceptor is a key part of our "Delegate and Enrich" tracing strategy, which relies on standard OTel instrumentation to generate baseline network spans (T4 spans) and enriches them with Google Cloud specific domain attributes.Intent & Motivation
To meet the Client Libraries Observability Product Requirements, our generated libraries need to inject standard domain-specific attributes (like repository name, service name, and destination resource ID) onto generated tracing spans.
Rather than manual implementation of span lifecycle and propagation boilerplate in each client transport, this base interceptor allows client libraries to register a callback (
attribute_extractor) during channel initialization. This callback extracts dynamic telemetry attributes directly from request payloads (e.g. database name in Spanner, queue name in PubSub) and injects them into the active OTel span.Changes Included
OtelSpanEnricherclass, inheriting fromgrpc.UnaryUnaryClientInterceptor, which resolves static attributes and extracts dynamic attributes from request payloads to attach to the active recording span.opentelemetry-apito the dependencies block ofpyproject.tomland mapped its lower bound (opentelemetry-api==1.1.0) in the test constraints.grpcbeing installed.pytestpatterns to verify enrichment behavior on recording/non-recording spans, as well as checking that attribute extraction failures do not crash runtime RPC calls.