Summary

Stacked on top of #11380 -- review that first; the merge base of this PR is dougqh/conflating-metrics-producer-wins, not master. The diff shown here is only the work that's new beyond that PR.

Moves the per-span MetricKey construction, cache lookups, and aggregation off the producer thread into the existing aggregator thread, replacing the Batch-based conflation pipeline with a thin per-span SpanSnapshot posted to the inbox.

What the producer does now (per span)

• filter (shouldComputeMetric, resource-ignored, longRunning)
• collect tag values into a SpanSnapshot (one allocation per span)
• inbox.offer(snapshot) + return error flag for forceKeep

What moved off the producer

• MetricKey construction and its hash computation
• SERVICE_NAMES.computeIfAbsent (UTF8 encoding of service name)
• SPAN_KINDS.computeIfAbsent (UTF8 encoding of span.kind)
• PEER_TAGS_CACHE lookups (peer-tag name+value UTF8 encoding)
• pending / keys ConcurrentHashMap operations
• Batch pooling, atomic ops, contributeTo

Removed entirely

• Batch.java -- the aggregator's existing LRUCache<MetricKey, AggregateMetric> IS the conflation point now
• pending ConcurrentHashMap<MetricKey, Batch>
• keys ConcurrentHashMap<MetricKey, MetricKey> (canonical dedup)
• batchPool MessagePassingQueue<Batch>
• CommonKeyCleaner's keys.keySet() tracking; AggregateExpiry now just reports LRU drops to health metrics

Added

• SpanSnapshot: immutable value carrying the raw MetricKey inputs + a tagAndDuration long (duration OR-ed with ERROR_TAG / TOP_LEVEL_TAG).
• AggregateMetric.recordOneDuration(long) -- single-hit equivalent of the existing recordDurations(int, AtomicLongArray).
• Peer-tag values flow through the snapshot as a flattened String[] of [name0, value0, name1, value1, ...]; the aggregator encodes them through PEER_TAGS_CACHE on its own thread.
• HealthMetrics.onStatsInboxFull() + a TracerHealthMetrics counter reported as stats.dropped_aggregates{reason:inbox_full} -- parallel to the existing reason:lru_eviction. Without conflation the producer can lose snapshots when the bounded MPSC queue is full; this makes that visible without silencing it.

Benchmark results (2 forks × 5 iter × 15s)

ConflatingMetricsAggregatorDDSpanBenchmark:

~60% faster on the production DDSpan path. The SimpleSpan bench shows ~53% faster as well.

Caveat on the bench numbers

Without conflation, the producer pushes 1 inbox item per span instead of ~1 per 64. At the JMH bench's synthetic rate (effectively ~20M snapshots/sec from the producer) the consumer can't keep up and inbox.offer silently drops -- the new onStatsInboxFull counter would fire constantly. The headline numbers measure producer publish() latency only; consumer throughput at realistic span rates is a follow-up to validate. Tuning maxPending matters more in this design.

Real fixes for capacity (out of scope for this PR):

• Bump maxPending default; the conflating design used 2048 slots × ~64 conflation = ~131K effective capacity, the new design has 2048 slots flat.
• Producer-side mini-batching (TLAB-style accumulator per thread) to recover some compression.

Test plan

• ./gradlew :dd-trace-core:test --tests 'datadog.trace.common.metrics.*' passes
• ./gradlew :dd-trace-core:test --tests 'datadog.trace.core.monitor.*' passes
• ./gradlew :dd-trace-core:compileJava :dd-trace-core:compileTestGroovy :dd-trace-core:compileJmhJava :dd-trace-core:compileTraceAgentTestGroovy all green
• ./gradlew spotlessCheck clean
• CI muzzle / integration suites
• Validate stats.dropped_aggregates{reason:inbox_full} reports as expected under a synthetic high-load run (not in the JMH bench)

🤖 Generated with Claude Code