Skip to content

Stamp _dd.apm.enabled on every exported chunk when APM tracing is disabled#12028

Draft
claponcet wants to merge 4 commits into
masterfrom
clara.poncet/apm-enabled-service-entry-span
Draft

Stamp _dd.apm.enabled on every exported chunk when APM tracing is disabled#12028
claponcet wants to merge 4 commits into
masterfrom
clara.poncet/apm-enabled-service-entry-span

Conversation

@claponcet

@claponcet claponcet commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

When APM tracing is disabled (DD_APM_TRACING_ENABLED=false, i.e. standalone ASM), this stamps the _dd.apm.enabled:0 billing marker on the root-most span(s) of every exported trace chunk at export time (CoreTracer.write), instead of setting it once on the local root span at creation (Config.getLocalRootSpanTags()).

For each exported chunk:

  • Fast path — if the trace's local root (the service-entry span) is part of the chunk, the marker is stamped on it directly (the common case, matching the previous behavior).
  • Fallback — for a chunk flushed without its local root (a partial flush, or an orphaned subtree), the marker is stamped on every span whose parent is absent from the chunk, i.e. the span(s) that top the chunk.

This deliberately does not rely on span ordering: the local root is not necessarily the first span in finish order for asynchronous frameworks.

Motivation

Standalone ASM customers rely on _dd.apm.enabled:0 to opt out of APM host billing: the intake bills a trace chunk unless at least one of its spans carries the marker.

Previously the marker lived only on the trace's local root span, on the assumption that the local root is present in every exported chunk. That assumption breaks in two ways:

  • A child span outlives its local root — the root flushes in one chunk (marked), while the long-lived child (e.g. a delayed outbound http.request) flushes later, alone, in a separate chunk with no marker.
  • An HTTP client call creates an inferred, in-process service-entry span for a different service that never received the tag (see APPSEC-69334).

In both cases the intake charges APM host billing for an ASM-only service. Making the decision at chunk-export time — the one place that unambiguously knows chunk boundaries — guarantees every exported chunk carries the marker. This mirrors the equivalent Node.js fix (dd-trace-js#9483).

Additional Notes

The marker is now stamped after the sampling decision and only when APM tracing is disabled, so there is no effect on the default (APM-enabled) hot path. The fallback's array allocation and sort run only in the cold path (APM disabled + a multi-span chunk exported without its local root).

Covered at two levels:

  • ApmTracingDisabledChunkMarkerTest (dd-trace-core) — a deterministic unit reproduction of the split-chunk case, a positive single-chunk control that isolates chunk-splitting as the cause, and a case where the local root is not first in finish order (guarding against marking merely chunk.get(0)).
  • A new /late-outbound endpoint and smoke test in the apm-tracing-disabled module that exercises a real delayed outbound child flushed alone in its own chunk.

Contributor Checklist

  • Format the title according to the contribution guidelines
  • Assign the type: and (comp: or inst:) labels in addition to any other useful labels
  • Avoid using close, fix, or any linking keywords when referencing an issue
  • Update the CODEOWNERS file on source file addition, migration, or deletion — n/a (no ownership change)
  • Update public documentation with any new configuration flags or behaviors — n/a (no new config)

Jira ticket: APPSEC-69334

…abled

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@claponcet claponcet added tag: ai generated Largely based on code generated by an AI or LLM comp: core Tracer core type: bug fix Bug fix comp: asm waf Application Security Management (WAF) and removed comp: core Tracer core labels Jul 22, 2026
@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Jul 22, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 0.00%
Overall Coverage: 57.33% (-0.05%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 3258ce2 | Docs | Datadog PR Page | Give us feedback!

claponcet and others added 3 commits July 22, 2026 17:22
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@claponcet

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 3258ce2215

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

}
}

@GetMapping("/late-outbound")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice, I appreciate the test case for this

@dd-octo-sts

dd-octo-sts Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.98 s 13.95 s [-0.5%; +1.0%] (no difference)
startup:insecure-bank:tracing:Agent 12.91 s 12.90 s [-0.6%; +0.7%] (no difference)
startup:petclinic:appsec:Agent 16.94 s 16.79 s [-0.1%; +1.9%] (no difference)
startup:petclinic:iast:Agent 16.77 s 16.89 s [-1.5%; -0.0%] (maybe better)
startup:petclinic:profiling:Agent 16.63 s 16.67 s [-1.4%; +0.9%] (no difference)
startup:petclinic:sca:Agent 16.94 s 16.72 s [+0.4%; +2.2%] (maybe worse)
startup:petclinic:tracing:Agent 16.16 s 15.76 s [-1.8%; +6.9%] (no difference)

Commit: 3258ce22 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: asm waf Application Security Management (WAF) tag: ai generated Largely based on code generated by an AI or LLM type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants