Disable service discovery in CI Visibility agentless mode by daniel-mohedano · Pull Request #12908 · DataDog/dd-trace-java

daniel-mohedano · 2026-10-16T10:07:36Z

What Does This Do

Disables service discovery by default when CI Visibility runs in agentless mode. Setting DD_TRACE_SERVICE_DISCOVERY_ENABLED=true still turns it back on.

Motivation

On Linux, 1 second after startup, service discovery loads JNA on a background thread to write the tracer metadata file. JNA adds new system properties (jnidispatch.path, jna.loaded, jna.platform.library.path).

On Java 8, system properties are a Hashtable with fail-fast iterators. Any application thread iterating them at that moment throws ConcurrentModificationException. The Gradle launcher does exactly that at startup, so Gradle builds instrumented with the tracer can fail with:

java.util.ConcurrentModificationException
  at java.util.Hashtable$Enumerator.next
  at org.gradle.launcher.cli.converter.LayoutToPropertiesConverter.configureFromSystemPropertiesOfThisJvm

This caused 13 failures of GradleLauncherSmokeTest on Java 8 on master over the last 3 weeks.

The metadata file is only read by an Agent running on the same host. In CI Visibility agentless mode there is usually no Agent, and the instrumented processes are short-lived build tools and test JVMs. So the file brings little value there.

Additional Notes

The race still exists for other Java 8 applications on Linux that iterate system properties around startup. That would need a separate fix in service discovery.

test-environment-trigger: skip

…ess mode

cit-pr-commenter-54b7da · 2026-10-16T10:21:35Z

CI Visibility Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	54.44	57.69	$\color{green}{\blacktriangledown}$ -3.25	56.55	$\color{green}{\blacktriangledown}$ -2.11	64/312
agentEvpProxy	54.07	n/a	n/a	n/a	n/a	-

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

cit-pr-commenter-54b7da · 2026-10-16T10:24:58Z

CI Visibility Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	38.82	37.15	$\color{red}{\blacktriangle}$ +1.67	37.15	$\color{red}{\blacktriangle}$ +1.67	33/170
agentless	37.98	37.15	$\color{red}{\blacktriangle}$ +0.83	36.42	$\color{red}{\blacktriangle}$ +1.56	33/143
agentlessCodeCoverage	46.72	45.38	$\color{red}{\blacktriangle}$ +1.34	45.38	$\color{red}{\blacktriangle}$ +1.34	33/142
agentlessLineCoverage	58.01	56.55	$\color{red}{\blacktriangle}$ +1.46	56.55	$\color{red}{\blacktriangle}$ +1.46	32/141

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

cit-pr-commenter-54b7da · 2026-10-16T10:26:29Z

CI Visibility Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	87.16	91.39	$\color{green}{\blacktriangledown}$ -4.23	89.58	$\color{green}{\blacktriangledown}$ -2.42	33/171
agentless	81.04	82.69	$\color{green}{\blacktriangledown}$ -1.65	82.69	$\color{green}{\blacktriangledown}$ -1.65	33/141
agentlessCodeCoverage	95.49	99.00	$\color{green}{\blacktriangledown}$ -3.51	97.04	$\color{green}{\blacktriangledown}$ -1.55	33/141
agentlessLineCoverage	110.83	113.88	$\color{green}{\blacktriangledown}$ -3.05	113.88	$\color{green}{\blacktriangledown}$ -3.05	33/142

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

cit-pr-commenter-54b7da · 2026-10-16T10:26:33Z

CI Visibility Test Environment - pass4s

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	6.96	10.75	$\color{green}{\blacktriangledown}$ -3.79	10.13	$\color{green}{\blacktriangledown}$ -3.17	35/166
agentless	6.70	10.33	$\color{green}{\blacktriangledown}$ -3.63	10.13	$\color{green}{\blacktriangledown}$ -3.43	32/134
agentlessCodeCoverage	14.29	18.08	$\color{green}{\blacktriangledown}$ -3.79	17.73	$\color{green}{\blacktriangledown}$ -3.44	28/122

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

cit-pr-commenter-54b7da · 2026-10-16T10:27:33Z

CI Visibility Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	22.99	22.54	$\color{red}{\blacktriangle}$ +0.45	22.09	$\color{red}{\blacktriangle}$ +0.90	35/171
agentless	18.96	19.59	$\color{green}{\blacktriangledown}$ -0.63	19.59	$\color{green}{\blacktriangledown}$ -0.63	33/137
agentlessCodeCoverage	20.44	20.39	$\color{red}{\blacktriangle}$ +0.05	19.99	$\color{red}{\blacktriangle}$ +0.45	33/137
agentlessLineCoverage	27.03	27.52	$\color{green}{\blacktriangledown}$ -0.49	27.52	$\color{green}{\blacktriangledown}$ -0.49	20/93

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

cit-pr-commenter-54b7da · 2026-10-16T10:28:34Z

CI Visibility Test Environment - heliboard

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	9.05	9.92	$\color{green}{\blacktriangledown}$ -0.87	9.92	$\color{green}{\blacktriangledown}$ -0.87	32/139

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

dd-octo-sts · 2026-10-16T10:33:43Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.76 s	14.65 s	[+0.1%; +1.3%] (maybe worse)
startup:insecure-bank:tracing:Agent	13.64 s	13.65 s	[-0.9%; +0.7%] (no difference)
startup:petclinic:appsec:Agent	17.05 s	16.41 s	[-0.6%; +8.4%] (no difference)
startup:petclinic:iast:Agent	16.87 s	16.88 s	[-1.0%; +0.8%] (no difference)
startup:petclinic:profiling:Agent	16.81 s	16.60 s	[+0.3%; +2.3%] (maybe worse)
startup:petclinic:sca:Agent	16.98 s	16.88 s	[-0.4%; +1.6%] (no difference)
startup:petclinic:tracing:Agent	15.59 s	16.08 s	[-7.1%; +1.0%] (no difference)

Commit: baefd31a · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

cit-pr-commenter-54b7da · 2026-10-16T10:34:21Z

CI Visibility Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	11.65	13.67	$\color{green}{\blacktriangledown}$ -2.02	13.40	$\color{green}{\blacktriangledown}$ -1.75	35/171
agentless	10.71	12.12	$\color{green}{\blacktriangledown}$ -1.41	12.12	$\color{green}{\blacktriangledown}$ -1.41	35/141
agentlessCodeCoverage	14.85	15.72	$\color{green}{\blacktriangledown}$ -0.87	15.41	$\color{green}{\blacktriangledown}$ -0.56	34/140
agentlessLineCoverage	16.56	17.38	$\color{green}{\blacktriangledown}$ -0.82	17.38	$\color{green}{\blacktriangledown}$ -0.82	33/139

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

cit-pr-commenter-54b7da · 2026-10-16T10:36:44Z

CI Visibility Test Environment - jolokia

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	94.39	95.12	$\color{green}{\blacktriangledown}$ -0.73	95.12	$\color{green}{\blacktriangledown}$ -0.73	36/172
agentless	90.22	91.39	$\color{green}{\blacktriangledown}$ -1.17	89.58	$\color{red}{\blacktriangle}$ +0.64	35/143
agentlessCodeCoverage	97.59	101.00	$\color{green}{\blacktriangledown}$ -3.41	99.00	$\color{green}{\blacktriangledown}$ -1.41	34/142
agentlessLineCoverage	100.28	103.04	$\color{green}{\blacktriangledown}$ -2.76	101.00	$\color{green}{\blacktriangledown}$ -0.72	39/150

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

cit-pr-commenter-54b7da · 2026-10-16T10:43:09Z

CI Visibility Test Environment - okhttp

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	19.15	19.20	$\color{green}{\blacktriangledown}$ -0.05	19.99	$\color{green}{\blacktriangledown}$ -0.84	35/169
agentless	18.15	18.45	$\color{green}{\blacktriangledown}$ -0.30	19.20	$\color{green}{\blacktriangledown}$ -1.05	36/145
agentlessCodeCoverage	20.45	21.65	$\color{green}{\blacktriangledown}$ -1.20	22.99	$\color{green}{\blacktriangledown}$ -2.54	37/147
agentlessLineCoverage	37.37	37.91	$\color{green}{\blacktriangledown}$ -0.54	39.45	$\color{green}{\blacktriangledown}$ -2.08	33/141

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

cit-pr-commenter-54b7da · 2026-10-16T10:49:25Z

CI Visibility Test Environment - spring_boot

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	15.96	16.36	$\color{green}{\blacktriangledown}$ -0.40	16.36	$\color{green}{\blacktriangledown}$ -0.40	35/167
agentless	10.49	9.92	$\color{red}{\blacktriangle}$ +0.57	9.92	$\color{red}{\blacktriangle}$ +0.57	35/140
agentlessCodeCoverage	14.36	13.67	$\color{red}{\blacktriangle}$ +0.69	13.67	$\color{red}{\blacktriangle}$ +0.69	35/142
agentlessLineCoverage	23.58	22.54	$\color{red}{\blacktriangle}$ +1.04	22.54	$\color{red}{\blacktriangle}$ +1.04	32/138

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

chatgpt-codex-connector · 2026-10-16T10:52:02Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-16T10:54:27.827872Z	`baefd31`	Draft marked ready
🔒 Security Review	✅ Completed 2026-10-16T10:54:26.489135Z	`baefd31`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

datadog-prod-us1-5

More details

The new default disables service discovery only when CI Visibility and agentless mode are both enabled, while preserving explicit overrides. The completed static review found no actionable regressions.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit baefd31}

cit-pr-commenter-54b7da · 2026-10-16T11:09:09Z

CI Visibility Test Environment - sonar-java

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	44.96	17.38	$\color{red}{\blacktriangle}$ +27.58	15.41	$\color{red}{\blacktriangle}$ +29.55	35/169
agentless	41.46	13.94	$\color{red}{\blacktriangle}$ +27.52	13.13	$\color{red}{\blacktriangle}$ +28.33	35/141
agentlessCodeCoverage	71.91	79.45	$\color{green}{\blacktriangledown}$ -7.54	79.45	$\color{green}{\blacktriangledown}$ -7.54	35/142
agentlessLineCoverage	109.61	123.36	$\color{green}{\blacktriangledown}$ -13.75	123.36	$\color{green}{\blacktriangledown}$ -13.75	31/141

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

fix(ci-visibility): disable service discovery in CI Visibility agentl…

baefd31

…ess mode

daniel-mohedano added comp: ci visibility Continuous Integration Visibility tag: ai generated Largely based on code generated by an AI or LLM type: bug labels Oct 16, 2026

This comment has been minimized.

Sign in to view

daniel-mohedano marked this pull request as ready for review October 16, 2026 10:51

daniel-mohedano requested a review from a team as a code owner October 16, 2026 10:51

daniel-mohedano requested review from bric3 and removed request for a team October 16, 2026 10:51

datadog-prod-us1-5 Bot reviewed Oct 16, 2026

View reviewed changes

AlexeyKuznetsov-DD approved these changes Oct 16, 2026

View reviewed changes

Conversation

daniel-mohedano commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Uh oh!

cit-pr-commenter-54b7da Bot commented Oct 16, 2026

CI Visibility Test Environment - sbt-scalatest

Uh oh!

cit-pr-commenter-54b7da Bot commented Oct 16, 2026

CI Visibility Test Environment - nebula-release-plugin

Uh oh!

This comment has been minimized.

cit-pr-commenter-54b7da Bot commented Oct 16, 2026

CI Visibility Test Environment - netflix-zuul

Uh oh!

cit-pr-commenter-54b7da Bot commented Oct 16, 2026

CI Visibility Test Environment - pass4s

Uh oh!

cit-pr-commenter-54b7da Bot commented Oct 16, 2026

CI Visibility Test Environment - reactive-streams-jvm

Uh oh!

cit-pr-commenter-54b7da Bot commented Oct 16, 2026

CI Visibility Test Environment - heliboard

Uh oh!

dd-octo-sts Bot commented Oct 16, 2026

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

cit-pr-commenter-54b7da Bot commented Oct 16, 2026

CI Visibility Test Environment - sonar-kotlin

Uh oh!

cit-pr-commenter-54b7da Bot commented Oct 16, 2026

CI Visibility Test Environment - jolokia

Uh oh!

cit-pr-commenter-54b7da Bot commented Oct 16, 2026

CI Visibility Test Environment - okhttp

Uh oh!

cit-pr-commenter-54b7da Bot commented Oct 16, 2026

CI Visibility Test Environment - spring_boot

Uh oh!

chatgpt-codex-connector Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

datadog-prod-us1-5 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cit-pr-commenter-54b7da Bot commented Oct 16, 2026

CI Visibility Test Environment - sonar-java

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

daniel-mohedano commented Oct 16, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 16, 2026 •

edited

Loading