Fix Cucumber test retries running under Suite#11850

Open

daniel-mohedano wants to merge 1 commit into

daniel.mohedano/jep-500-junitfrom

daniel.mohedano/fix-cucumber-retries

daniel-mohedano commented Jul 3, 2026 •

edited

Loading

Contributor

What Does This Do

Resolve the per-engine test-identifier factory in TestDataFactory by the leaf engine instead of the root one. Under @Suite, the unique id is rooted at the suite engine ([engine:junit-platform-suite]/[suite:…]/[engine:cucumber]/…). The previous approach, using UniqueId.getEngineId() returns the root engine, which in these cases would point to JUnit instead of Cucumber.

Motivation

Test execution policies (ATR, EFD, TIA, etc.) were silently not applied to Cucumber scenarios run via the JUnit Platform Suite Engine (@Suite + junit-platform-suite-engine).

The bug was masked because the instrumentation tests launch Cucumber directly via the Launcher API, with Cucumber as the root engine.

Additional Notes

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Add your completed PR to the merge queue by commenting /merge. You can also:
- Customize the commit message associated with the merge with /merge --commit-message "..."
- Remove your PR from the merge queue with /merge -c
- Skip all merge queue checks with /merge -f --reason "reason"; please use this judiciously, as some checks do not run at the PR-level (note: the PR still needs to be mergeable, this will only skip the pre-merge build)
- Get more information in this doc

Jira ticket:


          fix: cucumber retries running through Suite

70060ee

daniel-mohedano added type: bug comp: ci visibility labels

daniel-mohedano changed the base branch from master to daniel.mohedano/jep-500-junit

July 3, 2026 07:58

daniel-mohedano changed the title ~~Daniel.mohedano/fix cucumber retries~~ Fix Cucumber test retries running under Suite

cit-pr-commenter-54b7da Bot commented Jul 3, 2026

Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	53.18	55.43	$\color{green}{\blacktriangledown}$ -2.25	55.43	$\color{green}{\blacktriangledown}$ -2.25	54/196
agentEvpProxy	53.54	n/a	n/a	n/a	n/a	-

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da Bot commented Jul 3, 2026

Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	87.61	86.07	$\color{red}{\blacktriangle}$ +1.54	86.07	$\color{red}{\blacktriangle}$ +1.54	30/144
agentless	80.74	81.05	$\color{green}{\blacktriangledown}$ -0.31	81.05	$\color{green}{\blacktriangledown}$ -0.31	31/144
agentlessCodeCoverage	96.57	95.12	$\color{red}{\blacktriangle}$ +1.45	95.12	$\color{red}{\blacktriangle}$ +1.45	30/141
agentlessLineCoverage	113.91	111.62	$\color{red}{\blacktriangle}$ +2.29	111.62	$\color{red}{\blacktriangle}$ +2.29	30/141

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da Bot commented Jul 3, 2026

Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	36.38	36.42	$\color{green}{\blacktriangledown}$ -0.04	36.42	$\color{green}{\blacktriangledown}$ -0.04	29/97
agentless	35.73	35.70	$\color{red}{\blacktriangle}$ +0.03	35.70	$\color{red}{\blacktriangle}$ +0.03	29/97
agentlessCodeCoverage	43.85	44.48	$\color{green}{\blacktriangledown}$ -0.63	43.60	$\color{red}{\blacktriangle}$ +0.25	29/97
agentlessLineCoverage	75.06	74.82	$\color{red}{\blacktriangle}$ +0.24	74.82	$\color{red}{\blacktriangle}$ +0.24	28/96

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da Bot commented Jul 3, 2026

Test Environment - pass4s

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	15.36	10.75	$\color{red}{\blacktriangle}$ +4.61	10.33	$\color{red}{\blacktriangle}$ +5.03	26/97
agentless	13.86	10.75	$\color{red}{\blacktriangle}$ +3.11	9.92	$\color{red}{\blacktriangle}$ +3.94	26/98
agentlessCodeCoverage	20.84	18.08	$\color{red}{\blacktriangle}$ +2.76	17.73	$\color{red}{\blacktriangle}$ +3.11	25/95

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da Bot commented Jul 3, 2026

Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	21.94	21.65	$\color{red}{\blacktriangle}$ +0.29	21.65	$\color{red}{\blacktriangle}$ +0.29	28/142
agentless	18.83	18.45	$\color{red}{\blacktriangle}$ +0.38	18.82	$\color{red}{\blacktriangle}$ +0.01	29/141
agentlessCodeCoverage	20.25	19.99	$\color{red}{\blacktriangle}$ +0.26	19.99	$\color{red}{\blacktriangle}$ +0.26	28/140
agentlessLineCoverage	30.20	29.82	$\color{red}{\blacktriangle}$ +0.38	29.82	$\color{red}{\blacktriangle}$ +0.38	27/139

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da Bot commented Jul 3, 2026

Test Environment - jolokia

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	93.12	93.23	$\color{green}{\blacktriangledown}$ -0.11	93.23	$\color{green}{\blacktriangledown}$ -0.11	28/100
agentless	87.44	89.58	$\color{green}{\blacktriangledown}$ -2.14	89.58	$\color{green}{\blacktriangledown}$ -2.14	28/100
agentlessCodeCoverage	98.28	99.00	$\color{green}{\blacktriangledown}$ -0.72	99.00	$\color{green}{\blacktriangledown}$ -0.72	27/97
agentlessLineCoverage	101.48	99.00	$\color{red}{\blacktriangle}$ +2.48	99.00	$\color{red}{\blacktriangle}$ +2.48	26/96

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da Bot commented Jul 3, 2026

Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	13.12	13.13	$\color{green}{\blacktriangledown}$ -0.01	13.13	$\color{green}{\blacktriangledown}$ -0.01	29/99
agentless	12.92	12.12	$\color{red}{\blacktriangle}$ +0.80	11.88	$\color{red}{\blacktriangle}$ +1.04	28/98
agentlessCodeCoverage	15.73	15.41	$\color{red}{\blacktriangle}$ +0.32	15.11	$\color{red}{\blacktriangle}$ +0.62	28/98
agentlessLineCoverage	20.24	19.20	$\color{red}{\blacktriangle}$ +1.04	19.20	$\color{red}{\blacktriangle}$ +1.04	28/98

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da Bot commented Jul 3, 2026

Test Environment - okhttp

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	18.92	19.20	$\color{green}{\blacktriangledown}$ -0.28	19.59	$\color{green}{\blacktriangledown}$ -0.67	31/105
agentless	19.04	19.20	$\color{green}{\blacktriangledown}$ -0.16	19.20	$\color{green}{\blacktriangledown}$ -0.16	31/105
agentlessCodeCoverage	23.56	22.54	$\color{red}{\blacktriangle}$ +1.02	22.09	$\color{red}{\blacktriangle}$ +1.47	30/102
agentlessLineCoverage	45.01	44.48	$\color{red}{\blacktriangle}$ +0.53	44.48	$\color{red}{\blacktriangle}$ +0.53	31/107

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da Bot commented Jul 3, 2026

Test Environment - spring_boot

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	16.68	16.04	$\color{red}{\blacktriangle}$ +0.64	16.04	$\color{red}{\blacktriangle}$ +0.64	25/94
agentless	9.30	9.73	$\color{green}{\blacktriangledown}$ -0.43	9.73	$\color{green}{\blacktriangledown}$ -0.43	25/94
agentlessCodeCoverage	12.98	13.13	$\color{green}{\blacktriangledown}$ -0.15	13.40	$\color{green}{\blacktriangledown}$ -0.42	25/93
agentlessLineCoverage	33.08	32.95	$\color{red}{\blacktriangle}$ +0.13	32.30	$\color{red}{\blacktriangle}$ +0.78	24/93

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da Bot commented Jul 3, 2026

Test Environment - sonar-java

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	17.64	14.81	$\color{red}{\blacktriangle}$ +2.83	15.72	$\color{red}{\blacktriangle}$ +1.92	29/101
agentless	4.48	24.41	$\color{green}{\blacktriangledown}$ -19.93	17.73	$\color{green}{\blacktriangledown}$ -13.25	29/100
agentlessCodeCoverage	116.33	87.80	$\color{red}{\blacktriangle}$ +28.53	89.58	$\color{red}{\blacktriangle}$ +26.75	29/100
agentlessLineCoverage	167.77	144.77	$\color{red}{\blacktriangle}$ +23.00	141.90	$\color{red}{\blacktriangle}$ +25.87	29/100

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

daniel-mohedano marked this pull request as ready for review

July 3, 2026 09:38

daniel-mohedano requested a review from a team as a code owner

July 3, 2026 09:38

datadog-datadog-prod-us1-2 Bot reviewed

View reviewed changes

datadog-datadog-prod-us1-2 Bot left a comment

Contributor

More details

The fix correctly uses JUnitPlatformUtils.getEngineId() — which walks the UniqueId segment list backward to return the innermost engine-type segment — instead of UniqueId.getEngineId(), which returns the root segment. Validated against 11 UniqueId shapes including all production patterns (plain Cucumber, Suite+Cucumber, Suite+Jupiter, Suite+Spock, no-engine, container variants); every case dispatches to the correct factory. No regressions found.

Was this helpful? React 👍 or 👎

_{📊 Validated against 11 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit 70060ee · What is Autotest? · Any feedback? Reach out in #autotest}

anmarchenko approved these changes

View reviewed changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: ci visibility type: bug