Finish Karate tests after scenario post-processing by daniel-mohedano · Pull Request #12014 · DataDog/dd-trace-java

daniel-mohedano · 2026-07-21T07:48:36Z

What Does This Do

Finishes Karate v2 test spans after ScenarioRuntime.call() completes. This ensures that Karate has applied afterScenario hooks and @fail result inversion before Test Optimization records the result and decides whether to retry.

Expected @fail failures are now reported as passed and are not retried. Failures introduced by afterScenario hooks are reported and remain eligible for retries.

The listener also keeps a fallback completion path when retry instrumentation is disabled.

Motivation

Karate can change a scenario result after SCENARIO_EXIT. Finishing the test at that event could miss hook failures or interpret an expected @fail failures as a real failure.

This caused incorrect test statuses and retry decisions. Removing the event-based completion path entirely could also leave test spans unfinished when retry instrumentation was disabled.

Additional Notes

Includes regression coverage for:

Failing afterScenario hooks and ATR retries.
Expected and unexpected @fail scenarios.
Expected failures not being retried by ATR.
Scenario completion when retry instrumentation is disabled.
EFD not executing retries when execution instrumentation is disabled.

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Once approved, use merge queue to merge the PR

Jira ticket: [PROJ-IDENT]

cit-pr-commenter-54b7da · 2026-07-21T07:57:39Z

Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	55.64	55.43	$\color{red}{\blacktriangle}$ +0.21	55.43	$\color{red}{\blacktriangle}$ +0.21	72/272
agentEvpProxy	54.17	n/a	n/a	n/a	n/a	-

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

datadog-datadog-prod-us1 · 2026-07-21T07:58:10Z

Bits found no code fix to apply

🟢 Investigated · ⚪ No code fix needed

validate_build fails because the generated dd-trace-api JAR differs from the reference artifact, but this branch changes only Karate instrumentation and test files. No code change applies; rerun or investigate the build-artifact reproducibility mismatch.

View in Datadog | Reviewed commit 43d9442

cit-pr-commenter-54b7da · 2026-07-21T08:01:20Z

Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	89.61	87.80	$\color{red}{\blacktriangle}$ +1.81	87.80	$\color{red}{\blacktriangle}$ +1.81	41/142
agentless	82.50	81.05	$\color{red}{\blacktriangle}$ +1.45	81.05	$\color{red}{\blacktriangle}$ +1.45	38/140
agentlessCodeCoverage	98.50	97.04	$\color{red}{\blacktriangle}$ +1.46	97.04	$\color{red}{\blacktriangle}$ +1.46	38/138
agentlessLineCoverage	113.25	111.62	$\color{red}{\blacktriangle}$ +1.63	111.62	$\color{red}{\blacktriangle}$ +1.63	37/137

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-21T08:01:53Z

Test Environment - pass4s

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	10.88	9.73	$\color{red}{\blacktriangle}$ +1.15	10.13	$\color{red}{\blacktriangle}$ +0.75	33/130
agentless	8.94	9.16	$\color{green}{\blacktriangledown}$ -0.22	9.92	$\color{green}{\blacktriangledown}$ -0.98	33/130
agentlessCodeCoverage	16.95	15.41	$\color{red}{\blacktriangle}$ +1.54	17.03	$\color{green}{\blacktriangledown}$ -0.08	32/128

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-21T08:02:57Z

Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	21.82	21.65	$\color{red}{\blacktriangle}$ +0.17	21.65	$\color{red}{\blacktriangle}$ +0.17	41/144
agentless	19.16	18.45	$\color{red}{\blacktriangle}$ +0.71	18.82	$\color{red}{\blacktriangle}$ +0.34	39/139
agentlessCodeCoverage	20.73	19.99	$\color{red}{\blacktriangle}$ +0.74	19.99	$\color{red}{\blacktriangle}$ +0.74	36/135
agentlessLineCoverage	27.17	29.23	$\color{green}{\blacktriangledown}$ -2.06	29.82	$\color{green}{\blacktriangledown}$ -2.65	35/134

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

datadog-datadog-prod-us1 · 2026-07-21T08:03:49Z

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 69.73% (+12.34%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 1d2c764 | Docs | Datadog PR Page | Give us feedback!}

cit-pr-commenter-54b7da · 2026-07-21T08:08:31Z

Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	12.38	13.13	$\color{green}{\blacktriangledown}$ -0.75	13.13	$\color{green}{\blacktriangledown}$ -0.75	36/130
agentless	11.07	12.12	$\color{green}{\blacktriangledown}$ -1.05	12.12	$\color{green}{\blacktriangledown}$ -1.05	34/127
agentlessCodeCoverage	14.12	15.11	$\color{green}{\blacktriangledown}$ -0.99	15.11	$\color{green}{\blacktriangledown}$ -0.99	34/127
agentlessLineCoverage	16.06	18.45	$\color{green}{\blacktriangledown}$ -2.39	18.82	$\color{green}{\blacktriangledown}$ -2.76	34/129

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-21T08:08:52Z

Test Environment - heliboard

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	10.50	9.54	$\color{red}{\blacktriangle}$ +0.96	9.54	$\color{red}{\blacktriangle}$ +0.96	33/77

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

dd-octo-sts · 2026-07-21T08:10:46Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.74 s	14.64 s	[-0.4%; +1.7%] (no difference)
startup:insecure-bank:tracing:Agent	13.65 s	13.74 s	[-1.5%; +0.1%] (no difference)
startup:petclinic:appsec:Agent	16.85 s	16.69 s	[-0.3%; +2.2%] (no difference)
startup:petclinic:iast:Agent	16.24 s	16.89 s	[-8.0%; +0.3%] (no difference)
startup:petclinic:profiling:Agent	16.17 s	16.86 s	[-9.6%; +1.4%] (unstable)
startup:petclinic:sca:Agent	16.88 s	16.85 s	[-0.9%; +1.2%] (no difference)
startup:petclinic:tracing:Agent	15.92 s	16.01 s	[-1.6%; +0.5%] (no difference)

Commit: 1d2c7643 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

cit-pr-commenter-54b7da · 2026-07-21T08:16:11Z

Test Environment - jolokia

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	91.88	93.23	$\color{green}{\blacktriangledown}$ -1.35	93.23	$\color{green}{\blacktriangledown}$ -1.35	36/144
agentless	87.16	89.58	$\color{green}{\blacktriangledown}$ -2.42	89.58	$\color{green}{\blacktriangledown}$ -2.42	37/144
agentlessCodeCoverage	97.45	99.00	$\color{green}{\blacktriangledown}$ -1.55	99.00	$\color{green}{\blacktriangledown}$ -1.55	35/140
agentlessLineCoverage	98.11	99.00	$\color{green}{\blacktriangledown}$ -0.89	99.00	$\color{green}{\blacktriangledown}$ -0.89	34/137

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-21T08:24:12Z

Test Environment - spring_boot

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	16.07	16.36	$\color{green}{\blacktriangledown}$ -0.29	16.36	$\color{green}{\blacktriangledown}$ -0.29	35/129
agentless	9.33	9.73	$\color{green}{\blacktriangledown}$ -0.40	9.73	$\color{green}{\blacktriangledown}$ -0.40	35/130
agentlessCodeCoverage	13.45	13.40	$\color{red}{\blacktriangle}$ +0.05	13.40	$\color{red}{\blacktriangle}$ +0.05	36/131
agentlessLineCoverage	21.90	32.30	$\color{green}{\blacktriangledown}$ -10.40	32.95	$\color{green}{\blacktriangledown}$ -11.05	36/131

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-21T08:24:13Z

Test Environment - okhttp

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	19.45	19.59	$\color{green}{\blacktriangledown}$ -0.14	19.20	$\color{red}{\blacktriangle}$ +0.25	31/138
agentless	19.06	19.20	$\color{green}{\blacktriangledown}$ -0.14	19.20	$\color{green}{\blacktriangledown}$ -0.14	31/135
agentlessCodeCoverage	22.94	22.54	$\color{red}{\blacktriangle}$ +0.40	22.54	$\color{red}{\blacktriangle}$ +0.40	31/134
agentlessLineCoverage	45.55	44.48	$\color{red}{\blacktriangle}$ +1.07	44.48	$\color{red}{\blacktriangle}$ +1.07	32/141

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

… daniel.mohedano/karate-finish-after-post-processing

cit-pr-commenter-54b7da · 2026-07-21T09:50:57Z

Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	36.38	36.42	$\color{green}{\blacktriangledown}$ -0.04	36.42	$\color{green}{\blacktriangledown}$ -0.04	35/138
agentless	36.01	36.42	$\color{green}{\blacktriangledown}$ -0.41	36.42	$\color{green}{\blacktriangledown}$ -0.41	35/138
agentlessCodeCoverage	44.41	44.48	$\color{green}{\blacktriangledown}$ -0.07	44.48	$\color{green}{\blacktriangledown}$ -0.07	35/138
agentlessLineCoverage	55.42	74.82	$\color{green}{\blacktriangledown}$ -19.40	74.82	$\color{green}{\blacktriangledown}$ -19.40	35/138

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-21T10:32:21Z

Test Environment - sonar-java

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	12.20	11.65	$\color{red}{\blacktriangle}$ +0.55	12.87	$\color{green}{\blacktriangledown}$ -0.67	36/137
agentless	51.89	16.04	$\color{red}{\blacktriangle}$ +35.85	16.04	$\color{red}{\blacktriangle}$ +35.85	36/136
agentlessCodeCoverage	128.65	84.36	$\color{red}{\blacktriangle}$ +44.29	84.36	$\color{red}{\blacktriangle}$ +44.29	35/135
agentlessLineCoverage	171.50	141.90	$\color{red}{\blacktriangle}$ +29.60	139.09	$\color{red}{\blacktriangle}$ +32.41	35/135

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

datadog-datadog-prod-us1

More details

The !scenario.isFail() guard added to SuppressErrorAdvice is the load-bearing fix: without it, expected @fail step errors were silently moved to the context and the step was replaced with a skipped one, causing Karate's result inversion to fire incorrectly (no failing step → Karate treated scenario as "passed" → @fail inverted to FAIL). Moving afterScenario completion to the advice (post-call()) ensures the retry policy always sees the fully post-processed result. Thirteen adversarial scenarios including the old-versus-new @fail+ATR path all pass; the fixture evidence confirms a single PASS span for the expected-failure scenario (no spurious retry).

Was this helpful? React 👍 or 👎

_{📊 Validated against 13 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit 5622501 · What is Autotest? · Any feedback? Reach out in #autotest}

…rocessing # Conflicts: # dd-java-agent/instrumentation/karate/karate-2.0/src/main/java21/datadog/trace/instrumentation/karate2/KarateScenarioAdvice.java

daniel-mohedano added 2 commits July 21, 2026 09:31

fix: avoid retrying untracked karate scenarios

3a1eec0

fix: finish karate tests after scenario post-processing

43d9442

daniel-mohedano added type: bug fix Bug fix comp: ci visibility Continuous Integration Visibility labels Jul 21, 2026

daniel-mohedano added 2 commits July 21, 2026 11:30

fix: merge test scenarios

42b9368

Merge branch 'daniel.mohedano/karate-ignore-untracked-scenarios' into…

d0a8d0a

… daniel.mohedano/karate-finish-after-post-processing

fix: honor expected karate failures

5622501

daniel-mohedano marked this pull request as ready for review July 21, 2026 10:56

daniel-mohedano requested a review from a team as a code owner July 21, 2026 10:56

datadog-datadog-prod-us1 Bot reviewed Jul 21, 2026

View reviewed changes

daniel-mohedano changed the title ~~Finish karate tests after scenario post-processing~~ Finish Karate tests after scenario post-processing Jul 21, 2026

juan-fernandez approved these changes Jul 21, 2026

View reviewed changes

Base automatically changed from daniel.mohedano/karate-ignore-untracked-scenarios to master July 22, 2026 13:36

Merge branch 'master' into daniel.mohedano/karate-finish-after-post-p…

1d2c764

…rocessing # Conflicts: # dd-java-agent/instrumentation/karate/karate-2.0/src/main/java21/datadog/trace/instrumentation/karate2/KarateScenarioAdvice.java

daniel-mohedano added the tag: override groovy enforcement Override the "Enforce Groovy Migration" check label Jul 22, 2026

Uh oh!

Conversation

daniel-mohedano commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - sbt-scalatest

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 21, 2026 • edited by datadog-datadog-us1-prod Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bits found no code fix to apply

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - netflix-zuul

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - pass4s

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - reactive-streams-jvm

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 21, 2026 • edited by datadog-datadog-prod-us1-2 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - sonar-kotlin

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - heliboard

Uh oh!

dd-octo-sts Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - jolokia

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - spring_boot

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - okhttp

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - nebula-release-plugin

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - sonar-java

Uh oh!

datadog-datadog-prod-us1 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

daniel-mohedano commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Jul 21, 2026 •

edited by datadog-datadog-us1-prod Bot

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Jul 21, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

dd-octo-sts Bot commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading