Skip to content

Stop retrying aborted Karate scenarios#12015

Open
daniel-mohedano wants to merge 4 commits into
daniel.mohedano/karate-finish-after-post-processingfrom
daniel.mohedano/karate-stop-aborted-retries
Open

Stop retrying aborted Karate scenarios#12015
daniel-mohedano wants to merge 4 commits into
daniel.mohedano/karate-finish-after-post-processingfrom
daniel.mohedano/karate-stop-aborted-retries

Conversation

@daniel-mohedano

@daniel-mohedano daniel-mohedano commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What Does This Do

Traces whether a Karate 2 scenario emitted SCENARIO_ENTER. Scenarios that return before entering are not finished or retried.

Motivation

With abortSuiteOnFailure enabled, ScenarioRuntime.call() can return before the scenario starts. The retry advice previously attempted to finish a test that did not exist.

Since the execution policy was not advanced, it remained applicable and the retry loop could repeatedly execute the aborted scenario without terminating.

Additional Notes

Includes regression coverage for an aborted suite during retry handling.

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@daniel-mohedano daniel-mohedano added type: bug fix Bug fix comp: ci visibility Continuous Integration Visibility labels Jul 21, 2026
@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 55.62 55.43 $\color{red}{\blacktriangle}$ +0.19 55.43 $\color{red}{\blacktriangle}$ +0.19 64/270
agentEvpProxy 55.02 n/a n/a n/a n/a -

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 89.16 87.80 $\color{red}{\blacktriangle}$ +1.36 87.80 $\color{red}{\blacktriangle}$ +1.36 37/141
agentless 81.72 81.05 $\color{red}{\blacktriangle}$ +0.67 81.05 $\color{red}{\blacktriangle}$ +0.67 34/139
agentlessCodeCoverage 96.19 97.04 $\color{green}{\blacktriangledown}$ -0.85 97.04 $\color{green}{\blacktriangledown}$ -0.85 34/137
agentlessLineCoverage 112.44 111.62 $\color{red}{\blacktriangle}$ +0.82 111.62 $\color{red}{\blacktriangle}$ +0.82 33/136

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@datadog-datadog-prod-us1-2

This comment has been minimized.

Sign in to view
@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 36.44 36.42 $\color{red}{\blacktriangle}$ +0.02 36.42 $\color{red}{\blacktriangle}$ +0.02 33/135
agentless 36.32 36.42 $\color{green}{\blacktriangledown}$ -0.10 36.42 $\color{green}{\blacktriangledown}$ -0.10 33/135
agentlessCodeCoverage 44.16 44.48 $\color{green}{\blacktriangledown}$ -0.32 44.48 $\color{green}{\blacktriangledown}$ -0.32 33/135
agentlessLineCoverage 74.44 74.82 $\color{green}{\blacktriangledown}$ -0.38 74.82 $\color{green}{\blacktriangledown}$ -0.38 33/135

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - pass4s

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 12.14 9.54 $\color{red}{\blacktriangle}$ +2.60 10.13 $\color{red}{\blacktriangle}$ +2.01 29/127
agentless 9.78 9.73 $\color{red}{\blacktriangle}$ +0.05 10.13 $\color{green}{\blacktriangledown}$ -0.35 29/127
agentlessCodeCoverage 15.72 15.11 $\color{red}{\blacktriangle}$ +0.61 17.03 $\color{green}{\blacktriangledown}$ -1.31 28/125

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - heliboard

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 8.58 9.54 $\color{green}{\blacktriangledown}$ -0.96 9.54 $\color{green}{\blacktriangledown}$ -0.96 32/73

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 23.71 21.65 $\color{red}{\blacktriangle}$ +2.06 21.65 $\color{red}{\blacktriangle}$ +2.06 42/145
agentless 19.44 18.45 $\color{red}{\blacktriangle}$ +0.99 18.82 $\color{red}{\blacktriangle}$ +0.62 38/139
agentlessCodeCoverage 19.51 19.99 $\color{green}{\blacktriangledown}$ -0.48 19.99 $\color{green}{\blacktriangledown}$ -0.48 36/136
agentlessLineCoverage 29.51 29.23 $\color{red}{\blacktriangle}$ +0.28 29.82 $\color{green}{\blacktriangledown}$ -0.31 35/135

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 11.46 13.13 $\color{green}{\blacktriangledown}$ -1.67 13.13 $\color{green}{\blacktriangledown}$ -1.67 33/129
agentless 11.45 12.12 $\color{green}{\blacktriangledown}$ -0.67 12.12 $\color{green}{\blacktriangledown}$ -0.67 31/126
agentlessCodeCoverage 16.17 15.11 $\color{red}{\blacktriangle}$ +1.06 15.11 $\color{red}{\blacktriangle}$ +1.06 31/126
agentlessLineCoverage 17.33 18.45 $\color{green}{\blacktriangledown}$ -1.12 18.82 $\color{green}{\blacktriangledown}$ -1.49 32/128

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@dd-octo-sts

dd-octo-sts Bot commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.02 s 14.05 s [-1.1%; +0.7%] (no difference)
startup:insecure-bank:tracing:Agent 13.05 s 13.00 s [-0.6%; +1.4%] (no difference)
startup:petclinic:appsec:Agent 16.80 s 16.68 s [-0.5%; +1.9%] (no difference)
startup:petclinic:iast:Agent 16.33 s 16.46 s [-7.0%; +5.5%] (unstable)
startup:petclinic:profiling:Agent 16.70 s 16.78 s [-1.7%; +0.8%] (no difference)
startup:petclinic:sca:Agent 16.88 s 16.78 s [-0.3%; +1.4%] (no difference)
startup:petclinic:tracing:Agent 15.97 s 16.10 s [-1.7%; +0.1%] (no difference)

Commit: 2f857865 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - jolokia

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 94.66 93.23 $\color{red}{\blacktriangle}$ +1.43 93.23 $\color{red}{\blacktriangle}$ +1.43 37/145
agentless 91.28 89.58 $\color{red}{\blacktriangle}$ +1.70 89.58 $\color{red}{\blacktriangle}$ +1.70 39/144
agentlessCodeCoverage 100.65 99.00 $\color{red}{\blacktriangle}$ +1.65 99.00 $\color{red}{\blacktriangle}$ +1.65 36/140
agentlessLineCoverage 101.68 99.00 $\color{red}{\blacktriangle}$ +2.68 99.00 $\color{red}{\blacktriangle}$ +2.68 35/137

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - okhttp

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 19.35 19.59 $\color{green}{\blacktriangledown}$ -0.24 19.20 $\color{red}{\blacktriangle}$ +0.15 32/139
agentless 19.64 19.20 $\color{red}{\blacktriangle}$ +0.44 19.20 $\color{red}{\blacktriangle}$ +0.44 32/136
agentlessCodeCoverage 24.70 22.54 $\color{red}{\blacktriangle}$ +2.16 22.54 $\color{red}{\blacktriangle}$ +2.16 32/135
agentlessLineCoverage 44.79 44.48 $\color{red}{\blacktriangle}$ +0.31 44.48 $\color{red}{\blacktriangle}$ +0.31 33/142

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - spring_boot

Job Status: 🟢 success

Scenario Overhead (%)
agent 15.50
agentless 9.31
agentlessCodeCoverage 13.12
agentlessLineCoverage 32.47

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - sonar-java

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 15.17 8.46 $\color{red}{\blacktriangle}$ +6.71 12.87 $\color{red}{\blacktriangle}$ +2.30 36/137
agentless 46.53 11.65 $\color{red}{\blacktriangle}$ +34.88 15.72 $\color{red}{\blacktriangle}$ +30.81 36/136
agentlessCodeCoverage 110.65 82.69 $\color{red}{\blacktriangle}$ +27.96 84.36 $\color{red}{\blacktriangle}$ +26.29 36/136
agentlessLineCoverage 187.64 136.34 $\color{red}{\blacktriangle}$ +51.30 136.34 $\color{red}{\blacktriangle}$ +51.30 36/136

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@daniel-mohedano
daniel-mohedano marked this pull request as ready for review July 21, 2026 10:56
@daniel-mohedano
daniel-mohedano requested a review from a team as a code owner July 21, 2026 10:56
datadog-datadog-prod-us1-2[bot]
datadog-datadog-prod-us1-2 Bot reviewed Jul 21, 2026
View reviewed changes

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

The PR introduces sound defensive tracking to prevent infinite retry loops when scenarios abort before starting. By recording whether SCENARIO_ENTER fires, the code now correctly skips retry processing for aborted scenarios where the execution policy wouldn't advance. The fix is minimal, targeted, and preserves existing behavior for normal scenarios while fixing the infinite loop on abort.

Was this helpful? React 👍 or 👎

📊 Validated against 4 scenarios · Open Bits AI session

🤖 Datadog Autotest · Commit 2f85786 · What is Autotest? · Any feedback? Reach out in #autotest

@daniel-mohedano daniel-mohedano changed the title Stop retrying aborted karate scenarios Stop retrying aborted Karate scenarios Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2[bot] datadog-datadog-prod-us1-2[bot] left review comments
@juan-fernandez juan-fernandez juan-fernandez approved these changes

Assignees

No one assigned

Labels

comp: ci visibility Continuous Integration Visibility type: bug fix Bug fix

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@daniel-mohedano @juan-fernandez