Skip to content

Fix TestNG session status reported with EFD#11087

Open
daniel-mohedano wants to merge 4 commits intomasterfrom
daniel.mohedano/testng-failure-propagation
Open

Fix TestNG session status reported with EFD#11087
daniel-mohedano wants to merge 4 commits intomasterfrom
daniel.mohedano/testng-failure-propagation

Conversation

@daniel-mohedano
Copy link
Copy Markdown
Contributor

@daniel-mohedano daniel-mohedano commented Apr 13, 2026 •
edited
Loading

What Does This Do

  • Fixes a bug that would make the test session status dependent of the result orders when using EFD with TestNG:
    • If the result order was pass and fail, the test session was correctly failed.
    • If the result order was fail and pass, the test session would incorrectly pass, while EFD should fail the session when a flaky is found.
  • This was caused by TestNG's retry analyzer suppressing failures on retried tests automatically, which would only trigger if the first test execution was a fail.
  • This bug only affects EFD as it depends on tests being retried after a failure and the failures not being suppressed:
    • ATR suppresses failures by default
    • Attempt to Fix will not retry a test if the first execution is a failure

Motivation

Aligns EFD behavior for TestNG with the rest of the testing frameworks.

Additional Information

Two unit tests have been introduced to verify the behavior.

Contributor Checklist

Jira ticket: SDTEST-3720

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

@daniel-mohedano daniel-mohedano added type: bug Bug report and fix comp: ci visibility Continuous Integration Visibility labels Apr 13, 2026
@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - sbt-scalatest

Job Status: success

Scenario Overhead (%)
agent 56.52
agentEvpProxy 56.73

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - nebula-release-plugin

Job Status: success

Scenario Overhead (%)
agent 37.84
agentless 37.47
agentlessCodeCoverage 45.33
agentlessLineCoverage 76.99

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - netflix-zuul

Job Status: success

Scenario Overhead (%)
agent 87.89
agentless 80.97
agentlessCodeCoverage 98.70
agentlessLineCoverage 115.42

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - reactive-streams-jvm

Job Status: success

Scenario Overhead (%)
agent 20.94
agentless 19.44
agentlessCodeCoverage 20.38
agentlessLineCoverage 29.97

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - pass4s

Job Status: success

Scenario Overhead (%)
agent 11.17
agentless 8.96
agentlessCodeCoverage 19.84

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - sonar-kotlin

Job Status: success

Scenario Overhead (%)
agent 14.72
agentless 12.42
agentlessCodeCoverage 16.93
agentlessLineCoverage 19.85

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - jolokia

Job Status: success

Scenario Overhead (%)
agent 93.79
agentless 88.99
agentlessCodeCoverage 97.86
agentlessLineCoverage 99.92

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - okhttp

Job Status: success

Scenario Overhead (%)
agent 19.38
agentless 18.23
agentlessCodeCoverage 22.72
agentlessLineCoverage 44.60

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - spring_boot

Job Status: success

Scenario Overhead (%)
agent 15.73
agentless 9.82
agentlessCodeCoverage 13.75
agentlessLineCoverage 32.84

@pr-commenter
Copy link
Copy Markdown

pr-commenter bot commented Apr 13, 2026 •
edited
Loading

Benchmarks

⚠️ Warning: Baseline build not found for merge-base commit. Comparing against the latest commit on master instead.

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master daniel.mohedano/testng-failure-propagation
git_commit_date 1776158162 1776160067
git_commit_sha 16f8277 bc787c0
release_version 1.62.0-SNAPSHOT~16f8277b2e 1.62.0-SNAPSHOT~bc787c05dc
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1776161895 1776161895
ci_job_id 1592671496 1592671496
ci_pipeline_id 107541390 107541390
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-4iv7qok5 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-4iv7qok5 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 60 metrics, 11 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.62.0-SNAPSHOT~bc787c05dc, baseline=1.62.0-SNAPSHOT~16f8277b2e

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.055 s) : 0, 1054592
Total [baseline] (8.848 s) : 0, 8848133
Agent [candidate] (1.057 s) : 0, 1056684
Total [candidate] (8.842 s) : 0, 8841607
section iast
Agent [baseline] (1.223 s) : 0, 1222872
Total [baseline] (9.635 s) : 0, 9635190
Agent [candidate] (1.224 s) : 0, 1224220
Total [candidate] (9.575 s) : 0, 9575242
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.055 s -
Agent iast 1.223 s 168.28 ms (16.0%)
Total tracing 8.848 s -
Total iast 9.635 s 787.057 ms (8.9%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.057 s -
Agent iast 1.224 s 167.536 ms (15.9%)
Total tracing 8.842 s -
Total iast 9.575 s 733.635 ms (8.3%) 
gantt
    title insecure-bank - break down per module: candidate=1.62.0-SNAPSHOT~bc787c05dc, baseline=1.62.0-SNAPSHOT~16f8277b2e

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.233 ms) : 0, 1233
crashtracking [candidate] (1.225 ms) : 0, 1225
BytebuddyAgent [baseline] (632.43 ms) : 0, 632430
BytebuddyAgent [candidate] (633.847 ms) : 0, 633847
AgentMeter [baseline] (29.365 ms) : 0, 29365
AgentMeter [candidate] (29.28 ms) : 0, 29280
GlobalTracer [baseline] (248.497 ms) : 0, 248497
GlobalTracer [candidate] (248.317 ms) : 0, 248317
AppSec [baseline] (32.01 ms) : 0, 32010
AppSec [candidate] (31.958 ms) : 0, 31958
Debugger [baseline] (59.389 ms) : 0, 59389
Debugger [candidate] (59.06 ms) : 0, 59060
Remote Config [baseline] (602.472 µs) : 0, 602
Remote Config [candidate] (584.167 µs) : 0, 584
Telemetry [baseline] (8.071 ms) : 0, 8071
Telemetry [candidate] (8.001 ms) : 0, 8001
Flare Poller [baseline] (6.767 ms) : 0, 6767
Flare Poller [candidate] (8.176 ms) : 0, 8176
section iast
crashtracking [baseline] (1.246 ms) : 0, 1246
crashtracking [candidate] (1.223 ms) : 0, 1223
BytebuddyAgent [baseline] (800.974 ms) : 0, 800974
BytebuddyAgent [candidate] (801.402 ms) : 0, 801402
AgentMeter [baseline] (11.358 ms) : 0, 11358
AgentMeter [candidate] (11.393 ms) : 0, 11393
GlobalTracer [baseline] (239.021 ms) : 0, 239021
GlobalTracer [candidate] (239.218 ms) : 0, 239218
AppSec [baseline] (32.581 ms) : 0, 32581
AppSec [candidate] (30.263 ms) : 0, 30263
Debugger [baseline] (59.239 ms) : 0, 59239
Debugger [candidate] (59.933 ms) : 0, 59933
Remote Config [baseline] (540.39 µs) : 0, 540
Remote Config [candidate] (531.463 µs) : 0, 531
Telemetry [baseline] (12.373 ms) : 0, 12373
Telemetry [candidate] (14.271 ms) : 0, 14271
Flare Poller [baseline] (3.466 ms) : 0, 3466
Flare Poller [candidate] (3.606 ms) : 0, 3606
IAST [baseline] (25.848 ms) : 0, 25848
IAST [candidate] (25.903 ms) : 0, 25903
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.62.0-SNAPSHOT~bc787c05dc, baseline=1.62.0-SNAPSHOT~16f8277b2e

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.059 s) : 0, 1058809
Total [baseline] (11.093 s) : 0, 11093174
Agent [candidate] (1.056 s) : 0, 1056323
Total [candidate] (11.081 s) : 0, 11081483
section appsec
Agent [baseline] (1.248 s) : 0, 1247814
Total [baseline] (11.153 s) : 0, 11153392
Agent [candidate] (1.247 s) : 0, 1246683
Total [candidate] (11.154 s) : 0, 11154256
section iast
Agent [baseline] (1.223 s) : 0, 1222775
Total [baseline] (11.336 s) : 0, 11336171
Agent [candidate] (1.225 s) : 0, 1225121
Total [candidate] (11.194 s) : 0, 11193659
section profiling
Agent [baseline] (1.184 s) : 0, 1183807
Total [baseline] (11.079 s) : 0, 11078658
Agent [candidate] (1.192 s) : 0, 1191724
Total [candidate] (11.301 s) : 0, 11300809
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.059 s -
Agent appsec 1.248 s 189.005 ms (17.9%)
Agent iast 1.223 s 163.965 ms (15.5%)
Agent profiling 1.184 s 124.998 ms (11.8%)
Total tracing 11.093 s -
Total appsec 11.153 s 60.218 ms (0.5%)
Total iast 11.336 s 242.996 ms (2.2%)
Total profiling 11.079 s -14.516 ms (-0.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.056 s -
Agent appsec 1.247 s 190.361 ms (18.0%)
Agent iast 1.225 s 168.798 ms (16.0%)
Agent profiling 1.192 s 135.402 ms (12.8%)
Total tracing 11.081 s -
Total appsec 11.154 s 72.773 ms (0.7%)
Total iast 11.194 s 112.176 ms (1.0%)
Total profiling 11.301 s 219.327 ms (2.0%) 
gantt
    title petclinic - break down per module: candidate=1.62.0-SNAPSHOT~bc787c05dc, baseline=1.62.0-SNAPSHOT~16f8277b2e

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.241 ms) : 0, 1241
crashtracking [candidate] (1.22 ms) : 0, 1220
BytebuddyAgent [baseline] (633.345 ms) : 0, 633345
BytebuddyAgent [candidate] (632.732 ms) : 0, 632732
AgentMeter [baseline] (29.474 ms) : 0, 29474
AgentMeter [candidate] (29.327 ms) : 0, 29327
GlobalTracer [baseline] (249.071 ms) : 0, 249071
GlobalTracer [candidate] (248.492 ms) : 0, 248492
AppSec [baseline] (32.155 ms) : 0, 32155
AppSec [candidate] (32.007 ms) : 0, 32007
Debugger [baseline] (60.417 ms) : 0, 60417
Debugger [candidate] (60.104 ms) : 0, 60104
Remote Config [baseline] (607.256 µs) : 0, 607
Remote Config [candidate] (595.815 µs) : 0, 596
Telemetry [baseline] (8.131 ms) : 0, 8131
Telemetry [candidate] (8.147 ms) : 0, 8147
Flare Poller [baseline] (8.202 ms) : 0, 8202
Flare Poller [candidate] (7.606 ms) : 0, 7606
section appsec
crashtracking [baseline] (1.217 ms) : 0, 1217
crashtracking [candidate] (1.216 ms) : 0, 1216
BytebuddyAgent [baseline] (661.687 ms) : 0, 661687
BytebuddyAgent [candidate] (660.875 ms) : 0, 660875
AgentMeter [baseline] (12.044 ms) : 0, 12044
AgentMeter [candidate] (12.061 ms) : 0, 12061
GlobalTracer [baseline] (249.168 ms) : 0, 249168
GlobalTracer [candidate] (248.737 ms) : 0, 248737
AppSec [baseline] (184.401 ms) : 0, 184401
AppSec [candidate] (184.295 ms) : 0, 184295
Debugger [baseline] (65.732 ms) : 0, 65732
Debugger [candidate] (65.855 ms) : 0, 65855
Remote Config [baseline] (609.279 µs) : 0, 609
Remote Config [candidate] (592.289 µs) : 0, 592
Telemetry [baseline] (8.591 ms) : 0, 8591
Telemetry [candidate] (8.631 ms) : 0, 8631
Flare Poller [baseline] (3.535 ms) : 0, 3535
Flare Poller [candidate] (3.556 ms) : 0, 3556
IAST [baseline] (24.479 ms) : 0, 24479
IAST [candidate] (24.538 ms) : 0, 24538
section iast
crashtracking [baseline] (1.228 ms) : 0, 1228
crashtracking [candidate] (1.223 ms) : 0, 1223
BytebuddyAgent [baseline] (799.99 ms) : 0, 799990
BytebuddyAgent [candidate] (801.12 ms) : 0, 801120
AgentMeter [baseline] (11.372 ms) : 0, 11372
AgentMeter [candidate] (11.383 ms) : 0, 11383
GlobalTracer [baseline] (238.794 ms) : 0, 238794
GlobalTracer [candidate] (239.402 ms) : 0, 239402
AppSec [baseline] (31.104 ms) : 0, 31104
AppSec [candidate] (31.914 ms) : 0, 31914
Debugger [baseline] (60.168 ms) : 0, 60168
Debugger [candidate] (61.007 ms) : 0, 61007
Remote Config [baseline] (1.133 ms) : 0, 1133
Remote Config [candidate] (543.048 µs) : 0, 543
Telemetry [baseline] (13.378 ms) : 0, 13378
Telemetry [candidate] (12.594 ms) : 0, 12594
Flare Poller [baseline] (3.435 ms) : 0, 3435
Flare Poller [candidate] (3.479 ms) : 0, 3479
IAST [baseline] (25.865 ms) : 0, 25865
IAST [candidate] (25.852 ms) : 0, 25852
section profiling
crashtracking [baseline] (1.19 ms) : 0, 1190
crashtracking [candidate] (1.174 ms) : 0, 1174
BytebuddyAgent [baseline] (691.154 ms) : 0, 691154
BytebuddyAgent [candidate] (695.861 ms) : 0, 695861
AgentMeter [baseline] (9.114 ms) : 0, 9114
AgentMeter [candidate] (9.196 ms) : 0, 9196
GlobalTracer [baseline] (207.088 ms) : 0, 207088
GlobalTracer [candidate] (208.47 ms) : 0, 208470
AppSec [baseline] (32.553 ms) : 0, 32553
AppSec [candidate] (32.785 ms) : 0, 32785
Debugger [baseline] (64.884 ms) : 0, 64884
Debugger [candidate] (66.224 ms) : 0, 66224
Remote Config [baseline] (586.063 µs) : 0, 586
Remote Config [candidate] (573.368 µs) : 0, 573
Telemetry [baseline] (8.624 ms) : 0, 8624
Telemetry [candidate] (7.903 ms) : 0, 7903
Flare Poller [baseline] (3.537 ms) : 0, 3537
Flare Poller [candidate] (3.624 ms) : 0, 3624
ProfilingAgent [baseline] (93.849 ms) : 0, 93849
ProfilingAgent [candidate] (94.568 ms) : 0, 94568
Profiling [baseline] (94.412 ms) : 0, 94412
Profiling [candidate] (95.134 ms) : 0, 95134
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master daniel.mohedano/testng-failure-propagation
git_commit_date 1776158162 1776160067
git_commit_sha 16f8277 bc787c0
release_version 1.62.0-SNAPSHOT~16f8277b2e 1.62.0-SNAPSHOT~bc787c05dc
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1776162337 1776162337
ci_job_id 1592671497 1592671497
ci_pipeline_id 107541390 107541390
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-760gsm05 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-760gsm05 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 2 performance improvements and 2 performance regressions! Performance is the same for 16 metrics, 16 unstable metrics.

scenario Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p95 Δ mean throughput candidate mean agg_http_req_duration_p50 candidate mean agg_http_req_duration_p95 candidate mean throughput baseline mean agg_http_req_duration_p50 baseline mean agg_http_req_duration_p95 baseline mean throughput
scenario:load:insecure-bank:iast:high_load worse
[+121.997µs; +279.818µs] or [+4.625%; +10.609%]		 same
[-26.618µs; +565.274µs] or [-0.342%; +7.265%]		 unstable
[-227.094op/s; +74.094op/s] or [-16.797%; +5.480%]		 2.838ms 8.050ms 1275.469op/s 2.638ms 7.781ms 1351.969op/s
scenario:load:petclinic:no_agent:high_load better
[-3.038ms; -1.829ms] or [-15.999%; -9.634%]		 better
[-4.357ms; -1.582ms] or [-13.997%; -5.081%]		 unstable
[+3.563op/s; +62.000op/s] or [+1.486%; +25.864%]		 16.554ms 28.159ms 272.500op/s 18.987ms 31.128ms 239.719op/s
scenario:load:petclinic:profiling:high_load worse
[+0.957ms; +1.971ms] or [+5.422%; +11.167%]		 unsure
[+0.581ms; +2.234ms] or [+1.990%; +7.657%]		 unstable
[-44.530op/s; +10.655op/s] or [-17.251%; +4.128%]		 19.115ms 30.586ms 241.188op/s 17.651ms 29.179ms 258.125op/s
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~bc787c05dc, baseline=1.62.0-SNAPSHOT~16f8277b2e
    dateFormat X
    axisFormat %s
section baseline
no_agent (19.472 ms) : 19275, 19670
.   : milestone, 19472,
appsec (18.707 ms) : 18520, 18894
.   : milestone, 18707,
code_origins (18.08 ms) : 17903, 18257
.   : milestone, 18080,
iast (18.018 ms) : 17838, 18199
.   : milestone, 18018,
profiling (18.075 ms) : 17894, 18255
.   : milestone, 18075,
tracing (17.739 ms) : 17564, 17914
.   : milestone, 17739,
section candidate
no_agent (17.119 ms) : 16950, 17287
.   : milestone, 17119,
appsec (19.031 ms) : 18837, 19226
.   : milestone, 19031,
code_origins (17.917 ms) : 17739, 18095
.   : milestone, 17917,
iast (17.851 ms) : 17673, 18028
.   : milestone, 17851,
profiling (19.356 ms) : 19161, 19552
.   : milestone, 19356,
tracing (17.683 ms) : 17511, 17855
.   : milestone, 17683,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 19.472 ms [19.275 ms, 19.67 ms] -
appsec 18.707 ms [18.52 ms, 18.894 ms] -765.333 µs (-3.9%)
code_origins 18.08 ms [17.903 ms, 18.257 ms] -1.392 ms (-7.1%)
iast 18.018 ms [17.838 ms, 18.199 ms] -1.454 ms (-7.5%)
profiling 18.075 ms [17.894 ms, 18.255 ms] -1.398 ms (-7.2%)
tracing 17.739 ms [17.564 ms, 17.914 ms] -1.733 ms (-8.9%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 17.119 ms [16.95 ms, 17.287 ms] -
appsec 19.031 ms [18.837 ms, 19.226 ms] 1.912 ms (11.2%)
code_origins 17.917 ms [17.739 ms, 18.095 ms] 798.165 µs (4.7%)
iast 17.851 ms [17.673 ms, 18.028 ms] 732.202 µs (4.3%)
profiling 19.356 ms [19.161 ms, 19.552 ms] 2.238 ms (13.1%)
tracing 17.683 ms [17.511 ms, 17.855 ms] 564.406 µs (3.3%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~bc787c05dc, baseline=1.62.0-SNAPSHOT~16f8277b2e
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.272 ms) : 1260, 1284
.   : milestone, 1272,
iast (3.387 ms) : 3342, 3432
.   : milestone, 3387,
iast_FULL (6.075 ms) : 6012, 6137
.   : milestone, 6075,
iast_GLOBAL (3.708 ms) : 3656, 3760
.   : milestone, 3708,
profiling (2.167 ms) : 2147, 2187
.   : milestone, 2167,
tracing (1.862 ms) : 1847, 1877
.   : milestone, 1862,
section candidate
no_agent (1.256 ms) : 1244, 1269
.   : milestone, 1256,
iast (3.595 ms) : 3543, 3647
.   : milestone, 3595,
iast_FULL (6.01 ms) : 5950, 6071
.   : milestone, 6010,
iast_GLOBAL (3.782 ms) : 3720, 3845
.   : milestone, 3782,
profiling (2.227 ms) : 2206, 2247
.   : milestone, 2227,
tracing (1.931 ms) : 1912, 1950
.   : milestone, 1931,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.272 ms [1.26 ms, 1.284 ms] -
iast 3.387 ms [3.342 ms, 3.432 ms] 2.115 ms (166.2%)
iast_FULL 6.075 ms [6.012 ms, 6.137 ms] 4.802 ms (377.5%)
iast_GLOBAL 3.708 ms [3.656 ms, 3.76 ms] 2.436 ms (191.4%)
profiling 2.167 ms [2.147 ms, 2.187 ms] 894.264 µs (70.3%)
tracing 1.862 ms [1.847 ms, 1.877 ms] 589.692 µs (46.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.256 ms [1.244 ms, 1.269 ms] -
iast 3.595 ms [3.543 ms, 3.647 ms] 2.339 ms (186.2%)
iast_FULL 6.01 ms [5.95 ms, 6.071 ms] 4.754 ms (378.5%)
iast_GLOBAL 3.782 ms [3.72 ms, 3.845 ms] 2.526 ms (201.1%)
profiling 2.227 ms [2.206 ms, 2.247 ms] 970.388 µs (77.3%)
tracing 1.931 ms [1.912 ms, 1.95 ms] 674.746 µs (53.7%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master daniel.mohedano/testng-failure-propagation
git_commit_date 1776158162 1776160067
git_commit_sha 16f8277 bc787c0
release_version 1.62.0-SNAPSHOT~16f8277b2e 1.62.0-SNAPSHOT~bc787c05dc
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1776162114 1776162114
ci_job_id 1592671498 1592671498
ci_pipeline_id 107541390 107541390
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-o891n5zk 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-o891n5zk 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 0 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:dacapo:tomcat:appsec better
[-1.452ms; -1.101ms] or [-38.042%; -28.840%]		 2.541ms 3.818ms
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~bc787c05dc, baseline=1.62.0-SNAPSHOT~16f8277b2e
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.484 ms) : 1473, 1496
.   : milestone, 1484,
appsec (3.818 ms) : 3594, 4042
.   : milestone, 3818,
iast (2.276 ms) : 2206, 2345
.   : milestone, 2276,
iast_GLOBAL (2.319 ms) : 2250, 2389
.   : milestone, 2319,
profiling (2.106 ms) : 2051, 2161
.   : milestone, 2106,
tracing (2.085 ms) : 2031, 2139
.   : milestone, 2085,
section candidate
no_agent (1.483 ms) : 1471, 1494
.   : milestone, 1483,
appsec (2.541 ms) : 2486, 2596
.   : milestone, 2541,
iast (2.27 ms) : 2200, 2339
.   : milestone, 2270,
iast_GLOBAL (2.308 ms) : 2239, 2378
.   : milestone, 2308,
profiling (2.099 ms) : 2045, 2154
.   : milestone, 2099,
tracing (2.095 ms) : 2040, 2149
.   : milestone, 2095,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.484 ms [1.473 ms, 1.496 ms] -
appsec 3.818 ms [3.594 ms, 4.042 ms] 2.334 ms (157.2%)
iast 2.276 ms [2.206 ms, 2.345 ms] 791.434 µs (53.3%)
iast_GLOBAL 2.319 ms [2.25 ms, 2.389 ms] 834.824 µs (56.2%)
profiling 2.106 ms [2.051 ms, 2.161 ms] 622.027 µs (41.9%)
tracing 2.085 ms [2.031 ms, 2.139 ms] 600.691 µs (40.5%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.483 ms [1.471 ms, 1.494 ms] -
appsec 2.541 ms [2.486 ms, 2.596 ms] 1.059 ms (71.4%)
iast 2.27 ms [2.2 ms, 2.339 ms] 787.0 µs (53.1%)
iast_GLOBAL 2.308 ms [2.239 ms, 2.378 ms] 825.714 µs (55.7%)
profiling 2.099 ms [2.045 ms, 2.154 ms] 616.59 µs (41.6%)
tracing 2.095 ms [2.04 ms, 2.149 ms] 611.962 µs (41.3%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~bc787c05dc, baseline=1.62.0-SNAPSHOT~16f8277b2e
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.387 s) : 15387000, 15387000
.   : milestone, 15387000,
appsec (14.606 s) : 14606000, 14606000
.   : milestone, 14606000,
iast (18.331 s) : 18331000, 18331000
.   : milestone, 18331000,
iast_GLOBAL (18.027 s) : 18027000, 18027000
.   : milestone, 18027000,
profiling (14.874 s) : 14874000, 14874000
.   : milestone, 14874000,
tracing (14.96 s) : 14960000, 14960000
.   : milestone, 14960000,
section candidate
no_agent (15.127 s) : 15127000, 15127000
.   : milestone, 15127000,
appsec (14.795 s) : 14795000, 14795000
.   : milestone, 14795000,
iast (18.293 s) : 18293000, 18293000
.   : milestone, 18293000,
iast_GLOBAL (18.086 s) : 18086000, 18086000
.   : milestone, 18086000,
profiling (15.031 s) : 15031000, 15031000
.   : milestone, 15031000,
tracing (14.973 s) : 14973000, 14973000
.   : milestone, 14973000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.387 s [15.387 s, 15.387 s] -
appsec 14.606 s [14.606 s, 14.606 s] -781.0 ms (-5.1%)
iast 18.331 s [18.331 s, 18.331 s] 2.944 s (19.1%)
iast_GLOBAL 18.027 s [18.027 s, 18.027 s] 2.64 s (17.2%)
profiling 14.874 s [14.874 s, 14.874 s] -513.0 ms (-3.3%)
tracing 14.96 s [14.96 s, 14.96 s] -427.0 ms (-2.8%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.127 s [15.127 s, 15.127 s] -
appsec 14.795 s [14.795 s, 14.795 s] -332.0 ms (-2.2%)
iast 18.293 s [18.293 s, 18.293 s] 3.166 s (20.9%)
iast_GLOBAL 18.086 s [18.086 s, 18.086 s] 2.959 s (19.6%)
profiling 15.031 s [15.031 s, 15.031 s] -96.0 ms (-0.6%)
tracing 14.973 s [14.973 s, 14.973 s] -154.0 ms (-1.0%)

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - sonar-java

Job Status: success

Scenario Overhead (%)
agent -7.23
agentless 7.36
agentlessCodeCoverage 73.85
agentlessLineCoverage 144.07

@daniel-mohedano daniel-mohedano marked this pull request as ready for review April 15, 2026 08:15
@daniel-mohedano daniel-mohedano requested a review from a team as a code owner April 15, 2026 08:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

comp: ci visibility Continuous Integration Visibility type: bug Bug report and fix

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@daniel-mohedano