Skip to content

Update system-tests to 268c7ddfba2c45085dc29434e677ff13307445ef#6257

Merged
mcculls merged 1 commit into
masterfrom
mcculls/enable-tracer-flare-system-tests
Nov 22, 2023
Merged

Update system-tests to 268c7ddfba2c45085dc29434e677ff13307445ef#6257
mcculls merged 1 commit into
masterfrom
mcculls/enable-tracer-flare-system-tests

Conversation

@mcculls
Copy link
Copy Markdown
Contributor

@mcculls mcculls commented Nov 22, 2023 •
edited
Loading

What Does This Do

Enables tracer-flare system tests for the development version of the Java tracer (1.25.0)

[gw2] [ 98%] PASSED tests/parametric/test_tracer_flare.py::TestTracerFlareV1::test_flare_log_level_order[library_env0] 
[gw0] [ 98%] PASSED tests/parametric/test_tracer_flare.py::TestTracerFlareV1::test_telemetry_app_started[library_env0] 
tests/parametric/test_tracer_flare.py::TestTracerFlareV1::test_no_tracer_flare_for_other_task_types[library_env0] 
[gw1] [ 99%] PASSED tests/parametric/test_tracer_flare.py::TestTracerFlareV1::test_tracer_flare_with_debug[library_env0] 
[gw3] [ 99%] PASSED tests/parametric/test_tracer_flare.py::TestTracerFlareV1::test_tracer_flare[library_env0] 
[gw0] [100%] PASSED tests/parametric/test_tracer_flare.py::TestTracerFlareV1::test_no_tracer_flare_for_other_task_types[library_env0]

Additional Notes

Jira ticket: APMJAVA-1125

@mcculls mcculls added comp: testing Testing tag: no release notes Changes to exclude from release notes labels Nov 22, 2023
@mcculls mcculls force-pushed the mcculls/enable-tracer-flare-system-tests branch from d4db77d to d251924 Compare November 22, 2023 09:30
@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Nov 22, 2023 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mcculls/enable-tracer-flare-system-tests
git_commit_date 1700644990 1700646069
git_commit_sha 05aa9e0 d251924
release_version 1.25.0-SNAPSHOT~05aa9e092c 1.25.0-SNAPSHOT~d25192478f
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1700648048 1700648048
ci_job_id 374929934 374929934
ci_pipeline_id 23709649 23709649
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 46 metrics, 8 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.25.0-SNAPSHOT~d25192478f, baseline=1.25.0-SNAPSHOT~05aa9e092c

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.036 s) : 0, 1035851
Total [baseline] (8.654 s) : 0, 8653855
Agent [candidate] (1.033 s) : 0, 1032789
Total [candidate] (8.649 s) : 0, 8649366
section iast
Agent [baseline] (1.155 s) : 0, 1155174
Total [baseline] (9.21 s) : 0, 9209967
Agent [candidate] (1.157 s) : 0, 1157192
Total [candidate] (9.255 s) : 0, 9254648
section iast_TELEMETRY_OFF
Agent [baseline] (1.139 s) : 0, 1138761
Total [baseline] (9.184 s) : 0, 9183591
Agent [candidate] (1.156 s) : 0, 1155578
Total [candidate] (9.278 s) : 0, 9278471
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.036 s -
Agent iast 1.155 s 119.323 ms (11.5%)
Agent iast_TELEMETRY_OFF 1.139 s 102.91 ms (9.9%)
Total tracing 8.654 s -
Total iast 9.21 s 556.112 ms (6.4%)
Total iast_TELEMETRY_OFF 9.184 s 529.736 ms (6.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.033 s -
Agent iast 1.157 s 124.403 ms (12.0%)
Agent iast_TELEMETRY_OFF 1.156 s 122.789 ms (11.9%)
Total tracing 8.649 s -
Total iast 9.255 s 605.282 ms (7.0%)
Total iast_TELEMETRY_OFF 9.278 s 629.105 ms (7.3%) 
gantt
    title insecure-bank - break down per module: candidate=1.25.0-SNAPSHOT~d25192478f, baseline=1.25.0-SNAPSHOT~05aa9e092c

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (645.758 ms) : 0, 645758
BytebuddyAgent [candidate] (644.212 ms) : 0, 644212
GlobalTracer [baseline] (298.995 ms) : 0, 298995
GlobalTracer [candidate] (298.177 ms) : 0, 298177
AppSec [baseline] (48.533 ms) : 0, 48533
AppSec [candidate] (48.089 ms) : 0, 48089
Remote Config [baseline] (675.409 µs) : 0, 675
Remote Config [candidate] (660.949 µs) : 0, 661
Telemetry [baseline] (7.355 ms) : 0, 7355
Telemetry [candidate] (7.234 ms) : 0, 7234
section iast
BytebuddyAgent [baseline] (767.463 ms) : 0, 767463
BytebuddyAgent [candidate] (769.0 ms) : 0, 769000
GlobalTracer [baseline] (278.627 ms) : 0, 278627
GlobalTracer [candidate] (279.824 ms) : 0, 279824
AppSec [baseline] (49.754 ms) : 0, 49754
AppSec [candidate] (48.875 ms) : 0, 48875
IAST [baseline] (14.062 ms) : 0, 14062
IAST [candidate] (15.653 ms) : 0, 15653
Remote Config [baseline] (570.475 µs) : 0, 570
Remote Config [candidate] (577.979 µs) : 0, 578
Telemetry [baseline] (10.177 ms) : 0, 10177
Telemetry [candidate] (8.671 ms) : 0, 8671
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (754.835 ms) : 0, 754835
BytebuddyAgent [candidate] (764.369 ms) : 0, 764369
GlobalTracer [baseline] (277.209 ms) : 0, 277209
GlobalTracer [candidate] (282.684 ms) : 0, 282684
AppSec [baseline] (46.297 ms) : 0, 46297
AppSec [candidate] (47.078 ms) : 0, 47078
IAST [baseline] (19.399 ms) : 0, 19399
IAST [candidate] (19.802 ms) : 0, 19802
Remote Config [baseline] (563.218 µs) : 0, 563
Remote Config [candidate] (578.673 µs) : 0, 579
Telemetry [baseline] (6.297 ms) : 0, 6297
Telemetry [candidate] (6.429 ms) : 0, 6429
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.25.0-SNAPSHOT~d25192478f, baseline=1.25.0-SNAPSHOT~05aa9e092c

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.036 s) : 0, 1036331
Total [baseline] (9.297 s) : 0, 9296736
Agent [candidate] (1.032 s) : 0, 1032041
Total [candidate] (9.307 s) : 0, 9306952
section appsec
Agent [baseline] (1.119 s) : 0, 1118869
Total [baseline] (9.383 s) : 0, 9382616
Agent [candidate] (1.126 s) : 0, 1125834
Total [candidate] (9.471 s) : 0, 9470816
section iast
Agent [baseline] (1.149 s) : 0, 1148803
Total [baseline] (9.474 s) : 0, 9474220
Agent [candidate] (1.149 s) : 0, 1149294
Total [candidate] (9.516 s) : 0, 9515846
section profiling
Agent [baseline] (1.228 s) : 0, 1227531
Total [baseline] (9.61 s) : 0, 9610316
Agent [candidate] (1.222 s) : 0, 1221768
Total [candidate] (9.606 s) : 0, 9606079
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.036 s -
Agent appsec 1.119 s 82.538 ms (8.0%)
Agent iast 1.149 s 112.472 ms (10.9%)
Agent profiling 1.228 s 191.2 ms (18.4%)
Total tracing 9.297 s -
Total appsec 9.383 s 85.88 ms (0.9%)
Total iast 9.474 s 177.483 ms (1.9%)
Total profiling 9.61 s 313.58 ms (3.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.032 s -
Agent appsec 1.126 s 93.794 ms (9.1%)
Agent iast 1.149 s 117.254 ms (11.4%)
Agent profiling 1.222 s 189.728 ms (18.4%)
Total tracing 9.307 s -
Total appsec 9.471 s 163.864 ms (1.8%)
Total iast 9.516 s 208.893 ms (2.2%)
Total profiling 9.606 s 299.127 ms (3.2%) 
gantt
    title petclinic - break down per module: candidate=1.25.0-SNAPSHOT~d25192478f, baseline=1.25.0-SNAPSHOT~05aa9e092c

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (646.735 ms) : 0, 646735
BytebuddyAgent [candidate] (643.224 ms) : 0, 643224
GlobalTracer [baseline] (298.826 ms) : 0, 298826
GlobalTracer [candidate] (298.304 ms) : 0, 298304
AppSec [baseline] (48.407 ms) : 0, 48407
AppSec [candidate] (48.291 ms) : 0, 48291
Remote Config [baseline] (664.755 µs) : 0, 665
Remote Config [candidate] (663.539 µs) : 0, 664
Telemetry [baseline] (7.248 ms) : 0, 7248
Telemetry [candidate] (7.216 ms) : 0, 7216
section appsec
BytebuddyAgent [baseline] (643.495 ms) : 0, 643495
BytebuddyAgent [candidate] (647.654 ms) : 0, 647654
GlobalTracer [baseline] (297.457 ms) : 0, 297457
GlobalTracer [candidate] (299.645 ms) : 0, 299645
AppSec [baseline] (136.358 ms) : 0, 136358
AppSec [candidate] (136.726 ms) : 0, 136726
Remote Config [baseline] (639.505 µs) : 0, 640
Remote Config [candidate] (662.885 µs) : 0, 663
Telemetry [baseline] (6.776 ms) : 0, 6776
Telemetry [candidate] (6.78 ms) : 0, 6780
section iast
BytebuddyAgent [baseline] (762.959 ms) : 0, 762959
BytebuddyAgent [candidate] (762.896 ms) : 0, 762896
GlobalTracer [baseline] (277.376 ms) : 0, 277376
GlobalTracer [candidate] (277.645 ms) : 0, 277645
AppSec [baseline] (52.159 ms) : 0, 52159
AppSec [candidate] (48.531 ms) : 0, 48531
Remote Config [baseline] (553.144 µs) : 0, 553
Remote Config [candidate] (571.037 µs) : 0, 571
Telemetry [baseline] (7.114 ms) : 0, 7114
Telemetry [candidate] (10.653 ms) : 0, 10653
IAST [baseline] (14.532 ms) : 0, 14532
IAST [candidate] (14.757 ms) : 0, 14757
section profiling
BytebuddyAgent [baseline] (658.265 ms) : 0, 658265
BytebuddyAgent [candidate] (655.017 ms) : 0, 655017
GlobalTracer [baseline] (369.645 ms) : 0, 369645
GlobalTracer [candidate] (367.401 ms) : 0, 367401
AppSec [baseline] (48.644 ms) : 0, 48644
AppSec [candidate] (48.575 ms) : 0, 48575
Remote Config [baseline] (710.493 µs) : 0, 710
Remote Config [candidate] (700.341 µs) : 0, 700
Telemetry [baseline] (7.454 ms) : 0, 7454
Telemetry [candidate] (7.405 ms) : 0, 7405
ProfilingAgent [baseline] (88.478 ms) : 0, 88478
ProfilingAgent [candidate] (88.591 ms) : 0, 88591
Profiling [baseline] (88.501 ms) : 0, 88501
Profiling [candidate] (88.614 ms) : 0, 88614
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2023-11-22T09:53:26 2023-11-22T10:09:57
git_branch master mcculls/enable-tracer-flare-system-tests
git_commit_date 1700644990 1700646069
git_commit_sha 05aa9e0 d251924
release_version 1.25.0-SNAPSHOT~05aa9e092c 1.25.0-SNAPSHOT~d25192478f
start_time 2023-11-22T09:53:13 2023-11-22T10:09:44
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1700648048 1700648048
ci_job_id 374929934 374929934
ci_pipeline_id 23709649 23709649
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 7 metrics, 14 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:petclinic:profiling better
[-89.291µs; -34.336µs] or [-5.738%; -2.206%]		 unstable
[-470.518op/s; +698.438op/s] or [-15.880%; +23.572%]		 1.494ms 3076.923op/s 1.556ms 2962.963op/s
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.25.0-SNAPSHOT~d25192478f, baseline=1.25.0-SNAPSHOT~05aa9e092c
    dateFormat X
    axisFormat %s
section baseline
no_agent (366.875 µs) : 345, 389
.   : milestone, 367,
iast (466.519 µs) : 446, 487
.   : milestone, 467,
iast_FULL (523.232 µs) : 503, 544
.   : milestone, 523,
iast_INACTIVE (439.793 µs) : 419, 460
.   : milestone, 440,
iast_TELEMETRY_OFF (462.585 µs) : 442, 483
.   : milestone, 463,
tracing (435.078 µs) : 415, 455
.   : milestone, 435,
section candidate
no_agent (364.142 µs) : 343, 386
.   : milestone, 364,
iast (462.75 µs) : 442, 483
.   : milestone, 463,
iast_FULL (530.233 µs) : 510, 551
.   : milestone, 530,
iast_INACTIVE (444.498 µs) : 423, 465
.   : milestone, 444,
iast_TELEMETRY_OFF (457.884 µs) : 438, 478
.   : milestone, 458,
tracing (434.959 µs) : 413, 456
.   : milestone, 435,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 366.875 µs [344.725 µs, 389.025 µs] -
iast 466.519 µs [446.381 µs, 486.657 µs] 99.644 µs (27.2%)
iast_FULL 523.232 µs [502.854 µs, 543.611 µs] 156.357 µs (42.6%)
iast_INACTIVE 439.793 µs [419.258 µs, 460.329 µs] 72.918 µs (19.9%)
iast_TELEMETRY_OFF 462.585 µs [441.848 µs, 483.323 µs] 95.711 µs (26.1%)
tracing 435.078 µs [414.87 µs, 455.286 µs] 68.203 µs (18.6%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 364.142 µs [342.5 µs, 385.784 µs] -
iast 462.75 µs [442.382 µs, 483.118 µs] 98.607 µs (27.1%)
iast_FULL 530.233 µs [509.837 µs, 550.63 µs] 166.091 µs (45.6%)
iast_INACTIVE 444.498 µs [423.498 µs, 465.497 µs] 80.355 µs (22.1%)
iast_TELEMETRY_OFF 457.884 µs [437.633 µs, 478.134 µs] 93.741 µs (25.7%)
tracing 434.959 µs [413.438 µs, 456.479 µs] 70.816 µs (19.4%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.25.0-SNAPSHOT~d25192478f, baseline=1.25.0-SNAPSHOT~05aa9e092c
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.339 ms) : 1320, 1357
.   : milestone, 1339,
appsec (1.751 ms) : 1726, 1777
.   : milestone, 1751,
iast (1.52 ms) : 1496, 1544
.   : milestone, 1520,
profiling (1.556 ms) : 1530, 1582
.   : milestone, 1556,
tracing (1.504 ms) : 1479, 1528
.   : milestone, 1504,
section candidate
no_agent (1.338 ms) : 1319, 1357
.   : milestone, 1338,
appsec (1.76 ms) : 1734, 1785
.   : milestone, 1760,
iast (1.513 ms) : 1489, 1537
.   : milestone, 1513,
profiling (1.494 ms) : 1470, 1519
.   : milestone, 1494,
tracing (1.497 ms) : 1472, 1522
.   : milestone, 1497,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.339 ms [1.32 ms, 1.357 ms] -
appsec 1.751 ms [1.726 ms, 1.777 ms] 412.669 µs (30.8%)
iast 1.52 ms [1.496 ms, 1.544 ms] 181.157 µs (13.5%)
profiling 1.556 ms [1.53 ms, 1.582 ms] 217.472 µs (16.2%)
tracing 1.504 ms [1.479 ms, 1.528 ms] 164.91 µs (12.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.338 ms [1.319 ms, 1.357 ms] -
appsec 1.76 ms [1.734 ms, 1.785 ms] 421.442 µs (31.5%)
iast 1.513 ms [1.489 ms, 1.537 ms] 174.373 µs (13.0%)
profiling 1.494 ms [1.47 ms, 1.519 ms] 156.08 µs (11.7%)
tracing 1.497 ms [1.472 ms, 1.522 ms] 158.745 µs (11.9%)

@mcculls mcculls marked this pull request as ready for review November 22, 2023 10:30
@mcculls mcculls requested a review from a team as a code owner November 22, 2023 10:30
@mcculls mcculls requested review from bantonsson and dougqh November 22, 2023 10:30
@mcculls mcculls merged commit bc43877 into master Nov 22, 2023
@mcculls mcculls deleted the mcculls/enable-tracer-flare-system-tests branch November 22, 2023 10:31
@github-actions github-actions Bot added this to the 1.25.0 milestone Nov 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@smola smola smola approved these changes

@dougqh dougqh Awaiting requested review from dougqh dougqh is a code owner automatically assigned from DataDog/apm-java

@bantonsson bantonsson Awaiting requested review from bantonsson bantonsson is a code owner automatically assigned from DataDog/apm-java

Assignees

No one assigned

Labels

comp: testing Testing tag: no release notes Changes to exclude from release notes

Projects

None yet

Milestone

1.25.0

Development

Successfully merging this pull request may close these issues.

2 participants

@mcculls @smola