Fix root coverage packages calculation in large repositories by nikita-tkachenko-datadog · Pull Request #6317 · DataDog/dd-trace-java

nikita-tkachenko-datadog · 2023-12-05T12:37:11Z

What Does This Do

Updates repository indexer to only scan the project folder (and not the entire repository).
Adds a new config property that sets the limit on the number of root packages calculated by the indexed.

Motivation

Both changes are needed to handle builds in repositories that contain multiple projects.

One of the things that the repository indexer does is determining the set of packages that should be instrumented with code coverage (limiting the set of packages is needed to reduce coverage overhead, as instrumenting 3rd party dependencies and core JDK classes adds a lot of overhead and serves no purpose).
The set of packages determined by the indexer is passed to Jacoco agent via the command line.
The length of the command line is finite (exact max length is OS-dependant), so the number of packages added to the command line has to be limited.

The new config property allows to manually increase the limit for those projects that might need it.
Indexing only the project folder allows to exclude packages that are known to be irrelevant.

The problem with the previous version of the code was that in a large repository (for example, CI Visibility test environment repo) the total number of packages was too big to fit the hard-coded limit, so the coverage logic degraded to instrumenting all the classes, which caused huge performance overhead.

Additional Notes

Jira ticket: CIVIS-8227

pr-commenter · 2023-12-05T13:22:03Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/coverage-root-packages-fix
git_commit_date	1701730732	1701779797
git_commit_sha	`d1eda60`	`68cb650`
release_version	1.25.0~d1eda60b66	1.26.0-SNAPSHOT~68cb650aad

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1701782482	1701782482
ci_job_id	381845112	381845112
ci_pipeline_id	24390089	24390089
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 43 metrics, 11 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.26.0-SNAPSHOT~68cb650aad, baseline=1.25.0~d1eda60b66

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.045 s) : 0, 1044848
Total [baseline] (9.372 s) : 0, 9371885
Agent [candidate] (1.043 s) : 0, 1042754
Total [candidate] (9.327 s) : 0, 9327240
section appsec
Agent [baseline] (1.14 s) : 0, 1139965
Total [baseline] (9.514 s) : 0, 9514340
Agent [candidate] (1.133 s) : 0, 1132836
Total [candidate] (9.51 s) : 0, 9509518
section iast
Agent [baseline] (1.171 s) : 0, 1170882
Total [baseline] (9.604 s) : 0, 9604159
Agent [candidate] (1.162 s) : 0, 1162049
Total [candidate] (9.615 s) : 0, 9615013
section profiling
Agent [baseline] (1.244 s) : 0, 1244388
Total [baseline] (9.651 s) : 0, 9651464
Agent [candidate] (1.24 s) : 0, 1240003
Total [candidate] (9.689 s) : 0, 9689149

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.045 s	-
Agent	appsec	1.14 s	95.118 ms (9.1%)
Agent	iast	1.171 s	126.034 ms (12.1%)
Agent	profiling	1.244 s	199.541 ms (19.1%)
Total	tracing	9.372 s	-
Total	appsec	9.514 s	142.455 ms (1.5%)
Total	iast	9.604 s	232.274 ms (2.5%)
Total	profiling	9.651 s	279.579 ms (3.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.043 s	-
Agent	appsec	1.133 s	90.082 ms (8.6%)
Agent	iast	1.162 s	119.295 ms (11.4%)
Agent	profiling	1.24 s	197.249 ms (18.9%)
Total	tracing	9.327 s	-
Total	appsec	9.51 s	182.278 ms (2.0%)
Total	iast	9.615 s	287.772 ms (3.1%)
Total	profiling	9.689 s	361.909 ms (3.9%)

gantt
    title petclinic - break down per module: candidate=1.26.0-SNAPSHOT~68cb650aad, baseline=1.25.0~d1eda60b66

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (646.772 ms) : 0, 646772
BytebuddyAgent [candidate] (645.379 ms) : 0, 645379
GlobalTracer [baseline] (307.106 ms) : 0, 307106
GlobalTracer [candidate] (306.502 ms) : 0, 306502
AppSec [baseline] (48.827 ms) : 0, 48827
AppSec [candidate] (48.812 ms) : 0, 48812
Remote Config [baseline] (680.385 µs) : 0, 680
Remote Config [candidate] (676.374 µs) : 0, 676
Telemetry [baseline] (7.141 ms) : 0, 7141
Telemetry [candidate] (7.088 ms) : 0, 7088
section appsec
BytebuddyAgent [baseline] (651.123 ms) : 0, 651123
BytebuddyAgent [candidate] (645.937 ms) : 0, 645937
GlobalTracer [baseline] (308.675 ms) : 0, 308675
GlobalTracer [candidate] (306.354 ms) : 0, 306354
AppSec [baseline] (137.443 ms) : 0, 137443
AppSec [candidate] (136.721 ms) : 0, 136721
Remote Config [baseline] (650.064 µs) : 0, 650
Remote Config [candidate] (646.43 µs) : 0, 646
Telemetry [baseline] (7.514 ms) : 0, 7514
Telemetry [candidate] (8.806 ms) : 0, 8806
section iast
BytebuddyAgent [baseline] (773.287 ms) : 0, 773287
BytebuddyAgent [candidate] (767.569 ms) : 0, 767569
GlobalTracer [baseline] (287.423 ms) : 0, 287423
GlobalTracer [candidate] (285.567 ms) : 0, 285567
AppSec [baseline] (46.751 ms) : 0, 46751
AppSec [candidate] (46.358 ms) : 0, 46358
IAST [baseline] (19.432 ms) : 0, 19432
IAST [candidate] (20.932 ms) : 0, 20932
Remote Config [baseline] (637.877 µs) : 0, 638
Remote Config [candidate] (638.918 µs) : 0, 639
Telemetry [baseline] (8.844 ms) : 0, 8844
Telemetry [candidate] (6.585 ms) : 0, 6585
section profiling
BytebuddyAgent [baseline] (661.591 ms) : 0, 661591
BytebuddyAgent [candidate] (660.722 ms) : 0, 660722
GlobalTracer [baseline] (380.832 ms) : 0, 380832
GlobalTracer [candidate] (378.42 ms) : 0, 378420
AppSec [baseline] (49.064 ms) : 0, 49064
AppSec [candidate] (48.943 ms) : 0, 48943
Remote Config [baseline] (695.142 µs) : 0, 695
Remote Config [candidate] (700.904 µs) : 0, 701
Telemetry [baseline] (8.147 ms) : 0, 8147
Telemetry [candidate] (7.46 ms) : 0, 7460
ProfilingAgent [baseline] (89.491 ms) : 0, 89491
ProfilingAgent [candidate] (89.195 ms) : 0, 89195
Profiling [baseline] (89.516 ms) : 0, 89516
Profiling [candidate] (89.22 ms) : 0, 89220

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.26.0-SNAPSHOT~68cb650aad, baseline=1.25.0~d1eda60b66

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.051 s) : 0, 1050853
Total [baseline] (8.759 s) : 0, 8758502
Agent [candidate] (1.051 s) : 0, 1051432
Total [candidate] (8.791 s) : 0, 8791421
section iast
Agent [baseline] (1.169 s) : 0, 1168664
Total [baseline] (9.291 s) : 0, 9291304
Agent [candidate] (1.163 s) : 0, 1163203
Total [candidate] (9.296 s) : 0, 9296024
section iast_TELEMETRY_OFF
Agent [baseline] (1.155 s) : 0, 1154511
Total [baseline] (9.239 s) : 0, 9238722
Agent [candidate] (1.159 s) : 0, 1158762
Total [candidate] (9.296 s) : 0, 9295803

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.051 s	-
Agent	iast	1.169 s	117.81 ms (11.2%)
Agent	iast_TELEMETRY_OFF	1.155 s	103.657 ms (9.9%)
Total	tracing	8.759 s	-
Total	iast	9.291 s	532.802 ms (6.1%)
Total	iast_TELEMETRY_OFF	9.239 s	480.219 ms (5.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.051 s	-
Agent	iast	1.163 s	111.771 ms (10.6%)
Agent	iast_TELEMETRY_OFF	1.159 s	107.33 ms (10.2%)
Total	tracing	8.791 s	-
Total	iast	9.296 s	504.603 ms (5.7%)
Total	iast_TELEMETRY_OFF	9.296 s	504.382 ms (5.7%)

gantt
    title insecure-bank - break down per module: candidate=1.26.0-SNAPSHOT~68cb650aad, baseline=1.25.0~d1eda60b66

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (651.193 ms) : 0, 651193
BytebuddyAgent [candidate] (651.149 ms) : 0, 651149
GlobalTracer [baseline] (308.378 ms) : 0, 308378
GlobalTracer [candidate] (308.588 ms) : 0, 308588
AppSec [baseline] (48.891 ms) : 0, 48891
AppSec [candidate] (49.293 ms) : 0, 49293
Remote Config [baseline] (677.734 µs) : 0, 678
Remote Config [candidate] (679.299 µs) : 0, 679
Telemetry [baseline] (7.088 ms) : 0, 7088
Telemetry [candidate] (7.198 ms) : 0, 7198
section iast
BytebuddyAgent [baseline] (773.33 ms) : 0, 773330
BytebuddyAgent [candidate] (768.535 ms) : 0, 768535
GlobalTracer [baseline] (287.023 ms) : 0, 287023
GlobalTracer [candidate] (286.4 ms) : 0, 286400
AppSec [baseline] (46.567 ms) : 0, 46567
AppSec [candidate] (46.114 ms) : 0, 46114
IAST [baseline] (19.019 ms) : 0, 19019
IAST [candidate] (20.61 ms) : 0, 20610
Remote Config [baseline] (607.169 µs) : 0, 607
Remote Config [candidate] (601.031 µs) : 0, 601
Telemetry [baseline] (7.406 ms) : 0, 7406
Telemetry [candidate] (6.63 ms) : 0, 6630
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (760.433 ms) : 0, 760433
BytebuddyAgent [candidate] (764.536 ms) : 0, 764536
GlobalTracer [baseline] (285.614 ms) : 0, 285614
GlobalTracer [candidate] (285.894 ms) : 0, 285894
AppSec [baseline] (47.745 ms) : 0, 47745
AppSec [candidate] (47.121 ms) : 0, 47121
IAST [baseline] (16.937 ms) : 0, 16937
IAST [candidate] (18.865 ms) : 0, 18865
Remote Config [baseline] (620.526 µs) : 0, 621
Remote Config [candidate] (591.897 µs) : 0, 592
Telemetry [baseline] (8.687 ms) : 0, 8687
Telemetry [candidate] (7.135 ms) : 0, 7135

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2023-12-05T13:00:29	2023-12-05T13:17:08
git_branch	master	nikita-tkachenko/coverage-root-packages-fix
git_commit_date	1701730732	1701779797
git_commit_sha	`d1eda60`	`68cb650`
release_version	1.25.0~d1eda60b66	1.26.0-SNAPSHOT~68cb650aad
start_time	2023-12-05T13:00:16	2023-12-05T13:16:55

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1701782482	1701782482
ci_job_id	381845112	381845112
ci_pipeline_id	24390089	24390089
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 12 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~68cb650aad, baseline=1.25.0~d1eda60b66
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.372 ms) : 1352, 1392
.   : milestone, 1372,
appsec (1.747 ms) : 1722, 1772
.   : milestone, 1747,
iast (1.534 ms) : 1510, 1558
.   : milestone, 1534,
profiling (1.576 ms) : 1548, 1603
.   : milestone, 1576,
tracing (1.512 ms) : 1487, 1537
.   : milestone, 1512,
section candidate
no_agent (1.349 ms) : 1330, 1368
.   : milestone, 1349,
appsec (1.762 ms) : 1737, 1788
.   : milestone, 1762,
iast (1.529 ms) : 1504, 1554
.   : milestone, 1529,
profiling (1.522 ms) : 1496, 1548
.   : milestone, 1522,
tracing (1.501 ms) : 1477, 1526
.   : milestone, 1501,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.372 ms [1.352 ms, 1.392 ms]	-
appsec	1.747 ms [1.722 ms, 1.772 ms]	374.755 µs (27.3%)
iast	1.534 ms [1.51 ms, 1.558 ms]	162.103 µs (11.8%)
profiling	1.576 ms [1.548 ms, 1.603 ms]	203.586 µs (14.8%)
tracing	1.512 ms [1.487 ms, 1.537 ms]	139.684 µs (10.2%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.349 ms [1.33 ms, 1.368 ms]	-
appsec	1.762 ms [1.737 ms, 1.788 ms]	413.191 µs (30.6%)
iast	1.529 ms [1.504 ms, 1.554 ms]	179.628 µs (13.3%)
profiling	1.522 ms [1.496 ms, 1.548 ms]	172.915 µs (12.8%)
tracing	1.501 ms [1.477 ms, 1.526 ms]	152.375 µs (11.3%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~68cb650aad, baseline=1.25.0~d1eda60b66
    dateFormat X
    axisFormat %s
section baseline
no_agent (370.571 µs) : 351, 390
.   : milestone, 371,
iast (479.771 µs) : 459, 500
.   : milestone, 480,
iast_FULL (542.787 µs) : 522, 563
.   : milestone, 543,
iast_INACTIVE (453.494 µs) : 433, 474
.   : milestone, 453,
iast_TELEMETRY_OFF (478.453 µs) : 457, 500
.   : milestone, 478,
tracing (444.758 µs) : 424, 465
.   : milestone, 445,
section candidate
no_agent (368.526 µs) : 349, 388
.   : milestone, 369,
iast (474.708 µs) : 454, 495
.   : milestone, 475,
iast_FULL (540.494 µs) : 520, 561
.   : milestone, 540,
iast_INACTIVE (449.606 µs) : 429, 470
.   : milestone, 450,
iast_TELEMETRY_OFF (484.144 µs) : 463, 505
.   : milestone, 484,
tracing (456.827 µs) : 436, 478
.   : milestone, 457,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	370.571 µs [350.789 µs, 390.354 µs]	-
iast	479.771 µs [459.438 µs, 500.104 µs]	109.2 µs (29.5%)
iast_FULL	542.787 µs [522.396 µs, 563.178 µs]	172.216 µs (46.5%)
iast_INACTIVE	453.494 µs [432.619 µs, 474.37 µs]	82.923 µs (22.4%)
iast_TELEMETRY_OFF	478.453 µs [457.223 µs, 499.684 µs]	107.882 µs (29.1%)
tracing	444.758 µs [424.28 µs, 465.236 µs]	74.187 µs (20.0%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	368.526 µs [348.596 µs, 388.457 µs]	-
iast	474.708 µs [454.185 µs, 495.232 µs]	106.182 µs (28.8%)
iast_FULL	540.494 µs [520.293 µs, 560.695 µs]	171.968 µs (46.7%)
iast_INACTIVE	449.606 µs [429.1 µs, 470.113 µs]	81.08 µs (22.0%)
iast_TELEMETRY_OFF	484.144 µs [462.884 µs, 505.405 µs]	115.618 µs (31.4%)
tracing	456.827 µs [436.054 µs, 477.601 µs]	88.301 µs (24.0%)

Cherrypick: Fix root coverage packages calculation in large repositories (#6317)

Fix root coverage packages caclculation

68cb650

nikita-tkachenko-datadog added type: bug Bug report and fix comp: ci visibility Continuous Integration Visibility labels Dec 5, 2023

nikita-tkachenko-datadog changed the title ~~Fix root coverage packages caclulation~~ Fix root coverage packages calculation in large repositories Dec 5, 2023

nikita-tkachenko-datadog marked this pull request as ready for review December 5, 2023 13:09

nikita-tkachenko-datadog requested review from a team as code owners December 5, 2023 13:09

nikita-tkachenko-datadog requested review from am312, anmarchenko and mcculls December 5, 2023 13:09

romainkomorn-exdatadog approved these changes Dec 5, 2023

View reviewed changes

nikita-tkachenko-datadog merged commit c88f535 into master Dec 5, 2023

nikita-tkachenko-datadog deleted the nikita-tkachenko/coverage-root-packages-fix branch December 5, 2023 13:26

github-actions Bot added this to the 1.26.0 milestone Dec 5, 2023

bm1549 restored the nikita-tkachenko/coverage-root-packages-fix branch December 5, 2023 20:47

bm1549 deleted the nikita-tkachenko/coverage-root-packages-fix branch December 5, 2023 20:48

bm1549 pushed a commit that referenced this pull request Dec 5, 2023

Fix root coverage packages calculation in large repositories (#6317)

8908066

bm1549 added a commit that referenced this pull request Dec 6, 2023

Merge pull request #6321 from DataDog/bm1549/cherry-pick

476c346

Cherrypick: Fix root coverage packages calculation in large repositories (#6317)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix root coverage packages calculation in large repositories#6317

Fix root coverage packages calculation in large repositories#6317
nikita-tkachenko-datadog merged 1 commit into
masterfrom
nikita-tkachenko/coverage-root-packages-fix

nikita-tkachenko-datadog commented Dec 5, 2023 •

edited by atlassian Bot

Loading

Uh oh!

pr-commenter Bot commented Dec 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nikita-tkachenko-datadog commented Dec 5, 2023 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Uh oh!

pr-commenter Bot commented Dec 5, 2023

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nikita-tkachenko-datadog commented Dec 5, 2023 •

edited by atlassian Bot

Loading