diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md new file mode 100644 index 00000000000..a34874f6957 --- /dev/null +++ b/.claude/CLAUDE.md @@ -0,0 +1,5 @@ +@../AGENTS.md + +## Claude Code workflow + +- Before creating a pull request, run `/techdebt` to check for technical debt, code duplication, and unnecessary complexity in the branch changes. diff --git a/.claude/skills/add-apm-integrations/SKILL.md b/.claude/skills/add-apm-integrations/SKILL.md new file mode 100644 index 00000000000..e29b93ef3a2 --- /dev/null +++ b/.claude/skills/add-apm-integrations/SKILL.md @@ -0,0 +1,232 @@ +--- +name: add-apm-integrations +description: Write a new library instrumentation end-to-end. Use when the user ask to add a new APM integration or a library instrumentation. +context: fork +allowed-tools: + - Bash + - Read + - Write + - Edit + - Glob + - Grep +--- + +Write a new APM end-to-end integration for dd-trace-java, based on library instrumentations, following all project conventions. + +## Step 1 – Read the authoritative docs and sync this skill (mandatory, always first) + +Before writing any code, read all three files in full: + +1. [`docs/how_instrumentations_work.md`](docs/how_instrumentations_work.md) — full reference (types, methods, advice, helpers, context stores, decorators) +2. [`docs/add_new_instrumentation.md`](docs/add_new_instrumentation.md) — step-by-step walkthrough +3. [`docs/how_to_test.md`](docs/how_to_test.md) — test types and how to run them + +These files are the single source of truth. Reference them while implementing. + +**After reading the docs, sync this skill with them:** + +Compare the content of the three docs against the rules encoded in Steps 2–11 of this skill file. Look for: +- Patterns, APIs, or conventions described in the docs but absent or incorrect here +- Steps that are out of date relative to the current docs (e.g. renamed methods, new base classes) +- Advice constraints or test requirements that have changed + +For every discrepancy found, edit this file (`.claude/skills/apm-integrations/SKILL.md`) to correct it using the +`Edit` tool before continuing. Keep changes targeted: fix what diverged, add what is missing, remove what is wrong. +Do not touch content that already matches the docs. + +## Step 2 – Clarify the task + +If the user has not already provided all of the following, ask before proceeding: + +- **Framework name** and **minimum supported version** (e.g. `okhttp-3.0`) +- **Target class(es) and method(s)** to instrument (fully qualified class names preferred) +- **Target system**: one of `Tracing`, `Profiling`, `AppSec`, `Iast`, `CiVisibility`, `Usm`, `ContextTracking` +- **Whether this is a bootstrap instrumentation** (affects allowed imports) + +## Step 3 – Find a reference instrumentation + +Search `dd-java-agent/instrumentation/` for a structurally similar integration: +- Same target system +- Comparable type-matching strategy (single type, hierarchy, known types) + +Read the reference integration's `InstrumenterModule`, Advice, Decorator, and test files to understand the established +pattern before writing new code. Use it as a template. + +## Step 4 – Set up the module + +1. Create directory: `dd-java-agent/instrumentation/$framework/$framework-$minVersion/` +2. Under it, create the standard Maven source layout: + - `src/main/java/` — instrumentation code + - `src/test/groovy/` — Spock tests +3. Create `build.gradle` with: + - `compileOnly` dependencies for the target framework + - `testImplementation` dependencies for tests + - `muzzle { pass { } }` directives (see Step 9) +4. Register the new module in `settings.gradle.kts` in **alphabetical order** + +## Step 5 – Write the InstrumenterModule + +Conventions to enforce: + +- Add `@AutoService(InstrumenterModule.class)` annotation — required for auto-discovery +- Extend the correct `InstrumenterModule.*` subclass (never the bare abstract class) +- Implement the **narrowest** `Instrumenter` interface possible: + - Prefer `ForSingleType` > `ForKnownTypes` > `ForTypeHierarchy` +- Add `classLoaderMatcher()` if a sentinel class identifies the framework on the classpath +- Declare **all** helper class names in `helperClassNames()`: + - Include inner classes (`Foo$Bar`), anonymous classes (`Foo$1`), and enum synthetic classes +- Declare `contextStore()` entries if context stores are needed (key class → value class) +- Keep method matchers as narrow as possible (name, parameter types, visibility) + +## Step 6 – Write the Decorator + +- Extend the most specific available base decorator: + - `HttpClientDecorator`, `DatabaseClientDecorator`, `ServerDecorator`, `MessagingClientDecorator`, etc. +- One `public static final DECORATE` instance +- Define `UTF8BytesString` constants for the component name and operation name +- Keep all tag/naming/error logic here — not in the Advice class +- Override `spanType()`, `component()`, `spanKind()` as appropriate + +## Step 7 – Write the Advice class (highest-risk step) + +### Must do + +- Advice methods **must** be `static` +- Annotate enter: `@Advice.OnMethodEnter(suppress = Throwable.class)` +- Annotate exit: `@Advice.OnMethodExit(onThrowable = Throwable.class, suppress = Throwable.class)` + - **Exception**: do NOT use `suppress` when hooking a constructor +- Use `@Advice.Local("...")` for values shared between enter and exit (span, scope) +- Use the correct parameter annotations: + - `@Advice.This` — the receiver object + - `@Advice.Argument(N)` — a method argument by index + - `@Advice.Return` — the return value (exit only) + - `@Advice.Thrown` — the thrown exception (exit only) + - `@Advice.Enter` — the return value of the enter method (exit only) +- Use `CallDepthThreadLocalMap` to guard against recursive instrumentation of the same method + +### Span lifecycle (in order) + +Enter method: +1. `AgentSpan span = startSpan(DECORATE.operationName(), ...)` +2. `DECORATE.afterStart(span)` + set domain-specific tags +3. `AgentScope scope = activateSpan(span)` — return or store via `@Advice.Local` + +Exit method: +4. `DECORATE.onError(span, throwable)` — only if throwable is non-null +5. `DECORATE.beforeFinish(span)` +6. `span.finish()` +7. `scope.close()` + +### Must NOT do + +- **No logger fields** in the Advice class or the Instrumentation class (loggers only in helpers/decorators) +- **No code in the Advice constructor** — it is never called +- **Do not use lambdas in advice methods** — they create synthetic classes that will be missing from helper declarations +- **No references** to other methods in the same Advice class or in the InstrumenterModule class +- **No `InstrumentationContext.get()`** outside of Advice code +- **No `inline=false`** in production code (only for debugging; must be removed before committing) +- **No `java.util.logging.*`, `java.nio.*`, or `javax.management.*`** in bootstrap instrumentations + +## Step 8 – Add SETTER/GETTER adapters (if applicable) + +For context propagation to and from upstream services, like HTTP headers, +implement `AgentPropagation.Setter` / `AgentPropagation.Getter` adapters that wrap the framework's specific header API. +Place them in the helpers package, declare them in `helperClassNames()`. + +## Step 9 – Write tests + +Cover all mandatory test types: + +### 1. Instrumentation test (mandatory) + +- Spock spec extending `InstrumentationSpecification` +- Place in `src/test/groovy/` +- Verify: spans created, tags set, errors propagated, resource names correct +- Use `TEST_WRITER.waitForTraces(N)` for assertions +- Use `runUnderTrace("root") { ... }` for synchronous code + +For tests that need a separate JVM, suffix the test class with `ForkedTest` and run via the `forkedTest` task. + +### 2. Muzzle directives (mandatory) + +In `build.gradle`, add `muzzle` blocks: +```groovy +muzzle { + pass { + group = "com.example" + module = "framework" + versions = "[$minVersion,)" + assertInverse = true // ensures versions below $minVersion fail muzzle + } +} +``` + +### 3. Latest dependency test (mandatory) + +Use the `latestDepTestLibrary` helper in `build.gradle` to pin the latest available version. Run with: +```bash +./gradlew :dd-java-agent:instrumentation:$framework-$version:latestDepTest +``` + +### 4. Smoke test (optional) + +Add a smoke test in `dd-smoke-tests/` only if the framework warrants a full end-to-end demo-app test. + +## Step 10 – Build and verify + +Run these commands in order and fix any failures before proceeding: + +```bash +./gradlew :dd-java-agent:instrumentation:$framework-$version:muzzle +./gradlew :dd-java-agent:instrumentation:$framework-$version:test +./gradlew :dd-java-agent:instrumentation:$framework-$version:latestDepTest +./gradlew spotlessCheck +``` + +**If muzzle fails:** check for missing helper class names in `helperClassNames()`. + +**If tests fail:** verify span lifecycle order (start → activate → error → finish → close), helper registration, +and `contextStore()` map entries match actual usage. + +**If spotlessCheck fails:** run `./gradlew spotlessApply` to auto-format, then re-check. + +## Step 11 – Checklist before finishing + +Output this checklist and confirm each item is satisfied: + +- [ ] `settings.gradle.kts` entry added in alphabetical order +- [ ] `build.gradle` has `compileOnly` deps and `muzzle` directives with `assertInverse = true` +- [ ] `@AutoService(InstrumenterModule.class)` annotation present on the module class +- [ ] `helperClassNames()` lists ALL referenced helpers (including inner, anonymous, and enum synthetic classes) +- [ ] Advice methods are `static` with `@Advice.OnMethodEnter` / `@Advice.OnMethodExit` annotations +- [ ] `suppress = Throwable.class` on enter/exit (unless the hooked method is a constructor) +- [ ] No logger field in the Advice class or InstrumenterModule class +- [ ] No `inline=false` left in production code +- [ ] No `java.util.logging.*` / `java.nio.*` / `javax.management.*` in bootstrap path +- [ ] Span lifecycle order is correct: startSpan → afterStart → activateSpan (enter); onError → beforeFinish → finish → close (exit) +- [ ] Muzzle passes +- [ ] Instrumentation tests pass +- [ ] `latestDepTest` passes +- [ ] `spotlessCheck` passes + +## Step 12 – Retrospective: update this skill with what was learned + +After the instrumentation is complete (or abandoned), review the full session and improve this skill for future use. + +**Collect lessons from four sources:** + +1. **Build/test failures** — did any Gradle task fail with an error that this skill did not anticipate or gave wrong + guidance for? (e.g. a muzzle failure that wasn't caused by missing helpers, a test pattern that didn't work) +2. **Docs vs. skill gaps** — did Step 1's sync miss anything? Did you consult the docs for something not captured here? +3. **Reference instrumentation insights** — did the reference integration use a pattern, API, or convention not + reflected in any step of this skill? +4. **User corrections** — did the user correct an output, override a decision, or point out a mistake? + +**For each lesson identified**, edit this file (`.claude/skills/apm-integrations/SKILL.md`) using the `Edit` tool: +- Wrong rule → fix it in place +- Missing rule → add it to the most relevant step +- Wrong failure guidance → update the relevant "If X fails" section in Step 10 +- Misleading or obsolete content → remove it + +Keep each change minimal and targeted. Do not rewrite sections that worked correctly. +After editing, confirm to the user which improvements were made to the skill. diff --git a/.claude/skills/migrate-groovy-to-java/SKILL.md b/.claude/skills/migrate-groovy-to-java/SKILL.md new file mode 100644 index 00000000000..c08c85ba296 --- /dev/null +++ b/.claude/skills/migrate-groovy-to-java/SKILL.md @@ -0,0 +1,61 @@ +--- +name: migrate-groovy-to-java +description: > + Converts Spock/Groovy test files in a Gradle module to equivalent JUnit 5 Java tests. + Use when asked to "migrate groovy", "convert groovy to java", "g2j", or when a module + has .groovy test files that need to be replaced with .java equivalents. +--- + +Migrate test Groovy files to Java using JUnit 5 + +1. List all Groovy files of the current Gradle module +2. Convert Groovy files to Java using JUnit 5 +3. Make sure the tests are still passing after migration and that the test count has not changed +4. Remove Groovy files +5. Add the migrated module path(s) to `.github/g2j-migrated-modules.txt` + +When converting Groovy code to Java code, make sure that: +- The Java code generated is compatible with JDK 8 +- When translating Spock tests, prefer `@TableTest` for data rows that are naturally tabular. See detailed guidance in the "TableTest usage" section. +- `@TableTest` and `@MethodSource` may be combined on the same `@ParameterizedTest` when most cases are tabular but a few cases require programmatic setup. +- In combined mode, keep table-friendly cases in `@TableTest`, and put only non-tabular/complex cases in `@MethodSource`. +- If `@TableTest` is not viable for the test at all, use `@MethodSource` only. +- If `@TableTest` was successfully used and if the `@ParameterizedTest` is not used to specify the test name, `@ParameterizedTest` can then be removed as `@TableTest` replace it fully. +- For `@MethodSource`, name the arguments method `Arguments` (camelCase, e.g. `testMethodArguments`) and return `Stream` using `Stream.of(...)` and `arguments(...)` with static import. +- Ensure parameterized test names are human-readable (i.e. no hashcodes); instead add a description string as the first `Arguments.arguments(...)` value or index the test case +- When converting tuples, create a light dedicated structure instead to keep the typing system +- Instead of checking a state and throwing an exception, use JUnit asserts +- Instead of using `assertTrue(a.equals(b))` or `assertFalse(a.equals(b))`, use `assertEquals(expected, actual)` and `assertNotEquals(unexpected, actual)` +- Import frequently used types rather than using fully-qualified names inline, to improve readability +- Do not wrap checked exceptions and throw a Runtime exception; prefer adding a throws clause at method declaration +- Do not mark local variables `final` +- Ensure variables are human-readable; avoid single-letter names and pre-define variables that are referenced multiple times +- When translating Spock `Mock(...)` usage, use `libs.bundles.mockito` instead of writing manual recording/stub implementations + +TableTest usage + Import: `import org.tabletest.junit.TableTest;` + + JDK 8 rules: + - No text blocks. + - @TableTest must use String[] annotation array syntax: + ``` + @TableTest({ + "a | b", + "1 | 2" + }) + ``` + + Spock `where:` → @TableTest: + - First row = header (column names = method parameters). + - Add `scenario` column as first column (display name, not a method parameter). + - Use `|` delimiter; align columns so pipes line up vertically. + - Prefer single quotes for strings with special chars (e.g., `'a|b'`, `'[]'`). + - Blank cell = null (object types); `''` = empty string. + - Collections: `[a, b]` = List/array, `{a, b}` = Set, `[k: v]` = Map. + + Mixed eligibility: + - Prefer combining `@TableTest` + `@MethodSource` on one `@ParameterizedTest` when only some cases are complex. + - Use `@MethodSource` only when tabular representation is not practical for the test. + + Do NOT use @TableTest when: + - Majority of rows require complex objects or custom converters. diff --git a/.claude/skills/migrate-junit-source-to-tabletest/SKILL.md b/.claude/skills/migrate-junit-source-to-tabletest/SKILL.md new file mode 100644 index 00000000000..062abd87d72 --- /dev/null +++ b/.claude/skills/migrate-junit-source-to-tabletest/SKILL.md @@ -0,0 +1,79 @@ +--- +name: migrate-junit-source-to-tabletest +description: Convert JUnit 5 @MethodSource/@CsvSource/@ValueSource parameterized tests to @TableTest (JDK8) +--- +Goal: Migrate JUnit 5 parameterized tests using @MethodSource/@CsvSource/@ValueSource to @TableTest with minimal churn and passing tests. + +Process (do in this order): +1) Locate targets via Grep (no agent subprocess). Search for: "@ParameterizedTest", "@MethodSource", "@CsvSource", "@ValueSource". +2) Read all matching files up front (parallel is OK). +3) Convert eligible tests to @TableTest. +4) Write each modified file once in full using Write (no incremental per-test edits). +5) Run module tests once and verify "BUILD SUCCESSFUL". If failed, inspect JUnit XML report. + +Import: `import org.tabletest.junit.TableTest;` + +JDK 8 rules: +- No text blocks. +- @TableTest must use String[] annotation array syntax: + ``` + @TableTest({ + "a | b", + "1 | 2" + }) + ``` + +Table formatting rules (mandatory): +- Always include a header row (parameter names). +- Always add a "scenario" column; using common sense for naming; scenario is NOT a method parameter. +- Use '|' as delimiter. +- Align columns with spaces so pipes line up vertically. +- Prefer single quotes for strings requiring quotes (e.g., 'a|b', '[]', '{}', ' '). +- Use value sets (`{a, b, c}`) instead of matrix-style repetition when only one dimension varies across otherwise-identical rows. + +Conversions: +A) @CsvSource +- Remove @ParameterizedTest and @CsvSource. +- If delimiter is '|': rows map directly to @TableTest. +- If delimiter is ',' (default): replace ',' with '|' in rows. + +B) @ValueSource +- Keep single-parameter tests on `@ValueSource` (and `@NullSource` when null cases are needed). +- Otherwise convert to @TableTest with header from parameter name. +- Each value becomes one row. +- Add "scenario" column using common sense for name. + +C) @MethodSource (convert only if values are representable as strings) +- Convert when argument values are primitives, strings, enums, booleans, nulls, and simple collection literals supported by TableTest: + - Array: [a, b, ...] + - List: [a, b, ...] + - Set: {a, b, ...} + - Map: [k: v, ...] +- `@TableTest` and `@MethodSource` may be combined on the same `@ParameterizedTest` when most cases are tabular but a few cases require programmatic setup. +- In combined mode, keep table-friendly cases in `@TableTest`, and put only non-tabular/complex cases in `@MethodSource`. +- If `@TableTest` is not viable for the test at all, use `@MethodSource` only. +- For `@MethodSource`, name the arguments method `Arguments` (camelCase, e.g. `testMethodArguments`) and return `Stream` using `Stream.of(...)` and `arguments(...)` with static import. +- Blank cell = null (non-primitive). +- '' = empty string. +- For String params that start with '[' or '{', quote to avoid collection parsing (prefer '[]'/'{}'). + +D) @TypeConverter +- Use `@TypeConverter` for symbolic constants used by migrated table rows (e.g. `Long.MAX_VALUE`, `DDSpanId.MAX`). +- Prefer explicit one-case-one-return mappings. +- Prefer shared converter utilities (e.g. in `utils/test-utils`) when reuse across modules is likely. + +Scenario handling: +- If MethodSource includes a leading description string OR @ParameterizedTest(name=...) uses {0}, convert that to a scenario column and remove that parameter from method signature. + +Cleanup: +- Delete now-unused @MethodSource provider methods and unused imports. + +Mixed eligibility: +- Prefer combining `@TableTest` + `@MethodSource` on one `@ParameterizedTest` when only some cases are complex. + +Do NOT convert when: +- Most rows require complex builders/mocks. + +Test command (exact): +./gradlew :path:to:module:test --rerun-tasks 2>&1 | tail -20 +- If BUILD FAILED: cat path/to/module/build/test-results/test/TEST-*.xml diff --git a/.claude/skills/techdebt/SKILL.md b/.claude/skills/techdebt/SKILL.md new file mode 100644 index 00000000000..6929e6e2240 --- /dev/null +++ b/.claude/skills/techdebt/SKILL.md @@ -0,0 +1,67 @@ +--- +name: techdebt +description: Analyze branch changes for technical debt, code duplication, and unnecessary complexity +user-invocable: true +context: fork +allowed-tools: + - Bash + - Read + - Grep + - Glob +--- + +# Techdebt Cleanup Skill + +Analyze changes on the current branch to identify and fix technical debt, code duplication, and unnecessary complexity. + +## Instructions + +### Step 1: Get Branch Changes + +Find the merge-base (where this branch diverged from master) and compare against it: + +```bash +# Find upstream (DataDog org repo) +UPSTREAM=$(git remote -v | grep -E 'DataDog/[^/]+(.git)?\s' | head -1 | awk '{print $1}') +if [ -z "$UPSTREAM" ]; then + echo "No DataDog upstream found, using origin" + UPSTREAM="origin" +fi + +# Find the merge-base (commit where this branch diverged from master) +MERGE_BASE=$(git merge-base HEAD ${UPSTREAM}/master) +echo "Comparing changes introduced on this branch since diverging from master using base commit: $MERGE_BASE" + +git diff $MERGE_BASE --stat +git diff $MERGE_BASE --name-status +``` + +If no changes exist, inform the user and stop. + +If changes exist, read the diff and the full content of modified source files (not test files) to understand context. + +### Step 2: Analyze for Issues + +Look for: + +**Code Duplication** +- Similar code blocks that should be extracted into shared functions +- Copy-pasted logic with minor variations + +**Unnecessary Complexity** +- Over-engineered solutions (abstractions used only once) +- Excessive indirection or layers +- Backward compatibility shims that aren't needed + +**Redundant Code** +- Dead code paths +- Overly defensive checks for impossible scenarios + +### Step 3: Report and Fix + +Present a concise summary of issues found with file:line references. + +Then ask the user if they want you to fix the issues. When fixing: +- Make one logical change at a time +- Do NOT change behavior, only refactor +- Skip trivial or stylistic issues diff --git a/buildSrc/.kotlin/sessions/kotlin-compiler-8328714832012535211.salive b/.codex similarity index 100% rename from buildSrc/.kotlin/sessions/kotlin-compiler-8328714832012535211.salive rename to .codex diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index e4dd8a0f757..41cc9a818a9 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -4,9 +4,22 @@ # Default owners, overridden by file/directory specific owners below * @DataDog/apm-java + # @DataDog/apm-idm-java /dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/ @DataDog/apm-idm-java /dd-java-agent/instrumentation/ @DataDog/apm-idm-java +/dd-smoke-tests/armeria-grpc/ @DataDog/apm-idm-java +/dd-smoke-tests/cli/ @DataDog/apm-idm-java +/dd-smoke-tests/jersey-2/ @DataDog/apm-idm-java +/dd-smoke-tests/jersey-3/ @DataDog/apm-idm-java +/dd-smoke-tests/play-*/ @DataDog/apm-idm-java +/dd-smoke-tests/quarkus/ @DataDog/apm-idm-java +/dd-smoke-tests/quarkus-native/ @DataDog/apm-idm-java +/dd-smoke-tests/ratpack-1.5/ @DataDog/apm-idm-java +/dd-smoke-tests/resteasy/ @DataDog/apm-idm-java +/dd-smoke-tests/spring*/ @DataDog/apm-idm-java +/dd-smoke-tests/vertx-*/ @DataDog/apm-idm-java +/dd-smoke-tests/wildfly/ @DataDog/apm-idm-java # @DataDog/apm-release-platform /.gitlab/ @DataDog/apm-release-platform @@ -14,7 +27,11 @@ # @DataDog/apm-sdk-capabilities-java /dd-java-agent/agent-otel @DataDog/apm-sdk-capabilities-java +/dd-smoke-tests/log-injection @DataDog/apm-sdk-capabilities-java +/dd-smoke-tests/opentelemetry @DataDog/apm-sdk-capabilities-java +/dd-smoke-tests/opentracing @DataDog/apm-sdk-capabilities-java /dd-smoke-tests/sample-trace @DataDog/apm-sdk-capabilities-java +/dd-smoke-tests/tracer-flare @DataDog/apm-sdk-capabilities-java /dd-trace-core/src/main/java/datadog/trace/core/baggage @DataDog/apm-sdk-capabilities-java /dd-trace-core/src/test/groovy/datadog/trace/core/baggage @DataDog/apm-sdk-capabilities-java /dd-trace-core/src/main/java/datadog/trace/core/propagation @DataDog/apm-sdk-capabilities-java @@ -28,21 +45,26 @@ /internal-api/src/test/groovy/datadog/trace/api/sampling @DataDog/apm-sdk-capabilities-java # @DataDog/apm-serverless -/dd-trace-core/src/main/java/datadog/trace/lambda/ @DataDog/apm-serverless -/dd-trace-core/src/test/groovy/datadog/trace/lambda/ @DataDog/apm-serverless -**/InferredProxy*.java @DataDog/apm-serverless -**/InferredProxy*.groovy @DataDog/apm-serverless +/dd-java-agent/instrumentation/aws-java/aws-java-lambda-handler-1.2/ @DataDog/apm-serverless +/dd-java-agent/instrumentation/azure-functions/ @DataDog/apm-serverless +/dd-java-agent/instrumentation/azure-functions-1.2.2/ @DataDog/apm-serverless +/dd-trace-core/src/main/java/datadog/trace/lambda/ @DataDog/apm-serverless +/dd-trace-core/src/test/groovy/datadog/trace/lambda/ @DataDog/apm-serverless +/utils/container-utils/ @DataDog/apm-serverless +**/InferredProxy*.java @DataDog/apm-serverless +**/InferredProxy*.groovy @DataDog/apm-serverless # @DataDog/apm-lang-platform-java -/.circleci/ @DataDog/apm-lang-platform-java -/.github/ @DataDog/apm-lang-platform-java -/benchmark/ @DataDog/apm-lang-platform-java -/components/ @DataDog/apm-lang-platform-java -/dd-java-agent/instrumentation/java-* @DataDog/apm-lang-platform-java -/metadata/ @DataDog/apm-lang-platform-java -/remote-config/ @DataDog/apm-lang-platform-java -/telemetry/ @DataDog/apm-lang-platform-java -/test-published-dependencies/ @DataDog/apm-lang-platform-java +/.github/ @DataDog/apm-lang-platform-java +/benchmark/ @DataDog/apm-lang-platform-java +/components/ @DataDog/apm-lang-platform-java +/dd-java-agent/instrumentation/java/ @DataDog/apm-lang-platform-java +/dd-smoke-tests/concurrent/ @DataDog/apm-lang-platform-java +/dd-smoke-tests/lib-injection/ @DataDog/apm-lang-platform-java +/metadata/ @DataDog/apm-lang-platform-java +/remote-config/ @DataDog/apm-lang-platform-java +/telemetry/ @DataDog/apm-lang-platform-java +/test-published-dependencies/ @DataDog/apm-lang-platform-java # @DataDog/asm-java (AppSec/IAST) /buildSrc/call-site-instrumentation-plugin/ @DataDog/asm-java @@ -51,13 +73,17 @@ /dd-java-agent/appsec/appsec-test-fixtures/ @DataDog/asm-java /dd-java-agent/instrumentation/*iast* @DataDog/asm-java /dd-java-agent/instrumentation/*appsec* @DataDog/asm-java -/dd-java-agent/instrumentation/json/ @DataDog/asm-java -/dd-java-agent/instrumentation/snakeyaml/ @DataDog/asm-java -/dd-java-agent/instrumentation/velocity/ @DataDog/asm-java +/dd-java-agent/instrumentation/org-json-20230227/ @DataDog/asm-java +/dd-java-agent/instrumentation/snakeyaml-1.33/ @DataDog/asm-java +/dd-java-agent/instrumentation/velocity-1.5/ @DataDog/asm-java /dd-java-agent/instrumentation/freemarker/ @DataDog/asm-java +/dd-java-agent/instrumentation/datadog/asm/ @DataDog/asm-java +/dd-smoke-tests/apm-tracing-disabled/ @DataDog/asm-java +/dd-smoke-tests/appsec/ @DataDog/asm-java +/dd-smoke-tests/iast-propagation/ @DataDog/asm-java /dd-smoke-tests/iast-util/ @DataDog/asm-java /dd-smoke-tests/spring-security/ @DataDog/asm-java -/dd-java-agent/instrumentation/commons-fileupload/ @DataDog/asm-java +/dd-java-agent/instrumentation/commons-fileupload-1.5/ @DataDog/asm-java /dd-java-agent/instrumentation/spring/spring-security/ @DataDog/asm-java /dd-trace-api/src/main/java/datadog/trace/api/aiguard/ @DataDog/asm-java /dd-trace-api/src/main/java/datadog/trace/api/EventTracker.java @DataDog/asm-java @@ -75,67 +101,97 @@ **/*Waf*.java @DataDog/asm-java **/*Waf*.groovy @DataDog/asm-java -# @DataDog/ci-app-libraries-java -/dd-java-agent/agent-ci-visibility/ @DataDog/ci-app-libraries-java -/dd-java-agent/instrumentation/cucumber/ @DataDog/ci-app-libraries-java -/dd-java-agent/instrumentation/jacoco/ @DataDog/ci-app-libraries-java -/dd-java-agent/instrumentation/junit @DataDog/ci-app-libraries-java -/dd-java-agent/instrumentation/karate/ @DataDog/ci-app-libraries-java -/dd-java-agent/instrumentation/scalatest/ @DataDog/ci-app-libraries-java -/dd-java-agent/instrumentation/selenium/ @DataDog/ci-app-libraries-java -/dd-java-agent/instrumentation/testng/ @DataDog/ci-app-libraries-java -/dd-java-agent/instrumentation/gradle/ @DataDog/ci-app-libraries-java -/dd-java-agent/instrumentation/gradle-testing/ @DataDog/ci-app-libraries-java -/dd-java-agent/instrumentation/maven @DataDog/ci-app-libraries-java -/dd-java-agent/instrumentation/maven-surefire/ @DataDog/ci-app-libraries-java -/dd-java-agent/instrumentation/weaver/ @DataDog/ci-app-libraries-java -/dd-smoke-tests/gradle/ @DataDog/ci-app-libraries-java -/dd-smoke-tests/junit-console/ @DataDog/ci-app-libraries-java -/dd-smoke-tests/maven/ @DataDog/ci-app-libraries-java -/internal-api/src/main/java/datadog/trace/api/git/ @DataDog/ci-app-libraries-java -**/civisibility/ @DataDog/ci-app-libraries-java -**/CiVisibility*.java @DataDog/ci-app-libraries-java -**/CiVisibility*.groovy @DataDog/ci-app-libraries-java +# @DataDog/ci-app-libraries +/dd-java-agent/agent-ci-visibility/ @DataDog/ci-app-libraries +/dd-java-agent/instrumentation/cucumber-5.4/ @DataDog/ci-app-libraries +/dd-java-agent/instrumentation/jacoco-0.8.9/ @DataDog/ci-app-libraries +/dd-java-agent/instrumentation/junit @DataDog/ci-app-libraries +/dd-java-agent/instrumentation/karate-1.0/ @DataDog/ci-app-libraries +/dd-java-agent/instrumentation/scalatest-3.0.8/ @DataDog/ci-app-libraries +/dd-java-agent/instrumentation/selenium-3.13/ @DataDog/ci-app-libraries +/dd-java-agent/instrumentation/testng/ @DataDog/ci-app-libraries +/dd-java-agent/instrumentation/gradle/ @DataDog/ci-app-libraries +/dd-java-agent/instrumentation/gradle-testing-5.1/ @DataDog/ci-app-libraries +/dd-java-agent/instrumentation/maven @DataDog/ci-app-libraries +/dd-java-agent/instrumentation/weaver-0.9/ @DataDog/ci-app-libraries +/dd-smoke-tests/gradle/ @DataDog/ci-app-libraries +/dd-smoke-tests/junit-console/ @DataDog/ci-app-libraries +/dd-smoke-tests/maven/ @DataDog/ci-app-libraries +/internal-api/src/main/java/datadog/trace/api/git/ @DataDog/ci-app-libraries +**/civisibility/ @DataDog/ci-app-libraries +**/CiVisibility*.java @DataDog/ci-app-libraries +**/CiVisibility*.groovy @DataDog/ci-app-libraries # @DataDog/debugger-java (Live Debugger) -/dd-java-agent/agent-debugger/ @DataDog/debugger-java -/dd-smoke-tests/debugger-integration-tests/ @DataDog/debugger-java -/internal-api/src/main/java/datadog/trace/api/debugger/ @DataDog/debugger-java +/dd-java-agent/agent-debugger/ @DataDog/debugger-java +/dd-java-agent/instrumentation/datadog/dynamic-instrumentation/ @DataDog/debugger-java +/dd-smoke-tests/debugger-integration-tests/ @DataDog/debugger-java +/internal-api/src/main/java/datadog/trace/api/debugger/ @DataDog/debugger-java + # @DataDog/data-jobs-monitoring /dd-java-agent/instrumentation/spark/ @DataDog/data-jobs-monitoring -/dd-java-agent/instrumentation/spark-executor/ @DataDog/data-jobs-monitoring # @DataDog/data-streams-monitoring -/dd-java-agent/testing/src/main/groovy/datadog/trace/agent/test/datastreams @DataDog/data-streams-monitoring -/dd-trace-core/src/main/java/datadog/trace/core/datastreams @DataDog/data-streams-monitoring -/dd-trace-core/src/test/groovy/datadog/trace/core/datastreams @DataDog/data-streams-monitoring -/internal-api/src/main/java/datadog/trace/api/datastreams @DataDog/data-streams-monitoring -/internal-api/src/test/groovy/datadog/trace/api/datastreams @DataDog/data-streams-monitoring -**/datastreams/ @DataDog/data-streams-monitoring -**/DataStreams* @DataDog/data-streams-monitoring +/dd-java-agent/instrumentation-testing/src/main/groovy/datadog/trace/agent/test/datastreams @DataDog/data-streams-monitoring +/dd-trace-core/src/main/java/datadog/trace/core/datastreams @DataDog/data-streams-monitoring +/dd-trace-core/src/test/groovy/datadog/trace/core/datastreams @DataDog/data-streams-monitoring +/internal-api/src/main/java/datadog/trace/api/datastreams @DataDog/data-streams-monitoring +/dd-smoke-tests/datastreams/ @DataDog/data-streams-monitoring +/internal-api/src/test/groovy/datadog/trace/api/datastreams @DataDog/data-streams-monitoring +**/datastreams/ @DataDog/data-streams-monitoring +**/DataStreams* @DataDog/data-streams-monitoring +**/dsmTest/** @DataDog/data-streams-monitoring + +/dd-java-agent/instrumentation/armeria/armeria-grpc-0.84/ @DataDog/data-streams-monitoring @DataDog/apm-idm-java +/dd-java-agent/instrumentation/avro-1.11.3/ @DataDog/data-streams-monitoring @DataDog/apm-idm-java +/dd-java-agent/instrumentation/aws-java/aws-java-sns-1.0/ @DataDog/data-streams-monitoring @DataDog/apm-idm-java +/dd-java-agent/instrumentation/aws-java/aws-java-sns-2.0/ @DataDog/data-streams-monitoring @DataDog/apm-idm-java +/dd-java-agent/instrumentation/aws-java/aws-java-sqs-1.0/ @DataDog/data-streams-monitoring @DataDog/apm-idm-java +/dd-java-agent/instrumentation/aws-java/aws-java-sqs-2.0/ @DataDog/data-streams-monitoring @DataDog/apm-idm-java +/dd-java-agent/instrumentation/confluent-schema-registry/ @DataDog/data-streams-monitoring @DataDog/apm-idm-java +/dd-java-agent/instrumentation/google-pubsub-1.116/ @DataDog/data-streams-monitoring @DataDog/apm-idm-java +/dd-java-agent/instrumentation/grpc-1.5/ @DataDog/data-streams-monitoring @DataDog/apm-idm-java +/dd-java-agent/instrumentation/kafka/kafka-clients-0.11/ @DataDog/data-streams-monitoring @DataDog/apm-idm-java +/dd-java-agent/instrumentation/kafka/kafka-clients-3.8/ @DataDog/data-streams-monitoring @DataDog/apm-idm-java +/dd-java-agent/instrumentation/kafka/kafka-connect-0.11/ @DataDog/data-streams-monitoring @DataDog/apm-idm-java +/dd-java-agent/instrumentation/kafka/kafka-streams-0.11/ @DataDog/data-streams-monitoring @DataDog/apm-idm-java +/dd-java-agent/instrumentation/protobuf-3.0/ @DataDog/data-streams-monitoring @DataDog/apm-idm-java +/dd-java-agent/instrumentation/rabbitmq-amqp-2.7/ @DataDog/data-streams-monitoring @DataDog/apm-idm-java +/dd-smoke-tests/kafka-2/ @DataDog/data-streams-monitoring @DataDog/apm-idm-java +/dd-smoke-tests/kafka-3/ @DataDog/data-streams-monitoring @DataDog/apm-idm-java + +# @DataDog/feature-flagging-and-experimentation-sdk +/dd-smoke-tests/openfeature/ @DataDog/feature-flagging-and-experimentation-sdk +/products/feature-flagging/ @DataDog/feature-flagging-and-experimentation-sdk # @DataDog/profiling-java /dd-java-agent/agent-profiling/ @DataDog/profiling-java /dd-java-agent/agent-crashtracking/ @DataDog/profiling-java -/dd-java-agent/instrumentation/exception-profiling/ @DataDog/profiling-java +/dd-java-agent/instrumentation/datadog/profiling/ @DataDog/profiling-java /dd-java-agent/instrumentation/java/java-nio-1.8/ @DataDog/profiling-java /dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/jfr/ @DataDog/profiling-java /dd-java-agent/agent-bootstrap/src/main/java11/datadog/trace/bootstrap/instrumentation/jfr/ @DataDog/profiling-java +/dd-smoke-tests/crashtracking/ @DataDog/profiling-java +/dd-smoke-tests/profiling-integration-tests/ @DataDog/profiling-java /dd-trace-api/src/main/java/datadog/trace/api/profiling @DataDog/profiling-java /internal-api/src/main/java/datadog/trace/api/profiling @DataDog/profiling-java /internal-api/src/main/java/datadog/trace/api/EndpointCheckpointer.java @DataDog/profiling-java /internal-api/src/main/java/datadog/trace/api/EndpointTracker.java @DataDog/profiling-java -/dd-smoke-tests/profiling-integration-tests/ @DataDog/profiling-java # @DataDog/ml-observability -dd-trace-api/src/main/java/datadog/trace/api/llmobs/ @DataDog/ml-observability -dd-java-agent/agent-llmobs/ @DataDog/ml-observability -dd-trace-core/src/main/java/datadog/trace/llmobs/ @DataDog/ml-observability -dd-trace-core/src/test/groovy/datadog/trace/llmobs/ @DataDog/ml-observability +/dd-trace-api/src/main/java/datadog/trace/api/llmobs/ @DataDog/ml-observability +/dd-java-agent/agent-llmobs/ @DataDog/ml-observability +/dd-trace-core/src/main/java/datadog/trace/llmobs/ @DataDog/ml-observability +/dd-trace-core/src/test/groovy/datadog/trace/llmobs/ @DataDog/ml-observability + +# @DataDog/database-monitoring +/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/dbm/ @DataDog/database-monitoring @DataDog/apm-idm-java +/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/dbm/ @DataDog/database-monitoring @DataDog/apm-idm-java # @DataDog/rum /internal-api/src/main/java/datadog/trace/api/rum/ @DataDog/rum /internal-api/src/test/groovy/datadog/trace/api/rum/ @DataDog/rum +/dd-smoke-tests/rum/ @DataDog/rum /telemetry/src/main/java/datadog/telemetry/rum/ @DataDog/rum /telemetry/src/test/groovy/datadog/telemetry/rum/ @DataDog/rum diff --git a/.github/chainguard/self.enforce-datadog-merge-queue.comment-pr.sts.yaml b/.github/chainguard/self.enforce-datadog-merge-queue.comment-pr.sts.yaml new file mode 100644 index 00000000000..6c682eb1543 --- /dev/null +++ b/.github/chainguard/self.enforce-datadog-merge-queue.comment-pr.sts.yaml @@ -0,0 +1,11 @@ +issuer: https://token.actions.githubusercontent.com + +subject: repo:DataDog/dd-trace-java:pull_request + +claim_pattern: + event_name: pull_request + job_workflow_ref: DataDog/dd-trace-java/\.github/workflows/enforce-datadog-merge-queue\.yaml@refs/pull/[0-9]+/merge + +permissions: + issues: write + pull_requests: write diff --git a/.github/chainguard/self.pin-system-tests.create-pr.sts.yaml b/.github/chainguard/self.pin-system-tests.create-pr.sts.yaml new file mode 100644 index 00000000000..6a8a8fea85c --- /dev/null +++ b/.github/chainguard/self.pin-system-tests.create-pr.sts.yaml @@ -0,0 +1,13 @@ +issuer: https://token.actions.githubusercontent.com + +subject_pattern: repo:DataDog/dd-trace-java:ref:refs/(heads/master|tags/v\d+\.\d+\.0) + +claim_pattern: + event_name: (workflow_dispatch|push) + ref: refs/(heads/master|tags/v\d+\.\d+\.0) + job_workflow_ref: DataDog/dd-trace-java/\.github/workflows/create-release-branch\.yaml@refs/(heads/master|tags/v\d+\.\d+\.0) + +permissions: + contents: write + pull_requests: write + workflows: write diff --git a/.github/chainguard/self.update-smoke-test-latest-versions.create-pr.sts.yaml b/.github/chainguard/self.update-smoke-test-latest-versions.create-pr.sts.yaml new file mode 100644 index 00000000000..fc84cb8d5a4 --- /dev/null +++ b/.github/chainguard/self.update-smoke-test-latest-versions.create-pr.sts.yaml @@ -0,0 +1,13 @@ +issuer: https://token.actions.githubusercontent.com + +subject: repo:DataDog/dd-trace-java:ref:refs/heads/master + +claim_pattern: + event_name: (schedule|workflow_dispatch) + ref: refs/heads/master + ref_protected: "true" + job_workflow_ref: DataDog/dd-trace-java/\.github/workflows/update-smoke-test-latest-versions\.yaml@refs/heads/master + +permissions: + contents: write + pull_requests: write diff --git a/.github/chainguard/self.update-system-tests.create-pr.sts.yaml b/.github/chainguard/self.update-system-tests.create-pr.sts.yaml deleted file mode 100644 index 361668e8735..00000000000 --- a/.github/chainguard/self.update-system-tests.create-pr.sts.yaml +++ /dev/null @@ -1,12 +0,0 @@ -issuer: https://token.actions.githubusercontent.com - -subject_pattern: repo:DataDog/dd-trace-java:ref:refs/(heads/master|tags/v[0-9]+.[0-9]+.0) - -claim_pattern: - event_name: (push|workflow_dispatch) - ref: refs/(heads/master|tags/v[0-9]+\.[0-9]+\.0) - job_workflow_ref: DataDog/dd-trace-java/\.github/workflows/create-release-branch\.yaml@refs/heads/master - -permissions: - contents: write - pull_requests: write diff --git a/.github/dependabot.yml b/.github/dependabot.yml index f914fd12ade..143febaaeb0 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -1,21 +1,38 @@ -# To get started with Dependabot version updates, you'll need to specify which -# package ecosystems to update and where the package manifests are located. -# Please see the documentation for all configuration options: # https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file - version: 2 updates: - - package-ecosystem: "github-actions" - directory: "/" + - package-ecosystem: github-actions + directory: / schedule: - interval: "weekly" + interval: weekly labels: - - "comp: tooling" - - "tag: dependencies" - - "tag: no release notes" + - 'comp: tooling' + - 'tag: dependencies' + - 'tag: no release notes' commit-message: - prefix: "chore(ci): " + prefix: 'chore(ci): ' groups: gh-actions-packages: patterns: - - "*" + - '*' + cooldown: + default-days: 2 + + - package-ecosystem: gradle + directory: / + schedule: + interval: weekly + allow: + - dependency-name: gradle + ignore: + - dependency-name: gradle + update-types: + - version-update:semver-major + labels: + - 'comp: tooling' + - 'tag: dependencies' + - 'tag: no release notes' + commit-message: + prefix: 'chore(build): ' + cooldown: + default-days: 2 diff --git a/.github/g2j-migrated-modules.txt b/.github/g2j-migrated-modules.txt new file mode 100644 index 00000000000..c1f05f83b15 --- /dev/null +++ b/.github/g2j-migrated-modules.txt @@ -0,0 +1,11 @@ +# This file lists modules that have been migrated from Groovy to Java / JUnit 5. +# New *.groovy files under any src/<*test*>/groovy/ path +# in these modules will fail the 'enforce-groovy-migration' PR check. +# +# After a module is migrated, add it on a new line here. +# Use the filesystem path prefix as seen below. + +buildSrc/call-site-instrumentation-plugin +components/json +dd-java-agent/instrumentation/sofarpc/sofarpc-5.0 +dd-trace-api diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 9c15da43d42..c4c494c58b1 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -6,15 +6,17 @@ # Contributor Checklist -- Format the title [according the contribution guidelines](https://github.com/DataDog/dd-trace-java/blob/master/CONTRIBUTING.md#title-format) -- Assign the `type:` and (`comp:` or `inst:`) labels in addition to [any useful labels](https://github.com/DataDog/dd-trace-java/blob/master/CONTRIBUTING.md#labels) -- Don't use `close`, `fix` or any [linking keywords](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) when referencing an issue. +- Format the title according to [the contribution guidelines](https://github.com/DataDog/dd-trace-java/blob/master/CONTRIBUTING.md#title-format) +- Assign the `type:` and (`comp:` or `inst:`) labels in addition to [any other useful labels](https://github.com/DataDog/dd-trace-java/blob/master/CONTRIBUTING.md#labels) +- Avoid using `close`, `fix`, or [any linking keywords](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) when referencing an issue Use `solves` instead, and assign the PR [milestone](https://github.com/DataDog/dd-trace-java/milestones) to the issue -- Update the [CODEOWNERS](https://github.com/DataDog/dd-trace-java/blob/master/.github/CODEOWNERS) file on source file addition, move, or deletion -- Update the [public documentation](https://docs.datadoghq.com/tracing/trace_collection/library_config/java/) in case of new configuration flag or behavior +- Update the [CODEOWNERS](https://github.com/DataDog/dd-trace-java/blob/master/.github/CODEOWNERS) file on source file addition, migration, or deletion +- Update [public documentation](https://docs.datadoghq.com/tracing/trace_collection/library_config/java/) with any new configuration flags or behaviors Jira ticket: [PROJ-IDENT] +***Note:*** **Once your PR is ready to merge, add it to the merge queue by commenting `/merge`.** `/merge -c` cancels the queue request. `/merge -f --reason "reason"` skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see [this doc](https://datadoghq.atlassian.net/wiki/spaces/DEVX/pages/3121612126/MergeQueue). + ' - const blockingComment = comments.data.find(comment => comment.body.includes(commentMarker)) + let blockingComment = comments.data.find(comment => comment.body.includes(commentMarker)) // Create or update blocking comment if there are invalid labels if (hasInvalidLabels) { const commentBody = '**PR Blocked - Invalid Label**\n\n' + @@ -54,7 +174,7 @@ jobs: '**This PR is blocked until:**\n' + '1. The invalid labels are deleted, and\n' + '2. A maintainer deletes this comment to unblock the PR\n\n' + - '**Note:** Simply removing labels from the pull request is not enough - a maintainer must remove the label and delete this comment to remove the block.\n\n' + + '**Note:** Simply removing labels from the pull request is not enough - a maintainer must delete this comment then remove the label to remove the block.\n\n' + commentMarker if (blockingComment) { @@ -78,5 +198,9 @@ jobs: } if (blockingComment) { // Block the PR by failing the workflow - core.setFailed(`PR blocked: Invalid labels detected: (${invalidLabels.join(', ')}). A maintainer must delete the blocking comment after fixing the labels to allow merging.`) + if (hasInvalidLabels) { + core.setFailed(`PR blocked: Invalid labels detected: (${invalidLabels.join(', ')}). A maintainer must delete the blocking comment after fixing the labels to allow merging.`) + } else { + core.setFailed(`PR blocked: A previous blocking comment still exists. A maintainer must delete it to allow merging.`) + } } diff --git a/.github/workflows/check-pull-requests.yaml b/.github/workflows/check-pull-requests.yaml index 803900f2bc6..42058eb42ab 100644 --- a/.github/workflows/check-pull-requests.yaml +++ b/.github/workflows/check-pull-requests.yaml @@ -17,7 +17,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Check pull requests - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # 8.0.0 + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # 9.0.0 with: github-token: ${{secrets.GITHUB_TOKEN}} script: | diff --git a/.github/workflows/comment-on-submodule-update.yaml b/.github/workflows/comment-on-submodule-update.yaml index 336e15f9b05..c81a3004bf5 100644 --- a/.github/workflows/comment-on-submodule-update.yaml +++ b/.github/workflows/comment-on-submodule-update.yaml @@ -17,7 +17,7 @@ jobs: steps: - name: Post comment on submodule update - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # 8.0.0 + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # 9.0.0 with: github-token: ${{secrets.GITHUB_TOKEN}} script: | diff --git a/.github/workflows/create-release-branch.yaml b/.github/workflows/create-release-branch.yaml index 0fadc2d4996..f4cf50ad31f 100644 --- a/.github/workflows/create-release-branch.yaml +++ b/.github/workflows/create-release-branch.yaml @@ -1,9 +1,9 @@ -name: Create Release Branch and Pin System-Tests +name: Create Release Branch and Pin System Tests on: push: tags: - - 'v[0-9]+.[0-9]+.0' # Trigger on minor release tags (e.g. v1.54.0) + - 'v[0-9]+.[0-9]+.0' # Trigger on minor release tags (e.g. v1.54.0) workflow_dispatch: inputs: tag: @@ -15,19 +15,10 @@ jobs: create-release-branch: runs-on: ubuntu-latest permissions: - # contents: write # Allow pushing the empty release branch - contents: read - id-token: write # Required for OIDC token federation + contents: write # Allow pushing the release branch + outputs: + release-branch-name: ${{ steps.define-release-branch.outputs.branch }} steps: - - uses: DataDog/dd-octo-sts-action@acaa02eee7e3bb0839e4272dacb37b8f3b58ba80 # v1.0.3 - id: octo-sts - with: - scope: DataDog/dd-trace-java - policy: self.update-system-tests.create-pr - - - name: Checkout dd-trace-java at tag - uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # 5.0.0 - - name: Determine tag id: determine-tag run: | @@ -42,67 +33,112 @@ jobs: fi echo "tag=${TAG}" >> "$GITHUB_OUTPUT" - - name: Define branch name from tag - id: define-branch + - name: Define release branch name from tag + id: define-release-branch run: | TAG=${{ steps.determine-tag.outputs.tag }} echo "branch=release/${TAG%.0}.x" >> "$GITHUB_OUTPUT" - # - name: Check if branch already exists - # id: check-branch - # run: | - # BRANCH=${{ steps.define-branch.outputs.branch }} - # if git ls-remote --heads origin "$BRANCH" | grep -q "$BRANCH"; then - # echo "creating_new_branch=false" >> "$GITHUB_OUTPUT" - # echo "Branch $BRANCH already exists - skipping following steps" - # else - # echo "creating_new_branch=true" >> "$GITHUB_OUTPUT" - # echo "Branch $BRANCH does not exist - proceeding with following steps" - # fi + - name: Check out repo at tag + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # 6.0.2 + with: + ref: ${{ steps.determine-tag.outputs.tag }} - # - name: Create and push empty release branch - # if: steps.check-branch.outputs.creating_new_branch == 'true' - # run: | - # git checkout -b "${{ steps.define-branch.outputs.branch }}" - # git push -u origin "${{ steps.define-branch.outputs.branch }}" + - name: Check if branch already exists + id: check-release-branch + run: | + BRANCH=${{ steps.define-release-branch.outputs.branch }} + if git ls-remote --heads origin "$BRANCH" | grep -q "$BRANCH"; then + echo "creating_new_branch=false" >> "$GITHUB_OUTPUT" + echo "Branch $BRANCH already exists - skipping creation" + else + echo "creating_new_branch=true" >> "$GITHUB_OUTPUT" + echo "Branch $BRANCH does not exist - creating it now" + fi - - name: Update system-tests references to latest commit SHA on main - # if: steps.check-branch.outputs.creating_new_branch == 'true' - run: BRANCH=main ./tooling/update_system_test_reference.sh + - name: Create and push release branch + if: steps.check-release-branch.outputs.creating_new_branch == 'true' + run: | + git checkout -b "${{ steps.define-release-branch.outputs.branch }}" + git push -u origin "${{ steps.define-release-branch.outputs.branch }}" - - name: Define temp branch name - # if: steps.check-branch.outputs.creating_new_branch == 'true' - id: define-temp-branch - run: echo "temp-branch=ci/pin-system-tests-$(date +'%Y%m%d')" >> $GITHUB_OUTPUT + pin-system-tests: + needs: create-release-branch + runs-on: ubuntu-latest + permissions: + contents: write + id-token: write # required for OIDC token federation + steps: + - uses: DataDog/dd-octo-sts-action@96a25462dbcb10ebf0bfd6e2ccc917d2ab235b9a # v1.0.4 + id: octo-sts + with: + scope: DataDog/dd-trace-java + policy: self.pin-system-tests.create-pr + + - name: Check out repo at release branch + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # 6.0.2 + with: + ref: ${{ needs.create-release-branch.outputs.release-branch-name }} + + - name: Get latest commit SHA of base release branch + id: get-latest-commit-sha + run: | + echo "sha=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT + - name: Define pin-system-tests branch name from date + id: define-pin-branch + run: echo "branch=ci/pin-system-tests-$(date +'%Y%m%d')" >> $GITHUB_OUTPUT + + - name: Check if pin-system-tests branch already exists + id: check-pin-branch + run: | + BRANCH=${{ steps.define-pin-branch.outputs.branch }} + if git ls-remote --heads origin "$BRANCH" | grep -q "$BRANCH"; then + echo "ERROR: Branch $BRANCH already exists - please delete it and re-run the workflow." + exit 1 + else + echo "Branch $BRANCH does not exist - creating it now." + fi + + - name: Update system-tests references to latest commit SHA of system-tests main + run: ./tooling/update_system_test_reference.sh + + - name: Check if changes should be committed + id: check-changes + run: | + if [[ -z "$(git status -s)" ]]; then + echo "ERROR: No changes to commit - the system-tests reference was not updated." + exit 1 + else + echo "Changes to commit:" + git status -s + fi + - name: Commit changes - # if: steps.check-branch.outputs.creating_new_branch == 'true' id: create-commit run: | git config user.name "github-actions[bot]" git config user.email "41898282+github-actions[bot]@users.noreply.github.com" - git commit -m "chore: Pin system-tests for release branch" .github/workflows/run-system-tests.yaml + git commit -m "chore: Pin system-tests for release branch" .github/workflows/run-system-tests.yaml .gitlab-ci.yml echo "commit=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT - - - name: Push changes to temp branch - # if: steps.check-branch.outputs.creating_new_branch == 'true' - uses: DataDog/commit-headless@5a0f3876e0fbdd3a86b3e008acf4ec562db59eee # action/v2.0.1 + + - name: Push changes + uses: DataDog/commit-headless@ad3668640012ec69186398f43d61923f6878bbbe # action/v3.2.0 with: token: "${{ steps.octo-sts.outputs.token }}" - branch: "${{ steps.define-temp-branch.outputs.temp-branch }}" - head-sha: "${{ github.sha }}" + branch: "${{ steps.define-pin-branch.outputs.branch }}" + head-sha: "${{ steps.get-latest-commit-sha.outputs.sha }}" create-branch: true command: push commits: "${{ steps.create-commit.outputs.commit }}" - - name: Create pull request from temp branch to release branch - # if: steps.check-branch.outputs.creating_new_branch == 'true' + - name: Create pull request env: GH_TOKEN: ${{ steps.octo-sts.outputs.token }} run: | - gh pr create --title "Pin system-tests for ${{ steps.define-branch.outputs.branch }}" \ - --base "${{ steps.define-branch.outputs.branch }}" \ - --head "${{ steps.define-temp-branch.outputs.temp-branch }}" \ + gh pr create --title "Pin system tests for release branch" \ + --base ${{ needs.create-release-branch.outputs.release-branch-name }} \ + --head ${{ steps.define-pin-branch.outputs.branch }} \ --label "tag: dependencies" \ --label "tag: no release notes" \ --body "This PR pins the system-tests reference for the release branch." diff --git a/.github/workflows/draft-release-notes-on-tag.yaml b/.github/workflows/draft-release-notes-on-tag.yaml index f0dec1a1ae9..d471015ffe4 100644 --- a/.github/workflows/draft-release-notes-on-tag.yaml +++ b/.github/workflows/draft-release-notes-on-tag.yaml @@ -13,7 +13,7 @@ jobs: steps: - name: Get milestone title id: milestoneTitle - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # 8.0.0 + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # 9.0.0 with: result-encoding: string script: | diff --git a/.github/workflows/enforce-datadog-merge-queue.yaml b/.github/workflows/enforce-datadog-merge-queue.yaml new file mode 100644 index 00000000000..6577861e271 --- /dev/null +++ b/.github/workflows/enforce-datadog-merge-queue.yaml @@ -0,0 +1,40 @@ +name: Enforce Datadog Merge Queue + +on: + pull_request: + types: [opened, synchronize, reopened, enqueued] + branches: + - master + merge_group: + +jobs: + enforce_datadog_merge_queue: + name: Merge queue check + runs-on: ubuntu-latest + permissions: + id-token: write # required for OIDC token federation + steps: + - name: Block GitHub merge queue + if: github.event_name == 'merge_group' + run: | + echo "Merge is handled by the Datadog merge queue system. Use the /merge command to enqueue your PR for merging." + exit 1 + - name: Get OIDC token + if: github.event.action == 'enqueued' + uses: DataDog/dd-octo-sts-action@96a25462dbcb10ebf0bfd6e2ccc917d2ab235b9a # v1.0.4 + id: octo-sts + with: + scope: DataDog/dd-trace-java + policy: self.enforce-datadog-merge-queue.comment-pr + - name: Post /merge comment + if: github.event.action == 'enqueued' + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # 9.0.0 + with: + github-token: ${{ steps.octo-sts.outputs.token }} + script: | + await github.rest.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: context.payload.pull_request.number, + body: '/merge' + }); diff --git a/.github/workflows/enforce-groovy-migration.yaml b/.github/workflows/enforce-groovy-migration.yaml new file mode 100644 index 00000000000..89757b1cf82 --- /dev/null +++ b/.github/workflows/enforce-groovy-migration.yaml @@ -0,0 +1,170 @@ +name: Enforce Groovy Migration +on: + pull_request: + types: [opened, edited, ready_for_review, labeled, unlabeled, synchronize] + branches: + - master + - 'release/v*' + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + enforce_groovy_migration: + name: Enforce Groovy migration + permissions: + issues: write # Required to create a comment on the pull request + pull-requests: write # Required to create a comment on the pull request + contents: read # Required to read migrated modules file + runs-on: ubuntu-latest + steps: + - name: Check for Groovy regressions + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # 9.0.0 + with: + github-token: ${{ secrets.GITHUB_TOKEN }} + script: | + // Skip draft pull requests + if (context.payload.pull_request.draft) { + return + } + + // Check for override label — skip all checks if label present + const labels = context.payload.pull_request.labels.map(l => l.name) + if (labels.includes('tag: override-groovy-enforcement')) { + console.log('tag: override-groovy-enforcement label detected — skipping all checks.') + return + } + + // Read migrated modules list from master + const migratedMods = await github.rest.repos.getContent({ + owner: context.repo.owner, + repo: context.repo.repo, + path: '.github/g2j-migrated-modules.txt', + ref: 'master' + }) + const migratedPrefixes = Buffer.from(migratedMods.data.content, 'base64') + .toString() + .split('\n') + .map(l => l.trim()) + .filter(l => l && !l.startsWith('#')) + + // Get all files changed in this PR + const allFiles = await github.paginate(github.rest.pulls.listFiles, { + owner: context.repo.owner, + repo: context.repo.repo, + pull_number: context.payload.pull_request.number + }) + + // Filter these changed files to newly added Groovy files in any test source set + const addedGroovy = allFiles.filter(f => + f.status === 'added' && + /\/src\/[^/]*[tT]est[^/]*\/groovy\/.*\.groovy$/.test(f.filename) + ) + + // Extract module prefix from file path (everything before /src/(test|testFixtures)/groovy/) + const moduleOf = path => { + const m = path.match(/^(.*?)\/src\/(test|testFixtures)\/groovy\//) + return m ? m[1] : null + } + + // Classify each added Groovy file + const regressions = [] + const warnings = [] + for (const file of addedGroovy) { + const path = file.filename + const mod = moduleOf(path) + if (migratedPrefixes.some(prefix => path.startsWith(prefix + '/'))) { + regressions.push({ path, mod }) + } else if ( + path.startsWith('dd-java-agent/instrumentation/') || + path.startsWith('dd-smoke-tests/') + ) { + // ignore Groovy file additions to instrumentations and smoke-tests for now + } else { + warnings.push({ path, mod }) + } + } + + // Fetch existing comments once + const comments = await github.rest.issues.listComments({ + issue_number: context.payload.pull_request.number, + owner: context.repo.owner, + repo: context.repo.repo + }) + + const regressionMarker = '' + const warningMarker = '' + const existingRegressionComment = comments.data.find(c => c.body.includes(regressionMarker)) + const existingWarningComment = comments.data.find(c => c.body.includes(warningMarker)) + + // Handle regression comment + if (regressions.length > 0) { + const fileList = regressions + .map(({ path, mod }) => `- \`${path}\` (module: \`${mod}\`)`) + .join('\n') + const body = `**❌ Groovy Test Regression Detected**\n\n` + + `The following files add Groovy tests to modules that have been fully migrated to Java / JUnit 5:\n\n` + + `${fileList}\n\n` + + `These modules no longer accept Groovy test files. Please rewrite the test in Java / JUnit 5 instead.\n\n` + + regressionMarker + if (existingRegressionComment) { + await github.rest.issues.updateComment({ + comment_id: existingRegressionComment.id, + owner: context.repo.owner, + repo: context.repo.repo, + body + }) + } else { + await github.rest.issues.createComment({ + issue_number: context.payload.pull_request.number, + owner: context.repo.owner, + repo: context.repo.repo, + body + }) + } + } else if (existingRegressionComment) { + await github.rest.issues.deleteComment({ + comment_id: existingRegressionComment.id, + owner: context.repo.owner, + repo: context.repo.repo + }) + } + + // Handle warning comment + if (warnings.length > 0) { + const fileList = warnings + .map(({ path, mod }) => `- \`${path}\` (module: \`${mod}\`)`) + .join('\n') + const body = `**⚠️ New Groovy Test Files Added**\n\n` + + `The following files add Groovy tests to modules that are candidates for migration to Java / JUnit 5:\n\n` + + `${fileList}\n\n` + + `Consider writing these tests in Java / JUnit 5 instead to help with the ongoing migration effort.\n\n` + + warningMarker + if (existingWarningComment) { + await github.rest.issues.updateComment({ + comment_id: existingWarningComment.id, + owner: context.repo.owner, + repo: context.repo.repo, + body + }) + } else { + await github.rest.issues.createComment({ + issue_number: context.payload.pull_request.number, + owner: context.repo.owner, + repo: context.repo.repo, + body + }) + } + } else if (existingWarningComment) { + await github.rest.issues.deleteComment({ + comment_id: existingWarningComment.id, + owner: context.repo.owner, + repo: context.repo.repo + }) + } + + // Fail the check if there are regressions + if (regressions.length > 0) { + core.setFailed(`${regressions.length} Groovy regression(s) detected in migrated module(s). See PR comment for details. To skip this check entirely, add the 'tag: override-groovy-enforcement' label.`) + } diff --git a/.github/workflows/increment-milestone-on-tag.yaml b/.github/workflows/increment-milestone-on-tag.yaml index 8be620e0612..ef4608582d4 100644 --- a/.github/workflows/increment-milestone-on-tag.yaml +++ b/.github/workflows/increment-milestone-on-tag.yaml @@ -11,7 +11,7 @@ jobs: steps: - name: Close current milestone id: close-milestone - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # 8.0.0 + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # 9.0.0 with: script: | // Get the milestone title ("X.Y.Z") from tag name ("vX.Y.Z") diff --git a/.github/workflows/prune-old-pull-requests.yaml b/.github/workflows/prune-old-pull-requests.yaml index 6993d7204e6..e5633a39415 100644 --- a/.github/workflows/prune-old-pull-requests.yaml +++ b/.github/workflows/prune-old-pull-requests.yaml @@ -13,7 +13,7 @@ jobs: pull-requests: write steps: - name: Prune old pull requests - uses: actions/stale@5f858e3efba33a5ca4407a664cc011ad407f2008 # v10.1.0 + uses: actions/stale@b5d41d4e1d5dceea10e7104786b73624c18a190f # v10.2.0 with: days-before-stale: -1 # Disable general stale bot days-before-pr-stale: 90 # Only enable stale bot for PRs with no activity for 90 days diff --git a/.github/workflows/run-system-tests.yaml b/.github/workflows/run-system-tests.yaml index f2a9487bbf5..2c441f495de 100644 --- a/.github/workflows/run-system-tests.yaml +++ b/.github/workflows/run-system-tests.yaml @@ -1,6 +1,7 @@ name: Run system tests on: + merge_group: pull_request: workflow_dispatch: schedule: @@ -23,13 +24,13 @@ jobs: group: APM Larger Runners steps: - name: Checkout repository - uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # 5.0.0 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # 6.0.2 with: submodules: 'recursive' fetch-depth: 0 - name: Cache Gradle dependencies - uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 with: path: | ~/.gradle/caches @@ -52,7 +53,7 @@ jobs: --build-cache --parallel --stacktrace --no-daemon --max-workers=4 - name: Upload artifact - uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a with: name: binaries path: workspace/dd-java-agent/build/libs/ @@ -61,8 +62,7 @@ jobs: needs: - build # If you change the following comment, update the pattern in the update_system_test_reference.sh script to match. - uses: DataDog/system-tests/.github/workflows/system-tests.yml@main # system tests are pinned for releases only - secrets: inherit + uses: DataDog/system-tests/.github/workflows/system-tests.yml@main # system tests are pinned on release branches only permissions: contents: read id-token: write @@ -70,12 +70,16 @@ jobs: with: library: java # If you change the following comment, update the pattern in the update_system_test_reference.sh script to match. - ref: main # system tests are pinned for releases only + ref: "main" # system tests are pinned on release branches only binaries_artifact: binaries desired_execution_time: 900 # 15 minutes scenarios_groups: tracer-release - excluded_scenarios: CROSSED_TRACING_LIBRARIES,INTEGRATIONS_AWS,APM_TRACING_E2E_OTEL,APM_TRACING_E2E_SINGLE_SPAN,PROFILING # require AWS and datadog credentials - skip_empty_scenarios: true + excluded_scenarios: APM_TRACING_E2E_OTEL,APM_TRACING_E2E_SINGLE_SPAN,PROFILING # exclude flaky scenarios + skip_empty_scenarios: ${{ github.event_name != 'push' && github.event_name != 'schedule' }} + push_to_test_optimization: true + secrets: + DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }} + DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }} # Ensure the main job is run to completion check: @@ -84,4 +88,15 @@ jobs: if: ${{ always() }} needs: [build, main] steps: - - run: exit 0 + - name: Fail if build failed + if: ${{ needs.build.result != 'success' }} + run: | + echo "❌ Build job did not succeed: ${{ needs.build.result }}" + exit 1 + - name: Fail if main failed or is skipped + if: ${{ needs.main.result != 'success' }} + run: | + echo "❌ Main job did not succeed: ${{ needs.main.result }}" + exit 1 + - name: Success + run: echo "✅ All required jobs succeeded." diff --git a/.github/workflows/update-docker-build-image.yaml b/.github/workflows/update-docker-build-image.yaml deleted file mode 100644 index 5346f54763a..00000000000 --- a/.github/workflows/update-docker-build-image.yaml +++ /dev/null @@ -1,96 +0,0 @@ -name: Update Docker Build Image - -on: - schedule: - # A day after creating the tag from https://github.com/DataDog/dd-trace-java-docker-build/blob/master/.github/workflows/docker-tag.yml - - cron: "0 0 1 2,5,8,11 *" - workflow_dispatch: - inputs: - tag: - description: "The tag to use for the Docker build image" - required: true - default: "vYY.MM" - -jobs: - update-docker-build-image: - runs-on: ubuntu-latest - permissions: - contents: read - id-token: write # Required for OIDC token federation - steps: - - uses: DataDog/dd-octo-sts-action@acaa02eee7e3bb0839e4272dacb37b8f3b58ba80 # v1.0.3 - id: octo-sts - with: - scope: DataDog/dd-trace-java - policy: self.update-docker-build-image.create-pr - - - name: Checkout the repository - uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0 - - name: Define the Docker build image tag to use - id: define-tag - run: | - if [ -n "${{ github.event.inputs.tag }}" ]; then - TAG=${{ github.event.inputs.tag }} - else - CURRENT_MONTH=$(date +%m) - CURRENT_YEAR=$(date +%y) - case $CURRENT_MONTH in - 01) TAG_DATE="$(($CURRENT_YEAR - 1)).10" ;; - 02|03|04) TAG_DATE="${CURRENT_YEAR}.01" ;; - 05|06|07) TAG_DATE="${CURRENT_YEAR}.04" ;; - 08|09|10) TAG_DATE="${CURRENT_YEAR}.07" ;; - 11|12) TAG_DATE="${CURRENT_YEAR}.10" ;; - esac - TAG="v${TAG_DATE}" - fi - echo "tag=${TAG}" >> "$GITHUB_OUTPUT" - echo "::notice::Using Docker build image tag: ${TAG}" - - name: Update the Docker build image in GitLab CI config - run: | - sed -i -E 's|(BUILDER_IMAGE_VERSION_PREFIX:)[^#]*([#].*)|\1 "${{ steps.define-tag.outputs.tag }}-" \2|' .gitlab-ci.yml - - name: Check if changes should be committed - id: check-changes - run: | - if [[ -z "$(git status -s)" ]]; then - echo "No changes to commit, exiting." - echo "commit_changes=false" >> "$GITHUB_OUTPUT" - exit 0 - else - echo "commit_changes=true" >> "$GITHUB_OUTPUT" - fi - - name: Pick a branch name - if: steps.check-changes.outputs.commit_changes == 'true' - id: define-branch - run: echo "branch=ci/update-docker-build-image-$(date +'%Y%m%d')" >> $GITHUB_OUTPUT - - name: Commit changes - if: steps.check-changes.outputs.commit_changes == 'true' - id: create-commit - run: | - git config user.name "github-actions[bot]" - git config user.email "41898282+github-actions[bot]@users.noreply.github.com" - git commit -m "feat(ci): Update Docker build image" .gitlab-ci.yml - echo "commit=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT - - name: Push changes - uses: DataDog/commit-headless@5a0f3876e0fbdd3a86b3e008acf4ec562db59eee # action/v2.0.1 - if: steps.check-changes.outputs.commit_changes == 'true' - with: - token: "${{ steps.octo-sts.outputs.token }}" - branch: "${{ steps.define-branch.outputs.branch }}" - # for scheduled runs, sha is the tip of the default branch - # for dispatched runs, sha is the tip of the branch it was dispatched on - head-sha: "${{ github.sha }}" - create-branch: true - command: push - commits: "${{ steps.create-commit.outputs.commit }}" - - name: Create pull request - if: steps.check-changes.outputs.commit_changes == 'true' - env: - GH_TOKEN: ${{ steps.octo-sts.outputs.token }} - run: | - gh pr create --title "Update Docker build image" \ - --base master \ - --head ${{ steps.define-branch.outputs.branch }} \ - --label "comp: tooling" \ - --label "type: enhancement" \ - --label "tag: no release notes" \ - --body "This PR updates the Docker build image to ${{ steps.define-tag.outputs.tag }}." diff --git a/.github/workflows/update-gradle-dependencies.yaml b/.github/workflows/update-gradle-dependencies.yaml index 521618d38cd..ef2ff1e301b 100644 --- a/.github/workflows/update-gradle-dependencies.yaml +++ b/.github/workflows/update-gradle-dependencies.yaml @@ -12,74 +12,161 @@ jobs: contents: read id-token: write # Required for OIDC token federation steps: - - uses: DataDog/dd-octo-sts-action@acaa02eee7e3bb0839e4272dacb37b8f3b58ba80 # v1.0.3 + - uses: DataDog/dd-octo-sts-action@96a25462dbcb10ebf0bfd6e2ccc917d2ab235b9a # v1.0.4 id: octo-sts with: scope: DataDog/dd-trace-java policy: self.update-gradle-dependencies.create-pr - name: Checkout repository - uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # 5.0.0 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # 6.0.2 with: submodules: "recursive" + + - uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5.2.0 + with: + distribution: 'temurin' + java-version: '21' + cache: 'gradle' + + - name: Configure git + run: | + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + + - name: Define branch names + id: define-branches + run: | + DATE=$(date +'%Y%m%d') + echo "core_branch=ci/update-gradle-dependencies-${DATE}" >> $GITHUB_OUTPUT + echo "instrumentation_branch=ci/update-gradle-dependencies-instrumentation-${DATE}" >> $GITHUB_OUTPUT + - name: Update Gradle dependencies env: ORG_GRADLE_PROJECT_akkaRepositoryToken: ${{ secrets.AKKA_REPO_TOKEN }} run: | find . -name 'gradle.lockfile' -delete - GRADLE_OPTS="-Dorg.gradle.jvmargs='-Xmx3G -Xms2G'" \ - JAVA_HOME=$JAVA_HOME_8_X64 \ - JAVA_8_HOME=$JAVA_HOME_8_X64 \ - JAVA_11_HOME=$JAVA_HOME_11_X64 \ - JAVA_17_HOME=$JAVA_HOME_17_X64 \ - JAVA_21_HOME=$JAVA_HOME_21_X64 \ - JAVA_25_HOME=$JAVA_HOME_25_X64 \ + GRADLE_OPTS="-Dorg.gradle.jvmargs='-Xms2G -Xmx3G'" \ ./gradlew resolveAndLockAll --write-locks --parallel --stacktrace --no-daemon --max-workers=4 - - name: Check if changes should be committed - id: check-changes + + - name: Save instrumentation lock files + run: | + mkdir -p /tmp/instrumentation-lockfiles + find dd-smoke-tests dd-java-agent/instrumentation -name 'gradle.lockfile' -exec cp --parents {} /tmp/instrumentation-lockfiles/ \; + # Restore instrumentation dirs to original state (keep only core changes) + git restore -- 'dd-smoke-tests/' 'dd-java-agent/instrumentation/' + + # ==================== Core modules PR ==================== + - name: Check if core changes exist + id: check-core-changes run: | if [[ -z "$(git status -s)" ]]; then - echo "No changes to commit, exiting." + echo "No core changes to commit." echo "commit_changes=false" >> "$GITHUB_OUTPUT" - exit 0 else echo "commit_changes=true" >> "$GITHUB_OUTPUT" fi - - name: Pick a branch name - if: steps.check-changes.outputs.commit_changes == 'true' - id: define-branch - run: echo "branch=ci/update-gradle-dependencies-$(date +'%Y%m%d')" >> $GITHUB_OUTPUT - - name: Commit changes - if: steps.check-changes.outputs.commit_changes == 'true' - id: create-commit + + - name: Create core commit + if: steps.check-core-changes.outputs.commit_changes == 'true' + id: create-core-commit run: | - git config user.name "github-actions[bot]" - git config user.email "41898282+github-actions[bot]@users.noreply.github.com" git add --all git commit -m "chore: Update Gradle dependencies" echo "commit=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT - - name: Push changes - uses: DataDog/commit-headless@5a0f3876e0fbdd3a86b3e008acf4ec562db59eee # action/v2.0.1 - if: steps.check-changes.outputs.commit_changes == 'true' + + - name: Push core changes + if: steps.check-core-changes.outputs.commit_changes == 'true' + uses: DataDog/commit-headless@ad3668640012ec69186398f43d61923f6878bbbe # action/v3.2.0 with: token: "${{ steps.octo-sts.outputs.token }}" - branch: "${{ steps.define-branch.outputs.branch }}" - # for scheduled runs, sha is the tip of the default branch - # for dispatched runs, sha is the tip of the branch it was dispatched on + branch: "${{ steps.define-branches.outputs.core_branch }}" head-sha: "${{ github.sha }}" create-branch: true command: push - commits: "${{ steps.create-commit.outputs.commit }}" - - name: Create pull request - if: steps.check-changes.outputs.commit_changes == 'true' + commits: "${{ steps.create-core-commit.outputs.commit }}" + + - name: Create core pull request + if: steps.check-core-changes.outputs.commit_changes == 'true' env: GH_TOKEN: ${{ steps.octo-sts.outputs.token }} run: | - # use echo to set a multiline body for the PR - echo -e "This PR updates the Gradle dependencies. ⚠️ Don't forget to squash commits before merging. ⚠️\n\n- [ ] Update PR title if a code change is needed to support one of those new dependencies" | \ - gh pr create --title "Update Gradle dependencies" \ + gh pr create --title "Update Gradle dependencies" \ --base master \ - --head ${{ steps.define-branch.outputs.branch }} \ + --head ${{ steps.define-branches.outputs.core_branch }} \ --label "tag: dependencies" \ --label "tag: no release notes" \ - --body-file - + --body "$(cat <<'EOF' + # What Does This Do + + This PR updates the Gradle dependency locks for common and product modules. + + # Motivation + + Refresh Gradle dependencies to make sure to apply the latest version available when bumping dependencies. + + # Contributor Checklist + + - [ ] Update PR title if a code change is needed to support one of those new dependencies + EOF + )" + + # ==================== Instrumentation PR ==================== + - name: Reset and apply instrumentation changes + run: | + git reset --hard ${{ github.sha }} + cp -r /tmp/instrumentation-lockfiles/* . + + - name: Check if instrumentation changes exist + id: check-instrumentation-changes + run: | + if [[ -z "$(git status -s)" ]]; then + echo "No instrumentation changes to commit." + echo "commit_changes=false" >> "$GITHUB_OUTPUT" + else + echo "commit_changes=true" >> "$GITHUB_OUTPUT" + fi + + - name: Create instrumentation commit + if: steps.check-instrumentation-changes.outputs.commit_changes == 'true' + id: create-instrumentation-commit + run: | + git add --all + git commit -m "chore: Update instrumentation Gradle dependencies" + echo "commit=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT + + - name: Push instrumentation changes + if: steps.check-instrumentation-changes.outputs.commit_changes == 'true' + uses: DataDog/commit-headless@ad3668640012ec69186398f43d61923f6878bbbe # action/v3.2.0 + with: + token: "${{ steps.octo-sts.outputs.token }}" + branch: "${{ steps.define-branches.outputs.instrumentation_branch }}" + head-sha: "${{ github.sha }}" + create-branch: true + command: push + commits: "${{ steps.create-instrumentation-commit.outputs.commit }}" + + - name: Create instrumentation pull request + if: steps.check-instrumentation-changes.outputs.commit_changes == 'true' + env: + GH_TOKEN: ${{ steps.octo-sts.outputs.token }} + run: | + gh pr create --title "Update instrumentation Gradle dependencies" \ + --base master \ + --head ${{ steps.define-branches.outputs.instrumentation_branch }} \ + --label "tag: dependencies" \ + --label "tag: no release notes" \ + --body "$(cat <<'EOF' + # What Does This Do + + This PR updates the Gradle dependency locks for instrumentations and their tests. + + # Motivation + + Refresh Gradle dependencies to make sure to test latest versions of dependencies within their supported versions. + + # Contributor Checklist + + - [ ] Update PR title if a code change is needed to support one of those new dependencies + EOF + )" diff --git a/.github/workflows/update-issues-on-release.yaml b/.github/workflows/update-issues-on-release.yaml index 348a6a3a0ad..8ed8a9de1b1 100644 --- a/.github/workflows/update-issues-on-release.yaml +++ b/.github/workflows/update-issues-on-release.yaml @@ -15,7 +15,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Get milestone for release - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # 8.0.0 + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # 9.0.0 with: github-token: ${{secrets.GITHUB_TOKEN}} script: | diff --git a/.github/workflows/update-jmxfetch-submodule.yaml b/.github/workflows/update-jmxfetch-submodule.yaml index 88e1eb35c6a..272bd649562 100644 --- a/.github/workflows/update-jmxfetch-submodule.yaml +++ b/.github/workflows/update-jmxfetch-submodule.yaml @@ -12,14 +12,14 @@ jobs: contents: read id-token: write # Required for OIDC token federation steps: - - uses: DataDog/dd-octo-sts-action@acaa02eee7e3bb0839e4272dacb37b8f3b58ba80 # v1.0.3 + - uses: DataDog/dd-octo-sts-action@96a25462dbcb10ebf0bfd6e2ccc917d2ab235b9a # v1.0.4 id: octo-sts with: scope: DataDog/dd-trace-java policy: self.update-jmxfetch-submodule.create-pr - name: Checkout repository - uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # 5.0.0 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # 6.0.2 - name: Update Submodule run: | @@ -47,7 +47,7 @@ jobs: git commit -m "feat(ci): Update agent-jmxfetch submodule" dd-java-agent/agent-jmxfetch/integrations-core echo "commit=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT - name: Push changes - uses: DataDog/commit-headless@5a0f3876e0fbdd3a86b3e008acf4ec562db59eee # action/v2.0.1 + uses: DataDog/commit-headless@ad3668640012ec69186398f43d61923f6878bbbe # action/v3.2.0 if: steps.check-changes.outputs.commit_changes == 'true' with: token: "${{ steps.octo-sts.outputs.token }}" diff --git a/.github/workflows/update-smoke-test-latest-versions.yaml b/.github/workflows/update-smoke-test-latest-versions.yaml new file mode 100644 index 00000000000..2fd41117634 --- /dev/null +++ b/.github/workflows/update-smoke-test-latest-versions.yaml @@ -0,0 +1,168 @@ +name: Update smoke test latest versions +on: + schedule: + - cron: "0 5 * * 0" + workflow_dispatch: + +jobs: + update-smoke-test-latest-versions: + runs-on: ubuntu-latest + name: Update smoke test latest versions + permissions: + contents: read + id-token: write # Required for OIDC token federation + steps: + - uses: DataDog/dd-octo-sts-action@96a25462dbcb10ebf0bfd6e2ccc917d2ab235b9a # v1.0.4 + id: octo-sts + with: + scope: DataDog/dd-trace-java + policy: self.update-smoke-test-latest-versions.create-pr + + - name: Checkout repository + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # 6.0.2 + + - name: Define branch name + id: define-branch + run: | + DATE=$(date +'%Y%m%d') + echo "branch=ci/update-smoke-test-latest-versions-${DATE}" >> "$GITHUB_OUTPUT" + + - name: Fetch latest Gradle version + id: gradle + run: | + VERSION=$(curl -sf https://services.gradle.org/versions/current | jq -r '.version') + if [ -z "$VERSION" ] || [ "$VERSION" = "null" ]; then + echo "::error::Failed to fetch latest Gradle version" + exit 1 + fi + echo "version=$VERSION" >> "$GITHUB_OUTPUT" + echo "Latest Gradle version: $VERSION" + + - name: Fetch latest stable Maven version + id: maven + run: | + METADATA=$(curl -sf https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/maven-metadata.xml) + # Get all versions, filter out alpha/beta/rc, take the latest + VERSION=$(echo "$METADATA" \ + | grep -o '[^<]*' \ + | sed 's/<[^>]*>//g' \ + | grep -v -E '(alpha|beta|rc)' \ + | sort -V \ + | tail -1) + if [ -z "$VERSION" ]; then + echo "::error::Failed to fetch latest stable Maven version" + exit 1 + fi + echo "version=$VERSION" >> "$GITHUB_OUTPUT" + echo "Latest stable Maven version: $VERSION" + + - name: Fetch latest stable Maven Surefire version + id: surefire + run: | + METADATA=$(curl -sf https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/maven-metadata.xml) + # Get all versions, filter out alpha/beta, take the latest + VERSION=$(echo "$METADATA" \ + | grep -o '[^<]*' \ + | sed 's/<[^>]*>//g' \ + | grep -v -E '(alpha|beta)' \ + | sort -V \ + | tail -1) + if [ -z "$VERSION" ]; then + echo "::error::Failed to fetch latest stable Maven Surefire version" + exit 1 + fi + echo "version=$VERSION" >> "$GITHUB_OUTPUT" + echo "Latest stable Maven Surefire version: $VERSION" + + - name: Update properties files + env: + GRADLE_VERSION: ${{ steps.gradle.outputs.version }} + MAVEN_VERSION: ${{ steps.maven.outputs.version }} + SUREFIRE_VERSION: ${{ steps.surefire.outputs.version }} + run: | + echo "Writing resolved versions to properties files:" + echo " Gradle: ${GRADLE_VERSION}" + echo " Maven: ${MAVEN_VERSION}" + echo " Maven Surefire: ${SUREFIRE_VERSION}" + + printf '%s\n' \ + "# Pinned \"latest\" versions for CI Visibility Gradle smoke tests." \ + "# Updated automatically by the update-smoke-test-latest-versions workflow." \ + "gradle.version=${GRADLE_VERSION}" \ + > dd-smoke-tests/gradle/src/test/resources/latest-tool-versions.properties + + printf '%s\n' \ + "# Pinned \"latest\" versions for CI Visibility Maven smoke tests." \ + "# Updated automatically by the update-smoke-test-latest-versions workflow." \ + "maven.version=${MAVEN_VERSION}" \ + "maven-surefire.version=${SUREFIRE_VERSION}" \ + > dd-smoke-tests/maven/src/test/resources/latest-tool-versions.properties + + - name: Check for changes + id: check-changes + run: | + if [[ -z "$(git status -s)" ]]; then + echo "No changes detected — pinned versions are already up to date." + echo "has_changes=false" >> "$GITHUB_OUTPUT" + else + echo "Changes detected in the following files:" + git status -s + echo "" + echo "Diff:" + git diff + echo "has_changes=true" >> "$GITHUB_OUTPUT" + fi + + - name: Configure git + if: steps.check-changes.outputs.has_changes == 'true' + run: | + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + + - name: Create commit + if: steps.check-changes.outputs.has_changes == 'true' + id: create-commit + run: | + git add dd-smoke-tests/gradle/src/test/resources/latest-tool-versions.properties \ + dd-smoke-tests/maven/src/test/resources/latest-tool-versions.properties + git commit -m "chore: Update smoke test latest tool versions" + echo "commit=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT" + + - name: Push changes + if: steps.check-changes.outputs.has_changes == 'true' + uses: DataDog/commit-headless@ad3668640012ec69186398f43d61923f6878bbbe # action/v3.2.0 + with: + token: "${{ steps.octo-sts.outputs.token }}" + branch: "${{ steps.define-branch.outputs.branch }}" + head-sha: "${{ github.sha }}" + create-branch: true + command: push + commits: "${{ steps.create-commit.outputs.commit }}" + + - name: Create pull request + if: steps.check-changes.outputs.has_changes == 'true' + env: + GH_TOKEN: ${{ steps.octo-sts.outputs.token }} + run: | + gh pr create --title "Update smoke test latest tool versions" \ + --base master \ + --head ${{ steps.define-branch.outputs.branch }} \ + --label "tag: dependencies" \ + --label "tag: no release notes" \ + --body "$(cat <<'EOF' + # What Does This Do + + This PR updates the pinned "latest" tool versions used by CI Visibility smoke tests: + - Gradle: ${{ steps.gradle.outputs.version }} + - Maven: ${{ steps.maven.outputs.version }} + - Maven Surefire: ${{ steps.surefire.outputs.version }} + + # Motivation + + Keep smoke tests running against the latest stable versions of build tools. + + # Contributor Checklist + + - [ ] Verify smoke tests pass with the new versions + EOF + )" diff --git a/.gitignore b/.gitignore index c0dcdb32d1b..efe0ddbf28b 100644 --- a/.gitignore +++ b/.gitignore @@ -46,8 +46,21 @@ out/ ###################### .vscode +# Cursor # +########## +.cursor + +# Claude Code local custom settings # +##################################### +.claude/*.local.* + +# Vim # +####### +*.sw[nop] + # Others # ########## +/dumps /logs/* /bin /out @@ -77,3 +90,6 @@ mise*.local.toml .config/mise*.toml # asdf .tool-versions + +# Exclude kotlin build files +.kotlin diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 4d25d4a80fd..ab93e6d2171 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -4,14 +4,34 @@ include: - local: ".gitlab/macrobenchmarks.yml" - local: ".gitlab/exploration-tests.yml" - local: ".gitlab/ci-visibility-tests.yml" + - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' + file: '.gitlab/ci-java-spring-petclinic-parallel.yml' + ref: 'main' + - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' + file: '.gitlab/ci-java-load-parallel.yml' + ref: 'main' + - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' + file: '.gitlab/ci-java-startup-parallel.yml' + ref: 'main' + - project: 'DataDog/apm-reliability/apm-sdks-benchmarks' + file: '.gitlab/ci-java-dacapo-parallel.yml' + ref: 'main' + - local: ".gitlab/java-benchmark-configs.yml" stages: - build - publish + # These benchmarks are intended to replace the legacy benchmarks in the future + - java-spring-petclinic-parallel + - java-spring-petclinic-parallel-slo + - java-startup-parallel + - java-load-parallel + - java-dacapo-parallel - shared-pipeline - benchmarks - macrobenchmarks - tests + - test-summary - exploration-tests - ci-visibility-tests - generate-signing-key @@ -25,12 +45,13 @@ variables: BUILD_JOB_NAME: "build" DEPENDENCY_CACHE_POLICY: pull BUILD_CACHE_POLICY: pull - GRADLE_VERSION: "8.14.3" # must match gradle-wrapper.properties + GRADLE_VERSION: "8.14.4" # must match gradle-wrapper.properties MAVEN_REPOSITORY_PROXY: "https://depot-read-api-java.us1.ddbuild.io/magicmirror/magicmirror/@current/" GRADLE_PLUGIN_PROXY: "https://depot-read-api-java.us1.ddbuild.io/magicmirror/magicmirror/@current/" - BUILDER_IMAGE_VERSION_PREFIX: "v25.10-" # use either an empty string (e.g. "") for latest images or a version followed by a hyphen (e.g. "v25.05-") + BUILDER_IMAGE_REPO: "registry.ddbuild.io/images/mirror/dd-trace-java-docker-build" # images are pinned in images/mirror.lock.yaml in the DataDog/images repo + BUILDER_IMAGE_VERSION_PREFIX: "ci-" # use either an empty string (e.g. "") for latest images or a version followed by a hyphen (e.g. "ci-" or "123_merge-") REPO_NOTIFICATION_CHANNEL: "#apm-java-escalations" - DEFAULT_TEST_JVMS: /^(8|11|17|21|25)$/ # the latest "stable" version is LTS v25 + DEFAULT_TEST_JVMS: /^(8|11|17|21|25|tip)$/ # the latest "tip" version is 26 PROFILE_TESTS: description: "Enable profiling of tests" value: "false" @@ -41,6 +62,10 @@ variables: description: "Enable flaky tests" value: "false" + # One pipeline injection package size ratchet + OCI_PACKAGE_MAX_SIZE_BYTES: 40_000_000 + LIB_INJECTION_IMAGE_MAX_SIZE_BYTES: 40_000_000 + # trigger new commit cancel workflow: auto_cancel: @@ -68,7 +93,7 @@ workflow: - "ibm8" - "zulu11" - "semeru17" - # - "stable" + - "tip" CI_SPLIT: ["1/1"] # Gitlab doesn't support "parallel" and "parallel:matrix" at the same time @@ -108,10 +133,7 @@ default: .normalize_node_index: &normalize_node_index - if [ "$CI_NO_SPLIT" == "true" ] ; then CI_NODE_INDEX=1; CI_NODE_TOTAL=1; fi # A job uses parallel but doesn't intend to split by index - if [ -n "$CI_SPLIT" ]; then CI_NODE_INDEX="${CI_SPLIT%%/*}"; CI_NODE_TOTAL="${CI_SPLIT##*/}"; fi - - echo "CI_NODE_TOTAL=${CI_NODE_TOTAL}, CI_NODE_INDEX=$CI_NODE_INDEX" - - export NORMALIZED_NODE_TOTAL=${CI_NODE_TOTAL:-1} - - ONE_INDEXED_NODE_INDEX=${CI_NODE_INDEX:-1}; export NORMALIZED_NODE_INDEX=$((ONE_INDEXED_NODE_INDEX - 1)) - - echo "NORMALIZED_NODE_TOTAL=${NORMALIZED_NODE_TOTAL}, NORMALIZED_NODE_INDEX=$NORMALIZED_NODE_INDEX" + - echo "CI_NODE_INDEX=$CI_NODE_INDEX, CI_NODE_TOTAL=${CI_NODE_TOTAL}" .cgroup_info: &cgroup_info - source .gitlab/gitlab-utils.sh @@ -119,6 +141,21 @@ default: - .gitlab/cgroup-info.sh - gitlab_section_end "cgroup-info" +.container_info: &container_info + - | + # Containers and processes are limited to a 1-hour window, and expire after 36 hours + if [ -n "$CI_JOB_STARTED_AT" ]; then + FROM_EPOCH_MS=$(date -d "$CI_JOB_STARTED_AT" +%s)000 + TO_EPOCH_MS=$((FROM_EPOCH_MS + 3600000)) + TIME_PARAMS="from_ts=${FROM_EPOCH_MS}&to_ts=${TO_EPOCH_MS}&" + else + TIME_PARAMS="" + fi + + echo -e "${TEXT_BOLD}${TEXT_YELLOW}Runner dashboard, these are live (limited to a 1-hour window, and expire after 36 hours)${TEXT_CLEAR}" + echo -e "${TEXT_BOLD}${TEXT_YELLOW} Containers:${TEXT_CLEAR} https://app.datadoghq.com/containers?${TIME_PARAMS}query=image_name%3A%2A%2Fdatadog%2Fdd-trace-java-docker-build%20AND%20pod_name%3A${POD_NAME}&live=false" + echo -e "${TEXT_BOLD}${TEXT_YELLOW} Processes:${TEXT_CLEAR} https://app.datadoghq.com/process?${TIME_PARAMS}query=image_name%3A%2A%2Fdatadog%2Fdd-trace-java-docker-build%20AND%20pod_name%3A${POD_NAME}&live=false" + .gitlab_base_ref_params: &gitlab_base_ref_params - | export GIT_BASE_REF=$(.gitlab/find-gh-base-ref.sh) @@ -129,15 +166,16 @@ default: fi .gradle_build: &gradle_build - image: ghcr.io/datadog/dd-trace-java-docker-build:${BUILDER_IMAGE_VERSION_PREFIX}base + image: ${BUILDER_IMAGE_REPO}:${BUILDER_IMAGE_VERSION_PREFIX}base stage: build variables: - MAVEN_OPTS: "-Xms64M -Xmx512M" - GRADLE_WORKERS: 2 - GRADLE_MEM: 2560M - KUBERNETES_CPU_REQUEST: 8 - KUBERNETES_MEMORY_REQUEST: 8Gi - KUBERNETES_MEMORY_LIMIT: 8Gi + MAVEN_OPTS: "-Xms256M -Xmx1024M" + GRADLE_WORKERS: 6 + GRADLE_MEMORY_MIN: 1G + GRADLE_MEMORY_MAX: 4G + KUBERNETES_CPU_REQUEST: 10 + KUBERNETES_MEMORY_REQUEST: 20Gi + KUBERNETES_MEMORY_LIMIT: 20Gi CACHE_TYPE: "lib" #default FF_USE_FASTZIP: "true" CACHE_COMPRESSION_LEVEL: "slowest" @@ -167,20 +205,28 @@ default: unprotect: true before_script: - source .gitlab/gitlab-utils.sh + - *container_info # Akka token added to SSM from https://account.akka.io/token - export ORG_GRADLE_PROJECT_akkaRepositoryToken=$(aws ssm get-parameter --region us-east-1 --name ci.dd-trace-java.akka_repo_token --with-decryption --query "Parameter.Value" --out text) - export ORG_GRADLE_PROJECT_mavenRepositoryProxy=$MAVEN_REPOSITORY_PROXY - export ORG_GRADLE_PROJECT_gradlePluginProxy=$GRADLE_PLUGIN_PROXY + - | + JAVA_HOMES=$(env | grep -E '^JAVA_[A-Z0-9_]+_HOME=' | sed 's/=.*//' | paste -sd,) + cat >> gradle.properties < /dev/null; then + echo "Failed to authenticate tokens against maven central staging API. Check credentials and see https://datadoghq.atlassian.net/wiki/x/Oog5OgE" exit 1 fi @@ -273,10 +326,6 @@ build_tests: variables: BUILD_CACHE_POLICY: push DEPENDENCY_CACHE_POLICY: pull - GRADLE_MEM: 4G - GRADLE_WORKERS: 3 - KUBERNETES_MEMORY_REQUEST: 18Gi - KUBERNETES_MEMORY_LIMIT: 18Gi parallel: matrix: - GRADLE_TARGET: ":baseTest" @@ -289,8 +338,7 @@ build_tests: CACHE_TYPE: "latestdep" - GRADLE_TARGET: ":smokeTest" CACHE_TYPE: "smoke" - MAVEN_OPTS: "-Xms64M -Xmx512M" - + MAVEN_OPTS: "-Xms256M -Xmx1024M" script: - *gitlab_base_ref_params - ./gradlew --version @@ -357,13 +405,31 @@ spotless: extends: .gradle_build stage: tests needs: [] + variables: + GRADLE_MEMORY_MAX: 6G script: - ./gradlew --version - ./gradlew spotlessCheck $GRADLE_ARGS +check-instrumentation-naming: + extends: .gradle_build + stage: tests + needs: [ ] + script: + - ./gradlew --version + - ./gradlew checkInstrumentationNaming + +config-inversion-linter: + extends: .gradle_build + stage: tests + needs: [] + script: + - ./gradlew --version + - ./gradlew checkConfigurations + test_published_artifacts: extends: .gradle_build - image: ghcr.io/datadog/dd-trace-java-docker-build:${BUILDER_IMAGE_VERSION_PREFIX}7 # Needs Java7 for some tests + image: ${BUILDER_IMAGE_REPO}:${BUILDER_IMAGE_VERSION_PREFIX}7 # Needs Java7 for some tests stage: tests needs: [ build ] variables: @@ -399,18 +465,23 @@ test_published_artifacts: script: - *gitlab_base_ref_params - ./gradlew --version - - ./gradlew $GRADLE_TARGET $GRADLE_PARAMS -PskipTests -PrunBuildSrcTests -PskipSpotless -PtaskPartitionCount=$NORMALIZED_NODE_TOTAL -PtaskPartition=$NORMALIZED_NODE_INDEX $GRADLE_ARGS + - ./gradlew $GRADLE_TARGET -x spotlessCheck $GRADLE_PARAMS -PskipTests -PrunBuildSrcTests -Pslot=$CI_NODE_INDEX/$CI_NODE_TOTAL $GRADLE_ARGS after_script: + - *container_info - *cgroup_info - source .gitlab/gitlab-utils.sh - gitlab_section_start "collect-reports" "Collecting reports" - .gitlab/collect_reports.sh --destination ./check_reports --move + - .gitlab/collect_results.sh - gitlab_section_end "collect-reports" artifacts: when: always paths: - ./check_reports + - ./results - '.gradle/daemon/*/*.out.log' + reports: + junit: results/*.xml retry: max: 2 when: @@ -421,6 +492,12 @@ test_published_artifacts: - scheduler_failure - data_integrity_failure +check_build_src: + extends: .check_job + needs: [] + variables: + GRADLE_TARGET: ":buildSrc:build" + check_base: extends: .check_job variables: @@ -462,18 +539,25 @@ muzzle: script: - export SKIP_BUILDSCAN="true" - ./gradlew --version - - ./gradlew :runMuzzle -PtaskPartitionCount=$NORMALIZED_NODE_TOTAL -PtaskPartition=$NORMALIZED_NODE_INDEX $GRADLE_ARGS + - ./gradlew :runMuzzle -Pslot=$CI_NODE_INDEX/$CI_NODE_TOTAL $GRADLE_ARGS after_script: + - *container_info - *cgroup_info + - *set_datadog_api_keys - source .gitlab/gitlab-utils.sh - gitlab_section_start "collect-reports" "Collecting reports" - .gitlab/collect_reports.sh + - .gitlab/collect_results.sh + - .gitlab/upload_ciapp.sh $CACHE_TYPE - gitlab_section_end "collect-reports" artifacts: when: always paths: - ./reports + - ./results - '.gradle/daemon/*/*.out.log' + reports: + junit: results/*.xml muzzle-dep-report: extends: .gradle_build @@ -486,6 +570,7 @@ muzzle-dep-report: - ./gradlew --version - ./gradlew generateMuzzleReport muzzleInstrumentationReport $GRADLE_ARGS after_script: + - *container_info - *cgroup_info - .gitlab/collect_muzzle_deps.sh artifacts: @@ -511,16 +596,11 @@ muzzle-dep-report: .test_job: extends: .gradle_build - image: ghcr.io/datadog/dd-trace-java-docker-build:${BUILDER_IMAGE_VERSION_PREFIX}$testJvm + image: ${BUILDER_IMAGE_REPO}:${BUILDER_IMAGE_VERSION_PREFIX}$testJvm tags: [ "docker-in-docker:amd64" ] # use docker-in-docker runner for testcontainers needs: [ build_tests ] stage: tests variables: - KUBERNETES_MEMORY_REQUEST: 17Gi - KUBERNETES_MEMORY_LIMIT: 17Gi - KUBERNETES_CPU_REQUEST: 10 - GRADLE_WORKERS: 4 - GRADLE_MEM: 3G GRADLE_PARAMS: "-PskipFlakyTests" CONTINUE_ON_FAILURE: "false" TESTCONTAINERS_CHECKS_DISABLE: "true" @@ -534,6 +614,10 @@ muzzle-dep-report: when: on_success - if: $CI_COMMIT_BRANCH == "master" when: on_success + - if: '$CI_COMMIT_BRANCH =~ /^mq-working-branch-/' + when: on_success + - if: '$CI_COMMIT_BRANCH =~ /^gh-readonly-queue/' + when: on_success script: - *gitlab_base_ref_params - > @@ -542,12 +626,13 @@ muzzle-dep-report: export PROFILER_COMMAND="-XX:StartFlightRecording=settings=profile,filename=/tmp/${CI_JOB_NAME_SLUG}.jfr,dumponexit=true"; fi - *prepare_test_env - - export GRADLE_OPTS="-Dorg.gradle.jvmargs='-Xms$GRADLE_MEM -Xmx$GRADLE_MEM $PROFILER_COMMAND -XX:ErrorFile=/tmp/hs_err_pid%p.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp' -Ddatadog.forkedMaxHeapSize=1024M -Ddatadog.forkedMinHeapSize=128M" + - export GRADLE_OPTS="-Dorg.gradle.jvmargs='-Xms$GRADLE_MEMORY_MIN -Xmx$GRADLE_MEMORY_MAX $PROFILER_COMMAND -XX:ErrorFile=/tmp/hs_err_pid%p.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djava.util.prefs.userRoot=/tmp/.java/.userPrefs-${CI_JOB_ID}' -Ddatadog.forkedMinHeapSize=128M -Ddatadog.forkedMaxHeapSize=1024M" - ./gradlew --version - - ./gradlew $GRADLE_TARGET $GRADLE_PARAMS -PtestJvm=$testJvm -PtaskPartitionCount=$NORMALIZED_NODE_TOTAL -PtaskPartition=$NORMALIZED_NODE_INDEX $GRADLE_ARGS --continue || $CONTINUE_ON_FAILURE + - ./gradlew $GRADLE_TARGET $GRADLE_PARAMS -PtestJvm=$testJvm -Pslot=$CI_NODE_INDEX/$CI_NODE_TOTAL $GRADLE_ARGS --continue || $CONTINUE_ON_FAILURE after_script: - *restore_pretest_env - *set_datadog_api_keys + - *container_info - *cgroup_info - source .gitlab/gitlab-utils.sh - gitlab_section_start "collect-reports" "Collecting reports" @@ -556,6 +641,7 @@ muzzle-dep-report: - .gitlab/collect_results.sh - .gitlab/upload_ciapp.sh $CACHE_TYPE $testJvm - gitlab_section_end "collect-reports" + - .gitlab/count_tests.sh "$GRADLE_TARGET" "$testJvm" "./results" "./test_counts_${CI_JOB_ID}.json" - URL_ENCODED_JOB_NAME=$(jq -rn --arg x "$CI_JOB_NAME" '$x|@uri') - echo -e "${TEXT_BOLD}${TEXT_YELLOW}See test results in Datadog:${TEXT_CLEAR} https://app.datadoghq.com/ci/test/runs?query=test_level%3Atest%20%40test.service%3Add-trace-java%20%40ci.pipeline.id%3A${CI_PIPELINE_ID}%20%40ci.job.name%3A%22${URL_ENCODED_JOB_NAME}%22" artifacts: @@ -564,6 +650,7 @@ muzzle-dep-report: - ./reports.tar - ./profiles.tar - ./results + - './test_counts_*.json' - '.gradle/daemon/*/*.out.log' reports: junit: results/*.xml @@ -583,7 +670,7 @@ muzzle-dep-report: CI_USE_TEST_AGENT: "true" CI_AGENT_HOST: local-agent services: - - name: ghcr.io/datadog/dd-apm-test-agent/ddapm-test-agent:v1.33.1 + - name: registry.ddbuild.io/images/mirror/dd-apm-test-agent/ddapm-test-agent:v1.44.0 alias: local-agent variables: LOG_LEVEL: "DEBUG" @@ -630,7 +717,7 @@ test_inst: GRADLE_TARGET: ":instrumentationTest" CACHE_TYPE: "inst" parallel: - matrix: *test_matrix_6 + matrix: *test_matrix_8 test_inst_latest: extends: .test_job_with_test_agent @@ -639,7 +726,7 @@ test_inst_latest: CACHE_TYPE: "latestdep" parallel: matrix: - - testJvm: ["8", "17", "21", "25"] # the latest "stable" version is LTS v25 + - testJvm: ["8", "17", "21", "25"] # the latest "tip" version is LTS v25 # Gitlab doesn't support "parallel" and "parallel:matrix" at the same time # This emulates "parallel" by including it in the matrix CI_SPLIT: [ "1/6", "2/6", "3/6", "4/6", "5/6", "6/6"] @@ -691,7 +778,7 @@ test_debugger: variables: GRADLE_TARGET: ":debuggerTest" CACHE_TYPE: "base" - DEFAULT_TEST_JVMS: /^(8|11|17|21|25|semeru8)$/ # the latest "stable" version is LTS v25 + DEFAULT_TEST_JVMS: /^(8|11|17|21|25|semeru8)$/ # the latest "tip" version is LTS v25 parallel: matrix: *test_matrix @@ -702,7 +789,7 @@ test_smoke: GRADLE_PARAMS: "-PskipFlakyTests" CACHE_TYPE: "smoke" parallel: - matrix: *test_matrix_4 + matrix: *test_matrix_8 test_ssi_smoke: extends: .test_job @@ -713,7 +800,7 @@ test_ssi_smoke: DD_INJECT_FORCE: "true" DD_INJECTION_ENABLED: "tracer" parallel: - matrix: *test_matrix_4 + matrix: *test_matrix_8 test_smoke_graalvm: extends: .test_job @@ -736,32 +823,32 @@ test_smoke_semeru8_debugger: NON_DEFAULT_JVMS: "true" testJvm: "semeru8" -deploy_to_profiling_backend: - stage: publish - needs: [ build ] +aggregate_test_counts: + image: ${BUILDER_IMAGE_REPO}:${BUILDER_IMAGE_VERSION_PREFIX}base + stage: test-summary + # Note: No explicit 'needs' or 'dependencies' required + # By default, GitLab CI automatically downloads artifacts from ALL jobs in previous stages + # This job collects test_counts_*.json files from all test/check jobs via stage ordering rules: - if: '$POPULATE_CACHE' when: never - - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH' + - if: '$CI_COMMIT_BRANCH == "master"' when: on_success - - if: '$CI_COMMIT_TAG =~ /^v.*/' + - if: '$CI_COMMIT_BRANCH =~ /^mq-working-branch-/' when: on_success - - when: manual - allow_failure: true - trigger: - project: DataDog/profiling-backend - branch: dogfooding - variables: - UPSTREAM_PACKAGE_JOB: $BUILD_JOB_NAME - UPSTREAM_PACKAGE_JOB_ID: $BUILD_JOB_ID - UPSTREAM_PROJECT_ID: $CI_PROJECT_ID - UPSTREAM_PROJECT_NAME: $CI_PROJECT_NAME - UPSTREAM_PIPELINE_ID: $CI_PIPELINE_ID - UPSTREAM_BRANCH: $CI_COMMIT_BRANCH - UPSTREAM_TAG: $CI_COMMIT_TAG + - if: '$CI_COMMIT_BRANCH =~ /^gh-readonly-queue/' + when: on_success + script: + - *set_datadog_api_keys + - .gitlab/aggregate_test_counts.sh + artifacts: + when: always + paths: + - test_counts_summary.json + - test_counts_report.md -trigger_tibco_tests: - stage: tests +deploy_to_profiling_backend: + stage: publish needs: [ build ] rules: - if: '$POPULATE_CACHE' @@ -773,9 +860,8 @@ trigger_tibco_tests: - when: manual allow_failure: true trigger: - project: DataDog/tibco-testing - branch: main - strategy: depend + project: DataDog/profiling-backend + branch: dogfooding variables: UPSTREAM_PACKAGE_JOB: $BUILD_JOB_NAME UPSTREAM_PACKAGE_JOB_ID: $BUILD_JOB_ID @@ -784,7 +870,6 @@ trigger_tibco_tests: UPSTREAM_PIPELINE_ID: $CI_PIPELINE_ID UPSTREAM_BRANCH: $CI_COMMIT_BRANCH UPSTREAM_TAG: $CI_COMMIT_TAG - FORCE_TRIGGER: $FORCE_TRIGGER deploy_to_di_backend:manual: stage: publish @@ -836,6 +921,31 @@ deploy_to_maven_central: - 'workspace/dd-trace-api/build/libs/*.jar' - 'workspace/dd-trace-ot/build/libs/*.jar' +deploy_snapshot_with_ddprof_snapshot: + extends: .gradle_build + stage: publish + needs: [ build ] + variables: + CACHE_TYPE: "lib" + rules: + - if: '$POPULATE_CACHE' + when: never + # Manual trigger only - for testing with ddprof snapshot versions + - when: manual + allow_failure: true + script: + - export MAVEN_CENTRAL_USERNAME=$(aws ssm get-parameter --region us-east-1 --name ci.dd-trace-java.central_username --with-decryption --query "Parameter.Value" --out text) + - export MAVEN_CENTRAL_PASSWORD=$(aws ssm get-parameter --region us-east-1 --name ci.dd-trace-java.central_password --with-decryption --query "Parameter.Value" --out text) + - export GPG_PRIVATE_KEY=$(aws ssm get-parameter --region us-east-1 --name ci.dd-trace-java.signing.gpg_private_key --with-decryption --query "Parameter.Value" --out text) + - export GPG_PASSWORD=$(aws ssm get-parameter --region us-east-1 --name ci.dd-trace-java.signing.gpg_passphrase --with-decryption --query "Parameter.Value" --out text) + - echo "Publishing dd-trace-java snapshot with ddprof snapshot dependency" + - ./gradlew -PbuildInfo.build.number=$CI_JOB_ID -PddprofUseSnapshot publishToSonatype -PskipTests $GRADLE_ARGS + artifacts: + paths: + - 'workspace/dd-java-agent/build/libs/*.jar' + - 'workspace/dd-trace-api/build/libs/*.jar' + - 'workspace/dd-trace-ot/build/libs/*.jar' + deploy_artifacts_to_github: stage: publish image: registry.ddbuild.io/images/dd-octo-sts-ci-base:2025.06-1 @@ -958,6 +1068,7 @@ publishing-gate: configure_system_tests: variables: + SYSTEM_TESTS_REF: "main" # system tests are pinned on release branches only SYSTEM_TESTS_SCENARIOS_GROUPS: "simple_onboarding,simple_onboarding_profiling,simple_onboarding_appsec,docker-ssi,lib-injection" create_key: @@ -975,13 +1086,14 @@ create_key: paths: - pubkeys -validate_supported_configurations_local_file: - extends: .validate_supported_configurations_local_file +validate_supported_configurations_v2_local_file: + extends: .validate_supported_configurations_v2_local_file variables: LOCAL_JSON_PATH: "metadata/supported-configurations.json" + BACKFILLED: "false" -update_central_configurations_version_range: - extends: .update_central_configurations_version_range +update_central_configurations_version_range_v2: + extends: .update_central_configurations_version_range_v2 variables: LOCAL_REPO_NAME: "dd-trace-java" LOCAL_JSON_PATH: "metadata/supported-configurations.json" diff --git a/.gitlab/TagInitializationErrors.java b/.gitlab/TagInitializationErrors.java new file mode 100644 index 00000000000..3fdf8357a74 --- /dev/null +++ b/.gitlab/TagInitializationErrors.java @@ -0,0 +1,114 @@ +import org.w3c.dom.Element; +import javax.xml.XMLConstants; +import javax.xml.parsers.DocumentBuilderFactory; +import javax.xml.transform.OutputKeys; +import javax.xml.transform.TransformerFactory; +import javax.xml.transform.dom.DOMSource; +import javax.xml.transform.stream.StreamResult; +import java.io.File; +import java.nio.file.Files; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +/// Tags intermediate `initializationError` retries with `dd_tags[test.final_status]=skip`. +/// +/// Gradle generates synthetic "initializationError" testcases in JUnit reports for setup methods. +/// When a setup is retried and eventually succeeds, multiple testcases are created, with only the +/// last one passing. All intermediate attempts are marked skip so Test Optimization is not misled. +/// +/// For any suite with multiple `initializationError` test cases (when retries occurred), all entries +/// but the last one are tagged by this script with `dd_tags[test.final_status]=skip`. The last +/// entry is left unmodified, allowing **Test Optimization** to apply its default status inference based +/// on the actual outcome. Files with only one (or zero) `initializationError` test cases are left unmodified. +/// +/// Before: +/// +/// ``` +/// +/// ``` +/// +/// After: +/// +/// ``` +/// +/// +/// +/// +/// +/// ``` +/// +/// Usage (Java 25): `java TagInitializationErrors.java junit-report.xml` + +class TagInitializationErrors { + public static void main(String[] args) throws Exception { + if (args.length == 0) { + System.err.println("Usage: java TagInitializationErrors.java "); + System.exit(1); + } + var xmlFile = new File(args[0]); + if (!xmlFile.exists()) { + System.err.println("File not found: " + xmlFile); + System.exit(1); + } + var dbf = DocumentBuilderFactory.newInstance(); + dbf.setFeature(XMLConstants.FEATURE_SECURE_PROCESSING, true); + dbf.setFeature("http://apache.org/xml/features/disallow-doctype-decl", true); + dbf.setFeature("http://xml.org/sax/features/external-general-entities", false); + dbf.setFeature("http://xml.org/sax/features/external-parameter-entities", false); + dbf.setExpandEntityReferences(false); + var doc = dbf.newDocumentBuilder().parse(xmlFile); + var testcases = doc.getElementsByTagName("testcase"); + Map> byClassname = new LinkedHashMap<>(); + for (int i = 0; i < testcases.getLength(); i++) { + var e = (Element) testcases.item(i); + if ("initializationError".equals(e.getAttribute("name"))) { + byClassname.computeIfAbsent(e.getAttribute("classname"), k -> new ArrayList<>()).add(e); + } + } + boolean modified = false; + for (var group : byClassname.values()) { + if (group.size() <= 1) continue; + for (int i = 0; i < group.size() - 1; i++) { + var testcase = group.get(i); + var existingProperties = testcase.getElementsByTagName("properties"); + if (existingProperties.getLength() > 0) { + var props = (Element) existingProperties.item(0); + var existingProps = props.getElementsByTagName("property"); + boolean alreadyTagged = false; + for (int j = 0; j < existingProps.getLength(); j++) { + if ("dd_tags[test.final_status]".equals(((Element) existingProps.item(j)).getAttribute("name"))) { + alreadyTagged = true; + break; + } + } + if (alreadyTagged) continue; + var property = doc.createElement("property"); + property.setAttribute("name", "dd_tags[test.final_status]"); + property.setAttribute("value", "skip"); + props.appendChild(property); + } else { + var properties = doc.createElement("properties"); + var property = doc.createElement("property"); + property.setAttribute("name", "dd_tags[test.final_status]"); + property.setAttribute("value", "skip"); + properties.appendChild(property); + testcase.appendChild(properties); + } + modified = true; + } + } + if (!modified) return; + var tmpFile = File.createTempFile("TagInitializationErrors", ".xml", xmlFile.getParentFile()); + try { + var transformer = TransformerFactory.newInstance().newTransformer(); + transformer.setOutputProperty(OutputKeys.ENCODING, "UTF-8"); + transformer.transform(new DOMSource(doc), new StreamResult(tmpFile)); + Files.move(tmpFile.toPath(), xmlFile.toPath(), java.nio.file.StandardCopyOption.REPLACE_EXISTING); + } catch (Exception e) { + tmpFile.delete(); + throw e; + } + } +} diff --git a/.gitlab/add_final_status.xsl b/.gitlab/add_final_status.xsl new file mode 100644 index 00000000000..4b4c0da17fd --- /dev/null +++ b/.gitlab/add_final_status.xsl @@ -0,0 +1,58 @@ + + + + + + + + + + + + + + + + + + + + fail + skip + pass + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/.gitlab/aggregate_test_counts.sh b/.gitlab/aggregate_test_counts.sh new file mode 100755 index 00000000000..57f271313bb --- /dev/null +++ b/.gitlab/aggregate_test_counts.sh @@ -0,0 +1,506 @@ +#!/usr/bin/env bash +# spotless:off - This file uses heredocs with <<- which require tabs, not spaces +# +# aggregate_test_counts.sh - Aggregate test counts from GitLab CI test jobs +# +# This script collects and aggregates test execution data from multiple test jobs +# in a GitLab CI pipeline. It processes test_counts_*.json files generated by +# individual test jobs and produces comprehensive summaries and reports. +# +# Workflow: +# 1. Finds all test_counts_*.json files from test jobs +# 2. Validates JSON structure and filters out invalid files +# 3. Aggregates data using jq (grouping by JVM version, job kind, etc.) +# 4. Generates summaries with breakdowns by JVM version and job kind +# 5. Creates aggregated JSON summary file (test_counts_summary.json) +# 6. Displays formatted output with GitLab CI collapsible sections +# 7. Detects and alerts on jobs with zero tests +# +# Usage: +# aggregate_test_counts.sh [-v] [aggregate_dir] +# +# Options: +# -v Enable verbose output for debugging +# aggregate_dir Directory containing test_counts_*.json files +# (default: ./test_counts_aggregate) +# +# Outputs: +# test_counts_summary.json - Aggregated JSON data with full breakdowns +# Console output - Formatted summary with links to Datadog CI Visibility +# +# Environment Variables (from GitLab CI): +# CI_PIPELINE_ID - GitLab pipeline ID +# CI_COMMIT_SHA - Git commit SHA +# CI_COMMIT_BRANCH - Git branch name +# CI_PROJECT_URL - GitLab project URL (for artifact links) +# +# See: docs/test-coverage.md for more details on the test counting system +# +# https://docs.gitlab.com/ci/variables/predefined_variables/ + +set -e + +# Source GitLab utilities for section formatting +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +if [ -f "$SCRIPT_DIR/gitlab-utils.sh" ]; then + source "$SCRIPT_DIR/gitlab-utils.sh" +fi + + +# ============================================================================ +# Configuration +# ============================================================================ + +VERBOSE=0 +if [ "$1" = "-v" ]; then + VERBOSE=1 + shift +fi + +AGGREGATE_DIR="${1:-./test_counts_aggregate}" +OUTPUT_FILE="test_counts_summary.json" + + +# ============================================================================ +# Utility Functions +# ============================================================================ + +# IMPORTANT: Heredocs in this function use <<- (with hyphen) to allow indentation in source code +# while producing unindented output. This requires TABS (not spaces) for leading whitespace. +# Using tabs is not a matter of preference or style; it's how the language is defined. + +log_verbose() { + if [ $VERBOSE -eq 1 ]; then + echo "[aggregate] $*" >&2 + fi +} + + +# ============================================================================ +# File Discovery and Validation +# ============================================================================ + +find_and_validate_test_files() { + local aggregate_dir="$1" + local output_file="$2" + + log_verbose "Searching for test count files (test_counts_.json)" + + # Find all test count files (exclude summary to avoid reprocessing previous runs) + local -a found_files + mapfile -t found_files < <(find "$aggregate_dir" -name "test_counts_*.json" -not -name "$output_file" -type f 2>/dev/null | sort) + + local job_count=${#found_files[@]} + log_verbose "Found $job_count test count files" + + if [ "$job_count" -eq 0 ]; then + echo "No test count files found" >&2 + return 1 + fi + + # Validate and filter out invalid JSON files + log_verbose "Validating JSON files" + local -a valid_files=() + local invalid_count=0 + local gitlab_base_url="${CI_PROJECT_URL}" + + for file in "${found_files[@]}"; do + # More thorough validation: try to parse the structure we expect + if jq -e 'type == "object" and has("ci_job_id") and has("total_tests")' "$file" >/dev/null 2>&1; then + valid_files+=("$file") + else + # Extract job ID from filename (e.g., test_counts_1234.json -> 1234) + local filename + filename=$(basename "$file") + local job_id="${filename#test_counts_}" + job_id="${job_id%.json}" + local artifact_url="${gitlab_base_url}/-/jobs/${job_id}/artifacts/file/${filename}" + + echo "⚠️ WARNING: Skipping invalid/empty JSON file: $file" >&2 + echo " Artifact URL: $artifact_url" >&2 + log_verbose "Invalid JSON file: $file ($artifact_url)" + ((invalid_count++)) + fi + done + + local valid_count=${#valid_files[@]} + log_verbose "Valid files: $valid_count, Invalid files: $invalid_count" + + if [ "$valid_count" -eq 0 ]; then + echo "ERROR: No valid test count files found (all $job_count files are invalid)" >&2 + return 1 + fi + + # Return valid files by printing them (caller will capture) + printf '%s\n' "${valid_files[@]}" + return 0 +} + + +# ============================================================================ +# Data Aggregation with jq +# ============================================================================ + +aggregate_test_data() { + local -a count_files=("$@") + + log_verbose "Processing all files with jq" + + # Capture jq output and errors + local jq_error_file + jq_error_file=$(mktemp) + trap 'rm -f "$jq_error_file"' RETURN + + # Aggregate data using jq (heredoc with <<- strips leading tabs) + # Using tabs is not a matter of preference or style; it's how the language is defined. + local jq_program + jq_program=$(cat <<-'JQ' + # Reusable function to generate JVM version sort keys + # Returns: [numeric_value, is_prefixed, version_string] + # This ensures: 8 < 11 < 17 < 21 < graalvm17 < semeru21 < stable + def jvm_sort_key: + [ + # First: extract numeric value (8, 11, 17, 21, 25, etc.) + (if . == "stable" then 1000 + elif ((. | gsub("[^0-9]"; "")) as $nums | $nums == "") then 0 + else (. | gsub("[^0-9]"; "") | tonumber) + end), + # Second: plain numbers before prefixed versions (8 before semeru8, 17 before graalvm17) + (if (. | test("^[0-9]+$")) then 0 else 1 end), + # Third: alphabetically by full version string + . + ]; + + # Sort by base job name (before colon), then jvm_version (numeric), then test_category + . | sort_by([ + (.ci_job_name | split(":")[0]), + (.jvm_version | jvm_sort_key), + .test_category + ]) | + { + # Pipeline metadata + pipeline_id: $pipeline_id, + commit_sha: $commit_sha, + branch: $branch, + timestamp: $timestamp, + test_jobs: ., + + # Overall summary across all jobs + summary: { + total_tests: (map(.total_tests) | add), + total_passed: (map(.passed_tests) | add), + total_failed: (map(.failed_tests) | add), + total_skipped: (map(.skipped_tests) | add) + }, + + # Aggregation by JVM version (e.g., 8, 11, 17, graalvm17, stable) + by_jvm: (group_by(.jvm_version) | map({ + jvm_version: .[0].jvm_version, + total_tests: (map(.total_tests) | add), + total_passed: (map(.passed_tests) | add), + total_failed: (map(.failed_tests) | add), + total_skipped: (map(.skipped_tests) | add), + job_count: length + }) | sort_by(.jvm_version | jvm_sort_key)), + + # Aggregation by job kind and JVM (ignoring test splits) + # This groups all splits of the same job+JVM together + by_job_kind_and_jvm: ( + # Extract base job name (before the matrix suffix like ": [8, 2/6]") + map(. + {job_kind: (.ci_job_name | split(":")[0])}) | + group_by([.job_kind, .jvm_version]) | + map({ + job_kind: .[0].job_kind, + jvm_version: .[0].jvm_version, + total_tests: (map(.total_tests) | add), + total_passed: (map(.passed_tests) | add), + total_failed: (map(.failed_tests) | add), + total_skipped: (map(.skipped_tests) | add), + split_count: length + }) | sort_by([.job_kind, (.jvm_version | jvm_sort_key)]) + ), + + # Pre-formatted table rows for markdown report (detailed breakdown) + table_rows: map( + [.test_category, .jvm_version, .ci_job_name, .total_tests, .passed_tests, .failed_tests, .skipped_tests] | + "| \(.[0]) | \(.[1]) | \(.[2]) | \(.[3]) | \(.[4]) | \(.[5]) | \(.[6]) |" + ), + + # Detection of jobs with zero tests (potential issues) + zero_test_jobs: map( + select(.total_tests == 0) | + { + category: .test_category, + jvm: .jvm_version, + job: .ci_job_name, + alert: "⚠️ **WARNING**: Zero tests in \(.test_category) on JVM \(.jvm_version) (job: \(.ci_job_name))" + } + ), + + # Detection of job failures without test failures (timeouts, infrastructure issues) + job_failures_no_tests: map( + select(.job_failed == true and .failed_tests == 0 and .error_tests == 0) | + { + category: .test_category, + jvm: .jvm_version, + job: .ci_job_name, + ci_job_id: .ci_job_id, + ci_job_status: .ci_job_status, + alert: "🚨 **JOB FAILURE**: Job failed without test failures in \(.test_category) on JVM \(.jvm_version) (likely timeout/infrastructure)" + } + ) + } + JQ + ) + + local aggregated_data + if ! aggregated_data=$(jq -s "$jq_program" \ + --arg pipeline_id "${CI_PIPELINE_ID:-unknown}" \ + --arg commit_sha "${CI_COMMIT_SHA:-unknown}" \ + --arg branch "${CI_COMMIT_BRANCH:-unknown}" \ + --arg timestamp "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + "${count_files[@]}" 2>"$jq_error_file"); then + + # Extract problematic file from jq error message + local error_msg + error_msg=$(cat "$jq_error_file") + echo "ERROR: jq processing failed: $error_msg" >&2 + + # Try to extract filename and job ID from error + if [[ "$error_msg" =~ test_counts_([0-9]+)\.json ]]; then + local problem_job_id="${BASH_REMATCH[1]}" + local problem_file="./test_counts_${problem_job_id}.json" + local gitlab_base_url="${CI_PROJECT_URL}" + + echo "" >&2 + echo "⚠️ Problematic file: $problem_file" >&2 + + if [ -n "$gitlab_base_url" ]; then + local artifact_url="${gitlab_base_url}/-/jobs/${problem_job_id}/artifacts/file/test_counts_${problem_job_id}.json" + echo " Artifact URL: $artifact_url" >&2 + fi + + # Show a snippet of the problematic file + if [ -f "$problem_file" ]; then + echo "" >&2 + echo "File contents around the error:" >&2 + head -20 "$problem_file" >&2 + fi + fi + + return 1 + fi + + # Return aggregated data + echo "$aggregated_data" + return 0 +} + + +# ============================================================================ +# Display Functions +# ============================================================================ + +display_summary() { + local aggregated_data="$1" + + # Extract summary values + local total_tests + local total_passed + local total_failed + local total_skipped + local zero_test_count + + total_tests=$(echo "$aggregated_data" | jq -r '.summary.total_tests') + total_passed=$(echo "$aggregated_data" | jq -r '.summary.total_passed') + total_failed=$(echo "$aggregated_data" | jq -r '.summary.total_failed') + total_skipped=$(echo "$aggregated_data" | jq -r '.summary.total_skipped') + zero_test_count=$(echo "$aggregated_data" | jq -r '.zero_test_jobs | length') + job_failure_count=$(echo "$aggregated_data" | jq -r '.job_failures_no_tests | length') + + log_verbose "Overall totals: $total_tests tests ($total_passed passed, $total_failed failed, $total_skipped skipped)" + log_verbose "Jobs with zero tests: $zero_test_count" + + + # ANSI color codes + local bold='\033[1m' + local green='\033[32m' + local red='\033[31m' + local yellow='\033[33m' + local cyan='\033[36m' + local gray='\033[90m' + local reset='\033[0m' + + # Color based on values + local failed_color=$gray + local skipped_color=$gray + [ "$total_failed" -gt 0 ] && failed_color=$red + [ "$total_skipped" -gt 0 ] && skipped_color=$yellow + + + # Display summary + echo "" + echo -e "${cyan}${bold}Pipeline Test Summary:${reset}" + echo -e " ${bold}Total:${reset} $total_tests" + echo -e " ${green}Passed:${reset} $total_passed" + echo -e " ${failed_color}Failed:${reset} $total_failed" + echo -e " ${skipped_color}Skipped:${reset} $total_skipped" + echo "" + + + # Links to Datadog CI Visibility and Test Optimization + if [ -n "${CI_PIPELINE_ID}" ]; then + echo -e "${bold}${yellow}See test results in Datadog:${reset}" + echo -e " ${cyan}CI Visibility:${reset} https://app.datadoghq.com/ci/test/runs?query=test_level%3Atest%20%40test.service%3Add-trace-java%20%40ci.pipeline.id%3A${CI_PIPELINE_ID}" + echo -e " ${cyan}Test Optimization:${reset} https://app.datadoghq.com/ci/settings/test-optimization?search=dd-trace-java" + echo "" + fi + + + # Display alerts in log output + if [ "$total_tests" -eq 0 ] || [ "$zero_test_count" -gt 0 ] || [ "$job_failure_count" -gt 0 ]; then + echo -e "${red}${bold}Alerts:${reset}" + + if [ "$total_tests" -eq 0 ]; then + echo -e " ${red}🚨 CRITICAL: No tests were executed in this pipeline!${reset}" + fi + + if [ "$job_failure_count" -gt 0 ]; then + echo -e " ${red}🚨 CRITICAL: $job_failure_count job(s) failed without test failures:${reset}" + echo -e " ${gray} (These likely indicate timeouts or infrastructure issues)${reset}" + echo "$aggregated_data" | jq -r '.job_failures_no_tests[] | " • \(.job) (JVM \(.jvm), status: \(.ci_job_status))"' + echo "" + fi + + if [ "$zero_test_count" -gt 0 ]; then + echo -e " ${yellow}⚠️ WARNING: $zero_test_count job(s) with zero tests:${reset}" + echo "$aggregated_data" | jq -r '.zero_test_jobs[] | " • \(.job) (\(.category), JVM \(.jvm))"' + fi + + echo "" + fi +} + +display_jvm_breakdown() { + local aggregated_data="$1" + + gitlab_section_start "test-jvm-breakdown" "Test Breakdown by JVM Version" + + # Header + printf "%-15s %12s %12s %12s %12s %10s\n" "JVM Version" "Total Tests" "Passed" "Failed" "Skipped" "Jobs" >&2 + printf "%-15s %12s %12s %12s %12s %10s\n" "---------------" "------------" "------------" "------------" "------------" "----------" >&2 + + # Data rows + echo "$aggregated_data" | jq -r '.by_jvm[] | + "\(.jvm_version)|\(.total_tests)|\(.total_passed)|\(.total_failed)|\(.total_skipped)|\(.job_count)"' | while IFS='|' read -r jvm total passed failed skipped jobs; do + printf "%-15s %12s %12s %12s %12s %10s\n" "$jvm" "$total" "$passed" "$failed" "$skipped" "$jobs" >&2 + done + + gitlab_section_end "test-jvm-breakdown" +} + +display_job_kind_breakdown() { + local aggregated_data="$1" + + gitlab_section_start "test-job-kind-breakdown" "Test Breakdown by Job Kind and JVM (ignoring splits)" + + # Header + printf "%-40s %-15s %12s %12s %12s %12s %8s\n" "Job Kind" "JVM" "Total Tests" "Passed" "Failed" "Skipped" "Splits" >&2 + printf "%-40s %-15s %12s %12s %12s %12s %8s\n" "----------------------------------------" "---------------" "------------" "------------" "------------" "------------" "--------" >&2 + + # Data rows + echo "$aggregated_data" | jq -r '.by_job_kind_and_jvm[] | + "\(.job_kind)|\(.jvm_version)|\(.total_tests)|\(.total_passed)|\(.total_failed)|\(.total_skipped)|\(.split_count)"' | while IFS='|' read -r job_kind jvm total passed failed skipped splits; do + printf "%-40s %-15s %12s %12s %12s %12s %8s\n" "$job_kind" "$jvm" "$total" "$passed" "$failed" "$skipped" "$splits" >&2 + done + + gitlab_section_end "test-job-kind-breakdown" +} + +display_detailed_results() { + local aggregated_data="$1" + local gitlab_base_url="${CI_PROJECT_URL}" + + gitlab_section_start "test-job-details" "Detailed Test Results by Job" + echo "$aggregated_data" | jq -r --arg base_url "$gitlab_base_url" '.test_jobs[] | + " → Job: \(.ci_job_name) | Tests: \(.total_tests) (passed: \(.passed_tests), failed: \(.failed_tests), skipped: \(.skipped_tests))\n Artifact: \($base_url)/-/jobs/\(.ci_job_id)/artifacts/file/test_counts_\(.ci_job_id).json"' >&2 + gitlab_section_end "test-job-details" +} + + +# ============================================================================ +# JSON Summary Generation +# ============================================================================ + +write_json_summary() { + local aggregated_data="$1" + local output_file="$2" + + log_verbose "Creating summary JSON file" + echo "$aggregated_data" | jq '{ + pipeline_id, + commit_sha, + branch, + timestamp, + test_jobs, + summary, + by_jvm, + by_job_kind_and_jvm + }' > "$output_file" + echo "Summary written to $output_file" +} + + +# ============================================================================ +# Main Script Logic +# ============================================================================ + +mkdir -p "$AGGREGATE_DIR" + +echo "Aggregating test counts..." + +# Log configuration details +while IFS= read -r line; do + log_verbose "$line" +done <<-EOF # <<- strips leading tabs + Pipeline ID: ${CI_PIPELINE_ID:-unknown} + Commit: ${CI_COMMIT_SHA:-unknown} + Branch: ${CI_COMMIT_BRANCH:-unknown} + Aggregate directory: $AGGREGATE_DIR + Output file: $OUTPUT_FILE + EOF + +# Find and validate test count files +mapfile -t VALID_FILES < <(find_and_validate_test_files "." "$OUTPUT_FILE") +if [ ${#VALID_FILES[@]} -eq 0 ]; then + exit 0 +fi + +# Aggregate test data +if ! AGGREGATED_DATA=$(aggregate_test_data "${VALID_FILES[@]}"); then + exit 1 +fi + + +# ============================================================================ +# Display Results +# ============================================================================ + +# Display summary and alerts in log +display_summary "$AGGREGATED_DATA" + +# Display breakdowns in collapsible sections +display_jvm_breakdown "$AGGREGATED_DATA" +display_job_kind_breakdown "$AGGREGATED_DATA" + +# Display detailed results in collapsible section +display_detailed_results "$AGGREGATED_DATA" + + +# ============================================================================ +# Write Output File +# ============================================================================ + +write_json_summary "$AGGREGATED_DATA" "$OUTPUT_FILE" + +# spotless:on diff --git a/.gitlab/benchmarks.yml b/.gitlab/benchmarks.yml index 7127d9d995d..28aef7ad58b 100644 --- a/.gitlab/benchmarks.yml +++ b/.gitlab/benchmarks.yml @@ -1,6 +1,5 @@ .benchmarks: stage: benchmarks - interruptible: true timeout: 1h tags: ["runner:apm-k8s-tweaked-metal"] image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/benchmarking-platform:dd-trace-java-benchmarks @@ -11,7 +10,11 @@ - if: '$CI_COMMIT_TAG =~ /^v?[0-9]+\.[0-9]+\.[0-9]+$/' when: manual allow_failure: true + - if: '$CI_COMMIT_BRANCH == "master"' + when: on_success + interruptible: false - when: on_success + interruptible: true script: - export ARTIFACTS_DIR="$(pwd)/reports" && mkdir -p "${ARTIFACTS_DIR}" - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ddbuild.io/DataDog/".insteadOf "https://github.com/DataDog/" @@ -71,13 +74,15 @@ check-big-regressions: needs: - job: benchmarks-startup artifacts: true - - job: benchmarks-load + - job: benchmarks-dacapo artifacts: true when: on_success tags: ["arch:amd64"] rules: - if: '$POPULATE_CACHE' when: never + - if: '$CI_COMMIT_BRANCH =~ /backport-pr-/' + when: never - if: '$CI_COMMIT_BRANCH !~ /^(master|release\/)/' when: on_success - when: never @@ -86,7 +91,7 @@ check-big-regressions: script: - !reference [ .benchmarks, script ] - | - for benchmarkType in startup load; do + for benchmarkType in startup dacapo; do find "$ARTIFACTS_DIR/$benchmarkType" -name "benchmark-baseline.json" -o -name "benchmark-candidate.json" | while read file; do relpath="${file#$ARTIFACTS_DIR/$benchmarkType/}" prefix="${relpath%/benchmark-*}" # Remove the trailing /benchmark-(baseline|candidate).json diff --git a/.gitlab/benchmarks/bp-runner.fail-on-breach.yml b/.gitlab/benchmarks/bp-runner.fail-on-breach.yml index bb2211a27fe..2e7c600bf03 100644 --- a/.gitlab/benchmarks/bp-runner.fail-on-breach.yml +++ b/.gitlab/benchmarks/bp-runner.fail-on-breach.yml @@ -1,4 +1,17 @@ -# Thresholds set based on guidance in https://datadoghq.atlassian.net/wiki/x/LgI1LgE#How-to-choose-thresholds-for-pre-release-gates%3F +# Auto-generated SLO Thresholds +# Generated: 2026-03-31 +# +# Generation Strategy: tight +# Formula: CI_bound / (1 ± T) (T = 10.0%) +# +# SLO Checking: +# - BREACH: 90% CI boundary crosses threshold +# - WARNING: 90% CI boundary crosses warning_threshold (constant) +# +# DO NOT EDIT MANUALLY - Regenerate using: +# benchmark_analyzer generate slos --help +# +# link to documentation on autogenerated thresholds https://github.com/DataDog/relenv-benchmark-analyzer/blob/main/README.md#generate-slo-thresholds experiments: - name: Run SLO breach check @@ -18,7 +31,7 @@ experiments: # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=normal_operation%2Fonly-tracing&trendsType=scenario - name: normal_operation/only-tracing thresholds: - - agg_http_req_duration_p50 < 2.6 ms + - agg_http_req_duration_p50 < 2.526 ms - agg_http_req_duration_p99 < 8.5 ms # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=normal_operation%2Fotel-latest&trendsType=scenario - name: normal_operation/otel-latest @@ -36,13 +49,13 @@ experiments: - throughput > 1100.0 op/s # Startup macrobenchmarks - # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Atracing%3AGlobalTracer&trendsType=scenario - # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Aappsec%3AGlobalTracer&trendsType=scenario - # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Aiast%3AGlobalTracer&trendsType=scenario - - name: "startup:petclinic:(tracing|appsec|iast):GlobalTracer" + # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Atracing%3AAgent.start&trendsType=scenario + - name: "startup:petclinic:tracing:Agent.start" thresholds: - - execution_time < 280 ms - # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Aprofiling%3AGlobalTracer&trendsType=scenario - - name: "startup:petclinic:profiling:GlobalTracer" + - execution_time < 1120 ms + # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Aprofiling%3AAgent.start&trendsType=scenario + # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Aappsec%3AAgent.start&trendsType=scenario + # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Aiast%3AAgent.start&trendsType=scenario + - name: "startup:petclinic:(profiling|appsec|iast):Agent.start" thresholds: - - execution_time < 420 ms + - execution_time < 1300 ms diff --git a/.gitlab/ci_visibility_generate_job.sh b/.gitlab/ci_visibility_generate_job.sh index cc3800949c3..b4abe3aaf44 100755 --- a/.gitlab/ci_visibility_generate_job.sh +++ b/.gitlab/ci_visibility_generate_job.sh @@ -42,7 +42,19 @@ if [ -z "$pr_number" ]; then exit 0 fi -echo "PR #${pr_number} found, checking labels..." +echo "PR #${pr_number} found, checking target branch..." +set +e +base_branch=$(gh pr view "$pr_number" --repo DataDog/dd-trace-java --json baseRefName --jq '.baseRefName' 2>&1) +base_branch_status=$? +set -e + +if [ $base_branch_status -eq 0 ] && [[ "$base_branch" == release/* ]]; then + echo "PR #$pr_number targets release branch '$base_branch' - skipping trigger" + add_dummy_job + exit 0 +fi + +echo "Checking labels..." set +e labels=$(gh pr view "$pr_number" --repo DataDog/dd-trace-java --json labels --jq '.labels[].name' 2>&1) labels_status=$? @@ -59,14 +71,40 @@ if [ -z "$labels" ] || ! echo "$labels" | grep -q "comp: ci visibility"; then exit 0 fi -echo "PR #$pr_number is a CI Visibility PR - triggering test environment" +echo "PR #$pr_number is a CI Visibility PR" + +# Check for test-environment configuration in PR body +set +e +echo "Checking additional trigger configuration" +target_branch="main" +pr_body=$(gh pr view "$pr_number" --repo DataDog/dd-trace-java --json body --jq '.body' 2>&1) +pr_body_status=$? +if [ $pr_body_status -eq 0 ] && [ -n "$pr_body" ]; then + # Check for skip directive: "test-environment-trigger: skip" (must be at start of line) + if echo "$pr_body" | grep -qP '^test-environment-trigger:\s*skip'; then + echo "Found 'test-environment-trigger: skip' in PR body - skipping trigger" + add_dummy_job + exit 0 + fi + # Look for "test-environment-branch: " at start of line in PR body + override_branch=$(echo "$pr_body" | grep -oP '^test-environment-branch:\s*\K[\S]+' | head -1) + if [ -n "$override_branch" ]; then + echo "Found test-environment branch override in PR body: '$override_branch'" + target_branch="$override_branch" + else + echo "No test-environment-branch override in PR body - using default 'main' for downstream pipeline" + fi +else + echo "Could not read PR body (status=$pr_body_status) - using default 'main' for downstream pipeline" +fi +set -e cat <>ci-visibility-test-environment.yml ci-visibility-test-environment: stage: ci-visibility-tests trigger: project: DataDog/apm-reliability/test-environment - branch: main + branch: $target_branch strategy: depend variables: UPSTREAM_PACKAGE_JOB: build diff --git a/.gitlab/collect_results.sh b/.gitlab/collect_results.sh index 03d69d080d6..dec34ed5d59 100755 --- a/.gitlab/collect_results.sh +++ b/.gitlab/collect_results.sh @@ -12,7 +12,13 @@ WORKSPACE_DIR=workspace mkdir -p $TEST_RESULTS_DIR mkdir -p $WORKSPACE_DIR -mapfile -t TEST_RESULT_DIRS < <(find $WORKSPACE_DIR -name test-results -type d) +# Main project modules redirect their build directory to workspace//build/ in CI +# (see build.gradle.kts layout.buildDirectory override). buildSrc is a separate Gradle build +# that runs before the main build is configured, so this redirect never applies to it; +# its test results always land in buildSrc/**/build/test-results/, not under workspace/. +SEARCH_DIRS=($WORKSPACE_DIR buildSrc) + +mapfile -t TEST_RESULT_DIRS < <(find "${SEARCH_DIRS[@]}" -name test-results -type d) if [[ ${#TEST_RESULT_DIRS[@]} -eq 0 ]]; then echo "No test results found" @@ -26,16 +32,30 @@ function get_source_file () { class="${RESULT_XML_FILE%.xml}" class="${class##*"TEST-"}" class="${class##*"."}" - common_root=$(grep -rl "class $class" "$file_path" | head -n 1) - while IFS= read -r line; do - while [[ $line != "$common_root"* ]]; do - common_root=$(dirname "$common_root") - if [[ "$common_root" == "$common_root/.." ]]; then - break - fi - done - done < <(grep -rl "class $class" "$file_path") - file_path="/$common_root" + class="${class##*"$"}" # remove inner class name if it exists + set +e # allow grep to fail + common_root=$(grep -rl "class $class\|static class $class" "$file_path" 2>/dev/null | head -n 1) + set -e + + if [[ -n "$common_root" ]]; then + while IFS= read -r line; do + while [[ $line != "$common_root"* ]]; do + common_root=$(dirname "$common_root") + if [[ "$common_root" == "$common_root/.." ]] || [[ "$common_root" == "/" ]]; then + common_root="" + break + fi + done + done < <(grep -rl "class $class\|static class $class" "$file_path" 2>/dev/null) + + if [[ -n "$common_root" && "$common_root" != "/" ]]; then + file_path="/$common_root" + else + file_path="UNKNOWN" + fi + else + file_path="UNKNOWN" + fi fi } @@ -43,19 +63,40 @@ echo "Saving test results:" while IFS= read -r -d '' RESULT_XML_FILE do echo -n "- $RESULT_XML_FILE" + # Assuming the path looks like that: dd-java-agent/instrumentation/tomcat/tomcat-5.5/build/test-results/forkedTest/TEST-TomcatServletV1ForkedTest.xml + # it will extracts 3 components from the path (counting from the end), to form the new name AGGREGATED_FILE_NAME: + # + # 1. Field 1 (from end): The XML filename itself + # 2. Field 2 (from end): The test suite type (test, forkedTest, etc.) + # 3. Field 5 (from end): The module/subproject name + # + # E.g. for the example path: tomcat-5.5_forkedTest_TEST-TomcatServletV1ForkedTest.xml AGGREGATED_FILE_NAME=$(echo "$RESULT_XML_FILE" | rev | cut -d "/" -f 1,2,5 | rev | tr "/" "_") echo -n " as $AGGREGATED_FILE_NAME" - cp "$RESULT_XML_FILE" "$TEST_RESULTS_DIR/$AGGREGATED_FILE_NAME" + TARGET_DIR="$TEST_RESULTS_DIR" + mkdir -p "$TARGET_DIR" + cp "$RESULT_XML_FILE" "$TARGET_DIR/$AGGREGATED_FILE_NAME" # Insert file attribute to testcase XML nodes get_source_file - sed -i "/ + +# https://docs.gitlab.com/ci/variables/predefined_variables/ + +VERBOSE=0 +if [ "$1" = "-v" ]; then + VERBOSE=1 + shift +fi + +TEST_CATEGORY="${1:-unknown}" +JVM_VERSION="${2:-unknown}" +RESULTS_DIR="${3:-./results}" +OUTPUT_FILE="${4:-./test_counts.json}" + +log_verbose() { + if [ $VERBOSE -eq 1 ]; then + echo "[count_tests] $*" >&2 + fi +} + +log_verbose "Job: ${CI_JOB_NAME:-unknown} (ID: ${CI_JOB_ID:-unknown})" +log_verbose "Test category: $TEST_CATEGORY" +log_verbose "JVM version: $JVM_VERSION" +log_verbose "Results directory: $RESULTS_DIR" +log_verbose "Output file: $OUTPUT_FILE" + +if [ ! -d "$RESULTS_DIR" ]; then + echo "Results directory not found: $RESULTS_DIR" + log_verbose "Directory does not exist, exiting" + exit 0 +fi + +# Count tests from JUnit XML files +TOTAL_TESTS=0 +TOTAL_FAILURES=0 +TOTAL_ERRORS=0 +TOTAL_SKIPPED=0 +XML_FILE_COUNT=0 + +echo "Counting tests in $RESULTS_DIR for $TEST_CATEGORY on JVM $JVM_VERSION" + +# Find all XML files and count tests +log_verbose "Searching for XML files in $RESULTS_DIR" +while IFS= read -r -d '' xml_file; do + XML_FILE_COUNT=$((XML_FILE_COUNT + 1)) + log_verbose "Processing file $XML_FILE_COUNT: $xml_file" + if [ -f "$xml_file" ]; then + # Count actual elements (more accurate than testsuite attributes) + # This matches how datadog-ci counts tests + tests=$(grep -c '/dev/null || echo "0") + tests="${tests//[$'\n\r']/}" # Strip any newlines/carriage returns + + # Count , , and tags + # These are more reliable than trying to match testcase+failure in one grep + failures=$(grep -c '/dev/null || echo "0") + failures="${failures//[$'\n\r']/}" # Strip any newlines/carriage returns + + errors=$(grep -c '/dev/null || echo "0") + errors="${errors//[$'\n\r']/}" # Strip any newlines/carriage returns + + skipped=$(grep -c '/dev/null || echo "0") + skipped="${skipped//[$'\n\r']/}" # Strip any newlines/carriage returns + + log_verbose " → tests=$tests, failures=$failures, errors=$errors, skipped=$skipped" + + TOTAL_TESTS=$((TOTAL_TESTS + tests)) + TOTAL_FAILURES=$((TOTAL_FAILURES + failures)) + TOTAL_ERRORS=$((TOTAL_ERRORS + errors)) + TOTAL_SKIPPED=$((TOTAL_SKIPPED + skipped)) + fi +done < <(find "$RESULTS_DIR" -name "*.xml" -type f -print0 2>/dev/null) + +log_verbose "Processed $XML_FILE_COUNT XML files" + +TOTAL_PASSED=$((TOTAL_TESTS - TOTAL_FAILURES - TOTAL_ERRORS - TOTAL_SKIPPED)) + +# ANSI color codes +BOLD='\033[1m' +GREEN='\033[32m' +RED='\033[31m' +YELLOW='\033[33m' +CYAN='\033[36m' +GRAY='\033[90m' +RESET='\033[0m' + +# Color based on values +FAILED_COLOR=$GRAY +ERRORS_COLOR=$GRAY +SKIPPED_COLOR=$GRAY +[ $TOTAL_FAILURES -gt 0 ] && FAILED_COLOR=$RED +[ $TOTAL_ERRORS -gt 0 ] && ERRORS_COLOR=$RED +[ $TOTAL_SKIPPED -gt 0 ] && SKIPPED_COLOR=$YELLOW + +echo -e "${CYAN}${BOLD}Test counts for $TEST_CATEGORY on JVM $JVM_VERSION:${RESET}" +echo -e " ${BOLD}Total:${RESET} $TOTAL_TESTS" +echo -e " ${GREEN}Passed:${RESET} $TOTAL_PASSED" +echo -e " ${FAILED_COLOR}Failed:${RESET} $TOTAL_FAILURES" +echo -e " ${ERRORS_COLOR}Errors:${RESET} $TOTAL_ERRORS" +echo -e " ${SKIPPED_COLOR}Skipped:${RESET} $TOTAL_SKIPPED" + +# ============================================================================ +# Detect Job Failures (using CI_JOB_STATUS) +# ============================================================================ + +echo "" +if [ "${CI_JOB_STATUS:-}" = "failed" ]; then + JOB_FAILED="true" + echo -e "${RED}${BOLD}CI Job Status:${RESET} ${RED}failed${RESET}" +else + JOB_FAILED="false" + echo -e "${GREEN}${BOLD}CI Job Status:${RESET} ${GREEN}${CI_JOB_STATUS:-unknown}${RESET}" +fi + +# ============================================================================ +# Create JSON output with failure metadata +# ============================================================================ + +log_verbose "Writing JSON output to $OUTPUT_FILE" +cat > "$OUTPUT_FILE" <&2 + exit 1 +fi + CURRENT_HEAD_SHA="$(git rev-parse HEAD)" if [[ -z "${CURRENT_HEAD_SHA:-}" ]]; then echo "Failed to determine current HEAD SHA" >&2 diff --git a/.gitlab/java-benchmark-configs.yml b/.gitlab/java-benchmark-configs.yml new file mode 100644 index 00000000000..7b9d7188c40 --- /dev/null +++ b/.gitlab/java-benchmark-configs.yml @@ -0,0 +1,21 @@ +# Ensure the tracer artifact publish finishes before the benchmark jobs start. +linux-java-spring-petclinic-parallel: + needs: ["publish-artifacts-to-s3"] + +linux-java-insecure-bank-load-parallel: + needs: ["publish-artifacts-to-s3"] + +linux-java-spring-petclinic-load-parallel: + needs: ["publish-artifacts-to-s3"] + +linux-java-insecure-bank-startup-parallel: + needs: ["publish-artifacts-to-s3"] + +linux-java-spring-petclinic-startup-parallel: + needs: ["publish-artifacts-to-s3"] + +linux-java-dacapo-parallel-1: + needs: ["publish-artifacts-to-s3"] + +linux-java-dacapo-parallel-2: + needs: ["publish-artifacts-to-s3"] diff --git a/.gitlab/macrobenchmarks.yml b/.gitlab/macrobenchmarks.yml index d2ad417c2dd..b1c5681fb5b 100644 --- a/.gitlab/macrobenchmarks.yml +++ b/.gitlab/macrobenchmarks.yml @@ -11,11 +11,12 @@ include: when: never - if: ($NIGHTLY_BENCHMARKS || $CI_PIPELINE_SOURCE != "schedule") && $CI_COMMIT_REF_NAME == "master" when: always + interruptible: false - when: manual + interruptible: true allow_failure: true tags: ["runner:apm-k8s-same-cpu"] needs: ["build"] - interruptible: true timeout: 1h image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/benchmarking-platform:dd-trace-java-petclinic script: @@ -142,4 +143,4 @@ notify-slo-breaches: when: never - when: always variables: - CHANNEL: "apm-java" + CHANNEL: "apm-java-ops-bot" diff --git a/.gitlab/one-pipeline.locked.yml b/.gitlab/one-pipeline.locked.yml index 14fae58a18e..746078fae73 100644 --- a/.gitlab/one-pipeline.locked.yml +++ b/.gitlab/one-pipeline.locked.yml @@ -1,4 +1,4 @@ # DO NOT EDIT THIS FILE MANUALLY # This file is auto-generated by automation. include: - - remote: https://gitlab-templates.ddbuild.io/libdatadog/one-pipeline/ca/53cec1ca53804e5abff804aefdd5cffbcaa5cb546c7e6fcf4c35df6796e06bf1/one-pipeline.yml + - remote: https://gitlab-templates.ddbuild.io/libdatadog/one-pipeline/ca/0a900a87c53d3a57a5ab007b3147db4601f15c73ff31dc65f3791c803f2651d9/one-pipeline.yml diff --git a/.gitlab/upload_ciapp.sh b/.gitlab/upload_ciapp.sh index ecf0388813f..a4c36438796 100755 --- a/.gitlab/upload_ciapp.sh +++ b/.gitlab/upload_ciapp.sh @@ -1,17 +1,42 @@ #!/usr/bin/env bash SERVICE_NAME="dd-trace-java" CACHE_TYPE=$1 -TEST_JVM=$2 +TEST_JVM=${2:-} + +# CI_JOB_NAME, CI_NODE_INDEX, and CI_NODE_TOTAL are read from GitLab CI environment # JAVA_???_HOME are set in the base image for each used JDK https://github.com/DataDog/dd-trace-java-docker-build/blob/master/Dockerfile#L86 -JAVA_HOME="JAVA_${TEST_JVM}_HOME" -JAVA_BIN="${!JAVA_HOME}/bin/java" -if [ ! -x "$JAVA_BIN" ]; then - JAVA_BIN=$(which java) +JAVA_PROPS="" +if [ -n "$TEST_JVM" ]; then + JAVA_BIN="" + RESOLVED_JVM="$TEST_JVM" + if [ "$TEST_JVM" = "tip" ]; then + # Resolve "tip" to the highest available JAVA_*_HOME version + MAX_VER=0 + for var in $(compgen -v JAVA_ | grep -E '^JAVA_[0-9]+_HOME$'); do + ver="${var#JAVA_}" + ver="${ver%_HOME}" + if [ "$ver" -gt "$MAX_VER" ] 2>/dev/null; then + MAX_VER="$ver" + fi + done + if [ "$MAX_VER" -gt 0 ] 2>/dev/null; then + RESOLVED_JVM="$MAX_VER" + fi + fi + if [[ "$RESOLVED_JVM" =~ ^[A-Za-z0-9_]+$ ]]; then + JAVA_HOME_VAR="JAVA_${RESOLVED_JVM}_HOME" + JAVA_HOME_VALUE="${!JAVA_HOME_VAR}" + if [ -n "$JAVA_HOME_VALUE" ] && [ -x "$JAVA_HOME_VALUE/bin/java" ]; then + JAVA_BIN="$JAVA_HOME_VALUE/bin/java" + fi + fi + if [ -z "$JAVA_BIN" ]; then + JAVA_BIN="$(command -v java)" + fi + JAVA_PROPS=$($JAVA_BIN -XshowSettings:properties -version 2>&1) fi -# Extract Java properties from the JVM used to run the tests -JAVA_PROPS=$($JAVA_BIN -XshowSettings:properties -version 2>&1) java_prop() { local PROP_NAME=$1 echo "$JAVA_PROPS" | grep "$PROP_NAME" | head -n1 | cut -d'=' -f2 | xargs @@ -21,17 +46,40 @@ java_prop() { junit_upload() { # based on tracer implementation: https://github.com/DataDog/dd-trace-java/blob/master/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/TestDecorator.java#L55-L77 # Overwriting the tag with the GitHub repo URL instead of the GitLab one. Otherwise, some Test Optimization features won't work. + + # Build custom tags array directly from arguments + local custom_tags_args=() + + # Extract job base name from CI_JOB_NAME. + # Handles: + # - matrix suffix format: "job-name: [value, 1/6]" -> "job-name" + # - split suffix format: "job-name 1/6" -> "job-name" + local job_base_name="${CI_JOB_NAME%%:*}" + job_base_name="$(echo "$job_base_name" | sed -E 's/[[:space:]]+[0-9]+\/[0-9]+$//')" + + # Add custom test configuration tags + if [ -n "$TEST_JVM" ]; then + custom_tags_args+=(--tags "test.configuration.jvm:${TEST_JVM}") + custom_tags_args+=(--tags "runtime.name:$(java_prop java.runtime.name)") + custom_tags_args+=(--tags "runtime.vendor:$(java_prop java.vendor)") + custom_tags_args+=(--tags "runtime.version:$(java_prop java.version)") + custom_tags_args+=(--tags "os.architecture:$(java_prop os.arch)") + custom_tags_args+=(--tags "os.platform:$(java_prop os.name)") + custom_tags_args+=(--tags "os.version:$(java_prop os.version)") + fi + if [ -n "$CI_NODE_INDEX" ] && [ -n "$CI_NODE_TOTAL" ]; then + custom_tags_args+=(--tags "test.configuration.split:${CI_NODE_INDEX}/${CI_NODE_TOTAL}") + fi + if [ -n "$job_base_name" ]; then + custom_tags_args+=(--tags "test.configuration.job_name:${job_base_name}") + fi + DD_API_KEY=$1 \ datadog-ci junit upload --service $SERVICE_NAME \ --logs \ --tags "test.traits:{\"category\":[\"$CACHE_TYPE\"]}" \ - --tags "runtime.name:$(java_prop java.runtime.name)" \ - --tags "runtime.vendor:$(java_prop java.vendor)" \ - --tags "runtime.version:$(java_prop java.version)" \ - --tags "os.architecture:$(java_prop os.arch)" \ - --tags "os.platform:$(java_prop os.name)" \ - --tags "os.version:$(java_prop os.version)" \ --tags "git.repository_url:https://github.com/DataDog/dd-trace-java" \ + "${custom_tags_args[@]}" \ ./results } diff --git a/.gitmodules b/.gitmodules index 27fc232a993..593a3fb913e 100644 --- a/.gitmodules +++ b/.gitmodules @@ -1,3 +1,3 @@ [submodule "dd-java-agent/agent-jmxfetch/integrations-core"] path = dd-java-agent/agent-jmxfetch/integrations-core - url = https://github.com/DataDog/integrations-core.git + url = https://github.com/GuanceCloud/integrations-core.git diff --git a/.mvn/wrapper/MavenWrapperDownloader.java b/.mvn/wrapper/MavenWrapperDownloader.java deleted file mode 100644 index 9ac51e49730..00000000000 --- a/.mvn/wrapper/MavenWrapperDownloader.java +++ /dev/null @@ -1,118 +0,0 @@ -/* - * Copyright 2007-present the original author or authors. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -import java.io.*; -import java.net.*; -import java.nio.channels.*; -import java.util.Properties; - -public class MavenWrapperDownloader { - - private static final String WRAPPER_VERSION = "0.5.6"; - /** Default URL to download the maven-wrapper.jar from, if no 'downloadUrl' is provided. */ - private static final String DEFAULT_DOWNLOAD_URL = - "https://repo.maven.apache.org/maven2/io/takari/maven-wrapper/" - + WRAPPER_VERSION - + "/maven-wrapper-" - + WRAPPER_VERSION - + ".jar"; - - /** - * Path to the maven-wrapper.properties file, which might contain a downloadUrl property to use - * instead of the default one. - */ - private static final String MAVEN_WRAPPER_PROPERTIES_PATH = - ".mvn/wrapper/maven-wrapper.properties"; - - /** Path where the maven-wrapper.jar will be saved to. */ - private static final String MAVEN_WRAPPER_JAR_PATH = ".mvn/wrapper/maven-wrapper.jar"; - - /** - * Name of the property which should be used to override the default download url for the wrapper. - */ - private static final String PROPERTY_NAME_WRAPPER_URL = "wrapperUrl"; - - public static void main(String args[]) { - System.out.println("- Downloader started"); - File baseDirectory = new File(args[0]); - System.out.println("- Using base directory: " + baseDirectory.getAbsolutePath()); - - // If the maven-wrapper.properties exists, read it and check if it contains a custom - // wrapperUrl parameter. - File mavenWrapperPropertyFile = new File(baseDirectory, MAVEN_WRAPPER_PROPERTIES_PATH); - String url = DEFAULT_DOWNLOAD_URL; - if (mavenWrapperPropertyFile.exists()) { - FileInputStream mavenWrapperPropertyFileInputStream = null; - try { - mavenWrapperPropertyFileInputStream = new FileInputStream(mavenWrapperPropertyFile); - Properties mavenWrapperProperties = new Properties(); - mavenWrapperProperties.load(mavenWrapperPropertyFileInputStream); - url = mavenWrapperProperties.getProperty(PROPERTY_NAME_WRAPPER_URL, url); - } catch (IOException e) { - System.out.println("- ERROR loading '" + MAVEN_WRAPPER_PROPERTIES_PATH + "'"); - } finally { - try { - if (mavenWrapperPropertyFileInputStream != null) { - mavenWrapperPropertyFileInputStream.close(); - } - } catch (IOException e) { - // Ignore ... - } - } - } - System.out.println("- Downloading from: " + url); - - File outputFile = new File(baseDirectory.getAbsolutePath(), MAVEN_WRAPPER_JAR_PATH); - if (!outputFile.getParentFile().exists()) { - if (!outputFile.getParentFile().mkdirs()) { - System.out.println( - "- ERROR creating output directory '" - + outputFile.getParentFile().getAbsolutePath() - + "'"); - } - } - System.out.println("- Downloading to: " + outputFile.getAbsolutePath()); - try { - downloadFileFromURL(url, outputFile); - System.out.println("Done"); - System.exit(0); - } catch (Throwable e) { - System.out.println("- Error downloading"); - e.printStackTrace(); - System.exit(1); - } - } - - private static void downloadFileFromURL(String urlString, File destination) throws Exception { - if (System.getenv("MVNW_USERNAME") != null && System.getenv("MVNW_PASSWORD") != null) { - String username = System.getenv("MVNW_USERNAME"); - char[] password = System.getenv("MVNW_PASSWORD").toCharArray(); - Authenticator.setDefault( - new Authenticator() { - @Override - protected PasswordAuthentication getPasswordAuthentication() { - return new PasswordAuthentication(username, password); - } - }); - } - URL website = new URL(urlString); - ReadableByteChannel rbc; - rbc = Channels.newChannel(website.openStream()); - FileOutputStream fos = new FileOutputStream(destination); - fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE); - fos.close(); - rbc.close(); - } -} diff --git a/.mvn/wrapper/maven-wrapper.jar b/.mvn/wrapper/maven-wrapper.jar deleted file mode 100644 index 2cc7d4a55c0..00000000000 Binary files a/.mvn/wrapper/maven-wrapper.jar and /dev/null differ diff --git a/.mvn/wrapper/maven-wrapper.properties b/.mvn/wrapper/maven-wrapper.properties index 642d572ce90..a20bf2f300f 100644 --- a/.mvn/wrapper/maven-wrapper.properties +++ b/.mvn/wrapper/maven-wrapper.properties @@ -1,2 +1,3 @@ -distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.6.3/apache-maven-3.6.3-bin.zip -wrapperUrl=https://repo.maven.apache.org/maven2/io/takari/maven-wrapper/0.5.6/maven-wrapper-0.5.6.jar +wrapperVersion=3.3.4 +distributionType=only-script +distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.8.9/apache-maven-3.8.9-bin.zip diff --git a/.profileconfig.json b/.profileconfig.json new file mode 100644 index 00000000000..b2f42d90d09 --- /dev/null +++ b/.profileconfig.json @@ -0,0 +1,64 @@ +{ + "jfrConfig": { + "settings": "profile" + }, + "asyncProfilerConfig": { + "jfrsync": true, + "alloc": true, + "event": "wall", + "misc": "" + }, + "file": "$PROJECT_DIR/profile.jfr", + "conversionConfig": { + "nonProjectPackagePrefixes": [ + "java.", + "javax.", + "kotlin.", + "jdk.", + "com.google.", + "org.apache.", + "org.spring.", + "sun.", + "scala." + ], + "enableMarkers": true, + "initialVisibleThreads": 10, + "initialSelectedThreads": 10, + "includeGCThreads": false, + "includeInitialSystemProperty": false, + "includeInitialEnvironmentVariables": false, + "includeSystemProcesses": false, + "ignoredEvents": [ + "jdk.ActiveSetting", + "jdk.ActiveRecording", + "jdk.BooleanFlag", + "jdk.IntFlag", + "jdk.DoubleFlag", + "jdk.LongFlag", + "jdk.NativeLibrary", + "jdk.StringFlag", + "jdk.UnsignedIntFlag", + "jdk.UnsignedLongFlag", + "jdk.InitialSystemProperty", + "jdk.InitialEnvironmentVariable", + "jdk.SystemProcess", + "jdk.ModuleExport", + "jdk.ModuleRequire" + ], + "minRequiredItemsPerThread": 3 + }, + "additionalGradleTargets": [ + { + "targetPrefix": "quarkus", + "optionForVmArgs": "-Djvm.args", + "description": "Example quarkus config, adding profiling arguments via -Djvm.args option to the Gradle task run" + } + ], + "additionalMavenTargets": [ + { + "targetPrefix": "quarkus:", + "optionForVmArgs": "-Djvm.args", + "description": "Example quarkus config, adding profiling arguments via -Djvm.args option to the Maven goal run" + } + ] +} \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000000..ef91103f717 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,80 @@ +# Agent context for dd-trace-java + +## What is this project? + +Datadog Java APM agent (`dd-trace-java`): a Java agent that auto-instruments JVM applications at runtime via bytecode manipulation. +It ships ~120 integrations (~200 instrumentations) for tracing, profiling, AppSec, IAST, CI Visibility, USM, and LLM Observability. + +## Project layout + +See [ARCHITECTURE.md](ARCHITECTURE.md) for detailed module descriptions. + +``` +dd-java-agent/ Main agent (shadow jar, instrumentations, product modules) +dd-trace-api/ Public API & configuration constants +dd-trace-core/ Core tracing engine (spans, propagation, writer) +dd-trace-ot/ Legacy OpenTracing compatibility library +internal-api/ Internal shared API across modules +components/ Shared low-level components (context, environment, json) +products/ Sub-products (feature flagging, metrics) +communication/ HTTP transport to Datadog Agent +remote-config/ Remote configuration support +telemetry/ Agent telemetry +utils/ Shared utility modules (config, time, socket, test, etc.) +dd-smoke-tests/ Smoke tests (real apps + agent) +docs/ Developer documentation (see below) +``` + +## Key documentation (read on demand, don't load upfront) + +| Topic | File | +|---|---| +| Architecture & design | [ARCHITECTURE.md](ARCHITECTURE.md) | +| Building from source | [BUILDING.md](BUILDING.md) | +| Contributing & PR guidelines | [CONTRIBUTING.md](CONTRIBUTING.md) | +| How instrumentations work | [docs/how_instrumentations_work.md](docs/how_instrumentations_work.md) | +| Adding a new instrumentation | [docs/add_new_instrumentation.md](docs/add_new_instrumentation.md) | +| Adding a new configuration | [docs/add_new_configurations.md](docs/add_new_configurations.md) | +| Testing guide (6 test types) | [docs/how_to_test.md](docs/how_to_test.md) | +| Working with Gradle | [docs/how_to_work_with_gradle.md](docs/how_to_work_with_gradle.md) | +| Bootstrap/premain constraints | [docs/bootstrap_design_guidelines.md](docs/bootstrap_design_guidelines.md) | +| CI/CD workflows | [.github/workflows/README.md](.github/workflows/README.md) | + +**When working on a topic above, read the linked file first** — they are the source of truth maintained by humans. + +## Build quick reference + +```shell +./gradlew clean assemble # Build without tests +./gradlew :dd-java-agent:shadowJar # Build agent jar only (dd-java-agent/build/libs/) +./gradlew :path:to:module:test # Run tests for a specific module +./gradlew :path:to:module:test -PtestJvm=11 # Test on a specific JVM version +./gradlew spotlessApply # Auto-format code (google-java-format) +./gradlew spotlessCheck # Verify formatting +``` + +## Code conventions + +- **Formatting**: google-java-format enforced via Spotless. Run `./gradlew spotlessApply` before committing. +- **Instrumentation layout**: `dd-java-agent/instrumentation/{framework}/{framework}-{minVersion}/` +- **Instrumentation pattern**: Type matching → Method matching → Advice class (bytecode advice, not AOP) +- **Test frameworks**: JUnit 5 (preferred for unit tests), Spock 2 (for complex scenarios needing Groovy) +- **Forked tests**: Use `ForkedTest` suffix when tests need a separate JVM +- **Flaky tests**: Annotate with `@Flaky` — they are skipped in CI by default + +## PR conventions + +- Title: imperative verb sentence describing user-visible change (e.g. "Fix span sampling rule parsing") +- Labels: at least one `comp:` or `inst:` label + one `type:` label +- Use `tag: no release note` for internal/refactoring changes +- Use `tag: ai generated` for AI generated code +- Open as draft first, convert to ready when reviewable + +## Bootstrap constraints (critical) + +Code running in the agent's `premain` phase must **not** use: +- `java.util.logging.*` — locks in log manager before app configures it +- `java.nio.*` — triggers premature provider initialization +- `javax.management.*` — causes class loading issues + +See [docs/bootstrap_design_guidelines.md](docs/bootstrap_design_guidelines.md) for details and alternatives. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 00000000000..ce12897532f --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,292 @@ +# Architecture of dd-trace-java + +High-level architecture of the Datadog Java APM agent. +Start here to orient yourself in the codebase. + +## Bird's Eye View + +dd-trace-java is a Java agent that auto-instruments JVM applications at runtime via bytecode manipulation. +It attaches to a running JVM using the `-javaagent` flag, intercepts class loading, and rewrites method bytecode +to inject tracing, security, profiling, and observability logic. No application code changes required. + +Ships ~120 integrations (~200 instrumentations) covering major frameworks +(Spring, Servlet, gRPC, JDBC, Kafka, etc.) and supports multiple Datadog products through a single jar: +**Tracing**, **Profiling**, **Application Security (AppSec)**, **IAST**, **CI Visibility**, +**Dynamic Instrumentation**, **LLM Observability**, **Crash Tracking**, **Data Streams**, +**Feature Flagging**, and **USM**. + +Communicates with a local Datadog Agent process (or directly with the Datadog intake APIs) +to send collected telemetry. + +## Startup Sequence + +1. **`AgentBootstrap.premain()`** — JVM entry point. Runs on the application classloader + with minimal logic: locates the agent jar, creates an isolated classloader, jumps to `Agent.start()`. + Must remain tiny and side-effect-free. + +2. **`Agent.start()`** — Runs on the bootstrap classloader. Creates the agent classloader, + reads configuration, determines which products are enabled, starts each subsystem on dedicated threads. + +3. **`AgentInstaller`** — Installs the ByteBuddy `ClassFileTransformer` that intercepts all class loading. + Discovers all `InstrumenterModule` implementations via service loading, registers their + type matchers and advice classes. + +4. **Product subsystems start** — Each enabled product is started via its own `*System.start()` method, + receiving shared communication objects. + +## Codemap + +### `dd-java-agent/` + +Main agent module. Produces the final shadow jar (`dd-java-agent.jar`) using a composite shadow jar +strategy. Each product module builds its own shadow jar, embedded as a nested directory inside the +main jar (`inst/`, `profiling/`, `appsec/`, `iast/`, `debugger/`, `ci-visibility/`, `llm-obs/`, +`shared/`, `trace/`, etc.). A dedicated `sharedShadowJar` bundles common transitive dependencies +(OkHttp, JCTools, LZ4, etc.) to avoid duplication across feature jars. All dependencies are relocated +under `datadog.` prefixes to prevent classpath conflicts. Class files inside feature jars are renamed +to `.classdata` to prevent unintended loading. See [`docs/how_to_work_with_gradle.md`](docs/how_to_work_with_gradle.md). + +- **`src/`** — `AgentBootstrap` and `AgentJar`, the entry point loaded by `-javaagent`. + Deliberately minimal. + +- **`agent-bootstrap/`** — Classes on the bootstrap classloader: `Agent` (startup orchestrator), + decorator base classes (`HttpServerDecorator`, `DatabaseClientDecorator`, etc.), and bootstrap-safe + utilities. Visible to all classloaders, so instrumentation advice and helpers can use them directly. + See [`docs/bootstrap_design_guidelines.md`](docs/bootstrap_design_guidelines.md). + +- **`agent-installer/`** — Product installers and ByteBuddy integration layer. Class transformer pipeline: + `DDClassFileTransformer` intercepts every class load, `GlobalIgnoresMatcher` applies early + filtering, `CombiningMatcher` evaluates instrumentation matchers, `SplittingTransformer` + applies matched transformations. The `ignored_class_name.trie` is a compiled trie built at + build time that short-circuits matcher evaluation for known non-transformable classes (JVM + internals, agent infrastructure, monitoring libraries, large framework packages). When a class + is unexpectedly not instrumented, check the trie first. + +- **`agent-tooling/`** — Instrumentation framework. Key types: + - `InstrumenterModule` — Base class for all instrumentation modules. Declares a target system + (Tracing, AppSec, IAST, Profiling, CiVisibility, USM, etc.) and one or more instrumentations. + - `Instrumenter` — Type matching interface: `ForSingleType`, `ForKnownTypes`, + `ForTypeHierarchy`, `ForBootstrap`. + - `muzzle/` — Build-time and runtime safety checks. Verifies that expected types and methods + exist in the library version at runtime. If not, the instrumentation is silently skipped. + See [`docs/how_instrumentations_work.md`](docs/how_instrumentations_work.md) and [`docs/add_new_instrumentation.md`](docs/add_new_instrumentation.md). + +- **`instrumentation/`** — All auto-instrumentations, organized as `{framework}/{framework}-{minVersion}/`. + Nearly 200 framework directories. Each follows the same pattern: an `InstrumenterModule` declares the + target system and integration name, one or more `Instrumenter` implementations select target types + via matchers, advice classes inject bytecode via `@Advice.OnMethodEnter`/`@Advice.OnMethodExit`, + and decorator/helper classes contain the actual product logic. Instrumentations are discovered + via `@AutoService(InstrumenterModule.class)` (Java SPI) and validated by Muzzle at build time. + See [`docs/how_instrumentations_work.md`](docs/how_instrumentations_work.md) and [`docs/add_new_instrumentation.md`](docs/add_new_instrumentation.md) for details. + +- **`appsec/`** — Application Security. Entry point: `AppSecSystem.start()`. Runs the Datadog WAF + to detect and block attacks in real-time. Hooks into the gateway to intercept HTTP requests. + +- **`agent-iast/`** — Interactive Application Security Testing. Entry point: `IastSystem.start()`. + Performs taint tracking: marks user input as tainted, propagates taint through string operations, + and reports when tainted data reaches dangerous sinks (SQL injection, XSS, command injection, etc.). + +- **`agent-ci-visibility/`** — CI Visibility. Entry point: `CiVisibilitySystem.start()`. + Instruments test frameworks (JUnit, TestNG, Gradle, Maven, Cucumber) to collect test results, + code coverage, and performance metrics. + +- **`agent-profiling/`** — Continuous Profiling. Entry point: `ProfilingAgent`. + Collects CPU, memory, and wall-clock profiles using JFR or the Datadog native profiler (`ddprof`). + Uploads profiles to the Datadog backend. + +- **`agent-debugger/`** — Dynamic Instrumentation. Entry point: `DebuggerAgent`. + Probes, snapshot capture, exception replay, code origin mapping. + Driven by remote configuration. + +- **`agent-llmobs/`** — LLM Observability. Entry point: `LLMObsSystem.start()`. + Monitors LLM API calls (OpenAI, LangChain, etc.): token usage, model inference, evaluations. + +- **`agent-crashtracking/`** — Crash Tracking. Detects JVM crashes and fatal exceptions, + collects system metadata, and uploads crash reports to Datadog's error tracking intake. + +- **`agent-otel/`** — OpenTelemetry compatibility shim. `OtelTracerProvider`, `OtelSpan`, + `OtelContext` and other wrappers implement the OTel API by delegating to the Datadog tracer. + Paired with instrumentations in `instrumentation/opentelemetry/` that intercept OTel API calls + and redirect them to shim instances. + +### `dd-trace-core/` + +Core tracing engine. Grew organically and now also hosts product-specific features that depend on +tight integration with span creation, interception, or serialization. New code should go in +`products/` or `components/` instead. Core tracing types: + +- `CoreTracer` — Tracer implementation. Creates spans, manages sampling, drives the writer pipeline. + Implements `AgentTracer.TracerAPI`. +- `DDSpan` / `DDSpanContext` — Concrete span and context implementations with Datadog-specific metadata. +- `PendingTrace` — Collects all spans in a trace. Flushes to the writer when the root span finishes. +- `scopemanager/` — `ContinuableScopeManager`, `ContinuableScope`, `ScopeContinuation`. Active span + per thread, async context propagation via continuations. +- `propagation/` — Trace context propagation codecs: Datadog, W3C TraceContext, B3, Haystack, X-Ray. +- `common/writer/` — Writer pipeline. `DDAgentWriter` buffers traces and dispatches via + `PayloadDispatcherImpl` to the Datadog Agent's `/v0.4/traces` endpoint. `DDIntakeWriter` for + direct API submission. `TraceProcessingWorker` for async processing. +- `common/sampling/` — Sampling logic: `RuleBasedTraceSampler`, `RateByServiceTraceSampler`, + `SingleSpanSampler`. Supports both head-based and rule-based sampling. +- `tagprocessor/` — Post-processing of span tags: peer service calculation, base service naming, + query obfuscation, endpoint resolution. + +Non-tracing code that also lives here due to organic growth: + +- `datastreams/` — Data Streams Monitoring. Tracks message pipeline latency across Kafka, RabbitMQ, SQS, etc. +- `civisibility/` — CI Visibility trace interceptors and protocol adapters. Hooks into the trace + completion pipeline to filter and reformat test spans for the CI Test Cycle intake. +- `lambda/` — AWS Lambda support. Coordinates span creation with the serverless extension, + handling invocation start/end and trace context propagation. +- `llmobs/` — LLM Observability span mapper. Serializes LLM-specific spans (messages, tool calls) + to the dedicated LLM Obs intake format. + +### `dd-trace-api/` + +Public API. Types application developers may use directly: `Tracer`, `GlobalTracer`, `DDTags`, +`DDSpanTypes`, `Trace` (annotation), `ConfigDefaults`. Also houses all configuration key constants +by domain: `TracerConfig`, `GeneralConfig`, `AppSecConfig`, `ProfilingConfig`, `CiVisibilityConfig`, +`IastConfig`, `DebuggerConfig`, etc. + +### `internal-api/` + +Internal shared API across all agent modules (not public). Like `dd-trace-core`, grew organically +and now hosts interfaces for many products beyond tracing. New product APIs should go in +`products/` or `components/`. + +Core tracing abstractions: + +- `AgentTracer` — Static tracer facade. Instrumentations call `AgentTracer.startSpan()`, + `AgentTracer.activateSpan()`, etc. +- `AgentSpan` / `AgentScope` / `AgentSpanContext` — Internal span/scope/context interfaces. +- `AgentPropagation` — Context propagation interfaces (`Getter`, `Setter`) that instrumentations + implement to inject/extract trace context from framework-specific carriers (HTTP headers, message + properties, etc.). +- `Config` / `InstrumenterConfig` — Master configuration class and instrumenter-specific config, + centralizing settings for all products. `InstrumenterConfig` is separated from `Config` due to + GraalVM native-image constraints: in native-image builds, all bytecode instrumentation must be + applied at build time (ahead-of-time compilation), so configuration that controls instrumentation + decisions (which classes to instrument, which integrations to enable, resolver behavior, field + injection flags) must be frozen into the native image binary. Runtime-only settings (agent + endpoints, service names, sampling rates) remain in `Config`. + See [`docs/add_new_configurations.md`](docs/add_new_configurations.md). + +Cross-product abstractions: + +- `gateway/` — Instrumentation Gateway: event bus (`InstrumentationGateway`, + `SubscriptionService`, `Events`, `CallbackProvider`, `RequestContext`) decoupling + instrumentations from product modules. Primarily used by AppSec and IAST to hook into + the HTTP request lifecycle without modifying instrumentations. +- `cache/` — Shared caching primitives (`DDCache`, `FixedSizeCache`, `RadixTreeCache`) used + throughout the agent. +- `naming/` — Service and span operation naming schemas (v0, v1) for databases, messaging, + cloud services, etc. +- `telemetry/` — Multi-product telemetry collection interfaces (`MetricCollector`, + `WafMetricCollector`, `LLMObsMetricCollector`, etc.). + +Product-specific APIs that also live here: + +- `iast/` — IAST vulnerability detection interfaces: taint tracking (`Taintable`, `IastContext`), + sink definitions for each vulnerability type (SQL injection, XSS, command injection, etc.), + and call site instrumentation hooks. About 60 files. +- `civisibility/` — CI Visibility interfaces: test identification, code coverage, build/test + event handlers, and CI-specific telemetry metrics. About 95 files. +- `datastreams/` — Data Streams Monitoring interfaces: pathway context, stats points, + and schema registry integration. +- `appsec/` — AppSec interfaces: HTTP client request/response payloads for WAF analysis, + RASP call sites. +- `profiling/` — Profiler integration: recording data, timing, and enablement interfaces. +- `llmobs/` — LLM Observability context. + +### `components/` + +Low-level shared platform components. Not tied to any product, no external dependencies, +bootstrap-safe: + +- `context` — Immutable context propagation framework. Provides `Context`, `ContextKey`, + and `Propagator` abstractions for storing and propagating key-value pairs across threads + and carrier objects. +- `environment` — JVM and OS detection utilities. `JavaVersion` for version parsing, + `JavaVirtualMachine` for JVM implementation detection (OpenJDK, Graal, J9), + `OperatingSystem` for OS/architecture detection, and `EnvironmentVariables`/`SystemProperties` + for safe access and mocking. +- `json` — Lightweight, dependency-free JSON serialization. `JsonWriter` for building JSON + with a fluent API, `JsonReader` for streaming parsing. +- `native-loader` — Platform-aware native library loading with pluggable strategies. + `NativeLoader` handles OS/architecture detection, resource extraction from JARs, + and temp file management. + +### `products/` + +Self-contained product modules following a layered submodule pattern: + +- `{product}-api/` — Public API interfaces, zero dependencies. +- `{product}-bootstrap/` — Data classes safe for the bootstrap classloader. +- `{product}-lib/` — Core implementation (shadow jar, excludes shared dependencies). +- `{product}-agent/` — Agent integration entry point (shadow jar). + +Current products: + +- `metrics/` — StatsD client and monitoring abstraction. Provides `Monitoring` interface with + counters, timers, and histograms for internal agent metrics collection. +- `feature-flagging/` — Server-side feature flag evaluation driven by remote configuration. + Implements the OpenFeature SDK, handles the Unified Feature Control (UFC) protocol, + and tracks flag exposure per user/session. + +### `communication/` + +HTTP transport to the Datadog Agent and intake APIs. `SharedCommunicationObjects` holds shared +`OkHttpClient` instances (Unix domain socket and named pipe support), agent URL, feature discovery, +and the configuration poller. All product modules receive this at startup. + +### `remote-config/` + +Remote configuration client. `DefaultConfigurationPoller` periodically polls the Datadog Agent +for configuration updates (AppSec rules, debugger probes, sampling rates, feature flags). +Uses TUF (The Update Framework) for signature validation. + +### `telemetry/` + +Agent telemetry. `TelemetrySystem` collects and reports which features are enabled, +which integrations loaded, performance metrics, and product-specific counters. +Each product registers periodic actions that collect domain-specific metrics. + +### `utils/` + +Shared utilities, each in its own submodule: + +- `config-utils` — `ConfigProvider` for reading and merging configuration from environment variables, + system properties, properties files, and CI environment. +- `container-utils` — Parses container runtime information (Docker, Kubernetes, ECS). +- `filesystem-utils` — Permission-safe file existence checks that handle `SecurityException`. +- `flare-utils` — Tracer flare collection (`TracerFlareService`) that gathers diagnostics + (logs, spans, system info) and sends them to Datadog for troubleshooting. +- `queue-utils` — High-performance lock-free queues (`MpscArrayQueue`, `SpscArrayQueue`) for + inter-thread communication and span buffering. +- `socket-utils` — Socket factories (`UnixDomainSocketFactory`, `NamedPipeSocket`) for connecting + to the local Datadog Agent via Unix sockets or named pipes. +- `time-utils` — Time source abstractions (`TimeSource`, `ControllableTimeSource`) for testable + time handling and delay parsing. +- `version-utils` — Agent version string (`VersionInfo.VERSION`) read from packaged resources. +- `test-utils` — Testing utilities: `@Flaky` annotation, log capture, GC control, + forked test configuration. +- `test-agent-utils` — Message decoders for parsing v04/v05 binary protocol frames in tests. + +### `dd-trace-ot/` + +Legacy OpenTracing compatibility library. Publishes a standalone JAR artifact (`dd-trace-ot.jar`) +that implements the `io.opentracing.Tracer` interface by wrapping the Datadog `CoreTracer`. +This is a pure library for manual instrumentation only — there is no auto-instrumentation or +bytecode advice. + +### `dd-smoke-tests/` + +End-to-end smoke tests. Each boots a real application with the agent jar and verifies traces, spans, +and product behavior. Covers Spring Boot, Play, Vert.x, Quarkus, WildFly, and more. +Core test hierarchy (Groovy/Spock): +- `ProcessManager` — Base. Spawns forked JVM processes with the agent via `ProcessBuilder`, + captures stdout to log files, tears down on cleanup. `assertNoErrorLogs()` scans logs for errors. +- `AbstractSmokeTest` extends `ProcessManager` — Adds a mock Datadog Agent (`TestHttpServer`) + receiving traces (v0.4/v0.5), telemetry, remote config, and EVP proxy requests. Polling helpers: + `waitForTraceCount`, `waitForSpan`, `waitForTelemetryFlat`. +- `AbstractServerSmokeTest` extends `AbstractSmokeTest` — For HTTP server apps. Adds port + management, waits for server port to open, verifies expected trace output. diff --git a/BUILDING.md b/BUILDING.md index 6fc6ed031d2..5a557933b54 100644 --- a/BUILDING.md +++ b/BUILDING.md @@ -5,7 +5,7 @@ This documentation provides information for developers to set up their environme * [Development environment](#development-environment) * [Quick check](#quick-check) * [Requirements](#requirements) - * [Install the required JDKs](#install-the-required-jdks) + * [Install JDK](#install-jdk) * [Install git](#install-git) * [Install Docker Desktop](#install-docker-desktop) * [Configure Akka Token](#configure-akka-token) @@ -16,7 +16,7 @@ This documentation provides information for developers to set up their environme ### Quick check -To check that your development environment is properly set up to build the project, from the project root run on macOS or Linux: +To check that your development environment is properly set up to build the project, run the following command from the project root on macOS or Linux: ```shell ./setup.sh ``` @@ -29,14 +29,15 @@ or on Windows: Your output should look something like the following: ``` -ℹ️ Checking required JVMs: -✅ JAVA_HOME is set to /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home. -✅ JAVA_8_HOME is set to /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home. -✅ JAVA_11_HOME is set to /Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home. -✅ JAVA_17_HOME is set to /Library/Java/JavaVirtualMachines/zulu-17.jdk/Contents/Home. -✅ JAVA_21_HOME is set to /Library/Java/JavaVirtualMachines/zulu-21.jdk/Contents/Home. -✅ JAVA_25_HOME is set to /Library/Java/JavaVirtualMachines/zulu-25.jdk/Contents/Home. -✅ JAVA_GRAALVM17_HOME is set to /Library/Java/JavaVirtualMachines/graalvm-ce-java17-22.3.1/Contents/Home. +ℹ️ Checking required JVM: +✅ JAVA_HOME is set to /Library/Java/JavaVirtualMachines/zulu-21.jdk/Contents/Home. +ℹ️ Checking other JVMs available for testing: +✅ Azul Zulu JDK 1.8.0_462-b08 from /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home. +✅ Azul Zulu JDK 11.0.28+6-LTS from /Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home. +✅ Azul Zulu JDK 17.0.16+8-LTS from /Library/Java/JavaVirtualMachines/zulu-17.jdk/Contents/Home. +✅ Azul Zulu JDK 21.0.8+9-LTS from /Library/Java/JavaVirtualMachines/zulu-21.jdk/Contents/Home. +✅ Azul Zulu JDK 25+36-LTS from /Library/Java/JavaVirtualMachines/zulu-25.jdk/Contents/Home. +✅ GraalVM Community JDK 17.0.9+9-jvmci-23.0-b22 from /Library/Java/JavaVirtualMachines/graalvm-ce-java17-22.3.1/Contents/Home. ℹ️ Checking git configuration: ✅ The git command line is installed. ✅ pre-commit hook is installed in repository. @@ -47,112 +48,56 @@ Your output should look something like the following: ✅ The Docker server is running. ``` -If there is any issue with your output, check the requirements above and use the following guide to install and configure the required tools. +If there is any issue with your output, check the requirements below and use the following guide to install and configure the required tools. ### Requirements Requirements to build the full project: -* The JDK versions 8, 11, 17, 21, and 25 must be installed. -* The `JAVA_8_HOME`, `JAVA_11_HOME`, `JAVA_17_HOME`, `JAVA_21_HOME`, `JAVA_25_HOME`, and `JAVA_GRAALVM17_HOME` must point to their respective JDK location. -* The JDK 8 `bin` directory must be the only JDK on the PATH (e.g. `$JAVA_8_HOME/bin`). -* The `JAVA_HOME` environment variable may be unset. If set, it must point to the JDK 8 location (same as `JAVA_8_HOME`). -* The `git` command line must be installed. -* A container runtime environment must be available to run all tests (e.g. Docker Desktop). +* JDK version is 21+, +* The `git` command line is installed, +* A container runtime environment is available to run all tests (e.g. Docker Desktop). -### Install the required JDKs +### Install JDK -Download and install JDK versions 8, 11, 17, 21 and 25, and GraalVM 17 for your OS. +Java is required to run Gradle, the project build tool. +Gradle will find any locally installed JDK and download any missing JDK versions needed for the project build and testing. #### macOS -* Install the required JDKs using `brew`: - ```shell - brew install --cask zulu@8 zulu@11 zulu@17 zulu@21 zulu graalvm/tap/graalvm-ce-java17 - ``` -* Identify your local version of GraalVM: - ``` - ls /Library/Java/JavaVirtualMachines | grep graalvm - ``` - Example: `graalvm-ce-java17-22.3.1` -* Use this version in the following command to fix the GraalVM installation by [removing the quarantine flag](https://www.graalvm.org/latest/docs/getting-started/macos/): - ``` - sudo xattr -r -d com.apple.quarantine /Library/Java/JavaVirtualMachines/graalvm- - ``` - Example: `/Library/Java/JavaVirtualMachines/graalvm-ce-java17-22.3.1` -* Add the required environment variables to your shell using the `export` command. You can permanently install the environment variables by appending the `export` commands into your shell configuration file `~/.zshrc` or `.bashrc` or other. - ```shell - export JAVA_8_HOME=/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home - export JAVA_11_HOME=/Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home - export JAVA_17_HOME=/Library/Java/JavaVirtualMachines/zulu-17.jdk/Contents/Home - export JAVA_21_HOME=/Library/Java/JavaVirtualMachines/zulu-21.jdk/Contents/Home - export JAVA_25_HOME=/Library/Java/JavaVirtualMachines/zulu-25.jdk/Contents/Home - export JAVA_GRAALVM17_HOME=/Library/Java/JavaVirtualMachines/graalvm-/Contents/Home - export JAVA_HOME=$JAVA_8_HOME - ``` -* Restart your shell after applying the changes if you appended the commands to your shell configuration file. - -> [!NOTE] -> ARM users: there is no Oracle JDK v8 for ARM. -> It's recommended to use [Azul's Zulu](https://www.azul.com/downloads/?version=java-8-lts&architecture=arm-64-bit&package=jdk#zulu) builds of Java 8. -> [Amazon Corretto](https://aws.amazon.com/corretto/) builds have also been proven to work. - -> [!NOTE] -> macOS users: remember that `/usr/libexec/java_home` may control which JDK is in your path. +Install JDK 21 using `brew`: +```shell +brew install --cask zulu@21 +``` #### Linux -* Download and extract JDK 8, 11, 17, 21, and 25 from [Eclipse Temurin releases](https://adoptium.net/temurin/releases/) and GraalVM 17 from [Oracle downloads](https://www.graalvm.org/downloads/). -* Install the GraalVM native image requirements for native builds by following [the GraalVM official documentation](https://www.graalvm.org/latest/reference-manual/native-image/#prerequisites). -* Add the required environment variables to your shell using the `export` command. You can permanently install the environment variables by appending the `export` commands into your shell configuration file `~/.zshrc` or `~/.bashrc` or other. - ```shell - export JAVA_8_HOME=//jdk8u - export JAVA_11_HOME=//jdk-11. - export JAVA_17_HOME=//jdk-17. - export JAVA_21_HOME=//jdk-21. - export JAVA_25_HOME=//jdk-25. - export JAVA_GRAALVM17_HOME=//graalvm-jdk-17./Contents/Home - export JAVA_HOME=$JAVA_8_HOME - ``` -* Restart your shell after applying the changes if you appended the commands to your shell configuration file. - -#### Windows - -* Download and install JDK 8, 11, 17, 21, and 25 [Eclipse Temurin releases](https://adoptium.net/temurin/releases/). - -
- Alternatively, install JDKs using winget or scoop. (click here to expand) +Use your distribution package manager to install JDK 21: +```shell +apt install openjdk-21-jdk +``` +Alternatively, manually download and install from [Eclipse Temurin releases](https://adoptium.net/temurin/releases/). - ```pwsh - winget install --id EclipseAdoptium.Temurin.8.JDK - winget install --id EclipseAdoptium.Temurin.11.JDK - winget install --id EclipseAdoptium.Temurin.17.JDK - winget install --id EclipseAdoptium.Temurin.21.JDK - winget install --id EclipseAdoptium.Temurin.25.JDK - ``` +Add the `JAVA_HOME` environment variable to your shell using the `export` command. +You can permanently set it by appending the `export` command to your shell configuration file such as `~/.zshrc`, `~/.bashrc` or similar. +```shell +export JAVA_HOME=//jdk-21.x.x +``` +If you appended the commands to your shell configuration file, restart your shell after applying the changes. - ```pwsh - scoop bucket add java - scoop install temurin8-jdk - scoop install temurin11-jdk - scoop install temurin17-jdk - scoop install temurin21-jdk - scoop install temurin25-jdk - ``` +#### Windows -
+Install JDK 21 using the Windows package manager `winget`: +```pwsh +winget install --id EclipseAdoptium.Temurin.21.JDK +``` +Or manually download and install it from [Eclipse Temurin releases](https://adoptium.net/temurin/releases/). -* To add the required environment variables, run the following PowerShell commands for each SDK version, replacing the paths with the correct version installed: - ```pwsh - [Environment]::SetEnvironmentVariable("JAVA_8_HOME", "C:\Program Files\Eclipse Adoptium\jdk-8.0.432.6-hotspot", [EnvironmentVariableTarget]::User) - [Environment]::SetEnvironmentVariable("JAVA_11_HOME", "C:\Program Files\Eclipse Adoptium\jdk-11.0.25.9-hotspot", [EnvironmentVariableTarget]::User) - [Environment]::SetEnvironmentVariable("JAVA_17_HOME", "C:\Program Files\Eclipse Adoptium\jdk-17.0.12.7-hotspot", [EnvironmentVariableTarget]::User) - [Environment]::SetEnvironmentVariable("JAVA_21_HOME", "C:\Program Files\Eclipse Adoptium\jdk-21.0.5.11-hotspot", [EnvironmentVariableTarget]::User) - [Environment]::SetEnvironmentVariable("JAVA_25_HOME", "C:\Program Files\Eclipse Adoptium\jdk-25.0.1.9-hotspot", [EnvironmentVariableTarget]::User) +Set the `JAVA_HOME` environment variable, replacing the path with your JDK 21 installation: +```pwsh +[Environment]::SetEnvironmentVariable("JAVA_HOME", "C:\Program Files\Eclipse Adoptium\jdk-21.x.x-hotspot", [EnvironmentVariableTarget]::User) +``` - # JAVA_HOME = JAVA_8_HOME - [Environment]::SetEnvironmentVariable("JAVA_HOME", "C:\Program Files\Eclipse Adoptium\jdk-8.0.432.6-hotspot", [EnvironmentVariableTarget]::User) - ``` ### Install git @@ -164,26 +109,17 @@ If not installed, the terminal will prompt you to install it. #### Linux ```shell -apt-get install git +apt install git ``` #### Windows -Download and install the installer from [the official website](https://git-scm.com/download/win). - -
-Alternatively, install git using winget or scoop. (click here to expand) +Download and install the installer from [the official website](https://git-scm.com/download/win), or install it using the Windows package manager `winget`: ```pwsh winget install --id git.git ``` -```pwsh -scoop install git -``` - -
- ### Install Docker Desktop > [!NOTE] @@ -230,8 +166,8 @@ winget install --id Docker.DockerDesktop > You can alternatively use the `core.hooksPath` configuration to point to the `.githooks` folder using `git config --local core.hooksPath .githooks` if you don't already have a hooks path defined system-wide. > [!NOTE] -> The git hooks will check that your code is properly formatted before commiting. -> This is done both to avoid future merge conflict and ensure uniformity inside the code base. +> The git hooks will check that your code is properly formatted before committing. +> This is done both to avoid future merge conflicts and ensure uniformity across the code base. * Configure git to automatically update submodules. ```shell @@ -252,18 +188,18 @@ winget install --id Docker.DockerDesktop ### Configure Akka Token > [!NOTE] > You can skip this step if you don’t need instrumentation for the **akka-http-10.6** module. -> For background on why Akka now requires authentication, see this [article](https://akka.io/blog/why-we-are-changing-the-license-for-akka). +> For background on why Akka now requires authentication, see [this article](https://akka.io/blog/why-we-are-changing-the-license-for-akka). To enable access to Akka artifacts hosted on Lightbend’s private repository, you’ll need to configure an authentication token. -1. Obtain a repository token. Visit the Akka account [page](https://account.akka.io/token) to generate a secure repository token. -2. Set up the environment variable. Create an environment variable named: +1. Obtain a repository token by visiting the Akka account [page](https://account.akka.io/token) to generate a secure repository token. +2. Create an environment variable named: ```shell ORG_GRADLE_PROJECT_akkaRepositoryToken= ``` ## Building the project -After everything is properly set up, you can move on to the next section to start a build or check [the contribution guidelines](CONTRIBUTING.md). +After everything is properly set up, you can build the project or check [the contribution guidelines](CONTRIBUTING.md). To build the project without running tests, run: ```shell diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 85698bf5ffe..1399caf947a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -50,10 +50,6 @@ To run the formatting verify task only: For IntelliJ IDEA, we suggest the following settings and plugin. -The default JVM to build and run tests from the command line should be Java 8. - -* Use Java 8 to build and run tests: - * `Project Structure` -> `Project` -> `SDK` -> `Download JDK...` -> `Version: 1.8` -> `Download` * Configure Java and Groovy import formatting: * `Settings...` ->`Editor` > `Code Style` > `Java` > `Imports` * `Use single class import`: checked @@ -64,6 +60,7 @@ The default JVM to build and run tests from the command line should be Java 8. * top right Settings icon -> `Settings...` ->`Editor` > `Code Style` > `Groovy` > `Imports` * `Class count to use import with '*'`: `9999` (some number sufficiently large that is unlikely to matter) * `Names count to use static import with '*'`: `9999` +* To run test in a specific JDK use the `testJvm` property, e.g. `-PtestJvm=11` * Install the [Google Java Format](https://plugins.jetbrains.com/plugin/8527-google-java-format) plugin ### Troubleshooting @@ -82,11 +79,6 @@ The default JVM to build and run tests from the command line should be Java 8. * IntelliJ 2021.3 complains `Failed to find KotlinGradleProjectData for GradleSourceSetData` https://youtrack.jetbrains.com/issue/KTIJ-20173. * Switch to `IntelliJ IDEA CE 2021.2.3`. - -* IntelliJ Gradle fails to import the project with `JAVA_11_HOME must be set to build Java 11 code`. - * A workaround is to run IntelliJ from your terminal with `JAVA_11_HOME`. - * In order to verify what's visible from IntelliJ, use the `Add Configuration` bar and go - to `Add New` -> `Gradle` -> `Environmental Variables`. ## Pull request guidelines diff --git a/README.md b/README.md index f98ef7ca129..4c9e32ff166 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,7 @@ Take a look at the [APM Glossary][visualization docs]. Before contributing to the project, please take a moment to read our brief [Contribution Guidelines](CONTRIBUTING.md). Then check our guides: -* [How to setup a development and build the project](BUILDING.md), +* [How to set up a development environment and build the project](BUILDING.md), * [How to create a new instrumentation](docs/add_new_instrumentation.md), * [How to test](docs/how_to_test.md), @@ -43,6 +43,6 @@ Or our reference documents: * [How instrumentations work](docs/how_instrumentations_work.md). ## Releases -Datadog will generally create a new minor release the first full week of every month. +Datadog will generally release a new minor version during the first full week of every month. See [release.md](docs/releases.md) for more information. diff --git a/build.gradle.kts b/build.gradle.kts index abcd900cd3f..38169896069 100644 --- a/build.gradle.kts +++ b/build.gradle.kts @@ -2,20 +2,23 @@ import com.diffplug.gradle.spotless.SpotlessExtension import datadog.gradle.plugin.ci.testAggregate plugins { - id("datadog.gradle-debug") - id("datadog.dependency-locking") - id("datadog.tracer-version") - id("datadog.dump-hanged-test") - id("datadog.ci-jobs") - - id("com.diffplug.spotless") version "6.13.0" - id("com.github.spotbugs") version "5.0.14" - id("de.thetaphi.forbiddenapis") version "3.8" + kotlin("jvm") version libs.versions.kotlin.plugin apply false + + id("dd-trace-java.gradle-debug") + id("dd-trace-java.dependency-locking") + id("dd-trace-java.tracer-version") + id("dd-trace-java.dump-hanged-test") + id("dd-trace-java.config-inversion-linter") + id("dd-trace-java.ci-jobs") + + id("com.diffplug.spotless") version "8.4.0" + id("me.champeau.gradle.japicmp") version "0.4.3" + id("com.github.spotbugs") version "6.5.0" + id("de.thetaphi.forbiddenapis") version "3.10" id("io.github.gradle-nexus.publish-plugin") version "2.0.0" - id("com.gradleup.shadow") version "8.3.6" apply false + id("com.gradleup.shadow") version "8.3.9" apply false id("me.champeau.jmh") version "0.7.3" apply false - id("org.gradle.playframework") version "0.13" apply false - kotlin("jvm") version libs.versions.kotlin.plugin apply false + id("org.gradle.playframework") version "0.16.0" apply false } description = "dd-trace-java" @@ -23,6 +26,7 @@ description = "dd-trace-java" val isCI = providers.environmentVariable("CI") apply(from = rootDir.resolve("gradle/repositories.gradle")) +apply(from = rootDir.resolve("gradle/ddprof-override.gradle")) spotless { // only resolve the spotless dependencies once in the build @@ -34,8 +38,8 @@ with(extensions["spotlessPredeclare"] as SpotlessExtension) { java { removeUnusedImports() - // This is the last Google Java Format version that supports Java 8 - googleJavaFormat("1.7") + googleJavaFormat("1.35.0") + tableTestFormatter("1.1.1") } groovyGradle { greclipse() @@ -44,13 +48,14 @@ with(extensions["spotlessPredeclare"] as SpotlessExtension) { greclipse() } kotlinGradle { - ktlint("0.41.0") + ktlint("1.8.0") } kotlin { - ktlint("0.41.0") + ktlint("1.8.0") } scala { - scalafmt("2.7.5") + // TODO: For some reason Scala format is working correctly with this version only. + scalafmt("3.8.6") } } apply(from = rootDir.resolve("gradle/spotless.gradle")) @@ -144,7 +149,8 @@ testAggregate("instrumentation", listOf(":dd-java-agent:instrumentation"), empty testAggregate("profiling", listOf(":dd-java-agent:agent-profiling"), emptyList()) testAggregate("debugger", listOf(":dd-java-agent:agent-debugger"), forceCoverage = true) testAggregate( - "base", listOf(":"), + "base", + listOf(":"), listOf( ":dd-java-agent:instrumentation", ":dd-smoke-tests", @@ -152,3 +158,29 @@ testAggregate( ":dd-java-agent:agent-debugger" ) ) + +// JApiCmp configuration example +// Usage: ./gradlew japicmp -Partifact=groupId:artifactId -Pbaseline=1.0.0 -Ptarget=2.0.0 +tasks.register("japicmp") { + val artifact = providers.gradleProperty("artifact").orNull + val baseline = providers.gradleProperty("baseline").orNull + val target = providers.gradleProperty("target").orNull + + if (artifact != null && baseline != null && target != null) { + oldClasspath.from( + configurations.detachedConfiguration( + dependencies.create("$artifact:$baseline") + ) + ) + newClasspath.from( + configurations.detachedConfiguration( + dependencies.create("$artifact:$target") + ) + ) + onlyModified.set(true) + failOnModification.set(false) + ignoreMissingClasses.set(true) + txtOutputFile.set(layout.buildDirectory.file("reports/japicmp.txt")) + htmlOutputFile.set(layout.buildDirectory.file("reports/japicmp.html")) + } +} diff --git a/buildSrc/build.gradle.kts b/buildSrc/build.gradle.kts index d93c261886a..8083b423297 100644 --- a/buildSrc/build.gradle.kts +++ b/buildSrc/build.gradle.kts @@ -1,47 +1,63 @@ plugins { - groovy `java-gradle-plugin` `kotlin-dsl` `jvm-test-suite` - id("com.diffplug.spotless") version "6.13.0" + id("com.diffplug.spotless") version "8.4.0" } +// The buildSrc still needs to target Java 8 as build time instrumentation and muzzle plugin +// allow to schedule workers on different JDK version. java { - toolchain { - languageVersion = JavaLanguageVersion.of(8) + sourceCompatibility = JavaVersion.VERSION_1_8 + targetCompatibility = JavaVersion.VERSION_1_8 +} + +kotlin { + compilerOptions { + jvmTarget.set(org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_1_8) } } gradlePlugin { plugins { create("instrument-plugin") { - id = "instrument" - implementationClass = "InstrumentPlugin" + id = "dd-trace-java.build-time-instrumentation" + implementationClass = "datadog.gradle.plugin.instrument.BuildTimeInstrumentationPlugin" } + create("muzzle-plugin") { - id = "muzzle" + id = "dd-trace-java.muzzle" implementationClass = "datadog.gradle.plugin.muzzle.MuzzlePlugin" } create("call-site-instrumentation-plugin") { - id = "call-site-instrumentation" - implementationClass = "datadog.gradle.plugin.CallSiteInstrumentationPlugin" + id = "dd-trace-java.call-site-instrumentation" + implementationClass = "datadog.gradle.plugin.csi.CallSiteInstrumentationPlugin" } + create("tracer-version-plugin") { - id = "datadog.tracer-version" + id = "dd-trace-java.tracer-version" implementationClass = "datadog.gradle.plugin.version.TracerVersionPlugin" } + create("dump-hanged-test-plugin") { - id = "datadog.dump-hanged-test" + id = "dd-trace-java.dump-hanged-test" implementationClass = "datadog.gradle.plugin.dump.DumpHangedTestPlugin" } + create("supported-config-generation") { - id = "supported-config-generator" + id = "dd-trace-java.supported-config-generator" implementationClass = "datadog.gradle.plugin.config.SupportedConfigPlugin" } + create("supported-config-linter") { - id = "config-inversion-linter" + id = "dd-trace-java.config-inversion-linter" implementationClass = "datadog.gradle.plugin.config.ConfigInversionLinter" } + + create("instrumentation-naming") { + id = "dd-trace-java.instrumentation-naming" + implementationClass = "datadog.gradle.plugin.naming.InstrumentationNamingPlugin" + } } } @@ -49,17 +65,22 @@ apply { from("$rootDir/../gradle/repositories.gradle") } +repositories { + gradlePluginPortal() +} + dependencies { implementation(gradleApi()) - implementation(localGroovy()) - implementation("net.bytebuddy", "byte-buddy-gradle-plugin", "1.17.7") + implementation("net.bytebuddy", "byte-buddy-gradle-plugin", "1.18.8") implementation("org.eclipse.aether", "aether-connector-basic", "1.1.0") implementation("org.eclipse.aether", "aether-transport-http", "1.1.0") + implementation("org.eclipse.aether", "aether-transport-file", "1.1.0") implementation("org.apache.maven", "maven-aether-provider", "3.3.9") implementation("com.github.zafarkhaja:java-semver:0.10.2") + implementation("com.github.javaparser", "javaparser-symbol-solver-core", "3.24.4") implementation("com.google.guava", "guava", "20.0") implementation(libs.asm) @@ -69,6 +90,8 @@ dependencies { implementation("com.fasterxml.jackson.core:jackson-databind") implementation("com.fasterxml.jackson.core:jackson-annotations") implementation("com.fasterxml.jackson.core:jackson-core") + + compileOnly(libs.develocity) } tasks.compileKotlin { @@ -80,24 +103,15 @@ testing { suites { val test by getting(JvmTestSuite::class) { dependencies { - implementation(libs.spock.core) - implementation(libs.groovy) + implementation(libs.assertj.core) } targets.configureEach { testTask.configure { - enabled = project.hasProperty("runBuildSrcTests") + enabled = providers.gradleProperty("runBuildSrcTests").isPresent or providers.systemProperty("idea.active").isPresent } } } - val integTest by registering(JvmTestSuite::class) { - dependencies { - implementation(gradleTestKit()) - } - // Makes the gradle plugin publish its declared plugins to this source set - gradlePlugin.testSourceSet(sources) - } - withType(JvmTestSuite::class).configureEach { useJUnitJupiter(libs.versions.junit5) targets.configureEach { diff --git a/buildSrc/call-site-instrumentation-plugin/build.gradle.kts b/buildSrc/call-site-instrumentation-plugin/build.gradle.kts index 1987bd5f91d..562ab980e91 100644 --- a/buildSrc/call-site-instrumentation-plugin/build.gradle.kts +++ b/buildSrc/call-site-instrumentation-plugin/build.gradle.kts @@ -1,8 +1,7 @@ plugins { java - groovy - id("com.diffplug.spotless") version "6.13.0" - id("com.gradleup.shadow") version "8.3.6" + id("com.diffplug.spotless") version "8.4.0" + id("com.gradleup.shadow") version "8.3.9" } java { @@ -17,8 +16,7 @@ spotless { target("src/**/*.java") // ignore embedded test projects targetExclude("src/test/resources/**") - // This is the last Google Java Format version that supports Java 8 - googleJavaFormat("1.7") + googleJavaFormat("1.35.0") } } @@ -32,14 +30,14 @@ dependencies { implementation("org.freemarker", "freemarker", "2.3.30") implementation(libs.asm) implementation(libs.asm.tree) - implementation("com.github.javaparser", "javaparser-symbol-solver-core", "3.24.4") + implementation(libs.javaparser.symbol.solver) testImplementation(libs.bytebuddy) - testImplementation(libs.spock.core) - testImplementation("org.objenesis", "objenesis", "3.0.1") - testImplementation(libs.groovy) + testImplementation(libs.bundles.junit5) + testRuntimeOnly(libs.junit.platform.launcher) + testImplementation(libs.bundles.mockito) testImplementation("javax.servlet", "javax.servlet-api", "3.0.1") - testImplementation("com.github.spotbugs", "spotbugs-annotations", "4.2.0") + testImplementation(libs.spotbugs.annotations) } sourceSets { diff --git a/buildSrc/call-site-instrumentation-plugin/src/main/java/datadog/trace/plugin/csi/impl/AdviceGeneratorImpl.java b/buildSrc/call-site-instrumentation-plugin/src/main/java/datadog/trace/plugin/csi/impl/AdviceGeneratorImpl.java index 14efc4a42d4..d517834c3a6 100644 --- a/buildSrc/call-site-instrumentation-plugin/src/main/java/datadog/trace/plugin/csi/impl/AdviceGeneratorImpl.java +++ b/buildSrc/call-site-instrumentation-plugin/src/main/java/datadog/trace/plugin/csi/impl/AdviceGeneratorImpl.java @@ -266,7 +266,6 @@ private static void writeStackOperations(final AdviceSpecification advice, final final List parameterIndicesValues = advice .getArguments() - .sorted() .map(argSpec -> intLiteral(argSpec.getIndex())) .collect(Collectors.toList()); final VariableDeclarator parameterIndices = diff --git a/buildSrc/call-site-instrumentation-plugin/src/main/java/datadog/trace/plugin/csi/util/CallSiteUtils.java b/buildSrc/call-site-instrumentation-plugin/src/main/java/datadog/trace/plugin/csi/util/CallSiteUtils.java index 5f2e2fa8d75..1f9ecdc5aad 100644 --- a/buildSrc/call-site-instrumentation-plugin/src/main/java/datadog/trace/plugin/csi/util/CallSiteUtils.java +++ b/buildSrc/call-site-instrumentation-plugin/src/main/java/datadog/trace/plugin/csi/util/CallSiteUtils.java @@ -3,6 +3,7 @@ import java.io.File; import java.net.MalformedURLException; import java.net.URL; +import java.nio.file.Files; import java.nio.file.Path; import java.util.stream.Collectors; import java.util.stream.IntStream; @@ -47,7 +48,20 @@ public static String repeat(final char value, int count) { public static URL toURL(final Path path) { try { - return path.toUri().toURL(); + URL url = path.toUri().toURL(); + // URLClassLoader interprets URLs ending with '/' as directories. If the trailing '/' is + // missing, a directory URL is treated as a JAR. If the path does yet exist on disk + // assumes that paths not ending with ".jar" are directories. + boolean shouldAddSlash = + Files.exists(path) ? Files.isDirectory(path) : !path.toString().endsWith(".jar"); + + if (shouldAddSlash) { + String urlString = url.toString(); + if (!urlString.endsWith("/")) { + url = new URL(urlString + "/"); + } + } + return url; } catch (MalformedURLException e) { throw new RuntimeException(e); } diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/AdviceGeneratorTest.groovy b/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/AdviceGeneratorTest.groovy deleted file mode 100644 index 2958c292cac..00000000000 --- a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/AdviceGeneratorTest.groovy +++ /dev/null @@ -1,518 +0,0 @@ -package datadog.trace.plugin.csi.impl - - -import datadog.trace.agent.tooling.csi.CallSite -import datadog.trace.agent.tooling.csi.CallSites -import datadog.trace.plugin.csi.AdviceGenerator -import datadog.trace.plugin.csi.impl.assertion.AssertBuilder -import datadog.trace.plugin.csi.impl.assertion.CallSiteAssert -import datadog.trace.plugin.csi.impl.ext.tests.IastCallSites -import datadog.trace.plugin.csi.impl.ext.tests.RaspCallSites -import groovy.transform.CompileDynamic -import spock.lang.Requires -import spock.lang.TempDir - -import javax.servlet.ServletRequest -import java.lang.invoke.MethodHandles -import java.lang.invoke.MethodType - -import static CallSiteFactory.pointcutParser - -@CompileDynamic -final class AdviceGeneratorTest extends BaseCsiPluginTest { - - @TempDir - private File buildDir - - @CallSite(spi = CallSites) - class BeforeAdvice { - @CallSite.Before('java.security.MessageDigest java.security.MessageDigest.getInstance(java.lang.String)') - static void before(@CallSite.Argument final String algorithm) {} - } - - void 'test before advice'() { - setup: - final spec = buildClassSpecification(BeforeAdvice) - final generator = buildAdviceGenerator(buildDir) - - when: - final result = generator.generate(spec) - - then: - assertNoErrors(result) - assertCallSites(result.file) { - interfaces(CallSites) - helpers(BeforeAdvice) - advices(0) { - type("BEFORE") - pointcut('java/security/MessageDigest', 'getInstance', '(Ljava/lang/String;)Ljava/security/MessageDigest;') - statements( - 'handler.dupParameters(descriptor, StackDupMode.COPY);', - 'handler.advice("datadog/trace/plugin/csi/impl/AdviceGeneratorTest$BeforeAdvice", "before", "(Ljava/lang/String;)V");', - 'handler.method(opcode, owner, name, descriptor, isInterface);' - ) - } - } - } - - @CallSite(spi = CallSites) - class AroundAdvice { - @CallSite.Around('java.lang.String java.lang.String.replaceAll(java.lang.String, java.lang.String)') - static String around(@CallSite.This final String self, @CallSite.Argument final String regexp, @CallSite.Argument final String replacement) { - return self.replaceAll(regexp, replacement); - } - } - - void 'test around advice'() { - setup: - final spec = buildClassSpecification(AroundAdvice) - final generator = buildAdviceGenerator(buildDir) - - when: - final result = generator.generate(spec) - - then: - assertNoErrors(result) - assertCallSites(result.file) { - interfaces(CallSites) - helpers(AroundAdvice) - advices(0) { - type("AROUND") - pointcut('java/lang/String', 'replaceAll', '(Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;') - statements( - 'handler.advice("datadog/trace/plugin/csi/impl/AdviceGeneratorTest$AroundAdvice", "around", "(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;");' - ) - } - } - } - - @CallSite(spi = CallSites) - class AfterAdvice { - @CallSite.After('java.lang.String java.lang.String.concat(java.lang.String)') - static String after(@CallSite.This final String self, @CallSite.Argument final String param, @CallSite.Return final String result) { - return result - } - } - - void 'test after advice'() { - setup: - final spec = buildClassSpecification(AfterAdvice) - final generator = buildAdviceGenerator(buildDir) - - when: - final result = generator.generate(spec) - - then: - assertNoErrors(result) - assertCallSites(result.file) { - interfaces(CallSites) - helpers(AfterAdvice) - advices(0) { - type("AFTER") - pointcut('java/lang/String', 'concat', '(Ljava/lang/String;)Ljava/lang/String;') - statements( - 'handler.dupInvoke(owner, descriptor, StackDupMode.COPY);', - 'handler.method(opcode, owner, name, descriptor, isInterface);', - 'handler.advice("datadog/trace/plugin/csi/impl/AdviceGeneratorTest$AfterAdvice", "after", "(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;");', - ) - } - } - } - - @CallSite(spi = CallSites) - class AfterAdviceCtor { - @CallSite.After('void java.net.URL.(java.lang.String)') - static URL after(@CallSite.AllArguments final Object[] args, @CallSite.Return final URL url) { - return url - } - } - - void 'test after advice ctor'() { - setup: - final spec = buildClassSpecification(AfterAdviceCtor) - final generator = buildAdviceGenerator(buildDir) - - when: - final result = generator.generate(spec) - - then: - assertNoErrors(result) - assertCallSites(result.file) { - interfaces(CallSites) - helpers(AfterAdviceCtor) - advices(0) { - pointcut('java/net/URL', '', '(Ljava/lang/String;)V') - statements( - 'handler.dupParameters(descriptor, StackDupMode.PREPEND_ARRAY_CTOR);', - 'handler.method(opcode, owner, name, descriptor, isInterface);', - 'handler.advice("datadog/trace/plugin/csi/impl/AdviceGeneratorTest$AfterAdviceCtor", "after", "([Ljava/lang/Object;Ljava/net/URL;)Ljava/net/URL;");', - ) - } - } - } - - @CallSite(spi = SampleSpi.class) - class SpiAdvice { - @CallSite.Before('java.security.MessageDigest java.security.MessageDigest.getInstance(java.lang.String)') - static void before(@CallSite.Argument final String algorithm) {} - - interface SampleSpi {} - } - - void 'test generator with spi'() { - setup: - final spec = buildClassSpecification(SpiAdvice) - final generator = buildAdviceGenerator(buildDir) - - when: - final result = generator.generate(spec) - - then: - assertNoErrors(result) - - assertCallSites(result.file) { - interfaces(CallSites, SpiAdvice.SampleSpi) - } - } - - @CallSite(spi = CallSites) - class InvokeDynamicAfterAdvice { - @CallSite.After( - value = 'java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])', - invokeDynamic = true - ) - static String after(@CallSite.AllArguments final Object[] arguments, @CallSite.Return final String result) { - result - } - } - - @Requires({ - jvm.java9Compatible - }) - void 'test invoke dynamic after advice'() { - setup: - final spec = buildClassSpecification(InvokeDynamicAfterAdvice) - final generator = buildAdviceGenerator(buildDir) - - when: - final result = generator.generate(spec) - - then: - assertNoErrors(result) - assertCallSites(result.file) { - interfaces(CallSites) - helpers(InvokeDynamicAfterAdvice) - advices(0) { - pointcut( - 'java/lang/invoke/StringConcatFactory', - 'makeConcatWithConstants', - '(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/invoke/CallSite;' - ) - statements( - 'handler.dupParameters(descriptor, StackDupMode.PREPEND_ARRAY);', - 'handler.invokeDynamic(name, descriptor, bootstrapMethodHandle, bootstrapMethodArguments);', - 'handler.advice("datadog/trace/plugin/csi/impl/AdviceGeneratorTest$InvokeDynamicAfterAdvice", "after", "([Ljava/lang/Object;Ljava/lang/String;)Ljava/lang/String;");' - ) - } - } - } - - @CallSite(spi = CallSites) - class InvokeDynamicAroundAdvice { - @CallSite.Around( - value = 'java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])', - invokeDynamic = true - ) - static java.lang.invoke.CallSite around(@CallSite.Argument final MethodHandles.Lookup lookup, - @CallSite.Argument final String name, - @CallSite.Argument final MethodType concatType, - @CallSite.Argument final String recipe, - @CallSite.Argument final Object... constants) { - return null; - } - } - - @Requires({ - jvm.java9Compatible - }) - void 'test invoke dynamic around advice'() { - setup: - final spec = buildClassSpecification(InvokeDynamicAroundAdvice) - final generator = buildAdviceGenerator(buildDir) - - when: - final result = generator.generate(spec) - - then: - assertNoErrors(result) - assertCallSites(result.file) { - interfaces(CallSites) - helpers(InvokeDynamicAroundAdvice) - advices(0) { - pointcut( - 'java/lang/invoke/StringConcatFactory', - 'makeConcatWithConstants', - '(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/invoke/CallSite;' - ) - statements( - 'handler.invokeDynamic(name, descriptor, new Handle(Opcodes.H_INVOKESTATIC, "datadog/trace/plugin/csi/impl/AdviceGeneratorTest$InvokeDynamicAroundAdvice", "around", "(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/invoke/CallSite;", false), bootstrapMethodArguments);', - ) - } - } - } - - @CallSite(spi = CallSites) - class InvokeDynamicWithConstantsAdvice { - @CallSite.After( - value = 'java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])', - invokeDynamic = true - ) - static String after(@CallSite.AllArguments final Object[] arguments, - @CallSite.Return final String result, - @CallSite.InvokeDynamicConstants final Object[] constants) { - return result; - } - } - - @Requires({ - jvm.java9Compatible - }) - void 'test invoke dynamic with constants advice'() { - setup: - final spec = buildClassSpecification(InvokeDynamicWithConstantsAdvice) - final generator = buildAdviceGenerator(buildDir) - - when: - final result = generator.generate(spec) - - then: - assertNoErrors(result) - assertCallSites(result.file) { - interfaces(CallSites) - helpers(InvokeDynamicWithConstantsAdvice) - advices(0) { - pointcut( - 'java/lang/invoke/StringConcatFactory', - 'makeConcatWithConstants', - '(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/invoke/CallSite;' - ) - statements( - 'handler.dupParameters(descriptor, StackDupMode.PREPEND_ARRAY);', - 'handler.invokeDynamic(name, descriptor, bootstrapMethodHandle, bootstrapMethodArguments);', - 'handler.loadConstantArray(bootstrapMethodArguments);', - 'handler.advice("datadog/trace/plugin/csi/impl/AdviceGeneratorTest$InvokeDynamicWithConstantsAdvice", "after", "([Ljava/lang/Object;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/String;");' - ) - } - } - } - - @CallSite(spi = CallSites) - class ArrayAdvice { - @CallSite.AfterArray([ - @CallSite.After('java.util.Map javax.servlet.ServletRequest.getParameterMap()'), - @CallSite.After('java.util.Map javax.servlet.ServletRequestWrapper.getParameterMap()') - ]) - static Map after(@CallSite.This final ServletRequest request, @CallSite.Return final Map parameters) { - return parameters - } - } - - void 'test array advice'() { - setup: - final spec = buildClassSpecification(ArrayAdvice) - final generator = buildAdviceGenerator(buildDir) - - when: - final result = generator.generate(spec) - - then: - assertNoErrors(result) - assertCallSites(result.file) { - advices(0) { - pointcut('javax/servlet/ServletRequest', 'getParameterMap', '()Ljava/util/Map;') - } - advices(1) { - pointcut('javax/servlet/ServletRequestWrapper', 'getParameterMap', '()Ljava/util/Map;') - } - } - } - - class MinJavaVersionCheck { - static boolean isAtLeast(final String version) { - return Integer.parseInt(version) >= 9 - } - } - - @CallSite(spi = CallSites, enabled = ['datadog.trace.plugin.csi.impl.AdviceGeneratorTest$MinJavaVersionCheck', 'isAtLeast', '18']) - class MinJavaVersionAdvice { - @CallSite.After('java.lang.String java.lang.String.concat(java.lang.String)') - static String after(@CallSite.This final String self, @CallSite.Argument final String param, @CallSite.Return final String result) { - return result - } - } - - void 'test custom enabled property'() { - setup: - final spec = buildClassSpecification(MinJavaVersionAdvice) - final generator = buildAdviceGenerator(buildDir) - - when: - final result = generator.generate(spec) - - then: - assertNoErrors(result) - assertCallSites(result.file) { callSites -> - interfaces(CallSites, CallSites.HasEnabledProperty) - enabled(MinJavaVersionCheck.getDeclaredMethod('isAtLeast', String), '18') - } - } - - - @CallSite(spi = CallSites) - class PartialArgumentsBeforeAdvice { - @CallSite.Before("int java.sql.Statement.executeUpdate(java.lang.String, java.lang.String[])") - static void before(@CallSite.Argument(0) String arg1) {} - - @CallSite.Before("java.lang.String java.lang.String.format(java.lang.String, java.lang.Object[])") - static void before(@CallSite.Argument(1) Object[] arg) {} - - @CallSite.Before("java.lang.CharSequence java.lang.String.subSequence(int, int)") - static void before(@CallSite.This String thiz, @CallSite.Argument(0) int arg) {} - } - - void 'partial arguments with before advice'() { - setup: - final spec = buildClassSpecification(PartialArgumentsBeforeAdvice) - final generator = buildAdviceGenerator(buildDir) - - when: - final result = generator.generate(spec) - - then: - assertNoErrors result - assertCallSites(result.file) { - advices(0) { - pointcut('java/sql/Statement', 'executeUpdate', '(Ljava/lang/String;[Ljava/lang/String;)I') - statements( - 'int[] parameterIndices = new int[] { 0 };', - 'handler.dupParameters(descriptor, parameterIndices, owner);', - 'handler.advice("datadog/trace/plugin/csi/impl/AdviceGeneratorTest$PartialArgumentsBeforeAdvice", "before", "(Ljava/lang/String;)V");', - 'handler.method(opcode, owner, name, descriptor, isInterface);', - ) - } - advices(1) { - pointcut('java/lang/String', 'format', '(Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/String;') - statements( - 'int[] parameterIndices = new int[] { 1 };', - 'handler.dupParameters(descriptor, parameterIndices, null);', - 'handler.advice("datadog/trace/plugin/csi/impl/AdviceGeneratorTest$PartialArgumentsBeforeAdvice", "before", "([Ljava/lang/Object;)V");', - 'handler.method(opcode, owner, name, descriptor, isInterface);', - ) - } - advices(2) { - pointcut('java/lang/String', 'subSequence', '(II)Ljava/lang/CharSequence;') - statements( - 'int[] parameterIndices = new int[] { 0 };', - 'handler.dupInvoke(owner, descriptor, parameterIndices);', - 'handler.advice("datadog/trace/plugin/csi/impl/AdviceGeneratorTest$PartialArgumentsBeforeAdvice", "before", "(Ljava/lang/String;I)V");', - 'handler.method(opcode, owner, name, descriptor, isInterface);', - ) - } - } - } - - - @CallSite(spi = CallSites) - class SuperTypeReturnAdvice { - @CallSite.After("void java.lang.StringBuilder.(java.lang.String)") - static Object after(@CallSite.AllArguments Object[] args, @CallSite.Return Object result) { - return result - } - } - - void 'test returning super type'() { - setup: - final spec = buildClassSpecification(SuperTypeReturnAdvice) - final generator = buildAdviceGenerator(buildDir) - - when: - final result = generator.generate(spec) - - then: - assertNoErrors result - assertCallSites(result.file) { - advices(0) { - pointcut('java/lang/StringBuilder', '', '(Ljava/lang/String;)V') - statements( - 'handler.dupParameters(descriptor, StackDupMode.PREPEND_ARRAY_CTOR);', - 'handler.method(opcode, owner, name, descriptor, isInterface);', - 'handler.advice("datadog/trace/plugin/csi/impl/AdviceGeneratorTest$SuperTypeReturnAdvice", "after", "([Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;");', - 'handler.instruction(Opcodes.CHECKCAST, "java/lang/StringBuilder");' - ) - } - } - } - - @CallSite(spi = [IastCallSites, RaspCallSites]) - class MultipleSpiClassesAdvice { - @CallSite.After("void java.lang.StringBuilder.(java.lang.String)") - static Object after(@CallSite.AllArguments Object[] args, @CallSite.Return Object result) { - return result - } - } - - void 'test multiple spi classes'() { - setup: - final spec = buildClassSpecification(MultipleSpiClassesAdvice) - final generator = buildAdviceGenerator(buildDir) - - when: - final result = generator.generate(spec) - - then: - assertNoErrors result - assertCallSites(result.file) { - spi(IastCallSites, RaspCallSites) - } - } - - - @CallSite(spi = CallSites) - class AfterAdviceWithVoidReturn { - @CallSite.After("void java.lang.StringBuilder.setLength(int)") - static void after(@CallSite.This StringBuilder self, @CallSite.Argument(0) int length) { - } - } - - void 'test after advice with void return'() { - setup: - final spec = buildClassSpecification(AfterAdviceWithVoidReturn) - final generator = buildAdviceGenerator(buildDir) - - when: - final result = generator.generate(spec) - - then: - assertNoErrors result - assertCallSites(result.file) { - advices(0) { - pointcut('java/lang/StringBuilder', 'setLength', '(I)V') - statements( - 'handler.dupInvoke(owner, descriptor, StackDupMode.COPY);', - 'handler.method(opcode, owner, name, descriptor, isInterface);', - 'handler.advice("datadog/trace/plugin/csi/impl/AdviceGeneratorTest$AfterAdviceWithVoidReturn", "after", "(Ljava/lang/StringBuilder;I)V");', - ) - } - } - } - - private static AdviceGenerator buildAdviceGenerator(final File targetFolder) { - return new AdviceGeneratorImpl(targetFolder, pointcutParser()) - } - - private static void assertCallSites(final File generated, @DelegatesTo(CallSiteAssert) final Closure closure) { - final asserter = new AssertBuilder(generated).build() - closure.delegate = asserter - closure(asserter) - } -} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/AdviceSpecificationTest.groovy b/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/AdviceSpecificationTest.groovy deleted file mode 100644 index f3407dd786f..00000000000 --- a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/AdviceSpecificationTest.groovy +++ /dev/null @@ -1,567 +0,0 @@ -package datadog.trace.plugin.csi.impl - -import datadog.trace.agent.tooling.csi.CallSite -import datadog.trace.agent.tooling.csi.CallSites -import datadog.trace.plugin.csi.HasErrors.Failure -import datadog.trace.plugin.csi.util.ErrorCode -import groovy.transform.CompileDynamic -import org.objectweb.asm.Type - -import datadog.trace.plugin.csi.impl.CallSiteSpecification.ThisSpecification as This -import datadog.trace.plugin.csi.impl.CallSiteSpecification.ReturnSpecification as Return -import datadog.trace.plugin.csi.impl.CallSiteSpecification.ArgumentSpecification as Arg -import datadog.trace.plugin.csi.impl.CallSiteSpecification.AllArgsSpecification as AllArgs -import datadog.trace.plugin.csi.impl.CallSiteSpecification.InvokeDynamicConstantsSpecification as DynConsts -import datadog.trace.plugin.csi.impl.CallSiteSpecification.ParameterSpecification -import spock.lang.Requires - -import javax.servlet.ServletRequest -import java.lang.invoke.MethodHandles -import java.lang.invoke.MethodType -import java.security.MessageDigest - -@CompileDynamic -class AdviceSpecificationTest extends BaseCsiPluginTest { - - @CallSite(spi = CallSites) - class EmptyAdvice {} - - void 'test class generator error, call site without advices'() { - setup: - final context = mockValidationContext() - final spec = buildClassSpecification(EmptyAdvice) - - when: - spec.validate(context) - - then: - 1 * context.addError(ErrorCode.CALL_SITE_SHOULD_HAVE_ADVICE_METHODS, _) - } - - @CallSite(spi = CallSites) - class NonPublicStaticMethodAdvice { - @CallSite.Before("void java.lang.Runnable.run()") - private void advice(@CallSite.This final Runnable run) {} - } - - void 'test class generator error, non public static method'() { - setup: - final context = mockValidationContext() - final spec = buildClassSpecification(NonPublicStaticMethodAdvice) - - when: - spec.advices.each { it.validate(context) } - - then: - 1 * context.addError(ErrorCode.ADVICE_METHOD_NOT_STATIC_AND_PUBLIC, _) - } - - class BeforeStringConcat { - static void concat(final String self, final String value) {} - } - - void 'test advice class should be on the classpath'(final Type type, final int errors) { - setup: - final context = mockValidationContext() - final spec = before { - advice { - method(BeforeStringConcat.getDeclaredMethod('concat', String, String)) - owner(type) // override owner - } - parameters(new This(), new Arg()) - signature('java.lang.String java.lang.String.concat(java.lang.String)') - } - - when: - spec.validate(context) - - then: - errors * context.addError { Failure failure -> failure.errorCode == ErrorCode.UNRESOLVED_TYPE } - 0 * context.addError(*_) - - where: - type | errors - Type.getType('Lfoo/bar/FooBar;') | 1 - Type.getType(BeforeStringConcat) | 0 - } - - void 'test before advice should return void'(final Class returnType, final int errors) { - setup: - final context = mockValidationContext() - final spec = before { - advice { - owner(BeforeStringConcat) - method('concat') - descriptor(returnType, String, String) // change return - } - parameters(new This(), new Arg()) - signature('java.lang.String java.lang.String.concat(java.lang.String)') - } - - when: - spec.validate(context) - - then: - errors * context.addError(ErrorCode.ADVICE_BEFORE_SHOULD_RETURN_VOID, _) - 0 * context.addError(*_) - - - where: - returnType || errors - String || 1 - void.class || 0 - } - - class AroundStringConcat { - static String concat(final String self, final String value) { - return self.concat(value) - } - } - - void 'test around advice should return type compatible with pointcut'(final Class returnType, final int errors) { - setup: - final context = mockValidationContext() - final spec = around { - advice { - owner(AroundStringConcat) - method('concat') - descriptor(returnType, String, String) // change return - } - parameters(new This(), new Arg()) - signature('java.lang.String java.lang.String.concat(java.lang.String)') - } - - when: - spec.validate(context) - - then: - errors * context.addError(ErrorCode.ADVICE_METHOD_RETURN_NOT_COMPATIBLE, _) - 0 * context.addError(*_) - - where: - returnType | errors - MessageDigest | 1 - Object | 0 - String | 0 - } - - class AfterStringConcat { - static String concat(final String self, final String value, final String result) { - return result - } - } - - void 'test after advice should return type compatible with pointcut'(final Class returnType, final int errors) { - setup: - final context = mockValidationContext() - final spec = after { - advice { - owner(AfterStringConcat) - method('concat') - descriptor(returnType, String, String, String) - // change return - } - parameters(new This(), new Arg(), new Return()) - signature('java.lang.String java.lang.String.concat(java.lang.String)') - } - - when: - spec.validate(context) - - then: - errors * context.addError(ErrorCode.ADVICE_METHOD_RETURN_NOT_COMPATIBLE, _) - 0 * context.addError(*_) - - where: - returnType | errors - MessageDigest | 1 - Object | 0 - String | 0 - } - - void 'test this parameter should always be the first'(final List params, final int errors) { - setup: - final context = mockValidationContext() - final spec = around { - advice { - method(AroundStringConcat.getDeclaredMethod('concat', String, String)) - } - parameters(params as ParameterSpecification[]) - signature('java.lang.String java.lang.String.concat(java.lang.String)') - } - - when: - spec.validate(context) - - then: - errors * context.addError(ErrorCode.ADVICE_PARAMETER_THIS_SHOULD_BE_FIRST, _) - 0 * context.addError(*_) - - where: - params | errors - [new This(), new Arg()] | 0 - [new Arg(), new This()] | 1 - } - - - void 'test this parameter should be compatible with pointcut'(final Class type, final int errors) { - setup: - final context = mockValidationContext() - final spec = around { - advice { - owner(AroundStringConcat) - method('concat') - descriptor(String, type, String) - } - parameters(new This(), new Arg()) - signature('java.lang.String java.lang.String.concat(java.lang.String)') - } - - when: - spec.validate(context) - - then: - errors * context.addError(ErrorCode.ADVICE_METHOD_PARAM_THIS_NOT_COMPATIBLE, _) - // advice returns String so other return types won't be able to find the method - if (type != String) { - 1 * context.addError { Failure failure -> failure.errorCode == ErrorCode.UNRESOLVED_METHOD } - } - 0 * context.addError(*_) - - where: - type | errors - MessageDigest | 1 - Object | 0 - String | 0 - } - - void 'test return parameter should always be the last'(final List params, final int errors) { - setup: - final context = mockValidationContext() - final spec = after { - advice { - method(AfterStringConcat.getDeclaredMethod('concat', String, String, String)) - } - parameters(params as ParameterSpecification[]) - signature('java.lang.String java.lang.String.concat(java.lang.String)') - } - - when: - spec.validate(context) - - then: - errors * context.addError(ErrorCode.ADVICE_PARAMETER_RETURN_SHOULD_BE_LAST, _) - // other errors are ignored - - where: - params | errors - [new This(), new Arg(), new Return()] | 0 - [new This(), new Return(), new Arg()] | 1 - } - - - void 'test return parameter should be compatible with pointcut'(final Class returnType, final int errors) { - setup: - final context = mockValidationContext() - final spec = after { - advice { - owner(AfterStringConcat) - method('concat') - descriptor(String, String, String, returnType) - } - parameters(new This(), new Arg(), new Return()) - signature('java.lang.String java.lang.String.concat(java.lang.String)') - } - - when: - spec.validate(context) - - then: - errors * context.addError(ErrorCode.ADVICE_METHOD_PARAM_RETURN_NOT_COMPATIBLE, _) - // advice returns String so other return types won't be able to find the method - if (returnType != String) { - 1 * context.addError { Failure failure -> failure.errorCode == ErrorCode.UNRESOLVED_METHOD } - } - 0 * context.addError(*_) - - where: - returnType | errors - MessageDigest | 1 - String | 0 - Object | 0 - } - - - void 'test argument parameter should be compatible with pointcut'(final Class parameterType, final int errors) { - setup: - final context = mockValidationContext() - final spec = after { - advice { - owner(AfterStringConcat) - method('concat') - descriptor(String, String, parameterType, String) - } - parameters(new This(), new Arg(), new Return()) - signature('java.lang.String java.lang.String.concat(java.lang.String)') - } - - when: - spec.validate(context) - - then: - errors * context.addError(ErrorCode.ADVICE_METHOD_PARAM_NOT_COMPATIBLE, _) - // advice parameter is a String so with other types won't be able to find the method - if (parameterType != String) { - 1 * context.addError { Failure failure -> failure.errorCode == ErrorCode.UNRESOLVED_METHOD } - } - 0 * context.addError(*_) - - where: - parameterType | errors - MessageDigest | 1 - String | 0 - Object | 0 - } - - class BadAfterStringConcat { - static String concat(final String param1, final String param2) { - return param2 - } - } - - void 'test after advice requires @This and @Return parameters'(final List params, final ErrorCode error) { - setup: - final context = mockValidationContext() - final spec = after { - advice { - method(BadAfterStringConcat.getDeclaredMethod('concat', String, String)) - } - parameters(params as ParameterSpecification[]) - signature('java.lang.String java.lang.String.concat(java.lang.String)') - } - - when: - spec.validate(context) - - then: - 1 * context.addError(error, _) - 0 * context.addError(*_) - - where: - params | error - [new Arg(), new Return()] | ErrorCode.ADVICE_AFTER_SHOULD_HAVE_THIS - [new This(), new Arg()] | ErrorCode.ADVICE_AFTER_SHOULD_HAVE_RETURN - } - - class BadAllArgsAfterStringConcat { - static String concat(final Object[] param1, final String param2, final String param3) { - return param3 - } - } - - void 'should not mix @AllArguments and @Argument'() { - setup: - final context = mockValidationContext() - final spec = after { - advice { - method(BadAllArgsAfterStringConcat.getDeclaredMethod('concat', Object[], String, String)) - } - parameters(new AllArgs(includeThis: true), new Arg(), new Return()) - signature('java.lang.String java.lang.String.concat(java.lang.String)') - } - - when: - spec.validate(context) - - then: - 1 * context.addError(ErrorCode.ADVICE_PARAMETER_ALL_ARGS_MIXED, _) - 1 * context.addError(ErrorCode.ADVICE_PARAMETER_ARGUMENT_OUT_OF_BOUNDS, _) // all args consumes all arguments - 0 * context.addError(*_) - } - - static class TestInheritedMethod { - static String after(final ServletRequest request, final String parameter, final String value) { - return value - } - } - - void 'test inherited methods'() { - setup: - final context = mockValidationContext() - final spec = after { - advice { - method(TestInheritedMethod.getDeclaredMethod('after', ServletRequest, String, String)) - } - parameters(new This(), new Arg(), new Return()) - signature('java.lang.String javax.servlet.http.HttpServletRequest.getParameter(java.lang.String)') - } - - when: - spec.validate(context) - - then: - 0 * context.addError(*_) - } - - static class TestInvokeDynamicConstants { - static Object after(final Object[] parameter, final Object result, final Object[] constants) { - return result - } - } - - @Requires({ - jvm.java9Compatible - }) - void 'test invoke dynamic constants'() { - setup: - final context = mockValidationContext() - final spec = after { - advice { - method(TestInvokeDynamicConstants.getDeclaredMethod('after', Object[], Object, Object[])) - } - parameters(new AllArgs(), new Return(), new DynConsts()) - signature('java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])') - invokeDynamic(true) - } - - when: - spec.validate(context) - - then: - 0 * context.addError(*_) - } - - @Requires({ - jvm.java9Compatible - }) - void 'test invoke dynamic constants should be last'(final List params, final ErrorCode error) { - setup: - final context = mockValidationContext() - final spec = after { - advice { - method(TestInvokeDynamicConstants.getDeclaredMethod('after', Object[], Object, Object[])) - } - parameters(params as ParameterSpecification[]) - signature('java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])') - invokeDynamic(true) - } - - when: - spec.validate(context) - - then: - if (error != null) { - 1 * context.addError(error, _) - } - 0 * context.addError(*_) - - where: - params | error - [new AllArgs(), new Return(), new DynConsts()] | null - [new AllArgs(), new DynConsts(), new Return()] | ErrorCode.ADVICE_PARAMETER_INVOKE_DYNAMIC_CONSTANTS_SHOULD_BE_LAST - } - - static class TestInvokeDynamicConstantsNonInvokeDynamic { - static Object after(final Object self, final Object[] parameter, final Object value, final Object[] constants) { - return value - } - } - - @Requires({ - jvm.java9Compatible - }) - void 'test invoke dynamic constants on non invoke dynamic pointcut'() { - setup: - final context = mockValidationContext() - final spec = after { - advice { - method(TestInvokeDynamicConstantsNonInvokeDynamic.getDeclaredMethod('after', Object, Object[], Object, Object[])) - } - parameters(new This(), new AllArgs(), new DynConsts(), new Return()) - signature('java.lang.String java.lang.String.concat(java.lang.String)') - } - - when: - spec.validate(context) - - then: - 1 * context.addError(ErrorCode.ADVICE_PARAMETER_INVOKE_DYNAMIC_CONSTANTS_ON_NON_INVOKE_DYNAMIC, _) - } - - static class TestInvokeDynamicConstantsBefore { - static void before(final Object[] parameter, final Object[] constants) { - } - } - - @Requires({ - jvm.java9Compatible - }) - void 'test invoke dynamic constants on non @After advice'() { - setup: - final context = mockValidationContext() - final spec = before { - advice { - method(TestInvokeDynamicConstantsBefore.getDeclaredMethod('before', Object[], Object[])) - } - parameters(new AllArgs(), new DynConsts()) - signature('java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])') - invokeDynamic(true) - } - - when: - spec.validate(context) - - then: - 1 * context.addError(ErrorCode.ADVICE_PARAMETER_INVOKE_DYNAMIC_CONSTANTS_NON_AFTER_ADVICE, _) - } - - static class TestInvokeDynamicConstantsAround { - static java.lang.invoke.CallSite around(final MethodHandles.Lookup lookup, final String name, final MethodType concatType, final String recipe, final Object... constants) { - return null - } - } - - @Requires({ - jvm.java9Compatible - }) - void 'test invoke dynamic on @Around advice'() { - setup: - final context = mockValidationContext() - final spec = around { - advice { - method(TestInvokeDynamicConstantsAround.getDeclaredMethod('around', MethodHandles.Lookup, String, MethodType, String, Object[])) - } - parameters(new Arg(), new Arg(), new Arg(), new Arg(), new Arg()) - signature('java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])') - invokeDynamic(true) - } - - when: - spec.validate(context) - - then: - 0 * context.addError(_, _) - } - - - @CallSite(spi = CallSites) - class AfterWithVoidWrongAdvice { - @CallSite.After("void java.lang.String.getChars(int, int, char[], int)") - static String after(@CallSite.AllArguments final Object[] args, @CallSite.Return final String result) { - return result; - } - } - - void 'test after advice with void should not use @Return'() { - setup: - final context = mockValidationContext() - final spec = buildClassSpecification(AfterWithVoidWrongAdvice) - - when: - spec.advices.each { it.validate(context) } - - then: - 1 * context.addError(ErrorCode.ADVICE_AFTER_VOID_METHOD_SHOULD_RETURN_VOID, _) - 1 * context.addError(ErrorCode.ADVICE_AFTER_VOID_METHOD_SHOULD_NOT_HAVE_RETURN, _) - } -} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/AsmSpecificationBuilderTest.groovy b/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/AsmSpecificationBuilderTest.groovy deleted file mode 100644 index ff6221c991c..00000000000 --- a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/AsmSpecificationBuilderTest.groovy +++ /dev/null @@ -1,516 +0,0 @@ -package datadog.trace.plugin.csi.impl - -import datadog.trace.agent.tooling.csi.CallSite -import datadog.trace.agent.tooling.csi.CallSites -import datadog.trace.plugin.csi.impl.CallSiteSpecification.AdviceSpecification -import datadog.trace.plugin.csi.impl.CallSiteSpecification.AfterSpecification -import datadog.trace.plugin.csi.impl.CallSiteSpecification.AroundSpecification -import datadog.trace.plugin.csi.impl.CallSiteSpecification.BeforeSpecification -import datadog.trace.plugin.csi.util.Types -import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; -import groovy.transform.CompileDynamic -import org.objectweb.asm.Type - -import javax.annotation.Nonnull -import javax.annotation.Nullable -import javax.servlet.ServletRequest -import java.lang.invoke.MethodHandles -import java.lang.invoke.MethodType -import java.util.stream.Collectors - -@CompileDynamic -final class AsmSpecificationBuilderTest extends BaseCsiPluginTest { - - static class NonCallSite {} - - void 'test specification builder for non call site'() { - setup: - final advice = fetchClass(NonCallSite) - final specificationBuilder = new AsmSpecificationBuilder() - - when: - final result = specificationBuilder.build(advice) - - then: - !result.present - } - - @CallSite(spi = Spi) - static class WithSpiClass { - interface Spi {} - } - - void 'test specification builder with custom spi class'() { - setup: - final advice = fetchClass(WithSpiClass) - final specificationBuilder = new AsmSpecificationBuilder() - - when: - final result = specificationBuilder.build(advice).orElseThrow(RuntimeException::new) - - then: - result.spi == [Type.getType(WithSpiClass.Spi)] as Type[] - } - - @CallSite(spi = CallSites, helpers = [SampleHelper1.class, SampleHelper2.class]) - static class HelpersAdvice { - static class SampleHelper1 {} - static class SampleHelper2 {} - } - - void 'test specification builder with custom helper classes'() { - setup: - final advice = fetchClass(HelpersAdvice) - final specificationBuilder = new AsmSpecificationBuilder() - - when: - final result = specificationBuilder.build(advice).orElseThrow(RuntimeException::new) - - then: - result.helpers.toList().containsAll([ - Type.getType(HelpersAdvice), - Type.getType(HelpersAdvice.SampleHelper1), - Type.getType(HelpersAdvice.SampleHelper2) - ]) - } - - @CallSite(spi = CallSites) - static class BeforeAdvice { - @CallSite.Before('java.lang.String java.lang.String.replaceAll(java.lang.String, java.lang.String)') - static void before(@CallSite.This final String self, @CallSite.Argument final String regexp, @CallSite.Argument final String replacement) { - } - } - - void 'test specification builder for before advice'() { - setup: - final advice = fetchClass(BeforeAdvice) - final specificationBuilder = new AsmSpecificationBuilder() - - when: - final result = specificationBuilder.build(advice).orElseThrow(RuntimeException::new) - - then: - result.clazz.className == BeforeAdvice.name - final beforeSpec = findAdvice(result, 'before') - beforeSpec instanceof BeforeSpecification - beforeSpec.advice.methodType.descriptor == '(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)V' - beforeSpec.signature == 'java.lang.String java.lang.String.replaceAll(java.lang.String, java.lang.String)' - beforeSpec.findThis() != null - beforeSpec.findReturn() == null - beforeSpec.findAllArguments() == null - beforeSpec.findInvokeDynamicConstants() == null - final arguments = getArguments(beforeSpec) - arguments == [0, 1] - } - - @CallSite(spi = CallSites) - static class AroundAdvice { - @CallSite.Around('java.lang.String java.lang.String.replaceAll(java.lang.String, java.lang.String)') - static String around(@CallSite.This final String self, @CallSite.Argument final String regexp, @CallSite.Argument final String replacement) { - return self.replaceAll(regexp, replacement) - } - } - - void 'test specification builder for around advice'() { - setup: - final advice = fetchClass(AroundAdvice) - final specificationBuilder = new AsmSpecificationBuilder() - - when: - final result = specificationBuilder.build(advice).orElseThrow(RuntimeException::new) - - then: - result.clazz.className == AroundAdvice.name - final aroundSpec = findAdvice(result, 'around') - aroundSpec instanceof AroundSpecification - aroundSpec.advice.methodType.descriptor == '(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;' - aroundSpec.signature == 'java.lang.String java.lang.String.replaceAll(java.lang.String, java.lang.String)' - aroundSpec.findThis() != null - aroundSpec.findReturn() == null - aroundSpec.findAllArguments() == null - aroundSpec.findInvokeDynamicConstants() == null - final arguments = getArguments(aroundSpec) - arguments == [0, 1] - } - - @CallSite(spi = CallSites) - static class AfterAdvice { - @CallSite.After('java.lang.String java.lang.String.replaceAll(java.lang.String, java.lang.String)') - static String after(@CallSite.This final String self, @CallSite.Argument final String regexp, @CallSite.Argument final String replacement, @CallSite.Return final String result) { - return result - } - } - - void 'test specification builder for after advice'() { - setup: - final advice = fetchClass(AfterAdvice) - final specificationBuilder = new AsmSpecificationBuilder() - - when: - final result = specificationBuilder.build(advice).orElseThrow(RuntimeException::new) - - then: - result.clazz.className == AfterAdvice.name - final afterSpec = findAdvice(result, 'after') - afterSpec instanceof AfterSpecification - afterSpec.advice.methodType.descriptor == '(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;' - afterSpec.signature == 'java.lang.String java.lang.String.replaceAll(java.lang.String, java.lang.String)' - afterSpec.findThis() != null - afterSpec.findReturn() != null - afterSpec.findAllArguments() == null - afterSpec.findInvokeDynamicConstants() == null - final arguments = getArguments(afterSpec) - arguments == [0, 1] - } - - @CallSite - static class AllArgsAdvice { - @CallSite.Around('java.lang.String java.lang.String.replaceAll(java.lang.String, java.lang.String)') - static String allArgs(@CallSite.AllArguments(includeThis = true) final Object[] arguments, @CallSite.Return final String result) { - return result - } - } - - void 'test specification builder for advice with @AllArguments'() { - setup: - final advice = fetchClass(AllArgsAdvice) - final specificationBuilder = new AsmSpecificationBuilder() - - when: - final result = specificationBuilder.build(advice).orElseThrow(RuntimeException::new) - - then: - result.clazz.className == AllArgsAdvice.name - final allArgsSpec = findAdvice(result, 'allArgs') - allArgsSpec instanceof AroundSpecification - allArgsSpec.advice.methodType.descriptor == '([Ljava/lang/Object;Ljava/lang/String;)Ljava/lang/String;' - allArgsSpec.signature == 'java.lang.String java.lang.String.replaceAll(java.lang.String, java.lang.String)' - allArgsSpec.findThis() == null - allArgsSpec.findReturn() != null - final allArguments = allArgsSpec.findAllArguments() - allArguments != null - allArguments.includeThis - allArgsSpec.findInvokeDynamicConstants() == null - final arguments = getArguments(allArgsSpec) - arguments == [] - } - - @CallSite(spi = CallSites) - static class InvokeDynamicBeforeAdvice { - @CallSite.After( - value = 'java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])', - invokeDynamic = true - ) - static String invokeDynamic(@CallSite.AllArguments final Object[] arguments, @CallSite.Return final String result) { - return result - } - } - - void 'test specification builder for before invoke dynamic'() { - setup: - final advice = fetchClass(InvokeDynamicBeforeAdvice) - final specificationBuilder = new AsmSpecificationBuilder() - - when: - final result = specificationBuilder.build(advice).orElseThrow(RuntimeException::new) - - then: - result.clazz.className == InvokeDynamicBeforeAdvice.name - final invokeDynamicSpec = findAdvice(result, 'invokeDynamic') - invokeDynamicSpec instanceof AfterSpecification - invokeDynamicSpec.advice.methodType.descriptor == '([Ljava/lang/Object;Ljava/lang/String;)Ljava/lang/String;' - invokeDynamicSpec.signature == 'java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])' - invokeDynamicSpec.findThis() == null - invokeDynamicSpec.findReturn() != null - final allArguments = invokeDynamicSpec.findAllArguments() - allArguments != null - !allArguments.includeThis - invokeDynamicSpec.findInvokeDynamicConstants() == null - final arguments = getArguments(invokeDynamicSpec) - arguments == [] - } - - @CallSite(spi = CallSites) - static class InvokeDynamicAroundAdvice { - @CallSite.Around( - value = 'java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])', - invokeDynamic = true - ) - static java.lang.invoke.CallSite invokeDynamic(@CallSite.Argument final MethodHandles.Lookup lookup, - @CallSite.Argument final String name, - @CallSite.Argument final MethodType concatType, - @CallSite.Argument final String recipe, - @CallSite.Argument final Object... constants) { - return null - } - } - - void 'test specification builder for around invoke dynamic'() { - setup: - final advice = fetchClass(InvokeDynamicAroundAdvice) - final specificationBuilder = new AsmSpecificationBuilder() - - when: - final result = specificationBuilder.build(advice).orElseThrow(RuntimeException::new) - - then: - result.clazz.className == InvokeDynamicAroundAdvice.name - final invokeDynamicSpec = findAdvice(result, 'invokeDynamic') - invokeDynamicSpec instanceof AroundSpecification - invokeDynamicSpec.advice.methodType.descriptor == '(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/invoke/CallSite;' - invokeDynamicSpec.signature == 'java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])' - invokeDynamicSpec.findThis() == null - invokeDynamicSpec.findReturn() == null - invokeDynamicSpec.findAllArguments() == null - invokeDynamicSpec.findInvokeDynamicConstants() == null - final arguments = getArguments(invokeDynamicSpec) - arguments == [0, 1, 2, 3, 4] - } - - @CallSite(spi = CallSites) - static class TestInvokeDynamicConstants { - @CallSite.After( - value = 'java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])', - invokeDynamic = true - ) - static String after(@CallSite.AllArguments final Object[] parameter, - @CallSite.InvokeDynamicConstants final Object[] constants, - @CallSite.Return final String value) { - return value - } - } - - void 'test invoke dynamic constants'() { - setup: - final advice = fetchClass(TestInvokeDynamicConstants) - final specificationBuilder = new AsmSpecificationBuilder() - - when: - final result = specificationBuilder.build(advice).orElseThrow(RuntimeException::new) - - then: - result.clazz.className == TestInvokeDynamicConstants.name - final inheritedSpec = findAdvice(result, 'after') - inheritedSpec instanceof AfterSpecification - inheritedSpec.advice.methodType.descriptor == '([Ljava/lang/Object;[Ljava/lang/Object;Ljava/lang/String;)Ljava/lang/String;' - inheritedSpec.signature == 'java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])' - inheritedSpec.findThis() == null - inheritedSpec.findReturn() != null - inheritedSpec.findInvokeDynamicConstants() != null - final arguments = getArguments(inheritedSpec) - arguments == [] - } - - @CallSite(spi = CallSites) - static class TestBeforeArray { - - @CallSite.BeforeArray([ - @CallSite.Before('java.util.Map javax.servlet.ServletRequest.getParameterMap()'), - @CallSite.Before('java.util.Map javax.servlet.ServletRequestWrapper.getParameterMap()') - ]) - static void before(@CallSite.This final ServletRequest request) { } - } - - void 'test specification builder for before advice array'() { - setup: - final advice = fetchClass(TestBeforeArray) - final specificationBuilder = new AsmSpecificationBuilder() - - when: - final result = specificationBuilder.build(advice).orElseThrow(RuntimeException::new) - - then: - result.clazz.className == TestBeforeArray.name - final list = result.advices - list.size() == 2 - list.each { - assert it instanceof BeforeSpecification - assert it.advice.methodType.descriptor == '(Ljavax/servlet/ServletRequest;)V' - assert it.signature in [ - 'java.util.Map javax.servlet.ServletRequest.getParameterMap()', - 'java.util.Map javax.servlet.ServletRequestWrapper.getParameterMap()' - ] - assert it.findThis() != null - assert it.findReturn() == null - assert it.findAllArguments() == null - assert it.findInvokeDynamicConstants() == null - final arguments = getArguments(it) - assert arguments == [] - } - } - - @CallSite(spi = CallSites) - static class TestAroundArray { - - @CallSite.AroundArray([ - @CallSite.Around('java.util.Map javax.servlet.ServletRequest.getParameterMap()'), - @CallSite.Around('java.util.Map javax.servlet.ServletRequestWrapper.getParameterMap()') - ]) - static Map around(@CallSite.This final ServletRequest request) { - return request.getParameterMap() - } - } - - void 'test specification builder for around advice array'() { - setup: - final advice = fetchClass(TestAroundArray) - final specificationBuilder = new AsmSpecificationBuilder() - - when: - final result = specificationBuilder.build(advice).orElseThrow(RuntimeException::new) - - then: - result.clazz.className == TestAroundArray.name - final list = result.advices - list.size() == 2 - list.each { - assert it instanceof AroundSpecification - assert it.advice.methodType.descriptor == '(Ljavax/servlet/ServletRequest;)Ljava/util/Map;' - assert it.signature in [ - 'java.util.Map javax.servlet.ServletRequest.getParameterMap()', - 'java.util.Map javax.servlet.ServletRequestWrapper.getParameterMap()' - ] - assert it.findThis() != null - assert it.findReturn() == null - assert it.findAllArguments() == null - assert it.findInvokeDynamicConstants() == null - final arguments = getArguments(it) - assert arguments == [] - } - } - - @CallSite(spi = CallSites) - static class TestAfterArray { - - @CallSite.AfterArray([ - @CallSite.After('java.util.Map javax.servlet.ServletRequest.getParameterMap()'), - @CallSite.After('java.util.Map javax.servlet.ServletRequestWrapper.getParameterMap()') - ]) - static Map after(@CallSite.This final ServletRequest request, @CallSite.Return final Map parameters) { - return parameters - } - } - - void 'test specification builder for before advice array'() { - setup: - final advice = fetchClass(TestAfterArray) - final specificationBuilder = new AsmSpecificationBuilder() - - when: - final result = specificationBuilder.build(advice).orElseThrow(RuntimeException::new) - - then: - result.clazz.className == TestAfterArray.name - final list = result.advices - list.size() == 2 - list.each { - assert it instanceof AfterSpecification - assert it.advice.methodType.descriptor == '(Ljavax/servlet/ServletRequest;Ljava/util/Map;)Ljava/util/Map;' - assert it.signature in [ - 'java.util.Map javax.servlet.ServletRequest.getParameterMap()', - 'java.util.Map javax.servlet.ServletRequestWrapper.getParameterMap()' - ] - assert it.findThis() != null - assert it.findReturn() != null - assert it.findAllArguments() == null - assert it.findInvokeDynamicConstants() == null - final arguments = getArguments(it) - assert arguments == [] - } - } - - @CallSite(spi = CallSites) - static class TestInheritedMethod { - @CallSite.After('java.lang.String javax.servlet.http.HttpServletRequest.getParameter(java.lang.String)') - static String after(@CallSite.This final ServletRequest request, @CallSite.Argument final String parameter, @CallSite.Return final String value) { - return value - } - } - - void 'test specification builder for inherited methods'() { - setup: - final advice = fetchClass(TestInheritedMethod) - final specificationBuilder = new AsmSpecificationBuilder() - - when: - final result = specificationBuilder.build(advice).orElseThrow(RuntimeException::new) - - then: - result.clazz.className == TestInheritedMethod.name - final inheritedSpec = findAdvice(result, 'after') - inheritedSpec instanceof AfterSpecification - inheritedSpec.advice.methodType.descriptor == '(Ljavax/servlet/ServletRequest;Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;' - inheritedSpec.signature == 'java.lang.String javax.servlet.http.HttpServletRequest.getParameter(java.lang.String)' - inheritedSpec.findThis() != null - inheritedSpec.findReturn() != null - inheritedSpec.findAllArguments() == null - inheritedSpec.findInvokeDynamicConstants() == null - final arguments = getArguments(inheritedSpec) - arguments == [0] - } - - static class IsEnabled { - static boolean isEnabled(final String defaultValue) { - return true - } - } - - @CallSite(spi = CallSites, enabled = ['datadog.trace.plugin.csi.impl.AsmSpecificationBuilderTest$IsEnabled', 'isEnabled', 'true']) - static class TestEnablement { - @CallSite.After('java.lang.String javax.servlet.http.HttpServletRequest.getParameter(java.lang.String)') - static String after(@CallSite.This final ServletRequest request, @CallSite.Argument final String parameter, @CallSite.Return final String value) { - return value - } - } - - void 'test specification builder with enabled property'() { - setup: - final advice = fetchClass(TestEnablement) - final specificationBuilder = new AsmSpecificationBuilder() - - when: - final result = specificationBuilder.build(advice).orElseThrow(RuntimeException::new) - - then: - result.clazz.className == TestEnablement.name - result.enabled != null - result.enabled.method.owner == Type.getType(IsEnabled) - result.enabled.method.methodName == 'isEnabled' - result.enabled.method.methodType == Type.getMethodType(Types.BOOLEAN, Types.STRING) - result.enabled.arguments == ['true'] - } - - @CallSite(spi = CallSites) - static class TestWithOtherAnnotations { - @CallSite.Around("java.lang.StringBuilder java.lang.StringBuilder.append(java.lang.Object)") - @CallSite.Around("java.lang.StringBuffer java.lang.StringBuffer.append(java.lang.Object)") - @Nonnull - @SuppressFBWarnings( - "NP_PARAMETER_MUST_BE_NONNULL_BUT_MARKED_AS_NULLABLE") // we do check for null on self - // parameter - static Appendable aroundAppend(@CallSite.This @Nullable final Appendable self, @CallSite.Argument(0) @Nullable final Object param) throws Throwable { - return self.append(param.toString()) - } - } - - void 'test specification builder with multiple method annotations'() { - setup: - final advice = fetchClass(TestWithOtherAnnotations) - final specificationBuilder = new AsmSpecificationBuilder() - - when: - final result = specificationBuilder.build(advice).orElseThrow(RuntimeException::new) - - then: - result.clazz.className == TestWithOtherAnnotations.name - result.advices.size() == 2 - } - - private static List getArguments(final AdviceSpecification advice) { - return advice.arguments.map(it -> it.index).collect(Collectors.toList()) - } - - private static AdviceSpecification findAdvice(final CallSiteSpecification result, final String name) { - return result.advices.find { it.advice.methodName == name } - } -} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/BaseCsiPluginTest.groovy b/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/BaseCsiPluginTest.groovy deleted file mode 100644 index cae4c42ba96..00000000000 --- a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/BaseCsiPluginTest.groovy +++ /dev/null @@ -1,202 +0,0 @@ -package datadog.trace.plugin.csi.impl - -import datadog.trace.plugin.csi.HasErrors -import datadog.trace.plugin.csi.ValidationContext -import datadog.trace.plugin.csi.util.MethodType -import groovy.transform.CompileDynamic -import org.objectweb.asm.Type -import spock.lang.Specification - -import java.lang.reflect.Constructor -import java.lang.reflect.Executable -import java.lang.reflect.Method -import java.nio.file.Files -import java.nio.file.Paths -import java.util.stream.Collectors -import datadog.trace.plugin.csi.impl.CallSiteSpecification.ParameterSpecification -import datadog.trace.plugin.csi.impl.CallSiteSpecification.AdviceSpecification -import datadog.trace.plugin.csi.impl.CallSiteSpecification.BeforeSpecification -import datadog.trace.plugin.csi.impl.CallSiteSpecification.AroundSpecification -import datadog.trace.plugin.csi.impl.CallSiteSpecification.AfterSpecification -import datadog.trace.plugin.csi.impl.CallSiteSpecification.ArgumentSpecification - -import static datadog.trace.plugin.csi.impl.CallSiteFactory.pointcutParser -import static datadog.trace.plugin.csi.impl.CallSiteFactory.specificationBuilder -import static datadog.trace.plugin.csi.impl.CallSiteFactory.typeResolver -import static datadog.trace.plugin.csi.util.CallSiteConstants.TYPE_RESOLVER - -@CompileDynamic -abstract class BaseCsiPluginTest extends Specification { - - protected static void assertNoErrors(final HasErrors hasErrors) { - final errors = hasErrors.errors.collect { error -> - "${error.message}: ${error.cause == null ? '-' : error.causeString}" - } - assert errors == [] - } - - protected static File fetchClass(final Class clazz) { - final folder = Paths.get(clazz.getResource('/').toURI()).resolve('../../') - final fileSeparatorPattern = File.separator == "\\" ? "\\\\" : File.separator - final classFile = clazz.getName().replaceAll('\\.', fileSeparatorPattern) + '.class' - final groovy = folder.resolve('groovy/test').resolve(classFile) - if (Files.exists(groovy)) { - return groovy.toFile() - } - return folder.resolve('java/test').resolve(classFile).toFile() - } - - protected static CallSiteSpecification buildClassSpecification(final Class clazz) { - final classFile = fetchClass(clazz) - final spec = specificationBuilder().build(classFile).get() - final pointcutParser = pointcutParser() - spec.advices.each { it.parseSignature(pointcutParser) } - return spec - } - - protected ValidationContext mockValidationContext() { - return Mock(ValidationContext) { - mock -> - mock.getContextProperty(TYPE_RESOLVER) >> typeResolver() - } - } - - protected static BeforeSpecification before(@DelegatesTo(strategy = Closure.DELEGATE_ONLY, value = BeforeAdviceSpecificationBuilder) final Closure cl) { - final spec = new BeforeAdviceSpecificationBuilder() - final code = cl.rehydrate(spec, this, this) - code.resolveStrategy = Closure.DELEGATE_ONLY - code() - return spec.build() - } - - protected static AroundSpecification around(@DelegatesTo(strategy = Closure.DELEGATE_ONLY, value = AroundAdviceSpecificationBuilder) final Closure cl) { - final spec = new AroundAdviceSpecificationBuilder() - final code = cl.rehydrate(spec, this, this) - code.resolveStrategy = Closure.DELEGATE_ONLY - code() - return spec.build() - } - - protected static AfterSpecification after(@DelegatesTo(strategy = Closure.DELEGATE_ONLY, value = AfterAdviceSpecificationBuilder) final Closure cl) { - final spec = new AfterAdviceSpecificationBuilder() - final code = cl.rehydrate(spec, this, this) - code.resolveStrategy = Closure.DELEGATE_ONLY - code() - return spec.build() - } - - private static class BeforeAdviceSpecificationBuilder extends AdviceSpecificationBuilder { - @Override - protected AdviceSpecification build(final MethodType advice, - final Map parameters, - final String signature, - final boolean invokeDynamic) { - return new BeforeSpecification(advice, parameters, signature, invokeDynamic) - } - } - - private static class AroundAdviceSpecificationBuilder extends AdviceSpecificationBuilder { - @Override - protected AroundSpecification build(final MethodType advice, - final Map parameters, - final String signature, - final boolean invokeDynamic) { - return new AroundSpecification(advice, parameters, signature, invokeDynamic) - } - } - - private static class AfterAdviceSpecificationBuilder extends AdviceSpecificationBuilder { - @Override - protected AfterSpecification build(final MethodType advice, - final Map parameters, - final String signature, - final boolean invokeDynamic) { - return new AfterSpecification(advice, parameters, signature, invokeDynamic) - } - } - - private abstract static class AdviceSpecificationBuilder { - protected MethodType advice - protected Map parameters = [:] - protected String signature - protected boolean invokeDynamic - - void advice(@DelegatesTo(strategy = Closure.DELEGATE_ONLY, value = MethodTypeBuilder) final Closure body) { - final spec = new MethodTypeBuilder() - final code = body.rehydrate(spec, this, this) - code.resolveStrategy = Closure.DELEGATE_ONLY - code() - advice = spec.build() - } - - void parameters(final ParameterSpecification... parameters) { - parameters.eachWithIndex { entry, int i -> this.parameters.put(i, entry) } - parameters.grep { it instanceof ArgumentSpecification } - .collect { it as ArgumentSpecification} - .eachWithIndex{ spec, int i -> spec.index = i} - } - - void signature(final String signature) { - this.signature = signature - } - - void invokeDynamic(final boolean invokeDynamic) { - this.invokeDynamic = invokeDynamic - } - - E build() { - final result = build(advice, parameters, signature, invokeDynamic) as E - result.parseSignature(pointcutParser()) - return result - } - - - protected abstract AdviceSpecification build(final MethodType advice, - final Map parameters, - final String signature, - final boolean invokeDynamic) - } - - private static class MethodTypeBuilder { - protected Type owner - protected String methodName - protected Type methodType - - void owner(final Type value) { - owner = value - } - - void owner(final Class value) { - owner = Type.getType(value) - } - - void method(final String value) { - methodName = value - } - - void descriptor(final Type value) { - methodType = value - } - - void descriptor(final Class returnType, final Class... args) { - methodType = Type.getMethodType(Type.getType(returnType), args.collect { Type.getType(it) } as Type[]) - } - - void method(final Executable executable) { - owner = Type.getType(executable.declaringClass) - final args = executable.parameterTypes.collect { Type.getType(it) }.toArray(new Type[0]) as Type[] - if (executable instanceof Constructor) { - methodName = '' - methodType = Type.getMethodType(Type.VOID_TYPE, args) - } else { - final method = executable as Method - methodName = method.name - methodType = Type.getMethodType(Type.getType(method.getReturnType()), args) - } - } - - private MethodType build() { - return new MethodType(owner, methodName, methodType) - } - } -} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/CallSiteSpecificationTest.groovy b/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/CallSiteSpecificationTest.groovy deleted file mode 100644 index 25a44004759..00000000000 --- a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/CallSiteSpecificationTest.groovy +++ /dev/null @@ -1,45 +0,0 @@ -package datadog.trace.plugin.csi.impl - -import datadog.trace.agent.tooling.csi.CallSiteAdvice -import datadog.trace.plugin.csi.util.ErrorCode -import org.objectweb.asm.Type -import datadog.trace.plugin.csi.impl.CallSiteSpecification.AdviceSpecification - -class CallSiteSpecificationTest extends BaseCsiPluginTest { - - def 'test call site spi should be an interface'() { - setup: - final context = mockValidationContext() - final spec = new CallSiteSpecification(Type.getType(String), [Mock(AdviceSpecification)], [Type.getType(String)] as Set, [] as List, [] as Set) - - when: - spec.validate(context) - - then: - 1 * context.addError(ErrorCode.CALL_SITE_SPI_SHOULD_BE_AN_INTERFACE, _) - } - - def 'test call site spi should not define any methods'() { - setup: - final context = mockValidationContext() - final spec = new CallSiteSpecification(Type.getType(String), [Mock(AdviceSpecification)], [Type.getType(Comparable)] as Set, [] as List, [] as Set) - - when: - spec.validate(context) - - then: - 1 * context.addError(ErrorCode.CALL_SITE_SPI_SHOULD_BE_EMPTY, _) - } - - def 'test call site should have advices'() { - setup: - final context = mockValidationContext() - final spec = new CallSiteSpecification(Type.getType(String), [], [Type.getType(CallSiteAdvice)] as Set, [] as List, [] as Set) - - when: - spec.validate(context) - - then: - 1 * context.addError(ErrorCode.CALL_SITE_SHOULD_HAVE_ADVICE_METHODS, _) - } -} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/RegexpAdvicePointcutParserTest.groovy b/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/RegexpAdvicePointcutParserTest.groovy deleted file mode 100644 index e7fdb727b38..00000000000 --- a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/RegexpAdvicePointcutParserTest.groovy +++ /dev/null @@ -1,136 +0,0 @@ -package datadog.trace.plugin.csi.impl - -import spock.lang.Specification - -final class RegexpAdvicePointcutParserTest extends Specification { - - def 'resolve constructor'() { - setup: - final pointcutParser = new RegexpAdvicePointcutParser() - - when: - final signature = pointcutParser.parse("void datadog.trace.plugin.csi.samples.SignatureParserExample.()") - - then: - signature.owner.className == 'datadog.trace.plugin.csi.samples.SignatureParserExample' - signature.methodName == '' - signature.methodType.descriptor == '()V' - } - - def 'resolve constructor with args'() { - setup: - final pointcutParser = new RegexpAdvicePointcutParser() - - when: - final signature = pointcutParser.parse("void datadog.trace.plugin.csi.samples.SignatureParserExample.(java.lang.String)") - - then: - signature.owner.className == 'datadog.trace.plugin.csi.samples.SignatureParserExample' - signature.methodName == '' - signature.methodType.descriptor == '(Ljava/lang/String;)V' - } - - def 'resolve without args'() { - setup: - final pointcutParser = new RegexpAdvicePointcutParser() - - when: - final signature = pointcutParser.parse("java.lang.String datadog.trace.plugin.csi.samples.SignatureParserExample.noParams()") - - then: - signature.owner.className == 'datadog.trace.plugin.csi.samples.SignatureParserExample' - signature.methodName == 'noParams' - signature.methodType.descriptor == '()Ljava/lang/String;' - } - - def 'resolve one param'() { - setup: - final pointcutParser = new RegexpAdvicePointcutParser() - - when: - final signature = pointcutParser.parse("java.lang.String datadog.trace.plugin.csi.samples.SignatureParserExample.oneParam(java.util.Map)") - - then: - signature.owner.className == 'datadog.trace.plugin.csi.samples.SignatureParserExample' - signature.methodName == 'oneParam' - signature.methodType.descriptor == '(Ljava/util/Map;)Ljava/lang/String;' - } - - def 'resolve multiple params'() { - setup: - final pointcutParser = new RegexpAdvicePointcutParser() - - when: - final signature = pointcutParser.parse("java.lang.String datadog.trace.plugin.csi.samples.SignatureParserExample.multipleParams(java.lang.String, int, java.util.List)") - - then: - signature.owner.className == 'datadog.trace.plugin.csi.samples.SignatureParserExample' - signature.methodName == 'multipleParams' - signature.methodType.descriptor == '(Ljava/lang/String;ILjava/util/List;)Ljava/lang/String;' - } - - def 'resolve varargs'() { - setup: - final pointcutParser = new RegexpAdvicePointcutParser() - - when: - final signature = pointcutParser.parse("java.lang.String datadog.trace.plugin.csi.samples.SignatureParserExample.varargs(java.lang.String[])") - - then: - signature.owner.className == 'datadog.trace.plugin.csi.samples.SignatureParserExample' - signature.methodName == 'varargs' - signature.methodType.descriptor == '([Ljava/lang/String;)Ljava/lang/String;' - } - - def 'resolve primitive'() { - setup: - final pointcutParser = new RegexpAdvicePointcutParser() - - when: - final signature = pointcutParser.parse("int datadog.trace.plugin.csi.samples.SignatureParserExample.primitive()") - - then: - signature.owner.className == 'datadog.trace.plugin.csi.samples.SignatureParserExample' - signature.methodName == 'primitive' - signature.methodType.descriptor == '()I' - } - - def 'resolve primitive array type'() { - setup: - final pointcutParser = new RegexpAdvicePointcutParser() - - when: - final signature = pointcutParser.parse("byte[] datadog.trace.plugin.csi.samples.SignatureParserExample.primitiveArray()") - - then: - signature.owner.className == 'datadog.trace.plugin.csi.samples.SignatureParserExample' - signature.methodName == 'primitiveArray' - signature.methodType.descriptor == '()[B' - } - - def 'resolve object array type'() { - setup: - final pointcutParser = new RegexpAdvicePointcutParser() - - when: - final signature = pointcutParser.parse("java.lang.Object[] datadog.trace.plugin.csi.samples.SignatureParserExample.objectArray()") - - then: - signature.owner.className == 'datadog.trace.plugin.csi.samples.SignatureParserExample' - signature.methodName == 'objectArray' - signature.methodType.descriptor == '()[Ljava/lang/Object;' - } - - def 'resolve multi dimensional object array type'() { - setup: - final pointcutParser = new RegexpAdvicePointcutParser() - - when: - final signature = pointcutParser.parse("java.lang.Object[][][] datadog.trace.plugin.csi.samples.SignatureParserExample.objectArray()") - - then: - signature.owner.className == 'datadog.trace.plugin.csi.samples.SignatureParserExample' - signature.methodName == 'objectArray' - signature.methodType.descriptor == '()[[[Ljava/lang/Object;' - } -} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/TypeResolverPoolTest.groovy b/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/TypeResolverPoolTest.groovy deleted file mode 100644 index ffeeb6b7f40..00000000000 --- a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/TypeResolverPoolTest.groovy +++ /dev/null @@ -1,109 +0,0 @@ -package datadog.trace.plugin.csi.impl - -import datadog.trace.plugin.csi.util.MethodType -import org.objectweb.asm.Type -import spock.lang.Specification - -import javax.servlet.ServletRequest -import javax.servlet.http.HttpServletRequest - -class TypeResolverPoolTest extends Specification { - - def 'test resolve primitive'() { - setup: - final resolver = new TypeResolverPool() - - when: - final result = resolver.resolveType(Type.INT_TYPE) - - then: - result == int.class - } - - def 'test resolve primitive array'() { - setup: - final resolver = new TypeResolverPool() - final type = Type.getType('[I') - - when: - final result = resolver.resolveType(type) - - then: - result == int[].class - } - - def 'test resolve primitive multidimensional array'() { - setup: - final resolver = new TypeResolverPool() - final type = Type.getType('[[[I') - - when: - final result = resolver.resolveType(type) - - then: - result == int[][][].class - } - - def 'test resolve class'() { - setup: - final resolver = new TypeResolverPool() - final type = Type.getType(String) - - when: - final result = resolver.resolveType(type) - - then: - result == String - } - - - def 'test resolve class array'() { - setup: - final resolver = new TypeResolverPool() - final type = Type.getType(String[]) - - when: - final result = resolver.resolveType(type) - - then: - result == String[] - } - - def 'test resolve class multidimensional array'() { - setup: - final resolver = new TypeResolverPool() - final type = Type.getType(String[][][]) - - when: - final result = resolver.resolveType(type) - - then: - result == String[][][] - } - - def 'test type resolver from method'() { - setup: - final resolver = new TypeResolverPool() - final type = Type.getMethodType(Type.getType(String[]), Type.getType(String), Type.getType(String)) - - when: - final result = resolver.resolveType(type.getReturnType()) - - then: - result == String[] - } - - def 'test inherited methods'() { - setup: - final resolver = new TypeResolverPool() - final owner = Type.getType(HttpServletRequest) - final name = 'getParameter' - final descriptor = Type.getMethodType(Type.getType(String), Type.getType(String)) - - when: - final result = resolver.resolveMethod(new MethodType(owner, name, descriptor)) - - then: - result == ServletRequest.getDeclaredMethod('getParameter', String) - } -} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/assertion/AdviceAssert.groovy b/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/assertion/AdviceAssert.groovy deleted file mode 100644 index d83120a6c52..00000000000 --- a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/assertion/AdviceAssert.groovy +++ /dev/null @@ -1,23 +0,0 @@ -package datadog.trace.plugin.csi.impl.assertion - -class AdviceAssert { - protected String type - protected String owner - protected String method - protected String descriptor - protected Collection statements - - void type(String type) { - assert type == this.type - } - - void pointcut(String owner, String method, String descriptor) { - assert owner == this.owner - assert method == this.method - assert descriptor == this.descriptor - } - - void statements(String... values) { - assert values.toList() == statements - } -} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/assertion/AssertBuilder.groovy b/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/assertion/AssertBuilder.groovy deleted file mode 100644 index f7dae8920cd..00000000000 --- a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/assertion/AssertBuilder.groovy +++ /dev/null @@ -1,122 +0,0 @@ -package datadog.trace.plugin.csi.impl.assertion - -import com.github.javaparser.JavaParser -import com.github.javaparser.ParserConfiguration -import com.github.javaparser.ast.CompilationUnit -import com.github.javaparser.ast.Node -import com.github.javaparser.ast.body.ClassOrInterfaceDeclaration -import com.github.javaparser.ast.body.MethodDeclaration -import com.github.javaparser.ast.expr.MethodCallExpr -import com.github.javaparser.symbolsolver.JavaSymbolSolver -import datadog.trace.agent.tooling.csi.CallSites - -import java.lang.reflect.Executable -import java.lang.reflect.Method - -import static datadog.trace.plugin.csi.impl.CallSiteFactory.typeResolver -import static datadog.trace.plugin.csi.util.CallSiteUtils.classNameToType - -class AssertBuilder { - private final File file - - AssertBuilder(final File file) { - this.file = file - } - - C build() { - final javaFile = parseJavaFile(file) - assert javaFile.parsed == Node.Parsedness.PARSED - final targetType = javaFile.primaryType.get().asClassOrInterfaceDeclaration() - final interfaces = getInterfaces(targetType) - def (enabled, enabledArgs) = getEnabledDeclaration(targetType, interfaces) - return (C) new CallSiteAssert([ - interfaces : getInterfaces(targetType), - spi : getSpi(targetType), - helpers : getHelpers(targetType), - advices : getAdvices(targetType), - enabled : enabled, - enabledArgs: enabledArgs - ]) - } - - protected Set> getSpi(final ClassOrInterfaceDeclaration type) { - return type.getAnnotationByName('AutoService').get().asNormalAnnotationExpr() - .collect { it.pairs.find { it.name.toString() == 'value' }.value.asArrayInitializerExpr() } - .collectMany { it.getValues() } - .collect { it.asClassExpr().getType().resolve().typeDeclaration.get().clazz } - .toSet() - } - - protected Set> getInterfaces(final ClassOrInterfaceDeclaration type) { - return type.asClassOrInterfaceDeclaration().implementedTypes.collect { - final resolved = it.asClassOrInterfaceType().resolve() - return resolved.typeDeclaration.get().clazz - }.toSet() - } - - protected def getEnabledDeclaration(final ClassOrInterfaceDeclaration type, final Set> interfaces) { - if (!interfaces.contains(CallSites.HasEnabledProperty)) { - return [null, null] - } - final isEnabled = type.getMethodsByName('isEnabled').first() - // JavaParser's NodeList has method getFirst() returning an Optional, however with Java 21's - // SequencedCollection, Groovy picks the getFirst() that returns the object itself. - // Using `first()` rather than `first` picks the groovy method instead, fixing the situation. - final returnStatement = isEnabled.body.get().statements.first().asReturnStmt() - final enabledMethodCall = returnStatement.expression.get().asMethodCallExpr() - final enabled = resolveMethod(enabledMethodCall) - final enabledArgs = enabledMethodCall.getArguments().collect { it.asStringLiteralExpr().asString() }.toSet() - return [enabled, enabledArgs] - } - - protected Set> getHelpers(final ClassOrInterfaceDeclaration type) { - final acceptMethod = type.getMethodsByName('accept').first() - final methodCalls = getMethodCalls(acceptMethod) - return methodCalls.findAll { - it.nameAsString == 'addHelpers' - }.collectMany { - it.arguments - }.collect { - typeResolver().resolveType(classNameToType(it.asStringLiteralExpr().asString())) - }.toSet() - } - - protected List getAdvices(final ClassOrInterfaceDeclaration type) { - final acceptMethod = type.getMethodsByName('accept').first() - return getMethodCalls(acceptMethod).findAll { - it.nameAsString == 'addAdvice' - }.collect { - final adviceType = it.arguments.get(0).asFieldAccessExpr().getName() - def (owner, method, descriptor) = it.arguments.subList(1, 4)*.asStringLiteralExpr()*.asString() - final handlerLambda = it.arguments[4].asLambdaExpr() - final advice = handlerLambda.body.asBlockStmt().statements*.toString() - return new AdviceAssert([ - type : adviceType, - owner : owner, - method : method, - descriptor: descriptor, - statements: advice - ]) - } - } - - protected static List getMethodCalls(final MethodDeclaration method) { - return method.body.get().asBlockStmt().getStatements().findAll { - it.isExpressionStmt() && it.asExpressionStmt().getExpression().isMethodCallExpr() - }.collect { - it.asExpressionStmt().getExpression().asMethodCallExpr() - } - } - - private static Executable resolveMethod(final MethodCallExpr methodCallExpr) { - final resolved = methodCallExpr.resolve() - return resolved.@method as Method - } - - private static CompilationUnit parseJavaFile(final File file) - throws FileNotFoundException { - final JavaSymbolSolver solver = new JavaSymbolSolver(typeResolver()); - final JavaParser parser = new JavaParser(new ParserConfiguration().setSymbolResolver(solver)); - return parser.parse(file).getResult().get(); - } -} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/assertion/CallSiteAssert.groovy b/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/assertion/CallSiteAssert.groovy deleted file mode 100644 index fdd557adc95..00000000000 --- a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/assertion/CallSiteAssert.groovy +++ /dev/null @@ -1,42 +0,0 @@ -package datadog.trace.plugin.csi.impl.assertion - -import java.lang.reflect.Method - -import static java.util.Arrays.asList - -class CallSiteAssert { - - protected Set> interfaces - protected Set> spi - protected Set> helpers - protected Collection advices - protected Method enabled - protected Set enabledArgs - - void interfaces(Class... values) { - assertSameElements(interfaces, values) - } - - void helpers(Class... values) { - assertSameElements(helpers, values) - } - - void spi(Class...values) { - assertSameElements(spi, values) - } - - void advices(int index, @DelegatesTo(AdviceAssert) Closure closure) { - final asserter = advices[index] - closure.delegate = asserter - closure(asserter) - } - - void enabled(Method method, String... args) { - assert method == enabled - assertSameElements(enabledArgs, args) - } - - private static void assertSameElements(final Set expected, final E...received) { - assert received.length == expected.size() && expected.containsAll(asList(received)) - } -} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/ext/IastExtensionTest.groovy b/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/ext/IastExtensionTest.groovy deleted file mode 100644 index bcd28fad4e4..00000000000 --- a/buildSrc/call-site-instrumentation-plugin/src/test/groovy/datadog/trace/plugin/csi/impl/ext/IastExtensionTest.groovy +++ /dev/null @@ -1,244 +0,0 @@ -package datadog.trace.plugin.csi.impl.ext - -import com.github.javaparser.JavaParser -import com.github.javaparser.ast.body.ClassOrInterfaceDeclaration -import com.github.javaparser.ast.stmt.IfStmt -import datadog.trace.agent.tooling.csi.CallSites -import datadog.trace.plugin.csi.AdviceGenerator -import datadog.trace.plugin.csi.PluginApplication.Configuration -import datadog.trace.plugin.csi.impl.AdviceGeneratorImpl -import datadog.trace.plugin.csi.impl.BaseCsiPluginTest -import datadog.trace.plugin.csi.impl.CallSiteSpecification -import datadog.trace.plugin.csi.impl.assertion.AdviceAssert -import datadog.trace.plugin.csi.impl.assertion.AssertBuilder -import datadog.trace.plugin.csi.impl.assertion.CallSiteAssert -import datadog.trace.plugin.csi.impl.ext.tests.IastExtensionCallSite -import datadog.trace.plugin.csi.impl.ext.tests.SourceTypes -import groovy.transform.CompileDynamic -import spock.lang.TempDir - -import java.nio.file.Files -import java.nio.file.Path -import java.nio.file.Paths - -import static datadog.trace.plugin.csi.impl.CallSiteFactory.pointcutParser -import static datadog.trace.plugin.csi.util.CallSiteUtils.classNameToType - -@CompileDynamic -class IastExtensionTest extends BaseCsiPluginTest { - - @TempDir - private File buildDir - private Path targetFolder - private Path projectFolder - private Path srcFolder - - void setup() { - targetFolder = buildDir.toPath().resolve('target') - Files.createDirectories(targetFolder) - projectFolder = buildDir.toPath().resolve('project') - Files.createDirectories(projectFolder) - srcFolder = projectFolder.resolve('src/main/java') - Files.createDirectories(srcFolder) - } - - void 'test that extension only applies to iast advices'() { - setup: - final type = classNameToType(typeName) - final callSite = Mock(CallSiteSpecification) { - getSpi() >> type - } - final extension = new IastExtension() - - when: - final applies = extension.appliesTo(callSite) - - then: - applies == expected - - where: - typeName | expected - CallSites.name | false - IastExtension.IAST_CALL_SITES_FQCN | true - } - - void 'test that extension generates a call site with telemetry'() { - setup: - final config = Mock(Configuration) { - getTargetFolder() >> targetFolder - getSrcFolder() >> getCallSiteSrcFolder() - getClassPath() >> [] - } - final spec = buildClassSpecification(IastExtensionCallSite) - final generator = buildAdviceGenerator(buildDir) - final result = generator.generate(spec) - if (!result.success) { - throw new IllegalArgumentException("Error with call site ${IastExtensionCallSite}") - } - final extension = new IastExtension() - - when: - extension.apply(config, result) - - then: 'the call site provider is modified with telemetry' - assertNoErrors result - assertCallSites(result.file) { - advices(0) { - pointcut('javax/servlet/http/HttpServletRequest', 'getHeader', '(Ljava/lang/String;)Ljava/lang/String;') - instrumentedMetric('IastMetric.INSTRUMENTED_SOURCE') { - metricStatements('IastMetricCollector.add(IastMetric.INSTRUMENTED_SOURCE, (byte) 3, 1);') - } - executedMetric('IastMetric.EXECUTED_SOURCE') { - metricStatements( - 'handler.field(net.bytebuddy.jar.asm.Opcodes.GETSTATIC, "datadog/trace/api/iast/telemetry/IastMetric", "EXECUTED_SOURCE", "Ldatadog/trace/api/iast/telemetry/IastMetric;");', - 'handler.instruction(net.bytebuddy.jar.asm.Opcodes.ICONST_3);', - 'handler.instruction(net.bytebuddy.jar.asm.Opcodes.ICONST_1);', - 'handler.method(net.bytebuddy.jar.asm.Opcodes.INVOKESTATIC, "datadog/trace/api/iast/telemetry/IastMetricCollector", "add", "(Ldatadog/trace/api/iast/telemetry/IastMetric;BI)V", false);' - ) - } - } - advices(1) { - pointcut('javax/servlet/http/HttpServletRequest', 'getInputStream', '()Ljavax/servlet/ServletInputStream;') - instrumentedMetric('IastMetric.INSTRUMENTED_SOURCE') { - metricStatements('IastMetricCollector.add(IastMetric.INSTRUMENTED_SOURCE, (byte) 127, 1);') - } - executedMetric('IastMetric.EXECUTED_SOURCE') { - metricStatements( - 'handler.field(net.bytebuddy.jar.asm.Opcodes.GETSTATIC, "datadog/trace/api/iast/telemetry/IastMetric", "EXECUTED_SOURCE", "Ldatadog/trace/api/iast/telemetry/IastMetric;");', - 'handler.instruction(net.bytebuddy.jar.asm.Opcodes.BIPUSH, 127);', - 'handler.instruction(net.bytebuddy.jar.asm.Opcodes.ICONST_1);', - 'handler.method(net.bytebuddy.jar.asm.Opcodes.INVOKESTATIC, "datadog/trace/api/iast/telemetry/IastMetricCollector", "add", "(Ldatadog/trace/api/iast/telemetry/IastMetric;BI)V", false);' - ) - } - } - advices(2) { - pointcut('javax/servlet/ServletRequest', 'getReader', '()Ljava/io/BufferedReader;') - instrumentedMetric('IastMetric.INSTRUMENTED_PROPAGATION') { - metricStatements('IastMetricCollector.add(IastMetric.INSTRUMENTED_PROPAGATION, 1);') - } - executedMetric('IastMetric.EXECUTED_PROPAGATION') { - metricStatements( - 'handler.field(net.bytebuddy.jar.asm.Opcodes.GETSTATIC, "datadog/trace/api/iast/telemetry/IastMetric", "EXECUTED_PROPAGATION", "Ldatadog/trace/api/iast/telemetry/IastMetric;");', - 'handler.instruction(net.bytebuddy.jar.asm.Opcodes.ICONST_1);', - 'handler.method(net.bytebuddy.jar.asm.Opcodes.INVOKESTATIC, "datadog/trace/api/iast/telemetry/IastMetricCollector", "add", "(Ldatadog/trace/api/iast/telemetry/IastMetric;I)V", false);' - ) - } - } - } - } - - private static AdviceGenerator buildAdviceGenerator(final File targetFolder) { - return new AdviceGeneratorImpl(targetFolder, pointcutParser()) - } - - private static Path getCallSiteSrcFolder() { - final file = Thread.currentThread().contextClassLoader.getResource('') - return Paths.get(file.toURI()).resolve('../../../../src/test/java') - } - - private static ClassOrInterfaceDeclaration parse(final File path) { - final parsedAdvice = new JavaParser().parse(path).getResult().get() - return parsedAdvice.primaryType.get().asClassOrInterfaceDeclaration() - } - - private static void assertCallSites(final File generated, @DelegatesTo(IastExtensionCallSiteAssert) final Closure closure) { - final asserter = new IastExtensionAssertBuilder(generated).build() - closure.delegate = asserter - closure(asserter) - } - - static class IastExtensionCallSiteAssert extends CallSiteAssert { - - IastExtensionCallSiteAssert(CallSiteAssert base) { - interfaces = base.interfaces - spi = base.spi - helpers = base.helpers - advices = base.advices - enabled = base.enabled - enabledArgs = base.enabledArgs - } - - void advices(int index, @DelegatesTo(IastExtensionAdviceAssert) Closure closure) { - final asserter = advices[index] - closure.delegate = asserter - closure(asserter) - } - - void advices(@DelegatesTo(IastExtensionAdviceAssert) Closure closure) { - advices.each { - closure.delegate = it - closure(it) - } - } - } - - static class IastExtensionAdviceAssert extends AdviceAssert { - - protected IastExtensionMetricAsserter instrumented - protected IastExtensionMetricAsserter executed - - void instrumentedMetric(final String metric, @DelegatesTo(IastExtensionMetricAsserter) Closure closure) { - assert metric == instrumented.metric - closure.delegate = instrumented - closure(instrumented) - } - - void executedMetric(final String metric, @DelegatesTo(IastExtensionMetricAsserter) Closure closure) { - assert metric == executed.metric - closure.delegate = executed - closure(executed) - } - } - - static class IastExtensionMetricAsserter { - protected String metric - protected Collection statements - - void metricStatements(String... values) { - assert values.toList() == statements - } - } - - static class IastExtensionAssertBuilder extends AssertBuilder { - - IastExtensionAssertBuilder(File file) { - super(file) - } - - @Override - IastExtensionCallSiteAssert build() { - final base = super.build() - return new IastExtensionCallSiteAssert(base) - } - - @Override - protected List getAdvices(ClassOrInterfaceDeclaration type) { - final acceptMethod = type.getMethodsByName('accept').first() - return getMethodCalls(acceptMethod).findAll { - it.nameAsString == 'addAdvice' - }.collect { - def (owner, method, descriptor) = it.arguments.subList(1, 4)*.asStringLiteralExpr()*.asString() - final handlerLambda = it.arguments[4].asLambdaExpr() - final statements = handlerLambda.body.asBlockStmt().statements - final instrumentedStmt = statements.get(0).asIfStmt() - final executedStmt = statements.get(1).asIfStmt() - return new IastExtensionAdviceAssert([ - owner : owner, - method : method, - descriptor: descriptor, - instrumented : buildMetricAsserter(instrumentedStmt), - executed: buildMetricAsserter(executedStmt), - statements: statements.findAll { !it.isIfStmt() } - ]) - } - } - - protected IastExtensionMetricAsserter buildMetricAsserter(final IfStmt ifStmt) { - final condition = ifStmt.getCondition().asMethodCallExpr() - return new IastExtensionMetricAsserter( - metric: condition.getScope().get().toString(), - statements: ifStmt.getThenStmt().asBlockStmt().statements*.toString() - ) - } - } -} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/AdviceGeneratorTest.java b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/AdviceGeneratorTest.java new file mode 100644 index 00000000000..3206b3c9b31 --- /dev/null +++ b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/AdviceGeneratorTest.java @@ -0,0 +1,578 @@ +package datadog.trace.plugin.csi.impl; + +import static datadog.trace.plugin.csi.impl.CallSiteFactory.pointcutParser; + +import datadog.trace.agent.tooling.csi.CallSite; +import datadog.trace.agent.tooling.csi.CallSites; +import datadog.trace.plugin.csi.AdviceGenerator; +import datadog.trace.plugin.csi.AdviceGenerator.CallSiteResult; +import datadog.trace.plugin.csi.impl.assertion.AssertBuilder; +import datadog.trace.plugin.csi.impl.assertion.CallSiteAssert; +import datadog.trace.plugin.csi.impl.ext.tests.IastCallSites; +import datadog.trace.plugin.csi.impl.ext.tests.RaspCallSites; +import java.io.File; +import java.lang.invoke.MethodHandles; +import java.lang.invoke.MethodType; +import java.net.URL; +import java.util.Map; +import javax.servlet.ServletRequest; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledForJreRange; +import org.junit.jupiter.api.condition.JRE; +import org.junit.jupiter.api.io.TempDir; + +class AdviceGeneratorTest extends BaseCsiPluginTest { + + @TempDir private File buildDir; + + @CallSite(spi = CallSites.class) + public static class BeforeAdvice { + @CallSite.Before( + "java.security.MessageDigest java.security.MessageDigest.getInstance(java.lang.String)") + public static void before(@CallSite.Argument String algorithm) {} + } + + @Test + void testBeforeAdvice() { + CallSiteSpecification spec = buildClassSpecification(BeforeAdvice.class); + AdviceGenerator generator = buildAdviceGenerator(buildDir); + + CallSiteResult result = generator.generate(spec); + + assertNoErrors(result); + CallSiteAssert asserter = assertCallSites(result.getFile()); + asserter.interfaces(CallSites.class); + asserter.helpers(BeforeAdvice.class); + asserter.advices( + 0, + advice -> { + advice.type("BEFORE"); + advice.pointcut( + "java/security/MessageDigest", + "getInstance", + "(Ljava/lang/String;)Ljava/security/MessageDigest;"); + advice.statements( + "handler.dupParameters(descriptor, StackDupMode.COPY);", + "handler.advice(\"datadog/trace/plugin/csi/impl/AdviceGeneratorTest$BeforeAdvice\", \"before\", \"(Ljava/lang/String;)V\");", + "handler.method(opcode, owner, name, descriptor, isInterface);"); + }); + } + + @CallSite(spi = CallSites.class) + public static class AroundAdvice { + @CallSite.Around( + "java.lang.String java.lang.String.replaceAll(java.lang.String, java.lang.String)") + public static String around( + @CallSite.This String self, + @CallSite.Argument String regexp, + @CallSite.Argument String replacement) { + return self.replaceAll(regexp, replacement); + } + } + + @Test + void testAroundAdvice() { + CallSiteSpecification spec = buildClassSpecification(AroundAdvice.class); + AdviceGenerator generator = buildAdviceGenerator(buildDir); + + CallSiteResult result = generator.generate(spec); + + assertNoErrors(result); + CallSiteAssert asserter = assertCallSites(result.getFile()); + asserter.interfaces(CallSites.class); + asserter.helpers(AroundAdvice.class); + asserter.advices( + 0, + advice -> { + advice.type("AROUND"); + advice.pointcut( + "java/lang/String", + "replaceAll", + "(Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;"); + advice.statements( + "handler.advice(\"datadog/trace/plugin/csi/impl/AdviceGeneratorTest$AroundAdvice\", \"around\", \"(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;\");"); + }); + } + + @CallSite(spi = CallSites.class) + public static class AfterAdvice { + @CallSite.After("java.lang.String java.lang.String.concat(java.lang.String)") + public static String after( + @CallSite.This String self, + @CallSite.Argument String param, + @CallSite.Return String result) { + return result; + } + } + + @Test + void testAfterAdvice() { + CallSiteSpecification spec = buildClassSpecification(AfterAdvice.class); + AdviceGenerator generator = buildAdviceGenerator(buildDir); + + CallSiteResult result = generator.generate(spec); + + assertNoErrors(result); + CallSiteAssert asserter = assertCallSites(result.getFile()); + asserter.interfaces(CallSites.class); + asserter.helpers(AfterAdvice.class); + asserter.advices( + 0, + advice -> { + advice.type("AFTER"); + advice.pointcut("java/lang/String", "concat", "(Ljava/lang/String;)Ljava/lang/String;"); + advice.statements( + "handler.dupInvoke(owner, descriptor, StackDupMode.COPY);", + "handler.method(opcode, owner, name, descriptor, isInterface);", + "handler.advice(\"datadog/trace/plugin/csi/impl/AdviceGeneratorTest$AfterAdvice\", \"after\", \"(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;\");"); + }); + } + + @CallSite(spi = CallSites.class) + public static class AfterAdviceCtor { + @CallSite.After("void java.net.URL.(java.lang.String)") + public static URL after(@CallSite.AllArguments Object[] args, @CallSite.Return URL url) { + return url; + } + } + + @Test + void testAfterAdviceCtor() { + CallSiteSpecification spec = buildClassSpecification(AfterAdviceCtor.class); + AdviceGenerator generator = buildAdviceGenerator(buildDir); + + CallSiteResult result = generator.generate(spec); + + assertNoErrors(result); + CallSiteAssert asserter = assertCallSites(result.getFile()); + asserter.interfaces(CallSites.class); + asserter.helpers(AfterAdviceCtor.class); + asserter.advices( + 0, + advice -> { + advice.pointcut("java/net/URL", "", "(Ljava/lang/String;)V"); + advice.statements( + "handler.dupParameters(descriptor, StackDupMode.PREPEND_ARRAY_CTOR);", + "handler.method(opcode, owner, name, descriptor, isInterface);", + "handler.advice(\"datadog/trace/plugin/csi/impl/AdviceGeneratorTest$AfterAdviceCtor\", \"after\", \"([Ljava/lang/Object;Ljava/net/URL;)Ljava/net/URL;\");"); + }); + } + + @CallSite(spi = SpiAdvice.SampleSpi.class) + public static class SpiAdvice { + @CallSite.Before( + "java.security.MessageDigest java.security.MessageDigest.getInstance(java.lang.String)") + public static void before(@CallSite.Argument String algorithm) {} + + interface SampleSpi {} + } + + @Test + void testGeneratorWithSpi() { + CallSiteSpecification spec = buildClassSpecification(SpiAdvice.class); + AdviceGenerator generator = buildAdviceGenerator(buildDir); + + CallSiteResult result = generator.generate(spec); + + assertNoErrors(result); + CallSiteAssert asserter = assertCallSites(result.getFile()); + asserter.interfaces(CallSites.class, SpiAdvice.SampleSpi.class); + } + + @CallSite(spi = CallSites.class) + public static class InvokeDynamicAfterAdvice { + @CallSite.After( + value = + "java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])", + invokeDynamic = true) + public static String after( + @CallSite.AllArguments Object[] arguments, @CallSite.Return String result) { + return result; + } + } + + @Test + @EnabledForJreRange(min = JRE.JAVA_9) + void testInvokeDynamicAfterAdvice() { + CallSiteSpecification spec = buildClassSpecification(InvokeDynamicAfterAdvice.class); + AdviceGenerator generator = buildAdviceGenerator(buildDir); + + CallSiteResult result = generator.generate(spec); + + assertNoErrors(result); + CallSiteAssert asserter = assertCallSites(result.getFile()); + asserter.interfaces(CallSites.class); + asserter.helpers(InvokeDynamicAfterAdvice.class); + asserter.advices( + 0, + advice -> { + advice.pointcut( + "java/lang/invoke/StringConcatFactory", + "makeConcatWithConstants", + "(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/invoke/CallSite;"); + advice.statements( + "handler.dupParameters(descriptor, StackDupMode.PREPEND_ARRAY);", + "handler.invokeDynamic(name, descriptor, bootstrapMethodHandle, bootstrapMethodArguments);", + "handler.advice(\"datadog/trace/plugin/csi/impl/AdviceGeneratorTest$InvokeDynamicAfterAdvice\", \"after\", \"([Ljava/lang/Object;Ljava/lang/String;)Ljava/lang/String;\");"); + }); + } + + @CallSite(spi = CallSites.class) + public static class InvokeDynamicAroundAdvice { + @CallSite.Around( + value = + "java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])", + invokeDynamic = true) + public static java.lang.invoke.CallSite around( + @CallSite.Argument MethodHandles.Lookup lookup, + @CallSite.Argument String name, + @CallSite.Argument MethodType concatType, + @CallSite.Argument String recipe, + @CallSite.Argument Object... constants) { + return null; + } + } + + @Test + @EnabledForJreRange(min = JRE.JAVA_9) + void testInvokeDynamicAroundAdvice() { + CallSiteSpecification spec = buildClassSpecification(InvokeDynamicAroundAdvice.class); + AdviceGenerator generator = buildAdviceGenerator(buildDir); + + CallSiteResult result = generator.generate(spec); + + assertNoErrors(result); + CallSiteAssert asserter = assertCallSites(result.getFile()); + asserter.interfaces(CallSites.class); + asserter.helpers(InvokeDynamicAroundAdvice.class); + asserter.advices( + 0, + advice -> { + advice.pointcut( + "java/lang/invoke/StringConcatFactory", + "makeConcatWithConstants", + "(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/invoke/CallSite;"); + advice.statements( + "handler.invokeDynamic(name, descriptor, new Handle(Opcodes.H_INVOKESTATIC, \"datadog/trace/plugin/csi/impl/AdviceGeneratorTest$InvokeDynamicAroundAdvice\", \"around\", \"(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/invoke/CallSite;\", false), bootstrapMethodArguments);"); + }); + } + + @CallSite(spi = CallSites.class) + public static class InvokeDynamicWithConstantsAdvice { + @CallSite.After( + value = + "java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])", + invokeDynamic = true) + public static String after( + @CallSite.AllArguments Object[] arguments, + @CallSite.Return String result, + @CallSite.InvokeDynamicConstants Object[] constants) { + return result; + } + } + + @Test + @EnabledForJreRange(min = JRE.JAVA_9) + void testInvokeDynamicWithConstantsAdvice() { + CallSiteSpecification spec = buildClassSpecification(InvokeDynamicWithConstantsAdvice.class); + AdviceGenerator generator = buildAdviceGenerator(buildDir); + + CallSiteResult result = generator.generate(spec); + + assertNoErrors(result); + CallSiteAssert asserter = assertCallSites(result.getFile()); + asserter.interfaces(CallSites.class); + asserter.helpers(InvokeDynamicWithConstantsAdvice.class); + asserter.advices( + 0, + advice -> { + advice.pointcut( + "java/lang/invoke/StringConcatFactory", + "makeConcatWithConstants", + "(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/invoke/CallSite;"); + advice.statements( + "handler.dupParameters(descriptor, StackDupMode.PREPEND_ARRAY);", + "handler.invokeDynamic(name, descriptor, bootstrapMethodHandle, bootstrapMethodArguments);", + "handler.loadConstantArray(bootstrapMethodArguments);", + "handler.advice(\"datadog/trace/plugin/csi/impl/AdviceGeneratorTest$InvokeDynamicWithConstantsAdvice\", \"after\", \"([Ljava/lang/Object;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/String;\");"); + }); + } + + @CallSite(spi = CallSites.class) + public static class ArrayAdvice { + @CallSite.AfterArray({ + @CallSite.After("java.util.Map javax.servlet.ServletRequest.getParameterMap()"), + @CallSite.After("java.util.Map javax.servlet.ServletRequestWrapper.getParameterMap()") + }) + public static Map after( + @CallSite.This ServletRequest request, @CallSite.Return Map parameters) { + return parameters; + } + } + + @Test + void testArrayAdvice() { + CallSiteSpecification spec = buildClassSpecification(ArrayAdvice.class); + AdviceGenerator generator = buildAdviceGenerator(buildDir); + + CallSiteResult result = generator.generate(spec); + + assertNoErrors(result); + CallSiteAssert asserter = assertCallSites(result.getFile()); + asserter.advices( + 0, + advice -> { + advice.pointcut("javax/servlet/ServletRequest", "getParameterMap", "()Ljava/util/Map;"); + }); + asserter.advices( + 1, + advice -> { + advice.pointcut( + "javax/servlet/ServletRequestWrapper", "getParameterMap", "()Ljava/util/Map;"); + }); + } + + public static class MinJavaVersionCheck { + public static boolean isAtLeast(String version) { + return Integer.parseInt(version) >= 9; + } + } + + @CallSite( + spi = CallSites.class, + enabled = { + "datadog.trace.plugin.csi.impl.AdviceGeneratorTest$MinJavaVersionCheck", + "isAtLeast", + "18" + }) + public static class MinJavaVersionAdvice { + @CallSite.After("java.lang.String java.lang.String.concat(java.lang.String)") + public static String after( + @CallSite.This String self, + @CallSite.Argument String param, + @CallSite.Return String result) { + return result; + } + } + + @Test + void testCustomEnabledProperty() throws Exception { + CallSiteSpecification spec = buildClassSpecification(MinJavaVersionAdvice.class); + AdviceGenerator generator = buildAdviceGenerator(buildDir); + + CallSiteResult result = generator.generate(spec); + + assertNoErrors(result); + CallSiteAssert asserter = assertCallSites(result.getFile()); + asserter.interfaces(CallSites.class, CallSites.HasEnabledProperty.class); + asserter.enabled(MinJavaVersionCheck.class.getDeclaredMethod("isAtLeast", String.class), "18"); + } + + @CallSite(spi = CallSites.class) + public static class PartialArgumentsBeforeAdvice { + @CallSite.Before("int java.sql.Statement.executeUpdate(java.lang.String, java.lang.String[])") + public static void before(@CallSite.Argument(0) String arg1) {} + + @CallSite.Before( + "java.lang.String java.lang.String.format(java.lang.String, java.lang.Object[])") + public static void before(@CallSite.Argument(1) Object[] arg) {} + + @CallSite.Before("java.lang.CharSequence java.lang.String.subSequence(int, int)") + public static void before(@CallSite.This String thiz, @CallSite.Argument(0) int arg) {} + } + + @Test + void partialArgumentsWithBeforeAdvice() { + CallSiteSpecification spec = buildClassSpecification(PartialArgumentsBeforeAdvice.class); + AdviceGenerator generator = buildAdviceGenerator(buildDir); + + CallSiteResult result = generator.generate(spec); + + assertNoErrors(result); + CallSiteAssert asserter = assertCallSites(result.getFile()); + asserter.advices( + 0, + advice -> { + advice.pointcut( + "java/sql/Statement", "executeUpdate", "(Ljava/lang/String;[Ljava/lang/String;)I"); + advice.statements( + "int[] parameterIndices = new int[] { 0 };", + "handler.dupParameters(descriptor, parameterIndices, owner);", + "handler.advice(\"datadog/trace/plugin/csi/impl/AdviceGeneratorTest$PartialArgumentsBeforeAdvice\", \"before\", \"(Ljava/lang/String;)V\");", + "handler.method(opcode, owner, name, descriptor, isInterface);"); + }); + asserter.advices( + 1, + advice -> { + advice.pointcut( + "java/lang/String", + "format", + "(Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/String;"); + advice.statements( + "int[] parameterIndices = new int[] { 1 };", + "handler.dupParameters(descriptor, parameterIndices, null);", + "handler.advice(\"datadog/trace/plugin/csi/impl/AdviceGeneratorTest$PartialArgumentsBeforeAdvice\", \"before\", \"([Ljava/lang/Object;)V\");", + "handler.method(opcode, owner, name, descriptor, isInterface);"); + }); + asserter.advices( + 2, + advice -> { + advice.pointcut("java/lang/String", "subSequence", "(II)Ljava/lang/CharSequence;"); + advice.statements( + "int[] parameterIndices = new int[] { 0 };", + "handler.dupInvoke(owner, descriptor, parameterIndices);", + "handler.advice(\"datadog/trace/plugin/csi/impl/AdviceGeneratorTest$PartialArgumentsBeforeAdvice\", \"before\", \"(Ljava/lang/String;I)V\");", + "handler.method(opcode, owner, name, descriptor, isInterface);"); + }); + } + + /** + * Captures two of three arguments positionally. Before the fix, Stream.sorted() without a + * comparator threw ClassCastException (ArgumentSpecification is not Comparable). The parameters + * TreeMap is keyed by advice-parameter index, so the stream is already in the correct order — + * sorted() must not be called. The reversed case verifies that parameterIndices follows the + * advice signature order, not the pointcut index order. + */ + @CallSite(spi = CallSites.class) + public static class MultiplePartialArgumentsBeforeAdvice { + /** Captures args 0 and 1 in the same order as the pointcut. */ + @CallSite.Before( + "java.lang.String java.lang.String.format(java.util.Locale, java.lang.String, java.lang.Object[])") + public static void before( + @CallSite.Argument(0) java.util.Locale locale, @CallSite.Argument(1) String format) {} + + /** + * Captures the same two args but with their advice positions reversed. parameterIndices must be + * {1, 0} (advice order), not {0, 1} (pointcut index order). + */ + @CallSite.Before( + "java.lang.String java.lang.String.format(java.util.Locale, java.lang.String, java.lang.Object[])") + public static void beforeReversed( + @CallSite.Argument(1) String format, @CallSite.Argument(0) java.util.Locale locale) {} + } + + @Test + void multiplePartialArgumentsWithBeforeAdvice() { + CallSiteSpecification spec = + buildClassSpecification(MultiplePartialArgumentsBeforeAdvice.class); + AdviceGenerator generator = buildAdviceGenerator(buildDir); + + CallSiteResult result = generator.generate(spec); + + assertNoErrors(result); + CallSiteAssert asserter = assertCallSites(result.getFile()); + // In-order capture: {0, 1} matches both advice and pointcut order + asserter.advices( + 0, + advice -> { + advice.pointcut( + "java/lang/String", + "format", + "(Ljava/util/Locale;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/String;"); + advice.statements( + "int[] parameterIndices = new int[] { 0, 1 };", + "handler.dupParameters(descriptor, parameterIndices, null);", + "handler.advice(\"datadog/trace/plugin/csi/impl/AdviceGeneratorTest$MultiplePartialArgumentsBeforeAdvice\", \"before\", \"(Ljava/util/Locale;Ljava/lang/String;)V\");", + "handler.method(opcode, owner, name, descriptor, isInterface);"); + }); + // Reversed capture: {1, 0} follows the advice signature, not the pointcut index order {0, 1} + asserter.advices( + 1, + advice -> { + advice.pointcut( + "java/lang/String", + "format", + "(Ljava/util/Locale;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/String;"); + advice.statements( + "int[] parameterIndices = new int[] { 1, 0 };", + "handler.dupParameters(descriptor, parameterIndices, null);", + "handler.advice(\"datadog/trace/plugin/csi/impl/AdviceGeneratorTest$MultiplePartialArgumentsBeforeAdvice\", \"beforeReversed\", \"(Ljava/lang/String;Ljava/util/Locale;)V\");", + "handler.method(opcode, owner, name, descriptor, isInterface);"); + }); + } + + @CallSite(spi = CallSites.class) + public static class SuperTypeReturnAdvice { + @CallSite.After("void java.lang.StringBuilder.(java.lang.String)") + public static Object after( + @CallSite.AllArguments Object[] args, @CallSite.Return Object result) { + return result; + } + } + + @Test + void testReturningSuperType() { + CallSiteSpecification spec = buildClassSpecification(SuperTypeReturnAdvice.class); + AdviceGenerator generator = buildAdviceGenerator(buildDir); + + CallSiteResult result = generator.generate(spec); + + assertNoErrors(result); + CallSiteAssert asserter = assertCallSites(result.getFile()); + asserter.advices( + 0, + advice -> { + advice.pointcut("java/lang/StringBuilder", "", "(Ljava/lang/String;)V"); + advice.statements( + "handler.dupParameters(descriptor, StackDupMode.PREPEND_ARRAY_CTOR);", + "handler.method(opcode, owner, name, descriptor, isInterface);", + "handler.advice(\"datadog/trace/plugin/csi/impl/AdviceGeneratorTest$SuperTypeReturnAdvice\", \"after\", \"([Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;\");", + "handler.instruction(Opcodes.CHECKCAST, \"java/lang/StringBuilder\");"); + }); + } + + @CallSite(spi = {IastCallSites.class, RaspCallSites.class}) + public static class MultipleSpiClassesAdvice { + @CallSite.After("void java.lang.StringBuilder.(java.lang.String)") + public static Object after( + @CallSite.AllArguments Object[] args, @CallSite.Return Object result) { + return result; + } + } + + @Test + void testMultipleSpiClasses() { + CallSiteSpecification spec = buildClassSpecification(MultipleSpiClassesAdvice.class); + AdviceGenerator generator = buildAdviceGenerator(buildDir); + + CallSiteResult result = generator.generate(spec); + + assertNoErrors(result); + CallSiteAssert asserter = assertCallSites(result.getFile()); + asserter.spi(IastCallSites.class, RaspCallSites.class); + } + + @CallSite(spi = CallSites.class) + public static class AfterAdviceWithVoidReturn { + @CallSite.After("void java.lang.StringBuilder.setLength(int)") + public static void after(@CallSite.This StringBuilder self, @CallSite.Argument(0) int length) {} + } + + @Test + void testAfterAdviceWithVoidReturn() { + CallSiteSpecification spec = buildClassSpecification(AfterAdviceWithVoidReturn.class); + AdviceGenerator generator = buildAdviceGenerator(buildDir); + + CallSiteResult result = generator.generate(spec); + + assertNoErrors(result); + CallSiteAssert asserter = assertCallSites(result.getFile()); + asserter.advices( + 0, + advice -> { + advice.pointcut("java/lang/StringBuilder", "setLength", "(I)V"); + advice.statements( + "handler.dupInvoke(owner, descriptor, StackDupMode.COPY);", + "handler.method(opcode, owner, name, descriptor, isInterface);", + "handler.advice(\"datadog/trace/plugin/csi/impl/AdviceGeneratorTest$AfterAdviceWithVoidReturn\", \"after\", \"(Ljava/lang/StringBuilder;I)V\");"); + }); + } + + private static AdviceGenerator buildAdviceGenerator(File targetFolder) { + return new AdviceGeneratorImpl(targetFolder, pointcutParser()); + } + + private static CallSiteAssert assertCallSites(File generated) { + return new AssertBuilder(generated).build(); + } +} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/AdviceSpecificationTest.java b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/AdviceSpecificationTest.java new file mode 100644 index 00000000000..3c9fe3cacb5 --- /dev/null +++ b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/AdviceSpecificationTest.java @@ -0,0 +1,723 @@ +package datadog.trace.plugin.csi.impl; + +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.argThat; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; + +import datadog.trace.agent.tooling.csi.CallSite; +import datadog.trace.agent.tooling.csi.CallSites; +import datadog.trace.plugin.csi.HasErrors.Failure; +import datadog.trace.plugin.csi.ValidationContext; +import datadog.trace.plugin.csi.impl.CallSiteSpecification.AfterSpecification; +import datadog.trace.plugin.csi.impl.CallSiteSpecification.AllArgsSpecification; +import datadog.trace.plugin.csi.impl.CallSiteSpecification.ArgumentSpecification; +import datadog.trace.plugin.csi.impl.CallSiteSpecification.AroundSpecification; +import datadog.trace.plugin.csi.impl.CallSiteSpecification.BeforeSpecification; +import datadog.trace.plugin.csi.impl.CallSiteSpecification.InvokeDynamicConstantsSpecification; +import datadog.trace.plugin.csi.impl.CallSiteSpecification.ParameterSpecification; +import datadog.trace.plugin.csi.impl.CallSiteSpecification.ReturnSpecification; +import datadog.trace.plugin.csi.impl.CallSiteSpecification.ThisSpecification; +import datadog.trace.plugin.csi.util.ErrorCode; +import datadog.trace.plugin.csi.util.MethodType; +import java.lang.invoke.MethodHandles; +import java.lang.reflect.Method; +import java.security.MessageDigest; +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Stream; +import javax.servlet.ServletRequest; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledForJreRange; +import org.junit.jupiter.api.condition.JRE; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; +import org.objectweb.asm.Type; + +class AdviceSpecificationTest extends BaseCsiPluginTest { + + @CallSite(spi = CallSites.class) + static class EmptyAdvice {} + + @Test + void testClassGeneratorErrorCallSiteWithoutAdvices() { + ValidationContext context = mockValidationContext(); + CallSiteSpecification spec = buildClassSpecification(EmptyAdvice.class); + + spec.validate(context); + verify(context).addError(eq(ErrorCode.CALL_SITE_SHOULD_HAVE_ADVICE_METHODS), any()); + } + + @CallSite(spi = CallSites.class) + static class NonPublicStaticMethodAdvice { + @CallSite.Before("void java.lang.Runnable.run()") + private void advice(@CallSite.This Runnable run) {} + } + + @Test + void testClassGeneratorErrorNonPublicStaticMethod() { + ValidationContext context = mockValidationContext(); + CallSiteSpecification spec = buildClassSpecification(NonPublicStaticMethodAdvice.class); + + spec.getAdvices().forEach(it -> it.validate(context)); + + verify(context).addError(eq(ErrorCode.ADVICE_METHOD_NOT_STATIC_AND_PUBLIC), any()); + } + + static class BeforeStringConcat { + static void concat(String self, String value) {} + } + + static Stream adviceClassShouldBeOnClasspathProvider() { + return Stream.of( + Arguments.of(Type.getType("Lfoo/bar/FooBar;"), 1), + Arguments.of(Type.getType(BeforeStringConcat.class), 0)); + } + + @ParameterizedTest + @MethodSource("adviceClassShouldBeOnClasspathProvider") + void testAdviceClassShouldBeOnTheClasspath(Type type, int errors) throws Exception { + ValidationContext context = mockValidationContext(); + BeforeSpecification spec = + createBeforeSpec( + BeforeStringConcat.class.getDeclaredMethod("concat", String.class, String.class), + type, + Arrays.asList(new ThisSpecification(), new ArgumentSpecification()), + "java.lang.String java.lang.String.concat(java.lang.String)"); + + spec.validate(context); + verify(context, times(errors)) + .addError( + argThat((Failure failure) -> failure.getErrorCode() == ErrorCode.UNRESOLVED_TYPE)); + } + + static Stream beforeAdviceShouldReturnVoidProvider() { + return Stream.of(Arguments.of(String.class, 1), Arguments.of(void.class, 0)); + } + + @ParameterizedTest + @MethodSource("beforeAdviceShouldReturnVoidProvider") + void testBeforeAdviceShouldReturnVoid(Class returnType, int errors) { + ValidationContext context = mockValidationContext(); + BeforeSpecification spec = + createBeforeSpec( + BeforeStringConcat.class, + "concat", + returnType, + new Class[] {String.class, String.class}, + Arrays.asList(new ThisSpecification(), new ArgumentSpecification()), + "java.lang.String java.lang.String.concat(java.lang.String)"); + + spec.validate(context); + verify(context, times(errors)).addError(eq(ErrorCode.ADVICE_BEFORE_SHOULD_RETURN_VOID), any()); + } + + static class AroundStringConcat { + static String concat(String self, String value) { + return self.concat(value); + } + } + + static Stream aroundAdviceReturnTypeProvider() { + return Stream.of( + Arguments.of(MessageDigest.class, 1), + Arguments.of(Object.class, 0), + Arguments.of(String.class, 0)); + } + + @ParameterizedTest + @MethodSource("aroundAdviceReturnTypeProvider") + void testAroundAdviceShouldReturnTypeCompatibleWithPointcut(Class returnType, int errors) { + ValidationContext context = mockValidationContext(); + AroundSpecification spec = + createAroundSpec( + AroundStringConcat.class, + "concat", + returnType, + new Class[] {String.class, String.class}, + Arrays.asList(new ThisSpecification(), new ArgumentSpecification()), + "java.lang.String java.lang.String.concat(java.lang.String)"); + + spec.validate(context); + verify(context, times(errors)) + .addError(eq(ErrorCode.ADVICE_METHOD_RETURN_NOT_COMPATIBLE), any()); + } + + static class AfterStringConcat { + static String concat(String self, String value, String result) { + return result; + } + } + + static Stream afterAdviceReturnTypeProvider() { + return Stream.of( + Arguments.of(MessageDigest.class, 1), + Arguments.of(Object.class, 0), + Arguments.of(String.class, 0)); + } + + @ParameterizedTest + @MethodSource("afterAdviceReturnTypeProvider") + void testAfterAdviceShouldReturnTypeCompatibleWithPointcut(Class returnType, int errors) { + ValidationContext context = mockValidationContext(); + AfterSpecification spec = + createAfterSpec( + AfterStringConcat.class, + "concat", + returnType, + new Class[] {String.class, String.class, String.class}, + Arrays.asList( + new ThisSpecification(), new ArgumentSpecification(), new ReturnSpecification()), + "java.lang.String java.lang.String.concat(java.lang.String)"); + + spec.validate(context); + verify(context, times(errors)) + .addError(eq(ErrorCode.ADVICE_METHOD_RETURN_NOT_COMPATIBLE), any()); + } + + static Stream thisParameterShouldBeFirstProvider() { + return Stream.of( + Arguments.of(Arrays.asList(new ThisSpecification(), new ArgumentSpecification()), 0), + Arguments.of(Arrays.asList(new ArgumentSpecification(), new ThisSpecification()), 1)); + } + + @ParameterizedTest + @MethodSource("thisParameterShouldBeFirstProvider") + void testThisParameterShouldAlwaysBeTheFirst(List params, int errors) + throws Exception { + ValidationContext context = mockValidationContext(); + AroundSpecification spec = + createAroundSpec( + AroundStringConcat.class.getDeclaredMethod("concat", String.class, String.class), + params, + "java.lang.String java.lang.String.concat(java.lang.String)"); + + spec.validate(context); + verify(context, times(errors)) + .addError(eq(ErrorCode.ADVICE_PARAMETER_THIS_SHOULD_BE_FIRST), any()); + } + + static Stream thisParameterCompatibilityProvider() { + return Stream.of( + Arguments.of(MessageDigest.class, 1), + Arguments.of(Object.class, 0), + Arguments.of(String.class, 0)); + } + + @ParameterizedTest + @MethodSource("thisParameterCompatibilityProvider") + void testThisParameterShouldBeCompatibleWithPointcut(Class type, int errors) { + ValidationContext context = mockValidationContext(); + AroundSpecification spec = + createAroundSpec( + AroundStringConcat.class, + "concat", + String.class, + new Class[] {type, String.class}, + Arrays.asList(new ThisSpecification(), new ArgumentSpecification()), + "java.lang.String java.lang.String.concat(java.lang.String)"); + + spec.validate(context); + verify(context, times(errors)) + .addError(eq(ErrorCode.ADVICE_METHOD_PARAM_THIS_NOT_COMPATIBLE), any()); + if (type != String.class) { + verify(context) + .addError( + argThat((Failure failure) -> failure.getErrorCode() == ErrorCode.UNRESOLVED_METHOD)); + } + } + + static Stream returnParameterShouldBeLastProvider() { + return Stream.of( + Arguments.of( + Arrays.asList( + new ThisSpecification(), new ArgumentSpecification(), new ReturnSpecification()), + 0), + Arguments.of( + Arrays.asList( + new ThisSpecification(), new ReturnSpecification(), new ArgumentSpecification()), + 1)); + } + + @ParameterizedTest + @MethodSource("returnParameterShouldBeLastProvider") + void testReturnParameterShouldAlwaysBeTheLast(List params, int errors) + throws Exception { + ValidationContext context = mockValidationContext(); + AfterSpecification spec = + createAfterSpec( + AfterStringConcat.class.getDeclaredMethod( + "concat", String.class, String.class, String.class), + params, + "java.lang.String java.lang.String.concat(java.lang.String)"); + + spec.validate(context); + verify(context, times(errors)) + .addError(eq(ErrorCode.ADVICE_PARAMETER_RETURN_SHOULD_BE_LAST), any()); + } + + static Stream returnParameterCompatibilityProvider() { + return Stream.of( + Arguments.of(MessageDigest.class, 1), + Arguments.of(String.class, 0), + Arguments.of(Object.class, 0)); + } + + @ParameterizedTest + @MethodSource("returnParameterCompatibilityProvider") + void testReturnParameterShouldBeCompatibleWithPointcut(Class returnType, int errors) { + ValidationContext context = mockValidationContext(); + AfterSpecification spec = + createAfterSpec( + AfterStringConcat.class, + "concat", + String.class, + new Class[] {String.class, String.class, returnType}, + Arrays.asList( + new ThisSpecification(), new ArgumentSpecification(), new ReturnSpecification()), + "java.lang.String java.lang.String.concat(java.lang.String)"); + + spec.validate(context); + verify(context, times(errors)) + .addError(eq(ErrorCode.ADVICE_METHOD_PARAM_RETURN_NOT_COMPATIBLE), any()); + if (returnType != String.class) { + verify(context) + .addError( + argThat((Failure failure) -> failure.getErrorCode() == ErrorCode.UNRESOLVED_METHOD)); + } + } + + static Stream argumentParameterCompatibilityProvider() { + return Stream.of( + Arguments.of(MessageDigest.class, 1), + Arguments.of(String.class, 0), + Arguments.of(Object.class, 0)); + } + + @ParameterizedTest + @MethodSource("argumentParameterCompatibilityProvider") + void testArgumentParameterShouldBeCompatibleWithPointcut(Class parameterType, int errors) { + ValidationContext context = mockValidationContext(); + AfterSpecification spec = + createAfterSpec( + AfterStringConcat.class, + "concat", + String.class, + new Class[] {String.class, parameterType, String.class}, + Arrays.asList( + new ThisSpecification(), new ArgumentSpecification(), new ReturnSpecification()), + "java.lang.String java.lang.String.concat(java.lang.String)"); + + spec.validate(context); + verify(context, times(errors)) + .addError(eq(ErrorCode.ADVICE_METHOD_PARAM_NOT_COMPATIBLE), any()); + if (parameterType != String.class) { + verify(context) + .addError( + argThat((Failure failure) -> failure.getErrorCode() == ErrorCode.UNRESOLVED_METHOD)); + } + } + + static class BadAfterStringConcat { + static String concat(String param1, String param2) { + return param2; + } + } + + static Stream afterAdviceRequiresThisAndReturnProvider() { + return Stream.of( + Arguments.of( + Arrays.asList(new ArgumentSpecification(), new ReturnSpecification()), + ErrorCode.ADVICE_AFTER_SHOULD_HAVE_THIS), + Arguments.of( + Arrays.asList(new ThisSpecification(), new ArgumentSpecification()), + ErrorCode.ADVICE_AFTER_SHOULD_HAVE_RETURN)); + } + + @ParameterizedTest + @MethodSource("afterAdviceRequiresThisAndReturnProvider") + void testAfterAdviceRequiresThisAndReturnParameters( + List params, ErrorCode error) throws Exception { + ValidationContext context = mockValidationContext(); + AfterSpecification spec = + createAfterSpec( + BadAfterStringConcat.class.getDeclaredMethod("concat", String.class, String.class), + params, + "java.lang.String java.lang.String.concat(java.lang.String)"); + + spec.validate(context); + verify(context).addError(eq(error), any()); + } + + static class BadAllArgsAfterStringConcat { + static String concat(Object[] param1, String param2, String param3) { + return param3; + } + } + + @Test + void shouldNotMixAllArgumentsAndArgument() throws Exception { + ValidationContext context = mockValidationContext(); + AllArgsSpecification allArgs = new AllArgsSpecification(); + allArgs.setIncludeThis(true); + AfterSpecification spec = + createAfterSpec( + BadAllArgsAfterStringConcat.class.getDeclaredMethod( + "concat", Object[].class, String.class, String.class), + Arrays.asList(allArgs, new ArgumentSpecification(), new ReturnSpecification()), + "java.lang.String java.lang.String.concat(java.lang.String)"); + + spec.validate(context); + verify(context).addError(eq(ErrorCode.ADVICE_PARAMETER_ALL_ARGS_MIXED), any()); + verify(context).addError(eq(ErrorCode.ADVICE_PARAMETER_ARGUMENT_OUT_OF_BOUNDS), any()); + } + + static class TestInheritedMethod { + static String after(ServletRequest request, String parameter, String value) { + return value; + } + } + + @Test + void testInheritedMethods() throws Exception { + ValidationContext context = mockValidationContext(); + AfterSpecification spec = + createAfterSpec( + TestInheritedMethod.class.getDeclaredMethod( + "after", ServletRequest.class, String.class, String.class), + Arrays.asList( + new ThisSpecification(), new ArgumentSpecification(), new ReturnSpecification()), + "java.lang.String javax.servlet.http.HttpServletRequest.getParameter(java.lang.String)"); + + spec.validate(context); + } + + static class TestInvokeDynamicConstants { + static Object after(Object[] parameter, Object result, Object[] constants) { + return result; + } + } + + @Test + @EnabledForJreRange(min = JRE.JAVA_9) + void testInvokeDynamicConstants() throws Exception { + ValidationContext context = mockValidationContext(); + AfterSpecification spec = + createAfterSpec( + TestInvokeDynamicConstants.class.getDeclaredMethod( + "after", Object[].class, Object.class, Object[].class), + Arrays.asList( + new AllArgsSpecification(), + new ReturnSpecification(), + new InvokeDynamicConstantsSpecification()), + "java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])", + true); + + spec.validate(context); + } + + static Stream invokeDynamicConstantsShouldBeLastProvider() { + return Stream.of( + Arguments.of( + Arrays.asList( + new AllArgsSpecification(), + new ReturnSpecification(), + new InvokeDynamicConstantsSpecification()), + null), + Arguments.of( + Arrays.asList( + new AllArgsSpecification(), + new InvokeDynamicConstantsSpecification(), + new ReturnSpecification()), + ErrorCode.ADVICE_PARAMETER_INVOKE_DYNAMIC_CONSTANTS_SHOULD_BE_LAST)); + } + + @ParameterizedTest + @MethodSource("invokeDynamicConstantsShouldBeLastProvider") + @EnabledForJreRange(min = JRE.JAVA_9) + void testInvokeDynamicConstantsShouldBeLast(List params, ErrorCode error) + throws Exception { + ValidationContext context = mockValidationContext(); + AfterSpecification spec = + createAfterSpec( + TestInvokeDynamicConstants.class.getDeclaredMethod( + "after", Object[].class, Object.class, Object[].class), + params, + "java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])", + true); + + spec.validate(context); + if (error != null) { + verify(context).addError(eq(error), any()); + } + } + + static class TestInvokeDynamicConstantsNonInvokeDynamic { + static Object after(Object self, Object[] parameter, Object value, Object[] constants) { + return value; + } + } + + @Test + @EnabledForJreRange(min = JRE.JAVA_9) + void testInvokeDynamicConstantsOnNonInvokeDynamicPointcut() throws Exception { + ValidationContext context = mockValidationContext(); + AfterSpecification spec = + createAfterSpec( + TestInvokeDynamicConstantsNonInvokeDynamic.class.getDeclaredMethod( + "after", Object.class, Object[].class, Object.class, Object[].class), + Arrays.asList( + new ThisSpecification(), + new AllArgsSpecification(), + new InvokeDynamicConstantsSpecification(), + new ReturnSpecification()), + "java.lang.String java.lang.String.concat(java.lang.String)"); + + spec.validate(context); + verify(context) + .addError( + eq(ErrorCode.ADVICE_PARAMETER_INVOKE_DYNAMIC_CONSTANTS_ON_NON_INVOKE_DYNAMIC), any()); + } + + static class TestInvokeDynamicConstantsBefore { + static void before(Object[] parameter, Object[] constants) {} + } + + @Test + @EnabledForJreRange(min = JRE.JAVA_9) + void testInvokeDynamicConstantsOnNonAfterAdvice() throws Exception { + ValidationContext context = mockValidationContext(); + BeforeSpecification spec = + createBeforeSpec( + TestInvokeDynamicConstantsBefore.class.getDeclaredMethod( + "before", Object[].class, Object[].class), + Arrays.asList(new AllArgsSpecification(), new InvokeDynamicConstantsSpecification()), + "java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])", + true); + + spec.validate(context); + verify(context) + .addError(eq(ErrorCode.ADVICE_PARAMETER_INVOKE_DYNAMIC_CONSTANTS_NON_AFTER_ADVICE), any()); + } + + static class TestInvokeDynamicConstantsAround { + static java.lang.invoke.CallSite around( + MethodHandles.Lookup lookup, + String name, + java.lang.invoke.MethodType concatType, + String recipe, + Object... constants) { + return null; + } + } + + @Test + @EnabledForJreRange(min = JRE.JAVA_9) + void testInvokeDynamicOnAroundAdvice() throws Exception { + ValidationContext context = mockValidationContext(); + AroundSpecification spec = + createAroundSpec( + TestInvokeDynamicConstantsAround.class.getDeclaredMethod( + "around", + MethodHandles.Lookup.class, + String.class, + java.lang.invoke.MethodType.class, + String.class, + Object[].class), + Arrays.asList( + new ArgumentSpecification(), + new ArgumentSpecification(), + new ArgumentSpecification(), + new ArgumentSpecification(), + new ArgumentSpecification()), + "java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])", + true); + + spec.validate(context); + } + + @CallSite(spi = CallSites.class) + static class AfterWithVoidWrongAdvice { + @CallSite.After("void java.lang.String.getChars(int, int, char[], int)") + static String after(@CallSite.AllArguments Object[] args, @CallSite.Return String result) { + return result; + } + } + + @Test + void testAfterAdviceWithVoidShouldNotUseReturn() { + ValidationContext context = mockValidationContext(); + CallSiteSpecification spec = buildClassSpecification(AfterWithVoidWrongAdvice.class); + + spec.getAdvices().forEach(it -> it.validate(context)); + + verify(context).addError(eq(ErrorCode.ADVICE_AFTER_VOID_METHOD_SHOULD_RETURN_VOID), any()); + verify(context).addError(eq(ErrorCode.ADVICE_AFTER_VOID_METHOD_SHOULD_NOT_HAVE_RETURN), any()); + } + + // Helper methods to create specifications + private BeforeSpecification createBeforeSpec( + Method method, List params, String signature) { + return createBeforeSpec(method, null, params, signature, false); + } + + private BeforeSpecification createBeforeSpec( + Method method, List params, String signature, boolean invokeDynamic) { + return createBeforeSpec(method, null, params, signature, invokeDynamic); + } + + private BeforeSpecification createBeforeSpec( + Method method, Type ownerOverride, List params, String signature) { + return createBeforeSpec(method, ownerOverride, params, signature, false); + } + + private BeforeSpecification createBeforeSpec( + Method method, + Type ownerOverride, + List params, + String signature, + boolean invokeDynamic) { + Type owner = ownerOverride != null ? ownerOverride : Type.getType(method.getDeclaringClass()); + Type[] argTypes = + Arrays.stream(method.getParameterTypes()).map(Type::getType).toArray(Type[]::new); + Type returnType = Type.getType(method.getReturnType()); + MethodType methodType = + new MethodType(owner, method.getName(), Type.getMethodType(returnType, argTypes)); + Map paramMap = new HashMap<>(); + for (int i = 0; i < params.size(); i++) { + paramMap.put(i, params.get(i)); + } + updateArgumentIndices(paramMap); + BeforeSpecification spec = + new BeforeSpecification(methodType, paramMap, signature, invokeDynamic); + spec.parseSignature(CallSiteFactory.pointcutParser()); + return spec; + } + + private BeforeSpecification createBeforeSpec( + Class clazz, + String methodName, + Class returnType, + Class[] argTypes, + List params, + String signature) { + Type owner = Type.getType(clazz); + Type[] argTypesAsm = Arrays.stream(argTypes).map(Type::getType).toArray(Type[]::new); + Type returnTypeAsm = Type.getType(returnType); + MethodType methodType = + new MethodType(owner, methodName, Type.getMethodType(returnTypeAsm, argTypesAsm)); + Map paramMap = new HashMap<>(); + for (int i = 0; i < params.size(); i++) { + paramMap.put(i, params.get(i)); + } + updateArgumentIndices(paramMap); + BeforeSpecification spec = new BeforeSpecification(methodType, paramMap, signature, false); + spec.parseSignature(CallSiteFactory.pointcutParser()); + return spec; + } + + private AroundSpecification createAroundSpec( + Method method, List params, String signature) { + return createAroundSpec(method, params, signature, false); + } + + private AroundSpecification createAroundSpec( + Method method, List params, String signature, boolean invokeDynamic) { + Type owner = Type.getType(method.getDeclaringClass()); + Type[] argTypes = + Arrays.stream(method.getParameterTypes()).map(Type::getType).toArray(Type[]::new); + Type returnType = Type.getType(method.getReturnType()); + MethodType methodType = + new MethodType(owner, method.getName(), Type.getMethodType(returnType, argTypes)); + Map paramMap = new HashMap<>(); + for (int i = 0; i < params.size(); i++) { + paramMap.put(i, params.get(i)); + } + updateArgumentIndices(paramMap); + AroundSpecification spec = + new AroundSpecification(methodType, paramMap, signature, invokeDynamic); + spec.parseSignature(CallSiteFactory.pointcutParser()); + return spec; + } + + private AroundSpecification createAroundSpec( + Class clazz, + String methodName, + Class returnType, + Class[] argTypes, + List params, + String signature) { + Type owner = Type.getType(clazz); + Type[] argTypesAsm = Arrays.stream(argTypes).map(Type::getType).toArray(Type[]::new); + Type returnTypeAsm = Type.getType(returnType); + MethodType methodType = + new MethodType(owner, methodName, Type.getMethodType(returnTypeAsm, argTypesAsm)); + Map paramMap = new HashMap<>(); + for (int i = 0; i < params.size(); i++) { + paramMap.put(i, params.get(i)); + } + updateArgumentIndices(paramMap); + AroundSpecification spec = new AroundSpecification(methodType, paramMap, signature, false); + spec.parseSignature(CallSiteFactory.pointcutParser()); + return spec; + } + + private AfterSpecification createAfterSpec( + Method method, List params, String signature) { + return createAfterSpec(method, params, signature, false); + } + + private AfterSpecification createAfterSpec( + Method method, List params, String signature, boolean invokeDynamic) { + Type owner = Type.getType(method.getDeclaringClass()); + Type[] argTypes = + Arrays.stream(method.getParameterTypes()).map(Type::getType).toArray(Type[]::new); + Type returnType = Type.getType(method.getReturnType()); + MethodType methodType = + new MethodType(owner, method.getName(), Type.getMethodType(returnType, argTypes)); + Map paramMap = new HashMap<>(); + for (int i = 0; i < params.size(); i++) { + paramMap.put(i, params.get(i)); + } + updateArgumentIndices(paramMap); + AfterSpecification spec = + new AfterSpecification(methodType, paramMap, signature, invokeDynamic); + spec.parseSignature(CallSiteFactory.pointcutParser()); + return spec; + } + + private AfterSpecification createAfterSpec( + Class clazz, + String methodName, + Class returnType, + Class[] argTypes, + List params, + String signature) { + Type owner = Type.getType(clazz); + Type[] argTypesAsm = Arrays.stream(argTypes).map(Type::getType).toArray(Type[]::new); + Type returnTypeAsm = Type.getType(returnType); + MethodType methodType = + new MethodType(owner, methodName, Type.getMethodType(returnTypeAsm, argTypesAsm)); + Map paramMap = new HashMap<>(); + for (int i = 0; i < params.size(); i++) { + paramMap.put(i, params.get(i)); + } + updateArgumentIndices(paramMap); + AfterSpecification spec = new AfterSpecification(methodType, paramMap, signature, false); + spec.parseSignature(CallSiteFactory.pointcutParser()); + return spec; + } + + private void updateArgumentIndices(Map paramMap) { + int index = 0; + for (ParameterSpecification param : paramMap.values()) { + if (param instanceof ArgumentSpecification) { + ((ArgumentSpecification) param).setIndex(index++); + } + } + } +} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/AsmSpecificationBuilderTest.java b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/AsmSpecificationBuilderTest.java new file mode 100644 index 00000000000..ec37662ff58 --- /dev/null +++ b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/AsmSpecificationBuilderTest.java @@ -0,0 +1,591 @@ +package datadog.trace.plugin.csi.impl; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import datadog.trace.agent.tooling.csi.CallSite; +import datadog.trace.agent.tooling.csi.CallSites; +import datadog.trace.plugin.csi.impl.CallSiteSpecification.AdviceSpecification; +import datadog.trace.plugin.csi.impl.CallSiteSpecification.AfterSpecification; +import datadog.trace.plugin.csi.impl.CallSiteSpecification.AllArgsSpecification; +import datadog.trace.plugin.csi.impl.CallSiteSpecification.AroundSpecification; +import datadog.trace.plugin.csi.impl.CallSiteSpecification.BeforeSpecification; +import datadog.trace.plugin.csi.util.Types; +import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; +import java.io.File; +import java.lang.invoke.MethodHandles; +import java.lang.invoke.MethodType; +import java.util.Arrays; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.stream.Collectors; +import javax.annotation.Nonnull; +import javax.annotation.Nullable; +import javax.servlet.ServletRequest; +import org.junit.jupiter.api.Test; +import org.objectweb.asm.Type; + +class AsmSpecificationBuilderTest extends BaseCsiPluginTest { + + static class NonCallSite {} + + @Test + void testSpecificationBuilderForNonCallSite() { + File advice = fetchClass(NonCallSite.class); + AsmSpecificationBuilder specificationBuilder = new AsmSpecificationBuilder(); + + Optional result = specificationBuilder.build(advice); + + assertFalse(result.isPresent()); + } + + @CallSite(spi = WithSpiClass.Spi.class) + static class WithSpiClass { + interface Spi {} + } + + @Test + void testSpecificationBuilderWithCustomSpiClass() { + File advice = fetchClass(WithSpiClass.class); + AsmSpecificationBuilder specificationBuilder = new AsmSpecificationBuilder(); + + CallSiteSpecification result = + specificationBuilder.build(advice).orElseThrow(RuntimeException::new); + + assertEquals( + Arrays.asList(Type.getType(WithSpiClass.Spi.class)), Arrays.asList(result.getSpi())); + } + + @CallSite( + spi = CallSites.class, + helpers = {HelpersAdvice.SampleHelper1.class, HelpersAdvice.SampleHelper2.class}) + static class HelpersAdvice { + static class SampleHelper1 {} + + static class SampleHelper2 {} + } + + @Test + void testSpecificationBuilderWithCustomHelperClasses() { + File advice = fetchClass(HelpersAdvice.class); + AsmSpecificationBuilder specificationBuilder = new AsmSpecificationBuilder(); + + CallSiteSpecification result = + specificationBuilder.build(advice).orElseThrow(RuntimeException::new); + + List helpers = Arrays.asList(result.getHelpers()); + assertTrue( + helpers.containsAll( + Arrays.asList( + Type.getType(HelpersAdvice.class), + Type.getType(HelpersAdvice.SampleHelper1.class), + Type.getType(HelpersAdvice.SampleHelper2.class)))); + } + + @CallSite(spi = CallSites.class) + static class BeforeAdvice { + @CallSite.Before( + "java.lang.String java.lang.String.replaceAll(java.lang.String, java.lang.String)") + static void before( + @CallSite.This String self, + @CallSite.Argument String regexp, + @CallSite.Argument String replacement) {} + } + + @Test + void testSpecificationBuilderForBeforeAdvice() { + File advice = fetchClass(BeforeAdvice.class); + AsmSpecificationBuilder specificationBuilder = new AsmSpecificationBuilder(); + + CallSiteSpecification result = + specificationBuilder.build(advice).orElseThrow(RuntimeException::new); + + assertEquals(BeforeAdvice.class.getName(), result.getClazz().getClassName()); + BeforeSpecification beforeSpec = (BeforeSpecification) findAdvice(result, "before"); + assertNotNull(beforeSpec); + assertEquals( + "(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)V", + beforeSpec.getAdvice().getMethodType().getDescriptor()); + assertEquals( + "java.lang.String java.lang.String.replaceAll(java.lang.String, java.lang.String)", + beforeSpec.getSignature()); + assertNotNull(beforeSpec.findThis()); + assertNull(beforeSpec.findReturn()); + assertNull(beforeSpec.findAllArguments()); + assertNull(beforeSpec.findInvokeDynamicConstants()); + List arguments = getArguments(beforeSpec); + assertEquals(Arrays.asList(0, 1), arguments); + } + + @CallSite(spi = CallSites.class) + static class AroundAdvice { + @CallSite.Around( + "java.lang.String java.lang.String.replaceAll(java.lang.String, java.lang.String)") + static String around( + @CallSite.This String self, + @CallSite.Argument String regexp, + @CallSite.Argument String replacement) { + return self.replaceAll(regexp, replacement); + } + } + + @Test + void testSpecificationBuilderForAroundAdvice() { + File advice = fetchClass(AroundAdvice.class); + AsmSpecificationBuilder specificationBuilder = new AsmSpecificationBuilder(); + + CallSiteSpecification result = + specificationBuilder.build(advice).orElseThrow(RuntimeException::new); + + assertEquals(AroundAdvice.class.getName(), result.getClazz().getClassName()); + AroundSpecification aroundSpec = (AroundSpecification) findAdvice(result, "around"); + assertNotNull(aroundSpec); + assertEquals( + "(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;", + aroundSpec.getAdvice().getMethodType().getDescriptor()); + assertEquals( + "java.lang.String java.lang.String.replaceAll(java.lang.String, java.lang.String)", + aroundSpec.getSignature()); + assertNotNull(aroundSpec.findThis()); + assertNull(aroundSpec.findReturn()); + assertNull(aroundSpec.findAllArguments()); + assertNull(aroundSpec.findInvokeDynamicConstants()); + List arguments = getArguments(aroundSpec); + assertEquals(Arrays.asList(0, 1), arguments); + } + + @CallSite(spi = CallSites.class) + static class AfterAdvice { + @CallSite.After( + "java.lang.String java.lang.String.replaceAll(java.lang.String, java.lang.String)") + static String after( + @CallSite.This String self, + @CallSite.Argument String regexp, + @CallSite.Argument String replacement, + @CallSite.Return String result) { + return result; + } + } + + @Test + void testSpecificationBuilderForAfterAdvice() { + File advice = fetchClass(AfterAdvice.class); + AsmSpecificationBuilder specificationBuilder = new AsmSpecificationBuilder(); + + CallSiteSpecification result = + specificationBuilder.build(advice).orElseThrow(RuntimeException::new); + + assertEquals(AfterAdvice.class.getName(), result.getClazz().getClassName()); + AfterSpecification afterSpec = (AfterSpecification) findAdvice(result, "after"); + assertNotNull(afterSpec); + assertEquals( + "(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;", + afterSpec.getAdvice().getMethodType().getDescriptor()); + assertEquals( + "java.lang.String java.lang.String.replaceAll(java.lang.String, java.lang.String)", + afterSpec.getSignature()); + assertNotNull(afterSpec.findThis()); + assertNotNull(afterSpec.findReturn()); + assertNull(afterSpec.findAllArguments()); + assertNull(afterSpec.findInvokeDynamicConstants()); + List arguments = getArguments(afterSpec); + assertEquals(Arrays.asList(0, 1), arguments); + } + + @CallSite(spi = CallSites.class) + static class AllArgsAdvice { + @CallSite.Around( + "java.lang.String java.lang.String.replaceAll(java.lang.String, java.lang.String)") + static String allArgs( + @CallSite.AllArguments(includeThis = true) Object[] arguments, + @CallSite.Return String result) { + return result; + } + } + + @Test + void testSpecificationBuilderForAdviceWithAllArguments() { + File advice = fetchClass(AllArgsAdvice.class); + AsmSpecificationBuilder specificationBuilder = new AsmSpecificationBuilder(); + + CallSiteSpecification result = + specificationBuilder.build(advice).orElseThrow(RuntimeException::new); + + assertEquals(AllArgsAdvice.class.getName(), result.getClazz().getClassName()); + AroundSpecification allArgsSpec = (AroundSpecification) findAdvice(result, "allArgs"); + assertNotNull(allArgsSpec); + assertEquals( + "([Ljava/lang/Object;Ljava/lang/String;)Ljava/lang/String;", + allArgsSpec.getAdvice().getMethodType().getDescriptor()); + assertEquals( + "java.lang.String java.lang.String.replaceAll(java.lang.String, java.lang.String)", + allArgsSpec.getSignature()); + assertNull(allArgsSpec.findThis()); + assertNotNull(allArgsSpec.findReturn()); + AllArgsSpecification allArguments = allArgsSpec.findAllArguments(); + assertNotNull(allArguments); + assertTrue(allArguments.isIncludeThis()); + assertNull(allArgsSpec.findInvokeDynamicConstants()); + List arguments = getArguments(allArgsSpec); + assertEquals(Arrays.asList(), arguments); + } + + @CallSite(spi = CallSites.class) + static class InvokeDynamicBeforeAdvice { + @CallSite.After( + value = + "java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])", + invokeDynamic = true) + static String invokeDynamic( + @CallSite.AllArguments Object[] arguments, @CallSite.Return String result) { + return result; + } + } + + @Test + void testSpecificationBuilderForBeforeInvokeDynamic() { + File advice = fetchClass(InvokeDynamicBeforeAdvice.class); + AsmSpecificationBuilder specificationBuilder = new AsmSpecificationBuilder(); + + CallSiteSpecification result = + specificationBuilder.build(advice).orElseThrow(RuntimeException::new); + + assertEquals(InvokeDynamicBeforeAdvice.class.getName(), result.getClazz().getClassName()); + AfterSpecification invokeDynamicSpec = (AfterSpecification) findAdvice(result, "invokeDynamic"); + assertNotNull(invokeDynamicSpec); + assertEquals( + "([Ljava/lang/Object;Ljava/lang/String;)Ljava/lang/String;", + invokeDynamicSpec.getAdvice().getMethodType().getDescriptor()); + assertEquals( + "java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])", + invokeDynamicSpec.getSignature()); + assertNull(invokeDynamicSpec.findThis()); + assertNotNull(invokeDynamicSpec.findReturn()); + AllArgsSpecification allArguments = invokeDynamicSpec.findAllArguments(); + assertNotNull(allArguments); + assertFalse(allArguments.isIncludeThis()); + assertNull(invokeDynamicSpec.findInvokeDynamicConstants()); + List arguments = getArguments(invokeDynamicSpec); + assertEquals(Arrays.asList(), arguments); + } + + @CallSite(spi = CallSites.class) + static class InvokeDynamicAroundAdvice { + @CallSite.Around( + value = + "java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])", + invokeDynamic = true) + static java.lang.invoke.CallSite invokeDynamic( + @CallSite.Argument MethodHandles.Lookup lookup, + @CallSite.Argument String name, + @CallSite.Argument MethodType concatType, + @CallSite.Argument String recipe, + @CallSite.Argument Object... constants) { + return null; + } + } + + @Test + void testSpecificationBuilderForAroundInvokeDynamic() { + File advice = fetchClass(InvokeDynamicAroundAdvice.class); + AsmSpecificationBuilder specificationBuilder = new AsmSpecificationBuilder(); + + CallSiteSpecification result = + specificationBuilder.build(advice).orElseThrow(RuntimeException::new); + + assertEquals(InvokeDynamicAroundAdvice.class.getName(), result.getClazz().getClassName()); + AroundSpecification invokeDynamicSpec = + (AroundSpecification) findAdvice(result, "invokeDynamic"); + assertNotNull(invokeDynamicSpec); + assertEquals( + "(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/invoke/CallSite;", + invokeDynamicSpec.getAdvice().getMethodType().getDescriptor()); + assertEquals( + "java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])", + invokeDynamicSpec.getSignature()); + assertNull(invokeDynamicSpec.findThis()); + assertNull(invokeDynamicSpec.findReturn()); + assertNull(invokeDynamicSpec.findAllArguments()); + assertNull(invokeDynamicSpec.findInvokeDynamicConstants()); + List arguments = getArguments(invokeDynamicSpec); + assertEquals(Arrays.asList(0, 1, 2, 3, 4), arguments); + } + + @CallSite(spi = CallSites.class) + static class TestInvokeDynamicConstants { + @CallSite.After( + value = + "java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])", + invokeDynamic = true) + static String after( + @CallSite.AllArguments Object[] parameter, + @CallSite.InvokeDynamicConstants Object[] constants, + @CallSite.Return String value) { + return value; + } + } + + @Test + void testInvokeDynamicConstants() { + File advice = fetchClass(TestInvokeDynamicConstants.class); + AsmSpecificationBuilder specificationBuilder = new AsmSpecificationBuilder(); + + CallSiteSpecification result = + specificationBuilder.build(advice).orElseThrow(RuntimeException::new); + + assertEquals(TestInvokeDynamicConstants.class.getName(), result.getClazz().getClassName()); + AfterSpecification inheritedSpec = (AfterSpecification) findAdvice(result, "after"); + assertNotNull(inheritedSpec); + assertEquals( + "([Ljava/lang/Object;[Ljava/lang/Object;Ljava/lang/String;)Ljava/lang/String;", + inheritedSpec.getAdvice().getMethodType().getDescriptor()); + assertEquals( + "java.lang.invoke.CallSite java.lang.invoke.StringConcatFactory.makeConcatWithConstants(java.lang.invoke.MethodHandles$Lookup, java.lang.String, java.lang.invoke.MethodType, java.lang.String, java.lang.Object[])", + inheritedSpec.getSignature()); + assertNull(inheritedSpec.findThis()); + assertNotNull(inheritedSpec.findReturn()); + assertNotNull(inheritedSpec.findInvokeDynamicConstants()); + List arguments = getArguments(inheritedSpec); + assertEquals(Arrays.asList(), arguments); + } + + @CallSite(spi = CallSites.class) + static class TestBeforeArray { + + @CallSite.BeforeArray({ + @CallSite.Before("java.util.Map javax.servlet.ServletRequest.getParameterMap()"), + @CallSite.Before("java.util.Map javax.servlet.ServletRequestWrapper.getParameterMap()") + }) + static void before(@CallSite.This ServletRequest request) {} + } + + @Test + void testSpecificationBuilderForBeforeAdviceArray() { + File advice = fetchClass(TestBeforeArray.class); + AsmSpecificationBuilder specificationBuilder = new AsmSpecificationBuilder(); + + CallSiteSpecification result = + specificationBuilder.build(advice).orElseThrow(RuntimeException::new); + + assertEquals(TestBeforeArray.class.getName(), result.getClazz().getClassName()); + List list = result.getAdvices(); + assertEquals(2, list.size()); + for (AdviceSpecification spec : list) { + assertInstanceOf(BeforeSpecification.class, spec); + assertEquals( + "(Ljavax/servlet/ServletRequest;)V", spec.getAdvice().getMethodType().getDescriptor()); + assertTrue( + spec.getSignature().equals("java.util.Map javax.servlet.ServletRequest.getParameterMap()") + || spec.getSignature() + .equals("java.util.Map javax.servlet.ServletRequestWrapper.getParameterMap()")); + assertNotNull(spec.findThis()); + assertNull(spec.findReturn()); + assertNull(spec.findAllArguments()); + assertNull(spec.findInvokeDynamicConstants()); + List arguments = getArguments(spec); + assertEquals(Arrays.asList(), arguments); + } + } + + @CallSite(spi = CallSites.class) + static class TestAroundArray { + + @CallSite.AroundArray({ + @CallSite.Around("java.util.Map javax.servlet.ServletRequest.getParameterMap()"), + @CallSite.Around("java.util.Map javax.servlet.ServletRequestWrapper.getParameterMap()") + }) + static Map around(@CallSite.This ServletRequest request) { + return request.getParameterMap(); + } + } + + @Test + void testSpecificationBuilderForAroundAdviceArray() { + File advice = fetchClass(TestAroundArray.class); + AsmSpecificationBuilder specificationBuilder = new AsmSpecificationBuilder(); + + CallSiteSpecification result = + specificationBuilder.build(advice).orElseThrow(RuntimeException::new); + + assertEquals(TestAroundArray.class.getName(), result.getClazz().getClassName()); + List list = result.getAdvices(); + assertEquals(2, list.size()); + for (AdviceSpecification spec : list) { + assertInstanceOf(AroundSpecification.class, spec); + assertEquals( + "(Ljavax/servlet/ServletRequest;)Ljava/util/Map;", + spec.getAdvice().getMethodType().getDescriptor()); + assertTrue( + spec.getSignature().equals("java.util.Map javax.servlet.ServletRequest.getParameterMap()") + || spec.getSignature() + .equals("java.util.Map javax.servlet.ServletRequestWrapper.getParameterMap()")); + assertNotNull(spec.findThis()); + assertNull(spec.findReturn()); + assertNull(spec.findAllArguments()); + assertNull(spec.findInvokeDynamicConstants()); + List arguments = getArguments(spec); + assertEquals(Arrays.asList(), arguments); + } + } + + @CallSite(spi = CallSites.class) + static class TestAfterArray { + + @CallSite.AfterArray({ + @CallSite.After("java.util.Map javax.servlet.ServletRequest.getParameterMap()"), + @CallSite.After("java.util.Map javax.servlet.ServletRequestWrapper.getParameterMap()") + }) + static Map after(@CallSite.This ServletRequest request, @CallSite.Return Map parameters) { + return parameters; + } + } + + @Test + void testSpecificationBuilderForAfterAdviceArray() { + File advice = fetchClass(TestAfterArray.class); + AsmSpecificationBuilder specificationBuilder = new AsmSpecificationBuilder(); + + CallSiteSpecification result = + specificationBuilder.build(advice).orElseThrow(RuntimeException::new); + + assertEquals(TestAfterArray.class.getName(), result.getClazz().getClassName()); + List list = result.getAdvices(); + assertEquals(2, list.size()); + for (AdviceSpecification spec : list) { + assertInstanceOf(AfterSpecification.class, spec); + assertEquals( + "(Ljavax/servlet/ServletRequest;Ljava/util/Map;)Ljava/util/Map;", + spec.getAdvice().getMethodType().getDescriptor()); + assertTrue( + spec.getSignature().equals("java.util.Map javax.servlet.ServletRequest.getParameterMap()") + || spec.getSignature() + .equals("java.util.Map javax.servlet.ServletRequestWrapper.getParameterMap()")); + assertNotNull(spec.findThis()); + assertNotNull(spec.findReturn()); + assertNull(spec.findAllArguments()); + assertNull(spec.findInvokeDynamicConstants()); + List arguments = getArguments(spec); + assertEquals(Arrays.asList(), arguments); + } + } + + @CallSite(spi = CallSites.class) + static class TestInheritedMethod { + @CallSite.After( + "java.lang.String javax.servlet.http.HttpServletRequest.getParameter(java.lang.String)") + static String after( + @CallSite.This ServletRequest request, + @CallSite.Argument String parameter, + @CallSite.Return String value) { + return value; + } + } + + @Test + void testSpecificationBuilderForInheritedMethods() { + File advice = fetchClass(TestInheritedMethod.class); + AsmSpecificationBuilder specificationBuilder = new AsmSpecificationBuilder(); + + CallSiteSpecification result = + specificationBuilder.build(advice).orElseThrow(RuntimeException::new); + + assertEquals(TestInheritedMethod.class.getName(), result.getClazz().getClassName()); + AfterSpecification inheritedSpec = (AfterSpecification) findAdvice(result, "after"); + assertNotNull(inheritedSpec); + assertEquals( + "(Ljavax/servlet/ServletRequest;Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;", + inheritedSpec.getAdvice().getMethodType().getDescriptor()); + assertEquals( + "java.lang.String javax.servlet.http.HttpServletRequest.getParameter(java.lang.String)", + inheritedSpec.getSignature()); + assertNotNull(inheritedSpec.findThis()); + assertNotNull(inheritedSpec.findReturn()); + assertNull(inheritedSpec.findAllArguments()); + assertNull(inheritedSpec.findInvokeDynamicConstants()); + List arguments = getArguments(inheritedSpec); + assertEquals(Arrays.asList(0), arguments); + } + + static class IsEnabled { + static boolean isEnabled(String defaultValue) { + return true; + } + } + + @CallSite( + spi = CallSites.class, + enabled = { + "datadog.trace.plugin.csi.impl.AsmSpecificationBuilderTest$IsEnabled", + "isEnabled", + "true" + }) + static class TestEnablement { + @CallSite.After( + "java.lang.String javax.servlet.http.HttpServletRequest.getParameter(java.lang.String)") + static String after( + @CallSite.This ServletRequest request, + @CallSite.Argument String parameter, + @CallSite.Return String value) { + return value; + } + } + + @Test + void testSpecificationBuilderWithEnabledProperty() { + File advice = fetchClass(TestEnablement.class); + AsmSpecificationBuilder specificationBuilder = new AsmSpecificationBuilder(); + + CallSiteSpecification result = + specificationBuilder.build(advice).orElseThrow(RuntimeException::new); + + assertEquals(TestEnablement.class.getName(), result.getClazz().getClassName()); + assertNotNull(result.getEnabled()); + assertEquals(Type.getType(IsEnabled.class), result.getEnabled().getMethod().getOwner()); + assertEquals("isEnabled", result.getEnabled().getMethod().getMethodName()); + assertEquals( + Type.getMethodType(Types.BOOLEAN, Types.STRING), + result.getEnabled().getMethod().getMethodType()); + assertEquals(Arrays.asList("true"), result.getEnabled().getArguments()); + } + + @CallSite(spi = CallSites.class) + static class TestWithOtherAnnotations { + @CallSite.Around("java.lang.StringBuilder java.lang.StringBuilder.append(java.lang.Object)") + @CallSite.Around("java.lang.StringBuffer java.lang.StringBuffer.append(java.lang.Object)") + @Nonnull + @SuppressFBWarnings("NP_PARAMETER_MUST_BE_NONNULL_BUT_MARKED_AS_NULLABLE") + static Appendable aroundAppend( + @CallSite.This @Nullable Appendable self, @CallSite.Argument(0) @Nullable Object param) + throws Throwable { + return self.append(param.toString()); + } + } + + @Test + void testSpecificationBuilderWithMultipleMethodAnnotations() { + File advice = fetchClass(TestWithOtherAnnotations.class); + AsmSpecificationBuilder specificationBuilder = new AsmSpecificationBuilder(); + + CallSiteSpecification result = + specificationBuilder.build(advice).orElseThrow(RuntimeException::new); + + assertEquals(TestWithOtherAnnotations.class.getName(), result.getClazz().getClassName()); + assertEquals(2, result.getAdvices().size()); + } + + private static List getArguments(AdviceSpecification advice) { + return advice.getArguments().map(arg -> arg.getIndex()).collect(Collectors.toList()); + } + + private static AdviceSpecification findAdvice(CallSiteSpecification result, String name) { + return result.getAdvices().stream() + .filter(it -> it.getAdvice().getMethodName().equals(name)) + .findFirst() + .orElse(null); + } +} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/BaseCsiPluginTest.java b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/BaseCsiPluginTest.java new file mode 100644 index 00000000000..a2bd158d20b --- /dev/null +++ b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/BaseCsiPluginTest.java @@ -0,0 +1,63 @@ +package datadog.trace.plugin.csi.impl; + +import static datadog.trace.plugin.csi.impl.CallSiteFactory.pointcutParser; +import static datadog.trace.plugin.csi.impl.CallSiteFactory.specificationBuilder; +import static datadog.trace.plugin.csi.impl.CallSiteFactory.typeResolver; +import static datadog.trace.plugin.csi.util.CallSiteConstants.TYPE_RESOLVER; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +import datadog.trace.plugin.csi.HasErrors; +import datadog.trace.plugin.csi.ValidationContext; +import java.io.File; +import java.net.URISyntaxException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.Collections; +import java.util.List; +import java.util.stream.Collectors; + +public abstract class BaseCsiPluginTest { + + protected static void assertNoErrors(HasErrors hasErrors) { + List errors = + hasErrors.getErrors().stream() + .map( + error -> { + String causeString = error.getCause() == null ? "-" : error.getCauseString(); + return error.getMessage() + ": " + causeString; + }) + .collect(Collectors.toList()); + assertEquals(Collections.emptyList(), errors); + } + + protected static File fetchClass(Class clazz) { + try { + Path folder = Paths.get(clazz.getResource("/").toURI()).resolve("../../"); + String fileSeparator = File.separator.equals("\\") ? "\\\\" : File.separator; + String classFile = clazz.getName().replaceAll("\\.", fileSeparator) + ".class"; + Path groovy = folder.resolve("groovy/test").resolve(classFile); + if (Files.exists(groovy)) { + return groovy.toFile(); + } + return folder.resolve("java/test").resolve(classFile).toFile(); + } catch (URISyntaxException e) { + throw new RuntimeException(e); + } + } + + protected static CallSiteSpecification buildClassSpecification(Class clazz) { + File classFile = fetchClass(clazz); + CallSiteSpecification spec = specificationBuilder().build(classFile).get(); + spec.getAdvices().forEach(advice -> advice.parseSignature(pointcutParser())); + return spec; + } + + protected ValidationContext mockValidationContext() { + ValidationContext context = mock(ValidationContext.class); + when(context.getContextProperty(TYPE_RESOLVER)).thenReturn(typeResolver()); + return context; + } +} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/CallSiteSpecificationTest.java b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/CallSiteSpecificationTest.java new file mode 100644 index 00000000000..a0d60d379c1 --- /dev/null +++ b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/CallSiteSpecificationTest.java @@ -0,0 +1,69 @@ +package datadog.trace.plugin.csi.impl; + +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; + +import datadog.trace.agent.tooling.csi.CallSiteAdvice; +import datadog.trace.plugin.csi.ValidationContext; +import datadog.trace.plugin.csi.impl.CallSiteSpecification.AdviceSpecification; +import datadog.trace.plugin.csi.util.ErrorCode; +import java.util.Collections; +import java.util.List; +import java.util.Set; +import org.junit.jupiter.api.Test; +import org.objectweb.asm.Type; + +class CallSiteSpecificationTest extends BaseCsiPluginTest { + + @Test + void testCallSiteSpiShouldBeAnInterface() { + ValidationContext context = mockValidationContext(); + AdviceSpecification mockAdvice = mock(AdviceSpecification.class); + List advices = Collections.singletonList(mockAdvice); + Set spiTypes = Collections.singleton(Type.getType(String.class)); + List helperClassNames = Collections.emptyList(); + Set constants = Collections.emptySet(); + CallSiteSpecification spec = + new CallSiteSpecification( + Type.getType(String.class), advices, spiTypes, helperClassNames, constants); + + spec.validate(context); + + verify(context).addError(eq(ErrorCode.CALL_SITE_SPI_SHOULD_BE_AN_INTERFACE), any()); + } + + @Test + void testCallSiteSpiShouldNotDefineAnyMethods() { + ValidationContext context = mockValidationContext(); + AdviceSpecification mockAdvice = mock(AdviceSpecification.class); + List advices = Collections.singletonList(mockAdvice); + Set spiTypes = Collections.singleton(Type.getType(Comparable.class)); + List helperClassNames = Collections.emptyList(); + Set constants = Collections.emptySet(); + CallSiteSpecification spec = + new CallSiteSpecification( + Type.getType(String.class), advices, spiTypes, helperClassNames, constants); + + spec.validate(context); + + verify(context).addError(eq(ErrorCode.CALL_SITE_SPI_SHOULD_BE_EMPTY), any()); + } + + @Test + void testCallSiteShouldHaveAdvices() { + ValidationContext context = mockValidationContext(); + List advices = Collections.emptyList(); + Set spiTypes = Collections.singleton(Type.getType(CallSiteAdvice.class)); + List helperClassNames = Collections.emptyList(); + Set constants = Collections.emptySet(); + CallSiteSpecification spec = + new CallSiteSpecification( + Type.getType(String.class), advices, spiTypes, helperClassNames, constants); + + spec.validate(context); + + verify(context).addError(eq(ErrorCode.CALL_SITE_SHOULD_HAVE_ADVICE_METHODS), any()); + } +} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/RegexpAdvicePointcutParserTest.java b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/RegexpAdvicePointcutParserTest.java new file mode 100644 index 00000000000..550082ccb7d --- /dev/null +++ b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/RegexpAdvicePointcutParserTest.java @@ -0,0 +1,162 @@ +package datadog.trace.plugin.csi.impl; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +import datadog.trace.plugin.csi.util.MethodType; +import org.junit.jupiter.api.Test; + +class RegexpAdvicePointcutParserTest { + + @Test + void resolveConstructor() { + RegexpAdvicePointcutParser pointcutParser = new RegexpAdvicePointcutParser(); + + MethodType signature = + pointcutParser.parse( + "void datadog.trace.plugin.csi.samples.SignatureParserExample.()"); + + assertEquals( + "datadog.trace.plugin.csi.samples.SignatureParserExample", + signature.getOwner().getClassName()); + assertEquals("", signature.getMethodName()); + assertEquals("()V", signature.getMethodType().getDescriptor()); + } + + @Test + void resolveConstructorWithArgs() { + RegexpAdvicePointcutParser pointcutParser = new RegexpAdvicePointcutParser(); + + MethodType signature = + pointcutParser.parse( + "void datadog.trace.plugin.csi.samples.SignatureParserExample.(java.lang.String)"); + + assertEquals( + "datadog.trace.plugin.csi.samples.SignatureParserExample", + signature.getOwner().getClassName()); + assertEquals("", signature.getMethodName()); + assertEquals("(Ljava/lang/String;)V", signature.getMethodType().getDescriptor()); + } + + @Test + void resolveWithoutArgs() { + RegexpAdvicePointcutParser pointcutParser = new RegexpAdvicePointcutParser(); + + MethodType signature = + pointcutParser.parse( + "java.lang.String datadog.trace.plugin.csi.samples.SignatureParserExample.noParams()"); + + assertEquals( + "datadog.trace.plugin.csi.samples.SignatureParserExample", + signature.getOwner().getClassName()); + assertEquals("noParams", signature.getMethodName()); + assertEquals("()Ljava/lang/String;", signature.getMethodType().getDescriptor()); + } + + @Test + void resolveOneParam() { + RegexpAdvicePointcutParser pointcutParser = new RegexpAdvicePointcutParser(); + + MethodType signature = + pointcutParser.parse( + "java.lang.String datadog.trace.plugin.csi.samples.SignatureParserExample.oneParam(java.util.Map)"); + + assertEquals( + "datadog.trace.plugin.csi.samples.SignatureParserExample", + signature.getOwner().getClassName()); + assertEquals("oneParam", signature.getMethodName()); + assertEquals("(Ljava/util/Map;)Ljava/lang/String;", signature.getMethodType().getDescriptor()); + } + + @Test + void resolveMultipleParams() { + RegexpAdvicePointcutParser pointcutParser = new RegexpAdvicePointcutParser(); + + MethodType signature = + pointcutParser.parse( + "java.lang.String datadog.trace.plugin.csi.samples.SignatureParserExample.multipleParams(java.lang.String, int, java.util.List)"); + + assertEquals( + "datadog.trace.plugin.csi.samples.SignatureParserExample", + signature.getOwner().getClassName()); + assertEquals("multipleParams", signature.getMethodName()); + assertEquals( + "(Ljava/lang/String;ILjava/util/List;)Ljava/lang/String;", + signature.getMethodType().getDescriptor()); + } + + @Test + void resolveVarargs() { + RegexpAdvicePointcutParser pointcutParser = new RegexpAdvicePointcutParser(); + + MethodType signature = + pointcutParser.parse( + "java.lang.String datadog.trace.plugin.csi.samples.SignatureParserExample.varargs(java.lang.String[])"); + + assertEquals( + "datadog.trace.plugin.csi.samples.SignatureParserExample", + signature.getOwner().getClassName()); + assertEquals("varargs", signature.getMethodName()); + assertEquals( + "([Ljava/lang/String;)Ljava/lang/String;", signature.getMethodType().getDescriptor()); + } + + @Test + void resolvePrimitive() { + RegexpAdvicePointcutParser pointcutParser = new RegexpAdvicePointcutParser(); + + MethodType signature = + pointcutParser.parse( + "int datadog.trace.plugin.csi.samples.SignatureParserExample.primitive()"); + + assertEquals( + "datadog.trace.plugin.csi.samples.SignatureParserExample", + signature.getOwner().getClassName()); + assertEquals("primitive", signature.getMethodName()); + assertEquals("()I", signature.getMethodType().getDescriptor()); + } + + @Test + void resolvePrimitiveArrayType() { + RegexpAdvicePointcutParser pointcutParser = new RegexpAdvicePointcutParser(); + + MethodType signature = + pointcutParser.parse( + "byte[] datadog.trace.plugin.csi.samples.SignatureParserExample.primitiveArray()"); + + assertEquals( + "datadog.trace.plugin.csi.samples.SignatureParserExample", + signature.getOwner().getClassName()); + assertEquals("primitiveArray", signature.getMethodName()); + assertEquals("()[B", signature.getMethodType().getDescriptor()); + } + + @Test + void resolveObjectArrayType() { + RegexpAdvicePointcutParser pointcutParser = new RegexpAdvicePointcutParser(); + + MethodType signature = + pointcutParser.parse( + "java.lang.Object[] datadog.trace.plugin.csi.samples.SignatureParserExample.objectArray()"); + + assertEquals( + "datadog.trace.plugin.csi.samples.SignatureParserExample", + signature.getOwner().getClassName()); + assertEquals("objectArray", signature.getMethodName()); + assertEquals("()[Ljava/lang/Object;", signature.getMethodType().getDescriptor()); + } + + @Test + void resolveMultiDimensionalObjectArrayType() { + RegexpAdvicePointcutParser pointcutParser = new RegexpAdvicePointcutParser(); + + MethodType signature = + pointcutParser.parse( + "java.lang.Object[][][] datadog.trace.plugin.csi.samples.SignatureParserExample.objectArray()"); + + assertEquals( + "datadog.trace.plugin.csi.samples.SignatureParserExample", + signature.getOwner().getClassName()); + assertEquals("objectArray", signature.getMethodName()); + assertEquals("()[[[Ljava/lang/Object;", signature.getMethodType().getDescriptor()); + } +} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/TypeResolverPoolTest.java b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/TypeResolverPoolTest.java new file mode 100644 index 00000000000..39a49d26c8b --- /dev/null +++ b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/TypeResolverPoolTest.java @@ -0,0 +1,67 @@ +package datadog.trace.plugin.csi.impl; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +import datadog.trace.plugin.csi.util.MethodType; +import java.lang.reflect.Method; +import javax.servlet.ServletRequest; +import javax.servlet.http.HttpServletRequest; +import org.junit.jupiter.api.Test; +import org.objectweb.asm.Type; + +class TypeResolverPoolTest { + + TypeResolverPool resolver = new TypeResolverPool(); + + @Test + void testResolvePrimitive() { + assertEquals(int.class, resolver.resolveType(Type.INT_TYPE)); + } + + @Test + void testResolvePrimitiveArray() { + Type type = Type.getType("[I"); + assertEquals(int[].class, resolver.resolveType(type)); + } + + @Test + void testResolvePrimitiveMultidimensionalArray() { + Type type = Type.getType("[[[I"); + assertEquals(int[][][].class, resolver.resolveType(type)); + } + + @Test + void testResolveClass() { + Type type = Type.getType(String.class); + assertEquals(String.class, resolver.resolveType(type)); + } + + @Test + void testResolveClassArray() { + Type type = Type.getType(String[].class); + assertEquals(String[].class, resolver.resolveType(type)); + } + + @Test + void testResolveClassMultidimensionalArray() { + Type type = Type.getType(String[][][].class); + assertEquals(String[][][].class, resolver.resolveType(type)); + } + + @Test + void testTypeResolverFromMethod() { + Type type = + Type.getMethodType( + Type.getType(String[].class), Type.getType(String.class), Type.getType(String.class)); + assertEquals(String[].class, resolver.resolveType(type.getReturnType())); + } + + @Test + void testInheritedMethods() throws Exception { + Type owner = Type.getType(HttpServletRequest.class); + String name = "getParameter"; + Type descriptor = Type.getMethodType(Type.getType(String.class), Type.getType(String.class)); + Method result = (Method) resolver.resolveMethod(new MethodType(owner, name, descriptor)); + assertEquals(ServletRequest.class.getDeclaredMethod("getParameter", String.class), result); + } +} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/assertion/AdviceAssert.java b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/assertion/AdviceAssert.java new file mode 100644 index 00000000000..9f814d7171d --- /dev/null +++ b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/assertion/AdviceAssert.java @@ -0,0 +1,37 @@ +package datadog.trace.plugin.csi.impl.assertion; + +import static org.junit.jupiter.api.Assertions.assertArrayEquals; +import static org.junit.jupiter.api.Assertions.assertEquals; + +import java.util.List; + +public class AdviceAssert { + protected String type; + protected String owner; + protected String method; + protected String descriptor; + protected List statements; + + public AdviceAssert( + String type, String owner, String method, String descriptor, List statements) { + this.type = type; + this.owner = owner; + this.method = method; + this.descriptor = descriptor; + this.statements = statements; + } + + public void type(String type) { + assertEquals(type, this.type); + } + + public void pointcut(String owner, String method, String descriptor) { + assertEquals(owner, this.owner); + assertEquals(method, this.method); + assertEquals(descriptor, this.descriptor); + } + + public void statements(String... values) { + assertArrayEquals(values, statements.toArray(new String[0])); + } +} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/assertion/AssertBuilder.java b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/assertion/AssertBuilder.java new file mode 100644 index 00000000000..42e87842adc --- /dev/null +++ b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/assertion/AssertBuilder.java @@ -0,0 +1,205 @@ +package datadog.trace.plugin.csi.impl.assertion; + +import static datadog.trace.plugin.csi.impl.CallSiteFactory.typeResolver; +import static datadog.trace.plugin.csi.util.CallSiteUtils.classNameToType; +import static org.junit.jupiter.api.Assertions.assertEquals; + +import com.github.javaparser.JavaParser; +import com.github.javaparser.ParserConfiguration; +import com.github.javaparser.ast.CompilationUnit; +import com.github.javaparser.ast.Node; +import com.github.javaparser.ast.body.ClassOrInterfaceDeclaration; +import com.github.javaparser.ast.body.MethodDeclaration; +import com.github.javaparser.ast.expr.MethodCallExpr; +import com.github.javaparser.resolution.declarations.ResolvedMethodDeclaration; +import com.github.javaparser.symbolsolver.JavaSymbolSolver; +import datadog.trace.agent.tooling.csi.CallSites; +import java.io.File; +import java.io.FileNotFoundException; +import java.lang.reflect.Field; +import java.lang.reflect.Method; +import java.util.Collections; +import java.util.List; +import java.util.Set; +import java.util.stream.Collectors; + +public class AssertBuilder { + protected final File file; + + public AssertBuilder(File file) { + this.file = file; + } + + public CallSiteAssert build() { + CompilationUnit javaFile; + javaFile = parseJavaFile(file); + assertEquals(Node.Parsedness.PARSED, javaFile.getParsed()); + ClassOrInterfaceDeclaration targetType = + javaFile.getPrimaryType().get().asClassOrInterfaceDeclaration(); + Set> interfaces = getInterfaces(targetType); + Method enabled = null; + Set enabledArgs = null; + Object[] enabledDeclaration = getEnabledDeclaration(targetType, interfaces); + enabled = (Method) enabledDeclaration[0]; + enabledArgs = (Set) enabledDeclaration[1]; + return new CallSiteAssert( + interfaces, + getSpi(targetType), + getHelpers(targetType), + getAdvices(targetType), + enabled, + enabledArgs); + } + + protected Set> getSpi(ClassOrInterfaceDeclaration type) { + return type.getAnnotationByName("AutoService") + .>>map( + annotation -> + annotation.asNormalAnnotationExpr().getPairs().stream() + .filter(pair -> pair.getNameAsString().equals("value")) + .flatMap( + pair -> + pair.getValue().asArrayInitializerExpr().getValues().stream() + .map( + value -> + value + .asClassExpr() + .getType() + .resolve() + .asReferenceType() + .getTypeDeclaration() + .get() + .getQualifiedName())) + .map(AssertBuilder::loadClass) + .collect(Collectors.toSet())) + .orElse(Collections.emptySet()); + } + + protected Set> getInterfaces(ClassOrInterfaceDeclaration type) { + return type.getImplementedTypes().stream() + .map( + implementedType -> { + String qualifiedName = + implementedType + .asClassOrInterfaceType() + .resolve() + .asReferenceType() + .getTypeDeclaration() + .get() + .getQualifiedName(); + return loadClass(qualifiedName); + }) + .collect(Collectors.toSet()); + } + + private static Class loadClass(String qualifiedName) { + // Try progressively replacing dots with $ from right to left for inner classes + String current = qualifiedName; + int lastDot = current.lastIndexOf('.'); + do { + try { + return Class.forName(current); + } catch (ClassNotFoundException e) { + if (lastDot <= 0) { + throw new RuntimeException(new ClassNotFoundException(qualifiedName)); + } + current = current.substring(0, lastDot) + "$" + current.substring(lastDot + 1); + lastDot = current.lastIndexOf('.', lastDot - 1); + } + } while (true); + } + + protected Object[] getEnabledDeclaration( + ClassOrInterfaceDeclaration type, Set> interfaces) { + if (!interfaces.contains(CallSites.HasEnabledProperty.class)) { + return new Object[] {null, null}; + } + MethodDeclaration isEnabled = type.getMethodsByName("isEnabled").get(0); + MethodCallExpr enabledMethodCall = + isEnabled + .getBody() + .get() + .getStatements() + .get(0) + .asReturnStmt() + .getExpression() + .get() + .asMethodCallExpr(); + Method enabled = resolveMethod(enabledMethodCall); + Set enabledArgs = + enabledMethodCall.getArguments().stream() + .map(arg -> arg.asStringLiteralExpr().asString()) + .collect(Collectors.toSet()); + return new Object[] {enabled, enabledArgs}; + } + + protected Set> getHelpers(ClassOrInterfaceDeclaration type) { + MethodDeclaration acceptMethod = type.getMethodsByName("accept").get(0); + List methodCalls = getMethodCalls(acceptMethod); + return methodCalls.stream() + .filter(methodCall -> methodCall.getNameAsString().equals("addHelpers")) + .flatMap(methodCall -> methodCall.getArguments().stream()) + .map( + arg -> { + String className = arg.asStringLiteralExpr().asString(); + return typeResolver().resolveType(classNameToType(className)); + }) + .collect(Collectors.toSet()); + } + + protected List getAdvices(ClassOrInterfaceDeclaration type) { + MethodDeclaration acceptMethod = type.getMethodsByName("accept").get(0); + return getMethodCalls(acceptMethod).stream() + .filter(methodCall -> methodCall.getNameAsString().equals("addAdvice")) + .map( + methodCall -> { + String adviceType = methodCall.getArgument(0).asFieldAccessExpr().getNameAsString(); + String owner = methodCall.getArgument(1).asStringLiteralExpr().asString(); + String method = methodCall.getArgument(2).asStringLiteralExpr().asString(); + String descriptor = methodCall.getArgument(3).asStringLiteralExpr().asString(); + List statements = + methodCall + .getArgument(4) + .asLambdaExpr() + .getBody() + .asBlockStmt() + .getStatements() + .stream() + .map(Object::toString) + .collect(Collectors.toList()); + return new AdviceAssert(adviceType, owner, method, descriptor, statements); + }) + .collect(Collectors.toList()); + } + + protected static List getMethodCalls(MethodDeclaration method) { + return method.getBody().get().asBlockStmt().getStatements().stream() + .filter( + stmt -> + stmt.isExpressionStmt() + && stmt.asExpressionStmt().getExpression().isMethodCallExpr()) + .map(stmt -> stmt.asExpressionStmt().getExpression().asMethodCallExpr()) + .collect(Collectors.toList()); + } + + private static Method resolveMethod(MethodCallExpr methodCallExpr) { + ResolvedMethodDeclaration resolved = methodCallExpr.resolve(); + try { + Field methodField = resolved.getClass().getDeclaredField("method"); + methodField.setAccessible(true); + return (Method) methodField.get(resolved); + } catch (IllegalAccessException | NoSuchFieldException e) { + throw new RuntimeException(e); + } + } + + private static CompilationUnit parseJavaFile(File file) { + JavaSymbolSolver solver = new JavaSymbolSolver(typeResolver()); + JavaParser parser = new JavaParser(new ParserConfiguration().setSymbolResolver(solver)); + try { + return parser.parse(file).getResult().get(); + } catch (FileNotFoundException e) { + throw new RuntimeException(e); + } + } +} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/assertion/CallSiteAssert.java b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/assertion/CallSiteAssert.java new file mode 100644 index 00000000000..57ce0146921 --- /dev/null +++ b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/assertion/CallSiteAssert.java @@ -0,0 +1,88 @@ +package datadog.trace.plugin.csi.impl.assertion; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.lang.reflect.Method; +import java.util.Arrays; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.function.Consumer; + +public class CallSiteAssert { + + protected Set> interfaces; + protected Set> spi; + protected Set> helpers; + protected List advices; + protected Method enabled; + protected Set enabledArgs; + + public CallSiteAssert( + Set> interfaces, + Set> spi, + Set> helpers, + List advices, + Method enabled, + Set enabledArgs) { + this.interfaces = interfaces; + this.spi = spi; + this.helpers = helpers; + this.advices = advices; + this.enabled = enabled; + this.enabledArgs = enabledArgs; + } + + public void interfaces(Class... values) { + assertSameElements(interfaces, values); + } + + public void helpers(Class... values) { + assertSameElements(helpers, values); + } + + public void spi(Class... values) { + assertSameElements(spi, values); + } + + public void advices(int index, Consumer assertions) { + AdviceAssert asserter = advices.get(index); + assertions.accept(asserter); + } + + public void enabled(Method method, String... args) { + assertEquals(method, enabled); + assertSameElements(enabledArgs, args); + } + + private static void assertSameElements(Set expected, E... received) { + assertEquals(received.length, expected.size()); + Set receivedSet = new HashSet<>(Arrays.asList(received)); + assertTrue(expected.containsAll(receivedSet) && receivedSet.containsAll(expected)); + } + + public Set> getInterfaces() { + return interfaces; + } + + public Set> getSpi() { + return spi; + } + + public Set> getHelpers() { + return helpers; + } + + public List getAdvices() { + return advices; + } + + public Method getEnabled() { + return enabled; + } + + public Set getEnabledArgs() { + return enabledArgs; + } +} diff --git a/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/ext/IastExtensionTest.java b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/ext/IastExtensionTest.java new file mode 100644 index 00000000000..cf42e18cbd5 --- /dev/null +++ b/buildSrc/call-site-instrumentation-plugin/src/test/java/datadog/trace/plugin/csi/impl/ext/IastExtensionTest.java @@ -0,0 +1,314 @@ +package datadog.trace.plugin.csi.impl.ext; + +import static datadog.trace.plugin.csi.impl.CallSiteFactory.pointcutParser; +import static datadog.trace.plugin.csi.util.CallSiteUtils.classNameToType; +import static org.junit.jupiter.api.Assertions.assertArrayEquals; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +import com.github.javaparser.JavaParser; +import com.github.javaparser.ast.body.ClassOrInterfaceDeclaration; +import com.github.javaparser.ast.stmt.IfStmt; +import datadog.trace.plugin.csi.AdviceGenerator; +import datadog.trace.plugin.csi.AdviceGenerator.CallSiteResult; +import datadog.trace.plugin.csi.PluginApplication.Configuration; +import datadog.trace.plugin.csi.impl.AdviceGeneratorImpl; +import datadog.trace.plugin.csi.impl.BaseCsiPluginTest; +import datadog.trace.plugin.csi.impl.CallSiteSpecification; +import datadog.trace.plugin.csi.impl.assertion.AdviceAssert; +import datadog.trace.plugin.csi.impl.assertion.AssertBuilder; +import datadog.trace.plugin.csi.impl.assertion.CallSiteAssert; +import datadog.trace.plugin.csi.impl.ext.tests.IastExtensionCallSite; +import java.io.File; +import java.io.IOException; +import java.lang.reflect.Method; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Collections; +import java.util.List; +import java.util.Set; +import java.util.function.Consumer; +import java.util.stream.Collectors; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; +import org.objectweb.asm.Type; + +class IastExtensionTest extends BaseCsiPluginTest { + + @TempDir private File buildDir; + private Path targetFolder; + private Path projectFolder; + private Path srcFolder; + + @BeforeEach + void setup() throws Exception { + targetFolder = createFolder("target"); + projectFolder = createFolder("project"); + srcFolder = createFolder("src/main/java"); + } + + private Path createFolder(String folderName) throws IOException { + Path folder = buildDir.toPath().resolve(folderName); + Files.createDirectories(folder); + return folder; + } + + @ParameterizedTest + @CsvSource( + delimiter = '|', + nullValues = "null", + value = { + "datadog.trace.agent.tooling.csi.CallSites | false", + "datadog.trace.api.iast.IastCallSites | true" + }) + void testThatExtensionOnlyAppliesToIastAdvices(String typeName, boolean expected) { + Type type = classNameToType(typeName); + Type[] types = new Type[] {type}; + CallSiteSpecification callSite = mock(CallSiteSpecification.class); + when(callSite.getSpi()).thenReturn(types); + IastExtension extension = new IastExtension(); + + boolean applies = extension.appliesTo(callSite); + + assertEquals(expected, applies); + } + + @Test + void testThatExtensionGeneratesACallSiteWithTelemetry() throws Exception { + Configuration config = mock(Configuration.class); + when(config.getTargetFolder()).thenReturn(targetFolder); + when(config.getSrcFolder()).thenReturn(getCallSiteSrcFolder()); + when(config.getClassPath()).thenReturn(Collections.emptyList()); + CallSiteSpecification spec = buildClassSpecification(IastExtensionCallSite.class); + AdviceGenerator generator = buildAdviceGenerator(buildDir); + CallSiteResult result = generator.generate(spec); + assertTrue(result.isSuccess()); + IastExtension extension = new IastExtension(); + + extension.apply(config, result); + + assertNoErrors(result); + IastExtensionCallSiteAssert asserter = assertCallSites(result.getFile()); + asserter.iastAdvices( + 0, + advice -> { + advice.pointcut( + "javax/servlet/http/HttpServletRequest", + "getHeader", + "(Ljava/lang/String;)Ljava/lang/String;"); + advice.instrumentedMetric( + "IastMetric.INSTRUMENTED_SOURCE", + metric -> { + metric.metricStatements( + "IastMetricCollector.add(IastMetric.INSTRUMENTED_SOURCE, (byte) 3, 1);"); + }); + advice.executedMetric( + "IastMetric.EXECUTED_SOURCE", + metric -> { + metric.metricStatements( + "handler.field(net.bytebuddy.jar.asm.Opcodes.GETSTATIC, \"datadog/trace/api/iast/telemetry/IastMetric\", \"EXECUTED_SOURCE\", \"Ldatadog/trace/api/iast/telemetry/IastMetric;\");", + "handler.instruction(net.bytebuddy.jar.asm.Opcodes.ICONST_3);", + "handler.instruction(net.bytebuddy.jar.asm.Opcodes.ICONST_1);", + "handler.method(net.bytebuddy.jar.asm.Opcodes.INVOKESTATIC, \"datadog/trace/api/iast/telemetry/IastMetricCollector\", \"add\", \"(Ldatadog/trace/api/iast/telemetry/IastMetric;BI)V\", false);"); + }); + }); + asserter.iastAdvices( + 1, + advice -> { + advice.pointcut( + "javax/servlet/http/HttpServletRequest", + "getInputStream", + "()Ljavax/servlet/ServletInputStream;"); + advice.instrumentedMetric( + "IastMetric.INSTRUMENTED_SOURCE", + metric -> { + metric.metricStatements( + "IastMetricCollector.add(IastMetric.INSTRUMENTED_SOURCE, (byte) 127, 1);"); + }); + advice.executedMetric( + "IastMetric.EXECUTED_SOURCE", + metric -> { + metric.metricStatements( + "handler.field(net.bytebuddy.jar.asm.Opcodes.GETSTATIC, \"datadog/trace/api/iast/telemetry/IastMetric\", \"EXECUTED_SOURCE\", \"Ldatadog/trace/api/iast/telemetry/IastMetric;\");", + "handler.instruction(net.bytebuddy.jar.asm.Opcodes.BIPUSH, 127);", + "handler.instruction(net.bytebuddy.jar.asm.Opcodes.ICONST_1);", + "handler.method(net.bytebuddy.jar.asm.Opcodes.INVOKESTATIC, \"datadog/trace/api/iast/telemetry/IastMetricCollector\", \"add\", \"(Ldatadog/trace/api/iast/telemetry/IastMetric;BI)V\", false);"); + }); + }); + asserter.iastAdvices( + 2, + advice -> { + advice.pointcut( + "javax/servlet/ServletRequest", "getReader", "()Ljava/io/BufferedReader;"); + advice.instrumentedMetric( + "IastMetric.INSTRUMENTED_PROPAGATION", + metric -> { + metric.metricStatements( + "IastMetricCollector.add(IastMetric.INSTRUMENTED_PROPAGATION, 1);"); + }); + advice.executedMetric( + "IastMetric.EXECUTED_PROPAGATION", + metric -> { + metric.metricStatements( + "handler.field(net.bytebuddy.jar.asm.Opcodes.GETSTATIC, \"datadog/trace/api/iast/telemetry/IastMetric\", \"EXECUTED_PROPAGATION\", \"Ldatadog/trace/api/iast/telemetry/IastMetric;\");", + "handler.instruction(net.bytebuddy.jar.asm.Opcodes.ICONST_1);", + "handler.method(net.bytebuddy.jar.asm.Opcodes.INVOKESTATIC, \"datadog/trace/api/iast/telemetry/IastMetricCollector\", \"add\", \"(Ldatadog/trace/api/iast/telemetry/IastMetric;I)V\", false);"); + }); + }); + } + + private static AdviceGenerator buildAdviceGenerator(File targetFolder) { + return new AdviceGeneratorImpl(targetFolder, pointcutParser()); + } + + private static Path getCallSiteSrcFolder() throws Exception { + File file = new File(Thread.currentThread().getContextClassLoader().getResource("").toURI()); + return file.toPath().resolve("../../../../src/test/java"); + } + + private static ClassOrInterfaceDeclaration parse(File path) throws Exception { + return new JavaParser() + .parse(path) + .getResult() + .get() + .getPrimaryType() + .get() + .asClassOrInterfaceDeclaration(); + } + + private static IastExtensionCallSiteAssert assertCallSites(File generated) { + try { + return new IastExtensionAssertBuilder(generated).build(); + } catch (Exception e) { + throw new RuntimeException(e); + } + } + + static class IastExtensionCallSiteAssert extends CallSiteAssert { + + IastExtensionCallSiteAssert( + Set> interfaces, + Set> spi, + Set> helpers, + List advices, + Method enabled, + Set enabledArgs) { + super(interfaces, spi, helpers, advices, enabled, enabledArgs); + } + + public void iastAdvices(int index, Consumer assertions) { + IastExtensionAdviceAssert asserter = (IastExtensionAdviceAssert) advices.get(index); + assertions.accept(asserter); + } + } + + static class IastExtensionAdviceAssert extends AdviceAssert { + + protected IastExtensionMetricAsserter instrumented; + protected IastExtensionMetricAsserter executed; + + IastExtensionAdviceAssert( + String owner, + String method, + String descriptor, + IastExtensionMetricAsserter instrumented, + IastExtensionMetricAsserter executed, + List statements) { + super(null, owner, method, descriptor, statements); + this.instrumented = instrumented; + this.executed = executed; + } + + public void instrumentedMetric( + String metric, Consumer assertions) { + assertEquals(metric, instrumented.metric); + assertions.accept(instrumented); + } + + public void executedMetric(String metric, Consumer assertions) { + assertEquals(metric, executed.metric); + assertions.accept(executed); + } + } + + static class IastExtensionMetricAsserter { + protected String metric; + protected List statements; + + IastExtensionMetricAsserter(String metric, List statements) { + this.metric = metric; + this.statements = statements; + } + + public void metricStatements(String... values) { + assertArrayEquals(values, statements.toArray(new String[0])); + } + } + + static class IastExtensionAssertBuilder extends AssertBuilder { + + IastExtensionAssertBuilder(File file) { + super(file); + } + + @Override + public IastExtensionCallSiteAssert build() { + CallSiteAssert base = super.build(); + return new IastExtensionCallSiteAssert( + base.getInterfaces(), + base.getSpi(), + base.getHelpers(), + base.getAdvices(), + base.getEnabled(), + base.getEnabledArgs()); + } + + @Override + protected List getAdvices(ClassOrInterfaceDeclaration type) { + return getMethodCalls(type.getMethodsByName("accept").get(0)).stream() + .filter(methodCall -> methodCall.getNameAsString().equals("addAdvice")) + .map( + methodCall -> { + String owner = methodCall.getArgument(1).asStringLiteralExpr().asString(); + String method = methodCall.getArgument(2).asStringLiteralExpr().asString(); + String descriptor = methodCall.getArgument(3).asStringLiteralExpr().asString(); + List statements = + methodCall + .getArgument(4) + .asLambdaExpr() + .getBody() + .asBlockStmt() + .getStatements(); + IfStmt instrumentedStmt = statements.get(0).asIfStmt(); + IfStmt executedStmt = statements.get(1).asIfStmt(); + List nonIfStatements = + statements.stream() + .filter(stmt -> !stmt.isIfStmt()) + .map(Object::toString) + .collect(Collectors.toList()); + return new IastExtensionAdviceAssert( + owner, + method, + descriptor, + buildMetricAsserter(instrumentedStmt), + buildMetricAsserter(executedStmt), + nonIfStatements); + }) + .collect(Collectors.toList()); + } + + protected IastExtensionMetricAsserter buildMetricAsserter(IfStmt ifStmt) { + String metric = ifStmt.getCondition().asMethodCallExpr().getScope().get().toString(); + List statements = + ifStmt.getThenStmt().asBlockStmt().getStatements().stream() + .map(Object::toString) + .collect(Collectors.toList()); + return new IastExtensionMetricAsserter(metric, statements); + } + } +} diff --git a/buildSrc/src/main/groovy/InstrumentPlugin.groovy b/buildSrc/src/main/groovy/InstrumentPlugin.groovy deleted file mode 100644 index 781e059d765..00000000000 --- a/buildSrc/src/main/groovy/InstrumentPlugin.groovy +++ /dev/null @@ -1,210 +0,0 @@ -import org.gradle.api.DefaultTask -import org.gradle.api.Plugin -import org.gradle.api.Project -import org.gradle.api.file.ConfigurableFileCollection -import org.gradle.api.file.Directory -import org.gradle.api.file.DirectoryProperty -import org.gradle.api.invocation.BuildInvocationDetails -import org.gradle.api.provider.ListProperty -import org.gradle.api.provider.Property -import org.gradle.api.tasks.Classpath -import org.gradle.api.tasks.Input -import org.gradle.api.tasks.InputDirectory -import org.gradle.api.tasks.InputFiles -import org.gradle.api.tasks.Optional -import org.gradle.api.tasks.OutputDirectory -import org.gradle.api.tasks.TaskAction -import org.gradle.api.tasks.compile.AbstractCompile -import org.gradle.jvm.toolchain.JavaLanguageVersion -import org.gradle.jvm.toolchain.JavaToolchainService -import org.gradle.workers.WorkAction -import org.gradle.workers.WorkParameters -import org.gradle.workers.WorkerExecutor - -import javax.inject.Inject -import java.util.concurrent.ConcurrentHashMap -import java.util.regex.Matcher - -/** - * instrument task plugin which performs build-time instrumentation of classes. - */ -@SuppressWarnings('unused') -class InstrumentPlugin implements Plugin { - @Override - void apply(Project project) { - InstrumentExtension extension = project.extensions.create('instrument', InstrumentExtension) - - project.tasks.matching { - it.name in ['compileJava', 'compileScala', 'compileGroovy'] || - it.name =~ /compileMain_.+Java/ - }.all { - AbstractCompile compileTask = it as AbstractCompile - Matcher versionMatcher = it.name =~ /compileMain_(.+)Java/ - project.afterEvaluate { - if (!compileTask.source.empty) { - String sourceSetSuffix = null - String javaVersion = null - if (versionMatcher.matches()) { - sourceSetSuffix = versionMatcher.group(1) - if (sourceSetSuffix ==~ /java\d+/) { - javaVersion = sourceSetSuffix[4..-1] - } - } - - // insert intermediate 'raw' directory for unprocessed classes - Directory classesDir = compileTask.destinationDirectory.get() - Directory rawClassesDir = classesDir.dir( - "../raw${sourceSetSuffix ? "_$sourceSetSuffix" : ''}/") - compileTask.destinationDirectory.set(rawClassesDir.asFile) - - // insert task between compile and jar, and before test* - String instrumentTaskName = compileTask.name.replace('compile', 'instrument') - def instrumentTask = project.tasks.register(instrumentTaskName, InstrumentTask) { - // Task configuration - it.group = 'Byte Buddy' - it.description = "Instruments the classes compiled by ${compileTask.name}" - it.inputs.dir(compileTask.destinationDirectory) - it.outputs.dir(classesDir) - // Task inputs - it.javaVersion = javaVersion - def instrumenterConfiguration = project.configurations.named('instrumentPluginClasspath') - if (instrumenterConfiguration.present) { - it.pluginClassPath.from(instrumenterConfiguration.get()) - } - it.plugins = extension.plugins - it.instrumentingClassPath.from( - findCompileClassPath(project, it.name) + - rawClassesDir + - findAdditionalClassPath(extension, it.name) - ) - it.sourceDirectory = rawClassesDir - // Task output - it.targetDirectory = classesDir - } - if (javaVersion) { - project.tasks.named(project.sourceSets."main_java${javaVersion}".classesTaskName) { - it.dependsOn(instrumentTask) - } - } else { - project.tasks.named(project.sourceSets.main.classesTaskName) { - it.dependsOn(instrumentTask) - } - } - } - } - } - } - - static findCompileClassPath(Project project, String taskName) { - def matcher = taskName =~ /instrument([A-Z].+)Java/ - def cfgName = matcher.matches() ? "${matcher.group(1).uncapitalize()}CompileClasspath" : 'compileClasspath' - project.configurations.named(cfgName).findAll { - it.name != 'previous-compilation-data.bin' && !it.name.endsWith('.gz') - } - } - - static findAdditionalClassPath(InstrumentExtension extension, String taskName) { - extension.additionalClasspath.getOrDefault(taskName, []).collect { - // insert intermediate 'raw' directory for unprocessed classes - def fileName = it.get().asFile.name - it.get().dir("../${fileName.replaceFirst('^main', 'raw')}") - } - } -} - -abstract class InstrumentExtension { - abstract ListProperty getPlugins() - Map> additionalClasspath = [:] -} - -abstract class InstrumentTask extends DefaultTask { - @Input @Optional - String javaVersion - @InputFiles @Classpath - abstract ConfigurableFileCollection getPluginClassPath() - @Input - ListProperty plugins - @InputFiles @Classpath - abstract ConfigurableFileCollection getInstrumentingClassPath() - @InputDirectory - Directory sourceDirectory - - @OutputDirectory - Directory targetDirectory - - @Inject - abstract JavaToolchainService getJavaToolchainService() - @Inject - abstract BuildInvocationDetails getInvocationDetails() - @Inject - abstract WorkerExecutor getWorkerExecutor() - - @TaskAction - instrument() { - workQueue().submit(InstrumentAction.class, parameters -> { - parameters.buildStartedTime.set(this.invocationDetails.buildStartedTime) - parameters.pluginClassPath.from(this.pluginClassPath) - parameters.plugins.set(this.plugins) - parameters.instrumentingClassPath.setFrom(this.instrumentingClassPath) - parameters.sourceDirectory.set(this.sourceDirectory.asFile) - parameters.targetDirectory.set(this.targetDirectory.asFile) - }) - } - - private workQueue() { - if (!this.javaVersion) { - this.javaVersion = "8" - } - def javaLauncher = this.javaToolchainService.launcherFor { spec -> - spec.languageVersion.set(JavaLanguageVersion.of(this.javaVersion)) - }.get() - return this.workerExecutor.processIsolation { spec -> - spec.forkOptions { fork -> - fork.executable = javaLauncher.executablePath - } - } - } -} - -interface InstrumentWorkParameters extends WorkParameters { - Property getBuildStartedTime() - ConfigurableFileCollection getPluginClassPath() - ListProperty getPlugins() - ConfigurableFileCollection getInstrumentingClassPath() - DirectoryProperty getSourceDirectory() - DirectoryProperty getTargetDirectory() -} - -abstract class InstrumentAction implements WorkAction { - private static final Object lock = new Object() - private static final Map classLoaderCache = new ConcurrentHashMap<>() - private static volatile long lastBuildStamp - - @Override - void execute() { - String[] plugins = parameters.getPlugins().get() as String[] - String classLoaderKey = plugins.join(':') - - // reset shared class-loaders each time a new build starts - long buildStamp = parameters.buildStartedTime.get() - ClassLoader pluginCL = classLoaderCache.get(classLoaderKey) - if (lastBuildStamp < buildStamp || !pluginCL) { - synchronized (lock) { - pluginCL = classLoaderCache.get(classLoaderKey) - if (lastBuildStamp < buildStamp || !pluginCL) { - pluginCL = createClassLoader(parameters.pluginClassPath) - classLoaderCache.put(classLoaderKey, pluginCL) - lastBuildStamp = buildStamp - } - } - } - File sourceDirectory = parameters.getSourceDirectory().get().asFile - File targetDirectory = parameters.getTargetDirectory().get().asFile - ClassLoader instrumentingCL = createClassLoader(parameters.instrumentingClassPath, pluginCL) - InstrumentingPlugin.instrumentClasses(plugins, instrumentingCL, sourceDirectory, targetDirectory) - } - - static ClassLoader createClassLoader(cp, parent = InstrumentAction.classLoader) { - return new URLClassLoader(cp*.toURI()*.toURL() as URL[], parent as ClassLoader) - } -} diff --git a/buildSrc/src/main/groovy/InstrumentingPlugin.groovy b/buildSrc/src/main/groovy/InstrumentingPlugin.groovy deleted file mode 100644 index 7710ee12539..00000000000 --- a/buildSrc/src/main/groovy/InstrumentingPlugin.groovy +++ /dev/null @@ -1,112 +0,0 @@ -import net.bytebuddy.ClassFileVersion -import net.bytebuddy.build.EntryPoint -import net.bytebuddy.build.Plugin -import net.bytebuddy.description.type.TypeDescription -import net.bytebuddy.dynamic.ClassFileLocator -import net.bytebuddy.dynamic.scaffold.inline.MethodNameTransformer.Suffixing -import net.bytebuddy.utility.StreamDrainer -import org.slf4j.Logger -import org.slf4j.LoggerFactory - -/** - * Performs build-time instrumentation of classes, called indirectly from InstrumentPlugin. - * (This is the byte-buddy side of the task; InstrumentPlugin contains the Gradle pieces.) - */ -class InstrumentingPlugin { - static final Logger log = LoggerFactory.getLogger(InstrumentingPlugin.class) - - static void instrumentClasses( - String[] plugins, ClassLoader instrumentingLoader, File sourceDirectory, File targetDirectory) - throws Exception { - - ClassLoader tccl = Thread.currentThread().getContextClassLoader() - try { - Thread.currentThread().setContextClassLoader(instrumentingLoader) - - List factories = new ArrayList<>() - for (String plugin : plugins) { - try { - Class pluginClass = (Class) instrumentingLoader.loadClass(plugin) - Plugin loadedPlugin = pluginClass.getConstructor(File.class).newInstance(targetDirectory) - factories.add(new Plugin.Factory.Simple(loadedPlugin)) - } catch (Throwable throwable) { - throw new IllegalStateException("Cannot resolve plugin: " + plugin, throwable) - } - } - - Plugin.Engine.Source source = new Plugin.Engine.Source.ForFolder(sourceDirectory) - Plugin.Engine.Target target = new Plugin.Engine.Target.ForFolder(targetDirectory) - - Plugin.Engine engine = - Plugin.Engine.Default.of( - EntryPoint.Default.REBASE, ClassFileVersion.ofThisVm(), Suffixing.withRandomSuffix()) - - Plugin.Engine.Summary summary = - engine - .with(Plugin.Engine.PoolStrategy.Default.FAST) - .with(new NonCachingClassFileLocator(instrumentingLoader)) - .with(new LoggingAdapter()) - .withErrorHandlers( - Plugin.Engine.ErrorHandler.Enforcing.ALL_TYPES_RESOLVED, - Plugin.Engine.ErrorHandler.Enforcing.NO_LIVE_INITIALIZERS, - new Plugin.Engine.ErrorHandler() { - @Delegate - Plugin.Engine.ErrorHandler delegate = Plugin.Engine.ErrorHandler.Failing.FAIL_LAST - - void onError(Map> throwables) { - throw new IllegalStateException("Failed to transform at least one type: " + throwables).tap { ise -> - throwables.values().flatten().each { - ise.addSuppressed(it) - } - }; - } - } - ) - .with(Plugin.Engine.Dispatcher.ForSerialTransformation.Factory.INSTANCE) - .apply(source, target, factories) - - if (!summary.getFailed().isEmpty()) { - throw new IllegalStateException("Failed to transform: " + summary.getFailed()) - } - } catch (Throwable e) { - Thread.currentThread().setContextClassLoader(tccl) - throw e - } - } - - static class LoggingAdapter extends Plugin.Engine.Listener.Adapter { - @Override - void onTransformation(TypeDescription typeDescription, List plugins) { - log.debug("Transformed {} using {}", typeDescription, plugins) - } - - @Override - void onError(TypeDescription typeDescription, Plugin plugin, Throwable throwable) { - log.warn("Failed to transform {} using {}", typeDescription, plugin, throwable) - } - } - - static class NonCachingClassFileLocator implements ClassFileLocator { - ClassLoader loader - - NonCachingClassFileLocator(ClassLoader loader) { - this.loader = loader - } - - @Override - Resolution locate(String name) throws IOException { - URL url = loader.getResource(name.replace('.', '/') + '.class') - if (null == url) { - return new Resolution.Illegal(name) - } - URLConnection uc = url.openConnection() - uc.setUseCaches(false) // avoid caching class-file resources in build-workers - try (InputStream is = uc.getInputStream()) { - return new Resolution.Explicit(StreamDrainer.DEFAULT.drain(is)) - } - } - - @Override - void close() {} - } -} diff --git a/buildSrc/src/main/kotlin/CIJobsExtensions.kt b/buildSrc/src/main/kotlin/CIJobsExtensions.kt deleted file mode 100644 index 3f837f27f6b..00000000000 --- a/buildSrc/src/main/kotlin/CIJobsExtensions.kt +++ /dev/null @@ -1,109 +0,0 @@ -package datadog.gradle.plugin.ci - -import org.gradle.api.Project -import org.gradle.api.Task -import org.gradle.kotlin.dsl.extra - -/** - * Returns the task's path, given affected projects, if this task or its dependencies are affected by git changes. - */ -internal fun findAffectedTaskPath(baseTask: Task, affectedProjects: Map>): String? { - val visited = mutableSetOf() - val queue = mutableListOf(baseTask) - - while (queue.isNotEmpty()) { - val t = queue.removeAt(0) - if (visited.contains(t)) { - continue - } - visited.add(t) - - val affectedTasks = affectedProjects[t.project] - if (affectedTasks != null) { - if (affectedTasks.contains("all")) { - return "${t.project.path}:${t.name}" - } - if (affectedTasks.contains(t.name)) { - return "${t.project.path}:${t.name}" - } - } - - t.taskDependencies.getDependencies(t).forEach { queue.add(it) } - } - return null -} - -/** - * Creates a single aggregate root task that depends on matching subproject tasks - */ -private fun Project.createRootTask( - rootTaskName: String, - subProjTaskName: String, - includePrefixes: List, - excludePrefixes: List, - forceCoverage: Boolean -) { - val coverage = forceCoverage || rootProject.providers.gradleProperty("checkCoverage").isPresent - tasks.register(rootTaskName) { - subprojects.forEach { subproject -> - val activePartition = subproject.extra.get("activePartition") as Boolean - if ( - activePartition && - includePrefixes.any { subproject.path.startsWith(it) } && - !excludePrefixes.any { subproject.path.startsWith(it) } - ) { - val testTask = subproject.tasks.findByName(subProjTaskName) - var isAffected = true - - if (testTask != null) { - val useGitChanges = rootProject.extra.get("useGitChanges") as Boolean - if (useGitChanges) { - @Suppress("UNCHECKED_CAST") - val affectedProjects = rootProject.extra.get("affectedProjects") as Map> - val affectedTaskPath = findAffectedTaskPath(testTask, affectedProjects) - if (affectedTaskPath != null) { - logger.warn("Selecting ${subproject.path}:$subProjTaskName (affected by $affectedTaskPath)") - } else { - logger.warn("Skipping ${subproject.path}:$subProjTaskName (not affected by changed files)") - isAffected = false - } - } - if (isAffected) { - dependsOn(testTask) - } - } - - if (isAffected && coverage) { - val coverageTask = subproject.tasks.findByName("jacocoTestReport") - if (coverageTask != null) { - dependsOn(coverageTask) - } - val verificationTask = subproject.tasks.findByName("jacocoTestCoverageVerification") - if (verificationTask != null) { - dependsOn(verificationTask) - } - } - } - } - } -} - -/** - * Creates aggregate test tasks for CI using createRootTask() above - * - * Creates three subtasks for the given base task name: - * - ${baseTaskName}Test - runs allTests - * - ${baseTaskName}LatestDepTest - runs allLatestDepTests - * - ${baseTaskName}Check - runs check - */ -fun Project.testAggregate( - baseTaskName: String, - includePrefixes: List, - excludePrefixes: List = emptyList(), - forceCoverage: Boolean = false -) { - createRootTask("${baseTaskName}Test", "allTests", includePrefixes, excludePrefixes, forceCoverage) - createRootTask("${baseTaskName}LatestDepTest", "allLatestDepTests", includePrefixes, excludePrefixes, forceCoverage) - createRootTask("${baseTaskName}Check", "check", includePrefixes, excludePrefixes, forceCoverage) -} - diff --git a/buildSrc/src/main/kotlin/datadog.ci-jobs.gradle.kts b/buildSrc/src/main/kotlin/datadog.ci-jobs.gradle.kts deleted file mode 100644 index 8d49d688cdb..00000000000 --- a/buildSrc/src/main/kotlin/datadog.ci-jobs.gradle.kts +++ /dev/null @@ -1,130 +0,0 @@ -/* - * This plugin defines a set of tasks to be used in CI. These aggregate tasks support partitioning (to parallelize - * jobs) with -PtaskPartitionCount and -PtaskPartition, and limiting tasks to those affected by git changes - * with -PgitBaseRef. - */ - -import datadog.gradle.plugin.ci.findAffectedTaskPath -import java.io.File -import kotlin.math.abs - -// Set up activePartition property on all projects -allprojects { - extra.set("activePartition", true) - - val taskPartitionCountProvider = rootProject.providers.gradleProperty("taskPartitionCount") - val taskPartitionProvider = rootProject.providers.gradleProperty("taskPartition") - if (taskPartitionCountProvider.isPresent && taskPartitionProvider.isPresent) { - val taskPartitionCount = taskPartitionCountProvider.get() - val taskPartition = taskPartitionProvider.get() - val currentTaskPartition = abs(project.path.hashCode() % taskPartitionCount.toInt()) - extra.set("activePartition", currentTaskPartition == taskPartition.toInt()) - } -} - -fun relativeToGitRoot(f: File): File { - return rootProject.projectDir.toPath().relativize(f.absoluteFile.toPath()).toFile() -} - -fun getChangedFiles(baseRef: String, newRef: String): List { - val stdout = StringBuilder() - val stderr = StringBuilder() - - val proc = Runtime.getRuntime().exec(arrayOf("git", "diff", "--name-only", "$baseRef..$newRef")) - proc.inputStream.bufferedReader().use { stdout.append(it.readText()) } - proc.errorStream.bufferedReader().use { stderr.append(it.readText()) } - proc.waitFor() - require(proc.exitValue() == 0) { "git diff command failed, stderr: $stderr" } - - val out = stdout.toString().trim() - if (out.isEmpty()) { - return emptyList() - } - - logger.debug("git diff output: $out") - return out.split("\n").map { File(rootProject.projectDir, it.trim()) } -} - -// Initialize git change tracking -rootProject.extra.set("useGitChanges", false) - -val gitBaseRefProvider = rootProject.providers.gradleProperty("gitBaseRef") -if (gitBaseRefProvider.isPresent) { - val baseRef = gitBaseRefProvider.get() - val newRef = rootProject.providers.gradleProperty("gitNewRef").orElse("HEAD").get() - - val changedFiles = getChangedFiles(baseRef, newRef) - rootProject.extra.set("changedFiles", changedFiles) - rootProject.extra.set("useGitChanges", true) - - val ignoredFiles = fileTree(rootProject.projectDir) { - include(".gitignore", ".editorconfig") - include("*.md", "**/*.md") - include("gradlew", "gradlew.bat", "mvnw", "mvnw.cmd") - include("NOTICE") - include("static-analysis.datadog.yml") - } - - changedFiles.forEach { f -> - if (ignoredFiles.contains(f)) { - logger.warn("Ignoring changed file: ${relativeToGitRoot(f)}") - } - } - - val filteredChangedFiles = changedFiles.filter { !ignoredFiles.contains(it) } - rootProject.extra.set("changedFiles", filteredChangedFiles) - - val globalEffectFiles = fileTree(rootProject.projectDir) { - include(".gitlab/**") - include("build.gradle") - include("gradle/**") - } - - for (f in filteredChangedFiles) { - if (globalEffectFiles.contains(f)) { - logger.warn("Global effect change: ${relativeToGitRoot(f)} (no tasks will be skipped)") - rootProject.extra.set("useGitChanges", false) - break - } - } - - if (rootProject.extra.get("useGitChanges") as Boolean) { - logger.warn("Git change tracking is enabled: $baseRef..$newRef") - - val projects = subprojects.sortedByDescending { it.projectDir.path.length } - val affectedProjects = mutableMapOf>() - - // Path prefixes mapped to affected task names. A file not matching any of these prefixes will affect all tasks in - // the project ("all" can be used a task name to explicitly state the same). Only the first matching prefix is used. - val matchers = listOf( - mapOf("prefix" to "src/testFixtures/", "task" to "testFixturesClasses"), - mapOf("prefix" to "src/test/", "task" to "testClasses"), - mapOf("prefix" to "src/jmh/", "task" to "jmhCompileGeneratedClasses") - ) - - for (f in filteredChangedFiles) { - val p = projects.find { f.toString().startsWith(it.projectDir.path + "/") } - if (p == null) { - logger.warn("Changed file: ${relativeToGitRoot(f)} at root project (no task will be skipped)") - rootProject.extra.set("useGitChanges", false) - break - } - - // Make sure path separator is / - val relPath = p.projectDir.toPath().relativize(f.toPath()).joinToString("/") - val task = matchers.find { relPath.startsWith(it["prefix"]!!) }?.get("task") ?: "all" - logger.warn("Changed file: ${relativeToGitRoot(f)} in project ${p.path} ($task)") - affectedProjects.computeIfAbsent(p) { mutableSetOf() }.add(task) - } - - rootProject.extra.set("affectedProjects", affectedProjects) - } -} - -tasks.register("runMuzzle") { - val muzzleSubprojects = subprojects.filter { p -> - val activePartition = p.extra.get("activePartition") as Boolean - activePartition && p.plugins.hasPlugin("java") && p.plugins.hasPlugin("muzzle") - } - dependsOn(muzzleSubprojects.map { p -> "${p.path}:muzzle" }) -} diff --git a/buildSrc/src/main/kotlin/datadog.dependency-locking.gradle.kts b/buildSrc/src/main/kotlin/datadog.dependency-locking.gradle.kts deleted file mode 100644 index 3435f9e5925..00000000000 --- a/buildSrc/src/main/kotlin/datadog.dependency-locking.gradle.kts +++ /dev/null @@ -1,34 +0,0 @@ -/* - * This plugin enables dependency locking. - * - * The goal is to be able to later rebuild any version, by pinning floating versions. - * It will also help IDEs not having to re-index any latest library release. - * Pinned versions will be updated by the CI on a weekly basis. - * - * Pinned version can be updated using: ./gradlew resolveAndLockAll --write-locks - * - * See https://docs.gradle.org/current/userguide/dependency_locking.html - */ - -project.dependencyLocking { - lockAllConfigurations() - //lockmode set to LENIENT because there are resolution - //errors in the build with an apiguardian dependency. - //See: https://docs.gradle.org/current/userguide/dependency_locking.html for more info - lockMode = LockMode.LENIENT -} - -tasks.register("resolveAndLockAll") { - notCompatibleWithConfigurationCache("Filters configurations at execution time") - doFirst { - require(gradle.startParameter.isWriteDependencyLocks) - } - doLast { - configurations.filter { - // Add any custom filtering on the configurations to be resolved: - // - Should be resolvable - // - Should skip Scala related task (https://github.com/ben-manes/gradle-versions-plugin/issues/816#issuecomment-1872264880) - it.isCanBeResolved && !it.name.startsWith("incrementalScalaAnalysis") - }.forEach { it.resolve() } - } -} diff --git a/buildSrc/src/main/kotlin/datadog.gradle-debug.gradle.kts b/buildSrc/src/main/kotlin/datadog.gradle-debug.gradle.kts deleted file mode 100644 index 38f9fac7411..00000000000 --- a/buildSrc/src/main/kotlin/datadog.gradle-debug.gradle.kts +++ /dev/null @@ -1,105 +0,0 @@ -/* - * Gradle debugging plugin for dd-trace-java builds. - */ - -val ddGradleDebugEnabled = project.hasProperty("ddGradleDebug") -val logPath = rootProject.layout.buildDirectory.file("datadog.gradle-debug.log") - -fun inferJdkFromJavaHome(javaHome: String?): String { - val effectiveJavaHome = javaHome ?: providers.environmentVariable("JAVA_HOME").orNull ?: error("JAVA_HOME is not set") - val javaExecutable = File(effectiveJavaHome, "bin/java").absolutePath - return try { - val process = ProcessBuilder(javaExecutable, "-version") - .redirectErrorStream(true) - .start() - val output = process.inputStream.bufferedReader().readText() - val versionLine = output.lines().firstOrNull() ?: "" - val versionMatch = Regex("version\\s+\"([0-9._]+)\"").find(versionLine) - versionMatch?.let { - val version = it.groupValues[1] - when { - version.startsWith("1.") -> version.substring(2, 3) - else -> version.split('.').first() - } - } ?: "unknown" - } catch (e: Exception) { - "error: ${e.message}" - } -} - -fun getJdkFromCompilerOptions(co: CompileOptions): String? { - if (co.isFork) { - val fo = co.forkOptions - val javaHome = fo.javaHome - if (javaHome != null) { - return inferJdkFromJavaHome(javaHome.toString()) - } - } - return null -} - -fun printJdkForProjectTasks(project: Project, logFile: File) { - project.tasks.forEach { task -> - val data = mutableMapOf() - data["task"] = task.path.toString() - if (task is JavaExec) { - val launcher = task.javaLauncher.get() - data["jdk"] = launcher.metadata.languageVersion.toString() - } else if (task is Javadoc) { - val tool = task.javadocTool.get() - data["jdk"] = tool.metadata.languageVersion.toString() - } else if (task is Test) { - val launcher = task.javaLauncher.get() - data["jdk"] = launcher.metadata.languageVersion.toString() - } else if (task is Exec) { - val java_home = task.environment.get("JAVA_HOME")?.toString() - data["jdk"] = inferJdkFromJavaHome(java_home) - } else if (task is JavaCompile) { - val compiler = task.javaCompiler.get() - data["jdk"] = compiler.metadata.languageVersion.toString() - val jdkFromJavaHome = getJdkFromCompilerOptions(task.options) - if (jdkFromJavaHome != null && jdkFromJavaHome != data["jdk"]) { - data["java_home"] = jdkFromJavaHome - } - } else if (task is GroovyCompile) { - val launcher = task.javaLauncher.get() - data["jdk"] = launcher.metadata.languageVersion.toString() - val jdkFromJavaHome = getJdkFromCompilerOptions(task.options) - if (jdkFromJavaHome != null && jdkFromJavaHome != data["jdk"]) { - data["java_home"] = jdkFromJavaHome - } - } else if (task is ScalaCompile) { - val launcher = task.javaLauncher.get() - data["jdk"] = launcher.metadata.languageVersion.toString() - val jdkFromJavaHome = getJdkFromCompilerOptions(task.options) - if (jdkFromJavaHome != null && jdkFromJavaHome != data["jdk"]) { - data["java_home"] = jdkFromJavaHome - } - } else { - data["jdk"] = "unknown" - } - val json = data.entries.joinToString(prefix = "{", postfix = "}") { (k, v) -> "\"$k\":\"$v\"" } - logFile.appendText("$json\n") - } -} - -class DebugBuildListener : org.gradle.BuildListener { - override fun settingsEvaluated(settings: Settings) = Unit - - override fun projectsLoaded(gradle: Gradle) = Unit - - override fun buildFinished(result: BuildResult) = Unit - - override fun projectsEvaluated(gradle: Gradle) { - val logFile = logPath.get().asFile - logFile.writeText("") - gradle.rootProject.allprojects.forEach { project -> - printJdkForProjectTasks(project, logFile) - } - } -} - -if (ddGradleDebugEnabled) { - logger.lifecycle("datadog.gradle-debug plugin is enabled") - gradle.addListener(DebugBuildListener()) -} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/CallSiteInstrumentationPlugin.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/CallSiteInstrumentationPlugin.kt deleted file mode 100644 index 40e1270dd54..00000000000 --- a/buildSrc/src/main/kotlin/datadog/gradle/plugin/CallSiteInstrumentationPlugin.kt +++ /dev/null @@ -1,245 +0,0 @@ -package datadog.gradle.plugin - -import org.gradle.api.GradleException -import org.gradle.api.JavaVersion -import org.gradle.api.Plugin -import org.gradle.api.Project -import org.gradle.api.file.DirectoryProperty -import org.gradle.api.file.ProjectLayout -import org.gradle.api.model.ObjectFactory -import org.gradle.api.plugins.JavaPluginExtension -import org.gradle.api.provider.ListProperty -import org.gradle.api.provider.Property -import org.gradle.api.tasks.JavaExec -import org.gradle.api.tasks.SourceSet -import org.gradle.api.tasks.SourceSetContainer -import org.gradle.api.tasks.compile.AbstractCompile -import org.gradle.api.tasks.testing.Test -import org.gradle.jvm.tasks.Jar -import org.gradle.jvm.toolchain.JavaLanguageVersion -import org.gradle.jvm.toolchain.JavaToolchainService -import java.io.File -import java.nio.file.Files -import java.nio.file.Paths -import javax.inject.Inject - -private const val CALL_SITE_INSTRUMENTER_MAIN_CLASS = "datadog.trace.plugin.csi.PluginApplication" -private const val CALL_SITE_CLASS_SUFFIX = "CallSite" -private const val CALL_SITE_CONSOLE_REPORTER = "CONSOLE" -private const val CALL_SITE_ERROR_CONSOLE_REPORTER = "ERROR_CONSOLE" - -/** - * This extension allows to configure the Call Site Instrumenter plugin execution. - */ -abstract class CallSiteInstrumentationExtension @Inject constructor(objectFactory: ObjectFactory, layout: ProjectLayout) { - /** - * The location of the source code to generate call site ({@code /src/main/java} by default). - */ - val srcFolder: DirectoryProperty = objectFactory.directoryProperty().convention( - layout.projectDirectory.dir("src").dir("main").dir("java") - ) - /** - * The location to generate call site source code ({@code /build/generated/sources/csi} by default). - */ - val targetFolder: DirectoryProperty = objectFactory.directoryProperty().convention( - layout.buildDirectory.dir("generated${File.separatorChar}sources${File.separatorChar}csi") - ) - /** - * The generated call site source file suffix (#CALL_SITE_CLASS_SUFFIX by default). - */ - val suffix: Property = objectFactory.property(String::class.java).convention(CALL_SITE_CLASS_SUFFIX) - /** - * The reporters to use after call site instrumenter run (only #CALL_SITE_CONSOLE_REPORTER and #CALL_SITE_ERROR_CONSOLE_REPORTER supported for now). - */ - val reporters: ListProperty = objectFactory.listProperty(String::class.java).convention(listOf( - CALL_SITE_ERROR_CONSOLE_REPORTER - )) - /** - * The location of the dd-trace-java project to look for the call site instrumenter (optional, current project root folder used if not set). - */ - abstract val rootFolder: Property - - /** - * The JVM to use to run the call site instrumenter (optional, default JVM used if not set). - */ - abstract val javaVersion: Property - - /** - * The JVM arguments to run the call site instrumenter. - */ - val jvmArgs: ListProperty = objectFactory.listProperty(String::class.java).convention(listOf("-Xmx128m", "-Xms64m")) -} - -abstract class CallSiteInstrumentationPlugin : Plugin{ - @get:Inject - abstract val javaToolchains: JavaToolchainService - - override fun apply(project: Project) { - // Create plugin extension - val extension = project.extensions.create("csi", CallSiteInstrumentationExtension::class.java) - project.afterEvaluate { - configureSourceSets(project, extension) - createTasks(project, extension) - } - } - - private fun configureSourceSets(project: Project, extension: CallSiteInstrumentationExtension) { - // create a new source set for the csi files - val targetFolder = newBuildFolder(project, extension.targetFolder.get().asFile.toString()) - val sourceSets = getSourceSets(project) - val mainSourceSet = sourceSets.named(SourceSet.MAIN_SOURCE_SET_NAME).get() - val csiSourceSet = sourceSets.create("csi") { - compileClasspath += mainSourceSet.output // mainly needed for the plugin tests - annotationProcessorPath += mainSourceSet.annotationProcessorPath - java.srcDir(targetFolder) - - } - project.configurations.named(csiSourceSet.compileClasspathConfigurationName) { - extendsFrom(project.configurations.named(mainSourceSet.compileClasspathConfigurationName).get()) - } - - project.tasks.named(csiSourceSet.getCompileTaskName("java"), AbstractCompile::class.java) { - sourceCompatibility = JavaVersion.VERSION_1_8.toString() - targetCompatibility = JavaVersion.VERSION_1_8.toString() - } - - // add csi classes to test classpath - sourceSets.named(SourceSet.TEST_SOURCE_SET_NAME) { - compileClasspath += csiSourceSet.output.classesDirs - runtimeClasspath += csiSourceSet.output.classesDirs - } - project.dependencies.add("testImplementation", csiSourceSet.output) - - // include classes in final JAR - project.tasks.named("jar", Jar::class.java) { - from(csiSourceSet.output.classesDirs) - } - } - - private fun newBuildFolder(project: Project, name: String): File { - val folder = project.layout.buildDirectory.dir(name).get().asFile - if (!folder.exists()) { - if (!folder.mkdirs()) { - throw GradleException("Cannot create folder $folder") - } - } - return folder - } - - private fun newTempFile(folder: File, name: String): File { - val file = File(folder, name) - if (!file.exists() && !file.createNewFile()) { - throw GradleException("Cannot create temporary file: $file") - } - file.deleteOnExit() - return file - } - - private fun getSourceSets(project: Project): SourceSetContainer { - return project.extensions.getByType(JavaPluginExtension::class.java).sourceSets - } - - private fun createTasks(project: Project, extension: CallSiteInstrumentationExtension) { - registerGenerateCallSiteTask(project, extension, "compileJava") - val targetFolder = extension.targetFolder.get().asFile - project.tasks.withType(AbstractCompile::class.java).matching { - task -> task.name.startsWith("compileTest") - }.configureEach { - inputs.dir(extension.targetFolder) - classpath += project.files(targetFolder) - } - project.tasks.withType(Test::class.java).configureEach { - inputs.dir(extension.targetFolder) - classpath += project.files(targetFolder) - } - } - - private fun configureLanguage(task: JavaExec, version: JavaLanguageVersion) { - task.javaLauncher.set(javaToolchains.launcherFor { - languageVersion.set(version) - }) - } - - private fun registerGenerateCallSiteTask(project: Project, - extension: CallSiteInstrumentationExtension, - compileTaskName: String) { - val taskName = compileTaskName.replace("compile", "generateCallSite") - val rootFolder = extension.rootFolder.getOrElse(project.rootDir) - val pluginJarFile = Paths.get( - rootFolder.toString(), - "buildSrc", - "call-site-instrumentation-plugin", - "build", - "libs", - "call-site-instrumentation-plugin-all.jar" - ).toFile() - val compileTask = project.tasks.named(compileTaskName, AbstractCompile::class.java) - val callSiteGeneratorTask = project.tasks.register(taskName, JavaExec::class.java) { - // Task description - group = "call site instrumentation" - description = "Generates call sites from ${compileTaskName}" - // Task input & output - val output = extension.targetFolder - val inputProvider = compileTask.map { it.destinationDirectory.get() } - inputs.dir(inputProvider) - outputs.dir(output) - // JavaExec configuration - if (extension.javaVersion.isPresent) { - configureLanguage(this, extension.javaVersion.get()) - } - jvmArgumentProviders.add({ extension.jvmArgs.get() }) - classpath(pluginJarFile) - mainClass.set(CALL_SITE_INSTRUMENTER_MAIN_CLASS) - // Write the call site instrumenter arguments into a temporary file - doFirst { - val programClassPath = getProgramClasspath(project).map { it.toString() } - val arguments = listOf( - extension.srcFolder.get().asFile.toString(), - inputProvider.get().asFile.toString(), - output.get().asFile.toString(), - extension.suffix.get(), - extension.reporters.get().joinToString(",") - ) + programClassPath - - val argumentFile = newTempFile(temporaryDir, "call-site-arguments") - Files.write(argumentFile.toPath(), arguments) - args(argumentFile.toString()) - } - - // make task depends on compile - dependsOn(compileTask) - } - - // make all sourcesets' class tasks depend on call site generator - val sourceSets = getSourceSets(project) - sourceSets.named(SourceSet.MAIN_SOURCE_SET_NAME) { - project.tasks.named(classesTaskName) { - dependsOn(callSiteGeneratorTask) - } - } - - // compile generated sources - sourceSets.named("csi") { - project.tasks.named(compileJavaTaskName) { - dependsOn(callSiteGeneratorTask) - } - } - } - - private fun getProgramClasspath(project: Project): List { - val classpath = ArrayList() - // 1. Compilation outputs - project.tasks.withType(AbstractCompile::class.java) - .map { it.destinationDirectory.asFile.get() } - .forEach(classpath::add) - // 2. Compile time dependencies - project.tasks.withType(AbstractCompile::class.java) - .flatMap { it.classpath } - .forEach(classpath::add) - // 3. Test time dependencies - project.tasks.withType(Test::class.java) - .flatMap { it.classpath } - .forEach(classpath::add) - return classpath - } -} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/ci/CIJobsExtensions.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/ci/CIJobsExtensions.kt new file mode 100644 index 00000000000..d6269176a0a --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/ci/CIJobsExtensions.kt @@ -0,0 +1,160 @@ +package datadog.gradle.plugin.ci + +import org.gradle.api.Project +import org.gradle.api.Task +import org.gradle.api.provider.Provider +import org.gradle.kotlin.dsl.extra +import kotlin.math.abs + +/** + * Determines if the current project is in the selected slot. + * + * The "slot" property should be provided in the format "X/Y", where X is the selected slot (1-based) + * and Y is the total number of slots. + * + * If the "slot" property is not provided, all projects are considered to be in the selected slot. + */ +val Project.isInSelectedSlot: Provider + get() = rootProject.providers.gradleProperty("slot").map { slot -> + val parts = slot.split("/") + if (parts.size != 2) { + project.logger.warn("Invalid slot format '{}', expected 'X/Y'. Treating all projects as selected.", slot) + return@map true + } + + // When CI_NODE_INDEX or CI_NODE_TOTAL is unset in non-parallel jobs, one part may be empty (e.g. slot="/1") — treat as no filtering + if (parts[0].isBlank() || parts[1].isBlank()) { + project.logger.info("Incomplete slot value '{}', CI_NODE_INDEX or CI_NODE_TOTAL not set. Treating all projects as selected.", slot) + return@map true + } + + val selectedSlot = parts[0].toIntOrNull() + val totalSlots = parts[1].toIntOrNull() + + if (selectedSlot == null || totalSlots == null || totalSlots <= 0) { + project.logger.warn("Invalid slot values '{}', expected numeric 'X/Y' with Y > 0. Treating all projects as selected.", slot) + return@map true + } + + // Distribution numbers when running on rootProject.allprojects indicates + // bucket sizes are reasonably balanced: + // + // * size 4 distribution: {2=146, 0=143, 1=157, 3=145} + // * size 6 distribution: {4=100, 0=92, 3=97, 2=97, 1=108, 5=97} + // * size 8 distribution: {2=62, 4=72, 0=71, 5=70, 7=78, 6=84, 1=87, 3=67} + // * size 10 distribution: {8=62, 0=65, 5=70, 9=59, 3=54, 1=56, 6=63, 4=47, 2=52, 7=63} + // * size 12 distribution: {10=55, 0=47, 4=45, 9=46, 8=51, 3=51, 2=46, 1=59, 5=52, 7=49, 11=45, 6=45} + val projectSlot = abs(project.path.hashCode() % totalSlots) + 1 // Convert to 1-based + + project.logger.info( + "Project {} assigned to slot {}/{}, active slot is {}", + project.path, + projectSlot, + totalSlots, + selectedSlot, + ) + + projectSlot == selectedSlot + }.orElse(true) + +/** + * Returns the task's path, given affected projects, if this task or its dependencies are affected by git changes. + */ +internal fun findAffectedTaskPath(baseTask: Task, affectedProjects: Map>): String? { + val visited = mutableSetOf() + val queue = mutableListOf(baseTask) + + while (queue.isNotEmpty()) { + val t = queue.removeAt(0) + if (visited.contains(t)) { + continue + } + visited.add(t) + + val affectedTasks = affectedProjects[t.project] + if (affectedTasks != null) { + if (affectedTasks.contains("all")) { + return "${t.project.path}:${t.name}" + } + if (affectedTasks.contains(t.name)) { + return "${t.project.path}:${t.name}" + } + } + + t.taskDependencies.getDependencies(t).forEach { queue.add(it) } + } + return null +} + +/** + * Creates a single aggregate root task that depends on matching subproject tasks + */ +private fun Project.createRootTask( + rootTaskName: String, + subProjTaskName: String, + includePrefixes: List, + excludePrefixes: List, + forceCoverage: Boolean +) { + val coverage = forceCoverage || rootProject.providers.gradleProperty("checkCoverage").isPresent + tasks.register(rootTaskName) { + subprojects.forEach { subproject -> + if ( + subproject.isInSelectedSlot.get() && + includePrefixes.any { subproject.path.startsWith(it) } && + !excludePrefixes.any { subproject.path.startsWith(it) } + ) { + val testTask = subproject.tasks.findByName(subProjTaskName) + var isAffected = true + + if (testTask != null) { + val useGitChanges = rootProject.extra.get("useGitChanges") as Boolean + if (useGitChanges) { + @Suppress("UNCHECKED_CAST") + val affectedProjects = rootProject.extra.get("affectedProjects") as Map> + val affectedTaskPath = findAffectedTaskPath(testTask, affectedProjects) + if (affectedTaskPath != null) { + logger.warn("Selecting ${subproject.path}:$subProjTaskName (affected by $affectedTaskPath)") + } else { + logger.warn("Skipping ${subproject.path}:$subProjTaskName (not affected by changed files)") + isAffected = false + } + } + if (isAffected) { + dependsOn(testTask) + } + } + + if (isAffected && coverage) { + val coverageTask = subproject.tasks.findByName("jacocoTestReport") + if (coverageTask != null) { + dependsOn(coverageTask) + } + val verificationTask = subproject.tasks.findByName("jacocoTestCoverageVerification") + if (verificationTask != null) { + dependsOn(verificationTask) + } + } + } + } + } +} + +/** + * Creates aggregate test tasks for CI using createRootTask() above + * + * Creates three subtasks for the given base task name: + * - ${baseTaskName}Test - runs allTests + * - ${baseTaskName}LatestDepTest - runs allLatestDepTests + * - ${baseTaskName}Check - runs check + */ +fun Project.testAggregate( + baseTaskName: String, + includePrefixes: List, + excludePrefixes: List = emptyList(), + forceCoverage: Boolean = false +) { + createRootTask("${baseTaskName}Test", "allTests", includePrefixes, excludePrefixes, forceCoverage) + createRootTask("${baseTaskName}LatestDepTest", "allLatestDepTests", includePrefixes, excludePrefixes, forceCoverage) + createRootTask("${baseTaskName}Check", "check", includePrefixes, excludePrefixes, forceCoverage) +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/ConfigCheckActions.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/ConfigCheckActions.kt new file mode 100644 index 00000000000..152cdf9832c --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/ConfigCheckActions.kt @@ -0,0 +1,145 @@ +package datadog.gradle.plugin.config + +import com.github.javaparser.StaticJavaParser +import com.github.javaparser.ast.Modifier +import com.github.javaparser.ast.body.FieldDeclaration +import com.github.javaparser.ast.body.VariableDeclarator +import com.github.javaparser.ast.expr.Expression +import com.github.javaparser.ast.expr.MethodCallExpr +import com.github.javaparser.ast.expr.NameExpr +import com.github.javaparser.ast.expr.StringLiteralExpr +import org.gradle.api.Action +import org.gradle.api.GradleException +import org.gradle.api.Task +import org.gradle.api.provider.Provider +import org.gradle.api.tasks.SourceSetOutput +import java.io.File + +/** Validates that all config definitions in the config directory are documented in supported-configurations.json. */ +internal class RegularConfigCheckAction( + private val mainSourceSetOutput: Provider>, + private val generatedClassName: Provider, + private val extension: SupportedTracerConfigurations +) : Action { + override fun execute(task: Task) { + val repoRoot = task.project.rootProject.projectDir.toPath() + val configDir = repoRoot.resolve("dd-trace-api/src/main/java/datadog/trace/api/config").toFile() + + if (!configDir.exists()) { + throw GradleException("Config directory not found: ${configDir.absolutePath}") + } + + val configFields = loadConfigFields(mainSourceSetOutput.get().get(), generatedClassName.get()) + val supported = configFields.supported + val aliasMapping = configFields.aliasMapping + + val violations = buildList { + configDir.listFiles()?.forEach { file -> + val fileName = file.name + extractStringConstants(file).forEach eachConstant@{ (fieldName, entry) -> + if (fieldName.endsWith("_DEFAULT")) return@eachConstant + val normalized = normalize(entry.value) + if (normalized !in supported && normalized !in aliasMapping) { + add("$fileName:${entry.line} -> Config '${entry.value}' normalizes to '$normalized' " + + "which is missing from '${extension.jsonFile.get()}'") + } + } + } + } + + if (violations.isNotEmpty()) { + task.logger.error("\nFound config definitions not in '${extension.jsonFile.get()}':") + violations.forEach { task.logger.lifecycle(it) } + throw GradleException("Undocumented Environment Variables found. Please add the above Environment Variables to '${extension.jsonFile.get()}'.") + } else { + task.logger.info("All config strings are present in '${extension.jsonFile.get()}'.") + } + } +} + +/** + * Validates that every `.ddprof.` config key used as a primary key in `DatadogProfilerConfig`'s + * static helpers also has its async-translated form (`profiling.ddprof.*` → `profiling.async.*`) + * documented in `supported-configurations.json`. + */ +internal class ProfilingConfigCheckAction( + private val mainSourceSetOutput: Provider>, + private val generatedClassName: Provider, + private val extension: SupportedTracerConfigurations +) : Action { + override fun execute(task: Task) { + val repoRoot = task.project.rootProject.projectDir.toPath() + + val constantMap = extractStringConstants( + repoRoot.resolve("dd-trace-api/src/main/java/datadog/trace/api/config/ProfilingConfig.java").toFile() + ) + + val configFields = loadConfigFields(mainSourceSetOutput.get().get(), generatedClassName.get()) + val supported = configFields.supported + val aliasMapping = configFields.aliasMapping + + val ddprofConfigFile = repoRoot.resolve( + "dd-java-agent/agent-profiling/profiling-ddprof/src/main/java/com/datadog/profiling/ddprof/DatadogProfilerConfig.java" + ).toFile() + val cu = StaticJavaParser.parse(ddprofConfigFile) + + val helperMethodNames = setOf("getBoolean", "getInteger", "getLong", "getString") + val violations = mutableListOf() + + cu.findAll(MethodCallExpr::class.java).forEach { call -> + if (call.scope.isPresent) return@forEach + if (call.nameAsString !in helperMethodNames) return@forEach + val args = call.arguments + if (args.size < 2 || args[0] !is NameExpr || (args[0] as NameExpr).nameAsString != "configProvider") return@forEach + + val primaryKeyEntry = resolveConstant(args[1], constantMap) ?: return@forEach + checkDocumented(primaryKeyEntry, supported, aliasMapping, call, violations, extension) + } + + if (violations.isNotEmpty()) { + violations.forEach { task.logger.error(it) } + throw GradleException("Undocumented configs found in DatadogProfilerConfig. Please add the above to '${extension.jsonFile.get()}'.") + } else { + task.logger.info("All DatadogProfilerConfig configs are documented.") + } + } +} + +internal data class ConstantEntry(val value: String, val line: Int) + +internal fun extractStringConstants(file: File): Map { + val map = mutableMapOf() + StaticJavaParser.parse(file).findAll(VariableDeclarator::class.java).forEach { varDecl -> + val field = varDecl.parentNode.map { it as? FieldDeclaration }.orElse(null) ?: return@forEach + if (field.hasModifiers(Modifier.Keyword.PUBLIC, Modifier.Keyword.STATIC, Modifier.Keyword.FINAL) + && varDecl.typeAsString == "String") { + val init = varDecl.initializer.orElse(null) as? StringLiteralExpr ?: return@forEach + val line = varDecl.range.map { it.begin.line }.orElse(-1) + map[varDecl.nameAsString] = ConstantEntry(init.value, line) + } + } + return map +} + +internal fun resolveConstant(expr: Expression?, constantMap: Map): ConstantEntry? = when (expr) { + is StringLiteralExpr -> ConstantEntry(expr.value, -1) + is NameExpr -> constantMap[expr.nameAsString] + else -> null +} + +// Only check the async-translated form produced by DatadogProfilerConfig.normalizeKey. +internal fun checkDocumented( + entry: ConstantEntry, + supported: Set, + aliasMapping: Map, + call: MethodCallExpr, + violations: MutableList, + extension: SupportedTracerConfigurations +) { + if (!entry.value.contains(".ddprof.")) return + val asyncNormalized = normalize(entry.value.replace(".ddprof.", ".async.")) + if (asyncNormalized !in supported && asyncNormalized !in aliasMapping) { + val callLine = call.range.map { it.begin.line }.orElse(-1) + violations.add("ProfilingConfig.java:${entry.line} (DatadogProfilerConfig.java:$callLine) -> '${entry.value}' (async form) → '$asyncNormalized' is missing from '${extension.jsonFile.get()}'") + } +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/ConfigInversionLinter.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/ConfigInversionLinter.kt index 890fea2c335..418f0e2b3e0 100644 --- a/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/ConfigInversionLinter.kt +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/ConfigInversionLinter.kt @@ -1,8 +1,12 @@ package datadog.gradle.plugin.config +import com.github.javaparser.ast.Modifier +import com.github.javaparser.ast.nodeTypes.NodeWithModifiers +import org.gradle.api.GradleException import org.gradle.api.Plugin import org.gradle.api.Project -import org.gradle.api.GradleException +import org.gradle.api.Task +import org.gradle.api.tasks.TaskProvider import org.gradle.api.tasks.SourceSet import org.gradle.api.tasks.SourceSetContainer import org.gradle.kotlin.dsl.getByType @@ -12,18 +16,64 @@ import java.nio.file.Path class ConfigInversionLinter : Plugin { override fun apply(target: Project) { val extension = target.extensions.create("supportedTracerConfigurations", SupportedTracerConfigurations::class.java) - registerLogEnvVarUsages(target, extension) - registerCheckEnvironmentVariablesUsage(target) + val logEnvVarUsages = registerLogEnvVarUsages(target, extension) + val checkEnvVarUsage = registerCheckEnvironmentVariablesUsage(target) + val checkConfigStrings = registerCheckConfigStringsTask(target, extension) + val checkInstrumenterModule = registerCheckInstrumenterModuleConfigurations(target, extension) + val checkDecoratorAnalytics = registerCheckDecoratorAnalyticsConfigurations(target, extension) + + target.tasks.register("checkConfigurations") { + group = "verification" + description = "Runs all config inversion validation checks" + dependsOn(logEnvVarUsages, checkEnvVarUsage, checkConfigStrings, checkInstrumenterModule, checkDecoratorAnalytics) + } + } +} + +// Data class for fields from generated class +data class LoadedConfigFields( + val supported: Set, + val aliasMapping: Map = emptyMap(), + val aliases: Map> = emptyMap() +) + +// Cache for fields from generated class +internal var cachedConfigFields: LoadedConfigFields? = null + +// Helper function to load fields from the generated class +internal fun loadConfigFields( + mainSourceSetOutput: org.gradle.api.file.FileCollection, + generatedClassName: String +): LoadedConfigFields { + return cachedConfigFields ?: run { + val urls = mainSourceSetOutput.files.map { it.toURI().toURL() }.toTypedArray() + URLClassLoader(urls, LoadedConfigFields::class.java.classLoader).use { cl -> + val clazz = Class.forName(generatedClassName, true, cl) + + val supportedField = clazz.getField("SUPPORTED").get(null) + @Suppress("UNCHECKED_CAST") + val supportedSet = when (supportedField) { + is Set<*> -> supportedField as Set + is Map<*, *> -> supportedField.keys as Set + else -> throw IllegalStateException("SUPPORTED field must be either Set or Map, but was ${supportedField?.javaClass}") + } + + @Suppress("UNCHECKED_CAST") + val aliasMappingMap = clazz.getField("ALIAS_MAPPING").get(null) as Map + @Suppress("UNCHECKED_CAST") + val aliasesMap = clazz.getField("ALIASES").get(null) as Map> + LoadedConfigFields(supportedSet, aliasMappingMap, aliasesMap) + }.also { cachedConfigFields = it } } } /** Registers `logEnvVarUsages` (scan for DD_/OTEL_ tokens and fail if unsupported). */ -private fun registerLogEnvVarUsages(target: Project, extension: SupportedTracerConfigurations) { +private fun registerLogEnvVarUsages(target: Project, extension: SupportedTracerConfigurations): TaskProvider { val ownerPath = extension.configOwnerPath val generatedFile = extension.className // token check that uses the generated class instead of JSON - target.tasks.register("logEnvVarUsages") { + return target.tasks.register("logEnvVarUsages") { group = "verification" description = "Scan Java files for DD_/OTEL_ tokens and fail if unsupported (using generated constants)" @@ -39,20 +89,17 @@ private fun registerLogEnvVarUsages(target: Project, extension: SupportedTracerC val javaFiles = target.fileTree(target.projectDir) { include("**/src/main/java/**/*.java") exclude("**/build/**", "**/dd-smoke-tests/**") + // Undertow uses DD_UNDERTOW_CONTINUATION as a legacy key to store an AgentScope. It is not related to an environment variable + exclude("dd-java-agent/instrumentation/undertow/undertow-common/src/main/java/datadog/trace/instrumentation/undertow/UndertowDecorator.java") } inputs.files(javaFiles) outputs.upToDateWhen { true } doLast { - // 1) Build classloader from the owner project’s runtime classpath - val urls = mainSourceSetOutput.get().get().files.map { it.toURI().toURL() }.toTypedArray() - val supported: Set = URLClassLoader(urls, javaClass.classLoader).use { cl -> - // 2) Load the generated class + read static field - val clazz = Class.forName(generatedFile.get(), true, cl) - @Suppress("UNCHECKED_CAST") - clazz.getField("SUPPORTED").get(null) as Set - } + // 1) Load configuration fields from the generated class + val configFields = loadConfigFields(mainSourceSetOutput.get().get(), generatedFile.get()) + val supported = configFields.supported - // 3) Scan our sources and compare + // 2) Scan our sources and compare val repoRoot = target.projectDir.toPath() val tokenRegex = Regex("\"(?:DD_|OTEL_)[A-Za-z0-9_]+\"") @@ -70,7 +117,7 @@ private fun registerLogEnvVarUsages(target: Project, extension: SupportedTracerC } tokenRegex.findAll(raw).forEach { m -> val token = m.value.trim('"') - if (token !in supported) add("$rel:${i + 1} -> Unsupported token'$token'") + if (token !in supported) add("$rel:${i + 1} -> Unsupported token '$token'") } } } @@ -87,8 +134,8 @@ private fun registerLogEnvVarUsages(target: Project, extension: SupportedTracerC } /** Registers `checkEnvironmentVariablesUsage` (forbid EnvironmentVariables.get(...)). */ -private fun registerCheckEnvironmentVariablesUsage(project: Project) { - project.tasks.register("checkEnvironmentVariablesUsage") { +private fun registerCheckEnvironmentVariablesUsage(project: Project): TaskProvider { + return project.tasks.register("checkEnvironmentVariablesUsage") { group = "verification" description = "Scans src/main/java for direct usages of EnvironmentVariables.get(...)" @@ -124,3 +171,84 @@ private fun registerCheckEnvironmentVariablesUsage(project: Project) { } } } + +// Helper functions for checking Config Strings +internal fun normalize(configValue: String) = + "DD_" + configValue.uppercase().replace("-", "_").replace(".", "_") + +// Checking "public" "static" "final" +internal fun NodeWithModifiers<*>.hasModifiers(vararg mods: Modifier.Keyword) = + mods.all { hasModifier(it) } + +/** Registers `checkConfigStrings` to validate config definitions against documented supported configurations. */ +private fun registerCheckConfigStringsTask(project: Project, extension: SupportedTracerConfigurations): TaskProvider { + val ownerPath = extension.configOwnerPath + val generatedFile = extension.className + + return project.tasks.register("checkConfigStrings") { + group = "verification" + description = "Validates that all config definitions in `dd-trace-api/src/main/java/datadog/trace/api/config` exist in `metadata/supported-configurations.json`" + + val mainSourceSetOutput = ownerPath.map { + project.project(it) + .extensions.getByType() + .named(SourceSet.MAIN_SOURCE_SET_NAME) + .map { main -> main.output } + } + inputs.files(mainSourceSetOutput) + + doLast("regular-config-check", RegularConfigCheckAction(mainSourceSetOutput, generatedFile, extension)) + doLast("profiling-config-check", ProfilingConfigCheckAction(mainSourceSetOutput, generatedFile, extension)) + } +} + + +/** Registers `checkInstrumenterModuleConfigurations` to verify each InstrumenterModule's integration name has proper entries in SUPPORTED and ALIASES. */ +private fun registerCheckInstrumenterModuleConfigurations(project: Project, extension: SupportedTracerConfigurations): TaskProvider { + val ownerPath = extension.configOwnerPath + val generatedFile = extension.className + + return project.tasks.register("checkInstrumenterModuleConfigurations", CheckInstrumenterModuleConfigTask::class.java) { + group = "verification" + description = "Validates that InstrumenterModule integration names have corresponding entries in SUPPORTED and ALIASES" + + mainSourceSetOutput.from(ownerPath.map { + project.project(it) + .extensions.getByType() + .named(SourceSet.MAIN_SOURCE_SET_NAME) + .map { main -> main.output } + }) + instrumentationFiles.from(project.fileTree(project.rootProject.projectDir) { + include("dd-java-agent/instrumentation/**/src/main/java/**/*.java") + }) + generatedClassName.set(generatedFile) + errorHeader.set("\nFound InstrumenterModule integration names with missing SUPPORTED/ALIASES entries:") + errorMessage.set("InstrumenterModule integration names are missing from SUPPORTED or ALIASES in '${extension.jsonFile.get()}'.") + successMessage.set("All InstrumenterModule integration names have proper SUPPORTED and ALIASES entries.") + } +} + +/** Registers `checkDecoratorAnalyticsConfigurations` to verify each BaseDecorator subclass's instrumentationNames have proper analytics entries in SUPPORTED and ALIASES. */ +private fun registerCheckDecoratorAnalyticsConfigurations(project: Project, extension: SupportedTracerConfigurations): TaskProvider { + val ownerPath = extension.configOwnerPath + val generatedFile = extension.className + + return project.tasks.register("checkDecoratorAnalyticsConfigurations", CheckDecoratorAnalyticsConfigTask::class.java) { + group = "verification" + description = "Validates that Decorator instrumentationNames have corresponding analytics entries in SUPPORTED and ALIASES" + + mainSourceSetOutput.from(ownerPath.map { + project.project(it) + .extensions.getByType() + .named(SourceSet.MAIN_SOURCE_SET_NAME) + .map { main -> main.output } + }) + instrumentationFiles.from(project.fileTree(project.rootProject.projectDir) { + include("dd-java-agent/instrumentation/**/src/main/java/**/*.java") + }) + generatedClassName.set(generatedFile) + errorHeader.set("\nFound Decorator instrumentationNames with missing analytics SUPPORTED/ALIASES entries:") + errorMessage.set("Decorator instrumentationNames are missing analytics entries from SUPPORTED or ALIASES in '${extension.jsonFile.get()}'.") + successMessage.set("All Decorator instrumentationNames have proper analytics SUPPORTED and ALIASES entries.") + } +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/InstrumentationConfigCheckTask.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/InstrumentationConfigCheckTask.kt new file mode 100644 index 00000000000..8e09c336011 --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/InstrumentationConfigCheckTask.kt @@ -0,0 +1,170 @@ +package datadog.gradle.plugin.config + +import com.github.javaparser.ParserConfiguration +import com.github.javaparser.StaticJavaParser +import com.github.javaparser.ast.CompilationUnit +import com.github.javaparser.ast.body.ClassOrInterfaceDeclaration +import com.github.javaparser.ast.body.MethodDeclaration +import com.github.javaparser.ast.expr.StringLiteralExpr +import com.github.javaparser.ast.stmt.ExplicitConstructorInvocationStmt +import com.github.javaparser.ast.stmt.ReturnStmt +import org.gradle.api.DefaultTask +import org.gradle.api.GradleException +import org.gradle.api.file.ConfigurableFileCollection +import org.gradle.api.provider.Property +import org.gradle.api.tasks.Input +import org.gradle.api.tasks.InputFiles +import org.gradle.api.tasks.TaskAction + +/** Abstract base for tasks that scan instrumentation source files against the generated config class. */ +abstract class InstrumentationConfigCheckTask : DefaultTask() { + @get:InputFiles + abstract val mainSourceSetOutput: ConfigurableFileCollection + + @get:InputFiles + abstract val instrumentationFiles: ConfigurableFileCollection + + @get:Input + abstract val generatedClassName: Property + + @get:Input + abstract val errorHeader: Property + + @get:Input + abstract val errorMessage: Property + + @get:Input + abstract val successMessage: Property + + @TaskAction + fun execute() { + val configFields = loadConfigFields(mainSourceSetOutput, generatedClassName.get()) + + val parserConfig = ParserConfiguration() + parserConfig.setLanguageLevel(ParserConfiguration.LanguageLevel.JAVA_8) + StaticJavaParser.setConfiguration(parserConfig) + + val repoRoot = project.rootProject.projectDir.toPath() + val violations = instrumentationFiles.files.flatMap { file -> + val rel = repoRoot.relativize(file.toPath()).toString() + val cu: CompilationUnit = try { + StaticJavaParser.parse(file) + } catch (_: Exception) { + return@flatMap emptyList() + } + collectPropertyViolations(configFields, rel, cu) + } + + if (violations.isNotEmpty()) { + logger.error(errorHeader.get()) + violations.forEach { logger.lifecycle(it) } + throw GradleException(errorMessage.get()) + } else { + logger.info(successMessage.get()) + } + } + + protected abstract fun collectPropertyViolations( + configFields: LoadedConfigFields, relativePath: String, cu: CompilationUnit + ): List + + /** Collects violations for [key] against [supported] and [aliases], checking that all [expectedAliases] are values of that alias entry. */ + protected fun collectMissingKeysAndAliases( + key: String, + expectedAliases: List, + supported: Set, + aliases: Map>, + location: String, + context: String + ): List = buildList { + if (key !in supported) { + add("$location -> $context: '$key' is missing from SUPPORTED") + } + if (key !in aliases) { + add("$location -> $context: '$key' is missing from ALIASES") + } else { + val aliasValues = aliases[key] ?: emptyList() + for (expected in expectedAliases) { + if (expected !in aliasValues) { + add("$location -> $context: '$expected' is missing from ALIASES['$key']") + } + } + } + } +} + +/** Checks that InstrumenterModule integration names have proper entries in SUPPORTED and ALIASES. */ +abstract class CheckInstrumenterModuleConfigTask : InstrumentationConfigCheckTask() { + override fun collectPropertyViolations( + configFields: LoadedConfigFields, relativePath: String, cu: CompilationUnit + ): List { + val violations = mutableListOf() + + cu.findAll(ClassOrInterfaceDeclaration::class.java).forEach classLoop@{ classDecl -> + val extendsModule = classDecl.extendedTypes.any { it.toString().startsWith("InstrumenterModule") } + if (!extendsModule) return@classLoop + + classDecl.findAll(ExplicitConstructorInvocationStmt::class.java) + .filter { !it.isThis } + .forEach { superCall -> + val names = superCall.arguments + .filterIsInstance() + .map { it.value } + val line = superCall.range.map { it.begin.line }.orElse(1) + + for (name in names) { + val normalized = name.uppercase().replace("-", "_").replace(".", "_") + val enabledKey = "DD_TRACE_${normalized}_ENABLED" + val context = "Integration '$name' (super arg)" + val location = "$relativePath:$line" + + violations.addAll(collectMissingKeysAndAliases( + enabledKey, + listOf("DD_TRACE_INTEGRATION_${normalized}_ENABLED", "DD_INTEGRATION_${normalized}_ENABLED"), + configFields.supported, configFields.aliases, location, context + )) + } + } + } + + return violations + } +} + +/** Checks that Decorator instrumentationNames have proper analytics entries in SUPPORTED and ALIASES. */ +abstract class CheckDecoratorAnalyticsConfigTask : InstrumentationConfigCheckTask() { + override fun collectPropertyViolations( + configFields: LoadedConfigFields, relativePath: String, cu: CompilationUnit + ): List { + val violations = mutableListOf() + + cu.findAll(MethodDeclaration::class.java) + .filter { it.nameAsString == "instrumentationNames" && it.parameters.isEmpty() } + .forEach { method -> + val names = method.findAll(ReturnStmt::class.java).flatMap { ret -> + ret.expression.map { it.findAll(StringLiteralExpr::class.java).map { s -> s.value } } + .orElse(emptyList()) + } + val line = method.range.map { it.begin.line }.orElse(1) + + for (name in names) { + val normalized = name.uppercase().replace("-", "_").replace(".", "_") + val context = "Decorator instrumentationName '$name'" + val location = "$relativePath:$line" + + violations.addAll(collectMissingKeysAndAliases( + "DD_TRACE_${normalized}_ANALYTICS_ENABLED", + listOf("DD_${normalized}_ANALYTICS_ENABLED"), + configFields.supported, configFields.aliases, location, context + )) + violations.addAll(collectMissingKeysAndAliases( + "DD_TRACE_${normalized}_ANALYTICS_SAMPLE_RATE", + listOf("DD_${normalized}_ANALYTICS_SAMPLE_RATE"), + configFields.supported, configFields.aliases, location, context + )) + } + } + + return violations + } +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/ParseSupportedConfigurationsTask.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/ParseSupportedConfigurationsTask.kt deleted file mode 100644 index 0864efbcbdc..00000000000 --- a/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/ParseSupportedConfigurationsTask.kt +++ /dev/null @@ -1,157 +0,0 @@ -package datadog.gradle.plugin.config - -import org.gradle.api.DefaultTask -import org.gradle.api.model.ObjectFactory -import org.gradle.api.tasks.Input -import org.gradle.api.tasks.InputFile -import org.gradle.api.tasks.OutputDirectory -import org.gradle.api.tasks.TaskAction -import com.fasterxml.jackson.core.type.TypeReference -import com.fasterxml.jackson.databind.ObjectMapper -import org.gradle.api.tasks.CacheableTask -import org.gradle.api.tasks.PathSensitive -import org.gradle.api.tasks.PathSensitivity -import java.io.File -import java.io.FileInputStream -import java.io.PrintWriter -import javax.inject.Inject - -@CacheableTask -abstract class ParseSupportedConfigurationsTask @Inject constructor( - private val objects: ObjectFactory -) : DefaultTask() { - @InputFile - @PathSensitive(PathSensitivity.NONE) - val jsonFile = objects.fileProperty() - - @get:OutputDirectory - val destinationDirectory = objects.directoryProperty() - - @Input - val className = objects.property(String::class.java) - - @TaskAction - fun generate() { - val input = jsonFile.get().asFile - val outputDir = destinationDirectory.get().asFile - val finalClassName = className.get() - outputDir.mkdirs() - - // Read JSON (directly from the file, not classpath) - val mapper = ObjectMapper() - val fileData: Map = FileInputStream(input).use { inStream -> - mapper.readValue(inStream, object : TypeReference>() {}) - } - - @Suppress("UNCHECKED_CAST") - val supported = fileData["supportedConfigurations"] as Map> - @Suppress("UNCHECKED_CAST") - val aliases = fileData["aliases"] as Map> - @Suppress("UNCHECKED_CAST") - val deprecated = (fileData["deprecations"] as? Map) ?: emptyMap() - - val aliasMapping = mutableMapOf() - for ((canonical, alist) in aliases) { - for (alias in alist) aliasMapping[alias] = canonical - } - - // Build the output .java path from the fully-qualified class name - val pkgName = finalClassName.substringBeforeLast('.', "") - val pkgPath = pkgName.replace('.', File.separatorChar) - val simpleName = finalClassName.substringAfterLast('.') - val pkgDir = if (pkgPath.isEmpty()) outputDir else File(outputDir, pkgPath).also { it.mkdirs() } - val generatedFile = File(pkgDir, "$simpleName.java").absolutePath - - // Call your existing generator (same signature as in your Java code) - generateJavaFile( - generatedFile, - simpleName, - pkgName, - supported.keys, - aliases, - aliasMapping, - deprecated - ) - } - - private fun generateJavaFile( - outputPath: String, - className: String, - packageName: String, - supportedKeys: Set, - aliases: Map>, - aliasMapping: Map, - deprecated: Map - ) { - val outFile = File(outputPath) - outFile.parentFile?.mkdirs() - - PrintWriter(outFile).use { out -> - // NOTE: adjust these if you want to match task's className - out.println("package $packageName;") - out.println() - out.println("import java.util.*;") - out.println() - out.println("public final class $className {") - out.println() - out.println(" public static final Set SUPPORTED;") - out.println() - out.println(" public static final Map> ALIASES;") - out.println() - out.println(" public static final Map ALIAS_MAPPING;") - out.println() - out.println(" public static final Map DEPRECATED;") - out.println() - out.println(" static {") - out.println() - - // SUPPORTED - out.print(" Set supportedSet = new HashSet<>(Arrays.asList(") - val supportedIter = supportedKeys.toSortedSet().iterator() - while (supportedIter.hasNext()) { - val key = supportedIter.next() - out.print("\"${esc(key)}\"") - if (supportedIter.hasNext()) out.print(", ") - } - out.println("));") - out.println(" SUPPORTED = Collections.unmodifiableSet(supportedSet);") - out.println() - - // ALIASES - out.println(" Map> aliasesMap = new HashMap<>();") - for ((canonical, list) in aliases.toSortedMap()) { - out.printf( - " aliasesMap.put(\"%s\", Collections.unmodifiableList(Arrays.asList(%s)));\n", - esc(canonical), - quoteList(list) - ) - } - out.println(" ALIASES = Collections.unmodifiableMap(aliasesMap);") - out.println() - - // ALIAS_MAPPING - out.println(" Map aliasMappingMap = new HashMap<>();") - for ((alias, target) in aliasMapping.toSortedMap()) { - out.printf(" aliasMappingMap.put(\"%s\", \"%s\");\n", esc(alias), esc(target)) - } - out.println(" ALIAS_MAPPING = Collections.unmodifiableMap(aliasMappingMap);") - out.println() - - // DEPRECATED - out.println(" Map deprecatedMap = new HashMap<>();") - for ((oldKey, note) in deprecated.toSortedMap()) { - out.printf(" deprecatedMap.put(\"%s\", \"%s\");\n", esc(oldKey), esc(note)) - } - out.println(" DEPRECATED = Collections.unmodifiableMap(deprecatedMap);") - out.println() - out.println(" }") - out.println("}") - } - } - - private fun quoteList(list: List): String = - list.joinToString(", ") { "\"${esc(it)}\"" } - - private fun esc(s: String): String = - s.replace("\\", "\\\\").replace("\"", "\\\"") -} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/ParseV2SupportedConfigurationsTask.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/ParseV2SupportedConfigurationsTask.kt new file mode 100644 index 00000000000..17fb4f07106 --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/ParseV2SupportedConfigurationsTask.kt @@ -0,0 +1,260 @@ +package datadog.gradle.plugin.config + +import org.gradle.api.DefaultTask +import org.gradle.api.model.ObjectFactory +import org.gradle.api.tasks.Input +import org.gradle.api.tasks.InputFile +import org.gradle.api.tasks.OutputDirectory +import org.gradle.api.tasks.TaskAction +import com.fasterxml.jackson.core.type.TypeReference +import com.fasterxml.jackson.databind.ObjectMapper +import org.gradle.api.tasks.CacheableTask +import org.gradle.api.tasks.PathSensitive +import org.gradle.api.tasks.PathSensitivity +import org.gradle.kotlin.dsl.property +import java.io.File +import java.io.FileInputStream +import java.io.PrintWriter +import javax.inject.Inject + +@CacheableTask +abstract class ParseV2SupportedConfigurationsTask @Inject constructor( + private val objects: ObjectFactory +) : DefaultTask() { + @InputFile + @PathSensitive(PathSensitivity.NONE) + val jsonFile = objects.fileProperty() + + @get:OutputDirectory + val destinationDirectory = objects.directoryProperty() + + @Input + val className = objects.property() + + @TaskAction + fun generate() { + val input = jsonFile.get().asFile + val outputDir = destinationDirectory.get().asFile + val finalClassName = className.get() + outputDir.mkdirs() + + // Read JSON (directly from the file, not classpath) + val mapper = ObjectMapper() + val fileData: Map = FileInputStream(input).use { inStream -> + mapper.readValue(inStream, object : TypeReference>() {}) + } + + // Fetch top-level keys of JSON file + @Suppress("UNCHECKED_CAST") + val supportedRaw = fileData["supportedConfigurations"] as Map>> + @Suppress("UNCHECKED_CAST") + val deprecated = (fileData["deprecations"] as? Map) ?: emptyMap() + + // Parse supportedConfigurations key to into a V2 format + val supported: Map> = supportedRaw.mapValues { (_, configList) -> + configList.map { configMap -> + @Suppress("UNCHECKED_CAST") + SupportedConfigurationItem( + configMap["version"] as? String, + configMap["type"] as? String, + configMap["default"] as? String, + (configMap["aliases"] as? List) ?: emptyList(), + (configMap["propertyKeys"] as? List) ?: emptyList() + ) + } + } + + // Generate top-level mapping from config -> list of aliases and reverse alias mapping from alias -> top-level config + // Note: This top-level alias mapping will be deprecated once Config Registry is mature enough to understand which version of a config a customer is using + val aliases: Map> = supported.mapValues { (_, configList) -> + configList.flatMap { it.aliases }.distinct() + } + + val aliasMapping = mutableMapOf() + for ((canonical, alist) in aliases) { + for (alias in alist) aliasMapping[alias] = canonical + } + + val reversePropertyKeysMap: Map = supported.flatMap { (canonical, configList) -> + configList.flatMap { config -> + config.propertyKeys.map { propertyKey -> propertyKey to canonical } + } + }.toMap() + + // Build the output .java path from the fully-qualified class name + val pkgName = finalClassName.substringBeforeLast('.', "") + val pkgPath = pkgName.replace('.', File.separatorChar) + val simpleName = finalClassName.substringAfterLast('.') + val pkgDir = if (pkgPath.isEmpty()) outputDir else File(outputDir, pkgPath).also { it.mkdirs() } + val generatedFile = File(pkgDir, "$simpleName.java").absolutePath + + // Call your existing generator (same signature as in your Java code) + generateJavaFile( + generatedFile, + simpleName, + pkgName, + supported, + aliases, + aliasMapping, + deprecated, + reversePropertyKeysMap + ) + } + + private fun generateJavaFile( + outputPath: String, + className: String, + packageName: String, + supported: Map>, + aliases: Map>, + aliasMapping: Map, + deprecated: Map, + reversePropertyKeysMap: Map + ) { + val outFile = File(outputPath) + outFile.parentFile?.mkdirs() + + PrintWriter(outFile).use { out -> + out.println("package $packageName;") + out.println() + out.println("import java.util.*;") + out.println() + out.println("public final class $className {") + out.println() + out.println(" public static final Map> SUPPORTED;") + out.println() + out.println(" public static final Map> ALIASES;") + out.println() + out.println(" public static final Map ALIAS_MAPPING;") + out.println() + out.println(" public static final Map DEPRECATED;") + out.println() + out.println(" public static final Map REVERSE_PROPERTY_KEYS_MAP;") + out.println() + out.println(" static {") + out.println(" SUPPORTED = initSupported();") + out.println(" ALIASES = initAliases();") + out.println(" ALIAS_MAPPING = initAliasMapping();") + out.println(" DEPRECATED = initDeprecated();") + out.println(" REVERSE_PROPERTY_KEYS_MAP = initReversePropertyKeysMap();") + out.println(" }") + out.println() + + // initSupported() - split into two helper functions to avoid "code too large" error + out.println(" private static Map> initSupported() {") + out.println(" Map> supportedMap = new HashMap<>();") + out.println(" initSupported1(supportedMap);") + out.println(" initSupported2(supportedMap);") + out.println(" return Collections.unmodifiableMap(supportedMap);") + out.println(" }") + out.println() + + val sortedSupported = supported.toSortedMap().entries.toList() + val midpoint = sortedSupported.size / 2 + + // initSupported1() - first half + out.println(" private static void initSupported1(Map> supportedMap) {") + for ((key, configList) in sortedSupported.take(midpoint)) { + out.print(" supportedMap.put(\"${esc(key)}\", Collections.unmodifiableList(Arrays.asList(") + val configIter = configList.iterator() + while (configIter.hasNext()) { + val config = configIter.next() + out.print("new SupportedConfiguration(") + out.print("${escNullableString(config.version)}, ") + out.print("${escNullableString(config.type)}, ") + out.print("${escNullableString(config.default)}, ") + out.print("Arrays.asList(${quoteList(config.aliases)}), ") + out.print("Arrays.asList(${quoteList(config.propertyKeys)})") + out.print(")") + if (configIter.hasNext()) out.print(", ") + } + out.println(")));") + } + out.println(" }") + out.println() + + // initSupported2() - second half + out.println(" private static void initSupported2(Map> supportedMap) {") + for ((key, configList) in sortedSupported.drop(midpoint)) { + out.print(" supportedMap.put(\"${esc(key)}\", Collections.unmodifiableList(Arrays.asList(") + val configIter = configList.iterator() + while (configIter.hasNext()) { + val config = configIter.next() + out.print("new SupportedConfiguration(") + out.print("${escNullableString(config.version)}, ") + out.print("${escNullableString(config.type)}, ") + out.print("${escNullableString(config.default)}, ") + out.print("Arrays.asList(${quoteList(config.aliases)}), ") + out.print("Arrays.asList(${quoteList(config.propertyKeys)})") + out.print(")") + if (configIter.hasNext()) out.print(", ") + } + out.println(")));") + } + out.println(" }") + out.println() + + // initAliases() + out.println(" // Note: This top-level alias mapping will be deprecated once Config Registry is mature enough to understand which version of a config a customer is using") + out.println(" private static Map> initAliases() {") + out.println(" Map> aliasesMap = new HashMap<>();") + for ((canonical, list) in aliases.toSortedMap()) { + out.printf( + " aliasesMap.put(\"%s\", Collections.unmodifiableList(Arrays.asList(%s)));\n", + esc(canonical), + quoteList(list) + ) + } + out.println(" return Collections.unmodifiableMap(aliasesMap);") + out.println(" }") + out.println() + + // initAliasMapping() + out.println(" private static Map initAliasMapping() {") + out.println(" Map aliasMappingMap = new HashMap<>();") + for ((alias, target) in aliasMapping.toSortedMap()) { + out.printf(" aliasMappingMap.put(\"%s\", \"%s\");\n", esc(alias), esc(target)) + } + out.println(" return Collections.unmodifiableMap(aliasMappingMap);") + out.println(" }") + out.println() + + // initDeprecated() + out.println(" private static Map initDeprecated() {") + out.println(" Map deprecatedMap = new HashMap<>();") + for ((oldKey, note) in deprecated.toSortedMap()) { + out.printf(" deprecatedMap.put(\"%s\", \"%s\");\n", esc(oldKey), esc(note)) + } + out.println(" return Collections.unmodifiableMap(deprecatedMap);") + out.println(" }") + out.println() + + // initReversePropertyKeysMap() + out.println(" private static Map initReversePropertyKeysMap() {") + out.println(" Map reversePropertyKeysMapping = new HashMap<>();") + for ((propertyKey, config) in reversePropertyKeysMap.toSortedMap()) { + out.printf(" reversePropertyKeysMapping.put(\"%s\", \"%s\");\n", esc(propertyKey), esc(config)) + } + out.println(" return Collections.unmodifiableMap(reversePropertyKeysMapping);") + out.println(" }") + out.println("}") + } + } + + private fun quoteList(list: List): String = + list.joinToString(", ") { "\"${esc(it)}\"" } + + private fun esc(s: String): String = + s.replace("\\", "\\\\").replace("\"", "\\\"") + + private fun escNullableString(s: String?): String = + if (s == null) "null" else "\"${esc(s)}\"" +} + +private data class SupportedConfigurationItem( + val version: String?, + val type: String?, + val default: String?, + val aliases: List, + val propertyKeys: List +) diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/SupportedConfigPlugin.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/SupportedConfigPlugin.kt index dd23dbbcd05..610aec60ebb 100644 --- a/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/SupportedConfigPlugin.kt +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/config/SupportedConfigPlugin.kt @@ -13,14 +13,14 @@ class SupportedConfigPlugin : Plugin { private fun generateSupportedConfigurations(targetProject: Project, extension: SupportedTracerConfigurations) { val generateTask = - targetProject.tasks.register("generateSupportedConfigurations", ParseSupportedConfigurationsTask::class.java) { + targetProject.tasks.register("generateSupportedConfigurations", ParseV2SupportedConfigurationsTask::class.java) { jsonFile.set(extension.jsonFile) destinationDirectory.set(extension.destinationDirectory) className.set(extension.className) } - val sourceset = targetProject.extensions.getByType(SourceSetContainer::class.java).named(SourceSet.MAIN_SOURCE_SET_NAME) - sourceset.configure { + val sourceSet = targetProject.extensions.getByType(SourceSetContainer::class.java).named(SourceSet.MAIN_SOURCE_SET_NAME) + sourceSet.configure { java.srcDir(generateTask) } diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/csi/CallSiteInstrumentationExtension.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/csi/CallSiteInstrumentationExtension.kt new file mode 100644 index 00000000000..324bc344d6b --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/csi/CallSiteInstrumentationExtension.kt @@ -0,0 +1,83 @@ +package datadog.gradle.plugin.csi + +import org.gradle.api.file.ConfigurableFileCollection +import org.gradle.api.file.DirectoryProperty +import org.gradle.api.file.ProjectLayout +import org.gradle.api.model.ObjectFactory +import org.gradle.api.provider.ListProperty +import org.gradle.api.provider.Property +import org.gradle.jvm.toolchain.JavaLanguageVersion +import org.gradle.kotlin.dsl.listProperty +import org.gradle.kotlin.dsl.property +import java.io.File +import javax.inject.Inject + +/** + * This extension allows to configure the Call Site Instrumenter plugin execution. + */ +abstract class CallSiteInstrumentationExtension @Inject constructor( + objectFactory: ObjectFactory, + layout: ProjectLayout +) { + companion object { + const val CALL_SITE_CLASS_SUFFIX = "CallSite" + const val CALL_SITE_CONSOLE_REPORTER = "CONSOLE" + const val CALL_SITE_ERROR_CONSOLE_REPORTER = "ERROR_CONSOLE" + } + + /** + * The location of the source code to generate call site ({@code /src/main/java} by default). + */ + val srcFolder: DirectoryProperty = objectFactory.directoryProperty().convention( + layout.projectDirectory.dir("src").dir("main").dir("java") + ) + + /** + * The location to generate call site source code ({@code /build/generated/sources/csi} by default). + */ + val targetFolder: DirectoryProperty = objectFactory.directoryProperty().convention( + layout.buildDirectory.dir("generated/sources/$CSI_SOURCE_SET") + ) + + /** + * The generated call site source file suffix (#CALL_SITE_CLASS_SUFFIX by default). + */ + val suffix: Property = objectFactory.property().convention(CALL_SITE_CLASS_SUFFIX) + + /** + * The reporters to use after call site instrumenter run (only #CALL_SITE_CONSOLE_REPORTER and #CALL_SITE_ERROR_CONSOLE_REPORTER supported for now). + */ + val reporters: ListProperty = objectFactory.listProperty().convention( + listOf( + CALL_SITE_CONSOLE_REPORTER, + CALL_SITE_ERROR_CONSOLE_REPORTER + ) + ) + + /** + * The location of the dd-trace-java project to look for the call site instrumenter (optional, current project root folder used if not set). + */ + abstract val rootFolder: Property + + /** + * The JVM to use to run the call site instrumenter (optional, default JVM used if not set). + */ + val javaVersion: Property = + objectFactory.property().convention(JavaLanguageVersion.current()) + + /** + * The JVM arguments to run the call site instrumenter. + */ + val jvmArgs: ListProperty = + objectFactory.listProperty().convention(listOf("-Xmx128m", "-Xms64m")) + + /** + * The paths used to look for the call site instrumenter dependencies. + * + * The plugin includes by default **only** the `main` and `test` source sets, and their + * related compilation classpath. As we don't want other test configurations by default. + * + * However, it's possible to contribute additional paths to look for dependencies. + */ + val additionalPaths: ConfigurableFileCollection = objectFactory.fileCollection() +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/csi/CallSiteInstrumentationPlugin.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/csi/CallSiteInstrumentationPlugin.kt new file mode 100644 index 00000000000..9fff60f498f --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/csi/CallSiteInstrumentationPlugin.kt @@ -0,0 +1,237 @@ +package datadog.gradle.plugin.csi + +import org.gradle.api.GradleException +import org.gradle.api.JavaVersion +import org.gradle.api.NamedDomainObjectSet +import org.gradle.api.Plugin +import org.gradle.api.Project +import org.gradle.api.artifacts.Configuration +import org.gradle.api.plugins.JavaPlugin +import org.gradle.api.plugins.JavaPlugin.COMPILE_CLASSPATH_CONFIGURATION_NAME +import org.gradle.api.plugins.JavaPlugin.RUNTIME_CLASSPATH_CONFIGURATION_NAME +import org.gradle.api.plugins.JavaPluginExtension +import org.gradle.api.plugins.jvm.JvmTestSuite +import org.gradle.api.tasks.JavaExec +import org.gradle.api.tasks.SourceSet.MAIN_SOURCE_SET_NAME +import org.gradle.api.tasks.SourceSet.TEST_SOURCE_SET_NAME +import org.gradle.api.tasks.SourceSetContainer +import org.gradle.api.tasks.TaskProvider +import org.gradle.api.tasks.compile.AbstractCompile +import org.gradle.internal.configuration.problems.projectPathFrom +import org.gradle.jvm.tasks.Jar +import org.gradle.jvm.toolchain.JavaToolchainService +import org.gradle.kotlin.dsl.apply +import org.gradle.kotlin.dsl.create +import org.gradle.kotlin.dsl.getByType +import org.gradle.kotlin.dsl.named +import org.gradle.kotlin.dsl.register +import org.gradle.kotlin.dsl.withType +import org.gradle.testing.base.TestingExtension +import java.io.File +import java.nio.file.Files +import java.nio.file.Paths +import java.util.Locale +import javax.inject.Inject + +private const val CALL_SITE_INSTRUMENTER_MAIN_CLASS = "datadog.trace.plugin.csi.PluginApplication" +const val CSI = "csi" +const val CSI_SOURCE_SET = CSI + +abstract class CallSiteInstrumentationPlugin : Plugin { + @get:Inject + abstract val javaToolchains: JavaToolchainService + + override fun apply(project: Project) { + project.pluginManager.apply(JavaPlugin::class) + + // Create plugin extension + val csiExtension = project.extensions.create(CSI) + configureSourceSets(project, csiExtension) + registerGenerateCallSiteTask(project, csiExtension, project.tasks.named("compileJava")) + configureTestConfigurations(project, csiExtension) + } + + private fun configureSourceSets(project: Project, extension: CallSiteInstrumentationExtension) { + // create a new source set for the csi files + val sourceSets = project.sourceSets + val mainSourceSet = sourceSets.named(MAIN_SOURCE_SET_NAME).get() + val csiSourceSet = sourceSets.create(CSI_SOURCE_SET) { + compileClasspath += mainSourceSet.output // mainly needed for the plugin tests + annotationProcessorPath += mainSourceSet.annotationProcessorPath + java.srcDir(extension.targetFolder) + } + + project.configurations.named(csiSourceSet.compileClasspathConfigurationName) { + extendsFrom(project.configurations.named(mainSourceSet.compileClasspathConfigurationName).get()) + } + + project.tasks.named(csiSourceSet.getCompileTaskName("java")) { + sourceCompatibility = JavaVersion.VERSION_1_8.toString() + targetCompatibility = JavaVersion.VERSION_1_8.toString() + } + + // add csi classes to test classpath + sourceSets.named(TEST_SOURCE_SET_NAME) { + compileClasspath += csiSourceSet.output.classesDirs + runtimeClasspath += csiSourceSet.output.classesDirs + } + project.dependencies.add("testImplementation", csiSourceSet.output) + + // include classes in final JAR + project.tasks.named("jar") { + from(csiSourceSet.output.classesDirs) + } + } + + private fun newTempFile(folder: File, name: String): File { + val file = File(folder, name) + if (!file.exists() && !file.createNewFile()) { + throw GradleException("Cannot create temporary file: $file") + } + file.deleteOnExit() + return file + } + + private fun configureTestConfigurations(project: Project, csiExtension: CallSiteInstrumentationExtension) { + project.pluginManager.withPlugin("jvm-test-suite") { + project.extensions.getByType().suites.withType().configureEach { + project.logger.info("Configuring jvm test suite '{}' to use csiExtension.targetFolder", name) + dependencies { + compileOnly.add(project.files(csiExtension.targetFolder)) + runtimeOnly.add(project.files(csiExtension.targetFolder)) + } + } + } + } + + private fun registerGenerateCallSiteTask( + project: Project, + csiExtension: CallSiteInstrumentationExtension, + mainCompileTask: TaskProvider + ) { + val genTaskName = mainCompileTask.name.replace("compile", "generateCallSite") + val pluginJarFile = Paths.get( + csiExtension.rootFolder.getOrElse(project.rootDir).toString(), + "buildSrc", + "call-site-instrumentation-plugin", + "build", + "libs", + "call-site-instrumentation-plugin-all.jar" + ) + + val callSiteGeneratorTask = project.tasks.register(genTaskName) { + // Task description + group = "call site instrumentation" + description = "Generates call sites from ${mainCompileTask.name}" + + // Remote Debug + if (project.providers.gradleProperty("debugCsiJar").isPresent) { + jvmArgs("-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=localhost:5005") + } + + // Task input & output + val output = csiExtension.targetFolder + val inputProvider = mainCompileTask.flatMap { it.destinationDirectory } + inputs.dir(inputProvider) + inputs.dir(csiExtension.srcFolder) + inputs.dir(csiExtension.rootFolder).optional() + inputs.file(pluginJarFile) + inputs.property("csi.suffix", csiExtension.suffix) + inputs.property("csi.javaVersion", csiExtension.javaVersion) + inputs.property("csi.jvmArgs", csiExtension.jvmArgs) + inputs.property("csi.reporters", csiExtension.reporters) + outputs.dir(output) + + // JavaExec configuration + javaLauncher.set( + javaToolchains.launcherFor { + languageVersion.set(csiExtension.javaVersion) + } + ) + + jvmArgumentProviders.add({ csiExtension.jvmArgs.get() }) + classpath(pluginJarFile) + mainClass.set(CALL_SITE_INSTRUMENTER_MAIN_CLASS) + + // Write the call site instrumenter arguments into a temporary file + doFirst { + val callsitesClassPath = project.files( + project.sourceSets.named(MAIN_SOURCE_SET_NAME).map { it.output }, + project.defaultConfigurations, + csiExtension.additionalPaths, + ) + + if (logger.isInfoEnabled) { + logger.info( + "Aggregated CSI classpath:\n{}", + callsitesClassPath.toSet().sorted().joinToString("\n") { it.toString() } + ) + } + + val argFile = buildList { + add(csiExtension.srcFolder.get().asFile.toString()) + add(inputProvider.get().asFile.toString()) + add(output.get().asFile.toString()) + add(csiExtension.suffix.get()) + add(csiExtension.reporters.get().joinToString(",")) + + // module program classpath + addAll(callsitesClassPath.map { it.toString() }) + } + + val argumentFile = newTempFile(temporaryDir, "call-site-arguments") + Files.write(argumentFile.toPath(), argFile) + args(argumentFile.toString()) + } + + // make task depends on compile + dependsOn(mainCompileTask) + } + + // make all sourceSets class tasks depend on call site generator + val sourceSets = project.sourceSets + sourceSets.named(MAIN_SOURCE_SET_NAME) { + project.tasks.named(classesTaskName) { + dependsOn(callSiteGeneratorTask) + } + } + + // compile generated sources + sourceSets.named(CSI_SOURCE_SET) { + project.tasks.named(compileJavaTaskName) { + dependsOn(callSiteGeneratorTask) + } + } + } + + private val Project.defaultConfigurations: NamedDomainObjectSet + get() = project.configurations.matching { + // Includes all main* source sets, but only the test (as wee don;t want other ) + // * For main => runtimeClasspath, compileClasspath + // * For test => testRuntimeClasspath, testCompileClasspath + // * For other main* => "main_javaXXRuntimeClasspath", "main_javaXXCompileClasspath" + + when (it.name) { + // Regular main and test source sets + RUNTIME_CLASSPATH_CONFIGURATION_NAME, + COMPILE_CLASSPATH_CONFIGURATION_NAME, + TEST_SOURCE_SET_NAME + RUNTIME_CLASSPATH_CONFIGURATION_NAME.capitalize(), + TEST_SOURCE_SET_NAME + COMPILE_CLASSPATH_CONFIGURATION_NAME.capitalize() -> true + + else -> false + } + } + + private fun String.capitalize(): String = replaceFirstChar { + if (it.isLowerCase()) { + it.titlecase( + Locale.getDefault() + ) + } else { + it.toString() + } + } + + private val Project.sourceSets: SourceSetContainer + get() = project.extensions.getByType().sourceSets +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/dump/DumpHangedTestPlugin.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/dump/DumpHangedTestPlugin.kt index a8297d4dfe6..19db11b8dd8 100644 --- a/buildSrc/src/main/kotlin/datadog/gradle/plugin/dump/DumpHangedTestPlugin.kt +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/dump/DumpHangedTestPlugin.kt @@ -20,10 +20,12 @@ import java.util.concurrent.ScheduledExecutorService import java.util.concurrent.ScheduledFuture import java.util.concurrent.TimeUnit import javax.inject.Inject +import kotlin.jvm.optionals.getOrElse /** * Plugin to collect thread and heap dumps for hanged tests. */ +@Suppress("unused") class DumpHangedTestPlugin : Plugin { companion object { private const val DUMP_FUTURE_KEY = "dumping_future" @@ -119,36 +121,27 @@ class DumpHangedTestPlugin : Plugin { dumpsDir.mkdirs() - fun file(name: String, ext: String = "log") = - File(dumpsDir, "$name-${System.currentTimeMillis()}.$ext") + ProcessHandle.current().children() + .filter { it.info().commandLine().getOrElse { "" }.contains("Gradle Test Executor") } + .forEach { process -> + collectDump(dumpsDir, process) - // For simplicity, use `0` as the PID, which collects all thread dumps across JVMs. - val allThreadsFile = file("all-thread-dumps") - runCmd(Redirect.to(allThreadsFile), "jcmd", "0", "Thread.print", "-l") - - // Collect all JVMs pids. - val allJavaProcessesFile = file("all-java-processes") - runCmd(Redirect.to(allJavaProcessesFile), "jcmd", "-l") - - // Collect pids for 'Gradle Test Executor'. - val pids = allJavaProcessesFile.readLines() - .filter { it.contains("Gradle Test Executor") } - .map { it.substringBefore(' ') } - - pids.forEach { pid -> - // Collect heap dump by pid. - val heapDumpPath = file("${pid}-heap-dump", "hprof").absolutePath - runCmd(Redirect.INHERIT, "jcmd", pid, "GC.heap_dump", heapDumpPath) + process.children().forEach { child -> + collectDump(dumpsDir, child) + } + } - // Collect thread dump by pid. - val threadDumpFile = file("${pid}-thread-dump") - runCmd(Redirect.to(threadDumpFile), "jcmd", pid, "Thread.print", "-l") - } + // Just in case collect all thread dumps by using special PID `0`. + val allThreadsFile = file(dumpsDir, "all-thread-dumps") + runCmd(Redirect.to(allThreadsFile), "jcmd", "0", "Thread.print", "-l") } catch (e: Throwable) { - t.logger.warn("Taking dumps failed with error: ${e.message}, for ${t.path}") + t.logger.warn("Taking dumps failed with error: ${e.message ?: e.javaClass.name}, for ${t.path}") } } + private fun file(baseDir: File, name: String, ext: String = "log") = + File(baseDir, "$name-${System.currentTimeMillis()}.$ext") + private fun cleanup(t: Task) { val future = t.extra .takeIf { it.has(DUMP_FUTURE_KEY) } @@ -174,4 +167,25 @@ class DumpHangedTestPlugin : Plugin { throw IOException("Process failed: ${args.joinToString(" ")}, exit code: $exitCode") } } + + private fun collectDump( + baseDir: File, + process: ProcessHandle + ) { + val pid = process.pid().toString() + + if (process.info().command().getOrElse { "" }.contains("/ibm8")) { + // On IBM JDK thread dump can be collected by signaling process with `kill -3`. + // It will be writen into `/tmp/javacore.YYYYMMDD.HHMMSS.PID.SEQ.txt + runCmd(Redirect.INHERIT, "kill", "-3", pid) + } else { + // Collect heap dump by pid. + val heapDumpPath = file(baseDir, "$pid-heap-dump", "hprof").absolutePath + runCmd(Redirect.INHERIT, "jcmd", pid, "GC.heap_dump", heapDumpPath) + + // Collect thread dump by pid. + val threadDumpFile = file(baseDir, "$pid-thread-dump", "log") + runCmd(Redirect.to(threadDumpFile), "jcmd", pid, "Thread.print", "-l") + } + } } diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/instrument/BuildTimeInstrumentationExtension.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/instrument/BuildTimeInstrumentationExtension.kt new file mode 100644 index 00000000000..ec07ba4e2a0 --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/instrument/BuildTimeInstrumentationExtension.kt @@ -0,0 +1,32 @@ +package datadog.gradle.plugin.instrument + +import org.gradle.api.file.ConfigurableFileCollection +import org.gradle.api.file.DirectoryProperty +import org.gradle.api.provider.ListProperty + +/** + * Extension for configuring the build-time instrumentation plugin. + * + * Exposed as `buildTimeInstrumentation { ... }`. + * + * @see BuildTimeInstrumentationPlugin + */ +abstract class BuildTimeInstrumentationExtension { + /** + * Fully qualified ByteBuddy plugin class names to apply during post-compilation instrumentation. + * + * Each plugin class must implement [net.bytebuddy.build.Plugin] and provide a constructor + * accepting a [java.io.File] target directory. + */ + abstract val plugins: ListProperty + + /** + * Additional classpath entries required to resolve instrumentation plugins and their dependencies. + */ + abstract val additionalClasspath: ListProperty + + /** + * Additional class directories to include in instrumentation processing. + */ + abstract val includeClassDirectories: ConfigurableFileCollection +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/instrument/BuildTimeInstrumentationPlugin.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/instrument/BuildTimeInstrumentationPlugin.kt new file mode 100644 index 00000000000..eb8c0358e04 --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/instrument/BuildTimeInstrumentationPlugin.kt @@ -0,0 +1,175 @@ +package datadog.gradle.plugin.instrument + +import org.gradle.api.Plugin +import org.gradle.api.Project +import org.gradle.api.logging.Logging +import org.gradle.api.tasks.SourceSet +import org.gradle.api.tasks.SourceSetContainer +import org.gradle.api.tasks.compile.AbstractCompile +import org.gradle.kotlin.dsl.create +import org.gradle.kotlin.dsl.getByType +import org.gradle.kotlin.dsl.newInstance +import org.gradle.kotlin.dsl.withType + +/** + * Gradle plugin that applies ByteBuddy plugins to perform build-time bytecode instrumentation. + * + * This plugin appends a post-processing action to existing `main` compilation tasks. It allows + * applying one or more ByteBuddy [net.bytebuddy.build.Plugin] implementations. + * + * Configuration: + * 1. `buildTimeInstrumentation` extension: `plugins`: list of ByteBuddy plugin class names to apply + * `additionalClasspath`: additional classpath entries required to load plugins + * 2. `buildTimeInstrumentationPlugin` configuration: dependencies containing ByteBuddy plugin implementations + * + * Example: + * ```kotlin + * buildTimeInstrumentation { + * plugins = listOf("com.example.MyByteBuddyPlugin", "com.example.AnotherPlugin") + * additionalClasspath = listOf(file("path/to/additional/classes")) + * } + * + * dependencies { + * buildTimeInstrumentationPlugin("com.example:my-bytebuddy-plugin:1.0.0") + * buildTimeInstrumentationPlugin(project( + * path = ":some:project", + * configuration = "buildTimeInstrumentationToolingPlugins" + * )) + * } + * ``` + * + * Requirements for ByteBuddy plugins: + * 1. Must implement [net.bytebuddy.build.Plugin] + * 2. Must have a constructor accepting a [java.io.File] parameter (target directory) + * 3. Plugin classes must be available on `buildTimeInstrumentationPlugin` configuration + * + * @see BuildTimeInstrumentationExtension + * @see InstrumentPostProcessingAction + * @see ByteBuddyInstrumenter + */ +class BuildTimeInstrumentationPlugin : Plugin { + private val logger = Logging.getLogger(BuildTimeInstrumentationPlugin::class.java) + + override fun apply(project: Project) { + val extension = + project.extensions.create("buildTimeInstrumentation") + + project.configurations.register(BUILD_TIME_INSTRUMENTATION_PLUGIN_CONFIGURATION) { + isVisible = false + isCanBeConsumed = false + isCanBeResolved = true + } + + for (languagePluginId in listOf("java", "kotlin", "scala", "groovy")) { + project.pluginManager.withPlugin(languagePluginId) { + configurePostCompilationInstrumentation(languagePluginId, project, extension) + } + } + } + + private fun configurePostCompilationInstrumentation( + language: String, + project: Project, + extension: BuildTimeInstrumentationExtension + ) { + val sourceSets = project.extensions.getByType() + // For any "main" source-set configure its compile task. + sourceSets.configureEach { + val sourceSetName = name + logger.info("[BuildTimeInstrumentationPlugin] source-set: {}, language: {}", sourceSetName, language) + + if (!sourceSetName.startsWith(SourceSet.MAIN_SOURCE_SET_NAME)) { + logger.debug( + "[BuildTimeInstrumentationPlugin] Skipping non-main source set {} for language {}", + sourceSetName, + language + ) + return@configureEach + } + + val compileTaskName = getCompileTaskName(language) + logger.info("[BuildTimeInstrumentationPlugin] compile task name: {}", compileTaskName) + + // For each _main_ compile task, append an instrumenting post-processing step. + // Examples of compile tasks: + // - compileJava, + // - compileMain_java17Java, + // - compileMain_jetty904Java, + // - compileMain_play25Java, + // - compileKotlin, + // - compileScala, + // - compileGroovy, + project.tasks.withType().matching { + val sourceIsEmpty = it.source.isEmpty + if (sourceIsEmpty) { + logger.debug( + "[BuildTimeInstrumentationPlugin] Skipping {} for source set {} as it has no source files", + compileTaskName, + sourceSetName + ) + } + it.name == compileTaskName && !sourceIsEmpty + }.configureEach { + logger.info( + "[BuildTimeInstrumentationPlugin] Applying '{}' configuration as compile task input", + BUILD_TIME_INSTRUMENTATION_PLUGIN_CONFIGURATION + ) + inputs.files(project.configurations.named(BUILD_TIME_INSTRUMENTATION_PLUGIN_CONFIGURATION)) + + // Compute optional Java version. + val match = Regex("compileMain_(.+)Java").matchEntire(compileTaskName) + var javaVersion: String? = null + if (match != null) { + val sourceSetSuffix = match.groupValues[1] + if (Regex("java\\d+").matches(sourceSetSuffix)) { + javaVersion = sourceSetSuffix.substring(4) + } + } + // Null is not accepted for task inputs. + val resolvedJavaVersion = javaVersion ?: DEFAULT_JAVA_VERSION + inputs.property("javaVersion", resolvedJavaVersion) + inputs.property("plugins", extension.plugins) + inputs.files(extension.additionalClasspath) + inputs.files(extension.includeClassDirectories) + + // Temporary location for raw (un-instrumented) classes. + val tmpUninstrumentedClasses = project.objects.directoryProperty().value( + project.layout.buildDirectory.dir("tmp/${name}-raw-classes") + ) + + // Class path to use for instrumentation post-processing. + val instrumentingClassPath = project.objects.fileCollection().apply { + setFrom( + classpath, + extension.additionalClasspath, + tmpUninstrumentedClasses + ) + } + + // This is where post-processing happens, i.e. where instrumentation is applied. + doLast( + "instrumentClasses", + project.objects.newInstance( + resolvedJavaVersion, + extension.plugins, + instrumentingClassPath, + destinationDirectory, + tmpUninstrumentedClasses, + extension.includeClassDirectories, + ) + ) + + logger.info( + "[BuildTimeInstrumentationPlugin] Configured post-compile instrumentation for {} for source-set {}", + compileTaskName, + sourceSetName + ) + } + } + } + + companion object { + const val DEFAULT_JAVA_VERSION = "default" + const val BUILD_TIME_INSTRUMENTATION_PLUGIN_CONFIGURATION = "buildTimeInstrumentationPlugin" + } +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/instrument/ByteBuddyInstrumenter.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/instrument/ByteBuddyInstrumenter.kt new file mode 100644 index 00000000000..576e0b01c2a --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/instrument/ByteBuddyInstrumenter.kt @@ -0,0 +1,114 @@ +package datadog.gradle.plugin.instrument + +import net.bytebuddy.ClassFileVersion +import net.bytebuddy.build.EntryPoint +import net.bytebuddy.build.Plugin +import net.bytebuddy.description.type.TypeDescription +import net.bytebuddy.dynamic.ClassFileLocator +import net.bytebuddy.dynamic.scaffold.inline.MethodNameTransformer.Suffixing +import net.bytebuddy.utility.StreamDrainer +import org.slf4j.LoggerFactory +import java.io.File +import java.io.IOException +import java.io.InputStream + +/** + * Performs build-time instrumentation of classes, called indirectly from + * [BuildTimeInstrumentationPlugin]. + * + * This is the ByteBuddy side of the task; [BuildTimeInstrumentationPlugin] contains the Gradle + * wiring and task configuration. + */ +object ByteBuddyInstrumenter { + private val log = LoggerFactory.getLogger(ByteBuddyInstrumenter::class.java) + + @Throws(Exception::class) + fun instrumentClasses( + plugins: Array, + instrumentingLoader: ClassLoader, + sourceDirectory: File, + targetDirectory: File + ) { + withThreadContextClassloader(instrumentingLoader) { + val factories = plugins.map { + try { + val pluginClass = instrumentingLoader.loadClass(it).asSubclass(Plugin::class.java) + val loadedPlugin = pluginClass.getConstructor(File::class.java).newInstance(targetDirectory) + Plugin.Factory.Simple(loadedPlugin) + } catch (throwable: Throwable) { + throw IllegalStateException("Cannot resolve plugin: $it", throwable) + } + } + + val source = Plugin.Engine.Source.ForFolder(sourceDirectory) + val target = Plugin.Engine.Target.ForFolder(targetDirectory) + val engine = Plugin.Engine.Default.of( + EntryPoint.Default.REBASE, + ClassFileVersion.ofThisVm(), + Suffixing.withRandomSuffix() + ) + + val summary = engine + .with(Plugin.Engine.PoolStrategy.Default.FAST) + .with(NonCachingClassFileLocator(instrumentingLoader)) + .with(LoggingAdapter()) + .withErrorHandlers( + Plugin.Engine.ErrorHandler.Enforcing.ALL_TYPES_RESOLVED, + Plugin.Engine.ErrorHandler.Enforcing.NO_LIVE_INITIALIZERS, + object : Plugin.Engine.ErrorHandler by Plugin.Engine.ErrorHandler.Failing.FAIL_LAST { + override fun onError(throwables: MutableMap>) { + throw IllegalStateException("Failed to transform at least one type: $throwables").apply { + throwables.values.flatten().forEach(::addSuppressed) + } + } + }) + .with(Plugin.Engine.Dispatcher.ForSerialTransformation.Factory.INSTANCE) + .apply(source, target, factories) + + if (summary.failed.isNotEmpty()) { + throw IllegalStateException("Failed to transform: ${summary.failed}") + } + } + } + + private inline fun withThreadContextClassloader( + classloader: ClassLoader, + block: () -> T + ): T { + val currentThread = Thread.currentThread() + val tccl = currentThread.contextClassLoader + try { + currentThread.contextClassLoader = classloader + return block() + } finally { + currentThread.contextClassLoader = tccl + } + } + + class LoggingAdapter : Plugin.Engine.Listener.Adapter() { + override fun onTransformation(typeDescription: TypeDescription, plugins: MutableList) { + log.debug("Transformed {} using {}", typeDescription, plugins) + } + + override fun onError(typeDescription: TypeDescription, plugin: Plugin, throwable: Throwable) { + log.warn("Failed to transform {} using {}", typeDescription, plugin, throwable) + } + } + + class NonCachingClassFileLocator(private val loader: ClassLoader) : ClassFileLocator { + @Throws(IOException::class) + override fun locate(name: String): ClassFileLocator.Resolution { + val url = loader.getResource(name.replace('.', '/') + ".class") + ?: return ClassFileLocator.Resolution.Illegal(name) + + return url.openConnection().run { + useCaches = false // avoid caching class-file resources in build-workers + getInputStream().use { input: InputStream -> + ClassFileLocator.Resolution.Explicit(StreamDrainer.DEFAULT.drain(input)) + } + } + } + + override fun close() {} + } +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/instrument/InstrumentAction.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/instrument/InstrumentAction.kt new file mode 100644 index 00000000000..330615209dd --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/instrument/InstrumentAction.kt @@ -0,0 +1,86 @@ +package datadog.gradle.plugin.instrument + +import java.io.File +import java.net.URLClassLoader +import java.util.concurrent.ConcurrentHashMap +import javax.inject.Inject +import org.gradle.api.file.FileSystemOperations +import org.gradle.api.model.ObjectFactory +import org.gradle.workers.WorkAction + +abstract class InstrumentAction : WorkAction { + @get:Inject + abstract val fileSystemOperations: FileSystemOperations + + @get:Inject + abstract val objects: ObjectFactory + + override fun execute() { + val plugins = parameters.plugins.get().toTypedArray() + val classLoaderKey = plugins.joinToString(":") + + // Reset shared class-loaders each time a new build starts. + val buildStamp = parameters.buildStartedTime.get() + var pluginClassLoader = classLoaderCache[classLoaderKey] + if (lastBuildStamp < buildStamp || pluginClassLoader == null) { + synchronized(lock) { + pluginClassLoader = classLoaderCache[classLoaderKey] + if (lastBuildStamp < buildStamp || pluginClassLoader == null) { + val created = createClassLoader(parameters.pluginClassPath.files) + pluginClassLoader = created + classLoaderCache[classLoaderKey] = created + lastBuildStamp = buildStamp + } + } + } + + val originalClassesDirectory = parameters.compilerOutputDirectory.get().asFile.toPath() + val tmpUninstrumentedDir = parameters.tmpDirectory.get().asFile.toPath() + + // Essentially a move of the original classes to the temp directory, + // because original classes will be replaced by post-processed ones. + run { + fileSystemOperations.sync { + from(originalClassesDirectory) + into(tmpUninstrumentedDir) + } + // Merge any additional class directories (e.g. unpacked dependency JARs) to be processed + parameters.includeClassDirectories.files.forEach { classesDir -> + if (classesDir.exists()) { + fileSystemOperations.copy { + from(classesDir) + into(tmpUninstrumentedDir) + } + } + } + fileSystemOperations.delete { + delete(objects.fileTree().from(originalClassesDirectory)) + } + } + + val instrumentingClassLoader = createClassLoader( + cp = parameters.instrumentingClassPath, + parent = pluginClassLoader + ) + ByteBuddyInstrumenter.instrumentClasses( + plugins = plugins, + instrumentingLoader = instrumentingClassLoader, + sourceDirectory = tmpUninstrumentedDir.toFile(), + targetDirectory = originalClassesDirectory.toFile() + ) + } + + companion object { + private val lock = Any() + private val classLoaderCache = ConcurrentHashMap() + @Volatile private var lastBuildStamp: Long = 0L + + @JvmStatic + fun createClassLoader( + cp: Iterable, + parent: ClassLoader? = InstrumentAction::class.java.classLoader + ): ClassLoader { + return URLClassLoader(cp.map { it.toURI().toURL() }.toTypedArray(), parent) + } + } +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/instrument/InstrumentPostProcessingAction.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/instrument/InstrumentPostProcessingAction.kt new file mode 100644 index 00000000000..b8c3e300afd --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/instrument/InstrumentPostProcessingAction.kt @@ -0,0 +1,79 @@ +package datadog.gradle.plugin.instrument + +import datadog.gradle.plugin.instrument.BuildTimeInstrumentationPlugin.Companion.BUILD_TIME_INSTRUMENTATION_PLUGIN_CONFIGURATION +import org.gradle.api.Action +import org.gradle.api.Project +import org.gradle.api.Task +import org.gradle.api.file.DirectoryProperty +import org.gradle.api.file.FileCollection +import org.gradle.api.invocation.BuildInvocationDetails +import org.gradle.api.logging.Logging +import org.gradle.api.provider.ListProperty +import org.gradle.jvm.toolchain.JavaLanguageVersion +import org.gradle.jvm.toolchain.JavaToolchainService +import org.gradle.kotlin.dsl.submit +import org.gradle.workers.WorkQueue +import org.gradle.workers.WorkerExecutor +import javax.inject.Inject + +abstract class InstrumentPostProcessingAction @Inject constructor( + javaVersion: String, + val plugins: ListProperty, + val instrumentingClassPath: FileCollection, + val compilerOutputDirectory: DirectoryProperty, + val tmpDirectory: DirectoryProperty, + val includeClassDirectories: FileCollection +) : Action { + private val logger = Logging.getLogger(InstrumentPostProcessingAction::class.java) + private val resolvedJavaVersion: JavaLanguageVersion = when (javaVersion) { + BuildTimeInstrumentationPlugin.DEFAULT_JAVA_VERSION -> JavaLanguageVersion.current() + else -> JavaLanguageVersion.of(javaVersion) + } + + @get:Inject + abstract val project: Project + + @get:Inject + abstract val javaToolchainService: JavaToolchainService + + @get:Inject + abstract val invocationDetails: BuildInvocationDetails + + @get:Inject + abstract val workerExecutor: WorkerExecutor + + override fun execute(task: Task) { + logger.info( + """ + [InstrumentPostProcessingAction] About to instrument classes + javaVersion=$resolvedJavaVersion, + plugins=${plugins.get()}, + instrumentingClassPath=${instrumentingClassPath.files}, + rawClassesDirectory=${compilerOutputDirectory.get().asFile} + """ + .trimIndent() + ) + + val action = this + workQueue().submit(InstrumentAction::class) { + buildStartedTime.set(invocationDetails.buildStartedTime) + pluginClassPath.from(project.configurations.named(BUILD_TIME_INSTRUMENTATION_PLUGIN_CONFIGURATION)) + plugins.set(action.plugins) + instrumentingClassPath.setFrom(action.instrumentingClassPath) + compilerOutputDirectory.set(action.compilerOutputDirectory) + tmpDirectory.set(action.tmpDirectory) + includeClassDirectories.setFrom(action.includeClassDirectories) + } + } + + private fun workQueue(): WorkQueue { + val javaLauncher = javaToolchainService.launcherFor { + languageVersion.set(resolvedJavaVersion) + }.get() + return workerExecutor.processIsolation { + forkOptions { + setExecutable(javaLauncher.executablePath.asFile.absolutePath) + } + } + } +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/instrument/InstrumentWorkParameters.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/instrument/InstrumentWorkParameters.kt new file mode 100644 index 00000000000..bf92234fd63 --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/instrument/InstrumentWorkParameters.kt @@ -0,0 +1,17 @@ +package datadog.gradle.plugin.instrument + +import org.gradle.api.file.ConfigurableFileCollection +import org.gradle.api.file.DirectoryProperty +import org.gradle.api.provider.ListProperty +import org.gradle.api.provider.Property +import org.gradle.workers.WorkParameters + +interface InstrumentWorkParameters : WorkParameters { + val buildStartedTime: Property + val pluginClassPath: ConfigurableFileCollection + val plugins: ListProperty + val instrumentingClassPath: ConfigurableFileCollection + val compilerOutputDirectory: DirectoryProperty + val tmpDirectory: DirectoryProperty + val includeClassDirectories: ConfigurableFileCollection +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleAction.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleAction.kt index d9d1793ca55..b01c9845054 100644 --- a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleAction.kt +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleAction.kt @@ -9,10 +9,14 @@ import java.net.URLClassLoader abstract class MuzzleAction : WorkAction { companion object { private val lock = Any() - private var bootCL: ClassLoader? = null - private var toolCL: ClassLoader? = null + @Volatile + private var lastBootCL: ClassLoader? = null + @Volatile + private var lastToolCL: ClassLoader? = null @Volatile private var lastBuildStamp: Long = 0 + @Volatile + private var lastBuildPathCount: Int = 0 fun createClassLoader(cp: FileCollection, parent: ClassLoader = ClassLoader.getSystemClassLoader()): ClassLoader { val urls = cp.map { it.toURI().toURL() }.toTypedArray() @@ -22,12 +26,21 @@ abstract class MuzzleAction : WorkAction { override fun execute() { val buildStamp = parameters.buildStartedTime.get() - if (bootCL == null || toolCL == null || lastBuildStamp < buildStamp) { + val buildPathCount = parameters.bootstrapClassPath.count() + parameters.toolingClassPath.count() + var bootCL : ClassLoader? = lastBootCL + var toolCL : ClassLoader? = lastToolCL + // cache boot and tool classloaders for each run; rebuild if either class-path is extended mid-build + if (bootCL == null || toolCL == null || lastBuildStamp < buildStamp || lastBuildPathCount < buildPathCount) { synchronized(lock) { - if (bootCL == null || toolCL == null || lastBuildStamp < buildStamp) { + bootCL = lastBootCL + toolCL = lastToolCL + if (bootCL == null || toolCL == null || lastBuildStamp < buildStamp || lastBuildPathCount < buildPathCount) { bootCL = createClassLoader(parameters.bootstrapClassPath) toolCL = createClassLoader(parameters.toolingClassPath, bootCL!!) + lastBootCL = bootCL + lastToolCL = toolCL lastBuildStamp = buildStamp + lastBuildPathCount = buildPathCount } } } diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleDirective.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleDirective.kt index d82e706e13d..4616363329b 100644 --- a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleDirective.kt +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleDirective.kt @@ -91,12 +91,9 @@ open class MuzzleDirective : Serializable { val nameSlug: String get() = name?.trim()?.replace(Regex("[^a-zA-Z0-9]+"), "-") ?: "" - override fun toString(): String { - return if (isCoreJdk) { - "${if (assertPass) "Pass" else "Fail"}-core-jdk" - } else { - "${if (assertPass) "pass" else "fail"} $group:$module:$versions" - } + override fun toString(): String = if (isCoreJdk) { + "${if (assertPass) "Pass" else "Fail"}-core-jdk" + } else { + "${if (assertPass) "pass" else "fail"} $group:$module:$versions" } } - diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleExtension.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleExtension.kt index 2bb88e8d69d..285301e2613 100644 --- a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleExtension.kt +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleExtension.kt @@ -3,8 +3,8 @@ package datadog.gradle.plugin.muzzle import org.gradle.api.Action import org.gradle.api.model.ObjectFactory import org.gradle.kotlin.dsl.newInstance -import javax.inject.Inject import java.util.Locale +import javax.inject.Inject /** * Muzzle extension containing all pass and fail directives. diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleMavenRepoUtils.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleMavenRepoUtils.kt index 13e8752cb27..998e0357b18 100644 --- a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleMavenRepoUtils.kt +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleMavenRepoUtils.kt @@ -12,6 +12,7 @@ import org.eclipse.aether.resolution.VersionRangeRequest import org.eclipse.aether.resolution.VersionRangeResult import org.eclipse.aether.spi.connector.RepositoryConnectorFactory import org.eclipse.aether.spi.connector.transport.TransporterFactory +import org.eclipse.aether.transport.file.FileTransporterFactory import org.eclipse.aether.transport.http.HttpTransporterFactory import org.eclipse.aether.version.Version import org.gradle.api.GradleException @@ -34,13 +35,15 @@ internal object MuzzleMavenRepoUtils { } /** - * Create new RepositorySystem for muzzle's Maven/Aether resoltions. + * Create new RepositorySystem for muzzle's Maven/Aether resolutions. + * Supports both HTTP/HTTPS and file:// repositories. */ @JvmStatic fun newRepositorySystem(): RepositorySystem { val locator = MavenRepositorySystemUtils.newServiceLocator().apply { addService(RepositoryConnectorFactory::class.java, BasicRepositoryConnectorFactory::class.java) addService(TransporterFactory::class.java, HttpTransporterFactory::class.java) + addService(TransporterFactory::class.java, FileTransporterFactory::class.java) } return locator.getService(RepositorySystem::class.java) } @@ -66,7 +69,8 @@ internal object MuzzleMavenRepoUtils { fun inverseOf( muzzleDirective: MuzzleDirective, system: RepositorySystem, - session: RepositorySystemSession + session: RepositorySystemSession, + defaultRepos: List = MUZZLE_REPOS ): Set { val allVersionsArtifact = DefaultArtifact( muzzleDirective.group, @@ -74,7 +78,7 @@ internal object MuzzleMavenRepoUtils { "jar", "[,)" ) - val repos = muzzleDirective.getRepositories(MUZZLE_REPOS) + val repos = muzzleDirective.getRepositories(defaultRepos) val allRangeRequest = VersionRangeRequest().apply { repositories = repos artifact = allVersionsArtifact @@ -119,7 +123,8 @@ internal object MuzzleMavenRepoUtils { fun resolveVersionRange( muzzleDirective: MuzzleDirective, system: RepositorySystem, - session: RepositorySystemSession + session: RepositorySystemSession, + defaultRepos: List = MUZZLE_REPOS ): VersionRangeResult { val directiveArtifact: Artifact = DefaultArtifact( muzzleDirective.group, @@ -129,7 +134,7 @@ internal object MuzzleMavenRepoUtils { muzzleDirective.versions ) val rangeRequest = VersionRangeRequest().apply { - repositories = muzzleDirective.getRepositories(MUZZLE_REPOS) + repositories = muzzleDirective.getRepositories(defaultRepos) artifact = directiveArtifact } diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzlePlugin.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzlePlugin.kt index b791f1f9a45..876c2234218 100644 --- a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzlePlugin.kt +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzlePlugin.kt @@ -1,18 +1,15 @@ package datadog.gradle.plugin.muzzle -import datadog.gradle.plugin.muzzle.MuzzleMavenRepoUtils.inverseOf -import datadog.gradle.plugin.muzzle.MuzzleMavenRepoUtils.muzzleDirectiveToArtifacts -import datadog.gradle.plugin.muzzle.MuzzleMavenRepoUtils.resolveVersionRange import datadog.gradle.plugin.muzzle.tasks.MuzzleEndTask import datadog.gradle.plugin.muzzle.tasks.MuzzleGenerateReportTask -import datadog.gradle.plugin.muzzle.tasks.MuzzleMergeReportsTask import datadog.gradle.plugin.muzzle.tasks.MuzzleGetReferencesTask +import datadog.gradle.plugin.muzzle.tasks.MuzzleMergeReportsTask import datadog.gradle.plugin.muzzle.tasks.MuzzleTask +import datadog.gradle.plugin.muzzle.planner.MuzzleTaskPlanner import org.eclipse.aether.artifact.Artifact import org.gradle.api.NamedDomainObjectProvider import org.gradle.api.Plugin import org.gradle.api.Project -import org.gradle.api.Task import org.gradle.api.artifacts.Configuration import org.gradle.api.tasks.TaskProvider import org.gradle.kotlin.dsl.create @@ -71,9 +68,8 @@ class MuzzlePlugin : Plugin { // compileMuzzle compiles all projects required to run muzzle validation. // Not adding group and description to keep this task from showing in `gradle tasks`. - @Suppress("UNCHECKED_CAST") val compileMuzzle = project.tasks.register("compileMuzzle") { - dependsOn(project.tasks.withType(Class.forName("InstrumentTask") as Class)) // kotlin can't see groovy code + inputs.files(project.providers.provider { project.allMainSourceSet }) dependsOn(bootstrapProject.tasks.named("compileJava")) dependsOn(bootstrapProject.tasks.named("compileMain_java11Java")) dependsOn(toolingProject.tasks.named("compileJava")) @@ -103,13 +99,15 @@ class MuzzlePlugin : Plugin { project.tasks.register("mergeMuzzleReports") val hasRelevantTask = project.gradle.startParameter.taskNames.any { taskName -> - // removing leading ':' if present - val muzzleTaskName = taskName.removePrefix(":") - val projectPath = project.path.removePrefix(":") - muzzleTaskName == "muzzle" || "$projectPath:muzzle" == muzzleTaskName + val taskProjectPath = taskName.substringBeforeLast(":", "") + val taskNameOnly = taskName.substringAfterLast(":") + val isRelevantForProject = taskProjectPath.isEmpty() || taskProjectPath == project.path + + isRelevantForProject && taskNameOnly.endsWith("muzzle", ignoreCase = true) } if (!hasRelevantTask) { // Adding muzzle dependencies has a large config overhead. Stop unless muzzle is explicitly run. + project.logger.info("No muzzle tasks invoked for ${project.path}, skipping muzzle task planification") return } @@ -118,37 +116,25 @@ class MuzzlePlugin : Plugin { val system = MuzzleMavenRepoUtils.newRepositorySystem() val session = MuzzleMavenRepoUtils.newRepositorySystemSession(system) + val taskPlanner = MuzzleTaskPlanner.from(system, session) project.afterEvaluate { // use runAfter to set up task finalizers in version order var runAfter: TaskProvider = muzzleTask + val muzzleReportTasks = mutableListOf>() + val directives = project.extensions.getByType().directives + taskPlanner.plan(directives).forEach { plan -> + runAfter = registerMuzzleTask(plan.directive, plan.artifact, project, runAfter, muzzleBootstrap, muzzleTooling) + muzzleReportTasks.add(runAfter) + project.logger.info("configured ${plan.directive}") + } - project.extensions.getByType().directives.forEach { directive -> - project.logger.debug("configuring {}", directive) - - if (directive.isCoreJdk) { - runAfter = addMuzzleTask(directive, null, project, runAfter, muzzleBootstrap, muzzleTooling) - } else { - val range = resolveVersionRange(directive, system, session) - - muzzleDirectiveToArtifacts(directive, range).forEach { - runAfter = addMuzzleTask(directive, it, project, runAfter, muzzleBootstrap, muzzleTooling) - } - - if (directive.assertInverse) { - inverseOf(directive, system, session).forEach { inverseDirective -> - val inverseRange = resolveVersionRange(inverseDirective, system, session) - - muzzleDirectiveToArtifacts(inverseDirective, inverseRange).forEach { - runAfter = addMuzzleTask(inverseDirective, it, project, runAfter, muzzleBootstrap, muzzleTooling) - } - } - } - } - project.logger.info("configured $directive") + if (muzzleReportTasks.isEmpty() && !directives.any { it.assertPass }) { + muzzleReportTasks.add(muzzleTask) } val timingTask = project.tasks.register("muzzle-end") { startTimeMs.set(startTime) + muzzleResultFiles.from(muzzleReportTasks.map { it.flatMap { task -> task.result } }) } // last muzzle task to run runAfter.configure { @@ -172,7 +158,7 @@ class MuzzlePlugin : Plugin { * @param muzzleTooling The configuration provider for agent tooling dependencies. * @return The muzzle task provider. */ - private fun addMuzzleTask( + private fun registerMuzzleTask( muzzleDirective: MuzzleDirective, versionArtifact: Artifact?, instrumentationProject: Project, diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleVersionUtils.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleVersionUtils.kt index b223f5c20ec..9a12471775b 100644 --- a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleVersionUtils.kt +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleVersionUtils.kt @@ -57,6 +57,7 @@ internal object MuzzleVersionUtils { v.contains("public_draft") || v.contains("-cr") || v.contains("-preview") || + v.contains("redhat") || // redhat releases often cause ArtifactNotFoundException skipVersions.contains(v) || END_NMN_PATTERN.matches(v) || GIT_SHA_PATTERN.matches(v)) @@ -67,7 +68,7 @@ internal object MuzzleVersionUtils { /** * Select a random set of versions to test */ - private val RANGE_COUNT_LIMIT = 25 + internal val RANGE_COUNT_LIMIT = 25 /** * Select a random set of versions to test, limiting the range for efficiency. diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleWorkParameters.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleWorkParameters.kt index 5af13d9a10d..6bb2c031657 100644 --- a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleWorkParameters.kt +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/MuzzleWorkParameters.kt @@ -6,13 +6,12 @@ import org.gradle.api.provider.Property import org.gradle.workers.WorkParameters interface MuzzleWorkParameters : WorkParameters { - val buildStartedTime: Property - val bootstrapClassPath: ConfigurableFileCollection - val toolingClassPath: ConfigurableFileCollection - val instrumentationClassPath: ConfigurableFileCollection - val testApplicationClassPath: ConfigurableFileCollection - val assertPass: Property - val muzzleDirective: Property - val resultFile: RegularFileProperty + val buildStartedTime: Property + val bootstrapClassPath: ConfigurableFileCollection + val toolingClassPath: ConfigurableFileCollection + val instrumentationClassPath: ConfigurableFileCollection + val testApplicationClassPath: ConfigurableFileCollection + val assertPass: Property + val muzzleDirective: Property + val resultFile: RegularFileProperty } - diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/TestedArtifact.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/TestedArtifact.kt index 81f6451d651..ac2a409b93e 100644 --- a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/TestedArtifact.kt +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/TestedArtifact.kt @@ -4,11 +4,11 @@ import org.eclipse.aether.version.Version // Changed from internal to public for cross-file accessibility internal data class TestedArtifact( - val instrumentation: String, - val group: String, - val module: String, - val lowVersion: Version, - val highVersion: Version + val instrumentation: String, + val group: String, + val module: String, + val lowVersion: Version, + val highVersion: Version ) { - fun key(): String = "$instrumentation:$group:$module" + fun key(): String = "$instrumentation:$group:$module" } diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/VersionSet.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/VersionSet.kt index de1422bb762..2574163f5ad 100644 --- a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/VersionSet.kt +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/VersionSet.kt @@ -4,11 +4,11 @@ import org.eclipse.aether.version.Version import java.util.SortedSet class VersionSet(versions: Collection) { - private val sortedVersions: SortedSet = sortedSetOf() + private val sortedVersions: SortedSet = sortedSetOf() - init { - versions.forEach { sortedVersions.add(ParsedVersion(it)) } - } + init { + versions.forEach { sortedVersions.add(ParsedVersion(it)) } + } val lowAndHighForMajorMinor: List get() { @@ -30,52 +30,49 @@ class VersionSet(versions: Collection) { return resultSet.map { it.version } } - private class ParsedVersion(val version: Version) : Comparable { - companion object { - private val dotPattern = Regex("\\.") - private const val VERSION_SHIFT = 12 - } - val versionNumber: Long - val ending: String - init { - var versionString = version.toString() - var ending = "" - val dash = versionString.indexOf('-') - if (dash > 0) { - ending = versionString.substring(dash + 1) - versionString = versionString.substring(0, dash) - } - val groups = versionString.split(dotPattern).toMutableList() - var versionNumber = 0L - var iteration = 0 - while (iteration < 3) { - versionNumber = versionNumber shl VERSION_SHIFT - if (groups.isNotEmpty() && groups[0].toIntOrNull() != null) { - versionNumber += groups.removeAt(0).toLong() - } - iteration++ - } - if (groups.isNotEmpty()) { - val rest = groups.joinToString(".") - ending = if (ending.isEmpty()) rest else "$rest-$ending" - } - this.versionNumber = versionNumber - this.ending = ending - } - val majorMinor: Int - get() = (versionNumber shr VERSION_SHIFT).toInt() - override fun compareTo(other: ParsedVersion): Int { - val diff = versionNumber - other.versionNumber - return if (diff != 0L) diff.toInt() else ending.compareTo(other.ending) - } - override fun equals(other: Any?): Boolean { - if (this === other) return true - if (other !is ParsedVersion) return false - return versionNumber == other.versionNumber && ending == other.ending - } - override fun hashCode(): Int { - return (versionNumber * 31 + ending.hashCode()).toInt() + internal class ParsedVersion(val version: Version) : Comparable { + companion object { + private val dotPattern = Regex("\\.") + private const val VERSION_SHIFT = 12 + } + val versionNumber: Long + val ending: String + init { + var versionString = version.toString() + var ending = "" + val dash = versionString.indexOf('-') + if (dash > 0) { + ending = versionString.substring(dash + 1) + versionString = versionString.substring(0, dash) + } + val groups = versionString.split(dotPattern).toMutableList() + var versionNumber = 0L + var iteration = 0 + while (iteration < 3) { + versionNumber = versionNumber shl VERSION_SHIFT + if (groups.isNotEmpty() && groups[0].toIntOrNull() != null) { + versionNumber += groups.removeAt(0).toLong() } + iteration++ + } + if (groups.isNotEmpty()) { + val rest = groups.joinToString(".") + ending = if (ending.isEmpty()) rest else "$rest-$ending" + } + this.versionNumber = versionNumber + this.ending = ending } + val majorMinor: Int + get() = (versionNumber shr VERSION_SHIFT).toInt() + override fun compareTo(other: ParsedVersion): Int { + val diff = versionNumber - other.versionNumber + return if (diff != 0L) diff.toInt() else ending.compareTo(other.ending) + } + override fun equals(other: Any?): Boolean { + if (this === other) return true + if (other !is ParsedVersion) return false + return versionNumber == other.versionNumber && ending == other.ending + } + override fun hashCode(): Int = (versionNumber * 31 + ending.hashCode()).toInt() + } } - diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/planner/MavenMuzzleResolutionService.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/planner/MavenMuzzleResolutionService.kt new file mode 100644 index 00000000000..ee9d7c38369 --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/planner/MavenMuzzleResolutionService.kt @@ -0,0 +1,23 @@ +package datadog.gradle.plugin.muzzle.planner + +import datadog.gradle.plugin.muzzle.MuzzleDirective +import datadog.gradle.plugin.muzzle.MuzzleMavenRepoUtils +import org.eclipse.aether.RepositorySystem +import org.eclipse.aether.RepositorySystemSession +import org.eclipse.aether.artifact.Artifact + +/** + * Default [MuzzleResolutionService] implementation backed by Maven/Aether resolution. + */ +internal class MavenMuzzleResolutionService( + private val system: RepositorySystem, + private val session: RepositorySystemSession, +) : MuzzleResolutionService { + override fun resolveArtifacts(directive: MuzzleDirective): Set { + val range = MuzzleMavenRepoUtils.resolveVersionRange(directive, system, session) + return MuzzleMavenRepoUtils.muzzleDirectiveToArtifacts(directive, range) + } + + override fun inverseOf(directive: MuzzleDirective): Set = + MuzzleMavenRepoUtils.inverseOf(directive, system, session) +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/planner/MuzzleResolutionService.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/planner/MuzzleResolutionService.kt new file mode 100644 index 00000000000..bcdd81427e9 --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/planner/MuzzleResolutionService.kt @@ -0,0 +1,19 @@ +package datadog.gradle.plugin.muzzle.planner + +import datadog.gradle.plugin.muzzle.MuzzleDirective +import org.eclipse.aether.artifact.Artifact + +/** + * Resolves muzzle directives into concrete artifacts and inverse directives. + */ +internal interface MuzzleResolutionService { + /** + * Resolves all dependency artifacts to test for the given directive. + */ + fun resolveArtifacts(directive: MuzzleDirective): Set + + /** + * Computes directives representing the inverse of the given directive. + */ + fun inverseOf(directive: MuzzleDirective): Set +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/planner/MuzzleTaskPlan.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/planner/MuzzleTaskPlan.kt new file mode 100644 index 00000000000..6b3a3dbd5f0 --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/planner/MuzzleTaskPlan.kt @@ -0,0 +1,14 @@ +package datadog.gradle.plugin.muzzle.planner + +import datadog.gradle.plugin.muzzle.MuzzleDirective +import org.eclipse.aether.artifact.Artifact + +/** + * Planned unit of muzzle work for task creation. + * + * For `coreJdk()` directives, [artifact] is `null`. + */ +internal data class MuzzleTaskPlan( + val directive: MuzzleDirective, + val artifact: Artifact?, +) diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/planner/MuzzleTaskPlanner.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/planner/MuzzleTaskPlanner.kt new file mode 100644 index 00000000000..107fbf2f2d9 --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/planner/MuzzleTaskPlanner.kt @@ -0,0 +1,55 @@ +package datadog.gradle.plugin.muzzle.planner + +import datadog.gradle.plugin.muzzle.MuzzleDirective +import org.eclipse.aether.RepositorySystem +import org.eclipse.aether.RepositorySystemSession + +/** + * Expands configured directives into ordered task plans. + */ +internal class MuzzleTaskPlanner( + private val resolutionService: MuzzleResolutionService, +) { + companion object { + fun from(system: RepositorySystem, session: RepositorySystemSession): MuzzleTaskPlanner = + MuzzleTaskPlanner(MavenMuzzleResolutionService(system, session)) + } + + /** + * Expands declared muzzle directives into executable task plans. + * + * Planning rules: + * - Core-JDK directives (`coreJdk()`) create exactly one [MuzzleTaskPlan] with `artifact = null`. + * - Non-core directives are resolved with [MuzzleResolutionService.resolveArtifacts], creating one + * plan per resolved artifact. + * - If a non-core directive has `assertInverse = true`, inverse directives are obtained from + * [MuzzleResolutionService.inverseOf], then each inverse directive is resolved and expanded with + * the same one-plan-per-artifact rule. + * + * Ordering: + * - The input [directives] order is preserved. + * - Direct plans for a directive are emitted before its inverse plans. + * - Artifact plan order follows the iteration order returned by the resolution service. + * + * No de-duplication is performed here. If needed, de-duplication must be handled by callers or by + * the resolution service implementation. + */ + fun plan(directives: List): List = buildList { + directives.forEach { directive -> + if (directive.isCoreJdk) { + add(MuzzleTaskPlan(directive, null)) + } else { + resolutionService.resolveArtifacts(directive).forEach { artifact -> + add(MuzzleTaskPlan(directive, artifact)) + } + if (directive.assertInverse) { + resolutionService.inverseOf(directive).forEach { inverseDirective -> + resolutionService.resolveArtifacts(inverseDirective).forEach { artifact -> + add(MuzzleTaskPlan(inverseDirective, artifact)) + } + } + } + } + } + } +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/tasks/MuzzleEndTask.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/tasks/MuzzleEndTask.kt index daf0ce44b06..47ab18bb9c9 100644 --- a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/tasks/MuzzleEndTask.kt +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/tasks/MuzzleEndTask.kt @@ -1,39 +1,167 @@ package datadog.gradle.plugin.muzzle.tasks import datadog.gradle.plugin.muzzle.pathSlug +import org.gradle.api.file.ConfigurableFileCollection import org.gradle.api.provider.Property import org.gradle.api.tasks.Input +import org.gradle.api.tasks.InputFiles import org.gradle.api.tasks.OutputFile +import org.gradle.api.tasks.PathSensitive +import org.gradle.api.tasks.PathSensitivity import org.gradle.api.tasks.TaskAction +import java.io.File +import java.io.StringWriter +import javax.xml.stream.XMLOutputFactory abstract class MuzzleEndTask : AbstractMuzzleTask() { @get:Input abstract val startTimeMs: Property + @get:InputFiles + @get:PathSensitive(PathSensitivity.RELATIVE) + abstract val muzzleResultFiles: ConfigurableFileCollection + + @get:OutputFile + val resultsFile = project + .layout + .buildDirectory + .file("test-results/muzzle/TEST-muzzle-${project.pathSlug}.xml") + @get:OutputFile - val resultsFile = project.rootProject + val legacyResultsFile = project.rootProject .layout .buildDirectory .file("${MUZZLE_TEST_RESULTS}/${project.pathSlug}_muzzle/results.xml") @TaskAction fun generatesResultFile() { + val report = buildJUnitReport() + writeReportFile(project.file(resultsFile), renderReportXml(report), "muzzle junit") + writeReportFile(project.file(legacyResultsFile), renderLegacyReportXml(report.durationSeconds), "muzzle legacy") + } + + private fun buildJUnitReport(): MuzzleJUnitReport { val endTimeMs = System.currentTimeMillis() val seconds = (endTimeMs - startTimeMs.get()).toDouble() / 1000.0 - with(project.file(resultsFile)) { - parentFile.mkdirs() - writeText( - """ - - - - - """.trimIndent() - ) - project.logger.info("Wrote muzzle results report to\n $this") + val testCases = muzzleResultFiles.files + .sortedBy { it.name } + .map { resultFile -> + val taskName = resultFile.name.removeSuffix(".txt") + when { + !resultFile.exists() -> { + MuzzleJUnitCase( + name = taskName, + failureMessage = "Muzzle result file missing", + failureText = "Expected ${resultFile.path}" + ) + } + + resultFile.readText() == "PASSING" -> MuzzleJUnitCase(name = taskName) + else -> { + MuzzleJUnitCase( + name = taskName, + failureMessage = "Muzzle validation failed", + failureText = resultFile.readText() + ) + } + } + } + return MuzzleJUnitReport( + suiteName = project.path, + module = project.path, + className = "muzzle.${project.pathSlug}", + durationSeconds = seconds, + testCases = testCases + ) + } + + private fun renderReportXml(report: MuzzleJUnitReport): String { + val output = StringWriter() + val xmlWriter = XMLOutputFactory.newInstance().createXMLStreamWriter(output) + with(xmlWriter) { + try { + writeStartDocument("UTF-8", "1.0") + writeCharacters("\n") + writeStartElement("testsuite") + writeAttribute("name", report.suiteName) + writeAttribute("tests", report.testCases.size.toString()) + writeAttribute("failures", report.failures.toString()) + writeAttribute("errors", "0") + writeAttribute("skipped", "0") + writeAttribute("time", report.durationSeconds.toString()) + writeCharacters("\n") + + writeStartElement("properties") + writeCharacters("\n") + writeEmptyElement("property") + writeAttribute("name", "category") + writeAttribute("value", "muzzle") + writeCharacters("\n") + writeEmptyElement("property") + writeAttribute("name", "module") + writeAttribute("value", report.module) + writeCharacters("\n") + writeEndElement() + writeCharacters("\n") + + report.testCases.forEach { testCase -> + writeStartElement("testcase") + writeAttribute("classname", report.className) + writeAttribute("name", testCase.name) + writeAttribute("time", "0") + if (testCase.failureMessage != null) { + writeCharacters("\n") + writeStartElement("failure") + writeAttribute("message", testCase.failureMessage) + writeCharacters(testCase.failureText ?: "") + writeEndElement() + writeCharacters("\n") + } + writeEndElement() + writeCharacters("\n") + } + writeEndElement() + writeEndDocument() + flush() + } finally { + close() + } } + return output.toString() + } + + private fun writeReportFile(file: File, xml: String, label: String) { + file.parentFile.mkdirs() + file.writeText(xml) + project.logger.info("Wrote $label report to\n $file") } + private fun renderLegacyReportXml(durationSeconds: Double): String { + return """ + + + + + """.trimIndent() + } + + private data class MuzzleJUnitReport( + val suiteName: String, + val module: String, + val className: String, + val durationSeconds: Double, + val testCases: List + ) { + val failures: Int + get() = testCases.count { it.failureMessage != null } + } + + private data class MuzzleJUnitCase( + val name: String, + val failureMessage: String? = null, + val failureText: String? = null + ) + companion object { private const val MUZZLE_TEST_RESULTS = "muzzle-test-results" } diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/tasks/MuzzleTask.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/tasks/MuzzleTask.kt index cefba57e610..2d5d830ea3b 100644 --- a/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/tasks/MuzzleTask.kt +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/tasks/MuzzleTask.kt @@ -31,12 +31,10 @@ abstract class MuzzleTask @Inject constructor( objects: ObjectFactory, providers: ProviderFactory, ) : AbstractMuzzleTask() { - override fun getDescription(): String { - return if (muzzleDirective.isPresent) { - "Run instrumentation muzzle on ${muzzleDirective.get().name} dependency" - } else { - "Run instrumentation muzzle on compile time dependencies" - } + override fun getDescription(): String = if (muzzleDirective.isPresent) { + "Run instrumentation muzzle on ${muzzleDirective.get().name} dependency" + } else { + "Run instrumentation muzzle on compile time dependencies" } @get:Inject @@ -68,23 +66,25 @@ abstract class MuzzleTask @Inject constructor( @get:Optional val muzzleDirective: Property = objects.property() - // This output is only used to make the task cacheable, this is not exposed @get:OutputFile - @get:Optional - protected val result: RegularFileProperty = objects.fileProperty().convention( - project.layout.buildDirectory.file("reports/${name}.txt") + val result: RegularFileProperty = objects.fileProperty().convention( + project.layout.buildDirectory.file("reports/$name.txt") ) @TaskAction fun muzzle() { when { + // Version-specific task: created by MuzzlePlugin for each resolved artifact. + muzzleDirective.isPresent -> { + assertMuzzle(muzzleDirective.get()) + } + // Fallback for the root "muzzle" lifecycle task when no pass{} directives are + // declared. In that case there are no version-specific pass tasks, so we assert + // the instrumentation against its own compile-time classpath as a basic sanity check. !project.extensions.getByType().directives.any { it.assertPass } -> { project.logger.info("No muzzle pass directives configured. Asserting pass against instrumentation compile-time dependencies") assertMuzzle() } - muzzleDirective.isPresent -> { - assertMuzzle(muzzleDirective.get()) - } } } @@ -98,6 +98,10 @@ abstract class MuzzleTask @Inject constructor( // See https://github.com/gradle/gradle/issues/33987 workerExecutor.processIsolation { forkOptions { + // datadog.trace.agent.tooling.muzzle.MuzzleVersionScanPlugin needs reflective access to ClassLoader.findLoadedClass + if(javaLauncher.metadata.languageVersion > JavaLanguageVersion.of(9)) { + jvmArgs("--add-opens=java.base/java.lang=ALL-UNNAMED") + } executable(javaLauncher.executablePath) } } diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/naming/InstrumentationNamingExtension.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/naming/InstrumentationNamingExtension.kt new file mode 100644 index 00000000000..24d11e1020e --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/naming/InstrumentationNamingExtension.kt @@ -0,0 +1,43 @@ +package datadog.gradle.plugin.naming + +import org.gradle.api.provider.Property +import org.gradle.api.provider.SetProperty + +/** + * Extension for configuring instrumentation naming convention checks. + * + * Example usage: + * ``` + * instrumentationNaming { + * instrumentationsDir.set(file("dd-java-agent/instrumentation")) + * exclusions.set(setOf("http-url-connection", "sslsocket")) + * suffixes.set(setOf("-common", "-stubs")) + * } + * ``` + */ +abstract class InstrumentationNamingExtension { + /** + * The directory containing instrumentation modules. + * Defaults to "dd-java-agent/instrumentation". + */ + abstract val instrumentationsDir: Property + + /** + * Set of module names to exclude from naming convention checks. + * These modules will not be validated against the naming rules. + */ + abstract val exclusions: SetProperty + + /** + * Set of allowed suffixes for module names (e.g., "-common", "-stubs"). + * Module names must end with either one of these suffixes or a version number. + * Defaults to ["-common", "-stubs"]. + */ + abstract val suffixes: SetProperty + + init { + instrumentationsDir.convention("dd-java-agent/instrumentation") + exclusions.convention(emptySet()) + suffixes.convention(setOf("-common", "-stubs")) + } +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/naming/InstrumentationNamingPlugin.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/naming/InstrumentationNamingPlugin.kt new file mode 100644 index 00000000000..e19c1b7c758 --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/naming/InstrumentationNamingPlugin.kt @@ -0,0 +1,202 @@ +package datadog.gradle.plugin.naming + +import org.gradle.api.GradleException +import org.gradle.api.Plugin +import org.gradle.api.Project +import org.gradle.kotlin.dsl.create +import java.io.File + +/** + * Plugin that validates naming conventions for instrumentation modules. + * + * Rules: + * 1. Module name must end with a version (e.g., "2.0", "3.1") OR end with "-common" + * 2. Module name must include the parent directory name + * (e.g., "couchbase-2.0" must contain "couchbase" which is the parent directory name) + * + * Apply this plugin: + * ``` + * plugins { + * id("dd-trace-java.instrumentation-naming") + * } + * ``` + */ +class InstrumentationNamingPlugin : Plugin { + private val versionPattern : Regex = Regex("""\d+\.\d+(\.\d+)?$""") + + override fun apply(target: Project) { + val extension = target.extensions.create("instrumentationNaming") + + target.tasks.register("checkInstrumentationNaming") { + group = "verification" + description = "Validates naming conventions for instrumentation modules" + + doLast { + val instrumentationsDir = target.rootProject.file(extension.instrumentationsDir) + val exclusions = extension.exclusions.get() + val suffixes = extension.suffixes.get() + + if (!instrumentationsDir.exists() || !instrumentationsDir.isDirectory) { + throw GradleException( + "Instrumentations directory not found: ${instrumentationsDir.absolutePath}" + ) + } + + val violations = validateInstrumentations(instrumentationsDir, exclusions, suffixes) + + if (violations.isNotEmpty()) { + val suffixesStr = suffixes.joinToString("', '", "'", "'") + val errorMessage = buildString { + appendLine(""" + + Instrumentation naming convention violations found: + + """.trimIndent()) + violations.forEach { violation -> + appendLine(""" + • ${violation.path} + ${violation.message} + """.trimIndent()) + } + append(""" + Naming rules: + 1. Module name must end with a version (e.g., '2.0', '3.1') OR one of: $suffixesStr + 2. Module name must include the parent directory name + Example: 'couchbase/couchbase-2.0' ✓ (contains 'couchbase') + + To exclude specific modules or customize suffixes, configure the plugin: + instrumentationNaming { + exclusions.set(setOf("module-name")) + suffixes.set(setOf("-common", "-stubs")) + } + """.trimIndent()) + } + throw GradleException(errorMessage) + } else { + target.logger.lifecycle("✓ All instrumentation modules follow naming conventions") + } + } + } + } + + private fun validateInstrumentations( + instrumentationsDir: File, + exclusions: Set, + suffixes: Set + ): List { + val violations = mutableListOf() + + fun hasBuildFile(dir: File): Boolean = dir.listFiles()?.any { + it.name == "build.gradle" || it.name == "build.gradle.kts" + } ?: false + + fun traverseModules(currentDir: File, parentName: String?) { + currentDir.listFiles { file -> file.isDirectory }?.forEach childLoop@{ childDir -> + val moduleName = childDir.name + + // Skip build directories and other non-instrumentation directories + if (moduleName in setOf("build", "src", ".gradle")) { + return@childLoop + } + // skip the special datadog top level instrumentation directory + if (parentName == null && moduleName == "datadog") { + return@childLoop + } + + val childHasBuildFile = hasBuildFile(childDir) + val nestedModules = childDir.listFiles { file -> file.isDirectory }?.filter { hasBuildFile(it) } ?: emptyList() + + if (childHasBuildFile && moduleName !in exclusions) { + val relativePath = childDir.relativeTo(instrumentationsDir).path + if (parentName == null && nestedModules.isEmpty()) { + validateLeafModuleName(moduleName, relativePath, suffixes)?.let { violations.add(it) } + } else if (parentName != null) { + violations.addAll(validateModuleName(moduleName, parentName, relativePath, suffixes)) + } + } + + // Continue traversing to validate deeply nested modules + if (nestedModules.isNotEmpty() || !childHasBuildFile) { + traverseModules(childDir, moduleName) + } + } + } + + traverseModules(instrumentationsDir, null) + + return violations + } + + private fun validateModuleName( + moduleName: String, + parentName: String, + relativePath: String, + suffixes: Set + ): List { + // Rule 1: Module name must end with version pattern or one of the configured suffixes + validateVersionOrSuffix(moduleName, relativePath, suffixes)?.let { return listOf(it) } + + // Rule 2: Module name must contain parent directory name (all characters in any order) + if (!containsAllChars(moduleName, parentName)) { + return listOf(NamingViolation( + relativePath, + "Module name '$moduleName' should contain all characters from parent directory name '$parentName'" + )) + } + + return emptyList() + } + + /** + * Validates naming for leaf modules (modules at the top level with no parent grouping). + * These only need to check the version/suffix requirement. + */ + private fun validateLeafModuleName( + moduleName: String, + relativePath: String, + suffixes: Set + ): NamingViolation? { + return validateVersionOrSuffix(moduleName, relativePath, suffixes) + } + + /** + * Validates that a module name ends with either a version or one of the configured suffixes. + */ + private fun validateVersionOrSuffix( + moduleName: String, + relativePath: String, + suffixes: Set + ): NamingViolation? { + val endsWithSuffix = suffixes.any { moduleName.endsWith(it) } + val endsWithVersion = versionPattern.containsMatchIn(moduleName) + + if (!endsWithVersion && !endsWithSuffix) { + val suffixesStr = suffixes.joinToString("', '", "'", "'") + return NamingViolation( + relativePath, + "Module name '$moduleName' must end with a version (e.g., '2.0', '3.1.0') or one of: $suffixesStr" + ) + } + + return null + } + + /** + * Checks if all characters from 'required' string appear in 'source' string (case-insensitive). + * Characters can appear in any order. + */ + private fun containsAllChars(source: String, required: String): Boolean { + val sourceChars = source.lowercase().toList() + val requiredChars = required.lowercase().groupingBy { it }.eachCount() + val sourceCharCounts = sourceChars.groupingBy { it }.eachCount() + + return requiredChars.all { (char, count) -> + sourceCharCounts.getOrDefault(char, 0) >= count + } + } + + private data class NamingViolation( + val path: String, + val message: String + ) +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/testJvmConstraints/ProvideJvmArgsOnJvmLauncherVersion.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/testJvmConstraints/ProvideJvmArgsOnJvmLauncherVersion.kt new file mode 100644 index 00000000000..52443474337 --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/testJvmConstraints/ProvideJvmArgsOnJvmLauncherVersion.kt @@ -0,0 +1,38 @@ +package datadog.gradle.plugin.testJvmConstraints + +import org.gradle.api.JavaVersion +import org.gradle.api.provider.Provider +import org.gradle.api.tasks.Input +import org.gradle.api.tasks.Internal +import org.gradle.api.tasks.Optional +import org.gradle.api.tasks.testing.Test +import org.gradle.process.CommandLineArgumentProvider + +class ProvideJvmArgsOnJvmLauncherVersion( + @get:Internal + val test: Test, + + @get:Input + val applyFromVersion: JavaVersion, + + @get:Input + val jvmArgsToApply: List, + + @get:Input + @get:Optional + val additionalCondition: Provider +) : CommandLineArgumentProvider { + + override fun asArguments(): Iterable { + val launcherVersion = test.javaLauncher + .map { JavaVersion.toVersion(it.metadata.languageVersion.asInt()) } + .orElse(JavaVersion.current()) + .get() + + return if (launcherVersion.isCompatibleWith(applyFromVersion) && additionalCondition.getOrElse(true)) { + jvmArgsToApply + } else { + emptyList() + } + } +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/testJvmConstraints/TestJvmConstraintsExtension.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/testJvmConstraints/TestJvmConstraintsExtension.kt new file mode 100644 index 00000000000..03b8c533d61 --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/testJvmConstraints/TestJvmConstraintsExtension.kt @@ -0,0 +1,43 @@ +package datadog.gradle.plugin.testJvmConstraints + +import org.gradle.api.JavaVersion +import org.gradle.api.provider.ListProperty +import org.gradle.api.provider.Property + +interface TestJvmConstraintsExtension { + /** + * Sets an explicit minimum bound to allowed JDK version + */ + val minJavaVersion: Property + + /** + * Sets an explicit maximum bound to allowed JDK version + */ + val maxJavaVersion: Property + + /** + * List of allowed JDK names (passed through the `testJvm` property). + */ + val forceJdk: ListProperty + + /** + * List of included JDK names (passed through the `testJvm` property). + */ + val includeJdk: ListProperty + + /** + * List of excluded JDK names (passed through the `testJvm` property). + */ + val excludeJdk: ListProperty + + /** + * Indicate if the test JVM allows reflective access to JDK + * `java.base/java.lang` and `java.base/java.util` modules by + * openning them. + */ + val allowReflectiveAccessToJdk: Property + + companion object { + const val TEST_JVM_CONSTRAINTS = "testJvmConstraints" + } +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/testJvmConstraints/TestJvmConstraintsUtils.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/testJvmConstraints/TestJvmConstraintsUtils.kt new file mode 100644 index 00000000000..a8c5ddf69f6 --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/testJvmConstraints/TestJvmConstraintsUtils.kt @@ -0,0 +1,46 @@ +package datadog.gradle.plugin.testJvmConstraints + +import org.gradle.api.JavaVersion +import org.gradle.api.logging.Logging + +private val logger = Logging.getLogger("TestJvmConstraintsUtils") + +internal fun TestJvmConstraintsExtension.isJavaVersionAllowed(version: JavaVersion): Boolean = withinAllowedRange(version) + +internal fun TestJvmConstraintsExtension.isTestJvmAllowed(testJvmSpec: TestJvmSpec): Boolean { + val testJvmName = testJvmSpec.normalizedTestJvm.get() + + val included = includeJdk.get() + if (included.isNotEmpty() && included.none { it.equals(testJvmName, ignoreCase = true) }) { + return false + } + + val excluded = excludeJdk.get() + if (excluded.isNotEmpty() && excluded.any { it.equals(testJvmName, ignoreCase = true) }) { + return false + } + + val launcherVersion = JavaVersion.toVersion(testJvmSpec.javaTestLauncher.get().metadata.languageVersion.asInt()) + if (!withinAllowedRange(launcherVersion) && forceJdk.get().none { it.equals(testJvmName, ignoreCase = true) }) { + return false + } + + return true +} + +private fun TestJvmConstraintsExtension.withinAllowedRange(currentJvmVersion: JavaVersion): Boolean { + val definedMin = minJavaVersion.isPresent + val definedMax = maxJavaVersion.isPresent + + if (definedMin && (minJavaVersion.get()) > currentJvmVersion) { + logger.info("'isWithinAllowedRange' returns false b/o testJvmConstraints.minJavaVersion=${minJavaVersion.get()} is defined and greater than test JVM version=$currentJvmVersion") + return false + } + + if (definedMax && (maxJavaVersion.get()) < currentJvmVersion) { + logger.info("'isWithinAllowedRange' returns false because testJvmConstraints.maxJavaVersion=${maxJavaVersion.get()} is defined and lower than test JVM version=$currentJvmVersion") + return false + } + + return true +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/testJvmConstraints/TestJvmSpec.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/testJvmConstraints/TestJvmSpec.kt new file mode 100644 index 00000000000..da9663f247a --- /dev/null +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/testJvmConstraints/TestJvmSpec.kt @@ -0,0 +1,209 @@ +package datadog.gradle.plugin.testJvmConstraints + +import org.gradle.api.GradleException +import org.gradle.api.Project +import org.gradle.api.internal.project.ProjectInternal +import org.gradle.api.internal.provider.PropertyFactory +import org.gradle.api.provider.Provider +import org.gradle.internal.jvm.inspection.JavaInstallationRegistry +import org.gradle.jvm.toolchain.JavaLanguageVersion +import org.gradle.jvm.toolchain.JavaLauncher +import org.gradle.jvm.toolchain.JavaToolchainService +import org.gradle.jvm.toolchain.JavaToolchainSpec +import org.gradle.jvm.toolchain.JvmImplementation +import org.gradle.jvm.toolchain.JvmVendorSpec +import org.gradle.jvm.toolchain.internal.DefaultToolchainSpec +import org.gradle.jvm.toolchain.internal.SpecificInstallationToolchainSpec +import org.gradle.kotlin.dsl.support.serviceOf +import java.nio.file.Files +import java.nio.file.Path +import java.nio.file.Paths + +/** + * Handles the `testJvm` property to resolve a Java launcher for testing. + * + * The `testJvm` property can be set via command line or environment variable to specify + * which JVM to use for running tests. E.g. + * + * ```shell + * ./gradlew test -PtestJvm=ZULU11 + * ``` + * + * This handles local setup, and CI environment, where the environment variables are defined here: + * * https://github.com/DataDog/dd-trace-java-docker-build/blob/a4f4bfa9d7fe0708858e595697dc67970a2a458f/Dockerfile#L182-L188 + * * https://github.com/DataDog/dd-trace-java-docker-build/blob/a4f4bfa9d7fe0708858e595697dc67970a2a458f/Dockerfile#L222-L241 + */ +class TestJvmSpec(val project: Project) { + companion object { + const val TEST_JVM = "testJvm" + } + + private val currentJavaHomePath = project.providers.systemProperty("java.home").map { it.normalizeToJDKJavaHome() } + + /** + * The raw `testJvm` property as passed via command line or environment variable. + */ + val testJvmProperty: Provider = project.providers.gradleProperty(TEST_JVM) + + /** + * Normalized `tip` string that's set to the highest JAVA_X_HOME version found in environment variables. + */ + val normalizedTestJvm: Provider = testJvmProperty.map { testJvm -> + if (testJvm.isBlank()) { + throw GradleException("testJvm property is blank") + } + + when (testJvm) { + "tip" -> { + val javaVersions = discoverJavaVersionsViaToolchains() + ?: discoverJavaVersionsViaEnvVars() + + if (javaVersions.isEmpty()) { + throw GradleException("No Java installations found via toolchains or JAVA_X_HOME environment variables.") + } + + javaVersions.max().toString() + } + + else -> testJvm + } + }.map { project.logger.info("normalized testJvm: {}", it); it } + + /** + * The home path of the test JVM. + * + * The `` string (`8`, `11`, `ZULU8`, `GRAALVM25`, etc.) is interpreted in that order: + * 1. Lookup for a valid path, + * 2. Look JVM via Gradle toolchains + * + * Holds the resolved JavaToolchainSpec for the test JVM. + */ + private val testJvmSpec = normalizedTestJvm.map { + val (distribution, version) = Regex("([a-zA-Z]*)([0-9]+)").matchEntire(it)?.groupValues?.drop(1) ?: listOf("", "") + + // Allow looking up JAVA__HOME environment variable (e.g., JAVA_IBM8_HOME, JAVA_SEMERU8_HOME), + // because gradle doesn't offer a way to distinguish ibm8 or semeru8 + val envVarValue = project.providers.environmentVariable("JAVA_${it.uppercase()}_HOME").orNull + + when { + Files.exists(Paths.get(it)) -> it.normalizeToJDKJavaHome().toToolchainSpec() + envVarValue != null && Files.exists(Paths.get(envVarValue)) -> envVarValue.normalizeToJDKJavaHome().toToolchainSpec() + version.isNotBlank() -> { + // Best effort to make a spec for the passed testJvm + // `8`, `11`, `ZULU8`, `GRAALVM25`, etc. + // if it is an integer, we assume it's a Java version + // also we can handle on macOs oracle, zulu, semeru, graalvm prefixes + + // This is using internal APIs + DefaultToolchainSpec(project.serviceOf()).apply { + languageVersion.set(JavaLanguageVersion.of(version.toInt())) + when (distribution.lowercase()) { + "" -> { + // No-op + } + + "oracle" -> { + vendor.set(JvmVendorSpec.ORACLE) + } + + "zulu" -> { + vendor.set(JvmVendorSpec.AZUL) + } + + // Note: Both IBM and Semeru report java.vendor = "IBM Corporation", + // so JvmVendorSpec cannot distinguish them. Use JAVA_IBM8_HOME or + // JAVA_SEMERU8_HOME env vars for reliable selection. + "ibm", "semeru" -> { + vendor.set(JvmVendorSpec.IBM) + implementation.set(JvmImplementation.J9) + } + + "graalvm" -> { + vendor.set(JvmVendorSpec.GRAAL_VM) + nativeImageCapable.set(true) + } + + else -> { + throw GradleException("Unknown JVM distribution '$distribution'. Supported: oracle, zulu, ibm, semeru, graalvm.") + } + } + } + } + + else -> throw GradleException( + """ + Unable to find launcher for Java '$it'. It needs to be: + 1. A valid path to a JDK home, or + 2. An environment variable named 'JAVA__HOME' or '' pointing to a JDK home, or + 3. A Java version or a known distribution+version combination (e.g. '11', 'zulu8', 'graalvm11', etc.) that can be resolved via Gradle toolchains. + 4. If using Gradle toolchains, ensure that the requested JDK is installed and configured correctly. + """.trimIndent() + ) + } + }.map { project.logger.info("testJvm home path: {}", it); it } + + /** + * The Java launcher for the test JVM. + * + * Current JVM or a launcher specified via the testJvm. + */ + val javaTestLauncher: Provider = + project.providers.zip(testJvmSpec, normalizedTestJvm) { jvmSpec, testJvm -> + // Only change test JVM if it's not the one we are running the gradle build with + if ((jvmSpec as? SpecificInstallationToolchainSpec)?.javaHome == currentJavaHomePath.get()) { + project.providers.provider { null } + } else { + // The provider always says that a value is present so we need to wrap it for proper error messages + project.javaToolchains.launcherFor(jvmSpec).orElse(project.providers.provider { + throw GradleException("Unable to find launcher for Java '$testJvm'. Does $TEST_JVM point to a JDK?") + }) + } + }.flatMap { it }.map { project.logger.info("testJvm launcher: {}", it.executablePath); it } + + private fun String.normalizeToJDKJavaHome(): Path { + val javaHome = project.file(this).toPath().toRealPath() + return if (javaHome.endsWith("jre")) javaHome.parent else javaHome + } + + private fun Path.toToolchainSpec(): JavaToolchainSpec = + // This is using internal APIs + SpecificInstallationToolchainSpec(project.serviceOf(), project.file(this)) + + private val Project.javaToolchains: JavaToolchainService + get() = + extensions.getByName("javaToolchains") as JavaToolchainService + + /** + * Discovers available Java versions via Gradle's internal JavaInstallationRegistry. + */ + private fun discoverJavaVersionsViaToolchains(): List? { + val registry = (project as ProjectInternal).services.get(JavaInstallationRegistry::class.java) + val versions = registry.toolchains().mapNotNull { installation -> + installation.metadata.languageVersion.majorVersion.toInt() + } + + return if (versions.isNotEmpty()) { + project.logger.info("Discovered Java versions via toolchains: {}", versions) + versions + } else { + null + } + } + + /** + * Discovers available Java versions via JAVA_X_HOME environment variables. + * Fallback method when toolchain discovery is not available. + */ + private fun discoverJavaVersionsViaEnvVars(): List { + val versions = project.providers.environmentVariablesPrefixedBy("JAVA_").map { javaHomes -> + javaHomes + .filter { it.key.matches(Regex("^JAVA_[0-9]+_HOME$")) } + .mapNotNull { Regex("^JAVA_(\\d+)_HOME$").find(it.key)?.groupValues?.get(1)?.toIntOrNull() } + }.get() + + if (versions.isNotEmpty()) { + project.logger.info("Discovered Java versions via JAVA_X_HOME env vars: {}", versions) + } + return versions + } +} diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/version/GitCommandValueSource.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/version/GitCommandValueSource.kt index dd60de6f162..acff64ae2bb 100644 --- a/buildSrc/src/main/kotlin/datadog/gradle/plugin/version/GitCommandValueSource.kt +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/version/GitCommandValueSource.kt @@ -46,7 +46,7 @@ abstract class GitCommandValueSource @Inject constructor( (exit code: ${result.exitValue}) Output: $output - """.trimIndent() + """.trimIndent() ) } } diff --git a/buildSrc/src/main/kotlin/datadog/gradle/plugin/version/TracerVersionPlugin.kt b/buildSrc/src/main/kotlin/datadog/gradle/plugin/version/TracerVersionPlugin.kt index 4e94c873457..1e6301b88ff 100644 --- a/buildSrc/src/main/kotlin/datadog/gradle/plugin/version/TracerVersionPlugin.kt +++ b/buildSrc/src/main/kotlin/datadog/gradle/plugin/version/TracerVersionPlugin.kt @@ -23,9 +23,13 @@ class TracerVersionPlugin @Inject constructor( val extension = targetProject.extensions.getByType(TracerVersionExtension::class.java) extension.detectDirty.set( - providerFactory.gradleProperty("tracerVersion.dirtiness") - .map { it.trim().toBoolean() } - .orElse(false) + providerFactory.gradleProperty("tracerVersion.dirtiness") + .map { it.trim().toBoolean() } + .orElse(false) + ) + + extension.versionQualifier.set( + providerFactory.gradleProperty("tracerVersion.qualifier") ) val versionProvider = versionProvider(targetProject, extension) @@ -54,6 +58,7 @@ class TracerVersionPlugin @Inject constructor( logger.info("Incrementing patch because release branch : $currentBranch") nextPatchVersion() } + else -> { logger.info("Incrementing minor") nextMinorVersion() @@ -104,7 +109,11 @@ class TracerVersionPlugin @Inject constructor( } } - private fun toTracerVersion(describeString: String, extension: TracerVersionExtension, nextVersion: Version.() -> Version): String { + private fun toTracerVersion( + describeString: String, + extension: TracerVersionExtension, + nextVersion: Version.() -> Version + ): String { logger.info("Git describe output: {}", describeString) val tagPrefix = extension.tagVersionPrefix.get() @@ -114,17 +123,26 @@ class TracerVersionPlugin @Inject constructor( val (lastTagVersion, describeTrailer) = matchResult.destructured val hasLaterCommits = describeTrailer.isNotBlank() - val version = Version.parse(lastTagVersion).let { - if (hasLaterCommits) { - it.nextVersion() - } else { - it + val version = if (describeString.contains("-ext")) { + // 如果包含 -ext，直接返回完整的 describeString，或者根据需求截取 + return describeString.replace(tagPrefix, "") + } else { + Version.parse(lastTagVersion).let { + if (hasLaterCommits) it.nextVersion() else it } } return buildString { append(version.toString()) + // Add optional version qualifier (e.g., "-ddprof") + if (extension.versionQualifier.isPresent) { + val qualifier = extension.versionQualifier.get() + if (qualifier.isNotBlank()) { + append("-").append(qualifier) + } + } + if (hasLaterCommits) { append(if (extension.useSnapshot.get()) "-SNAPSHOT" else describeTrailer) } @@ -143,5 +161,6 @@ class TracerVersionPlugin @Inject constructor( val useSnapshot = objectFactory.property(Boolean::class) .convention(true) val detectDirty = objectFactory.property(Boolean::class) + val versionQualifier = objectFactory.property(String::class) } } diff --git a/buildSrc/src/main/kotlin/dd-trace-java.ci-jobs.gradle.kts b/buildSrc/src/main/kotlin/dd-trace-java.ci-jobs.gradle.kts new file mode 100644 index 00000000000..4868f4f769b --- /dev/null +++ b/buildSrc/src/main/kotlin/dd-trace-java.ci-jobs.gradle.kts @@ -0,0 +1,132 @@ +import datadog.gradle.plugin.ci.isInSelectedSlot +import org.gradle.api.tasks.testing.Test +import java.io.File + +/* + * This plugin defines a set of tasks to be used in CI. + * + * These aggregate tasks support partitioning (to parallelize jobs) with + * `-Pslot=x/y`, and limiting tasks to those affected by git changes with + * `-PgitBaseRef`. + */ + +if (project != rootProject) { + logger.error("This plugin has been applied on a non-root project: ${project.path}") +} + +allprojects { + // Enable tests only on the selected slot (if -Pslot=n/t is provided) + tasks.withType().configureEach { + onlyIf("Project is in selected slot") { + project.isInSelectedSlot.get() + } + } +} + +fun relativeToGitRoot(f: File): File { + return rootProject.projectDir.toPath().relativize(f.absoluteFile.toPath()).toFile() +} + +fun getChangedFiles(baseRef: String, newRef: String): List { + val stdout = StringBuilder() + val stderr = StringBuilder() + + val proc = Runtime.getRuntime().exec(arrayOf("git", "diff", "--name-only", "$baseRef..$newRef")) + proc.inputStream.bufferedReader().use { stdout.append(it.readText()) } + proc.errorStream.bufferedReader().use { stderr.append(it.readText()) } + proc.waitFor() + require(proc.exitValue() == 0) { "git diff command failed, stderr: $stderr" } + + val out = stdout.toString().trim() + if (out.isEmpty()) { + return emptyList() + } + + logger.debug("git diff output: $out") + return out.split("\n").map { File(rootProject.projectDir, it.trim()) } +} + +// Initialize git change tracking +rootProject.extra.set("useGitChanges", false) + +val gitBaseRefProvider = rootProject.providers.gradleProperty("gitBaseRef") +if (gitBaseRefProvider.isPresent) { + val baseRef = gitBaseRefProvider.get() + val newRef = rootProject.providers.gradleProperty("gitNewRef").orElse("HEAD").get() + + val changedFiles = getChangedFiles(baseRef, newRef) + rootProject.extra.set("changedFiles", changedFiles) + rootProject.extra.set("useGitChanges", true) + + val ignoredFiles = fileTree(rootProject.projectDir) { + include(".gitignore", ".editorconfig") + include("*.md", "**/*.md") + include("gradlew", "gradlew.bat", "mvnw", "mvnw.cmd") + include("NOTICE") + include("static-analysis.datadog.yml") + } + + changedFiles.forEach { f -> + if (ignoredFiles.contains(f)) { + logger.warn("Ignoring changed file: ${relativeToGitRoot(f)}") + } + } + + val filteredChangedFiles = changedFiles.filter { !ignoredFiles.contains(it) } + rootProject.extra.set("changedFiles", filteredChangedFiles) + + val globalEffectFiles = fileTree(rootProject.projectDir) { + include(".gitlab/**") + include("build.gradle") + include("gradle/**") + } + + for (f in filteredChangedFiles) { + if (globalEffectFiles.contains(f)) { + logger.warn("Global effect change: ${relativeToGitRoot(f)} (no tasks will be skipped)") + rootProject.extra.set("useGitChanges", false) + break + } + } + + if (rootProject.extra.get("useGitChanges") as Boolean) { + logger.warn("Git change tracking is enabled: $baseRef..$newRef") + + val projects = subprojects.sortedByDescending { it.projectDir.path.length } + val affectedProjects = mutableMapOf>() + + // Path prefixes mapped to affected task names. A file not matching any of these prefixes will affect all tasks in + // the project ("all" can be used a task name to explicitly state the same). Only the first matching prefix is used. + val matchers = listOf( + mapOf("prefix" to "src/testFixtures/", "task" to "testFixturesClasses"), + mapOf("prefix" to "src/test/", "task" to "testClasses"), + mapOf("prefix" to "src/jmh/", "task" to "jmhCompileGeneratedClasses") + ) + + for (f in filteredChangedFiles) { + val p = projects.find { f.toString().startsWith(it.projectDir.path + "/") } + if (p == null) { + logger.warn("Changed file: ${relativeToGitRoot(f)} at root project (no task will be skipped)") + rootProject.extra.set("useGitChanges", false) + break + } + + // Make sure path separator is / + val relPath = p.projectDir.toPath().relativize(f.toPath()).joinToString("/") + val task = matchers.find { relPath.startsWith(it["prefix"]!!) }?.get("task") ?: "all" + logger.warn("Changed file: ${relativeToGitRoot(f)} in project ${p.path} ($task)") + affectedProjects.computeIfAbsent(p) { mutableSetOf() }.add(task) + } + + rootProject.extra.set("affectedProjects", affectedProjects) + } +} + +tasks.register("runMuzzle") { + val muzzleSubprojects = subprojects.filter { p -> + p.isInSelectedSlot.get() + && p.plugins.hasPlugin("java") + && p.plugins.hasPlugin("dd-trace-java.muzzle") + } + dependsOn(muzzleSubprojects.map { p -> "${p.path}:muzzle" }) +} diff --git a/buildSrc/src/main/kotlin/dd-trace-java.configure-tests.gradle.kts b/buildSrc/src/main/kotlin/dd-trace-java.configure-tests.gradle.kts new file mode 100644 index 00000000000..ad1d0a2d6ab --- /dev/null +++ b/buildSrc/src/main/kotlin/dd-trace-java.configure-tests.gradle.kts @@ -0,0 +1,119 @@ +import org.gradle.api.plugins.jvm.JvmTestSuite +import org.gradle.api.services.BuildService +import org.gradle.api.services.BuildServiceParameters +import org.gradle.api.tasks.testing.Test +import org.gradle.api.tasks.testing.junitplatform.JUnitPlatformOptions +import org.gradle.kotlin.dsl.develocity +import org.gradle.kotlin.dsl.findByType +import org.gradle.kotlin.dsl.withType +import org.gradle.testing.base.TestingExtension +import java.time.Duration +import java.time.temporal.ChronoUnit + +// Need concrete implementation of BuildService in Kotlin +abstract class ForkedTestLimit : BuildService +// Forked tests will fail with OOM if the memory is set too high. Gitlab allows at least a limit of 3. +val forkedTestsMemoryLimit = 3 + +val forkedTestLimit = gradle.sharedServices.registerIfAbsent("forkedTestLimit", ForkedTestLimit::class.java) { + maxParallelUsages.set(forkedTestsMemoryLimit) +} + +extensions.findByType()?.apply { + suites.withType().configureEach { + // Use JUnit 5 to run tests + useJUnitJupiter() + } +} + +// Use lazy providers to avoid evaluating the property until it is needed +val skipTestsProvider = rootProject.providers.gradleProperty("skipTests") +val skipForkedTestsProvider = rootProject.providers.gradleProperty("skipForkedTests") +val skipFlakyTestsProvider = rootProject.providers.gradleProperty("skipFlakyTests") +val runFlakyTestsProvider = rootProject.providers.gradleProperty("runFlakyTests") + +// Go through the Test tasks and configure them +tasks.withType().configureEach { + // Disable all tests if skipTests property was specified + onlyIf("skipTests are undefined or false") { !skipTestsProvider.isPresent } + + // Enable force rerun of tests with -Prerun.tests.${project.name} + outputs.upToDateWhen { + !rootProject.providers.gradleProperty("rerun.tests.${project.name}").isPresent + } + + // Trick to avoid on CI: "Couldn't flush user prefs: java.util.prefs.BackingStoreException: Couldn't get file lock." + // Use a task-specific user prefs directory + systemProperty("java.util.prefs.userRoot", layout.buildDirectory.dir("tmp/userPrefs/$name").get().asFile.absolutePath) + + // Enable JUnit 5 auto-detection so ConfigInversionExtension (STRICT mode) is loaded automatically + systemProperty("junit.jupiter.extensions.autodetection.enabled", "true") + + // Split up tests that want to run forked in their own separate JVM for generated tasks + if (name.startsWith("forkedTest") || name.endsWith("ForkedTest")) { + setExcludes(emptyList()) + setIncludes(listOf("**/*ForkedTest*")) + forkEvery = 1 + // Limit the number of concurrent forked tests + usesService(forkedTestLimit) + onlyIf("skipForkedTests are undefined or false") { !skipForkedTestsProvider.isPresent } + } else { + exclude("**/*ForkedTest*") + } + + // Set test timeout for 20 minutes. Default job timeout is 1h (configured on CI level). + timeout.set(Duration.of(20, ChronoUnit.MINUTES)) +} + +// Register a task "allTests" that depends on all non-latest and non-traceAgentTest Test tasks. +// This is used when we only want to run the 'main' test sets. +tasks.register("allTests") { + dependsOn(tasks.withType().matching { testTask -> + !testTask.name.contains("latest", ignoreCase = true) && testTask.name != "traceAgentTest" + }) +} + +// Register a task "allLatestDepTests" that depends on all Test tasks whose names include 'latest'. +// This is used when we want to run tests against the latest dependency versions. +tasks.register("allLatestDepTests") { + dependsOn(tasks.withType().matching { testTask -> + testTask.name.contains("latest", ignoreCase = true) + }) +} + +// Make the 'check' task depend on all Test tasks in the project. +// This means that when running the 'check' task, all Test tasks will run as well. +tasks.named("check") { + dependsOn(tasks.withType()) +} + +tasks.withType().configureEach { + // Flaky tests management for JUnit 5 + (options as? JUnitPlatformOptions)?.apply { + if (skipFlakyTestsProvider.isPresent) { + excludeTags("flaky") + } else if (runFlakyTestsProvider.isPresent) { + includeTags("flaky") + } + } + + // Set system property flag that is checked from tests to determine if they should be skipped or run + if (skipFlakyTestsProvider.isPresent) { + jvmArgs("-Drun.flaky.tests=false") + } else if (runFlakyTestsProvider.isPresent) { + jvmArgs("-Drun.flaky.tests=true") + } +} + +tasks.withType().configureEach { + // https://docs.gradle.com/develocity/flaky-test-detection/ + // https://docs.gradle.com/develocity/gradle-plugin/current/#test_retry + develocity.testRetry { + if (providers.environmentVariable("CI").isPresent()) { + maxRetries = 3 + filter { + excludeAnnotationClasses.add("*NonRetryable") // allow to mark classes non retryable + } + } + } +} diff --git a/buildSrc/src/main/kotlin/dd-trace-java.dependency-locking.gradle.kts b/buildSrc/src/main/kotlin/dd-trace-java.dependency-locking.gradle.kts new file mode 100644 index 00000000000..d1cc061d07e --- /dev/null +++ b/buildSrc/src/main/kotlin/dd-trace-java.dependency-locking.gradle.kts @@ -0,0 +1,34 @@ +/* + * This plugin enables dependency locking. + * + * The goal is to be able to later rebuild any version, by pinning floating versions. + * It will also help IDEs not having to re-index any latest library release. + * Pinned versions will be updated by the CI on a weekly basis. + * + * Pinned version can be updated using: ./gradlew resolveAndLockAll --write-locks + * + * See https://docs.gradle.org/current/userguide/dependency_locking.html + */ + +project.dependencyLocking { + lockAllConfigurations() + // lockmode set to LENIENT because there are resolution + // errors in the build with an apiguardian dependency. + // See: https://docs.gradle.org/current/userguide/dependency_locking.html for more info + lockMode = LockMode.LENIENT +} + +tasks.register("resolveAndLockAll") { + notCompatibleWithConfigurationCache("Filters configurations at execution time") + doFirst { + require(gradle.startParameter.isWriteDependencyLocks) + } + doLast { + configurations.filter { + // Add any custom filtering on the configurations to be resolved: + // - Should be resolvable + // - Should skip Scala related task (https://github.com/ben-manes/gradle-versions-plugin/issues/816#issuecomment-1872264880) + it.isCanBeResolved && !it.name.startsWith("incrementalScalaAnalysis") + }.forEach { it.resolve() } + } +} diff --git a/buildSrc/src/main/kotlin/dd-trace-java.gradle-debug.gradle.kts b/buildSrc/src/main/kotlin/dd-trace-java.gradle-debug.gradle.kts new file mode 100644 index 00000000000..2e50d9ee687 --- /dev/null +++ b/buildSrc/src/main/kotlin/dd-trace-java.gradle-debug.gradle.kts @@ -0,0 +1,105 @@ +/* + * Gradle debugging plugin for dd-trace-java builds. + */ + +val ddGradleDebugEnabled = project.hasProperty("ddGradleDebug") +val logPath = rootProject.layout.buildDirectory.file("datadog.gradle-debug.log") + +fun inferJdkFromJavaHome(javaHome: String?): String { + val effectiveJavaHome = javaHome ?: providers.environmentVariable("JAVA_HOME").orNull ?: error("JAVA_HOME is not set") + val javaExecutable = File(effectiveJavaHome, "bin/java").absolutePath + return try { + val process = ProcessBuilder(javaExecutable, "-version") + .redirectErrorStream(true) + .start() + val output = process.inputStream.bufferedReader().readText() + val versionLine = output.lines().firstOrNull() ?: "" + val versionMatch = Regex("version\\s+\"([0-9._]+)\"").find(versionLine) + versionMatch?.let { + val version = it.groupValues[1] + when { + version.startsWith("1.") -> version.substring(2, 3) + else -> version.split('.').first() + } + } ?: "unknown" + } catch (e: Exception) { + "error: ${e.message}" + } +} + +fun getJdkFromCompilerOptions(co: CompileOptions): String? { + if (co.isFork) { + val fo = co.forkOptions + val javaHome = fo.javaHome + if (javaHome != null) { + return inferJdkFromJavaHome(javaHome.toString()) + } + } + return null +} + +fun printJdkForProjectTasks(project: Project, logFile: File) { + project.tasks.forEach { task -> + val data = mutableMapOf() + data["task"] = task.path.toString() + if (task is JavaExec) { + val launcher = task.javaLauncher.get() + data["jdk"] = launcher.metadata.languageVersion.toString() + } else if (task is Javadoc) { + val tool = task.javadocTool.get() + data["jdk"] = tool.metadata.languageVersion.toString() + } else if (task is Test) { + val launcher = task.javaLauncher.get() + data["jdk"] = launcher.metadata.languageVersion.toString() + } else if (task is Exec) { + val javaHome = task.environment.get("JAVA_HOME")?.toString() + data["jdk"] = inferJdkFromJavaHome(javaHome) + } else if (task is JavaCompile) { + val compiler = task.javaCompiler.get() + data["jdk"] = compiler.metadata.languageVersion.toString() + val jdkFromJavaHome = getJdkFromCompilerOptions(task.options) + if (jdkFromJavaHome != null && jdkFromJavaHome != data["jdk"]) { + data["java_home"] = jdkFromJavaHome + } + } else if (task is GroovyCompile) { + val launcher = task.javaLauncher.get() + data["jdk"] = launcher.metadata.languageVersion.toString() + val jdkFromJavaHome = getJdkFromCompilerOptions(task.options) + if (jdkFromJavaHome != null && jdkFromJavaHome != data["jdk"]) { + data["java_home"] = jdkFromJavaHome + } + } else if (task is ScalaCompile) { + val launcher = task.javaLauncher.get() + data["jdk"] = launcher.metadata.languageVersion.toString() + val jdkFromJavaHome = getJdkFromCompilerOptions(task.options) + if (jdkFromJavaHome != null && jdkFromJavaHome != data["jdk"]) { + data["java_home"] = jdkFromJavaHome + } + } else { + data["jdk"] = "unknown" + } + val json = data.entries.joinToString(prefix = "{", postfix = "}") { (k, v) -> "\"$k\":\"$v\"" } + logFile.appendText("$json\n") + } +} + +class DebugBuildListener : org.gradle.BuildListener { + override fun settingsEvaluated(settings: Settings) = Unit + + override fun projectsLoaded(gradle: Gradle) = Unit + + override fun buildFinished(result: BuildResult) = Unit + + override fun projectsEvaluated(gradle: Gradle) { + val logFile = logPath.get().asFile + logFile.writeText("") + gradle.rootProject.allprojects.forEach { project -> + printJdkForProjectTasks(project, logFile) + } + } +} + +if (ddGradleDebugEnabled) { + logger.lifecycle("datadog.gradle-debug plugin is enabled") + gradle.addListener(DebugBuildListener()) +} diff --git a/buildSrc/src/main/kotlin/dd-trace-java.instrumentation.testing-framework-tests.gradle.kts b/buildSrc/src/main/kotlin/dd-trace-java.instrumentation.testing-framework-tests.gradle.kts new file mode 100644 index 00000000000..0f2af6fb44a --- /dev/null +++ b/buildSrc/src/main/kotlin/dd-trace-java.instrumentation.testing-framework-tests.gradle.kts @@ -0,0 +1,23 @@ +plugins { + java +} + +logger.info("Avoid executing classes used to test testing frameworks instrumentation") + +tasks.withType().configureEach { + exclude("**/TestAssumption*", "**/TestSuiteSetUpAssumption*") + exclude("**/TestDisableTestTrace*") + exclude("**/TestError*") + exclude("**/TestFactory*") + exclude("**/TestFailed*") + exclude("**/TestFailedWithSuccessPercentage*") + exclude("**/TestInheritance*", "**/BaseTestInheritance*") + exclude("**/TestParameterized*") + exclude("**/TestRepeated*") + exclude("**/TestSkipped*") + exclude("**/TestSkippedClass*") + exclude("**/TestSucceed*") + exclude("**/TestTemplate*") + exclude("**/TestUnskippable*") + exclude("**/TestWithSetup*") +} diff --git a/buildSrc/src/main/kotlin/dd-trace-java.profiling-ddprof-override.gradle.kts b/buildSrc/src/main/kotlin/dd-trace-java.profiling-ddprof-override.gradle.kts new file mode 100644 index 00000000000..8fe48f9dc41 --- /dev/null +++ b/buildSrc/src/main/kotlin/dd-trace-java.profiling-ddprof-override.gradle.kts @@ -0,0 +1,24 @@ +/** + * Convention plugin for overriding ddprof dependency version with snapshot. + * + * When the root project has the property 'ddprofUseSnapshot' set, this plugin: + * 1. Reads the calculated snapshot version from root project + * 2. Overrides all ddprof dependencies to use the snapshot version + * + * Apply this plugin only to projects that depend on ddprof. + */ + +if (rootProject.hasProperty("ddprofUseSnapshot")) { + val ddprofSnapshotVersion = rootProject.property("ddprofSnapshotVersion").toString() + + configurations.all { + resolutionStrategy.eachDependency { + if (requested.group == "com.datadoghq" && requested.name == "ddprof") { + useVersion(ddprofSnapshotVersion) + because("Using ddprof snapshot version for integration testing") + } + } + } + + logger.lifecycle("${project.name}: Configured to use ddprof SNAPSHOT version $ddprofSnapshotVersion") +} diff --git a/buildSrc/src/main/kotlin/dd-trace-java.test-jvm-constraints.gradle.kts b/buildSrc/src/main/kotlin/dd-trace-java.test-jvm-constraints.gradle.kts new file mode 100644 index 00000000000..aa4724183b0 --- /dev/null +++ b/buildSrc/src/main/kotlin/dd-trace-java.test-jvm-constraints.gradle.kts @@ -0,0 +1,128 @@ +import datadog.gradle.plugin.testJvmConstraints.ProvideJvmArgsOnJvmLauncherVersion +import datadog.gradle.plugin.testJvmConstraints.TestJvmConstraintsExtension +import datadog.gradle.plugin.testJvmConstraints.TestJvmConstraintsExtension.Companion.TEST_JVM_CONSTRAINTS +import datadog.gradle.plugin.testJvmConstraints.TestJvmSpec +import datadog.gradle.plugin.testJvmConstraints.isJavaVersionAllowed +import datadog.gradle.plugin.testJvmConstraints.isTestJvmAllowed + +plugins { + java +} + +val projectExtension = extensions.create(TEST_JVM_CONSTRAINTS) + +val testJvmSpec = TestJvmSpec(project) + +tasks.withType().configureEach { + if (extensions.findByName(TEST_JVM_CONSTRAINTS) != null) { + return@configureEach + } + + inputs.property("testJvm", testJvmSpec.testJvmProperty).optional(true) + + val taskExtension = project.objects.newInstance().also { + configureConventions(it, projectExtension) + } + + inputs.property("$TEST_JVM_CONSTRAINTS.allowReflectiveAccessToJdk", taskExtension.allowReflectiveAccessToJdk).optional(true) + inputs.property("$TEST_JVM_CONSTRAINTS.excludeJdk", taskExtension.excludeJdk) + inputs.property("$TEST_JVM_CONSTRAINTS.includeJdk", taskExtension.includeJdk) + inputs.property("$TEST_JVM_CONSTRAINTS.forceJdk", taskExtension.forceJdk) + inputs.property("$TEST_JVM_CONSTRAINTS.minJavaVersion", taskExtension.minJavaVersion).optional(true) + inputs.property("$TEST_JVM_CONSTRAINTS.maxJavaVersion", taskExtension.maxJavaVersion).optional(true) + + extensions.add(TEST_JVM_CONSTRAINTS, taskExtension) + + configureTestJvm(taskExtension) +} + +/** + * Provide arguments if condition is met. + */ +fun Test.conditionalJvmArgs( + applyFromVersion: JavaVersion, + jvmArgsToApply: List, + additionalCondition: Provider = project.providers.provider { true } +) { + jvmArgumentProviders.add( + ProvideJvmArgsOnJvmLauncherVersion( + this, + applyFromVersion, + jvmArgsToApply, + additionalCondition + ) + ) +} + +/** + * Configure the jvm launcher of the test task and ensure the test task + * can be run with the test task launcher. + */ +private fun Test.configureTestJvm(extension: TestJvmConstraintsExtension) { + if (testJvmSpec.javaTestLauncher.isPresent) { + javaLauncher = testJvmSpec.javaTestLauncher + onlyIf("Allowed or forced JDK") { + extension.isTestJvmAllowed(testJvmSpec) + } + } else { + onlyIf("Is current Daemon JVM allowed") { + extension.isJavaVersionAllowed(JavaVersion.current()) + } + } + + // temporary workaround when using Java16+: some tests require reflective access to java.lang/java.util + conditionalJvmArgs( + JavaVersion.VERSION_16, + listOf( + "--add-opens=java.base/java.lang=ALL-UNNAMED", + "--add-opens=java.base/java.util=ALL-UNNAMED" + ), + extension.allowReflectiveAccessToJdk + ) +} + +// Jacoco plugin is not applied on every project +pluginManager.withPlugin("org.gradle.jacoco") { + tasks.withType().configureEach { + // Disable jacoco for additional 'testJvm' tests to speed things up a bit + if (testJvmSpec.javaTestLauncher.isPresent) { + extensions.configure { + isEnabled = false + } + } + } +} + +/** + * Configures the convention, this tells Gradle where to look for values. + * + * Currently, the extension is still configured to look at project's _extra_ properties. + */ +private fun Test.configureConventions( + taskExtension: TestJvmConstraintsExtension, + projectExtension: TestJvmConstraintsExtension +) { + taskExtension.minJavaVersion.convention(projectExtension.minJavaVersion + .orElse(providers.provider { project.findProperty("${name}MinJavaVersionForTests") as? JavaVersion }) + .orElse(providers.provider { project.findProperty("minJavaVersion") as? JavaVersion }) + ) + taskExtension.maxJavaVersion.convention(projectExtension.maxJavaVersion + .orElse(providers.provider { project.findProperty("${name}MaxJavaVersionForTests") as? JavaVersion }) + .orElse(providers.provider { project.findProperty("maxJavaVersion") as? JavaVersion }) + ) + taskExtension.forceJdk.convention(projectExtension.forceJdk + .orElse(providers.provider { + @Suppress("UNCHECKED_CAST") + project.findProperty("forceJdk") as? List ?: emptyList() + }) + ) + taskExtension.excludeJdk.convention(projectExtension.excludeJdk + .orElse(providers.provider { + @Suppress("UNCHECKED_CAST") + project.findProperty("excludeJdk") as? List ?: emptyList() + }) + ) + taskExtension.allowReflectiveAccessToJdk.convention(projectExtension.allowReflectiveAccessToJdk + .orElse(providers.provider { project.findProperty("allowReflectiveAccessToJdk") as? Boolean }) + ) +} diff --git a/buildSrc/src/test/groovy/CallSiteInstrumentationPluginTest.groovy b/buildSrc/src/test/groovy/CallSiteInstrumentationPluginTest.groovy deleted file mode 100644 index f3453017a0c..00000000000 --- a/buildSrc/src/test/groovy/CallSiteInstrumentationPluginTest.groovy +++ /dev/null @@ -1,138 +0,0 @@ -import org.gradle.testkit.runner.BuildResult -import org.gradle.testkit.runner.GradleRunner -import org.gradle.testkit.runner.UnexpectedBuildFailure -import spock.lang.Specification -import spock.lang.TempDir - -class CallSiteInstrumentationPluginTest extends Specification { - - def buildGradle = ''' - plugins { - id 'java' - id 'call-site-instrumentation' - id 'com.diffplug.spotless' version '6.13.0' - } - - sourceCompatibility = JavaVersion.VERSION_1_8 - targetCompatibility = JavaVersion.VERSION_1_8 - - csi { - suffix = 'CallSite' - targetFolder = 'csi' - rootFolder = file('$$ROOT_FOLDER$$') - } - - repositories { - mavenCentral() - } - - dependencies { - implementation group: 'net.bytebuddy', name: 'byte-buddy', version: '1.17.7' - implementation group: 'com.google.auto.service', name: 'auto-service-annotations', version: '1.1.1' - } - ''' - - @TempDir - File buildDir - - def 'test call site instrumentation plugin'() { - setup: - createGradleProject(buildDir, buildGradle, ''' - import datadog.trace.agent.tooling.csi.*; - - @CallSite(spi = CallSites.class) - public class BeforeAdviceCallSite { - @CallSite.Before("java.lang.StringBuilder java.lang.StringBuilder.append(java.lang.String)") - public static void beforeAppend(@CallSite.This final StringBuilder self, @CallSite.Argument final String param) { - } - } - ''') - - when: - final result = buildGradleProject(buildDir) - - then: - final generated = resolve(buildDir, 'build', 'csi', 'BeforeAdviceCallSites.java') - generated.exists() - - final output = result.output - !output.contains('[⨉]') - output.contains('[✓] @CallSite BeforeAdviceCallSite') - } - - def 'test call site instrumentation plugin with error'() { - setup: - createGradleProject(buildDir, buildGradle, ''' - import datadog.trace.agent.tooling.csi.*; - - @CallSite(spi = CallSites.class) - public class BeforeAdviceCallSite { - @CallSite.Before("java.lang.StringBuilder java.lang.StringBuilder.append(java.lang.String)") - private void beforeAppend(@CallSite.This final StringBuilder self, @CallSite.Argument final String param) { - } - } - ''') - - when: - buildGradleProject(buildDir) - - then: - final error = thrown(UnexpectedBuildFailure) - - final generated = resolve(buildDir, 'build', 'csi', 'BeforeAdviceCallSites.java') - !generated.exists() - - final output = error.message - !output.contains('[✓]') - output.contains('ADVICE_METHOD_NOT_STATIC_AND_PUBLIC') - } - - private static void createGradleProject(final File buildDir, final String gradleFile, final String advice) { - final projectFolder = new File(System.getProperty('user.dir')).parentFile - final callSiteJar = resolve(projectFolder, 'buildSrc', 'call-site-instrumentation-plugin') - final gradleFileContent = gradleFile.replace('$$ROOT_FOLDER$$', projectFolder.toString().replace("\\","\\\\")) - - final buildGradle = resolve(buildDir, 'build.gradle') - buildGradle.text = gradleFileContent - - final javaFolder = resolve(buildDir, 'src', 'main', 'java') - javaFolder.mkdirs() - - final advicePackage = parsePackage(advice) - final adviceClassName = parseClassName(advice) - final adviceFolder = resolve(javaFolder, advicePackage.split('\\.')) - adviceFolder.mkdirs() - final adviceFile = resolve(adviceFolder, "${adviceClassName}.java") - adviceFile.text = advice - - final csiSource = resolve(projectFolder, 'dd-java-agent', 'agent-tooling', 'src', 'main', 'java', 'datadog', 'trace', 'agent', 'tooling', 'csi') - final csiTarget = resolve(javaFolder, 'datadog', 'trace', 'agent', 'tooling', 'csi') - csiTarget.mkdirs() - csiSource.listFiles().each { new File(csiTarget, it.name).text = it.text } - } - - private static BuildResult buildGradleProject(final File buildDir) { - return GradleRunner.create() - .withTestKitDir(new File(buildDir, '.gradle-test-kit')) // workaround in case the global test-kit cache becomes corrupted - .withDebug(true) // avoids starting daemon which can leave undeleted files post-cleanup - .withProjectDir(buildDir) - .withArguments('build', '--info', '--stacktrace') - .withPluginClasspath() - .forwardOutput() - .build() - } - - private static String parsePackage(final String advice) { - final advicePackageMatcher = advice =~ /(?s).*package\s+([\w\.]+)\s*;/ - return advicePackageMatcher ? advicePackageMatcher[0][1] as String : '' - } - - private static String parseClassName(final String advice) { - return (advice =~ /(?s).*class\s+(\w+)\s+\{\.*/)[0][1] - } - - private static File resolve(final File file, final String...path) { - final result = path.inject(file.toPath()) {parent, folder -> parent.resolve(folder)} - return result.toFile() - } -} diff --git a/buildSrc/src/test/groovy/InstrumentPluginTest.groovy b/buildSrc/src/test/groovy/InstrumentPluginTest.groovy deleted file mode 100644 index 8e5b84ae7d6..00000000000 --- a/buildSrc/src/test/groovy/InstrumentPluginTest.groovy +++ /dev/null @@ -1,121 +0,0 @@ -import net.bytebuddy.utility.OpenedClassReader -import org.gradle.testkit.runner.BuildResult -import org.gradle.testkit.runner.GradleRunner -import org.objectweb.asm.ClassReader -import org.objectweb.asm.ClassVisitor -import org.objectweb.asm.FieldVisitor -import spock.lang.Specification -import spock.lang.TempDir - -class InstrumentPluginTest extends Specification { - - def buildGradle = ''' - plugins { - id 'java' - id 'instrument' - } - - sourceCompatibility = JavaVersion.VERSION_1_8 - targetCompatibility = JavaVersion.VERSION_1_8 - - repositories { - mavenCentral() - } - - dependencies { - compileOnly group: 'net.bytebuddy', name: 'byte-buddy', version: '1.17.7' // just to build TestPlugin - } - - apply plugin: 'instrument' - - instrument.plugins = [ - 'TestPlugin' - ] - ''' - - def testPlugin = ''' - import java.io.File; - import java.io.IOException; - import net.bytebuddy.build.Plugin; - import net.bytebuddy.description.type.TypeDescription; - import net.bytebuddy.dynamic.ClassFileLocator; - import net.bytebuddy.dynamic.DynamicType; - - public class TestPlugin implements Plugin { - private final File targetDir; - - public TestPlugin(File targetDir) { - this.targetDir = targetDir; - } - - @Override - public boolean matches(TypeDescription target) { - return "ExampleCode".equals(target.getSimpleName()); - } - - @Override - public DynamicType.Builder apply( - DynamicType.Builder builder, - TypeDescription typeDescription, - ClassFileLocator classFileLocator) { - return builder.defineField("__TEST__FIELD__", Void.class); - } - - @Override - public void close() throws IOException { - // no-op - } - } - ''' - - def exampleCode = ''' - package example; public class ExampleCode {} - ''' - - @TempDir - File buildDir - - def 'test instrument plugin'() { - setup: - def tree = new FileTreeBuilder(buildDir) - tree.'build.gradle'(buildGradle) - tree.src { - main { - java { - 'TestPlugin.java'(testPlugin) - example { - 'ExampleCode.java'(exampleCode) - } - } - } - } - - when: - BuildResult result = GradleRunner.create() - .withTestKitDir(new File(buildDir, '.gradle-test-kit')) // workaround in case the global test-kit cache becomes corrupted - .withDebug(true) // avoids starting daemon which can leave undeleted files post-cleanup - .withProjectDir(buildDir) - .withArguments('build', '--stacktrace') - .withPluginClasspath() - .forwardOutput() - .build() - - File classFile = new File(buildDir, 'build/classes/java/main/example/ExampleCode.class') - - then: - assert classFile.isFile() - - boolean foundInsertedField = false - new ClassReader(new FileInputStream(classFile)).accept(new ClassVisitor(OpenedClassReader.ASM_API) { - @Override - FieldVisitor visitField(int access, String fieldName, String descriptor, String signature, Object value) { - if ('__TEST__FIELD__' == fieldName) { - foundInsertedField = true - } - return null - } - }, OpenedClassReader.ASM_API) - - assert foundInsertedField - } -} diff --git a/buildSrc/src/test/groovy/datadog/gradle/plugin/muzzle/RangeQueryTest.groovy b/buildSrc/src/test/groovy/datadog/gradle/plugin/muzzle/RangeQueryTest.groovy deleted file mode 100644 index d85aa3e5e1a..00000000000 --- a/buildSrc/src/test/groovy/datadog/gradle/plugin/muzzle/RangeQueryTest.groovy +++ /dev/null @@ -1,30 +0,0 @@ -package datadog.gradle.plugin.muzzle - -import org.eclipse.aether.RepositorySystem -import org.eclipse.aether.RepositorySystemSession -import org.eclipse.aether.artifact.Artifact -import org.eclipse.aether.artifact.DefaultArtifact -import org.eclipse.aether.resolution.VersionRangeRequest -import org.eclipse.aether.resolution.VersionRangeResult -import spock.lang.Specification - -class RangeQueryTest extends Specification { - - RepositorySystem system = MuzzleMavenRepoUtils.newRepositorySystem() - RepositorySystemSession session = MuzzleMavenRepoUtils.newRepositorySystemSession(system) - - def "test range request"() { - setup: -// compile group: 'org.codehaus.groovy', name: 'groovy-all', version: '2.5.0', ext: 'pom' - final Artifact directiveArtifact = new DefaultArtifact("org.codehaus.groovy", "groovy-all", "jar", "[2.5.0,2.5.8)") - final VersionRangeRequest rangeRequest = new VersionRangeRequest() - rangeRequest.setRepositories(MuzzleMavenRepoUtils.MUZZLE_REPOS) - rangeRequest.setArtifact(directiveArtifact) - - // This call makes an actual network request, which may fail if network access is limited. - final VersionRangeResult rangeResult = system.resolveVersionRange(session, rangeRequest) - - expect: - rangeResult.versions.size() >= 8 - } -} diff --git a/buildSrc/src/test/groovy/datadog/gradle/plugin/muzzle/VersionSetTest.groovy b/buildSrc/src/test/groovy/datadog/gradle/plugin/muzzle/VersionSetTest.groovy deleted file mode 100644 index 57877d9a964..00000000000 --- a/buildSrc/src/test/groovy/datadog/gradle/plugin/muzzle/VersionSetTest.groovy +++ /dev/null @@ -1,91 +0,0 @@ -package datadog.gradle.plugin.muzzle - - -import org.eclipse.aether.version.Version -import spock.lang.Specification - -class VersionSetTest extends Specification { - - def "parse versions properly"() { - when: - def parsed = new VersionSet.ParsedVersion(version) - - then: - parsed.versionNumber == versionNumber - parsed.ending == ending - parsed.majorMinor == versionNumber >> VersionSet.ParsedVersion.VERSION_SHIFT - - where: - version | versionNumber | ending - ver('1.2.3') | num(1, 2, 3) | "" - ver('4.5.6-foo') | num(4, 5, 6) | "foo" - ver('7.8.9.foo') | num(7, 8, 9) | "foo" - ver('10.11.12.foo-bar') | num(10, 11, 12) | "foo-bar" - ver('13.14.foo-bar') | num(13, 14, 0) | "foo-bar" - ver('15.foo') | num(15, 0, 0) | "foo" - ver('16-foo') | num(16, 0, 0) | "foo" - } - - def "select low and high from major.minor"() { - when: - def versionSet = new VersionSet(versions) - - then: - versionSet.lowAndHighForMajorMinor == expected - - where: - versions << [[ - ver('4.5.6'), - ver('1.2.3') - ], [ - ver('1.2.3'), - ver('1.2.1'), - ver('1.3.0'), - ver('1.2.7'), - ver('1.4.17'), - ver('1.4.1'), - ver('1.4.0'), - ver('1.4.10') - ]] - expected << [[ - ver('1.2.3'), - ver('4.5.6') - ], [ - ver('1.2.1'), - ver('1.2.7'), - ver('1.3.0'), - ver('1.4.0'), - ver('1.4.17') - ]] - } - - Version ver(String string) { - return new TestVersion(string) - } - - long num(int major, int minor, int micro) { - long result = major - return (((result << 12) + minor) << 12) + micro - } - - static class TestVersion implements Version { - private final String string - - TestVersion(String versionString) { - this.string = versionString - } - - @Override - int compareTo(Version o) { - if (o == null) return 1 - if (! o instanceof TestVersion) return 1 - TestVersion other = o as TestVersion - return string <=> other.string - } - - @Override - String toString() { - return string - } - } -} diff --git a/buildSrc/src/test/kotlin/datadog/gradle/plugin/GradleFixture.kt b/buildSrc/src/test/kotlin/datadog/gradle/plugin/GradleFixture.kt new file mode 100644 index 00000000000..5d6ca45b6fd --- /dev/null +++ b/buildSrc/src/test/kotlin/datadog/gradle/plugin/GradleFixture.kt @@ -0,0 +1,158 @@ +package datadog.gradle.plugin + +import org.gradle.testkit.runner.BuildResult +import org.gradle.testkit.runner.GradleRunner +import org.gradle.testkit.runner.UnexpectedBuildResultException +import org.intellij.lang.annotations.Language +import org.w3c.dom.Document +import java.io.File +import java.nio.file.Files +import javax.xml.parsers.DocumentBuilderFactory + +/** + * Base fixture for Gradle plugin integration tests. + * Provides common functionality for setting up test projects and running Gradle builds. + */ +internal open class GradleFixture(protected val projectDir: File) { + // Each fixture gets its own testkit dir in the system temp directory (NOT under + // projectDir) so that JUnit's @TempDir cleanup doesn't race with daemon file locks. + // See https://github.com/gradle/gradle/issues/12535 + // A fresh daemon is started per test — ensuring withEnvironment() vars (e.g. + // MAVEN_REPOSITORY_PROXY) are correctly set on the daemon JVM and not inherited + // from a previously-started daemon with a different test's environment. + // A JVM shutdown hook removes the directory after all tests have run (and daemons + // have been stopped), so file locks are guaranteed to be released by then. + private val testKitDir: File by lazy { + Files.createTempDirectory("gradle-testkit-").toFile().also { dir -> + Runtime.getRuntime().addShutdownHook(Thread { dir.deleteRecursively() }) + } + } + + /** + * Runs Gradle with the specified arguments. + * + * After the build completes, any Gradle daemons started by TestKit are killed + * so their file locks on the testkit cache are released before JUnit `@TempDir` + * cleanup. See https://github.com/gradle/gradle/issues/12535 + * + * @param args Gradle task names and arguments + * @param expectFailure Whether the build is expected to fail + * @param env Environment variables to set (merged with system environment) + * @return The build result + */ + fun run(vararg args: String, expectFailure: Boolean = false, env: Map = emptyMap()): BuildResult { + val runner = GradleRunner.create() + .withTestKitDir(testKitDir) + .withPluginClasspath() + .withProjectDir(projectDir) + // Using withDebug prevents starting a daemon, but it doesn't work with withEnvironment + .withEnvironment(System.getenv() + env) + .withArguments(*args) + return try { + if (expectFailure) runner.buildAndFail() else runner.build() + } catch (e: UnexpectedBuildResultException) { + e.buildResult + } finally { + stopDaemons() + } + } + + /** + * Kills Gradle daemons started by TestKit for this fixture's testkit dir. + * + * The Gradle Tooling API (used by [GradleRunner]) always spawns a daemon and + * provides no public API to stop it (https://github.com/gradle/gradle/issues/12535). + * We replicate the strategy Gradle uses in its own integration tests + * ([DaemonLogsAnalyzer.killAll()][1]): + * + * 1. Scan `/daemon//` for log files matching + * `DaemonLogConstants.DAEMON_LOG_PREFIX + pid + DaemonLogConstants.DAEMON_LOG_SUFFIX`, + * i.e. `daemon-.out.log`. + * 2. Extract the PID from the filename and kill the process. + * + * Trade-offs of the PID-from-filename approach: + * - **PID recycling**: between the build finishing and `kill` being sent, the OS + * could theoretically recycle the PID. In practice the window is short + * (the `finally` block runs immediately after the build) so the risk is negligible. + * - **Filename convention is internal**: Gradle's `DaemonLogConstants.DAEMON_LOG_PREFIX` + * (`"daemon-"`) / `DAEMON_LOG_SUFFIX` (`".out.log"`) are not public API; a future + * Gradle version could change them. The `toLongOrNull()` guard safely skips entries + * that don't parse as a PID (including the UUID fallback Gradle uses when the PID + * is unavailable). + * - **Java 8 compatible**: uses `kill`/`taskkill` via [ProcessBuilder] instead of + * `ProcessHandle` (Java 9+) because build logic targets JVM 1.8. + * + * [1]: https://github.com/gradle/gradle/blob/43b381d88/testing/internal-distribution-testing/src/main/groovy/org/gradle/integtests/fixtures/daemon/DaemonLogsAnalyzer.groovy + */ + private fun stopDaemons() { + val daemonDir = File(testKitDir, "daemon") + if (!daemonDir.exists()) return + + daemonDir.walkTopDown() + .filter { it.isFile && it.name.endsWith(".out.log") && !it.name.startsWith("hs_err") } + .forEach { logFile -> + val pid = logFile.nameWithoutExtension // daemon-12345.out + .removeSuffix(".out") // daemon-12345 + .removePrefix("daemon-") // 12345 + .toLongOrNull() ?: return@forEach // skip UUIDs / unparseable names + + val isWindows = System.getProperty("os.name").lowercase().contains("win") + val killProcess = if (isWindows) { + ProcessBuilder("taskkill", "/F", "/PID", pid.toString()) + } else { + ProcessBuilder("kill", pid.toString()) + } + try { + val process = killProcess.redirectErrorStream(true).start() + process.waitFor(5, java.util.concurrent.TimeUnit.SECONDS) + } catch (_: Exception) { + // best effort — daemon may already be stopped + } + } + } + + /** + * Adds a subproject to the build. + * Updates settings.gradle and creates the build script for the subproject. + * + * @param projectPath The project path (e.g., "dd-java-agent:instrumentation:other") + * @param buildScript The build script content for the subproject + */ + fun addSubproject(projectPath: String, @Language("Groovy") buildScript: String) { + // Add to settings.gradle + val settingsFile = file("settings.gradle") + if (settingsFile.exists()) { + settingsFile.appendText("\ninclude ':$projectPath'") + } else { + settingsFile.writeText("include ':$projectPath'") + } + + file("${projectPath.replace(':', '/')}/build.gradle") + .writeText(buildScript.trimIndent()) + } + + /** + * Writes the root project's build.gradle file. + * + * @param buildScript The build script content for the root project + */ + fun writeRootProject(@Language("Groovy") buildScript: String) { + file("build.gradle").writeText(buildScript.trimIndent()) + } + + /** + * Parses an XML file into a DOM Document. + */ + fun parseXml(xmlFile: File): Document { + val builder = DocumentBuilderFactory.newInstance().newDocumentBuilder() + return builder.parse(xmlFile) + } + + /** + * Creates or gets a file in the project directory, ensuring parent directories exist. + */ + protected fun file(path: String): File = + File(projectDir, path).also { file -> + file.parentFile?.mkdirs() + } +} diff --git a/buildSrc/src/test/kotlin/datadog/gradle/plugin/MavenRepoFixture.kt b/buildSrc/src/test/kotlin/datadog/gradle/plugin/MavenRepoFixture.kt new file mode 100644 index 00000000000..02216f1c7fb --- /dev/null +++ b/buildSrc/src/test/kotlin/datadog/gradle/plugin/MavenRepoFixture.kt @@ -0,0 +1,191 @@ +package datadog.gradle.plugin + +import java.io.File +import java.nio.file.Path +import java.security.MessageDigest +import java.util.jar.JarOutputStream + +/** + * Test fixture for creating and managing fake Maven repositories. + * Provides utilities to create Maven artifacts with proper structure and metadata. + * + * The fake Maven repository is automatically created in the constructor. + */ +class MavenRepoFixture(projectDir: File) { + + /** The root directory of the fake Maven repository */ + val repoDir: File = File(projectDir, "fake-maven-repo").apply { mkdirs() } + + /** + * Gets the repository URL for use in Gradle configuration. + */ + val repoUrl: String + get() = repoDir.toURI().toString() + + /** + * Publishes versions to the fake Maven repository for the specified module. + * If the module already exists, adds the new versions to the existing ones. + * Creates the module directory if it doesn't exist. + * + * @param group Maven group ID + * @param module Maven artifact ID + * @param versions List of versions to publish (will be merged with existing versions) + * @param jarContentBuilder Optional lambda to add entries to the JAR + */ + fun publishVersions( + group: String, + module: String, + versions: List, + jarContentBuilder: ((JarOutputStream) -> Unit)? = null + ) { + require(versions.isNotEmpty()) { "versions must not be empty" } + val groupPath = group.replace('.', '/') + val moduleDir = File(repoDir, "$groupPath/$module").apply { mkdirs() } + + // Create all version artifacts + versions.forEach { version -> + createMavenVersion(moduleDir, group, module, version, jarContentBuilder) + } + + // Read existing versions from metadata and merge with new versions + val metadataFile = File(moduleDir, "maven-metadata.xml") + val existingVersions = if (metadataFile.exists()) { + val content = metadataFile.readText() + val versionRegex = "([^<]+)".toRegex() + versionRegex.findAll(content).map { it.groupValues[1] }.toList() + } else { + emptyList() + } + + // Merge and sort all versions + val allVersions = (existingVersions + versions).distinct().sorted() + writeMavenMetadata(metadataFile, group, module, allVersions) + } + + /** + * Creates a single Maven version with POM and JAR artifacts (including checksums). + * + * @param moduleDir The module directory (e.g., repo/com/example/artifact) + * @param group Maven group ID + * @param module Maven artifact ID + * @param version Version to create + * @param jarContentBuilder Optional lambda to add entries to the JAR + */ + private fun createMavenVersion( + moduleDir: File, + group: String, + module: String, + version: String, + jarContentBuilder: ((JarOutputStream) -> Unit)? = null + ) { + val versionDir = File(moduleDir, version).apply { mkdirs() } + + val pomFile = File(versionDir, "$module-$version.pom") + pomFile.writeText( + """ + + 4.0.0 + $group + $module + $version + jar + + """.trimIndent() + ) + writeChecksum(pomFile) + + // Create JAR file + val jarFile = File(versionDir, "$module-$version.jar") + createJar(jarFile.toPath(), group, module, version, jarContentBuilder) + writeChecksum(jarFile) + } + + /** + * Writes maven-metadata.xml for a module with the given versions. + * + * @param metadataFile The maven-metadata.xml file to write + * @param group Maven group ID + * @param module Maven artifact ID + * @param versions List of versions (should be sorted) + */ + private fun writeMavenMetadata( + metadataFile: File, + group: String, + module: String, + versions: List + ) { + metadataFile.writeText( + """ + + $group + $module + + ${versions.last()} + ${versions.last()} + + ${versions.joinToString("\n") { " $it" }} + + ${System.currentTimeMillis() / 1000} + + + """.trimIndent() + ) + writeChecksum(metadataFile) + } + + /** + * Generates SHA-1 and MD5 checksum files for a given file. + */ + private fun writeChecksum(file: File) { + val content = file.readBytes() + val sha1 = MessageDigest.getInstance("SHA-1").digest(content) + .joinToString("") { "%02x".format(it) } + File(file.parentFile, "${file.name}.sha1").writeText(sha1) + + val md5 = MessageDigest.getInstance("MD5").digest(content) + .joinToString("") { "%02x".format(it) } + File(file.parentFile, "${file.name}.md5").writeText(md5) + } + + /** + * Creates a JAR file at the specified path with standard Maven metadata, optionally with custom content. + * + * @param path Path where the JAR should be created + * @param group Maven group ID + * @param module Maven artifact ID + * @param version Maven version + * @param contentBuilder Optional lambda to add custom entries to the JAR + */ + private fun createJar( + path: Path, + group: String, + module: String, + version: String, + contentBuilder: ((JarOutputStream) -> Unit)? = null + ) { + JarOutputStream(path.toFile().outputStream()).use { jos -> + // Add Maven metadata + val pomProperties = """ + groupId=$group + artifactId=$module + version=$version + """.trimIndent() + + val pomPropertiesPath = "META-INF/maven/$group/$module/pom.properties" + jos.putNextEntry(java.util.jar.JarEntry(pomPropertiesPath)) + jos.write(pomProperties.toByteArray()) + jos.closeEntry() + + // Add custom content if provided + contentBuilder?.invoke(jos) + + // Add manifest if not provided by contentBuilder + val manifestEntry = java.util.jar.JarEntry("META-INF/MANIFEST.MF") + jos.putNextEntry(manifestEntry) + jos.write("Manifest-Version: 1.0\n".toByteArray()) + jos.closeEntry() + } + } +} diff --git a/buildSrc/src/test/kotlin/datadog/gradle/plugin/config/ParseV2SupportedConfigurationsTest.kt b/buildSrc/src/test/kotlin/datadog/gradle/plugin/config/ParseV2SupportedConfigurationsTest.kt new file mode 100644 index 00000000000..bb6a04d52f1 --- /dev/null +++ b/buildSrc/src/test/kotlin/datadog/gradle/plugin/config/ParseV2SupportedConfigurationsTest.kt @@ -0,0 +1,199 @@ +package datadog.gradle.plugin.config + +import org.gradle.testkit.runner.BuildResult +import org.gradle.testkit.runner.GradleRunner +import org.gradle.testkit.runner.TaskOutcome +import org.junit.jupiter.api.Assertions.assertEquals +import org.junit.jupiter.api.Assertions.assertTrue +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.io.TempDir +import java.io.File +import java.nio.file.Paths + +class ParseV2SupportedConfigurationsTest { + @Test + fun `should generate Java file from JSON configuration`(@TempDir projectDir: File) { + val (buildResult, generatedFile) = runGradleTask(projectDir) + + assertEquals(TaskOutcome.SUCCESS, buildResult.task(":generateSupportedConfigurations")?.outcome) + + assertTrue(generatedFile.exists(), "Generated Java file should exist") + + val content = generatedFile.readText() + assertTrue(content.contains("package datadog.test;")) + assertTrue(content.contains("public final class TestGeneratedSupportedConfigurations {")) + + assertTrue(content.contains("public static final Map> SUPPORTED;")) + assertTrue(content.contains("public static final Map> ALIASES;")) + assertTrue(content.contains("public static final Map ALIAS_MAPPING;")) + assertTrue(content.contains("public static final Map DEPRECATED;")) + assertTrue(content.contains("public static final Map REVERSE_PROPERTY_KEYS_MAP;")) + + assertTrue(content.contains("private static Map> initSupported()")) + assertTrue(content.contains("private static void initSupported1(Map> supportedMap)")) + assertTrue(content.contains("private static void initSupported2(Map> supportedMap)")) + assertTrue(content.contains("private static Map> initAliases()")) + assertTrue(content.contains("private static Map initAliasMapping()")) + assertTrue(content.contains("private static Map initDeprecated()")) + assertTrue(content.contains("private static Map initReversePropertyKeysMap()")) + + assertContainsSupportedConfig( + content, + key = "DD_ACTION_EXECUTION_ID", + version = "A", + type = "string", + default = "null", + aliases = emptyList(), + propertyKeys = listOf("property.key") + ) + + assertContainsSupportedConfig( + content, + key = "DD_AGENTLESS_LOG_SUBMISSION_ENABLED", + version = "A", + type = "boolean", + default = "false", + aliases = emptyList() + ) + + assertContainsSupportedConfig( + content, + key = "DD_AGENTLESS_LOG_SUBMISSION_ENABLED", + version = "B", + type = "boolean", + default = "true", + aliases = listOf("DD_ALIAS") + ) + + assertTrue(content.contains("""aliasesMap.put("DD_ACTION_EXECUTION_ID", Collections.unmodifiableList(Arrays.asList()))""")) + assertTrue(content.contains("""aliasesMap.put("DD_AGENTLESS_LOG_SUBMISSION_ENABLED", Collections.unmodifiableList(Arrays.asList("DD_ALIAS")))""")) + + assertTrue(content.contains("""aliasMappingMap.put("DD_ALIAS", "DD_AGENTLESS_LOG_SUBMISSION_ENABLED")""")) + + assertTrue(content.contains("""deprecatedMap.put("old.config", "Use test.config instead")""")) + assertTrue(content.contains("""deprecatedMap.put("legacy.setting", "No longer supported")""")) + + assertTrue(content.contains("""reversePropertyKeysMapping.put("property.key", "DD_ACTION_EXECUTION_ID")""")) + } + + private fun runGradleTask(projectDir: File): Pair { + val jsonFile = file(projectDir, "test-supported-configurations.json") + jsonFile.writeText( + """ + { + "supportedConfigurations": { + "DD_ACTION_EXECUTION_ID": [ + { + "version": "A", + "type": "string", + "default": null, + "aliases": [], + "propertyKeys": ["property.key"] + } + ], + "DD_AGENTLESS_LOG_SUBMISSION_ENABLED": [ + { + "version": "A", + "type": "boolean", + "default": "false", + "aliases": [] + }, + { + "version": "B", + "type": "boolean", + "default": "true", + "aliases": ["DD_ALIAS"] + } + ] + }, + "deprecations": { + "old.config": "Use test.config instead", + "legacy.setting": "No longer supported" + } + } + """.trimIndent() + ) + + setupGradleProject(projectDir) + + val buildResult = GradleRunner.create() + .forwardOutput() + .withPluginClasspath() + .withArguments("generateSupportedConfigurations") + .withProjectDir(projectDir) + .build() + + val generatedFile = file(projectDir, "build", "generated", "supportedConfigurations", "datadog", "test", "TestGeneratedSupportedConfigurations.java") + return Pair(buildResult, generatedFile) + } + + private fun setupGradleProject(projectDir: File) { + file(projectDir, "settings.gradle.kts").writeText( + """ + rootProject.name = "test-config-project" + """.trimIndent() + ) + + file(projectDir, "build.gradle.kts").writeText( + """ + plugins { + id("java") + id("dd-trace-java.supported-config-generator") + } + + group = "datadog.config.test" + + supportedTracerConfigurations { + jsonFile.set(file("test-supported-configurations.json")) + destinationDirectory.set(file("build/generated/supportedConfigurations")) + className.set("datadog.test.TestGeneratedSupportedConfigurations") + } + """.trimIndent() + ) + } + + private fun file(projectDir: File, vararg parts: String, makeDirectory: Boolean = false): File { + val f = Paths.get(projectDir.absolutePath, *parts).toFile() + + if (makeDirectory) { + f.parentFile.mkdirs() + } + + return f + } + + private fun assertContainsSupportedConfig( + content: String, + key: String, + version: String, + type: String, + default: String, + aliases: List, + propertyKeys: List = emptyList() + ) { + val aliasesArray = aliases.joinToString(", ") { "\"$it\"" } + val propertyKeysArray = propertyKeys.joinToString(", ") { "\"$it\"" } + + assertTrue( + content.contains("""supportedMap.put("$key""""), + "Should contain supportedMap.put for key: $key" + ) + + val expectedPattern = buildString { + append("new SupportedConfiguration(") + append("\"$version\", ") + append("\"$type\", ") + append(if (default == "null") "null" else "\"$default\"") + append(", ") + append("Arrays.asList($aliasesArray)") + append(", ") + append("Arrays.asList($propertyKeysArray)") + append(")") + } + + assertTrue( + content.contains(expectedPattern), + "Should contain SupportedConfiguration: $expectedPattern" + ) + } +} diff --git a/buildSrc/src/test/kotlin/datadog/gradle/plugin/csi/CallSiteInstrumentationPluginTest.kt b/buildSrc/src/test/kotlin/datadog/gradle/plugin/csi/CallSiteInstrumentationPluginTest.kt new file mode 100644 index 00000000000..2f69abd5fc9 --- /dev/null +++ b/buildSrc/src/test/kotlin/datadog/gradle/plugin/csi/CallSiteInstrumentationPluginTest.kt @@ -0,0 +1,155 @@ +package datadog.gradle.plugin.csi + +import org.gradle.testkit.runner.BuildResult +import org.gradle.testkit.runner.GradleRunner +import org.gradle.testkit.runner.UnexpectedBuildFailure +import org.junit.jupiter.api.Assertions.assertFalse +import org.junit.jupiter.api.Assertions.assertThrows +import org.junit.jupiter.api.Assertions.assertTrue +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.io.TempDir +import java.io.File +import java.nio.file.Files + +class CallSiteInstrumentationPluginTest { + private val buildGradle = """ + plugins { + id 'java' + id 'dd-trace-java.call-site-instrumentation' + } + + sourceCompatibility = JavaVersion.VERSION_1_8 + targetCompatibility = JavaVersion.VERSION_1_8 + + csi { + suffix = 'CallSite' + targetFolder = project.layout.buildDirectory.dir('csi') + rootFolder = file('__ROOT_FOLDER__') + } + + repositories { + mavenCentral() + } + + dependencies { + implementation group: 'net.bytebuddy', name: 'byte-buddy', version: '1.18.8' + implementation group: 'com.google.auto.service', name: 'auto-service-annotations', version: '1.1.1' + } + """.trimIndent() + + @TempDir + lateinit var buildDir: File + + @Test + fun `test call site instrumentation plugin`() { + createGradleProject( + buildDir, buildGradle, + """ + import datadog.trace.agent.tooling.csi.*; + + @CallSite(spi = CallSites.class) + public class BeforeAdviceCallSite { + @CallSite.Before("java.lang.StringBuilder java.lang.StringBuilder.append(java.lang.String)") + public static void beforeAppend(@CallSite.This final StringBuilder self, @CallSite.Argument final String param) { + } + } + """.trimIndent() + ) + + val result = buildGradleProject(buildDir) + + val generated = resolve(buildDir, "build", "csi", "BeforeAdviceCallSites.java") + assertTrue(generated.exists()) + + val output = result.output + assertFalse(output.contains("[⨉]")) + assertTrue(output.contains("[✓] @CallSite BeforeAdviceCallSite")) + } + + @Test + fun `test call site instrumentation plugin with error`() { + createGradleProject( + buildDir, buildGradle, + """ + import datadog.trace.agent.tooling.csi.*; + + @CallSite(spi = CallSites.class) + public class BeforeAdviceCallSite { + @CallSite.Before("java.lang.StringBuilder java.lang.StringBuilder.append(java.lang.String)") + private void beforeAppend(@CallSite.This final StringBuilder self, @CallSite.Argument final String param) { + } + } + """.trimIndent() + ) + + val error = assertThrows(UnexpectedBuildFailure::class.java) { + buildGradleProject(buildDir) + } + + val generated = resolve(buildDir, "build", "csi", "BeforeAdviceCallSites.java") + assertFalse(generated.exists()) + + val output = error.message ?: "" + assertFalse(output.contains("[✓]")) + assertTrue(output.contains("ADVICE_METHOD_NOT_STATIC_AND_PUBLIC")) + } + + private fun createGradleProject(buildDir: File, gradleFile: String, advice: String) { + val projectFolder = File(System.getProperty("user.dir")).parentFile + val callSiteJar = resolve(projectFolder, "buildSrc", "call-site-instrumentation-plugin", "build", "libs", "call-site-instrumentation-plugin-all.jar") + val testCallSiteJarDir = resolve(buildDir, "buildSrc", "call-site-instrumentation-plugin", "build", "libs", makeDirs = true) + + Files.copy( + callSiteJar.toPath(), + testCallSiteJarDir.toPath().resolve(callSiteJar.name) + ) + + val gradleFileContent = gradleFile.replace("__ROOT_FOLDER__", projectFolder.toString().replace("\\", "\\\\")) + writeText(resolve(buildDir, "build.gradle"), gradleFileContent) + + val javaFolder = resolve(buildDir, "src", "main", "java", makeDirs = true) + val advicePackage = parsePackage(advice) + val adviceClassName = parseClassName(advice) + val adviceFolder = resolve(javaFolder, *advicePackage.split("\\.").toTypedArray(), makeDirs = true) + writeText(resolve(adviceFolder, "$adviceClassName.java"), advice) + + val csiSource = resolve(projectFolder, "dd-java-agent", "agent-tooling", "src", "main", "java", "datadog", "trace", "agent", "tooling", "csi") + val csiTarget = resolve(javaFolder, "datadog", "trace", "agent", "tooling", "csi", makeDirs = true) + csiSource.listFiles()?.forEach { + writeText(File(csiTarget, it.name), it.readText()) + } + } + + private fun buildGradleProject(buildDir: File): BuildResult { + return GradleRunner.create() + .withTestKitDir(File(buildDir, ".gradle-test-kit")) // workaround in case the global test-kit cache becomes corrupted + .withDebug(true) // avoids starting daemon which can leave undeleted files post-cleanup + .withProjectDir(buildDir) + .withArguments("build", "--info", "--stacktrace") + .withPluginClasspath() + .forwardOutput() + .build() + } + + private fun parsePackage(advice: String): String { + val regex = Regex("package\\s+([\\w.]+)\\s*;", RegexOption.DOT_MATCHES_ALL) + val match = regex.find(advice) + return match?.groupValues?.getOrNull(1) ?: "" + } + + private fun parseClassName(advice: String): String { + val regex = Regex("class\\s+(\\w+)\\s+\\{", RegexOption.DOT_MATCHES_ALL) + val match = regex.find(advice) + return match?.groupValues?.getOrNull(1) ?: "" + } + + private fun resolve(parent: File, vararg path: String, makeDirs: Boolean = false): File { + return path.fold(parent) { acc, next -> File(acc, next) }.apply { + if (makeDirs) { + mkdirs() + } + } + } + + private fun writeText(file: File, content: String) = file.writeText(content) +} diff --git a/buildSrc/src/integTest/kotlin/datadog/gradle/plugin/version/DumpHangedTestIntegrationTest.kt b/buildSrc/src/test/kotlin/datadog/gradle/plugin/dump/DumpHangedTestIntegrationTest.kt similarity index 97% rename from buildSrc/src/integTest/kotlin/datadog/gradle/plugin/version/DumpHangedTestIntegrationTest.kt rename to buildSrc/src/test/kotlin/datadog/gradle/plugin/dump/DumpHangedTestIntegrationTest.kt index 0b1fc6bbad0..d0a8279c85d 100644 --- a/buildSrc/src/integTest/kotlin/datadog/gradle/plugin/version/DumpHangedTestIntegrationTest.kt +++ b/buildSrc/src/test/kotlin/datadog/gradle/plugin/dump/DumpHangedTestIntegrationTest.kt @@ -1,4 +1,4 @@ -package datadog.gradle.plugin.version +package datadog.gradle.plugin.dump import org.gradle.testkit.runner.GradleRunner import org.gradle.testkit.runner.UnexpectedBuildFailure @@ -55,7 +55,7 @@ class DumpHangedTestIntegrationTest { plugins { id("java") - id("datadog.dump-hanged-test") + id("dd-trace-java.dump-hanged-test") } group = "datadog.dump.test" diff --git a/buildSrc/src/test/kotlin/datadog/gradle/plugin/instrument/BuildTimeInstrumentationPluginTest.kt b/buildSrc/src/test/kotlin/datadog/gradle/plugin/instrument/BuildTimeInstrumentationPluginTest.kt new file mode 100644 index 00000000000..2a76a65cc5d --- /dev/null +++ b/buildSrc/src/test/kotlin/datadog/gradle/plugin/instrument/BuildTimeInstrumentationPluginTest.kt @@ -0,0 +1,231 @@ +package datadog.gradle.plugin.instrument + +import net.bytebuddy.utility.OpenedClassReader +import org.gradle.testkit.runner.GradleRunner +import org.junit.jupiter.api.Assertions.assertTrue +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.io.TempDir +import org.objectweb.asm.ClassReader +import org.objectweb.asm.ClassVisitor +import org.objectweb.asm.ClassWriter +import org.objectweb.asm.FieldVisitor +import org.objectweb.asm.Opcodes +import java.io.File +import java.io.FileInputStream + +class BuildTimeInstrumentationPluginTest { + + private val buildGradle = """ + plugins { + id 'java' + id 'dd-trace-java.build-time-instrumentation' + } + + sourceCompatibility = JavaVersion.VERSION_1_8 + targetCompatibility = JavaVersion.VERSION_1_8 + + repositories { + mavenCentral() + } + + dependencies { + compileOnly group: 'net.bytebuddy', name: 'byte-buddy', version: '1.18.8' // just to build TestPlugin + } + + buildTimeInstrumentation.plugins = [ + 'TestPlugin' + ] + """.trimIndent() + + private val exampleCode = """ + package example; + public class ExampleCode {} + """.trimIndent() + + @TempDir + lateinit var buildDir: File + + @Test + fun `test instrument plugin`() { + val buildFile = File(buildDir, "build.gradle") + buildFile.writeText(buildGradle) + + val srcMainJava = testPlugin("src/main/java", "ExampleCode") + + val examplePackageDir = File(srcMainJava, "example").apply { mkdirs() } + File(examplePackageDir, "ExampleCode.java").writeText(exampleCode) + + // Run Gradle build with TestKit + GradleRunner.create().withTestKitDir(File(buildDir, ".gradle-test-kit")) // workaround in case the global test-kit cache becomes corrupted + .withDebug(true) // avoids starting daemon which can leave undeleted files post-cleanup + .withProjectDir(buildDir) + .withArguments("build", "--stacktrace") + .withPluginClasspath() + .forwardOutput() + .build() + + assertInstrumented(File(buildDir, "build/classes/java/main/example/ExampleCode.class")) + } + + @Test + fun `test instrument plugin processes includeClassDirectories`() { + val buildFile = File(buildDir, "build.gradle") + buildFile.writeText(""" + plugins { + id 'java' + id 'dd-trace-java.build-time-instrumentation' + } + + sourceCompatibility = JavaVersion.VERSION_1_8 + targetCompatibility = JavaVersion.VERSION_1_8 + + repositories { + mavenCentral() + } + + dependencies { + compileOnly group: 'net.bytebuddy', name: 'byte-buddy', version: '1.18.8' + } + + buildTimeInstrumentation { + plugins = ['TestPlugin'] + includeClassDirectories.from(file('external-classes')) + } + """.trimIndent()) + + testPlugin("src/main/java", "ExternalCode") + + // Pre-compile ExternalCode using ASM and place it in the external-classes directory + val externalClassesDir = File(buildDir, "external-classes").apply { mkdirs() } + precompiledClass("ExternalCode", externalClassesDir) + + GradleRunner.create() + .withTestKitDir(File(buildDir, ".gradle-test-kit")) + .withDebug(true) + .withProjectDir(buildDir) + .withArguments("build", "--stacktrace") + .withPluginClasspath() + .forwardOutput() + .build() + + // ExternalCode.class should have been copied from external-classes, instrumented, and placed in the output + assertInstrumented(File(buildDir, "build/classes/java/main/ExternalCode.class")) + } + + @Test + fun `test rerun-tasks does not lose includeClassDirectories classes`() { + val buildFile = File(buildDir, "build.gradle") + buildFile.writeText(""" + plugins { + id 'java' + id 'dd-trace-java.build-time-instrumentation' + } + + sourceCompatibility = JavaVersion.VERSION_1_8 + targetCompatibility = JavaVersion.VERSION_1_8 + + repositories { + mavenCentral() + } + + dependencies { + compileOnly group: 'net.bytebuddy', name: 'byte-buddy', version: '1.18.8' + } + + buildTimeInstrumentation { + plugins = ['TestPlugin'] + includeClassDirectories.from(file('external-classes')) + } + """.trimIndent()) + + val srcMainJava = testPlugin("src/main/java", "ExampleCode", "ExternalCode") + val examplePackageDir = File(srcMainJava, "example").apply { mkdirs() } + File(examplePackageDir, "ExampleCode.java").writeText("package example; public class ExampleCode {}") + + val externalClassesDir = File(buildDir, "external-classes").apply { mkdirs() } + precompiledClass("ExternalCode", externalClassesDir) + + val runner = GradleRunner.create() + .withTestKitDir(File(buildDir, ".gradle-test-kit")) + .withDebug(true) + .withProjectDir(buildDir) + .withPluginClasspath() + .forwardOutput() + + // First build + runner.withArguments("build", "--stacktrace").build() + + // Second build with --rerun-tasks: compileJava wipes classesDirectory, so without + // the fix InstrumentAction would only sync freshly-compiled classes and lose ExternalCode.class + runner.withArguments("build", "--rerun-tasks", "--stacktrace").build() + + assertInstrumented(File(buildDir, "build/classes/java/main/example/ExampleCode.class")) + assertInstrumented(File(buildDir, "build/classes/java/main/ExternalCode.class")) + } + + private fun testPlugin(srcDir: String, vararg classNames: String): File { + val dir = File(buildDir, srcDir).apply { mkdirs() } + val conditions = classNames.joinToString(" || ") { "\"$it\".equals(name)" } + File(dir, "TestPlugin.java").writeText(""" + import java.io.File; + import java.io.IOException; + import net.bytebuddy.build.Plugin; + import net.bytebuddy.description.type.TypeDescription; + import net.bytebuddy.dynamic.ClassFileLocator; + import net.bytebuddy.dynamic.DynamicType; + + public class TestPlugin implements Plugin { + private final File targetDir; + + public TestPlugin(File targetDir) { + this.targetDir = targetDir; + } + + @Override + public boolean matches(TypeDescription target) { + String name = target.getSimpleName(); + return $conditions; + } + + @Override + public DynamicType.Builder apply( + DynamicType.Builder builder, + TypeDescription typeDescription, + ClassFileLocator classFileLocator) { + return builder.defineField("__TEST__FIELD__", Void.class); + } + + @Override + public void close() throws IOException { + // no-op + } + } + """.trimIndent()) + return dir + } + + private fun precompiledClass(className: String, targetDir: File) { + val classWriter = ClassWriter(0) + classWriter.visit(Opcodes.V1_8, Opcodes.ACC_PUBLIC, className, null, "java/lang/Object", null) + classWriter.visitEnd() + File(targetDir, "$className.class").writeBytes(classWriter.toByteArray()) + } + + private fun assertInstrumented(classFile: File) { + assertTrue(classFile.isFile, "${classFile.name} should be present in the output directory") + var foundInsertedField = false + FileInputStream(classFile).use { input -> + val classReader = ClassReader(input) + classReader.accept( + object : ClassVisitor(OpenedClassReader.ASM_API) { + override fun visitField(access: Int, fieldName: String?, descriptor: String?, signature: String?, value: Any?): FieldVisitor? { + if ("__TEST__FIELD__" == fieldName) foundInsertedField = true + return null + } + }, + OpenedClassReader.ASM_API + ) + } + assertTrue(foundInsertedField, "${classFile.name} should have been instrumented with __TEST__FIELD__") + } +} diff --git a/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzleDirectiveTest.kt b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzleDirectiveTest.kt new file mode 100644 index 00000000000..b5c2ccaef42 --- /dev/null +++ b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzleDirectiveTest.kt @@ -0,0 +1,162 @@ +package datadog.gradle.plugin.muzzle + +import org.eclipse.aether.repository.RemoteRepository +import org.junit.jupiter.api.Test +import org.junit.jupiter.params.ParameterizedTest +import org.junit.jupiter.params.provider.CsvSource +import org.assertj.core.api.Assertions.assertThat + +class MuzzleDirectiveTest { + + @ParameterizedTest(name = "[{index}] nameSlug(''{0}'') == ''{1}''") + @CsvSource( + value = + [ + "simple, simple", + "My Directive, My-Directive", + "foo/bar@baz#123, foo-bar-baz-123", + ]) + fun `nameSlug replaces non-alphanumeric characters with dashes`(input: String, expected: String) { + val directive = MuzzleDirective().apply { name = input } + assertThat(directive.nameSlug).isEqualTo(expected.trim()) + } + + @Test + fun `nameSlug returns empty string for empty name`() { + val directive = MuzzleDirective().apply { name = "" } + assertThat(directive.nameSlug).isEmpty() + } + + @Test + fun `nameSlug trims leading and trailing whitespace before replacing`() { + val directive = MuzzleDirective().apply { name = " spaces " } + assertThat(directive.nameSlug).isEqualTo("spaces") + } + + @Test + fun `nameSlug returns empty string when name is null`() { + val directive = MuzzleDirective() // name defaults to null + assertThat(directive.nameSlug).isEmpty() + } + + @Test + fun `getRepositories returns defaults unchanged when no additional repos`() { + val directive = MuzzleDirective() + val defaults = listOf(RemoteRepository.Builder("central", "default", "https://repo1.maven.org/maven2/").build()) + + val repos = directive.getRepositories(defaults) + + // Same reference — no copy is made when additionalRepositories is empty + assertThat(repos).isSameAs(defaults) + } + + @Test + fun `getRepositories appends additional repositories after defaults`() { + val directive = + MuzzleDirective().apply { + extraRepository("myrepo", "https://example.com/repo") + extraRepository("otherrepo", "https://other.example.com/repo", "default") + } + val defaults = + listOf( + RemoteRepository.Builder("central", "default", "https://repo1.maven.org/maven2/").build()) + + val repos = directive.getRepositories(defaults) + + assertThat(repos.map { it.id }).containsExactly("central", "myrepo", "otherrepo") + } + + @Test + fun `coreJdk without version sets isCoreJdk true and javaVersion null`() { + val directive = MuzzleDirective() + directive.coreJdk() + + assertThat(directive.isCoreJdk).isTrue() + assertThat(directive.javaVersion).isNull() + } + + @Test + fun `coreJdk with version sets isCoreJdk true and javaVersion`() { + val directive = MuzzleDirective() + directive.coreJdk("17") + + assertThat(directive.isCoreJdk).isTrue() + assertThat(directive.javaVersion).isEqualTo("17") + } + + @ParameterizedTest(name = "[{index}] coreJdk={0}, assertPass={1} → {2}") + @CsvSource( + value = + [ + "true, true, Pass-core-jdk", + "true, false, Fail-core-jdk", + ]) + fun `toString for coreJdk directive`(isCoreJdk: Boolean, assertPass: Boolean, expected: String) { + val directive = + MuzzleDirective().apply { + if (isCoreJdk) coreJdk() + this.assertPass = assertPass + } + assertThat(directive.toString()).isEqualTo(expected) + } + + @ParameterizedTest(name = "[{index}] assertPass={0} → prefix ''{1}''") + @CsvSource( + value = + [ + "true, pass", + "false, fail", + ]) + fun `toString for non-coreJdk directive includes group module versions`( + assertPass: Boolean, + prefix: String + ) { + val directive = + MuzzleDirective().apply { + group = "com.example" + module = "mylib" + versions = "[1.0,2.0)" + this.assertPass = assertPass + } + + assertThat(directive.toString()).isEqualTo("$prefix com.example:mylib:[1.0,2.0)") + } + + @Test + fun `extraDependency accumulates multiple entries in order`() { + val directive = MuzzleDirective() + directive.extraDependency("com.example:dep1:1.0") + directive.extraDependency("com.example:dep2:2.0") + directive.extraDependency("com.example:dep3:3.0") + + assertThat(directive.additionalDependencies).containsExactly( + "com.example:dep1:1.0", + "com.example:dep2:2.0", + "com.example:dep3:3.0" + ) + } + + @Test + fun `excludeDependency accumulates multiple entries in order`() { + val directive = MuzzleDirective() + directive.excludeDependency("com.example:excluded1") + directive.excludeDependency("com.example:excluded2") + + assertThat(directive.excludedDependencies).containsExactly( + "com.example:excluded1", + "com.example:excluded2" + ) + } + + @Test + fun `extraRepository accumulates multiple entries in order`() { + val directive = MuzzleDirective() + directive.extraRepository("repo1", "https://repo1.example.com") + directive.extraRepository("repo2", "https://repo2.example.com", "p2") + + assertThat(directive.additionalRepositories).containsExactly( + Triple("repo1", "default", "https://repo1.example.com"), + Triple("repo2", "p2", "https://repo2.example.com"), + ) + } +} diff --git a/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzleMavenRepoUtilsTest.kt b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzleMavenRepoUtilsTest.kt new file mode 100644 index 00000000000..8c500a42f43 --- /dev/null +++ b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzleMavenRepoUtilsTest.kt @@ -0,0 +1,227 @@ +package datadog.gradle.plugin.muzzle + +import datadog.gradle.plugin.MavenRepoFixture +import org.eclipse.aether.artifact.DefaultArtifact +import org.eclipse.aether.repository.RemoteRepository +import org.eclipse.aether.resolution.VersionRangeRequest +import org.eclipse.aether.resolution.VersionRangeResult +import org.eclipse.aether.util.version.GenericVersionScheme +import org.gradle.api.GradleException +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.io.TempDir +import org.junit.jupiter.params.ParameterizedTest +import org.junit.jupiter.params.provider.CsvSource +import java.io.File +import org.assertj.core.api.Assertions.assertThat +import org.assertj.core.api.Assertions.assertThatThrownBy + +class MuzzleMavenRepoUtilsTest { + + @TempDir + lateinit var tempDir: File + + private val system = MuzzleMavenRepoUtils.newRepositorySystem() + + private val versionScheme = GenericVersionScheme() + + @Test + fun `resolveVersionRange resolves all versions matching an open range`() { + val repo = publishAndGetRepo("com.example", "mylib", listOf("1.0.0", "2.0.0", "3.0.0")) + val directive = MuzzleDirective().apply { + group = "com.example" + module = "mylib" + versions = "[1.0,)" + } + + val result = MuzzleMavenRepoUtils.resolveVersionRange(directive, system, newSession(), listOf(repo)) + + val resolvedVersions = result.versions.map { it.toString() } + assertThat(resolvedVersions).containsExactly("1.0.0", "2.0.0", "3.0.0") + } + + @Test + fun `resolveVersionRange respects bounded version range`() { + val repo = publishAndGetRepo("com.example", "mylib", listOf("1.0.0", "2.0.0", "3.0.0", "4.0.0", "5.0.0")) + val directive = MuzzleDirective().apply { + group = "com.example" + module = "mylib" + versions = "[2.0,4.0)" + } + + val result = MuzzleMavenRepoUtils.resolveVersionRange(directive, system, newSession(), listOf(repo)) + + val resolvedVersions = result.versions.map { it.toString() } + assertThat(resolvedVersions).containsExactly("2.0.0", "3.0.0") + } + + @Test + fun `resolveVersionRange throws IllegalStateException when resolution consistently fails`() { + val emptyRepo = RemoteRepository.Builder("empty", "default", File(tempDir, "empty").apply { mkdirs() }.toURI().toString()).build() + val directive = MuzzleDirective().apply { + group = "com.example" + module = "nonexistent" + versions = "[1.0,)" + } + + assertThatThrownBy { + MuzzleMavenRepoUtils.resolveVersionRange(directive, system, newSession(), listOf(emptyRepo)) + }.isInstanceOf(IllegalStateException::class.java) + } + + @Test + fun `resolveVersionRange includes directive extra repositories`() { + val repoA = publishAndGetRepo("com.example", "mylib", listOf("1.0.0", "2.0.0"), subDir = "repoA") + val fixtureB = MavenRepoFixture(File(tempDir, "repoB")) + fixtureB.publishVersions("com.example", "mylib", listOf("3.0.0")) + val directive = MuzzleDirective().apply { + group = "com.example" + module = "mylib" + versions = "[1.0,)" + extraRepository("repoB", fixtureB.repoUrl) + } + + val result = MuzzleMavenRepoUtils.resolveVersionRange(directive, system, newSession(), listOf(repoA)) + + val resolvedVersions = result.versions.map { it.toString() } + assertThat(resolvedVersions) + .withFailMessage("Expected all 3 versions from both repos, got: $resolvedVersions") + .containsAll(listOf("1.0.0", "2.0.0", "3.0.0")) + } + + @Test + fun `inverseOf returns directives outside range, inverts assertPass, and preserves properties`() { + val repo = publishAndGetRepo("com.example", "mylib", listOf("1.0.0", "2.0.0", "3.0.0", "4.0.0", "5.0.0")) + val directive = MuzzleDirective().apply { + name = "mytest" + group = "com.example" + module = "mylib" + versions = "[2.0,4.0)" + assertPass = true + excludeDependency("com.other:dep") + includeSnapshots = false + } + + val result = MuzzleMavenRepoUtils.inverseOf(directive, system, newSession(), listOf(repo)) + + val resultVersions = result.map { it.versions }.toSet() + // Versions inside [2.0, 4.0) are 2.0.0 and 3.0.0 — they should NOT appear + assertThat(resultVersions).doesNotContain("2.0.0", "3.0.0") + // Versions outside range: 1.0.0, 4.0.0, 5.0.0 + assertThat(resultVersions).contains("1.0.0", "4.0.0", "5.0.0") + + // assertPass must be inverted, and directive properties must be preserved + assertThat(result).allSatisfy { directive -> + assertThat(directive.assertPass).isFalse() + assertThat(directive.name).isEqualTo("mytest") + assertThat(directive.group).isEqualTo("com.example") + assertThat(directive.module).isEqualTo("mylib") + assertThat(directive.excludedDependencies).containsExactly("com.other:dep") + assertThat(directive.includeSnapshots).isFalse() + } + } + + @ParameterizedTest(name = "[{index}] highest({0}, {1}) == {2}") + @CsvSource( + value = + [ + "1.0.0, 2.0.0, 2.0.0", + "2.0.0, 1.0.0, 2.0.0", + "3.5.1, 3.5.1, 3.5.1", // equal — either is acceptable + ]) + fun `highest returns the greater version`(a: String, b: String, expected: String) { + val result = MuzzleMavenRepoUtils.highest(version(a), version(b)) + assertThat(result).isEqualTo(version(expected)) + } + + @ParameterizedTest(name = "[{index}] lowest({0}, {1}) == {2}") + @CsvSource( + value = + [ + "1.0.0, 2.0.0, 1.0.0", + "2.0.0, 1.0.0, 1.0.0", + "3.5.1, 3.5.1, 3.5.1", // equal — either is acceptable + ]) + fun `lowest returns the lesser version`(a: String, b: String, expected: String) { + val result = MuzzleMavenRepoUtils.lowest(version(a), version(b)) + assertThat(result).isEqualTo(version(expected)) + } + + @Test + fun `muzzleDirectiveToArtifacts throws GradleException when all versions are filtered out`() { + val directive = + MuzzleDirective().apply { + group = "com.example" + module = "test" + includeSnapshots = false // SNAPSHOT and RC will be filtered + } + // All versions are pre-release; none survive filterAndLimitVersions + val rangeResult = createVersionRangeResult("1.0.0-SNAPSHOT", "2.0.0-RC1") + + assertThatThrownBy { + MuzzleMavenRepoUtils.muzzleDirectiveToArtifacts(directive, rangeResult) + }.isInstanceOf(GradleException::class.java) + } + + @Test + fun `muzzleDirectiveToArtifacts produces artifacts with correct coordinates`() { + val directive = + MuzzleDirective().apply { + group = "com.example" + module = "mylib" + // classifier is null → DefaultArtifact receives "" + includeSnapshots = false + } + // Distinct major.minor versions so lowAndHighForMajorMinor keeps all three + val rangeResult = createVersionRangeResult("1.0.0", "2.0.0", "3.0.0") + + val artifacts = MuzzleMavenRepoUtils.muzzleDirectiveToArtifacts(directive, rangeResult) + + assertThat(artifacts).hasSize(3) + assertThat(artifacts).allSatisfy { artifact -> + assertThat(artifact.groupId).isEqualTo("com.example") + assertThat(artifact.artifactId).isEqualTo("mylib") + assertThat(artifact.extension).isEqualTo("jar") + assertThat(artifact.classifier).isEmpty() + } + assertThat(artifacts.map { it.version }).containsOnly("1.0.0", "2.0.0", "3.0.0") + } + + @Test + fun `muzzleDirectiveToArtifacts propagates classifier to artifacts`() { + val directive = + MuzzleDirective().apply { + group = "com.example" + module = "mylib" + classifier = "tests" + includeSnapshots = false + } + val rangeResult = createVersionRangeResult("1.0.0", "2.0.0") + + val artifacts = MuzzleMavenRepoUtils.muzzleDirectiveToArtifacts(directive, rangeResult) + + assertThat(artifacts).allSatisfy { assertThat(it.classifier).isEqualTo("tests") } + } + + private fun newSession() = MuzzleMavenRepoUtils.newRepositorySystemSession(system) + + private fun publishAndGetRepo( + group: String, + module: String, + versions: List, + subDir: String = "default" + ): RemoteRepository { + val fixture = MavenRepoFixture(File(tempDir, subDir)) + fixture.publishVersions(group, module, versions) + return RemoteRepository.Builder(subDir, "default", fixture.repoUrl).build() + } + + private fun version(v: String) = versionScheme.parseVersion(v) + + private fun createVersionRangeResult(vararg versionStrings: String): VersionRangeResult { + val artifact = DefaultArtifact("com.example:test:[1.0,)") + val request = VersionRangeRequest(artifact, emptyList(), null) + val versions = versionStrings.map { versionScheme.parseVersion(it) }.sorted() + // lowestVersion/highestVersion are computed as versions[0] and versions[last] + return VersionRangeResult(request).apply { this.versions = versions } + } +} diff --git a/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzlePluginFunctionalTest.kt b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzlePluginFunctionalTest.kt new file mode 100644 index 00000000000..00eb2514db0 --- /dev/null +++ b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzlePluginFunctionalTest.kt @@ -0,0 +1,869 @@ +package datadog.gradle.plugin.muzzle + +import datadog.gradle.plugin.GradleFixture +import datadog.gradle.plugin.MavenRepoFixture +import org.assertj.core.api.Assertions.assertThat +import org.gradle.testkit.runner.TaskOutcome.SUCCESS +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.io.TempDir +import org.junit.jupiter.params.ParameterizedTest +import org.junit.jupiter.params.provider.ValueSource +import java.io.File +import kotlin.io.path.readText + +class MuzzlePluginFunctionalTest { + @ParameterizedTest + @ValueSource(strings = ["muzzle", ":dd-java-agent:instrumentation:demo:muzzle", "runMuzzle"]) + fun `detects muzzle invocation with various task names`( + taskName: String, + @TempDir projectDir: File + ) { + val fixture = MuzzlePluginTestFixture(projectDir) + + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + + muzzle { + pass { + coreJdk() + } + } + """ + ) + + // Add runMuzzle aggregator task at root level (like in dd-trace-java.ci-jobs.gradle.kts) + fixture.writeRootProject( + """ + tasks.register('runMuzzle') { + dependsOn(':dd-java-agent:instrumentation:demo:muzzle') + } + """ + ) + + fixture.writeNoopScanPlugin() + + val result = fixture.run(taskName, "--stacktrace") + + assertThat(result.tasks) + .withFailMessage("Should create muzzle tasks when '$taskName' is requested") + .anyMatch { it.path.contains("muzzle") } + assertThat(result.output) + .withFailMessage("Should not skip muzzle task planification when '$taskName' is requested") + .doesNotContain("No muzzle tasks invoked, skipping muzzle task planification") + assertThat(result.tasks).withFailMessage("Should execute muzzle tasks when '$taskName' is requested") + .anyMatch { it.path == ":dd-java-agent:instrumentation:demo:muzzle" || it.path.contains("muzzle-Assert") } + } + + @Test + fun `muzzle with pass directive writes junit report`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + + muzzle { + pass { + name = 'expected-pass' + coreJdk() + } + } + """ + ) + fixture.writeScanPlugin( + """ + if (!assertPass) { + throw new IllegalStateException("unexpected fail assertion for " + muzzleDirective); + } + """ + ) + + val buildResult = fixture.run(":dd-java-agent:instrumentation:demo:muzzle", "--stacktrace") + assertThat(buildResult.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome).isEqualTo(SUCCESS) + assertThat(buildResult.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-core-jdk")?.outcome) + .isEqualTo(SUCCESS) + assertThat(buildResult.task(":dd-java-agent:instrumentation:demo:muzzle-end")?.outcome).isEqualTo(SUCCESS) + + val reportFile = fixture.findSingleMuzzleJUnitReport() + val report = fixture.parseXml(reportFile) + val suite = report.documentElement + assertThat(suite.tagName).isEqualTo("testsuite") + assertThat(suite.getAttribute("name")).isEqualTo(":dd-java-agent:instrumentation:demo") + assertThat(suite.getAttribute("tests")).isEqualTo("1") + assertThat(suite.getAttribute("failures")).isEqualTo("0") + + val passCase = findTestCase(report, "muzzle-AssertPass-core-jdk") + assertThat(passCase.getElementsByTagName("failure").length).isEqualTo(0) + } + + @Test + fun `muzzle without directives writes default junit report`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + """ + ) + fixture.writeScanPlugin( + """ + if (!assertPass) { + throw new IllegalStateException("unexpected fail assertion for " + muzzleDirective); + } + """ + ) + + val result = fixture.run(":dd-java-agent:instrumentation:demo:muzzle", "--stacktrace") + assertThat(result.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome).isEqualTo(SUCCESS) + + val reportFile = fixture.findSingleMuzzleJUnitReport() + val report = fixture.parseXml(reportFile) + val suite = report.documentElement + assertThat(suite.getAttribute("name")).isEqualTo(":dd-java-agent:instrumentation:demo") + assertThat(suite.getAttribute("tests")).isEqualTo("1") + assertThat(suite.getAttribute("failures")).isEqualTo("0") + + val defaultCase = findTestCase(report, "muzzle") + assertThat(defaultCase.getElementsByTagName("failure").length).isEqualTo(0) + } + + @Test + fun `non muzzle invocation does not register muzzle end task`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + + muzzle { + pass { + coreJdk() + } + } + """ + ) + + val buildResult = fixture.run(":dd-java-agent:instrumentation:demo:tasks", "--all") + + assertThat(buildResult.output).doesNotContain("muzzle-end") + } + + @Test + fun `muzzle plugin wires bootstrap and tooling project classpaths`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + """ + ) + + val bootstrapDependencies = fixture.run( + ":dd-java-agent:instrumentation:demo:dependencies", + "--configuration", + "muzzleBootstrap" + ) + assertThat(bootstrapDependencies.output).contains("project :dd-java-agent:agent-bootstrap") + + val toolingDependencies = fixture.run( + ":dd-java-agent:instrumentation:demo:dependencies", + "--configuration", + "muzzleTooling" + ) + assertThat(toolingDependencies.output).contains("project :dd-java-agent:agent-tooling") + } + + @Test + fun `muzzle executes exactly planned core-jdk tasks and writes task results`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + + muzzle { + pass { coreJdk() } + fail { coreJdk() } + } + """ + ) + fixture.writeScanPlugin( + """ + // pass + """ + ) + + val result = fixture.run(":dd-java-agent:instrumentation:demo:muzzle", "--stacktrace") + val muzzleTaskPath = ":dd-java-agent:instrumentation:demo:muzzle" + val passDirectiveTaskPath = ":dd-java-agent:instrumentation:demo:muzzle-AssertPass-core-jdk" + val failDirectiveTaskPath = ":dd-java-agent:instrumentation:demo:muzzle-AssertFail-core-jdk" + val endTaskPath = ":dd-java-agent:instrumentation:demo:muzzle-end" + + assertThat(result.task(muzzleTaskPath)?.outcome).isEqualTo(SUCCESS) + assertThat(result.task(passDirectiveTaskPath)?.outcome).isEqualTo(SUCCESS) + assertThat(result.task(failDirectiveTaskPath)?.outcome).isEqualTo(SUCCESS) + assertThat(result.task(endTaskPath)?.outcome).isEqualTo(SUCCESS) + + val muzzleChainInOrder = result.tasks + .map { it.path } + .filter { + it == muzzleTaskPath || + it == passDirectiveTaskPath || + it == failDirectiveTaskPath || + it == endTaskPath + } + assertThat(muzzleChainInOrder) + .containsExactly(muzzleTaskPath, passDirectiveTaskPath, failDirectiveTaskPath, endTaskPath) + + val passDirectiveResult = fixture.resultFile("muzzle-AssertPass-core-jdk") + val failDirectiveResult = fixture.resultFile("muzzle-AssertFail-core-jdk") + assertThat(passDirectiveResult).isRegularFile() + assertThat(failDirectiveResult).isRegularFile() + assertThat(passDirectiveResult.readText()).isEqualTo("PASSING") + assertThat(failDirectiveResult.readText()).isEqualTo("PASSING") + } + + @Test + fun `artifact directive resolves multiple versions from version range`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + val mavenRepoFixture = MavenRepoFixture(projectDir) + + mavenRepoFixture.publishVersions( + group = "com.example.test", + module = "demo-lib", + versions = listOf("1.0.0", "1.1.0", "1.2.0", "2.0.0") + ) + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + + // Gradle repositories for artifact download + repositories { + maven { + url = uri('${mavenRepoFixture.repoUrl}') + metadataSources { + mavenPom() + artifact() + // Disable checksum validation for fake repo + } + } + } + + muzzle { + pass { + group = 'com.example.test' + module = 'demo-lib' + versions = '[1.0.0,2.0.0)' // Should resolve 1.0.0, 1.1.0, 1.2.0 but NOT 2.0.0 + } + } + """ + ) + fixture.writeNoopScanPlugin() + + // Leveraging MAVEN_REPOSITORY_PROXY to point to our fake repo over maven central + val result = fixture.run( + ":dd-java-agent:instrumentation:demo:muzzle", + "--stacktrace", + env = mapOf("MAVEN_REPOSITORY_PROXY" to mavenRepoFixture.repoUrl) + ) + + assertThat(result.output) + .withFailMessage("Build should succeed. Output:\n${result.output.take(3000)}") + .contains("BUILD SUCCESSFUL") + + assertThat(result.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome).isEqualTo(SUCCESS) + assertThat(result.task(":dd-java-agent:instrumentation:demo:muzzle-end")?.outcome).isEqualTo(SUCCESS) + + assertThat(result.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-com.example.test-demo-lib-1.0.0")?.outcome) + .isEqualTo(SUCCESS) + assertThat(result.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-com.example.test-demo-lib-1.1.0")?.outcome) + .isEqualTo(SUCCESS) + assertThat(result.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-com.example.test-demo-lib-1.2.0")?.outcome) + .isEqualTo(SUCCESS) + assertThat(result.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-com.example.test-demo-lib-2.0.0")?.outcome) + .withFailMessage("Should not check against test-demo-lib:2.0.0") + .isNull() + + val reportFile = fixture.findSingleMuzzleJUnitReport() + val report = fixture.parseXml(reportFile) + val suite = report.documentElement + val testCount = suite.getAttribute("tests").toInt() + assertThat(testCount) + .withFailMessage("Should have at least 3 tests for 3 versions, got $testCount") + .isGreaterThanOrEqualTo(3) + assertThat(suite.getAttribute("failures")).withFailMessage("Should have no failures").isEqualTo("0") + + val testCases = (0 until report.getElementsByTagName("testcase").length) + .map { report.getElementsByTagName("testcase").item(it) as org.w3c.dom.Element } + .map { it.getAttribute("name") } + assertThat(testCases).withFailMessage("Should have test case for demo-lib-1.0.0. Found: ${testCases.take(5)}") + .anySatisfy { assertThat(it).contains("demo-lib-1.0.0") } + } + + @Test + fun `named directive is passed to scan plugin`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + + muzzle { + pass { + name = 'my-custom-check' + coreJdk() + } + } + """ + ) + + // The real MuzzleVersionScanPlugin uses the directive name to filter InstrumenterModules + fixture.writeScanPlugin( + """ + if (!"my-custom-check".equals(muzzleDirective)) { + throw new IllegalStateException( + "Expected muzzleDirective to be 'my-custom-check', but got: '" + muzzleDirective + "'" + ); + } + + System.out.println("Directive name passed correctly: " + muzzleDirective); + """ + ) + + val result = fixture.run(":dd-java-agent:instrumentation:demo:muzzle", "--stacktrace") + + assertThat(result.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-core-jdk")?.outcome) + .isEqualTo(SUCCESS) + assertThat(result.output).withFailMessage("Should confirm 'my-custom-check' was passed to scan plugin") + .contains("Directive name passed correctly: my-custom-check") + } + + @Test + fun `non-existent artifact fails with clear error message`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + + muzzle { + pass { + group = 'com.example.nonexistent' + module = 'does-not-exist' + versions = '[1.0.0,2.0.0)' + } + } + """ + ) + fixture.writeNoopScanPlugin() + + val result = fixture.run( + ":dd-java-agent:instrumentation:demo:muzzle", + "--stacktrace", + env = mapOf("MAVEN_REPOSITORY_PROXY" to "https://repo1.maven.org/maven2/") + ) + + assertThat(result.output).withFailMessage("Build should fail for non-existent artifact").contains("BUILD FAILED") + assertThat(result.output).withFailMessage("Should have error message about resolution failure") + .containsAnyOf( + "version range resolution failed", + "Could not resolve", + "not found", + "Failed to resolve" + ) + } + + @Test + fun `pass directive that fails validation causes build failure`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + + muzzle { + pass { + coreJdk() + } + } + """ + ) + + // Real implementation throws RuntimeException when !passed && assertPass (line 70 of MuzzleVersionScanPlugin) + fixture.writeScanPlugin( + """ + if (assertPass) { + System.err.println("FAILED MUZZLE VALIDATION: mismatches:"); + System.err.println("-- missing class Foo"); + throw new RuntimeException("Instrumentation failed Muzzle validation"); + } + """ + ) + + val result = fixture.run( + ":dd-java-agent:instrumentation:demo:muzzle", + "--stacktrace" + ) + + assertThat(result.output).withFailMessage("Build should fail when pass directive fails validation") + .contains("BUILD FAILED") + assertThat(result.output).withFailMessage("Should contain error message from scan plugin") + .containsAnyOf("Muzzle validation failed", "Instrumentation failed") + } + + @Test + fun `fail directive that passes validation causes build failure`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + + muzzle { + fail { + coreJdk() + } + } + """ + ) + + // Scan plugin simulates successful validation when it should fail + // Real MuzzleVersionScanPlugin throws RuntimeException when passed && !assertPass + fixture.writeScanPlugin( + """ + if (!assertPass) { + System.err.println("MUZZLE PASSED BUT FAILURE WAS EXPECTED"); + throw new RuntimeException("Instrumentation unexpectedly passed Muzzle validation"); + } + """ + ) + + val result = fixture.run( + ":dd-java-agent:instrumentation:demo:muzzle", + "--stacktrace" + ) + + // Expected behavior: build should fail when fail directive unexpectedly passes + assertThat(result.output) + .withFailMessage("Build should fail when fail directive unexpectedly passes") + .contains("BUILD FAILED") + assertThat(result.output).withFailMessage("Should indicate that fail directive passed when it shouldn't have") + .containsAnyOf("unexpectedly passed", "FAILURE WAS EXPECTED") + } + + @Test + fun `additional dependencies are added to muzzle test classpath`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + val mavenRepoFixture = MavenRepoFixture(projectDir) + + // Create a fake Maven repo with a fake additional dependency + // The JAR will automatically include standard Maven metadata + mavenRepoFixture.publishVersions( + group = "com.example.extra", + module = "extra-lib", + versions = listOf("1.0.0") + ) + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + + repositories { + maven { + url = uri('${mavenRepoFixture.repoUrl}') + metadataSources { + mavenPom() + artifact() + } + } + } + + muzzle { + pass { + coreJdk() + extraDependency('com.example.extra:extra-lib:1.0.0') + } + } + """ + ) + + // Scan plugin verifies that the additional dependency JAR is in the classpath + fixture.writeScanPlugin( + """ + java.io.InputStream resource = testApplicationClassLoader.getResourceAsStream("META-INF/maven/com.example.extra/extra-lib/pom.properties"); + if (resource != null) { + try { + resource.close(); + } catch (java.io.IOException e) { + // Ignore + } + System.out.println("Additional dependency (extra-lib) found in test classpath"); + } else { + throw new RuntimeException("Additional dependency (extra-lib) not found in test classpath"); + } + """ + ) + + val result = fixture.run( + ":dd-java-agent:instrumentation:demo:muzzle", + "--stacktrace", + env = mapOf("MAVEN_REPOSITORY_PROXY" to mavenRepoFixture.repoUrl) + ) + + assertThat(result.output) + .withFailMessage("Build should succeed. Output:\n${result.output.take(2000)}") + .contains("BUILD SUCCESSFUL") + assertThat(result.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome).isEqualTo(SUCCESS) + assertThat(result.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-core-jdk")?.outcome) + .isEqualTo(SUCCESS) + assertThat(result.output).withFailMessage("Additional dependency should be loadable from test classpath") + .contains("Additional dependency (extra-lib) found in test classpath") + } + + @Test + fun `excluded dependencies are removed from muzzle test classpath`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + val mavenRepoFixture = MavenRepoFixture(projectDir) + + // Create a fake repo with an artifact that has transitive dependencies + mavenRepoFixture.publishVersions( + group = "com.example.test", + module = "with-transitive", + versions = listOf("1.0.0") + ) + + // Manually create a POM with a transitive dependency + val pomFile = mavenRepoFixture.repoDir.resolve("com/example/test/with-transitive/1.0.0/with-transitive-1.0.0.pom") + pomFile.writeText( + """ + + 4.0.0 + com.example.test + with-transitive + 1.0.0 + + + com.google.guava + guava + 31.0-jre + + + + """.trimIndent() + ) + + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + + repositories { + maven { + url = uri('${mavenRepoFixture.repoUrl}') + metadataSources { + mavenPom() + artifact() + } + } + mavenCentral() + } + + muzzle { + pass { + group = 'com.example.test' + module = 'with-transitive' + versions = '1.0.0' + excludeDependency('com.google.guava:guava') + } + } + """ + ) + + // Scan plugin verifies that guava is NOT in the classpath (it was excluded) + fixture.writeScanPlugin( + """ + try { + testApplicationClassLoader.loadClass("com.google.common.collect.ImmutableList"); + throw new RuntimeException("Unexpected excluded dependency (guava) SHOULD NOT be in test classpath but was found"); + } catch (ClassNotFoundException e) { + System.out.println("Excluded dependency (guava) correctly not in test classpath"); + } + """ + ) + + val result = fixture.run( + ":dd-java-agent:instrumentation:demo:muzzle", + "--stacktrace", + env = mapOf("MAVEN_REPOSITORY_PROXY" to mavenRepoFixture.repoUrl) + ) + + assertThat(result.output).contains("BUILD SUCCESSFUL") + assertThat(result.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome).isEqualTo(SUCCESS) + assertThat(result.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-com.example.test-with-transitive-1.0.0")?.outcome) + .isEqualTo(SUCCESS) + assertThat(result.output).withFailMessage("Excluded dependency should not be loadable from test classpath") + .contains("Excluded dependency (guava) correctly not in test classpath") + } + + @Test + fun `java plugin applied after muzzle plugin`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + + fixture.writeProject( + """ + plugins { + id 'dd-trace-java.muzzle' + } + + // applied after muzzle plugin + apply plugin: 'java' + + muzzle { + pass { + coreJdk() + } + } + """ + ) + fixture.writeNoopScanPlugin() + + val result = fixture.run( + ":dd-java-agent:instrumentation:demo:muzzle", + "--stacktrace" + ) + + assertThat(result.output).contains("BUILD SUCCESSFUL") + assertThat(result.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome).isEqualTo(SUCCESS) + assertThat(result.task(":dd-java-agent:instrumentation:demo:muzzle-end")?.outcome).isEqualTo(SUCCESS) + } + + @Test + fun `java plugin applied before muzzle plugin`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' apply false // Declared but not applied + } + + // Apply muzzle plugin after java using imperative syntax + apply plugin: 'dd-trace-java.muzzle' + + muzzle { + pass { + coreJdk() + } + } + """ + ) + fixture.writeNoopScanPlugin() + + val result = fixture.run( + ":dd-java-agent:instrumentation:demo:muzzle", + "--stacktrace" + ) + + assertThat(result.output).contains("BUILD SUCCESSFUL") + assertThat(result.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome).isEqualTo(SUCCESS) + assertThat(result.task(":dd-java-agent:instrumentation:demo:muzzle-end")?.outcome).isEqualTo(SUCCESS) + } + + @Test + fun `plugin behavior without java plugin should no-op`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + + fixture.writeProject( + """ + plugins { + id 'dd-trace-java.muzzle' + // NO java plugin applied + } + + muzzle { + pass { + coreJdk() + } + } + """ + ) + fixture.writeNoopScanPlugin() + + val result = fixture.run( + ":dd-java-agent:instrumentation:demo:tasks", + "--all" + ) + + assertThat(result.output) + .withFailMessage("Should not create muzzle tasks without java plugin") + .doesNotContain("muzzle") + } + + @Test + fun `missing dd-java-agent projects error handling`(@TempDir projectDir: File) { + // Create a minimal settings.gradle without the dd-java-agent structure + File(projectDir, "settings.gradle").also { it.parentFile?.mkdirs() }.writeText( + """ + rootProject.name = 'muzzle-test' + include ':instrumentation:demo' + """.trimIndent() + ) + + File(projectDir, "instrumentation/demo/build.gradle").also { it.parentFile?.mkdirs() }.writeText( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + + muzzle { + pass { + coreJdk() + } + } + """.trimIndent() + ) + + // No need to create MuzzleVersionScanPlugin - the error happens during configuration + // phase before any task execution, so the scan plugin is never invoked + + val result = GradleFixture(projectDir).run( + ":instrumentation:demo:tasks", + "--stacktrace" + ) + + assertThat(result.output).withFailMessage("Should fail with clear error about missing dd-java-agent projects") + .containsAnyOf( + "BUILD FAILED", + ":dd-java-agent:agent-bootstrap project not found", + ":dd-java-agent:agent-tooling project not found" + ) + } + + @Test + fun `assertInverse creates pass and fail tasks for in-range and out-of-range versions`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + val mavenRepoFixture = MavenRepoFixture(projectDir) + + mavenRepoFixture.publishVersions( + group = "com.example.test", + module = "inverse-lib", + versions = listOf("1.0.0", "2.0.0", "3.0.0", "4.0.0") + ) + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + + // Gradle repositories for artifact download + repositories { + maven { + url = uri('${mavenRepoFixture.repoUrl}') + metadataSources { + mavenPom() + artifact() + } + } + } + + muzzle { + pass { + group = 'com.example.test' + module = 'inverse-lib' + versions = '[2.0.0,3.0.0]' + assertInverse = true + } + } + """ + ) + fixture.writeScanPlugin( + """ + System.out.println("MUZZLE_CHECK assertPass=" + assertPass); + """ + ) + + val result = fixture.run( + ":dd-java-agent:instrumentation:demo:muzzle", + "--stacktrace", + env = mapOf("MAVEN_REPOSITORY_PROXY" to mavenRepoFixture.repoUrl) + ) + + assertThat(result.output) + .withFailMessage("Build should succeed. Output:\n${result.output.take(3000)}") + .contains("BUILD SUCCESSFUL") + + val modulePrefix = ":dd-java-agent:instrumentation:demo" + assertThat(result.task("$modulePrefix:muzzle")?.outcome).isEqualTo(SUCCESS) + assertThat(result.task("$modulePrefix:muzzle-end")?.outcome).isEqualTo(SUCCESS) + + // In-range versions — assertPass=true + assertThat(result.task("$modulePrefix:muzzle-AssertPass-com.example.test-inverse-lib-2.0.0")?.outcome) + .isEqualTo(SUCCESS) + assertThat(result.task("$modulePrefix:muzzle-AssertPass-com.example.test-inverse-lib-3.0.0")?.outcome) + .isEqualTo(SUCCESS) + + // Out-of-range versions (inverse) — assertPass=false + assertThat(result.task("$modulePrefix:muzzle-AssertFail-com.example.test-inverse-lib-1.0.0")?.outcome) + .isEqualTo(SUCCESS) + assertThat(result.task("$modulePrefix:muzzle-AssertFail-com.example.test-inverse-lib-4.0.0")?.outcome) + .isEqualTo(SUCCESS) + + assertThat(result.output) + .withFailMessage("Should log assertPass=true for in-range versions") + .contains("MUZZLE_CHECK assertPass=true") + assertThat(result.output) + .withFailMessage("Should log assertPass=false for out-of-range (inverse) versions") + .contains("MUZZLE_CHECK assertPass=false") + + // Verify JUnit report contains all 4 test cases with no failures + val reportFile = fixture.findSingleMuzzleJUnitReport() + val report = fixture.parseXml(reportFile) + val suite = report.documentElement + assertThat(suite.getAttribute("tests")) + .withFailMessage("Should have 4 test cases (2 pass + 2 inverse fail)") + .isEqualTo("4") + assertThat(suite.getAttribute("failures")).withFailMessage("Should have no failures").isEqualTo("0") + + findTestCase(report, "muzzle-AssertPass-com.example.test-inverse-lib-2.0.0") + findTestCase(report, "muzzle-AssertPass-com.example.test-inverse-lib-3.0.0") + findTestCase(report, "muzzle-AssertFail-com.example.test-inverse-lib-1.0.0") + findTestCase(report, "muzzle-AssertFail-com.example.test-inverse-lib-4.0.0") + } + + private fun findTestCase(document: org.w3c.dom.Document, name: String): org.w3c.dom.Element = + (0 until document.getElementsByTagName("testcase").length) + .map { document.getElementsByTagName("testcase").item(it) as org.w3c.dom.Element } + .first { it.getAttribute("name") == name } +} diff --git a/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzlePluginPerformanceTest.kt b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzlePluginPerformanceTest.kt new file mode 100644 index 00000000000..79aaf409b7c --- /dev/null +++ b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzlePluginPerformanceTest.kt @@ -0,0 +1,407 @@ +package datadog.gradle.plugin.muzzle + +import datadog.gradle.plugin.MavenRepoFixture +import org.gradle.testkit.runner.TaskOutcome.SUCCESS +import org.gradle.testkit.runner.TaskOutcome.UP_TO_DATE +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.io.TempDir +import java.io.File +import org.assertj.core.api.Assertions.assertThat + +class MuzzlePluginPerformanceTest { + + @Test + fun `task graph does not include muzzle tasks when not requested`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + + muzzle { + pass { + group = 'com.example.test' + module = 'some-lib' + versions = '[1.0.0,2.0.0)' + } + } + """ + ) + fixture.writeNoopScanPlugin() + + val result = fixture.run( + ":dd-java-agent:instrumentation:demo:tasks", + "--all", + "--info" + ) + + assertThat(result.task(":dd-java-agent:instrumentation:demo:tasks")?.outcome).isEqualTo(SUCCESS) + + assertThat(result.tasks) + .withFailMessage("Should not create or execute any muzzle tasks when not requested") + .noneMatch { it.path.contains("muzzle") } + assertThat(result.output) + .withFailMessage("Should log early return when muzzle not requested") + .contains("No muzzle tasks invoked for :dd-java-agent:instrumentation:demo, skipping muzzle task planification") + } + + @Test + fun `does not configure muzzle when other project muzzle task is requested`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + muzzle { + pass { coreJdk() } + } + """ + ) + fixture.writeNoopScanPlugin() + fixture.addSubproject("dd-java-agent:instrumentation:other", + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + muzzle { + pass { coreJdk() } + } + """ + ) + + val result = fixture.run( + ":dd-java-agent:instrumentation:demo:muzzle", + "--stacktrace", + "--info" + ) + + assertThat(result.tasks) + .withFailMessage("Should execute muzzle tasks for demo project") + .anyMatch { it.path.contains("demo") && it.path.contains("muzzle") } + assertThat(result.tasks) + .withFailMessage("Should NOT create or register execute muzzle tasks for other project") + .noneMatch { it.path.contains("other") && it.path.contains("muzzle") } + assertThat(result.output.lines()) + .withFailMessage("Other project should skip muzzle configuration when demo project's muzzle is requested") + .anyMatch { it.contains("No muzzle tasks invoked for :dd-java-agent:instrumentation:other, skipping muzzle task planification") } + } + + @Test + fun `muzzle tasks are up-to-date when nothing changes`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + val mavenRepoFixture = MavenRepoFixture(projectDir) + + mavenRepoFixture.publishVersions( + group = "com.example.test", + module = "example-lib", + versions = listOf("1.0.0", "1.1.0") + ) + + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + + repositories { + maven { + url = uri('${mavenRepoFixture.repoUrl}') + metadataSources { + mavenPom() + artifact() + } + } + } + + muzzle { + pass { + group = 'com.example.test' + module = 'example-lib' + versions = '[1.0.0,2.0.0)' + } + } + """ + ) + fixture.writeNoopScanPlugin() + + // First run - should execute the tasks + run { + val firstRun = fixture.run( + ":dd-java-agent:instrumentation:demo:muzzle", + env = mapOf("MAVEN_REPOSITORY_PROXY" to mavenRepoFixture.repoUrl) + ) + + assertThat(firstRun.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome) + .withFailMessage("First run should execute muzzle task") + .isEqualTo(SUCCESS) + assertThat(firstRun.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-com.example.test-example-lib-1.0.0")?.outcome) + .isEqualTo(SUCCESS) + assertThat(firstRun.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-com.example.test-example-lib-1.1.0")?.outcome) + .isEqualTo(SUCCESS) + assertThat(firstRun.task(":dd-java-agent:instrumentation:demo:muzzle-end")?.outcome) + .withFailMessage("First run should execute muzzle-end task") + .isEqualTo(SUCCESS) + } + + // Second run without changes - assertion tasks should be up-to-date + run { + val secondRun = fixture.run( + ":dd-java-agent:instrumentation:demo:muzzle", + env = mapOf("MAVEN_REPOSITORY_PROXY" to mavenRepoFixture.repoUrl) + ) + + assertThat(secondRun.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome) + .withFailMessage("First run should execute muzzle task") + .isEqualTo(UP_TO_DATE) + assertThat(secondRun.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-com.example.test-example-lib-1.0.0")?.outcome) + .withFailMessage("1.0.0 assertion task should be up-to-date") + .isEqualTo(UP_TO_DATE) + assertThat(secondRun.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-com.example.test-example-lib-1.1.0")?.outcome) + .withFailMessage("1.1.0 assertion task should be up-to-date") + .isEqualTo(UP_TO_DATE) + assertThat(secondRun.task(":dd-java-agent:instrumentation:demo:muzzle-end")?.outcome) + .withFailMessage("First run should execute muzzle-end task") + .isEqualTo(SUCCESS) + } + + // Third run after adding new version - should NOT be up-to-date + // Add version 1.2.0 to the fake Maven repo + run { + mavenRepoFixture.publishVersions( + group = "com.example.test", + module = "example-lib", + versions = listOf("1.2.0") + ) + + val thirdRun = fixture.run( + ":dd-java-agent:instrumentation:demo:muzzle", + env = mapOf("MAVEN_REPOSITORY_PROXY" to mavenRepoFixture.repoUrl) + ) + + assertThat(thirdRun.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome) + .withFailMessage("First run should execute muzzle task") + .isEqualTo(UP_TO_DATE) + assertThat(thirdRun.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-com.example.test-example-lib-1.0.0")?.outcome) + .withFailMessage("1.0.0 assertion task should be up-to-date") + .isEqualTo(UP_TO_DATE) + assertThat(thirdRun.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-com.example.test-example-lib-1.1.0")?.outcome) + .withFailMessage("1.1.0 assertion task should be up-to-date") + .isEqualTo(UP_TO_DATE) + assertThat(thirdRun.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-com.example.test-example-lib-1.2.0")?.outcome) + .withFailMessage("New 1.2.0 assertion task should be created and execute") + .isEqualTo(SUCCESS) + assertThat(thirdRun.task(":dd-java-agent:instrumentation:demo:muzzle-end")?.outcome) + .withFailMessage("First run should execute muzzle-end task") + .isEqualTo(SUCCESS) + } + } + + @Test + fun `muzzle tasks invalidated when instrumentation code changes`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + + muzzle { + pass { coreJdk() } + } + """ + ) + fixture.writeNoopScanPlugin() + + // First run - should execute the tasks + run { + val firstRun = fixture.run(":dd-java-agent:instrumentation:demo:muzzle") + + assertThat(firstRun.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome) + .withFailMessage("First run should execute muzzle task") + .isEqualTo(SUCCESS) + assertThat(firstRun.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-core-jdk")?.outcome) + .withFailMessage("First run should execute coreJdk assertion task") + .isEqualTo(SUCCESS) + } + + // Second run without changes - assertion tasks should be up-to-date + run { + val secondRun = fixture.run(":dd-java-agent:instrumentation:demo:muzzle") + + assertThat(secondRun.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome) + .withFailMessage("Second run should be up-to-date") + .isEqualTo(UP_TO_DATE) + assertThat(secondRun.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-core-jdk")?.outcome) + .withFailMessage("coreJdk assertion task should be up-to-date") + .isEqualTo(UP_TO_DATE) + } + + // Third run after changing instrumentation code - should be invalidated + run { + val demoSourceDir = File(projectDir, "dd-java-agent/instrumentation/demo/src/main/java/com/example") + demoSourceDir.mkdirs() + File(demoSourceDir, "Demo.java").writeText( + """ + package com.example; + + public class Demo { + public void doSomething() {} + } + """.trimIndent() + ) + + val thirdRun = fixture.run(":dd-java-agent:instrumentation:demo:muzzle") + + assertThat(thirdRun.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome) + .withFailMessage("Third run should execute after instrumentation code change") + .isEqualTo(SUCCESS) + assertThat(thirdRun.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-core-jdk")?.outcome) + .withFailMessage("coreJdk assertion task should be invalidated and re-execute") + .isEqualTo(SUCCESS) + } + } + + @Test + fun `muzzle tasks invalidated when tooling classpath changes`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + + muzzle { + pass { coreJdk() } + } + """ + ) + fixture.writeNoopScanPlugin() + + // First run - should execute the tasks + run { + val firstRun = fixture.run(":dd-java-agent:instrumentation:demo:muzzle") + + assertThat(firstRun.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome) + .withFailMessage("First run should execute muzzle task") + .isEqualTo(SUCCESS) + assertThat(firstRun.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-core-jdk")?.outcome) + .withFailMessage("First run should execute coreJdk assertion task") + .isEqualTo(SUCCESS) + } + + // Second run without changes - assertion tasks should be up-to-date + run { + val secondRun = fixture.run(":dd-java-agent:instrumentation:demo:muzzle") + + assertThat(secondRun.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome) + .withFailMessage("Second run should be up-to-date") + .isEqualTo(UP_TO_DATE) + assertThat(secondRun.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-core-jdk")?.outcome) + .withFailMessage("coreJdk assertion task should be up-to-date") + .isEqualTo(UP_TO_DATE) + } + + // Third run after changing agent-tooling code - should be invalidated + run { + val toolingSourceDir = File(projectDir, "dd-java-agent/agent-tooling/src/main/java/datadog/trace/agent/tooling") + toolingSourceDir.mkdirs() + File(toolingSourceDir, "Extra.java").writeText( + """ + package datadog.trace.agent.tooling; + + public class Extra { + public void extraMethod() {} + } + """.trimIndent() + ) + + val thirdRun = fixture.run(":dd-java-agent:instrumentation:demo:muzzle") + + assertThat(thirdRun.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome) + .withFailMessage("Third run should execute after tooling classpath change") + .isEqualTo(SUCCESS) + assertThat(thirdRun.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-core-jdk")?.outcome) + .withFailMessage("coreJdk assertion task should be invalidated and re-execute") + .isEqualTo(SUCCESS) + } + } + + @Test + fun `muzzle tasks invalidated when bootstrap classpath changes`(@TempDir projectDir: File) { + val fixture = MuzzlePluginTestFixture(projectDir) + + fixture.writeProject( + """ + plugins { + id 'java' + id 'dd-trace-java.muzzle' + } + + muzzle { + pass { coreJdk() } + } + """ + ) + fixture.writeNoopScanPlugin() + + // First run - should execute the tasks + run { + val firstRun = fixture.run(":dd-java-agent:instrumentation:demo:muzzle") + + assertThat(firstRun.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome) + .withFailMessage("First run should execute muzzle task") + .isEqualTo(SUCCESS) + assertThat(firstRun.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-core-jdk")?.outcome) + .withFailMessage("First run should execute coreJdk assertion task") + .isEqualTo(SUCCESS) + } + + // Second run without changes - assertion tasks should be up-to-date + run { + val secondRun = fixture.run(":dd-java-agent:instrumentation:demo:muzzle") + + assertThat(secondRun.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome) + .withFailMessage("Second run should be up-to-date") + .isEqualTo(UP_TO_DATE) + assertThat(secondRun.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-core-jdk")?.outcome) + .withFailMessage("coreJdk assertion task should be up-to-date") + .isEqualTo(UP_TO_DATE) + } + + // Third run after changing agent-bootstrap code - should be invalidated + run { + val bootstrapSourceDir = File(projectDir, "dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap") + bootstrapSourceDir.mkdirs() + File(bootstrapSourceDir, "Helper.java").writeText( + """ + package datadog.trace.bootstrap; + + public class Helper { + public void help() {} + } + """.trimIndent() + ) + + val thirdRun = fixture.run(":dd-java-agent:instrumentation:demo:muzzle") + + assertThat(thirdRun.task(":dd-java-agent:instrumentation:demo:muzzle")?.outcome) + .withFailMessage("Third run should execute after bootstrap classpath change") + .isEqualTo(SUCCESS) + assertThat(thirdRun.task(":dd-java-agent:instrumentation:demo:muzzle-AssertPass-core-jdk")?.outcome) + .withFailMessage("coreJdk assertion task should be invalidated and re-execute") + .isEqualTo(SUCCESS) + } + } +} diff --git a/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzlePluginTestFixture.kt b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzlePluginTestFixture.kt new file mode 100644 index 00000000000..026ef5b0d9d --- /dev/null +++ b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzlePluginTestFixture.kt @@ -0,0 +1,100 @@ +package datadog.gradle.plugin.muzzle + +import datadog.gradle.plugin.GradleFixture +import org.intellij.lang.annotations.Language +import java.io.File + +/** + * Test fixture for muzzle plugin integration tests. + * Extends GradleFixture with muzzle-specific functionality. + */ +internal class MuzzlePluginTestFixture(projectDir: File) : GradleFixture(projectDir) { + + /** + * Writes the basic Gradle project structure for muzzle testing. + * Creates a multi-project build with agent-bootstrap, agent-tooling, and instrumentation modules. + */ + fun writeProject(@Language("Groovy") instrumentationBuildScript: String) { + file("settings.gradle").writeText( + // language=Groovy + """ + rootProject.name = 'muzzle-e2e' + """.trimIndent() + ) + + addSubproject("dd-java-agent:agent-bootstrap", + """ + plugins { + id 'java' + } + + tasks.register('compileMain_java11Java') + """ + ) + + addSubproject("dd-java-agent:agent-tooling", + """ + plugins { + id 'java' + } + """ + ) + + addSubproject("dd-java-agent:instrumentation:demo", instrumentationBuildScript) + } + + /** + * Writes a muzzle scan plugin that always passes. + */ + fun writeNoopScanPlugin() { + writeScanPlugin("// noop") + } + + /** + * Writes a muzzle scan plugin with custom assertion logic. + * The plugin is written to the agent-tooling project where it belongs. + * + * @param assertionBody Java code to execute in the assertion method + */ + fun writeScanPlugin(@Language("JAVA") assertionBody: String) { + file("dd-java-agent/agent-tooling/src/main/java/datadog/trace/agent/tooling/muzzle/MuzzleVersionScanPlugin.java") + .writeText( + // language=JAVA + """ + package datadog.trace.agent.tooling.muzzle; + + public final class MuzzleVersionScanPlugin { + private MuzzleVersionScanPlugin() {} + + public static void assertInstrumentationMuzzled( + ClassLoader instrumentationClassLoader, + ClassLoader testApplicationClassLoader, + boolean assertPass, + String muzzleDirective) { + $assertionBody + } + } + """.trimIndent() + ) + } + + /** + * Finds the single JUnit XML report generated by muzzle tests. + * Throws if zero or multiple reports are found. + */ + fun findSingleMuzzleJUnitReport(): File { + val reports = projectDir.walkTopDown() + .filter { it.isFile && it.name.startsWith("TEST-muzzle-") && it.extension == "xml" } + .toList() + require(reports.size == 1) { + "Expected exactly one JUnit muzzle report, but found ${reports.size}" + } + return reports.single() + } + + /** + * Returns the path to a muzzle result file for the given task name. + */ + fun resultFile(taskName: String) = + projectDir.toPath().resolve("dd-java-agent/instrumentation/demo/build/reports/$taskName.txt") +} diff --git a/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzlePluginUtilsTest.kt b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzlePluginUtilsTest.kt new file mode 100644 index 00000000000..3435a923d27 --- /dev/null +++ b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzlePluginUtilsTest.kt @@ -0,0 +1,64 @@ +package datadog.gradle.plugin.muzzle + +import org.gradle.api.tasks.SourceSetContainer +import org.gradle.kotlin.dsl.getByType +import org.gradle.testfixtures.ProjectBuilder +import org.junit.jupiter.api.Test +import org.junit.jupiter.params.ParameterizedTest +import org.junit.jupiter.params.provider.CsvSource +import org.assertj.core.api.Assertions.assertThat + +class MuzzlePluginUtilsTest { + @Test + fun `pathSlug for root project is empty`() { + val root = ProjectBuilder.builder().withName("root").build() + assertThat(root.pathSlug).isEmpty() + } + + @ParameterizedTest(name = "[{index}] path ''{0}'' → slug ''{1}''") + @CsvSource( + value = + [ + "foo, foo", + "foo_bar_baz, foo_bar_baz", // underscores are preserved (only colons are replaced) + ]) + fun `pathSlug for single-level child project`(childName: String, expectedSlug: String) { + val root = ProjectBuilder.builder().withName("root").build() + val child = ProjectBuilder.builder().withParent(root).withName(childName.trim()).build() + assertThat(child.pathSlug).isEqualTo(expectedSlug.trim()) + } + + @Test + fun `pathSlug for deeply nested project replaces colons with underscores`() { + val root = ProjectBuilder.builder().withName("root").build() + val foo = ProjectBuilder.builder().withParent(root).withName("foo").build() + val bar = ProjectBuilder.builder().withParent(foo).withName("bar").build() + val baz = ProjectBuilder.builder().withParent(bar).withName("baz").build() + + assertThat(baz.pathSlug).isEqualTo("foo_bar_baz") + } + + @Test + fun `allMainSourceSet includes main and excludes test`() { + val project = ProjectBuilder.builder().build() + project.plugins.apply("java") + + val sourceSets = project.allMainSourceSet + + assertThat(sourceSets.map { it.name }).contains("main").doesNotContain("test") + } + + @Test + fun `allMainSourceSet includes all source sets whose name starts with main`() { + val project = ProjectBuilder.builder().build() + project.plugins.apply("java") + project.extensions.getByType().apply { + create("mainLegacy") + create("mainJava8") + } + + val sourceSets = project.allMainSourceSet + + assertThat(sourceSets.map { it.name }).containsExactlyInAnyOrder("main", "mainLegacy", "mainJava8") + } +} diff --git a/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzleVersionUtilsTest.kt b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzleVersionUtilsTest.kt new file mode 100644 index 00000000000..9bc298a40f5 --- /dev/null +++ b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/MuzzleVersionUtilsTest.kt @@ -0,0 +1,156 @@ +package datadog.gradle.plugin.muzzle + +import datadog.gradle.plugin.muzzle.MuzzleVersionUtils.RANGE_COUNT_LIMIT +import org.eclipse.aether.artifact.DefaultArtifact +import org.eclipse.aether.resolution.VersionRangeRequest +import org.eclipse.aether.resolution.VersionRangeResult +import org.eclipse.aether.util.version.GenericVersionScheme +import org.junit.jupiter.api.Test +import org.junit.jupiter.params.ParameterizedTest +import org.junit.jupiter.params.provider.Arguments +import org.junit.jupiter.params.provider.MethodSource +import org.junit.jupiter.params.provider.ValueSource +import org.assertj.core.api.Assertions.assertThat + +class MuzzleVersionUtilsTest { + + private val versionScheme = GenericVersionScheme() + + @ParameterizedTest(name = "[{index}] filters pre-release: {0}") + @ValueSource( + strings = + [ + "2.0.0-SNAPSHOT", // -snapshot + "2.0.0-RC1", // rc + "2.0.0.CR1", // .cr + "2.0.0-alpha", // alpha + "2.0.0-beta.1", // beta + "2.0.0-b2", // -b + "2.0.0.M1", // .m + "2.0.0-m1", // -m + "2.0.0-dev", // -dev + "2.0.0-ea", // -ea + "2.0.0-atlassian-3", // -atlassian- + "2.0-public_draft", // public_draft + "2.0.0-cr1", // -cr + "2.0-preview", // -preview + "2.0.0.redhat-1", // redhat + "2.7.3m2", // END_NMN_PATTERN ^.*\.[0-9]+[mM][0-9]+$ + "2.0.0-1a2b3c4d", // GIT_SHA_PATTERN ^.*-[0-9a-f]{7,}$ + ]) + fun `filterAndLimitVersions filters out pre-release versions when includeSnapshots is false`( + preRelease: String + ) { + val result = createVersionRangeResult("1.0.0", preRelease, "3.0.0") + + val filtered = + MuzzleVersionUtils.filterAndLimitVersions(result, emptySet(), includeSnapshots = false) + + val filteredStrings = filtered.map { it.toString() } + assertThat(filteredStrings).withFailMessage("Expected '$preRelease' to be filtered out").doesNotContain(preRelease) + assertThat(filteredStrings).contains("1.0.0", "3.0.0") + } + + @ParameterizedTest(name = "[{index}] includeSnapshots=true keeps ''{0}'', skipVersions={1}") + @MethodSource("includeSnapshotsCases") + fun `with includeSnapshots=true, keeps pre-release versions and still respects skipVersions`( + preRelease: String, + skipVersions: Set + ) { + // preRelease major.minor = 1.0, surrounded by 2.0 and 3.0-RC1 (distinct major.minor) + val result = createVersionRangeResult(preRelease, "2.0.0", "3.0.0-RC1") + + val filtered = + MuzzleVersionUtils.filterAndLimitVersions(result, skipVersions, includeSnapshots = true) + + val filteredStrings = filtered.map { it.toString() } + assertThat(filteredStrings) + .withFailMessage("Expected '$preRelease' to be kept when includeSnapshots=true") + .contains(preRelease) + skipVersions.forEach { skipped -> + assertThat(filteredStrings) + .withFailMessage("Expected '$skipped' to be absent due to skipVersions") + .doesNotContain(skipped) + } + } + + @ParameterizedTest(name = "[{index}] skips exact version: {0}") + @ValueSource(strings = ["1.1.0", "1.3.0", "2.0.0"]) + fun `can skip exact versions`(versionToSkip: String) { + val result = createVersionRangeResult("1.0.0", "1.1.0", "1.2.0", "1.3.0", "2.0.0", "3.0.0") + + val filtered = + MuzzleVersionUtils.filterAndLimitVersions( + result, setOf(versionToSkip), includeSnapshots = false) + + assertThat(filtered.map { it.toString() }).doesNotContain(versionToSkip) + } + + @Test + fun `skip versions is case sensitive`() { + val result = createVersionRangeResult("1.0.0", "2.0.0-custom", "3.0.0") + + val filtered = + MuzzleVersionUtils.filterAndLimitVersions( + result, setOf("2.0.0-Custom"), includeSnapshots = false) + + assertThat(filtered.map { it.toString() }) + .withFailMessage("Expected '2.0.0-custom' to be kept because skipVersions entry 'Custom' does not match lowercased 'custom'") + .contains("2.0.0-custom") + } + + @Test + fun `trim version range larger than the limit`() { + // 30 versions with distinct major.minor: 1.0.0, 1.1.0, ..., 1.29.0 + val versions = (0..29).map { "1.$it.0" }.toTypedArray() + val result = createVersionRangeResult(*versions) + + val filtered = + MuzzleVersionUtils.filterAndLimitVersions(result, emptySet(), includeSnapshots = false) + + assertThat(filtered).withFailMessage("Expected fewer than 25 versions after trimming, got ${filtered.size}") + .hasSizeLessThan(RANGE_COUNT_LIMIT) + assertThat(filtered).isNotEmpty() + val filteredStrings = filtered.map { it.toString() } + assertThat(filteredStrings).withFailMessage("lowestVersion (${result.lowestVersion}) must be preserved") + .contains(result.lowestVersion.toString()) + assertThat(filteredStrings).withFailMessage("highestVersion (${result.highestVersion}) must be preserved") + .contains(result.highestVersion.toString()) + assertThat(filteredStrings).withFailMessage("All filtered versions must come from the original set") + .isSubsetOf(*versions) + } + + @ParameterizedTest(name = "[{index}] {0} version(s) pass through unchanged") + @ValueSource(ints = [1, 2, 3, 10, 24]) + fun `should limit large ranges`(count: Int) { + val versionStrings = (0 until count).map { "$it.0.0" }.toTypedArray() + val result = createVersionRangeResult(*versionStrings) + + val filtered = + MuzzleVersionUtils.filterAndLimitVersions(result, emptySet(), includeSnapshots = false) + + assertThat(filtered.map { it.toString() }).containsExactlyInAnyOrder(*versionStrings) + } + + companion object { + @JvmStatic + fun includeSnapshotsCases() = listOf( + Arguments.of("1.0.0-SNAPSHOT", emptySet()), + Arguments.of("1.0.0-RC1", emptySet()), + Arguments.of("1.0.0-alpha", emptySet()), + Arguments.of("1.0.0-beta.1", emptySet()), + Arguments.of("1.0.0-b2", emptySet()), + // skipVersions is still respected even when includeSnapshots=true + Arguments.of("1.0.0-SNAPSHOT", setOf("2.0.0")), + ) + } + + private fun createVersionRangeResult(vararg versionStrings: String): VersionRangeResult { + val artifact = DefaultArtifact("com.example:test:[1.0,)") + val request = VersionRangeRequest(artifact, emptyList(), null) + val versions = versionStrings.map { versionScheme.parseVersion(it) }.sorted() + // lowestVersion/highestVersion are computed as versions[0] and versions[last] + return VersionRangeResult(request).apply { this.versions = versions } + } +} + diff --git a/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/RangeQueryTest.kt b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/RangeQueryTest.kt new file mode 100644 index 00000000000..5ced9ed1032 --- /dev/null +++ b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/RangeQueryTest.kt @@ -0,0 +1,27 @@ +package datadog.gradle.plugin.muzzle + +import org.eclipse.aether.artifact.Artifact +import org.eclipse.aether.artifact.DefaultArtifact +import org.eclipse.aether.resolution.VersionRangeRequest +import org.assertj.core.api.Assertions.assertThat +import org.junit.jupiter.api.Test + +class RangeQueryTest { + private val system = MuzzleMavenRepoUtils.newRepositorySystem() + private val session = MuzzleMavenRepoUtils.newRepositorySystemSession(system) + + @Test + fun `test range request`() { + // compile group: 'org.codehaus.groovy', name: 'groovy-all', version: '2.5.0', ext: 'pom' + val directiveArtifact: Artifact = DefaultArtifact("org.codehaus.groovy", "groovy-all", "jar", "[2.5.0,2.5.8)") + val rangeRequest = VersionRangeRequest().apply { + repositories = MuzzleMavenRepoUtils.MUZZLE_REPOS + artifact = directiveArtifact + } + + // This call makes an actual network request, which may fail if network access is limited. + val rangeResult = system.resolveVersionRange(session, rangeRequest) + + assertThat(rangeResult.versions.size).isGreaterThanOrEqualTo(8) + } +} diff --git a/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/VersionSetTest.kt b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/VersionSetTest.kt new file mode 100644 index 00000000000..0d8c164e087 --- /dev/null +++ b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/VersionSetTest.kt @@ -0,0 +1,99 @@ +package datadog.gradle.plugin.muzzle + +import org.eclipse.aether.version.Version +import org.junit.jupiter.api.Test +import org.assertj.core.api.Assertions.assertThat + +class VersionSetTest { + + @Test + fun `parse versions properly`() { + data class Case( + val version: Version, + val versionNumber: Long, + val ending: String + ) + + val cases = listOf( + Case(ver("1.2.3"), num(1, 2, 3), ""), + Case(ver("4.5.6-foo"), num(4, 5, 6), "foo"), + Case(ver("7.8.9.foo"), num(7, 8, 9), "foo"), + Case(ver("10.11.12.foo-bar"), num(10, 11, 12), "foo-bar"), + Case(ver("13.14.foo-bar"), num(13, 14, 0), "foo-bar"), + Case(ver("15.foo"), num(15, 0, 0), "foo"), + Case(ver("16-foo"), num(16, 0, 0), "foo") + ) + + for (c in cases) { + val parsed = VersionSet.ParsedVersion(c.version) + assertThat(parsed.versionNumber).withFailMessage("versionNumber for ${c.version}").isEqualTo(c.versionNumber) + assertThat(parsed.ending).withFailMessage("ending for ${c.version}").isEqualTo(c.ending) + assertThat(parsed.majorMinor.toLong()) + .withFailMessage("majorMinor for ${c.version}") + .isEqualTo(c.versionNumber shr 12) + } + } + + @Test + fun `select low and high from major minor`() { + val versionsCases = listOf( + listOf( + ver("4.5.6"), + ver("1.2.3") + ), + listOf( + ver("1.2.3"), + ver("1.2.1"), + ver("1.3.0"), + ver("1.2.7"), + ver("1.4.17"), + ver("1.4.1"), + ver("1.4.0"), + ver("1.4.10") + ) + ) + + val expectedCases = listOf( + listOf( + ver("1.2.3"), + ver("4.5.6") + ), + listOf( + ver("1.2.1"), + ver("1.2.7"), + ver("1.3.0"), + ver("1.4.0"), + ver("1.4.17") + ) + ) + + versionsCases.zip(expectedCases).forEach { (versions, expected) -> + val versionSet = VersionSet(versions) + assertThat(versionSet.lowAndHighForMajorMinor).isEqualTo(expected) + } + } + + private fun ver(v: String): Version = TestVersion(v) + + private fun num(major: Int, minor: Int, micro: Int): Long { + var result = major.toLong() + result = (((result shl 12) + minor) shl 12) + micro + return result + } + + private class TestVersion(private val v: String) : Version { + override fun compareTo(other: Version?): Int { + if (other is TestVersion) { + return v.compareTo(other.v) + } + + return 1 + } + + override fun equals(other: Any?): Boolean = other is TestVersion && v == other.v + + override fun hashCode(): Int = v.hashCode() + + override fun toString(): String = v + } +} diff --git a/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/planner/MuzzleTaskPlannerTest.kt b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/planner/MuzzleTaskPlannerTest.kt new file mode 100644 index 00000000000..3dce77a8ad2 --- /dev/null +++ b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/planner/MuzzleTaskPlannerTest.kt @@ -0,0 +1,350 @@ +package datadog.gradle.plugin.muzzle.planner + +import datadog.gradle.plugin.muzzle.MuzzleDirective +import org.assertj.core.api.Assertions.assertThat +import org.eclipse.aether.artifact.Artifact +import org.eclipse.aether.artifact.DefaultArtifact +import org.junit.jupiter.api.Test + +class MuzzleTaskPlannerTest { + + @Test + fun `empty directives list returns empty plans`() { + val fakeService = FakeResolutionService() + + val plans = MuzzleTaskPlanner(fakeService).plan(emptyList()) + + assertThat(plans).isEmpty() + assertThat(fakeService.resolveCalls).isEqualTo(0) + assertThat(fakeService.inverseCalls).isEqualTo(0) + } + + @Test + fun `directive with no resolved artifacts returns empty plans`() { + val directive = MuzzleDirective().apply { + group = "com.example" + module = "nonexistent" + versions = "[99.0,100.0)" + assertPass = true + } + val fakeService = FakeResolutionService( + artifactsByDirective = mapOf(directive to emptySet()) + ) + + val plans = MuzzleTaskPlanner(fakeService).plan(listOf(directive)) + + assertThat(plans).isEmpty() + assertThat(fakeService.resolveCalls).isEqualTo(1) + assertThat(fakeService.inverseCalls).isEqualTo(0) + } + + @Test + fun `coreJdk directive does not call resolution service`() { + val directive = MuzzleDirective().apply { + assertPass = true + coreJdk() + } + val fakeService = FakeResolutionService() + + val plans = MuzzleTaskPlanner(fakeService).plan(listOf(directive)) + + assertThat(plans).containsExactly(MuzzleTaskPlan(directive, null)) + assertThat(fakeService.resolveCalls).isEqualTo(0) + assertThat(fakeService.inverseCalls).isEqualTo(0) + } + + @Test + fun `artifact directive creates one plan per resolved artifact version`() { + val directive = MuzzleDirective().apply { + group = "com.example" + module = "demo" + versions = "[1.0,2.0)" + assertPass = true + } + val artifacts = linkedSetOf( + artifact(version = "1.0.0"), + artifact(version = "1.1.0"), + artifact(version = "1.2.0"), + artifact(version = "1.3.0") + ) + val fakeService = FakeResolutionService( + artifactsByDirective = mapOf(directive to artifacts) + ) + + val plans = MuzzleTaskPlanner(fakeService).plan(listOf(directive)) + + assertThat(plans).containsExactly( + MuzzleTaskPlan(directive, artifact(version = "1.0.0")), + MuzzleTaskPlan(directive, artifact(version = "1.1.0")), + MuzzleTaskPlan(directive, artifact(version = "1.2.0")), + MuzzleTaskPlan(directive, artifact(version = "1.3.0")), + ) + assertThat(fakeService.resolveCalls).isEqualTo(1) + assertThat(fakeService.inverseCalls).isEqualTo(0) + } + + @Test + fun `multiple directives processed together preserves order`() { + val directive1 = MuzzleDirective().apply { + group = "com.example" + module = "first" + versions = "[1.0,2.0)" + assertPass = true + } + val directive2 = MuzzleDirective().apply { + group = "com.example" + module = "second" + versions = "[2.0,3.0)" + assertPass = true + } + val directive3 = MuzzleDirective().apply { + group = "com.example" + module = "third" + versions = "[3.0,4.0)" + assertPass = false + } + val fakeService = FakeResolutionService( + artifactsByDirective = mapOf( + directive1 to linkedSetOf(artifact("first", "1.5.0")), + directive2 to linkedSetOf(artifact("second", "2.5.0")), + directive3 to linkedSetOf(artifact("third", "3.5.0")) + ) + ) + + val plans = MuzzleTaskPlanner(fakeService).plan(listOf(directive1, directive2, directive3)) + + assertThat(plans).containsExactly( + MuzzleTaskPlan(directive1, artifact("first", "1.5.0")), + MuzzleTaskPlan(directive2, artifact("second", "2.5.0")), + MuzzleTaskPlan(directive3, artifact("third", "3.5.0")), + ) + assertThat(fakeService.resolveCalls).isEqualTo(3) + assertThat(fakeService.inverseCalls).isEqualTo(0) + } + + @Test + fun `assertInverse adds inverse plans on top of declared range plans`() { + val directive = MuzzleDirective().apply { + group = "com.example" + module = "demo" + versions = "[3.0,)" + assertPass = true + assertInverse = true + } + val inversedDirective = MuzzleDirective().apply { + group = "com.example" + module = "demo" + versions = "[2.7,3.0)" + assertPass = false + } + val directArtifactV1 = artifact(version = "3.12.13") + val directArtifactV2 = artifact(version = "4.4.1") + val directArtifactV3 = artifact(version = "5.3.2") + val inverseArtifactV1 = artifact(version = "2.7.5") + val inverseArtifactV2 = artifact(version = "2.8.1") + val fakeService = FakeResolutionService( + artifactsByDirective = mapOf( + directive to linkedSetOf(directArtifactV1, directArtifactV2, directArtifactV3), + inversedDirective to linkedSetOf(inverseArtifactV1, inverseArtifactV2) + ), + inverseByDirective = mapOf(directive to linkedSetOf(inversedDirective)) + ) + + val plans = MuzzleTaskPlanner(fakeService).plan(listOf(directive)) + + assertThat(plans).containsExactly( + MuzzleTaskPlan(directive, directArtifactV1), + MuzzleTaskPlan(directive, directArtifactV2), + MuzzleTaskPlan(directive, directArtifactV3), + MuzzleTaskPlan(inversedDirective, inverseArtifactV1), + MuzzleTaskPlan(inversedDirective, inverseArtifactV2), + ) + assertThat(fakeService.resolveCalls) + .withFailMessage("main directive + additional one for the inverse directive") + .isEqualTo(2) + assertThat(fakeService.inverseCalls).isEqualTo(1) + } + + @Test + fun `multiple artifacts with inverse creates comprehensive plan set`() { + val directive = MuzzleDirective().apply { + group = "io.netty" + module = "netty-codec-http" + versions = "[4.1.0,)" + assertPass = true + assertInverse = true + } + val inverseDirective = MuzzleDirective().apply { + group = "io.netty" + module = "netty-codec-http" + versions = "[4.0.0,4.1.0)" + assertPass = false + } + val passArtifacts = linkedSetOf( + artifact("netty-codec-http", "4.1.0"), + artifact("netty-codec-http", "4.1.50"), + artifact("netty-codec-http", "4.2.0") + ) + val failArtifacts = linkedSetOf( + artifact("netty-codec-http", "4.0.30") + ) + val fakeService = FakeResolutionService( + artifactsByDirective = mapOf( + directive to passArtifacts, + inverseDirective to failArtifacts + ), + inverseByDirective = mapOf( + directive to linkedSetOf(inverseDirective) + ) + ) + + val plans = MuzzleTaskPlanner(fakeService).plan(listOf(directive)) + + assertThat(plans).withFailMessage("Should have 3 pass plans + 1 inverse fail plan").hasSize(4) + assertThat(plans).containsExactly( + MuzzleTaskPlan(directive, artifact("netty-codec-http", "4.1.0")), + MuzzleTaskPlan(directive, artifact("netty-codec-http", "4.1.50")), + MuzzleTaskPlan(directive, artifact("netty-codec-http", "4.2.0")), + MuzzleTaskPlan(inverseDirective, artifact("netty-codec-http", "4.0.30")), + ) + assertThat(fakeService.resolveCalls).isEqualTo(2) + assertThat(fakeService.inverseCalls).isEqualTo(1) + } + + @Test + fun `mix of coreJdk and artifact directives`() { + val coreJdkDirective = MuzzleDirective().apply { + assertPass = true + coreJdk() + } + val artifactDirective = MuzzleDirective().apply { + group = "com.example" + module = "demo" + versions = "[1.0,2.0)" + assertPass = true + } + val fakeService = FakeResolutionService( + artifactsByDirective = mapOf( + artifactDirective to linkedSetOf(artifact("demo", "1.5.0")) + ) + ) + + val plans = MuzzleTaskPlanner(fakeService).plan(listOf(coreJdkDirective, artifactDirective)) + + assertThat(plans).containsExactly( + MuzzleTaskPlan(coreJdkDirective, null), + MuzzleTaskPlan(artifactDirective, artifact("demo", "1.5.0")), + ) + assertThat(fakeService.resolveCalls).isEqualTo(1) + assertThat(fakeService.inverseCalls).isEqualTo(0) + } + + @Test + fun `mix of pass and fail directives`() { + val passDirective = MuzzleDirective().apply { + group = "com.example" + module = "demo" + versions = "[2.0,)" + assertPass = true + } + val failDirective = MuzzleDirective().apply { + name = "before-2.0" + group = "com.example" + module = "demo" + versions = "[,2.0)" + assertPass = false + } + val fakeService = FakeResolutionService( + artifactsByDirective = mapOf( + passDirective to linkedSetOf(artifact("demo", "2.5.0"), artifact("demo", "3.0.0")), + failDirective to linkedSetOf(artifact("demo", "1.5.0")) + ) + ) + + val plans = MuzzleTaskPlanner(fakeService).plan(listOf(passDirective, failDirective)) + + assertThat(plans).containsExactly( + MuzzleTaskPlan(passDirective, artifact("demo", "2.5.0")), + MuzzleTaskPlan(passDirective, artifact("demo", "3.0.0")), + MuzzleTaskPlan(failDirective, artifact("demo", "1.5.0")), + ) + assertThat(fakeService.resolveCalls).isEqualTo(2) + assertThat(fakeService.inverseCalls).isEqualTo(0) + } + + @Test + fun `multiple directives with assertInverse`() { + val directive1 = MuzzleDirective().apply { + group = "com.example" + module = "first" + versions = "[3.0,)" + assertPass = true + assertInverse = true + } + val directive2 = MuzzleDirective().apply { + group = "com.example" + module = "second" + versions = "[2.0,)" + assertPass = true + assertInverse = true + } + val inverse1 = MuzzleDirective().apply { + group = "com.example" + module = "first" + versions = "[2.0,3.0)" + assertPass = false + } + val inverse2 = MuzzleDirective().apply { + group = "com.example" + module = "second" + versions = "[1.0,2.0)" + assertPass = false + } + val fakeService = FakeResolutionService( + artifactsByDirective = mapOf( + directive1 to linkedSetOf(artifact("first", "3.5.0")), + directive2 to linkedSetOf(artifact("second", "2.5.0")), + inverse1 to linkedSetOf(artifact("first", "2.5.0")), + inverse2 to linkedSetOf(artifact("second", "1.5.0")) + ), + inverseByDirective = mapOf( + directive1 to linkedSetOf(inverse1), + directive2 to linkedSetOf(inverse2) + ) + ) + + val plans = MuzzleTaskPlanner(fakeService).plan(listOf(directive1, directive2)) + + assertThat(plans).containsExactly( + MuzzleTaskPlan(directive1, artifact("first", "3.5.0")), + MuzzleTaskPlan(inverse1, artifact("first", "2.5.0")), + MuzzleTaskPlan(directive2, artifact("second", "2.5.0")), + MuzzleTaskPlan(inverse2, artifact("second", "1.5.0")), + ) + assertThat(fakeService.resolveCalls).isEqualTo(4) + assertThat(fakeService.inverseCalls).isEqualTo(2) + } + + private fun artifact(module: String = "demo", version: String) = + DefaultArtifact("com.example", module, "", "jar", version) + + private class FakeResolutionService( + private val artifactsByDirective: Map> = emptyMap(), + private val inverseByDirective: Map> = emptyMap(), + ) : MuzzleResolutionService { + var resolveCalls: Int = 0 + private set + var inverseCalls: Int = 0 + private set + + override fun resolveArtifacts(directive: MuzzleDirective): Set { + resolveCalls++ + return artifactsByDirective[directive].orEmpty() + } + + override fun inverseOf(directive: MuzzleDirective): Set { + inverseCalls++ + return inverseByDirective[directive].orEmpty() + } + } +} diff --git a/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/tasks/MuzzleEndTaskTest.kt b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/tasks/MuzzleEndTaskTest.kt new file mode 100644 index 00000000000..2c7dc920a30 --- /dev/null +++ b/buildSrc/src/test/kotlin/datadog/gradle/plugin/muzzle/tasks/MuzzleEndTaskTest.kt @@ -0,0 +1,122 @@ +package datadog.gradle.plugin.muzzle.tasks + +import org.gradle.kotlin.dsl.register +import org.gradle.testfixtures.ProjectBuilder +import org.junit.jupiter.api.BeforeEach +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.io.TempDir +import org.w3c.dom.Document +import org.w3c.dom.Element +import java.nio.file.Path +import javax.xml.parsers.DocumentBuilderFactory +import kotlin.io.path.createDirectories +import kotlin.io.path.readText +import kotlin.io.path.writeText +import org.assertj.core.api.Assertions.assertThat + +class MuzzleEndTaskTest { + + @TempDir + lateinit var tempDir: Path + + private lateinit var junitDoc: Document + private lateinit var legacyDoc: Document + + @BeforeEach + fun setup() { + val rootProject = ProjectBuilder.builder() + .withProjectDir(tempDir.toFile()) + .withName("root") + .build() + + val childProjectDir = tempDir.resolve("lettuce-5.0").createDirectories().toFile() + val project = ProjectBuilder.builder() + .withParent(rootProject) + .withName("lettuce-5.0") + .withProjectDir(childProjectDir) + .build() + + val passReportPath = project.layout.buildDirectory.file("reports/muzzle-pass.txt").get().asFile.toPath().apply { + parent.createDirectories() + writeText("PASSING") + } + + val failReportPath = project.layout.buildDirectory.file("reports/muzzle-fail.txt").get().asFile.toPath().apply { + parent.createDirectories() + writeText("java.lang.IllegalStateException: something is broken") + } + + val task = project.tasks.register("muzzle-end").get().apply { + startTimeMs.set(System.currentTimeMillis() - 1_000) + muzzleResultFiles.from(passReportPath.toFile(), failReportPath.toFile()) + } + + // Pre run the task + task.generatesResultFile() + + val junitReportXml = project.layout.buildDirectory + .file("test-results/muzzle/TEST-muzzle-lettuce-5.0.xml") + .get().asFile.toPath().readText() + junitDoc = parseXml(junitReportXml) + + val legacyReportXml = rootProject.layout.buildDirectory + .file("muzzle-test-results/lettuce-5.0_muzzle/results.xml") + .get().asFile.toPath().readText() + legacyDoc = parseXml(legacyReportXml) + } + + @Test + fun `junit report contains expected testsuite counters`() { + val suite = junitDoc.documentElement + assertThat(suite.tagName).isEqualTo("testsuite") + assertThat(suite.getAttribute("name")).isEqualTo(":lettuce-5.0") + assertThat(suite.getAttribute("tests")).isEqualTo("2") + assertThat(suite.getAttribute("failures")).isEqualTo("1") + assertThat(suite.getAttribute("errors")).isEqualTo("0") + assertThat(suite.getAttribute("skipped")).isEqualTo("0") + } + + @Test + fun `passed testcase has no failure node`() { + val passedTestCase = findTestCaseByName(junitDoc, "muzzle-pass") + assertThat(passedTestCase).isNotNull() + assertThat(passedTestCase.getElementsByTagName("failure").item(0)).isNull() + } + + @Test + fun `failed testcase contains failure node and message`() { + val failedTestCase = findTestCaseByName(junitDoc, "muzzle-fail") + assertThat(failedTestCase).isNotNull() + val failureNode = failedTestCase.getElementsByTagName("failure").item(0) as Element + assertThat(failureNode.getAttribute("message")).isEqualTo("Muzzle validation failed") + assertThat(failureNode.textContent).isEqualTo("java.lang.IllegalStateException: something is broken") + } + + @Test + fun `legacy report keeps historical shape`() { + val legacySuite = legacyDoc.documentElement + assertThat(legacySuite.tagName).isEqualTo("testsuite") + assertThat(legacySuite.getAttribute("tests")).isEqualTo("1") + assertThat(legacySuite.getAttribute("id")).isEqualTo("0") + assertThat(legacySuite.getAttribute("name")).isEqualTo("muzzle-end") + assertThat(legacySuite.getElementsByTagName("testcase").length).isEqualTo(1) + } + + private fun parseXml(xml: String): Document { + val builderFactory = DocumentBuilderFactory.newInstance() + builderFactory.isNamespaceAware = false + builderFactory.isIgnoringComments = true + return builderFactory.newDocumentBuilder().parse(xml.byteInputStream()) + } + + private fun findTestCaseByName(document: Document, name: String): Element { + val testCases = document.getElementsByTagName("testcase") + for (idx in 0 until testCases.length) { + val testCase = testCases.item(idx) as Element + if (testCase.getAttribute("name") == name) { + return testCase + } + } + throw IllegalStateException("Could not find testcase with name '$name'") + } +} diff --git a/buildSrc/src/integTest/kotlin/datadog/gradle/plugin/version/TracerVersionIntegrationTest.kt b/buildSrc/src/test/kotlin/datadog/gradle/plugin/version/TracerVersionIntegrationTest.kt similarity index 92% rename from buildSrc/src/integTest/kotlin/datadog/gradle/plugin/version/TracerVersionIntegrationTest.kt rename to buildSrc/src/test/kotlin/datadog/gradle/plugin/version/TracerVersionIntegrationTest.kt index 15fa42968c4..ac65bdf0910 100644 --- a/buildSrc/src/integTest/kotlin/datadog/gradle/plugin/version/TracerVersionIntegrationTest.kt +++ b/buildSrc/src/test/kotlin/datadog/gradle/plugin/version/TracerVersionIntegrationTest.kt @@ -41,10 +41,12 @@ class TracerVersionIntegrationTest { exec(projectDir, "git", "add", "-A") exec(projectDir, "git", "commit", "-m", "A commit") - File(projectDir, "settings.gradle.kts").appendText(""" + File(projectDir, "settings.gradle.kts").appendText( + """ // uncommitted change this file, - """.trimIndent()) + """.trimIndent() + ) } ) } @@ -101,10 +103,12 @@ class TracerVersionIntegrationTest { exec(projectDir, "git", "commit", "-m", "A commit") exec(projectDir, "git", "tag", "v1.52.0", "-m", "") - File(projectDir, "settings.gradle.kts").appendText(""" + File(projectDir, "settings.gradle.kts").appendText( + """ // uncommitted change this file, - """.trimIndent()) + """.trimIndent() + ) } ) } @@ -126,7 +130,7 @@ class TracerVersionIntegrationTest { """ // Committed change this file, - """.trimIndent() + """.trimIndent() ) exec(projectDir, "git", "commit", "-am", "Another commit") } @@ -153,16 +157,20 @@ class TracerVersionIntegrationTest { exec(projectDir, "git", "tag", "v1.52.0", "-m", "") val settingsFile = File(projectDir, "settings.gradle.kts") - settingsFile.appendText(""" + settingsFile.appendText( + """ // uncommitted change - """.trimIndent()) + """.trimIndent() + ) exec(projectDir, "git", "commit", "-am", "Another commit") - settingsFile.appendText(""" + settingsFile.appendText( + """ // An uncommitted modification - """.trimIndent()) + """.trimIndent() + ) } ) } @@ -181,10 +189,12 @@ class TracerVersionIntegrationTest { exec(projectDir, "git", "tag", "v1.52.0", "-m", "") val settingsFile = File(projectDir, "settings.gradle.kts") - settingsFile.appendText(""" + settingsFile.appendText( + """ // Committed change - """.trimIndent()) + """.trimIndent() + ) exec(projectDir, "git", "commit", "-am", "Another commit") exec(projectDir, "git", "switch", "-c", "release/v1.52.x") @@ -207,17 +217,21 @@ class TracerVersionIntegrationTest { exec(projectDir, "git", "switch", "-c", "release/v1.52.x") val settingsFile = File(projectDir, "settings.gradle.kts") - settingsFile.appendText(""" + settingsFile.appendText( + """ // Committed change - """.trimIndent()) + """.trimIndent() + ) exec(projectDir, "git", "commit", "-am", "Another commit") exec(projectDir, "git", "tag", "v1.52.1", "-m", "") - settingsFile.appendText(""" + settingsFile.appendText( + """ // Another committed change - """.trimIndent()) + """.trimIndent() + ) exec(projectDir, "git", "commit", "-am", "Another commit") } ) @@ -243,7 +257,7 @@ class TracerVersionIntegrationTest { """ // Committed change this file, - """.trimIndent() + """.trimIndent() ) exec(workTreeDir, "git", "commit", "-am", "Another commit") }, @@ -265,7 +279,7 @@ class TracerVersionIntegrationTest { File(projectDir, "build.gradle.kts").writeText( """ plugins { - id("datadog.tracer-version") + id("dd-trace-java.tracer-version") } tasks.register("printVersion") { diff --git a/communication/build.gradle.kts b/communication/build.gradle.kts index 4b631161bd6..b5f31fadb68 100644 --- a/communication/build.gradle.kts +++ b/communication/build.gradle.kts @@ -13,13 +13,16 @@ dependencies { implementation(project(":remote-config:remote-config-core")) implementation(project(":internal-api")) implementation(project(":utils:container-utils")) + implementation(project(":utils:filesystem-utils")) implementation(project(":utils:socket-utils")) implementation(project(":utils:version-utils")) api(libs.okio) api(libs.okhttp) api(libs.moshi) - implementation(libs.dogstatsd) + // metrics-lib is needed rather than metrics-api to change the default port of StatsD connection manager + // TODO Could help decoupling it later to only depend on metrics-api + implementation(project(":products:metrics:metrics-lib")) testImplementation(project(":utils:test-utils")) testImplementation(libs.bundles.junit5) @@ -48,9 +51,6 @@ val excludedClassesCoverage by extra( "datadog.communication.http.OkHttpUtils.GZipByteBufferRequestBody", "datadog.communication.http.OkHttpUtils.GZipRequestBodyDecorator", "datadog.communication.http.OkHttpUtils.JsonRequestBody", - "datadog.communication.monitor.DDAgentStatsDConnection", - "datadog.communication.monitor.DDAgentStatsDConnection.*", - "datadog.communication.monitor.LoggingStatsDClient", "datadog.communication.BackendApiFactory", "datadog.communication.BackendApiFactory.Intake", "datadog.communication.EvpProxyApi", diff --git a/communication/gradle.lockfile b/communication/gradle.lockfile index b392eda1d49..b8c84ea04db 100644 --- a/communication/gradle.lockfile +++ b/communication/gradle.lockfile @@ -5,99 +5,63 @@ cafe.cryptography:curve25519-elisabeth:0.1.0=runtimeClasspath,testRuntimeClasspa cafe.cryptography:ed25519-elisabeth:0.1.0=runtimeClasspath,testRuntimeClasspath ch.qos.logback:logback-classic:1.2.13=testCompileClasspath,testRuntimeClasspath ch.qos.logback:logback-core:1.2.13=testCompileClasspath,testRuntimeClasspath -com.beust:jcommander:1.78=testRuntimeClasspath com.datadoghq.okhttp3:okhttp:3.12.15=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.datadoghq.okio:okio:1.17.6=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.datadoghq:dd-javac-plugin-client:0.2.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.datadoghq:java-dogstatsd-client:4.4.3=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +com.datadoghq:java-dogstatsd-client:4.4.5=runtimeClasspath,testRuntimeClasspath +com.datadoghq:sketches-java:0.8.3=runtimeClasspath,testRuntimeClasspath com.fasterxml.jackson.core:jackson-annotations:2.9.0=testCompileClasspath,testRuntimeClasspath com.fasterxml.jackson.core:jackson-core:2.9.9=testCompileClasspath,testRuntimeClasspath com.fasterxml.jackson.core:jackson-databind:2.9.9.3=testCompileClasspath,testRuntimeClasspath -com.github.javaparser:javaparser-core:3.25.6=codenarc,testCompileClasspath,testRuntimeClasspath -com.github.jnr:jffi:1.2.23=compileClasspath,testCompileClasspath -com.github.jnr:jffi:1.3.13=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-a64asm:1.0.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +com.github.javaparser:javaparser-core:3.25.6=codenarc +com.github.jnr:jffi:1.3.14=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-a64asm:1.0.0=runtimeClasspath,testRuntimeClasspath com.github.jnr:jnr-constants:0.10.4=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-constants:0.9.17=compileClasspath,testCompileClasspath -com.github.jnr:jnr-enxio:0.30=compileClasspath,testCompileClasspath -com.github.jnr:jnr-enxio:0.32.17=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-ffi:2.1.16=compileClasspath,testCompileClasspath -com.github.jnr:jnr-ffi:2.2.16=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-posix:3.0.61=compileClasspath,testCompileClasspath -com.github.jnr:jnr-posix:3.1.19=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-unixsocket:0.36=compileClasspath,testCompileClasspath -com.github.jnr:jnr-unixsocket:0.38.22=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-x86asm:1.0.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.2.0=compileClasspath,testCompileClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.7.3=spotbugs -com.github.spotbugs:spotbugs:4.7.3=spotbugs +com.github.jnr:jnr-enxio:0.32.19=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-ffi:2.2.18=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-posix:3.1.21=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-unixsocket:0.38.24=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-x86asm:1.0.2=runtimeClasspath,testRuntimeClasspath +com.github.spotbugs:spotbugs-annotations:4.9.8=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath +com.github.spotbugs:spotbugs:4.9.8=spotbugs +com.github.stephenc.jcip:jcip-annotations:1.0-1=spotbugs com.google.code.findbugs:jsr305:3.0.2=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -com.google.code.gson:gson:2.9.1=spotbugs +com.google.code.gson:gson:2.13.2=spotbugs +com.google.errorprone:error_prone_annotations:2.41.0=spotbugs com.squareup.moshi:moshi:1.11.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.squareup.okhttp3:mockwebserver:3.12.12=testCompileClasspath,testRuntimeClasspath com.squareup.okhttp3:okhttp:3.12.12=testCompileClasspath,testRuntimeClasspath com.squareup.okio:okio:1.17.5=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.thoughtworks.qdox:qdox:1.12.1=codenarc,testRuntimeClasspath -commons-codec:commons-codec:1.15=spotbugs +com.thoughtworks.qdox:qdox:1.12.1=codenarc commons-fileupload:commons-fileupload:1.5=testCompileClasspath,testRuntimeClasspath commons-io:commons-io:2.11.0=testCompileClasspath,testRuntimeClasspath -de.thetaphi:forbiddenapis:3.8=compileClasspath -info.picocli:picocli:4.6.3=testRuntimeClasspath +commons-io:commons-io:2.20.0=spotbugs +de.thetaphi:forbiddenapis:3.10=compileClasspath,testCompileClasspath,testRuntimeClasspath io.leangen.geantyref:geantyref:1.3.16=testRuntimeClasspath -jaxen:jaxen:1.2.0=spotbugs -jline:jline:2.14.6=testRuntimeClasspath -junit:junit:4.13.2=testCompileClasspath,testRuntimeClasspath -net.bytebuddy:byte-buddy-agent:1.17.7=testCompileClasspath,testRuntimeClasspath -net.bytebuddy:byte-buddy:1.17.7=testCompileClasspath,testRuntimeClasspath -net.jcip:jcip-annotations:1.0=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -net.sf.saxon:Saxon-HE:11.4=spotbugs +jaxen:jaxen:2.0.0=spotbugs +junit:junit:4.12=testCompileClasspath,testRuntimeClasspath +net.bytebuddy:byte-buddy-agent:1.18.8=testCompileClasspath,testRuntimeClasspath +net.bytebuddy:byte-buddy:1.18.8=testCompileClasspath,testRuntimeClasspath +net.sf.saxon:Saxon-HE:12.9=spotbugs org.apache.ant:ant-antlr:1.10.14=codenarc -org.apache.ant:ant-antlr:1.10.15=testRuntimeClasspath org.apache.ant:ant-junit:1.10.14=codenarc -org.apache.ant:ant-junit:1.10.15=testRuntimeClasspath -org.apache.ant:ant-launcher:1.10.15=testRuntimeClasspath -org.apache.ant:ant:1.10.15=testCompileClasspath,testRuntimeClasspath -org.apache.bcel:bcel:6.5.0=spotbugs -org.apache.commons:commons-lang3:3.12.0=spotbugs -org.apache.commons:commons-text:1.10.0=spotbugs -org.apache.httpcomponents.client5:httpclient5:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5-h2:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5:5.1.3=spotbugs -org.apache.logging.log4j:log4j-api:2.19.0=spotbugs -org.apache.logging.log4j:log4j-core:2.19.0=spotbugs +org.apache.bcel:bcel:6.11.0=spotbugs +org.apache.commons:commons-lang3:3.19.0=spotbugs +org.apache.commons:commons-text:1.14.0=spotbugs +org.apache.logging.log4j:log4j-api:2.25.2=spotbugs +org.apache.logging.log4j:log4j-core:2.25.2=spotbugs org.apiguardian:apiguardian-api:1.1.2=testCompileClasspath -org.codehaus.groovy:groovy-all:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-ant:3.0.23=codenarc -org.codehaus.groovy:groovy-ant:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-astbuilder:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-cli-picocli:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-console:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-datetime:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-docgenerator:3.0.23=codenarc -org.codehaus.groovy:groovy-docgenerator:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-groovydoc:3.0.23=codenarc -org.codehaus.groovy:groovy-groovydoc:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-groovysh:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jmx:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-json:3.0.23=codenarc -org.codehaus.groovy:groovy-json:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jsr223:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-macro:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-nio:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-servlet:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-sql:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-swing:3.0.24=testCompileClasspath,testRuntimeClasspath +org.codehaus.groovy:groovy-json:3.0.25=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-templates:3.0.23=codenarc -org.codehaus.groovy:groovy-templates:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test-junit5:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-testng:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-xml:3.0.23=codenarc -org.codehaus.groovy:groovy-xml:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy:3.0.23=codenarc -org.codehaus.groovy:groovy:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codenarc:CodeNarc:3.6.0=codenarc -org.dom4j:dom4j:2.1.3=spotbugs +org.codehaus.groovy:groovy:3.0.25=testCompileClasspath,testRuntimeClasspath +org.codenarc:CodeNarc:3.7.0=codenarc +org.dom4j:dom4j:2.2.0=spotbugs org.gmetrics:GMetrics:2.1.0=codenarc org.hamcrest:hamcrest-core:1.3=testCompileClasspath,testRuntimeClasspath org.hamcrest:hamcrest:3.0=testCompileClasspath,testRuntimeClasspath @@ -105,50 +69,41 @@ org.jacoco:org.jacoco.agent:0.8.14=jacocoAgent,jacocoAnt org.jacoco:org.jacoco.ant:0.8.14=jacocoAnt org.jacoco:org.jacoco.core:0.8.14=jacocoAnt org.jacoco:org.jacoco.report:0.8.14=jacocoAnt -org.junit.jupiter:junit-jupiter-api:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-engine:5.12.2=testRuntimeClasspath -org.junit.jupiter:junit-jupiter-params:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-commons:1.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-engine:1.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-launcher:1.12.2=testRuntimeClasspath -org.junit:junit-bom:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit:junit-bom:5.9.1=spotbugs +org.junit.jupiter:junit-jupiter-api:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-engine:5.14.1=testRuntimeClasspath +org.junit.jupiter:junit-jupiter-params:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-commons:1.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-engine:1.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-launcher:1.14.1=testRuntimeClasspath +org.junit:junit-bom:5.14.0=spotbugs +org.junit:junit-bom:5.14.1=testCompileClasspath,testRuntimeClasspath org.mockito:mockito-core:4.4.0=testRuntimeClasspath org.msgpack:jackson-dataformat-msgpack:0.8.20=testCompileClasspath,testRuntimeClasspath org.msgpack:msgpack-core:0.8.20=testCompileClasspath,testRuntimeClasspath org.objenesis:objenesis:3.3=testCompileClasspath,testRuntimeClasspath org.opentest4j:opentest4j:1.3.0=testCompileClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:7.1=compileClasspath,testCompileClasspath -org.ow2.asm:asm-analysis:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.4=spotbugs -org.ow2.asm:asm-commons:7.1=compileClasspath,testCompileClasspath -org.ow2.asm:asm-commons:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-commons:9.4=spotbugs -org.ow2.asm:asm-commons:9.9=jacocoAnt -org.ow2.asm:asm-tree:7.1=compileClasspath,testCompileClasspath -org.ow2.asm:asm-tree:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-tree:9.4=spotbugs -org.ow2.asm:asm-tree:9.9=jacocoAnt -org.ow2.asm:asm-util:7.1=compileClasspath,testCompileClasspath -org.ow2.asm:asm-util:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-util:9.4=spotbugs -org.ow2.asm:asm:7.1=compileClasspath,testCompileClasspath -org.ow2.asm:asm:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm:9.4=spotbugs -org.ow2.asm:asm:9.9=jacocoAnt +org.ow2.asm:asm-analysis:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-analysis:9.9=spotbugs +org.ow2.asm:asm-commons:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-commons:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-tree:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-tree:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-util:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-util:9.9=spotbugs +org.ow2.asm:asm:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm:9.9=jacocoAnt,spotbugs org.slf4j:jcl-over-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath org.slf4j:jul-to-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath org.slf4j:log4j-over-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath org.slf4j:slf4j-api:1.7.30=compileClasspath,runtimeClasspath org.slf4j:slf4j-api:1.7.32=testCompileClasspath,testRuntimeClasspath -org.slf4j:slf4j-api:2.0.0=spotbugs,spotbugsSlf4j -org.slf4j:slf4j-simple:2.0.0=spotbugsSlf4j +org.slf4j:slf4j-api:2.0.17=spotbugs,spotbugsSlf4j +org.slf4j:slf4j-simple:2.0.17=spotbugsSlf4j org.snakeyaml:snakeyaml-engine:2.9=runtimeClasspath,testRuntimeClasspath -org.spockframework:spock-bom:2.4-M6-groovy-3.0=testCompileClasspath,testRuntimeClasspath -org.spockframework:spock-core:2.4-M6-groovy-3.0=testCompileClasspath,testRuntimeClasspath -org.testng:testng:7.5.1=testRuntimeClasspath -org.webjars:jquery:3.5.1=testRuntimeClasspath -org.xmlresolver:xmlresolver:4.4.3=spotbugs -xml-apis:xml-apis:1.4.01=spotbugs +org.spockframework:spock-bom:2.4-groovy-3.0=testCompileClasspath,testRuntimeClasspath +org.spockframework:spock-core:2.4-groovy-3.0=testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-junit:1.2.1=testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-parser:1.2.0=testCompileClasspath,testRuntimeClasspath +org.xmlresolver:xmlresolver:5.3.3=spotbugs empty=annotationProcessor,spotbugsPlugins,testAnnotationProcessor diff --git a/communication/src/main/java/datadog/communication/EvpProxyApi.java b/communication/src/main/java/datadog/communication/EvpProxyApi.java index 1ea6e79dbf8..83037ab9663 100644 --- a/communication/src/main/java/datadog/communication/EvpProxyApi.java +++ b/communication/src/main/java/datadog/communication/EvpProxyApi.java @@ -42,7 +42,7 @@ public EvpProxyApi( OkHttpClient httpClient, boolean responseCompression) { this.traceId = traceId; - this.evpProxyUrl = evpProxyUrl.resolve(String.format("api/%s/", API_VERSION)); + this.evpProxyUrl = evpProxyUrl.resolve("api/" + API_VERSION + "/"); this.subdomain = subdomain; this.retryPolicyFactory = retryPolicyFactory; this.httpClient = httpClient; diff --git a/communication/src/main/java/datadog/communication/ddagent/AgentVersion.java b/communication/src/main/java/datadog/communication/ddagent/AgentVersion.java new file mode 100644 index 00000000000..f3b30cb7baf --- /dev/null +++ b/communication/src/main/java/datadog/communication/ddagent/AgentVersion.java @@ -0,0 +1,72 @@ +package datadog.communication.ddagent; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class AgentVersion { + + private static final Logger log = LoggerFactory.getLogger(AgentVersion.class); + + /** + * Checks if the given version string represents a version that is below the specified major, + * minor, and patch version. + * + * @param version the version string to check (e.g., "7.64.0") + * @param maxMajor maximum major version (exclusive) + * @param maxMinor maximum minor version (exclusive) + * @param maxPatch maximum patch version (exclusive) + * @return true if version is below the specified maximum (or if not parseable), false otherwise + */ + public static boolean isVersionBelow(String version, int maxMajor, int maxMinor, int maxPatch) { + if (version == null || version.isEmpty()) { + return true; + } + + try { + // Parse version string in format "major.minor.patch" (e.g., "7.65.0") + // Assumes the 'version' is below if it can't be parsed. + int majorDot = version.indexOf('.'); + if (majorDot == -1) { + return true; + } + + int major = Integer.parseInt(version.substring(0, majorDot)); + + if (major < maxMajor) { + return true; + } else if (major > maxMajor) { + return false; + } + + // major == maxMajor + int minorDot = version.indexOf('.', majorDot + 1); + if (minorDot == -1) { + return true; + } + + int minor = Integer.parseInt(version.substring(majorDot + 1, minorDot)); + if (minor < maxMinor) { + return true; + } else if (minor > maxMinor) { + return false; + } + + // major == maxMajor && minor == maxMinor + // Find end of patch version (may have suffix like "-rc.1") + int patchEnd = minorDot + 1; + while (patchEnd < version.length() && Character.isDigit(version.charAt(patchEnd))) { + patchEnd++; + } + + int patch = Integer.parseInt(version.substring(minorDot + 1, patchEnd)); + if (patch != maxPatch) { + return patch < maxPatch; + } else { + // If there's a suffix (like "-rc.1"), consider it below the non-suffixed version + return patchEnd < version.length(); + } + } catch (NumberFormatException | IndexOutOfBoundsException e) { + return true; + } + } +} diff --git a/communication/src/main/java/datadog/communication/ddagent/DDAgentFeaturesDiscovery.java b/communication/src/main/java/datadog/communication/ddagent/DDAgentFeaturesDiscovery.java index 840a11c0b4c..54b1e146b5b 100644 --- a/communication/src/main/java/datadog/communication/ddagent/DDAgentFeaturesDiscovery.java +++ b/communication/src/main/java/datadog/communication/ddagent/DDAgentFeaturesDiscovery.java @@ -1,8 +1,11 @@ package datadog.communication.ddagent; -import static datadog.communication.http.OkHttpUtils.DATADOG_CONTAINER_ID; import static datadog.communication.http.OkHttpUtils.DATADOG_CONTAINER_TAGS_HASH; +import static datadog.communication.http.OkHttpUtils.msgpackRequestBodyOf; +import static datadog.communication.http.OkHttpUtils.prepareRequest; import static datadog.communication.serialization.msgpack.MsgPackWriter.FIXARRAY; +import static datadog.trace.api.ProtocolVersion.V0_4; +import static java.util.Collections.emptyMap; import static java.util.Collections.emptySet; import static java.util.Collections.singletonList; import static java.util.Collections.unmodifiableSet; @@ -11,15 +14,14 @@ import com.squareup.moshi.Moshi; import com.squareup.moshi.Types; import datadog.common.container.ContainerInfo; -import datadog.communication.http.OkHttpUtils; -import datadog.communication.monitor.DDAgentStatsDClientManager; -import datadog.communication.monitor.Monitoring; -import datadog.communication.monitor.Recording; +import datadog.metrics.api.Monitoring; +import datadog.metrics.api.Recording; +import datadog.metrics.impl.statsd.DDAgentStatsDClientManager; import datadog.trace.api.BaseHash; +import datadog.trace.api.ProtocolVersion; import datadog.trace.api.telemetry.LogCollector; import datadog.trace.util.Strings; import java.nio.ByteBuffer; -import java.security.NoSuchAlgorithmException; import java.util.HashSet; import java.util.List; import java.util.Map; @@ -46,12 +48,13 @@ public class DDAgentFeaturesDiscovery implements DroppingPolicy { (byte) FIXARRAY | 2, (byte) FIXARRAY, (byte) FIXARRAY }; - public static final String V3_ENDPOINT = "v0.3/traces"; - public static final String V4_ENDPOINT = "v0.4/traces"; - public static final String V5_ENDPOINT = "v0.5/traces"; + public static final String V03_ENDPOINT = "v0.3/traces"; + public static final String V04_ENDPOINT = "v0.4/traces"; + public static final String V05_ENDPOINT = "v0.5/traces"; + public static final String V1_ENDPOINT = "v1.0/traces"; - public static final String V6_METRICS_ENDPOINT = "v0.6/stats"; - public static final String V7_CONFIG_ENDPOINT = "v0.7/config"; + public static final String V06_METRICS_ENDPOINT = "v0.6/stats"; + public static final String V07_CONFIG_ENDPOINT = "v0.7/config"; public static final String V01_DATASTREAMS_ENDPOINT = "v0.1/pipeline_stats"; @@ -71,9 +74,9 @@ public class DDAgentFeaturesDiscovery implements DroppingPolicy { private final OkHttpClient client; private final HttpUrl agentBaseUrl; private final Recording discoveryTimer; - private final String[] traceEndpoints; - private final String[] metricsEndpoints = {V6_METRICS_ENDPOINT}; - private final String[] configEndpoints = {V7_CONFIG_ENDPOINT}; + private final ProtocolVersion protocolVersion; + private final String[] metricsEndpoints = {V06_METRICS_ENDPOINT}; + private final String[] configEndpoints = {V07_CONFIG_ENDPOINT}; private final boolean metricsEnabled; private final String[] dataStreamsEndpoints = {V01_DATASTREAMS_ENDPOINT}; // ordered from most recent to least recent, as the logic will stick with the first one that is @@ -86,6 +89,7 @@ private static class State { String metricsEndpoint; String dataStreamsEndpoint; boolean supportsLongRunning; + boolean supportsClientSideStats; boolean supportsDropping; String state; String configEndpoint; @@ -105,15 +109,12 @@ public DDAgentFeaturesDiscovery( OkHttpClient client, Monitoring monitoring, HttpUrl agentUrl, - boolean enableV05Traces, + ProtocolVersion protocolVersion, boolean metricsEnabled) { this.client = client; this.agentBaseUrl = agentUrl; this.metricsEnabled = metricsEnabled; - this.traceEndpoints = - enableV05Traces - ? new String[] {V5_ENDPOINT, V4_ENDPOINT, V3_ENDPOINT} - : new String[] {V4_ENDPOINT, V3_ENDPOINT}; + this.protocolVersion = protocolVersion != null ? protocolVersion : V0_4; this.discoveryTimer = monitoring.newTimer("trace.agent.discovery.time"); this.discoveryState = new State(); } @@ -150,13 +151,9 @@ private void doDiscovery(State newState) { // 3. fallback if the endpoint couldn't be found or the response couldn't be parsed try (Recording recording = discoveryTimer.start()) { boolean fallback = true; - final Request.Builder requestBuilder = - new Request.Builder().url(agentBaseUrl.resolve("info").url()); - final String containerId = ContainerInfo.get().getContainerId(); - if (containerId != null) { - requestBuilder.header(DATADOG_CONTAINER_ID, containerId); - } - try (Response response = client.newCall(requestBuilder.build()).execute()) { + final Request request = + prepareRequest(agentBaseUrl.resolve("info"), emptyMap()).get().build(); + try (Response response = client.newCall(request).execute()) { if (response.isSuccessful()) { processInfoResponseHeaders(response); fallback = !processInfoResponse(newState, response.body().string()); @@ -165,7 +162,7 @@ private void doDiscovery(State newState) { errorQueryingEndpoint("info", error); } if (fallback) { - newState.supportsDropping = false; + newState.supportsClientSideStats = false; newState.supportsLongRunning = false; log.debug("Falling back to probing, client dropping will be disabled"); // disable metrics unless the info endpoint is present, which prevents @@ -175,10 +172,10 @@ private void doDiscovery(State newState) { // don't want to rewire the traces pipeline if (null == newState.traceEndpoint) { - newState.traceEndpoint = probeTracesEndpoint(newState, traceEndpoints); + newState.traceEndpoint = probeTracesEndpoint(newState, protocolVersion.endpointsToProbe()); } else if (newState.state == null || newState.state.isEmpty()) { // Still need to probe so that state is correctly assigned - probeTracesEndpoint(newState, new String[] {newState.traceEndpoint}); + probeTracesEndpoint(newState, singletonList(newState.traceEndpoint)); } } @@ -196,16 +193,13 @@ private void doDiscovery(State newState) { } } - private String probeTracesEndpoint(State newState, String[] endpoints) { + private String probeTracesEndpoint(State newState, List endpoints) { for (String candidate : endpoints) { try (Response response = client .newCall( - new Request.Builder() - .put( - OkHttpUtils.msgpackRequestBodyOf( - singletonList(ByteBuffer.wrap(PROBE_MESSAGE)))) - .url(agentBaseUrl.resolve(candidate)) + prepareRequest(agentBaseUrl.resolve(candidate), emptyMap()) + .put(msgpackRequestBodyOf(singletonList(ByteBuffer.wrap(PROBE_MESSAGE)))) .build()) .execute()) { if (response.code() != 404) { @@ -216,7 +210,7 @@ private String probeTracesEndpoint(State newState, String[] endpoints) { errorQueryingEndpoint(candidate, e); } } - return V3_ENDPOINT; + return V03_ENDPOINT; } private void processInfoResponseHeaders(Response response) { @@ -236,9 +230,14 @@ private void processInfoResponseHeaders(Response response) { private boolean processInfoResponse(State newState, String response) { try { Map map = RESPONSE_ADAPTER.fromJson(response); + final Object endpointObj = map.get("endpoints"); + if (!(endpointObj instanceof List)) { + log.debug("Bad response received from the agent. Ignoring it."); + return false; + } discoverStatsDPort(map); newState.version = (String) map.get("version"); - Set endpoints = new HashSet<>((List) map.get("endpoints")); + Set endpoints = new HashSet<>((List) endpointObj); String foundMetricsEndpoint = null; if (metricsEnabled) { @@ -253,7 +252,7 @@ private boolean processInfoResponse(State newState, String response) { // This is done outside of the loop to set metricsEndpoint to null if not found newState.metricsEndpoint = foundMetricsEndpoint; - for (String endpoint : traceEndpoints) { + for (String endpoint : protocolVersion.endpointsToProbe()) { if (containsEndpoint(endpoints, endpoint)) { newState.traceEndpoint = endpoint; break; @@ -267,19 +266,7 @@ private boolean processInfoResponse(State newState, String response) { } } - if (containsEndpoint(endpoints, DEBUGGER_ENDPOINT_V1)) { - newState.debuggerLogEndpoint = DEBUGGER_ENDPOINT_V1; - } - // both debugger v2 and diagnostics endpoints are forwarding events to the DEBUGGER intake - // because older agents support diagnostics from DD agent 7.49 - if (containsEndpoint(endpoints, DEBUGGER_ENDPOINT_V2)) { - newState.debuggerSnapshotEndpoint = DEBUGGER_ENDPOINT_V2; - } else if (containsEndpoint(endpoints, DEBUGGER_DIAGNOSTICS_ENDPOINT)) { - newState.debuggerSnapshotEndpoint = DEBUGGER_DIAGNOSTICS_ENDPOINT; - } - if (containsEndpoint(endpoints, DEBUGGER_DIAGNOSTICS_ENDPOINT)) { - newState.debuggerDiagnosticsEndpoint = DEBUGGER_DIAGNOSTICS_ENDPOINT; - } + setDebuggerEndpoints(newState, endpoints); for (String endpoint : dataStreamsEndpoints) { if (containsEndpoint(endpoints, endpoint)) { @@ -312,6 +299,9 @@ private boolean processInfoResponse(State newState, String response) { && ("true".equalsIgnoreCase(String.valueOf(canDrop)) || Boolean.TRUE.equals(canDrop)); + newState.supportsClientSideStats = + newState.supportsDropping && !AgentVersion.isVersionBelow(newState.version, 7, 65, 0); + Object peer_tags = map.get("peer_tags"); newState.peerTags = peer_tags instanceof List @@ -320,7 +310,7 @@ private boolean processInfoResponse(State newState, String response) { } try { newState.state = Strings.sha256(response); - } catch (NoSuchAlgorithmException ex) { + } catch (Throwable ex) { log.debug( "Failed to hash trace agent /info response. Will probe {}", newState.traceEndpoint, ex); } @@ -331,6 +321,26 @@ private boolean processInfoResponse(State newState, String response) { return false; } + private static void setDebuggerEndpoints(State newState, Set endpoints) { + // both debugger v2 and diagnostics endpoints are forwarding events to the DEBUGGER intake + // because older agents support diagnostics from DD agent 7.49 + if (containsEndpoint(endpoints, DEBUGGER_ENDPOINT_V2)) { + newState.debuggerLogEndpoint = DEBUGGER_ENDPOINT_V2; + } else if (containsEndpoint(endpoints, DEBUGGER_DIAGNOSTICS_ENDPOINT)) { + newState.debuggerLogEndpoint = DEBUGGER_DIAGNOSTICS_ENDPOINT; + } else if (containsEndpoint(endpoints, DEBUGGER_ENDPOINT_V1)) { + newState.debuggerLogEndpoint = DEBUGGER_ENDPOINT_V1; + } + if (containsEndpoint(endpoints, DEBUGGER_ENDPOINT_V2)) { + newState.debuggerSnapshotEndpoint = DEBUGGER_ENDPOINT_V2; + } else if (containsEndpoint(endpoints, DEBUGGER_DIAGNOSTICS_ENDPOINT)) { + newState.debuggerSnapshotEndpoint = DEBUGGER_DIAGNOSTICS_ENDPOINT; + } + if (containsEndpoint(endpoints, DEBUGGER_DIAGNOSTICS_ENDPOINT)) { + newState.debuggerDiagnosticsEndpoint = DEBUGGER_DIAGNOSTICS_ENDPOINT; + } + } + private static boolean containsEndpoint(Set endpoints, String endpoint) { return endpoints.contains(endpoint) || endpoints.contains("/" + endpoint); } @@ -358,7 +368,7 @@ private static void discoverStatsDPort(final Map info) { public boolean supportsMetrics() { return metricsEnabled && null != discoveryState.metricsEndpoint - && discoveryState.supportsDropping; + && discoveryState.supportsClientSideStats; } public boolean supportsDebugger() { @@ -377,10 +387,6 @@ public boolean supportsDebuggerDiagnostics() { return discoveryState.debuggerDiagnosticsEndpoint != null; } - public boolean supportsDropping() { - return discoveryState.supportsDropping; - } - public boolean supportsLongRunning() { return discoveryState.supportsLongRunning; } diff --git a/communication/src/main/java/datadog/communication/ddagent/SharedCommunicationObjects.java b/communication/src/main/java/datadog/communication/ddagent/SharedCommunicationObjects.java index 3864ccbded1..71db6404842 100644 --- a/communication/src/main/java/datadog/communication/ddagent/SharedCommunicationObjects.java +++ b/communication/src/main/java/datadog/communication/ddagent/SharedCommunicationObjects.java @@ -4,13 +4,14 @@ import static datadog.trace.util.AgentThreadFactory.AGENT_THREAD_GROUP; import datadog.common.container.ContainerInfo; -import datadog.common.socket.SocketUtils; import datadog.communication.http.OkHttpUtils; -import datadog.communication.monitor.Monitoring; +import datadog.communication.http.SocketUtils; +import datadog.metrics.api.Monitoring; import datadog.remoteconfig.ConfigurationPoller; import datadog.remoteconfig.DefaultConfigurationPoller; import datadog.trace.api.Config; import datadog.trace.util.AgentTaskScheduler; +import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; import java.security.Security; import java.util.ArrayList; import java.util.List; @@ -31,6 +32,7 @@ public class SharedCommunicationObjects { * HTTP client for making requests to Datadog agent. Depending on configuration, this client may * use regular HTTP, UDS or named pipe. */ + @SuppressFBWarnings("PA_PUBLIC_PRIMITIVE_ATTRIBUTE") public OkHttpClient agentHttpClient; /** @@ -39,10 +41,18 @@ public class SharedCommunicationObjects { */ private volatile OkHttpClient intakeHttpClient; + @SuppressFBWarnings("PA_PUBLIC_PRIMITIVE_ATTRIBUTE") public long httpClientTimeout; + + @SuppressFBWarnings("PA_PUBLIC_PRIMITIVE_ATTRIBUTE") public boolean forceClearTextHttpForIntakeClient; + + @SuppressFBWarnings("PA_PUBLIC_PRIMITIVE_ATTRIBUTE") public HttpUrl agentUrl; + + @SuppressFBWarnings("PA_PUBLIC_PRIMITIVE_ATTRIBUTE") public Monitoring monitoring; + private volatile DDAgentFeaturesDiscovery featuresDiscovery; private ConfigurationPoller configurationPoller; @@ -163,7 +173,7 @@ public DDAgentFeaturesDiscovery featuresDiscovery(Config config) { agentHttpClient, monitoring, agentUrl, - config.isTraceAgentV05Enabled(), + config.getProtocolVersion(), config.isTracerMetricsEnabled()); if (paused) { diff --git a/communication/src/main/java/datadog/communication/http/OkHttpUtils.java b/communication/src/main/java/datadog/communication/http/OkHttpUtils.java index 979a5cdfb3c..9dc13229131 100644 --- a/communication/src/main/java/datadog/communication/http/OkHttpUtils.java +++ b/communication/src/main/java/datadog/communication/http/OkHttpUtils.java @@ -1,6 +1,6 @@ package datadog.communication.http; -import static datadog.common.socket.SocketUtils.discoverApmSocket; +import static datadog.communication.http.SocketUtils.discoverApmSocket; import static java.util.concurrent.TimeUnit.MILLISECONDS; import static java.util.concurrent.TimeUnit.SECONDS; @@ -15,6 +15,7 @@ import java.net.InetSocketAddress; import java.net.Proxy; import java.nio.ByteBuffer; +import java.util.Arrays; import java.util.Collections; import java.util.List; import java.util.Map; @@ -27,6 +28,7 @@ import okhttp3.HttpUrl; import okhttp3.MediaType; import okhttp3.OkHttpClient; +import okhttp3.Protocol; import okhttp3.Request; import okhttp3.RequestBody; import okhttp3.Response; @@ -62,22 +64,47 @@ public static OkHttpClient buildHttpClient(final HttpUrl url, final long timeout } public static OkHttpClient buildHttpClient( - final boolean isHttp, + final boolean isPlainHttp, final String unixDomainSocketPath, final String namedPipe, final long timeoutMillis) { return buildHttpClient( unixDomainSocketPath, + Config.get().isJdkSocketEnabled(), namedPipe, null, - isHttp, + isPlainHttp, + false, null, null, null, null, null, null, - timeoutMillis); + timeoutMillis, + Config.get().isAgentConfiguredUsingDefault()); + } + + public static OkHttpClient buildHttp2Client( + final boolean isPlainHttp, + final String unixDomainSocketPath, + final String namedPipe, + final long timeoutMillis) { + return buildHttpClient( + unixDomainSocketPath, + Config.get().isJdkSocketEnabled(), + namedPipe, + null, + isPlainHttp, + true, + null, + null, + null, + null, + null, + null, + timeoutMillis, + Config.get().isAgentConfiguredUsingDefault()); } public static OkHttpClient buildHttpClient( @@ -93,32 +120,38 @@ public static OkHttpClient buildHttpClient( final long timeoutMillis) { return buildHttpClient( discoverApmSocket(config), + config.isJdkSocketEnabled(), config.getAgentNamedPipe(), dispatcher, isPlainHttp(url), + false, retryOnConnectionFailure, maxRunningRequests, proxyHost, proxyPort, proxyUsername, proxyPassword, - timeoutMillis); + timeoutMillis, + config.isAgentConfiguredUsingDefault()); } public abstract static class CustomListener extends EventListener {} private static OkHttpClient buildHttpClient( final String unixDomainSocketPath, + final boolean useJdkUnixDomainSocket, final String namedPipe, final Dispatcher dispatcher, - final boolean isHttp, + final boolean isPlainHttp, + final boolean isHttp2, final Boolean retryOnConnectionFailure, final Integer maxRunningRequests, final String proxyHost, final Integer proxyPort, final String proxyUsername, final String proxyPassword, - final long timeoutMillis) { + final long timeoutMillis, + final boolean agentConfiguredUsingDefault) { final OkHttpClient.Builder builder = new OkHttpClient.Builder(); try { @@ -144,18 +177,28 @@ private static OkHttpClient buildHttpClient( dispatcher != null ? dispatcher : new Dispatcher(RejectingExecutorService.INSTANCE)); if (unixDomainSocketPath != null) { - builder.socketFactory(new UnixDomainSocketFactory(new File(unixDomainSocketPath))); + builder.socketFactory( + new UnixDomainSocketFactory( + new File(unixDomainSocketPath), useJdkUnixDomainSocket, agentConfiguredUsingDefault)); log.debug("Using UnixDomainSocket as http transport"); } else if (namedPipe != null) { builder.socketFactory(new NamedPipeSocketFactory(namedPipe)); log.debug("Using NamedPipe as http transport"); } - if (isHttp) { + if (isPlainHttp) { // force clear text when using http to avoid failures for JVMs without TLS builder.connectionSpecs(Collections.singletonList(ConnectionSpec.CLEARTEXT)); } + if (isHttp2) { + if (isPlainHttp) { + builder.protocols(Collections.singletonList(Protocol.H2_PRIOR_KNOWLEDGE)); + } else { + builder.protocols(Arrays.asList(Protocol.HTTP_2, Protocol.HTTP_1_1)); + } + } + if (retryOnConnectionFailure != null) { builder.retryOnConnectionFailure(retryOnConnectionFailure); } diff --git a/utils/socket-utils/src/main/java/datadog/common/socket/SocketUtils.java b/communication/src/main/java/datadog/communication/http/SocketUtils.java similarity index 89% rename from utils/socket-utils/src/main/java/datadog/common/socket/SocketUtils.java rename to communication/src/main/java/datadog/communication/http/SocketUtils.java index 53602428b64..ed0dcaa118d 100644 --- a/utils/socket-utils/src/main/java/datadog/common/socket/SocketUtils.java +++ b/communication/src/main/java/datadog/communication/http/SocketUtils.java @@ -1,7 +1,8 @@ -package datadog.common.socket; +package datadog.communication.http; import static datadog.trace.api.ConfigDefaults.DEFAULT_TRACE_AGENT_SOCKET_PATH; +import datadog.common.filesystem.Files; import datadog.environment.OperatingSystem; import datadog.environment.SystemProperties; import datadog.trace.api.Config; @@ -20,7 +21,7 @@ public static String discoverApmSocket(final Config config) { if (!OperatingSystem.isWindows()) { if (unixDomainSocket == null && Config.get().isAgentConfiguredUsingDefault() - && new File(DEFAULT_TRACE_AGENT_SOCKET_PATH).exists()) { + && Files.exists(new File(DEFAULT_TRACE_AGENT_SOCKET_PATH))) { log.info("Detected {}. Using it to send trace data.", DEFAULT_TRACE_AGENT_SOCKET_PATH); unixDomainSocket = DEFAULT_TRACE_AGENT_SOCKET_PATH; } diff --git a/communication/src/main/java/datadog/communication/monitor/Counter.java b/communication/src/main/java/datadog/communication/monitor/Counter.java deleted file mode 100644 index 4141c6b07fa..00000000000 --- a/communication/src/main/java/datadog/communication/monitor/Counter.java +++ /dev/null @@ -1,8 +0,0 @@ -package datadog.communication.monitor; - -public interface Counter { - - void increment(int delta); - - void incrementErrorCount(String cause, int delta); -} diff --git a/communication/src/main/java/datadog/communication/monitor/Monitoring.java b/communication/src/main/java/datadog/communication/monitor/Monitoring.java deleted file mode 100644 index 8f11fd1082b..00000000000 --- a/communication/src/main/java/datadog/communication/monitor/Monitoring.java +++ /dev/null @@ -1,37 +0,0 @@ -package datadog.communication.monitor; - -public interface Monitoring { - Monitoring DISABLED = new DisabledMonitoring(); - - Recording newTimer(String name); - - Recording newTimer(String name, String... tags); - - Recording newThreadLocalTimer(String name); - - Counter newCounter(String name); - - class DisabledMonitoring implements Monitoring { - private DisabledMonitoring() {} - - @Override - public Recording newTimer(String name) { - return NoOpRecording.NO_OP; - } - - @Override - public Recording newTimer(String name, String... tags) { - return NoOpRecording.NO_OP; - } - - @Override - public Recording newThreadLocalTimer(String name) { - return NoOpRecording.NO_OP; - } - - @Override - public Counter newCounter(String name) { - return NoOpCounter.NO_OP; - } - } -} diff --git a/communication/src/main/java/datadog/communication/monitor/NoOpCounter.java b/communication/src/main/java/datadog/communication/monitor/NoOpCounter.java deleted file mode 100644 index b44bea18339..00000000000 --- a/communication/src/main/java/datadog/communication/monitor/NoOpCounter.java +++ /dev/null @@ -1,10 +0,0 @@ -package datadog.communication.monitor; - -public final class NoOpCounter implements Counter { - - public static final Counter NO_OP = new NoOpCounter(); - - public void increment(int delta) {} - - public void incrementErrorCount(String cause, int delta) {} -} diff --git a/communication/src/main/java/datadog/communication/monitor/NoOpRecording.java b/communication/src/main/java/datadog/communication/monitor/NoOpRecording.java deleted file mode 100644 index 32abc8716c1..00000000000 --- a/communication/src/main/java/datadog/communication/monitor/NoOpRecording.java +++ /dev/null @@ -1,20 +0,0 @@ -package datadog.communication.monitor; - -public class NoOpRecording extends Recording { - - public static final Recording NO_OP = new NoOpRecording(); - - @Override - public Recording start() { - return this; - } - - @Override - public void reset() {} - - @Override - public void stop() {} - - @Override - public void flush() {} -} diff --git a/dd-trace-core/src/main/java/datadog/trace/common/writer/ddagent/Caching.java b/communication/src/main/java/datadog/communication/serialization/Caching.java similarity index 96% rename from dd-trace-core/src/main/java/datadog/trace/common/writer/ddagent/Caching.java rename to communication/src/main/java/datadog/communication/serialization/Caching.java index bc61b037784..868170fcac3 100644 --- a/dd-trace-core/src/main/java/datadog/trace/common/writer/ddagent/Caching.java +++ b/communication/src/main/java/datadog/communication/serialization/Caching.java @@ -1,4 +1,4 @@ -package datadog.trace.common.writer.ddagent; +package datadog.communication.serialization; import java.util.Arrays; @@ -14,7 +14,8 @@ private Caching() {} */ static final int cacheSizeFor(int requestedCapacity) { int pow; - for (pow = 1; pow < requestedCapacity; pow *= 2) ; + for (pow = 1; pow < requestedCapacity; pow *= 2) + ; return pow; } diff --git a/communication/src/main/java/datadog/communication/serialization/FlushingBuffer.java b/communication/src/main/java/datadog/communication/serialization/FlushingBuffer.java index d23f5b2358e..332434ad46a 100644 --- a/communication/src/main/java/datadog/communication/serialization/FlushingBuffer.java +++ b/communication/src/main/java/datadog/communication/serialization/FlushingBuffer.java @@ -27,8 +27,12 @@ public boolean isDirty() { @Override public void mark() { - mark = buffer.position(); - ++messageCount; + int current = buffer.position(); + if (current != mark) { + // count only non-empty messages + ++messageCount; + mark = current; + } } @Override @@ -101,4 +105,9 @@ public void reset() { buffer.limit(buffer.capacity()); mark = 0; } + + // for tests only + int getMessageCount() { + return messageCount; + } } diff --git a/dd-trace-core/src/main/java/datadog/trace/common/writer/ddagent/GenerationalUtf8Cache.java b/communication/src/main/java/datadog/communication/serialization/GenerationalUtf8Cache.java similarity index 98% rename from dd-trace-core/src/main/java/datadog/trace/common/writer/ddagent/GenerationalUtf8Cache.java rename to communication/src/main/java/datadog/communication/serialization/GenerationalUtf8Cache.java index be50ff7103e..036abfe6e79 100644 --- a/dd-trace-core/src/main/java/datadog/trace/common/writer/ddagent/GenerationalUtf8Cache.java +++ b/communication/src/main/java/datadog/communication/serialization/GenerationalUtf8Cache.java @@ -1,6 +1,5 @@ -package datadog.trace.common.writer.ddagent; +package datadog.communication.serialization; -import datadog.communication.serialization.EncodingCache; import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; import java.nio.charset.StandardCharsets; @@ -74,7 +73,7 @@ public final class GenerationalUtf8Cache implements EncodingCache { private static final int MAX_EDEN_PROBES = 4; private static final int MAX_TENURED_PROBES = 8; - private static final int MIN_PROMOTION_TRESHOLD = 2; + private static final int MIN_PROMOTION_THRESHOLD = 2; private static final int INITIAL_PROMOTION_THRESHOLD = 16; private static final double SCORE_DECAY = 0.5D; @@ -138,6 +137,7 @@ public int tenuredCapacity() { } /** Updates access time used @link {@link #getUtf8(String, String)} to the provided value */ + @SuppressFBWarnings("AT_NONATOMIC_64BIT_PRIMITIVE") public void updateAccessTime(long accessTimeMs) { this.accessTimeMs = accessTimeMs; } @@ -168,7 +168,7 @@ public synchronized void recalibrate(long accessTimeMs) { recalibrate(this.tenuredEntries); int totalPromotions = this.promotions + this.earlyPromotions; - if (totalPromotions == 0 && this.promotionThreshold >= MIN_PROMOTION_TRESHOLD) { + if (totalPromotions == 0 && this.promotionThreshold >= MIN_PROMOTION_THRESHOLD) { this.promotionThreshold /= PROMOTION_THRESHOLD_ADJ_FACTOR; } else if (totalPromotions > this.tenuredEvictions / 2) { this.promotionThreshold *= PROMOTION_THRESHOLD_ADJ_FACTOR; diff --git a/communication/src/main/java/datadog/communication/serialization/GrowableBuffer.java b/communication/src/main/java/datadog/communication/serialization/GrowableBuffer.java index 8f7a471e2d1..81b55aba7d2 100644 --- a/communication/src/main/java/datadog/communication/serialization/GrowableBuffer.java +++ b/communication/src/main/java/datadog/communication/serialization/GrowableBuffer.java @@ -17,11 +17,18 @@ public GrowableBuffer(int initialCapacity) { this.buffer = ByteBuffer.allocate(initialCapacity); } + /** Flips the buffer and returns a new slice which shares the buffered content. */ public ByteBuffer slice() { buffer.flip(); return buffer.slice(); } + /** Flips the buffer and returns the buffered content. */ + public ByteBuffer flip() { + buffer.flip(); + return buffer; + } + public int messageCount() { return messageCount; } diff --git a/dd-trace-core/src/main/java/datadog/trace/common/writer/ddagent/SimpleUtf8Cache.java b/communication/src/main/java/datadog/communication/serialization/SimpleUtf8Cache.java similarity index 98% rename from dd-trace-core/src/main/java/datadog/trace/common/writer/ddagent/SimpleUtf8Cache.java rename to communication/src/main/java/datadog/communication/serialization/SimpleUtf8Cache.java index ab751b6a5ac..bb2dcf11f5d 100644 --- a/dd-trace-core/src/main/java/datadog/trace/common/writer/ddagent/SimpleUtf8Cache.java +++ b/communication/src/main/java/datadog/communication/serialization/SimpleUtf8Cache.java @@ -1,6 +1,5 @@ -package datadog.trace.common.writer.ddagent; +package datadog.communication.serialization; -import datadog.communication.serialization.EncodingCache; import java.nio.charset.StandardCharsets; /** diff --git a/communication/src/main/java/datadog/communication/serialization/Writable.java b/communication/src/main/java/datadog/communication/serialization/Writable.java index 0f4c8d04414..7450e66d512 100644 --- a/communication/src/main/java/datadog/communication/serialization/Writable.java +++ b/communication/src/main/java/datadog/communication/serialization/Writable.java @@ -27,6 +27,14 @@ public interface Writable { void writeBinary(byte[] binary, int offset, int length); + /** + * Encodes 128 bits as binary. + * + * @param hi the high-order 64 bits. + * @param lo The low-order 64 bits. + */ + void writeBinary(long hi, long lo); + /** * Start a part of the message containing key-value pairs * diff --git a/communication/src/main/java/datadog/communication/serialization/custom/aiguard/MessageWriter.java b/communication/src/main/java/datadog/communication/serialization/custom/aiguard/MessageWriter.java index a1b089fbb1c..e8876b86df3 100644 --- a/communication/src/main/java/datadog/communication/serialization/custom/aiguard/MessageWriter.java +++ b/communication/src/main/java/datadog/communication/serialization/custom/aiguard/MessageWriter.java @@ -14,16 +14,51 @@ public void write( final AIGuard.Message value, final Writable writable, final EncodingCache encodingCache) { final int[] size = {0}; final boolean hasRole = isNotBlank(value.getRole(), size); - final boolean hasContent = isNotBlank(value.getContent(), size); final boolean hasToolCallId = isNotBlank(value.getToolCallId(), size); final boolean hasToolCalls = isNotEmpty(value.getToolCalls(), size); + + final boolean hasContentParts = isNotEmpty(value.getContentParts(), size); + final boolean hasContentString = !hasContentParts && isNotBlank(value.getContent(), size); + writable.startMap(size[0]); writeString(hasRole, "role", value.getRole(), writable, encodingCache); - writeString(hasContent, "content", value.getContent(), writable, encodingCache); + + if (hasContentParts) { + writeContentParts("content", value.getContentParts(), writable, encodingCache); + } else { + writeString(hasContentString, "content", value.getContent(), writable, encodingCache); + } + writeString(hasToolCallId, "tool_call_id", value.getToolCallId(), writable, encodingCache); writeToolCallArray(hasToolCalls, "tool_calls", value.getToolCalls(), writable, encodingCache); } + private static void writeContentParts( + final String key, + final List contentParts, + final Writable writable, + final EncodingCache encodingCache) { + writable.writeString(key, encodingCache); + writable.startArray(contentParts.size()); + + for (final AIGuard.ContentPart part : contentParts) { + writable.startMap(2); + + writable.writeString("type", encodingCache); + writable.writeString(part.getType().toString(), encodingCache); + + if (part.getType() == AIGuard.ContentPart.Type.TEXT) { + writable.writeString("text", encodingCache); + writable.writeString(part.getText(), encodingCache); + } else if (part.getType() == AIGuard.ContentPart.Type.IMAGE_URL) { + writable.writeString("image_url", encodingCache); + writable.startMap(1); + writable.writeString("url", encodingCache); + writable.writeString(part.getImageUrl().getUrl(), encodingCache); + } + } + } + private static void writeString( final boolean present, final String key, diff --git a/communication/src/main/java/datadog/communication/serialization/msgpack/MsgPackWriter.java b/communication/src/main/java/datadog/communication/serialization/msgpack/MsgPackWriter.java index 107e1f94efd..8c96226dc0f 100644 --- a/communication/src/main/java/datadog/communication/serialization/msgpack/MsgPackWriter.java +++ b/communication/src/main/java/datadog/communication/serialization/msgpack/MsgPackWriter.java @@ -213,6 +213,13 @@ public void writeBinary(byte[] binary, int offset, int length) { buffer.put(binary, offset, length); } + @Override + public void writeBinary(long hi, long lo) { + writeBinaryHeader(16); + buffer.putLong(hi); + buffer.putLong(lo); + } + @Override public void writeBinary(ByteBuffer binary) { ByteBuffer slice = binary.slice(); diff --git a/communication/src/main/java/okhttp3/internal/platform/PatchPlatform.java b/communication/src/main/java/okhttp3/internal/platform/PatchPlatform.java new file mode 100644 index 00000000000..dc9f9c2d4b6 --- /dev/null +++ b/communication/src/main/java/okhttp3/internal/platform/PatchPlatform.java @@ -0,0 +1,332 @@ +/* + * Copyright (C) 2012 Square, Inc. + * Copyright (C) 2012 The Android Open Source Project + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package okhttp3.internal.platform; + +import de.thetaphi.forbiddenapis.SuppressForbidden; +import java.io.IOException; +import java.lang.reflect.Field; +import java.net.InetSocketAddress; +import java.net.Socket; +import java.security.NoSuchAlgorithmException; +import java.util.ArrayList; +import java.util.List; +import java.util.logging.Level; +import java.util.logging.Logger; +import javax.annotation.Nullable; +import javax.net.ssl.SSLContext; +import javax.net.ssl.SSLSocket; +import javax.net.ssl.SSLSocketFactory; +import javax.net.ssl.X509TrustManager; +import okhttp3.OkHttpClient; +import okhttp3.Protocol; +import okhttp3.internal.tls.BasicCertificateChainCleaner; +import okhttp3.internal.tls.BasicTrustRootIndex; +import okhttp3.internal.tls.CertificateChainCleaner; +import okhttp3.internal.tls.TrustRootIndex; +import okio.Buffer; + +/** + * Replacement Platform class that avoids eager call to Security.getProviders() + * ---------------------------------------------------------------------------- + * + *

Access to platform-specific features. + * + *

Server name indication (SNI)

+ * + *

Supported on Android 2.3+. + * + *

Supported on OpenJDK 7+ + * + *

Session Tickets

+ * + *

Supported on Android 2.3+. + * + *

Android Traffic Stats (Socket Tagging)

+ * + *

Supported on Android 4.0+. + * + *

ALPN (Application Layer Protocol Negotiation)

+ * + *

Supported on Android 5.0+. The APIs were present in Android 4.4, but that implementation was + * unstable. + * + *

Supported on OpenJDK 7 and 8 (via the JettyALPN-boot library). + * + *

Supported on OpenJDK 9 via SSLParameters and SSLSocket features. + * + *

Trust Manager Extraction

+ * + *

Supported on Android 2.3+ and OpenJDK 7+. There are no public APIs to recover the trust + * manager that was used to create an {@link SSLSocketFactory}. + * + *

Android Cleartext Permit Detection

+ * + *

Supported on Android 6.0+ via {@code NetworkSecurityPolicy}. + */ +public class PatchPlatform { + private static final Platform PLATFORM = findPlatform(); + public static final int INFO = 4; + public static final int WARN = 5; + private static final Logger logger = Logger.getLogger(OkHttpClient.class.getName()); + + public static Platform get() { + return PLATFORM; + } + + /** Prefix used on custom headers. */ + public String getPrefix() { + return "OkHttp"; + } + + @SuppressForbidden // allow this use of Class.forName() + protected @Nullable X509TrustManager trustManager(SSLSocketFactory sslSocketFactory) { + // Attempt to get the trust manager from an OpenJDK socket factory. We attempt this on all + // platforms in order to support Robolectric, which mixes classes from both Android and the + // Oracle JDK. Note that we don't support HTTP/2 or other nice features on Robolectric. + try { + Class sslContextClass = Class.forName("sun.security.ssl.SSLContextImpl"); + Object context = readFieldOrNull(sslSocketFactory, sslContextClass, "context"); + if (context == null) return null; + return readFieldOrNull(context, X509TrustManager.class, "trustManager"); + } catch (ClassNotFoundException e) { + return null; + } + } + + /** + * Configure TLS extensions on {@code sslSocket} for {@code route}. + * + * @param hostname non-null for client-side handshakes; null for server-side handshakes. + */ + public void configureTlsExtensions( + SSLSocket sslSocket, @Nullable String hostname, List protocols) + throws IOException {} + + /** + * Called after the TLS handshake to release resources allocated by {@link + * #configureTlsExtensions}. + */ + public void afterHandshake(SSLSocket sslSocket) {} + + /** Returns the negotiated protocol, or null if no protocol was negotiated. */ + public @Nullable String getSelectedProtocol(SSLSocket socket) { + return null; + } + + public void connectSocket(Socket socket, InetSocketAddress address, int connectTimeout) + throws IOException { + socket.connect(address, connectTimeout); + } + + public void log(int level, String message, @Nullable Throwable t) { + Level logLevel = level == WARN ? Level.WARNING : Level.INFO; + logger.log(logLevel, message, t); + } + + public boolean isCleartextTrafficPermitted(String hostname) { + return true; + } + + /** + * Returns an object that holds a stack trace created at the moment this method is executed. This + * should be used specifically for {@link java.io.Closeable} objects and in conjunction with + * {@link #logCloseableLeak(String, Object)}. + */ + public Object getStackTraceForCloseable(String closer) { + if (logger.isLoggable(Level.FINE)) { + return new Throwable(closer); // These are expensive to allocate. + } + return null; + } + + public void logCloseableLeak(String message, Object stackTrace) { + if (stackTrace == null) { + message += + " To see where this was allocated, set the OkHttpClient logger level to FINE: " + + "Logger.getLogger(OkHttpClient.class.getName()).setLevel(Level.FINE);"; + } + log(WARN, message, (Throwable) stackTrace); + } + + public static List alpnProtocolNames(List protocols) { + List names = new ArrayList<>(protocols.size()); + for (int i = 0, size = protocols.size(); i < size; i++) { + Protocol protocol = protocols.get(i); + if (protocol == Protocol.HTTP_1_0) continue; // No HTTP/1.0 for ALPN. + names.add(protocol.toString()); + } + return names; + } + + public CertificateChainCleaner buildCertificateChainCleaner(X509TrustManager trustManager) { + return new BasicCertificateChainCleaner(buildTrustRootIndex(trustManager)); + } + + public CertificateChainCleaner buildCertificateChainCleaner(SSLSocketFactory sslSocketFactory) { + X509TrustManager trustManager = trustManager(sslSocketFactory); + + if (trustManager == null) { + throw new IllegalStateException( + "Unable to extract the trust manager on " + + Platform.get() + + ", sslSocketFactory is " + + sslSocketFactory.getClass()); + } + + return buildCertificateChainCleaner(trustManager); + } + + /* Avoid eager call to Security.getProviders() + ---------------------------------------------- + | public static boolean isConscryptPreferred() { + | // mainly to allow tests to run cleanly + | if ("conscrypt".equals(System.getProperty("okhttp.platform"))) { + | return true; + | } + | + | // check if Provider manually installed + | String preferredProvider = Security.getProviders()[0].getName(); + | return "Conscrypt".equals(preferredProvider); + | } + ---------------------------------------------- */ + + /** Attempt to match the host runtime to a capable Platform implementation. */ + private static Platform findPlatform() { + if (isAndroid()) { + return findAndroidPlatform(); + } else { + return findJvmPlatform(); + } + } + + public static boolean isAndroid() { + // This explicit check avoids activating in Android Studio with Android specific classes + // available when running plugins inside the IDE. + return "Dalvik".equals(System.getProperty("java.vm.name")); + } + + private static Platform findJvmPlatform() { + /* Avoid eager call to Security.getProviders() + ---------------------------------------------- + | if (isConscryptPreferred()) { + | Platform conscrypt = ConscryptPlatform.buildIfSupported(); + | + | if (conscrypt != null) { + | return conscrypt; + | } + | } + ---------------------------------------------- */ + + Platform jdk9 = Jdk9Platform.buildIfSupported(); + + if (jdk9 != null) { + return jdk9; + } + + Platform jdkWithJettyBoot = JdkWithJettyBootPlatform.buildIfSupported(); + + if (jdkWithJettyBoot != null) { + return jdkWithJettyBoot; + } + + // Probably an Oracle JDK like OpenJDK. + return new Platform(); + } + + private static Platform findAndroidPlatform() { + Platform android10 = Android10Platform.buildIfSupported(); + + if (android10 != null) { + return android10; + } + + Platform android = AndroidPlatform.buildIfSupported(); + + if (android == null) { + throw new NullPointerException("No platform found on Android"); + } + + return android; + } + + /** + * Returns the concatenation of 8-bit, length prefixed protocol names. + * http://tools.ietf.org/html/draft-agl-tls-nextprotoneg-04#page-4 + */ + static byte[] concatLengthPrefixed(List protocols) { + Buffer result = new Buffer(); + for (int i = 0, size = protocols.size(); i < size; i++) { + Protocol protocol = protocols.get(i); + if (protocol == Protocol.HTTP_1_0) continue; // No HTTP/1.0 for ALPN. + result.writeByte(protocol.toString().length()); + result.writeUtf8(protocol.toString()); + } + return result.readByteArray(); + } + + static @Nullable T readFieldOrNull(Object instance, Class fieldType, String fieldName) { + for (Class c = instance.getClass(); c != Object.class; c = c.getSuperclass()) { + try { + Field field = c.getDeclaredField(fieldName); + field.setAccessible(true); + Object value = field.get(instance); + if (value == null || !fieldType.isInstance(value)) return null; + return fieldType.cast(value); + } catch (NoSuchFieldException ignored) { + } catch (IllegalAccessException e) { + throw new AssertionError(); + } + } + + // Didn't find the field we wanted. As a last gasp attempt, try to find the value on a delegate. + if (!fieldName.equals("delegate")) { + Object delegate = readFieldOrNull(instance, Object.class, "delegate"); + if (delegate != null) return readFieldOrNull(delegate, fieldType, fieldName); + } + + return null; + } + + public SSLContext getSSLContext() { + String jvmVersion = System.getProperty("java.specification.version"); + if ("1.7".equals(jvmVersion)) { + try { + // JDK 1.7 (public version) only support > TLSv1 with named protocols + return SSLContext.getInstance("TLSv1.2"); + } catch (NoSuchAlgorithmException e) { + // fallback to TLS + } + } + + try { + return SSLContext.getInstance("TLS"); + } catch (NoSuchAlgorithmException e) { + throw new IllegalStateException("No TLS provider", e); + } + } + + public TrustRootIndex buildTrustRootIndex(X509TrustManager trustManager) { + return new BasicTrustRootIndex(trustManager.getAcceptedIssuers()); + } + + public void configureSslSocketFactory(SSLSocketFactory socketFactory) {} + + @Override + public String toString() { + return getClass().getSimpleName(); + } +} diff --git a/communication/src/test/groovy/datadog/communication/ddagent/DDAgentFeaturesDiscoveryTest.groovy b/communication/src/test/groovy/datadog/communication/ddagent/DDAgentFeaturesDiscoveryTest.groovy index 5724633e524..a8c76fb3fb5 100644 --- a/communication/src/test/groovy/datadog/communication/ddagent/DDAgentFeaturesDiscoveryTest.groovy +++ b/communication/src/test/groovy/datadog/communication/ddagent/DDAgentFeaturesDiscoveryTest.groovy @@ -1,9 +1,23 @@ package datadog.communication.ddagent +import static datadog.communication.ddagent.DDAgentFeaturesDiscovery.V01_DATASTREAMS_ENDPOINT +import static datadog.communication.ddagent.DDAgentFeaturesDiscovery.V04_ENDPOINT +import static datadog.communication.ddagent.DDAgentFeaturesDiscovery.V05_ENDPOINT +import static datadog.communication.ddagent.DDAgentFeaturesDiscovery.V06_METRICS_ENDPOINT +import static datadog.communication.ddagent.DDAgentFeaturesDiscovery.V07_CONFIG_ENDPOINT +import static datadog.communication.ddagent.DDAgentFeaturesDiscovery.V1_ENDPOINT +import static datadog.communication.http.OkHttpUtils.DATADOG_CONTAINER_ID +import static datadog.communication.http.OkHttpUtils.DATADOG_CONTAINER_TAGS_HASH +import static datadog.trace.api.ProtocolVersion.V0_4 +import static datadog.trace.api.ProtocolVersion.V0_5 +import static datadog.trace.api.ProtocolVersion.V1_0 + import datadog.common.container.ContainerInfo -import datadog.communication.monitor.Monitoring +import datadog.metrics.api.Monitoring import datadog.trace.test.util.DDSpecification import datadog.trace.util.Strings +import java.nio.file.Files +import java.nio.file.Paths import okhttp3.Call import okhttp3.Headers import okhttp3.HttpUrl @@ -15,15 +29,6 @@ import okhttp3.Response import okhttp3.ResponseBody import spock.lang.Shared -import java.nio.file.Files -import java.nio.file.Paths - -import static datadog.communication.ddagent.DDAgentFeaturesDiscovery.V01_DATASTREAMS_ENDPOINT -import static datadog.communication.ddagent.DDAgentFeaturesDiscovery.V6_METRICS_ENDPOINT -import static datadog.communication.ddagent.DDAgentFeaturesDiscovery.V7_CONFIG_ENDPOINT -import static datadog.communication.http.OkHttpUtils.DATADOG_CONTAINER_ID -import static datadog.communication.http.OkHttpUtils.DATADOG_CONTAINER_TAGS_HASH - class DDAgentFeaturesDiscoveryTest extends DDSpecification { @Shared @@ -50,21 +55,20 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { def "test parse /info response"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, true, true) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, protocol, true) when: "/info available" features.discover() then: 1 * client.newCall(_) >> { Request request -> infoResponse(request, INFO_RESPONSE) } - features.getMetricsEndpoint() == V6_METRICS_ENDPOINT + features.getMetricsEndpoint() == V06_METRICS_ENDPOINT !features.supportsMetrics() - features.getTraceEndpoint() == "v0.5/traces" - !features.supportsDropping() + features.getTraceEndpoint() == expectedTraceEndpoint features.getDataStreamsEndpoint() == V01_DATASTREAMS_ENDPOINT features.supportsDataStreams() features.state() == INFO_STATE - features.getConfigEndpoint() == V7_CONFIG_ENDPOINT + features.getConfigEndpoint() == V07_CONFIG_ENDPOINT features.supportsDebugger() features.getDebuggerSnapshotEndpoint() == "debugger/v2/input" features.supportsDebuggerDiagnostics() @@ -75,12 +79,35 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { !features.supportsLongRunning() !features.supportsTelemetryProxy() 0 * _ + + where: + protocol | expectedTraceEndpoint + V0_4 | V04_ENDPOINT + V0_5 | V05_ENDPOINT + V1_0 | V1_ENDPOINT + } + + def "null protocol version falls back to v0.4 trace endpoints"() { + setup: + OkHttpClient client = Mock(OkHttpClient) + DDAgentFeaturesDiscovery features = + new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, null, true) + + when: + features.discover() + + then: + 1 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/info" }) >> { Request request -> infoResponse(request, "{}") } + 0 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/v0.5/traces" }) >> { Request request -> success(request) } + 1 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/v0.4/traces" }) >> { Request request -> success(request) } + features.getTraceEndpoint() == V04_ENDPOINT + 0 * _ } def "Should change discovery state atomically after discovery happened"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, true, true) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_5, true) when: "/info available" features.discover() @@ -106,7 +133,7 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { def "test parse /info response with discoverIfOutdated"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, true, true) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_5, true) when: "/info available" features.discoverIfOutdated() @@ -115,14 +142,13 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { then: 1 * client.newCall(_) >> { Request request -> infoResponse(request, INFO_RESPONSE) } - features.getMetricsEndpoint() == V6_METRICS_ENDPOINT + features.getMetricsEndpoint() == V06_METRICS_ENDPOINT !features.supportsMetrics() - features.getTraceEndpoint() == "v0.5/traces" - !features.supportsDropping() + features.getTraceEndpoint() == V05_ENDPOINT features.getDataStreamsEndpoint() == V01_DATASTREAMS_ENDPOINT features.supportsDataStreams() features.state() == INFO_STATE - features.getConfigEndpoint() == V7_CONFIG_ENDPOINT + features.getConfigEndpoint() == V07_CONFIG_ENDPOINT features.supportsDebugger() features.supportsDebuggerDiagnostics() features.supportsEvpProxy() @@ -135,17 +161,16 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { def "test parse /info response with client dropping"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, true, true) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_5, true) when: "/info available" features.discover() then: 1 * client.newCall(_) >> { Request request -> infoResponse(request, INFO_WITH_CLIENT_DROPPING_RESPONSE) } - features.getMetricsEndpoint() == V6_METRICS_ENDPOINT + features.getMetricsEndpoint() == V06_METRICS_ENDPOINT features.supportsMetrics() - features.getTraceEndpoint() == "v0.5/traces" - features.supportsDropping() + features.getTraceEndpoint() == V05_ENDPOINT features.state() == INFO_WITH_CLIENT_DROPPING_STATE 0 * _ } @@ -154,16 +179,16 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { def "test parse /info response with data streams unavailable"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, true, true) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_5, true) when: "/info available" features.discover() then: 1 * client.newCall(_) >> { Request request -> infoResponse(request, INFO_WITHOUT_DATA_STREAMS_RESPONSE) } - features.getMetricsEndpoint() == V6_METRICS_ENDPOINT + features.getMetricsEndpoint() == V06_METRICS_ENDPOINT !features.supportsMetrics() - features.getTraceEndpoint() == "v0.5/traces" + features.getTraceEndpoint() == V05_ENDPOINT features.getDataStreamsEndpoint() == null !features.supportsDataStreams() features.state() == INFO_WITHOUT_DATA_STREAMS_STATE @@ -173,7 +198,7 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { def "test parse /info response with long running spans available"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, true, true) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_5, true) when: "/info available" features.discover() @@ -184,10 +209,32 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { 0 * _ } + def "test fallback when /info empty"() { + setup: + OkHttpClient client = Mock(OkHttpClient) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_4, true) + + when: "/info is empty" + features.discover() + + then: + 1 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/info" }) >> { Request request -> infoResponse(request, "{}") } + 0 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/v0.6/stats" }) >> { Request request -> clientError(request) } + 0 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/v0.5/traces" }) >> { Request request -> success(request) } + 1 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/v0.4/traces" }) >> { Request request -> success(request) } + 0 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/v0.3/traces" }) >> { Request request -> success(request) } + features.getMetricsEndpoint() == null + !features.supportsMetrics() + features.getTraceEndpoint() == V04_ENDPOINT + !features.supportsLongRunning() + features.state() == PROBE_STATE + 0 * _ + } + def "test fallback when /info not found"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, true, true) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_5, true) when: "/info unavailable" features.discover() @@ -200,8 +247,7 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { 0 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/v0.3/traces" }) >> { Request request -> success(request) } features.getMetricsEndpoint() == null !features.supportsMetrics() - features.getTraceEndpoint() == "v0.5/traces" - !features.supportsDropping() + features.getTraceEndpoint() == V05_ENDPOINT !features.supportsLongRunning() features.state() == PROBE_STATE 0 * _ @@ -210,7 +256,7 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { def "test fallback when /info not found and agent returns ok"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, true, true) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_5, true) when: "/info unavailable" features.discover() @@ -221,8 +267,7 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { 1 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/v0.5/traces" }) >> { Request request -> success(request) } features.getMetricsEndpoint() == null !features.supportsMetrics() - features.getTraceEndpoint() == "v0.5/traces" - !features.supportsDropping() + features.getTraceEndpoint() == V05_ENDPOINT !features.supportsLongRunning() features.state() == PROBE_STATE 0 * _ @@ -231,7 +276,7 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { def "test fallback when /info not found and v0.5 disabled"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, false, true) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_4, true) when: "/info unavailable" features.discover() @@ -245,7 +290,6 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { features.getMetricsEndpoint() == null !features.supportsMetrics() features.getTraceEndpoint() == "v0.4/traces" - !features.supportsDropping() features.state() == PROBE_STATE 0 * _ } @@ -253,7 +297,7 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { def "test fallback when /info not found and v0.5 unavailable agent side"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, true, true) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_5, true) when: "/info unavailable" features.discover() @@ -267,7 +311,6 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { features.getMetricsEndpoint() == null !features.supportsMetrics() features.getTraceEndpoint() == "v0.4/traces" - !features.supportsDropping() features.state() == PROBE_STATE 0 * _ } @@ -275,7 +318,7 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { def "test fallback on very old agent"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, true, true) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_5, true) when: "/info unavailable" features.discover() @@ -290,7 +333,6 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { !features.supportsMetrics() features.getTraceEndpoint() == "v0.3/traces" !features.supportsLongRunning() - !features.supportsDropping() features.state() == PROBE_STATE 0 * _ } @@ -298,7 +340,7 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { def "disabling metrics disables metrics and dropping"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, true, false) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_5, false) when: "/info unavailable" features.discover() @@ -308,7 +350,6 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { 1 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/v0.5/traces" }) >> { Request request -> success(request) } 0 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/v0.6/stats" }) !features.supportsMetrics() - !features.supportsDropping() !(features as DroppingPolicy).active() features.state() == PROBE_STATE @@ -318,7 +359,6 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { then: "metrics and dropping not supported" 1 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/info" }) >> { Request request -> infoResponse(request, INFO_WITH_CLIENT_DROPPING_RESPONSE) } !features.supportsMetrics() - !features.supportsDropping() !(features as DroppingPolicy).active() features.state() == INFO_WITH_CLIENT_DROPPING_STATE @@ -328,7 +368,6 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { then: "metrics and dropping not supported" 1 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/info" }) >> { Request request -> infoResponse(request, INFO_RESPONSE) } !features.supportsMetrics() - !features.supportsDropping() !(features as DroppingPolicy).active() features.state() == INFO_STATE 0 * _ @@ -337,7 +376,7 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { def "discovery of metrics endpoint after agent upgrade enables dropping and metrics"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, false, true) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_4, true) when: "/info unavailable" features.discover() @@ -346,7 +385,6 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { 1 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/info" }) >> { Request request -> notFound(request) } 1 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/v0.4/traces" }) >> { Request request -> success(request) } 0 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/v0.6/stats" }) - !features.supportsDropping() !features.supportsMetrics() !(features as DroppingPolicy).active() features.state() == PROBE_STATE @@ -357,7 +395,6 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { then: "metrics endpoint not probed, metrics and dropping enabled" 1 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/info" }) >> { Request request -> infoResponse(request, INFO_WITH_CLIENT_DROPPING_RESPONSE) } 0 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/v0.4/traces" }) >> { Request request -> success(request) } - features.supportsDropping() features.supportsMetrics() (features as DroppingPolicy).active() features.state() == INFO_WITH_CLIENT_DROPPING_STATE @@ -367,7 +404,7 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { def "disappearance of info endpoint after agent downgrade disables metrics and dropping"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, false, true) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_4, true) when: "/info available" features.discover() @@ -376,7 +413,6 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { 1 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/info" }) >> { Request request -> infoResponse(request, INFO_WITH_CLIENT_DROPPING_RESPONSE) } 0 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/v0.4/traces" }) >> { Request request -> success(request) } 0 * client.newCall(_) - features.supportsDropping() features.supportsMetrics() (features as DroppingPolicy).active() features.state() == INFO_WITH_CLIENT_DROPPING_STATE @@ -388,7 +424,6 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { 1 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/info" }) >> { Request request -> notFound(request) } 1 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/v0.4/traces" }) >> { Request request -> success(request) } 0 * client.newCall(_) - !features.supportsDropping() !features.supportsMetrics() !(features as DroppingPolicy).active() features.state() == PROBE_STATE @@ -398,7 +433,7 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { def "disappearance of metrics endpoint after agent downgrade disables metrics and dropping"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, false, true) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_4, true) when: "/info available" features.discover() @@ -407,7 +442,6 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { 1 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/info" }) >> { Request request -> infoResponse(request, INFO_WITH_CLIENT_DROPPING_RESPONSE) } 0 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/v0.4/traces" }) >> { Request request -> success(request) } 0 * client.newCall(_) - features.supportsDropping() features.supportsMetrics() (features as DroppingPolicy).active() features.state() == INFO_WITH_CLIENT_DROPPING_STATE @@ -418,8 +452,6 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { then: "metrics and dropping not supported" 1 * client.newCall({ Request request -> request.url().toString() == "http://localhost:8125/info" }) >> { Request request -> infoResponse(request, INFO_WITHOUT_METRICS_RESPONSE) } 0 * client.newCall(_) - // misconfigured agent allows dropping but not metrics - features.supportsDropping() !features.supportsMetrics() // but we don't permit dropping anyway !(features as DroppingPolicy).active() @@ -431,7 +463,7 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { def "test parse /info response with telemetry proxy"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, true, true) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_5, true) when: "/info available" features.discover() @@ -448,7 +480,7 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { def "test parse /info response with old EVP proxy"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, true, true) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_5, true) when: "/info available" features.discover() @@ -467,7 +499,7 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { def "test parse /info response with peer tag back propagation"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, true, true) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_5, true) when: "/info available" features.discover() @@ -481,7 +513,6 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { then: 1 * client.newCall(_) >> { Request request -> infoResponse(request, INFO_WITH_PEER_TAG_BACK_PROPAGATION_RESPONSE) } features.state() == INFO_WITH_PEER_TAG_BACK_PROPAGATION_STATE - features.supportsDropping() features.peerTags().containsAll( "_dd.base_service", "active_record.db.vendor", @@ -498,10 +529,70 @@ class DDAgentFeaturesDiscoveryTest extends DDSpecification { ) } + def "test metrics disabled for agent version below 7.65"() { + setup: + OkHttpClient client = Mock(OkHttpClient) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_5, true) + + when: "agent version is below 7.65" + features.discover() + + then: + 1 * client.newCall(_) >> { Request request -> + def response = """ + { + "version": "${version}", + "endpoints": ["/v0.5/traces", "/v0.6/stats"], + "client_drop_p0s": true, + "config": {} + } + """ + infoResponse(request, response) + } + features.getMetricsEndpoint() == V06_METRICS_ENDPOINT + features.supportsMetrics() == expected + + where: + version | expected + "7.64.0" | false + "7.64.9" | false + "7.64.9-rc.1" | false + "7.65.0" | true + "7.65.1" | true + "7.70.0" | true + "8.0.0" | true + } + + def "test metrics disabled for agent with unparseable version"() { + setup: + OkHttpClient client = Mock(OkHttpClient) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_5, true) + + when: "agent version is unparseable" + features.discover() + + then: + 1 * client.newCall(_) >> { Request request -> + def response = """ + { + "version": "${version}", + "endpoints": ["/v0.5/traces", "/v0.6/stats"], + "client_drop_p0s": true, + "config": {} + } + """ + infoResponse(request, response) + } + !features.supportsMetrics() + + where: + version << ["invalid", "7", "7.65", "", null] + } + def "should send container id as header on the info request and parse the hash in the response"() { setup: OkHttpClient client = Mock(OkHttpClient) - DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, true, true) + DDAgentFeaturesDiscovery features = new DDAgentFeaturesDiscovery(client, monitoring, agentUrl, V0_5, true) def oldContainerId = ContainerInfo.get().getContainerId() def oldContainerTagsHash = ContainerInfo.get().getContainerTagsHash() ContainerInfo.get().setContainerId("test") diff --git a/communication/src/test/groovy/datadog/communication/ddagent/SharedCommunicationsObjectsSpecification.groovy b/communication/src/test/groovy/datadog/communication/ddagent/SharedCommunicationsObjectsSpecification.groovy index 9795aca655c..6ee88e87115 100644 --- a/communication/src/test/groovy/datadog/communication/ddagent/SharedCommunicationsObjectsSpecification.groovy +++ b/communication/src/test/groovy/datadog/communication/ddagent/SharedCommunicationsObjectsSpecification.groovy @@ -1,13 +1,14 @@ package datadog.communication.ddagent -import datadog.communication.monitor.Monitoring +import static datadog.trace.api.ProtocolVersion.V0_4 +import static datadog.trace.api.config.TracerConfig.AGENT_HOST + +import datadog.metrics.api.Monitoring import datadog.trace.api.Config import datadog.trace.test.util.DDSpecification import okhttp3.HttpUrl import okhttp3.OkHttpClient -import static datadog.trace.api.config.TracerConfig.AGENT_HOST - class SharedCommunicationsObjectsSpecification extends DDSpecification { SharedCommunicationObjects sco = new SharedCommunicationObjects() @@ -31,7 +32,7 @@ class SharedCommunicationsObjectsSpecification extends DDSpecification { sco.featuresDiscovery(config) then: - 1 * config.traceAgentV05Enabled >> false + 1 * config.protocolVersion >> V0_4 1 * config.tracerMetricsEnabled >> false sco.featuresDiscovery != null diff --git a/communication/src/test/groovy/datadog/communication/http/RejectingExecutorServiceTest.groovy b/communication/src/test/groovy/datadog/communication/http/RejectingExecutorServiceTest.groovy index 6065c64a644..d236e32178e 100644 --- a/communication/src/test/groovy/datadog/communication/http/RejectingExecutorServiceTest.groovy +++ b/communication/src/test/groovy/datadog/communication/http/RejectingExecutorServiceTest.groovy @@ -1,19 +1,19 @@ package datadog.communication.http +import static org.junit.jupiter.api.Assertions.assertThrows + import org.junit.jupiter.api.Test import java.util.concurrent.ExecutorService import java.util.concurrent.RejectedExecutionException import java.util.concurrent.TimeUnit -import static groovy.test.GroovyAssert.shouldFail - class RejectingExecutorServiceTest { ExecutorService executorService = new RejectingExecutorService() @Test void 'execute throws exception'() { - shouldFail(RejectedExecutionException) { + assertThrows(RejectedExecutionException) { executorService.execute({}) } } diff --git a/communication/src/test/groovy/datadog/communication/monitor/NoopCounterTest.groovy b/communication/src/test/groovy/datadog/communication/monitor/NoopCounterTest.groovy deleted file mode 100644 index 4898b30e487..00000000000 --- a/communication/src/test/groovy/datadog/communication/monitor/NoopCounterTest.groovy +++ /dev/null @@ -1,13 +0,0 @@ -package datadog.communication.monitor - -import org.junit.jupiter.api.Test - -class NoopCounterTest { - Counter noopCounter = NoOpCounter.NO_OP - - @Test - void 'cover the empty methods'() { - noopCounter.increment(1) - noopCounter.incrementErrorCount('', 0) - } -} diff --git a/communication/src/test/groovy/datadog/communication/serialization/FlushingBufferTest.java b/communication/src/test/groovy/datadog/communication/serialization/FlushingBufferTest.java index 33b1a61a27e..708a5cb8514 100644 --- a/communication/src/test/groovy/datadog/communication/serialization/FlushingBufferTest.java +++ b/communication/src/test/groovy/datadog/communication/serialization/FlushingBufferTest.java @@ -10,4 +10,45 @@ public class FlushingBufferTest { public void testBufferCapacity() { assertEquals(5, new FlushingBuffer(5, (messageCount, buffer) -> {}).capacity()); } + + @Test + public void testMessageCount() { + FlushingBuffer fb = new FlushingBuffer(10, (messageCount, buffer) -> {}); + + // initial counter + assertEquals(0, fb.getMessageCount()); + + fb.mark(); + fb.mark(); + + // counter doesn't change if no data pushed into the buffer + assertEquals(0, fb.getMessageCount()); + + fb.put((byte) 1); + // still zero because the message counter increases on mark + assertEquals(0, fb.getMessageCount()); + + fb.mark(); + // expect increased message counter + assertEquals(1, fb.getMessageCount()); + + fb.mark(); + fb.mark(); + // no change to the counter expected for consecutive mark calls + + fb.putChar('a'); + fb.putChar('b'); + fb.putChar('c'); + // no change to the message counter expected before mark call + assertEquals(1, fb.getMessageCount()); + + fb.mark(); + // expect increased message counter + assertEquals(2, fb.getMessageCount()); + + fb.mark(); + fb.mark(); + // no change to the counter expected for consecutive mark calls + assertEquals(2, fb.getMessageCount()); + } } diff --git a/communication/src/test/groovy/datadog/communication/serialization/aiguard/MessageWriterTest.groovy b/communication/src/test/groovy/datadog/communication/serialization/aiguard/MessageWriterTest.groovy index 3328a2330ad..45191a2fc8f 100644 --- a/communication/src/test/groovy/datadog/communication/serialization/aiguard/MessageWriterTest.groovy +++ b/communication/src/test/groovy/datadog/communication/serialization/aiguard/MessageWriterTest.groovy @@ -116,4 +116,127 @@ class MessageWriterTest extends DDSpecification { private static String asString(final Value value) { return value.asStringValue().asString() } + + void 'test write message with text content parts'() { + given: + final message = AIGuard.Message.message('user', [ + AIGuard.ContentPart.text('Hello world') + ]) + + when: + writer.writeObject(message, encodingCache) + + then: + try (final unpacker = MessagePack.newDefaultUnpacker(buffer.slice())) { + final value = asStringKeyMap(unpacker.unpackValue()) + value.size() == 2 + asString(value.role) == 'user' + + final contentParts = value.content.asArrayValue().list() + contentParts.size() == 1 + + final part = asStringKeyMap(contentParts[0]) + asString(part.type) == 'text' + asString(part.text) == 'Hello world' + } + } + + void 'test write message with image_url content parts'() { + given: + final message = AIGuard.Message.message('user', [ + AIGuard.ContentPart.imageUrl('https://example.com/image.jpg') + ]) + + when: + writer.writeObject(message, encodingCache) + + then: + try (final unpacker = MessagePack.newDefaultUnpacker(buffer.slice())) { + final value = asStringKeyMap(unpacker.unpackValue()) + value.size() == 2 + asString(value.role) == 'user' + + final contentParts = value.content.asArrayValue().list() + contentParts.size() == 1 + + final part = asStringKeyMap(contentParts[0]) + asString(part.type) == 'image_url' + + final imageUrl = asStringKeyMap(part.image_url) + asString(imageUrl.url) == 'https://example.com/image.jpg' + } + } + + void 'test write message with mixed content parts'() { + given: + final message = AIGuard.Message.message('user', [ + AIGuard.ContentPart.text('Describe this:'), + AIGuard.ContentPart.imageUrl('https://example.com/image.jpg'), + AIGuard.ContentPart.text('What is it?') + ]) + + when: + writer.writeObject(message, encodingCache) + + then: + try (final unpacker = MessagePack.newDefaultUnpacker(buffer.slice())) { + final value = asStringKeyMap(unpacker.unpackValue()) + value.size() == 2 + asString(value.role) == 'user' + + final contentParts = value.content.asArrayValue().list() + contentParts.size() == 3 + + final part1 = asStringKeyMap(contentParts[0]) + asString(part1.type) == 'text' + asString(part1.text) == 'Describe this:' + + final part2 = asStringKeyMap(contentParts[1]) + asString(part2.type) == 'image_url' + final imageUrl = asStringKeyMap(part2.image_url) + asString(imageUrl.url) == 'https://example.com/image.jpg' + + final part3 = asStringKeyMap(contentParts[2]) + asString(part3.type) == 'text' + asString(part3.text) == 'What is it?' + } + } + + void 'test content parts type serializes as string not integer'() { + given: + final message = AIGuard.Message.message('user', [ + AIGuard.ContentPart.text('Test') + ]) + + when: + writer.writeObject(message, encodingCache) + + then: + try (final unpacker = MessagePack.newDefaultUnpacker(buffer.slice())) { + final value = asStringKeyMap(unpacker.unpackValue()) + final contentParts = value.content.asArrayValue().list() + final part = asStringKeyMap(contentParts[0]) + + // Verify type is a string value, not an integer + part.type.isStringValue() + !part.type.isIntegerValue() + asString(part.type) == 'text' + } + } + + void 'test backward compatibility with string content'() { + given: + final message = AIGuard.Message.message('user', 'Plain text message') + + when: + writer.writeObject(message, encodingCache) + + then: + try (final unpacker = MessagePack.newDefaultUnpacker(buffer.slice())) { + final value = asStringValueMap(unpacker.unpackValue()) + value.size() == 2 + value.role == 'user' + value.content == 'Plain text message' + } + } } diff --git a/communication/src/test/java/datadog/communication/ddagent/AgentVersionTest.java b/communication/src/test/java/datadog/communication/ddagent/AgentVersionTest.java new file mode 100644 index 00000000000..d9a1cd0d966 --- /dev/null +++ b/communication/src/test/java/datadog/communication/ddagent/AgentVersionTest.java @@ -0,0 +1,64 @@ +package datadog.communication.ddagent; + +import static org.junit.jupiter.api.Assertions.*; + +import org.junit.jupiter.api.Test; + +class AgentVersionTest { + + @Test + void testIsVersionBelow_VersionBelowThreshold() { + assertTrue(AgentVersion.isVersionBelow("7.64.0", 7, 65, 0)); + assertTrue(AgentVersion.isVersionBelow("7.64.9", 7, 65, 0)); + assertTrue(AgentVersion.isVersionBelow("6.99.99", 7, 65, 0)); + assertTrue(AgentVersion.isVersionBelow("7.0.0", 7, 65, 0)); + } + + @Test + void testIsVersionBelow_VersionEqualToThreshold() { + assertFalse(AgentVersion.isVersionBelow("7.65.0", 7, 65, 0)); + } + + @Test + void testIsVersionBelow_VersionAboveThreshold() { + assertFalse(AgentVersion.isVersionBelow("7.65.1", 7, 65, 0)); + assertFalse(AgentVersion.isVersionBelow("7.66.0", 7, 65, 0)); + assertFalse(AgentVersion.isVersionBelow("8.0.0", 7, 65, 0)); + assertFalse(AgentVersion.isVersionBelow("7.65.10", 7, 65, 0)); + } + + @Test + void testIsVersionBelow_MajorVersionComparison() { + assertTrue(AgentVersion.isVersionBelow("6.100.100", 7, 0, 0)); + assertFalse(AgentVersion.isVersionBelow("8.0.0", 7, 100, 100)); + } + + @Test + void testIsVersionBelow_MinorVersionComparison() { + assertTrue(AgentVersion.isVersionBelow("7.64.100", 7, 65, 0)); + assertFalse(AgentVersion.isVersionBelow("7.66.0", 7, 65, 100)); + } + + @Test + void testIsVersionBelow_WithSuffix() { + assertTrue(AgentVersion.isVersionBelow("7.64.0-rc.1", 7, 65, 0)); + assertTrue(AgentVersion.isVersionBelow("7.65.0-rc.1", 7, 65, 0)); + assertFalse(AgentVersion.isVersionBelow("7.65.1-snapshot", 7, 65, 0)); + } + + @Test + void testIsVersionBelow_NullOrEmpty() { + assertTrue(AgentVersion.isVersionBelow(null, 7, 65, 0)); + assertTrue(AgentVersion.isVersionBelow("", 7, 65, 0)); + } + + @Test + void testIsVersionBelow_InvalidFormat() { + assertTrue(AgentVersion.isVersionBelow("invalid", 7, 65, 0)); + assertTrue(AgentVersion.isVersionBelow("7", 7, 65, 0)); + assertTrue(AgentVersion.isVersionBelow("7.", 7, 65, 0)); + assertTrue(AgentVersion.isVersionBelow("7.65", 7, 65, 0)); + assertTrue(AgentVersion.isVersionBelow("7.65.", 7, 65, 0)); + assertTrue(AgentVersion.isVersionBelow("a.b.c", 7, 65, 0)); + } +} diff --git a/dd-trace-core/src/test/java/datadog/trace/common/writer/ddagent/CachingTest.java b/communication/src/test/java/datadog/communication/serialization/CachingTest.java similarity index 96% rename from dd-trace-core/src/test/java/datadog/trace/common/writer/ddagent/CachingTest.java rename to communication/src/test/java/datadog/communication/serialization/CachingTest.java index 51c6f1d79fe..1ca9cc9b12c 100644 --- a/dd-trace-core/src/test/java/datadog/trace/common/writer/ddagent/CachingTest.java +++ b/communication/src/test/java/datadog/communication/serialization/CachingTest.java @@ -1,4 +1,4 @@ -package datadog.trace.common.writer.ddagent; +package datadog.communication.serialization; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; diff --git a/dd-trace-core/src/test/java/datadog/trace/common/writer/ddagent/GenerationalUtf8CacheTest.java b/communication/src/test/java/datadog/communication/serialization/GenerationalUtf8CacheTest.java similarity index 98% rename from dd-trace-core/src/test/java/datadog/trace/common/writer/ddagent/GenerationalUtf8CacheTest.java rename to communication/src/test/java/datadog/communication/serialization/GenerationalUtf8CacheTest.java index facd64e71dc..b8fb3fde316 100644 --- a/dd-trace-core/src/test/java/datadog/trace/common/writer/ddagent/GenerationalUtf8CacheTest.java +++ b/communication/src/test/java/datadog/communication/serialization/GenerationalUtf8CacheTest.java @@ -1,4 +1,4 @@ -package datadog.trace.common.writer.ddagent; +package datadog.communication.serialization; import static org.junit.jupiter.api.Assertions.assertArrayEquals; import static org.junit.jupiter.api.Assertions.assertEquals; @@ -181,7 +181,7 @@ static final String nextTag() { int tagIndex = random.nextInt(TAGS.length + 1); if (tagIndex >= TAGS.length) { - return "tag-" + Integer.toString(random.nextInt()); + return "tag-" + random.nextInt(); } else { return TAGS[tagIndex]; } diff --git a/dd-trace-core/src/test/java/datadog/trace/common/writer/ddagent/SimpleUtf8CacheTest.java b/communication/src/test/java/datadog/communication/serialization/SimpleUtf8CacheTest.java similarity index 97% rename from dd-trace-core/src/test/java/datadog/trace/common/writer/ddagent/SimpleUtf8CacheTest.java rename to communication/src/test/java/datadog/communication/serialization/SimpleUtf8CacheTest.java index 521bf229e4d..d07221bd892 100644 --- a/dd-trace-core/src/test/java/datadog/trace/common/writer/ddagent/SimpleUtf8CacheTest.java +++ b/communication/src/test/java/datadog/communication/serialization/SimpleUtf8CacheTest.java @@ -1,4 +1,4 @@ -package datadog.trace.common.writer.ddagent; +package datadog.communication.serialization; import static org.junit.jupiter.api.Assertions.assertArrayEquals; import static org.junit.jupiter.api.Assertions.assertEquals; @@ -128,7 +128,7 @@ static final String nextTag() { int tagIndex = random.nextInt(TAGS.length + 1); if (tagIndex >= TAGS.length) { - return "tag-" + Integer.toString(random.nextInt()); + return "tag-" + random.nextInt(); } else { return TAGS[tagIndex]; } diff --git a/communication/src/test/java/datadog/communication/serialization/msgpack/MsgPackWriterTest.java b/communication/src/test/java/datadog/communication/serialization/msgpack/MsgPackWriterTest.java index 48a7b72cb0b..4a3f773fdc5 100644 --- a/communication/src/test/java/datadog/communication/serialization/msgpack/MsgPackWriterTest.java +++ b/communication/src/test/java/datadog/communication/serialization/msgpack/MsgPackWriterTest.java @@ -1,10 +1,12 @@ package datadog.communication.serialization.msgpack; import static datadog.trace.util.stacktrace.StackTraceEvent.DEFAULT_LANGUAGE; +import static java.nio.charset.StandardCharsets.UTF_8; import static org.junit.jupiter.api.Assertions.assertArrayEquals; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assertions.fail; import datadog.communication.serialization.ByteBufferConsumer; import datadog.communication.serialization.Codec; @@ -19,7 +21,6 @@ import java.math.BigDecimal; import java.nio.ByteBuffer; import java.nio.CharBuffer; -import java.nio.charset.StandardCharsets; import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; @@ -27,12 +28,15 @@ import java.util.List; import java.util.Map; import java.util.concurrent.atomic.AtomicInteger; -import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; import org.msgpack.core.MessagePack; import org.msgpack.core.MessageUnpacker; public class MsgPackWriterTest { + // Explicit escapes for non-ASCII chars to make test independent of container settings. + private static final String NON_ASCII_STRING = "foob\u00E1r_\u263a"; // foobár_☺ + private static final byte[] NON_ASCII_BYTES = NON_ASCII_STRING.getBytes(UTF_8); + private static final int NON_ASCII_BUFFER_CAPACITY = NON_ASCII_BYTES.length + 1; @Test public void testOverflow() { @@ -71,13 +75,16 @@ public void testFlushOfOverflow() { newBuffer( 2 + 25, // enough space for a 25 element string and its 2 byte header (messageCount, buffer) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer); - for (int i = 0; i < messageCount; ++i) { - try { - flushed.add(unpacker.unpackString()); - } catch (Exception error) { - Assertions.fail(error.getMessage()); + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer)) { + for (int i = 0; i < messageCount; ++i) { + try { + flushed.add(unpacker.unpackString()); + } catch (Exception error) { + fail(error.getMessage()); + } } + } catch (IOException error) { + fail(error.getMessage()); } })); assertTrue(packer.format("abcdefghijklm", mapper), "data fits in buffer"); @@ -117,13 +124,12 @@ public void testWriteBinary() { newBuffer( 25, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { int length = unpacker.unpackBinaryHeader(); assertEquals(4, length); assertArrayEquals(data, unpacker.readPayload(length)); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format( @@ -138,16 +144,36 @@ public void testWriteBinaryNoArgVariant() { newBuffer( 10, (messageCount, buffer) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer); - try { - assertEquals(unpacker.unpackBinaryHeader(), 6); - assertArrayEquals( - unpacker.readPayload(6), "foobar".getBytes(StandardCharsets.UTF_8)); + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer)) { + assertEquals(6, unpacker.unpackBinaryHeader()); + assertArrayEquals(unpacker.readPayload(6), "foobar".getBytes(UTF_8)); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); - writer.writeBinary("foobar".getBytes(StandardCharsets.UTF_8)); + writer.writeBinary("foobar".getBytes(UTF_8)); + } + + @Test + public void testWriteBinaryLongPair() { + final long hi = 0x1122334455667788L; + final long lo = 0x99AABBCCDDEEFF00L; + final byte[] data = ByteBuffer.allocate(16).putLong(hi).putLong(lo).array(); + MessageFormatter messageFormatter = + new MsgPackWriter( + newBuffer( + 25, + (messageCount, buffy) -> { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { + int length = unpacker.unpackBinaryHeader(); + assertEquals(16, length); + assertArrayEquals(data, unpacker.readPayload(length)); + } catch (IOException e) { + fail(e.getMessage()); + } + })); + messageFormatter.format(data, (ignored, writable) -> writable.writeBinary(hi, lo)); + messageFormatter.flush(); } @Test @@ -158,13 +184,12 @@ public void testWriteBinaryAsObject() { newBuffer( 25, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { int length = unpacker.unpackBinaryHeader(); assertEquals(4, length); assertArrayEquals(data, unpacker.readPayload(length)); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(data, (ba, writable) -> writable.writeObject(ba, null)); @@ -179,13 +204,12 @@ public void testWriteByteBuffer() { newBuffer( 25, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { int length = unpacker.unpackBinaryHeader(); assertEquals(4, length); assertArrayEquals(data, unpacker.readPayload(length)); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(ByteBuffer.wrap(data), (bb, writable) -> writable.writeBinary(bb)); @@ -200,13 +224,12 @@ public void testWriteByteBufferAsObject() { newBuffer( 25, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { int length = unpacker.unpackBinaryHeader(); assertEquals(4, length); assertArrayEquals(data, unpacker.readPayload(length)); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format( @@ -221,11 +244,10 @@ public void testWriteNull() { newBuffer( 25, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { unpacker.unpackNil(); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(null, (x, w) -> w.writeObject(x, null)); @@ -239,11 +261,10 @@ public void testWriteBooleanAsObject() { newBuffer( 25, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertTrue(unpacker.unpackBoolean()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(true, (x, w) -> w.writeObject(x, null)); @@ -257,11 +278,10 @@ public void testWriteBoolean() { newBuffer( 25, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertTrue(unpacker.unpackBoolean()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(true, (x, w) -> w.writeBoolean(x)); @@ -276,11 +296,10 @@ public void testWriteGenericNumber() { newBuffer( 25, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertEquals(data.doubleValue(), unpacker.unpackDouble(), 0); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(data, (Mapper) (x, w) -> w.writeObject(x, null)); @@ -295,11 +314,10 @@ public void testWriteCharArray() { newBuffer( 25, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertEquals(data, unpacker.unpackString()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(data.toCharArray(), (x, w) -> w.writeObject(x, null)); @@ -314,11 +332,10 @@ public void testWriteUTF8ByteString() { newBuffer( 25, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertEquals("xyz", unpacker.unpackString()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(utf8BytesString, (x, w) -> w.writeObject(x, null)); @@ -333,14 +350,13 @@ public void testWriteBooleanArray() { newBuffer( 25, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertEquals(4, unpacker.unpackArrayHeader()); for (boolean datum : data) { assertEquals(datum, unpacker.unpackBoolean()); } } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(data, (x, w) -> w.writeObject(x, null)); @@ -355,14 +371,13 @@ public void testWriteFloatArray() { newBuffer( 100, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertEquals(4, unpacker.unpackArrayHeader()); for (float datum : data) { assertEquals(datum, unpacker.unpackFloat(), 0.001); } } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(data, (x, w) -> w.writeObject(x, null)); @@ -377,14 +392,13 @@ public void testWriteDoubleArray() { newBuffer( 100, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertEquals(4, unpacker.unpackArrayHeader()); for (double datum : data) { assertEquals(datum, unpacker.unpackDouble(), 0.001); } } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(data, (x, w) -> w.writeObject(x, null)); @@ -399,14 +413,13 @@ public void testWriteLongArray() { newBuffer( 100, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertEquals(4, unpacker.unpackArrayHeader()); for (long datum : data) { assertEquals(datum, unpacker.unpackLong()); } } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(data, (x, w) -> w.writeObject(x, null)); @@ -421,14 +434,13 @@ public void testWriteIntArray() { newBuffer( 100, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertEquals(4, unpacker.unpackArrayHeader()); for (int datum : data) { assertEquals(datum, unpacker.unpackInt()); } } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(data, (x, w) -> w.writeObject(x, null)); @@ -443,14 +455,13 @@ public void testWriteShortArray() { newBuffer( 100, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertEquals(4, unpacker.unpackArrayHeader()); for (short datum : data) { assertEquals(datum, unpacker.unpackInt()); } } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(data, (x, w) -> w.writeObject(x, null)); @@ -465,11 +476,10 @@ public void testWriteLongBoxed() { newBuffer( 100, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertEquals(data, unpacker.unpackLong()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(data, (x, w) -> w.writeObject(x, null)); @@ -484,12 +494,11 @@ public void testWriteLongPrimitive() { newBuffer( 100, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertEquals(data, unpacker.unpackLong()); assertEquals(data, unpacker.unpackLong()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format( @@ -509,14 +518,13 @@ public void testWriteNegativeLongPrimitive() { newBuffer( 100, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertEquals(data, unpacker.unpackLong()); assertEquals(data, unpacker.unpackLong()); // Can't unpack unsigned long directly as the unpacker refuses negative values assertEquals(data, unpacker.unpackValue().asNumberValue().toLong()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format( @@ -537,11 +545,10 @@ public void testWriteIntBoxed() { newBuffer( 100, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertEquals(data, unpacker.unpackInt()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(data, (x, w) -> w.writeObject(x, null)); @@ -556,11 +563,10 @@ public void testWriteIntPrimitive() { newBuffer( 100, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertEquals(data, unpacker.unpackInt()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(data, (x, w) -> w.writeInt(x)); @@ -575,11 +581,10 @@ public void testWriteShortBoxed() { newBuffer( 100, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertEquals(data, unpacker.unpackInt()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(data, (x, w) -> w.writeObject(x, null)); @@ -594,11 +599,10 @@ public void testUnknownObject() { newBuffer( 100, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertEquals(data.toString(), unpacker.unpackString()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(data, (x, w) -> w.writeObject(x, null)); @@ -613,13 +617,12 @@ public void testWriteObjectArray() { newBuffer( 100, (messageCount, buffy) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffy)) { assertEquals(data.length, unpacker.unpackArrayHeader()); assertEquals(data[0].toString(), unpacker.unpackString()); assertEquals(data[1].toString(), unpacker.unpackString()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(data, (x, w) -> w.writeObject(x, null)); @@ -628,18 +631,17 @@ public void testWriteObjectArray() { @Test public void testWriteStringUTF8BytesString() { - UTF8BytesString value = UTF8BytesString.create("foobár"); + UTF8BytesString value = UTF8BytesString.create(NON_ASCII_STRING); MsgPackWriter writer = new MsgPackWriter( newBuffer( - 20, + NON_ASCII_BUFFER_CAPACITY * 2, (messageCount, buffer) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer); - try { - assertEquals(unpacker.unpackString(), "foobár"); - assertEquals(unpacker.unpackString(), "foobár"); + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer)) { + assertEquals(NON_ASCII_STRING, unpacker.unpackString()); + assertEquals(NON_ASCII_STRING, unpacker.unpackString()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); writer.writeObjectString(value, null); @@ -653,12 +655,11 @@ public void testWriteStringNull() { newBuffer( 20, (messageCount, buffer) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer)) { unpacker.unpackNil(); unpacker.unpackNil(); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); writer.writeObjectString(null, null); @@ -671,7 +672,7 @@ public void testWriteObjectStringGeneralPath() { new Object() { @Override public String toString() { - return "foobár"; + return NON_ASCII_STRING; } }; MsgPackWriter writer = @@ -679,16 +680,15 @@ public String toString() { newBuffer( 40, (messageCount, buffer) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer); - try { - assertEquals(unpacker.unpackString(), "foobár"); - assertEquals(unpacker.unpackString(), "foobàr"); + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer)) { + assertEquals(NON_ASCII_STRING, unpacker.unpackString()); + assertEquals(NON_ASCII_STRING, unpacker.unpackString()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); writer.writeObjectString(value, null); - writer.writeObjectString(value, s -> "foobàr".getBytes(StandardCharsets.UTF_8)); + writer.writeObjectString(value, s -> NON_ASCII_BYTES); } @Test @@ -696,16 +696,15 @@ public void testWriteStringGeneralCharSequence() { MsgPackWriter writer = new MsgPackWriter( newBuffer( - 10, + NON_ASCII_BUFFER_CAPACITY, (messageCount, buffer) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer); - try { - assertEquals(unpacker.unpackString(), "foobár"); + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer)) { + assertEquals(NON_ASCII_STRING, unpacker.unpackString()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); - CharBuffer charSeq = CharBuffer.wrap("foobár"); + CharBuffer charSeq = CharBuffer.wrap(NON_ASCII_STRING); writer.writeString(charSeq, null); } @@ -714,16 +713,15 @@ public void testWriteStringEncodingCache() { MsgPackWriter writer = new MsgPackWriter( newBuffer( - 10, + NON_ASCII_BUFFER_CAPACITY, (messageCount, buffer) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer); - try { - assertEquals(unpacker.unpackString(), "foobár"); + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer)) { + assertEquals(NON_ASCII_STRING, unpacker.unpackString()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); - writer.writeString("", s -> "foobár".getBytes(StandardCharsets.UTF_8)); + writer.writeString("", s -> NON_ASCII_BYTES); } @Test @@ -733,14 +731,13 @@ public void testStartArray() { newBuffer( 10, (messageCount, buffer) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer); - try { - assertEquals(unpacker.unpackArrayHeader(), 1); - assertEquals(unpacker.unpackArrayHeader(), 0xFFFF); - assertEquals(unpacker.unpackArrayHeader(), 0x10000); - assertEquals(unpacker.unpackArrayHeader(), 1); + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer)) { + assertEquals(1, unpacker.unpackArrayHeader()); + assertEquals(0xFFFF, unpacker.unpackArrayHeader()); + assertEquals(0x10000, unpacker.unpackArrayHeader()); + assertEquals(1, unpacker.unpackArrayHeader()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); writer.startArray(1); @@ -756,13 +753,12 @@ public void testStartMap() { newBuffer( 10, (messageCount, buffer) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer); - try { - assertEquals(unpacker.unpackMapHeader(), 1); - assertEquals(unpacker.unpackMapHeader(), 0xFFFF); - assertEquals(unpacker.unpackMapHeader(), 0x10000); + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer)) { + assertEquals(1, unpacker.unpackMapHeader()); + assertEquals(0xFFFF, unpacker.unpackMapHeader()); + assertEquals(0x10000, unpacker.unpackMapHeader()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); writer.startMap(1); @@ -777,13 +773,12 @@ public void testStartStringHeader() { newBuffer( 10, (messageCount, buffer) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer); - try { - assertEquals(unpacker.unpackRawStringHeader(), 1); - assertEquals(unpacker.unpackRawStringHeader(), 0xFFFF); - assertEquals(unpacker.unpackRawStringHeader(), 0x10000); + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer)) { + assertEquals(1, unpacker.unpackRawStringHeader()); + assertEquals(0xFFFF, unpacker.unpackRawStringHeader()); + assertEquals(0x10000, unpacker.unpackRawStringHeader()); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); writer.writeStringHeader(1); @@ -814,11 +809,10 @@ public void testSimpleStackTraceEvent() { newBuffer( 1000, (messageCount, buffer) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer)) { checkStackTraceEvent(data, unpacker); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(data, (x, w) -> w.writeObject(x, null)); @@ -833,11 +827,10 @@ public void testSimpleStackTraceFrame() { newBuffer( 1000, (messageCount, buffer) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer)) { checkStackTraceFrame(frame, unpacker); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(frame, (x, w) -> w.writeObject(x, null)); @@ -850,11 +843,10 @@ private void testStackTraceBatch(final Map> batch) newBuffer( 100000, (messageCount, buffer) -> { - MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer); - try { + try (MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(buffer)) { checkStackTraceBatch(batch, unpacker); } catch (IOException e) { - Assertions.fail(e.getMessage()); + fail(e.getMessage()); } })); messageFormatter.format(batch, (x, w) -> w.writeObject(x, null)); diff --git a/communication/src/test/resources/agent-features/agent-info-with-client-dropping.json b/communication/src/test/resources/agent-features/agent-info-with-client-dropping.json index 29baa64c04a..18467a1170b 100644 --- a/communication/src/test/resources/agent-features/agent-info-with-client-dropping.json +++ b/communication/src/test/resources/agent-features/agent-info-with-client-dropping.json @@ -1,5 +1,5 @@ { - "version": "0.99.0", + "version": "7.65.0", "git_commit": "fab047e10", "build_date": "2020-12-04 15:57:06.74187 +0200 EET m=+0.029001792", "endpoints": [ diff --git a/communication/src/test/resources/agent-features/agent-info.json b/communication/src/test/resources/agent-features/agent-info.json index 93b9f98f2d1..dabd457115b 100644 --- a/communication/src/test/resources/agent-features/agent-info.json +++ b/communication/src/test/resources/agent-features/agent-info.json @@ -8,6 +8,7 @@ "/v0.4/traces", "/v0.4/services", "/v0.5/traces", + "/v1.0/traces", "/v0.6/stats", "/profiling/v1/input", "/v0.1/pipeline_stats", diff --git a/components/context/gradle.lockfile b/components/context/gradle.lockfile index 0f0886b942b..e7ab997a313 100644 --- a/components/context/gradle.lockfile +++ b/components/context/gradle.lockfile @@ -3,109 +3,72 @@ # This file is expected to be part of source control. ch.qos.logback:logback-classic:1.2.13=testCompileClasspath,testRuntimeClasspath ch.qos.logback:logback-core:1.2.13=testCompileClasspath,testRuntimeClasspath -com.beust:jcommander:1.78=testRuntimeClasspath -com.github.javaparser:javaparser-core:3.25.6=codenarc,testCompileClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.2.0=compileClasspath,testCompileClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.7.3=spotbugs -com.github.spotbugs:spotbugs:4.7.3=spotbugs +com.github.javaparser:javaparser-core:3.25.6=codenarc +com.github.spotbugs:spotbugs-annotations:4.9.8=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath +com.github.spotbugs:spotbugs:4.9.8=spotbugs +com.github.stephenc.jcip:jcip-annotations:1.0-1=spotbugs com.google.code.findbugs:jsr305:3.0.2=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -com.google.code.gson:gson:2.9.1=spotbugs -com.thoughtworks.qdox:qdox:1.12.1=codenarc,testRuntimeClasspath -commons-codec:commons-codec:1.15=spotbugs -de.thetaphi:forbiddenapis:3.8=compileClasspath -info.picocli:picocli:4.6.3=testRuntimeClasspath +com.google.code.gson:gson:2.13.2=spotbugs +com.google.errorprone:error_prone_annotations:2.41.0=spotbugs +com.thoughtworks.qdox:qdox:1.12.1=codenarc +commons-io:commons-io:2.20.0=spotbugs +de.thetaphi:forbiddenapis:3.10=compileClasspath io.leangen.geantyref:geantyref:1.3.16=testRuntimeClasspath -jaxen:jaxen:1.2.0=spotbugs -jline:jline:2.14.6=testRuntimeClasspath -junit:junit:4.13.2=testCompileClasspath,testRuntimeClasspath +jaxen:jaxen:2.0.0=spotbugs net.bytebuddy:byte-buddy-agent:1.12.8=testRuntimeClasspath net.bytebuddy:byte-buddy:1.12.8=testRuntimeClasspath -net.jcip:jcip-annotations:1.0=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -net.sf.saxon:Saxon-HE:11.4=spotbugs +net.sf.saxon:Saxon-HE:12.9=spotbugs org.apache.ant:ant-antlr:1.10.14=codenarc -org.apache.ant:ant-antlr:1.10.15=testRuntimeClasspath org.apache.ant:ant-junit:1.10.14=codenarc -org.apache.ant:ant-junit:1.10.15=testRuntimeClasspath -org.apache.ant:ant-launcher:1.10.15=testRuntimeClasspath -org.apache.ant:ant:1.10.15=testCompileClasspath,testRuntimeClasspath -org.apache.bcel:bcel:6.5.0=spotbugs -org.apache.commons:commons-lang3:3.12.0=spotbugs -org.apache.commons:commons-text:1.10.0=spotbugs -org.apache.httpcomponents.client5:httpclient5:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5-h2:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5:5.1.3=spotbugs -org.apache.logging.log4j:log4j-api:2.19.0=spotbugs -org.apache.logging.log4j:log4j-core:2.19.0=spotbugs +org.apache.bcel:bcel:6.11.0=spotbugs +org.apache.commons:commons-lang3:3.19.0=spotbugs +org.apache.commons:commons-text:1.14.0=spotbugs +org.apache.logging.log4j:log4j-api:2.25.2=spotbugs +org.apache.logging.log4j:log4j-core:2.25.2=spotbugs org.apiguardian:apiguardian-api:1.1.2=testCompileClasspath -org.codehaus.groovy:groovy-all:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-ant:3.0.23=codenarc -org.codehaus.groovy:groovy-ant:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-astbuilder:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-cli-picocli:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-console:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-datetime:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-docgenerator:3.0.23=codenarc -org.codehaus.groovy:groovy-docgenerator:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-groovydoc:3.0.23=codenarc -org.codehaus.groovy:groovy-groovydoc:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-groovysh:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jmx:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-json:3.0.23=codenarc -org.codehaus.groovy:groovy-json:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jsr223:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-macro:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-nio:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-servlet:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-sql:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-swing:3.0.24=testCompileClasspath,testRuntimeClasspath +org.codehaus.groovy:groovy-json:3.0.25=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-templates:3.0.23=codenarc -org.codehaus.groovy:groovy-templates:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test-junit5:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-testng:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-xml:3.0.23=codenarc -org.codehaus.groovy:groovy-xml:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy:3.0.23=codenarc -org.codehaus.groovy:groovy:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codenarc:CodeNarc:3.6.0=codenarc -org.dom4j:dom4j:2.1.3=spotbugs +org.codehaus.groovy:groovy:3.0.25=testCompileClasspath,testRuntimeClasspath +org.codenarc:CodeNarc:3.7.0=codenarc +org.dom4j:dom4j:2.2.0=spotbugs org.gmetrics:GMetrics:2.1.0=codenarc -org.hamcrest:hamcrest-core:1.3=testCompileClasspath,testRuntimeClasspath org.hamcrest:hamcrest:3.0=testCompileClasspath,testRuntimeClasspath org.jacoco:org.jacoco.agent:0.8.14=jacocoAgent,jacocoAnt org.jacoco:org.jacoco.ant:0.8.14=jacocoAnt org.jacoco:org.jacoco.core:0.8.14=jacocoAnt org.jacoco:org.jacoco.report:0.8.14=jacocoAnt -org.junit.jupiter:junit-jupiter-api:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-engine:5.12.2=testRuntimeClasspath -org.junit.jupiter:junit-jupiter-params:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-commons:1.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-engine:1.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-launcher:1.12.2=testRuntimeClasspath -org.junit:junit-bom:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit:junit-bom:5.9.1=spotbugs +org.junit.jupiter:junit-jupiter-api:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-engine:5.14.1=testRuntimeClasspath +org.junit.jupiter:junit-jupiter-params:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-commons:1.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-engine:1.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-launcher:1.14.1=testRuntimeClasspath +org.junit:junit-bom:5.14.0=spotbugs +org.junit:junit-bom:5.14.1=testCompileClasspath,testRuntimeClasspath org.mockito:mockito-core:4.4.0=testRuntimeClasspath org.objenesis:objenesis:3.3=testCompileClasspath,testRuntimeClasspath org.opentest4j:opentest4j:1.3.0=testCompileClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.4=spotbugs -org.ow2.asm:asm-commons:9.4=spotbugs -org.ow2.asm:asm-commons:9.9=jacocoAnt -org.ow2.asm:asm-tree:9.4=spotbugs -org.ow2.asm:asm-tree:9.9=jacocoAnt -org.ow2.asm:asm-util:9.4=spotbugs -org.ow2.asm:asm:9.4=spotbugs -org.ow2.asm:asm:9.9=jacocoAnt +org.ow2.asm:asm-analysis:9.9=spotbugs +org.ow2.asm:asm-commons:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-tree:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-util:9.9=spotbugs +org.ow2.asm:asm:9.9=jacocoAnt,spotbugs org.slf4j:jcl-over-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath org.slf4j:jul-to-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath org.slf4j:log4j-over-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath org.slf4j:slf4j-api:1.7.32=testCompileClasspath,testRuntimeClasspath -org.slf4j:slf4j-api:2.0.0=spotbugs,spotbugsSlf4j -org.slf4j:slf4j-simple:2.0.0=spotbugsSlf4j -org.spockframework:spock-bom:2.4-M6-groovy-3.0=testCompileClasspath,testRuntimeClasspath -org.spockframework:spock-core:2.4-M6-groovy-3.0=testCompileClasspath,testRuntimeClasspath -org.testng:testng:7.5.1=testRuntimeClasspath -org.webjars:jquery:3.5.1=testRuntimeClasspath -org.xmlresolver:xmlresolver:4.4.3=spotbugs -xml-apis:xml-apis:1.4.01=spotbugs +org.slf4j:slf4j-api:2.0.17=spotbugs,spotbugsSlf4j +org.slf4j:slf4j-simple:2.0.17=spotbugsSlf4j +org.spockframework:spock-bom:2.4-groovy-3.0=testCompileClasspath,testRuntimeClasspath +org.spockframework:spock-core:2.4-groovy-3.0=testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-junit:1.2.1=testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-parser:1.2.0=testCompileClasspath,testRuntimeClasspath +org.xmlresolver:xmlresolver:5.3.3=spotbugs empty=annotationProcessor,runtimeClasspath,spotbugsPlugins,testAnnotationProcessor diff --git a/components/context/src/main/java/datadog/context/ContextKey.java b/components/context/src/main/java/datadog/context/ContextKey.java index 0308fe8c1dc..cc1cd6d8cb0 100644 --- a/components/context/src/main/java/datadog/context/ContextKey.java +++ b/components/context/src/main/java/datadog/context/ContextKey.java @@ -10,8 +10,10 @@ */ public final class ContextKey { private static final AtomicInteger NEXT_INDEX = new AtomicInteger(0); + /** The key name, for debugging purpose only. */ private final String name; + /** The key unique context, related to {@link IndexedContext} implementation. */ final int index; diff --git a/components/context/src/main/java/datadog/context/propagation/Concern.java b/components/context/src/main/java/datadog/context/propagation/Concern.java index 0e79a688c1c..ed7681369ac 100644 --- a/components/context/src/main/java/datadog/context/propagation/Concern.java +++ b/components/context/src/main/java/datadog/context/propagation/Concern.java @@ -8,8 +8,10 @@ public class Concern { /** The concern default priority. */ public static final int DEFAULT_PRIORITY = 100; + /** The concern name, for debugging purpose only. */ private final String name; + /** The concern priority, lower value means higher priority. */ private final int priority; diff --git a/components/environment/build.gradle.kts b/components/environment/build.gradle.kts index 8d9febfe679..7818fef2463 100644 --- a/components/environment/build.gradle.kts +++ b/components/environment/build.gradle.kts @@ -24,7 +24,7 @@ val excludedClassesCoverage by extra { "datadog.environment.JavaVirtualMachine.JvmOptionsHolder", // depends on OS and JVM vendor "datadog.environment.JvmOptions", // depends on OS and JVM vendor "datadog.environment.OperatingSystem**", // depends on OS - "datadog.environment.ThreadUtils", // depends on JVM version + "datadog.environment.ThreadSupport", // requires Java 21 ) } val excludedClassesBranchCoverage by extra { diff --git a/components/environment/gradle.lockfile b/components/environment/gradle.lockfile index 0ec563798fd..ec6c88c69ae 100644 --- a/components/environment/gradle.lockfile +++ b/components/environment/gradle.lockfile @@ -3,109 +3,72 @@ # This file is expected to be part of source control. ch.qos.logback:logback-classic:1.2.13=testCompileClasspath,testRuntimeClasspath ch.qos.logback:logback-core:1.2.13=testCompileClasspath,testRuntimeClasspath -com.beust:jcommander:1.78=testRuntimeClasspath -com.github.javaparser:javaparser-core:3.25.6=codenarc,testCompileClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.2.0=compileClasspath,testCompileClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.7.3=spotbugs -com.github.spotbugs:spotbugs:4.7.3=spotbugs +com.github.javaparser:javaparser-core:3.25.6=codenarc +com.github.spotbugs:spotbugs-annotations:4.9.8=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath +com.github.spotbugs:spotbugs:4.9.8=spotbugs +com.github.stephenc.jcip:jcip-annotations:1.0-1=spotbugs com.google.code.findbugs:jsr305:3.0.2=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -com.google.code.gson:gson:2.9.1=spotbugs -com.thoughtworks.qdox:qdox:1.12.1=codenarc,testRuntimeClasspath -commons-codec:commons-codec:1.15=spotbugs -de.thetaphi:forbiddenapis:3.8=compileClasspath -info.picocli:picocli:4.6.3=testRuntimeClasspath +com.google.code.gson:gson:2.13.2=spotbugs +com.google.errorprone:error_prone_annotations:2.41.0=spotbugs +com.thoughtworks.qdox:qdox:1.12.1=codenarc +commons-io:commons-io:2.20.0=spotbugs +de.thetaphi:forbiddenapis:3.10=compileClasspath io.leangen.geantyref:geantyref:1.3.16=testRuntimeClasspath -jaxen:jaxen:1.2.0=spotbugs -jline:jline:2.14.6=testRuntimeClasspath -junit:junit:4.13.2=testCompileClasspath,testRuntimeClasspath +jaxen:jaxen:2.0.0=spotbugs net.bytebuddy:byte-buddy-agent:1.12.8=testRuntimeClasspath net.bytebuddy:byte-buddy:1.12.8=testRuntimeClasspath -net.jcip:jcip-annotations:1.0=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -net.sf.saxon:Saxon-HE:11.4=spotbugs +net.sf.saxon:Saxon-HE:12.9=spotbugs org.apache.ant:ant-antlr:1.10.14=codenarc -org.apache.ant:ant-antlr:1.10.15=testRuntimeClasspath org.apache.ant:ant-junit:1.10.14=codenarc -org.apache.ant:ant-junit:1.10.15=testRuntimeClasspath -org.apache.ant:ant-launcher:1.10.15=testRuntimeClasspath -org.apache.ant:ant:1.10.15=testCompileClasspath,testRuntimeClasspath -org.apache.bcel:bcel:6.5.0=spotbugs -org.apache.commons:commons-lang3:3.12.0=spotbugs -org.apache.commons:commons-text:1.10.0=spotbugs -org.apache.httpcomponents.client5:httpclient5:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5-h2:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5:5.1.3=spotbugs -org.apache.logging.log4j:log4j-api:2.19.0=spotbugs -org.apache.logging.log4j:log4j-core:2.19.0=spotbugs +org.apache.bcel:bcel:6.11.0=spotbugs +org.apache.commons:commons-lang3:3.19.0=spotbugs +org.apache.commons:commons-text:1.14.0=spotbugs +org.apache.logging.log4j:log4j-api:2.25.2=spotbugs +org.apache.logging.log4j:log4j-core:2.25.2=spotbugs org.apiguardian:apiguardian-api:1.1.2=testCompileClasspath -org.codehaus.groovy:groovy-all:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-ant:3.0.23=codenarc -org.codehaus.groovy:groovy-ant:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-astbuilder:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-cli-picocli:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-console:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-datetime:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-docgenerator:3.0.23=codenarc -org.codehaus.groovy:groovy-docgenerator:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-groovydoc:3.0.23=codenarc -org.codehaus.groovy:groovy-groovydoc:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-groovysh:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jmx:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-json:3.0.23=codenarc -org.codehaus.groovy:groovy-json:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jsr223:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-macro:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-nio:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-servlet:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-sql:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-swing:3.0.24=testCompileClasspath,testRuntimeClasspath +org.codehaus.groovy:groovy-json:3.0.25=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-templates:3.0.23=codenarc -org.codehaus.groovy:groovy-templates:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test-junit5:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-testng:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-xml:3.0.23=codenarc -org.codehaus.groovy:groovy-xml:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy:3.0.23=codenarc -org.codehaus.groovy:groovy:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codenarc:CodeNarc:3.6.0=codenarc -org.dom4j:dom4j:2.1.3=spotbugs +org.codehaus.groovy:groovy:3.0.25=testCompileClasspath,testRuntimeClasspath +org.codenarc:CodeNarc:3.7.0=codenarc +org.dom4j:dom4j:2.2.0=spotbugs org.gmetrics:GMetrics:2.1.0=codenarc -org.hamcrest:hamcrest-core:1.3=testCompileClasspath,testRuntimeClasspath org.hamcrest:hamcrest:3.0=testCompileClasspath,testRuntimeClasspath org.jacoco:org.jacoco.agent:0.8.14=jacocoAgent,jacocoAnt org.jacoco:org.jacoco.ant:0.8.14=jacocoAnt org.jacoco:org.jacoco.core:0.8.14=jacocoAnt org.jacoco:org.jacoco.report:0.8.14=jacocoAnt -org.junit.jupiter:junit-jupiter-api:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-engine:5.12.2=testRuntimeClasspath -org.junit.jupiter:junit-jupiter-params:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-commons:1.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-engine:1.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-launcher:1.12.2=testRuntimeClasspath -org.junit:junit-bom:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit:junit-bom:5.9.1=spotbugs +org.junit.jupiter:junit-jupiter-api:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-engine:5.14.1=testRuntimeClasspath +org.junit.jupiter:junit-jupiter-params:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-commons:1.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-engine:1.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-launcher:1.14.1=testRuntimeClasspath +org.junit:junit-bom:5.14.0=spotbugs +org.junit:junit-bom:5.14.1=testCompileClasspath,testRuntimeClasspath org.mockito:mockito-core:4.4.0=testRuntimeClasspath org.objenesis:objenesis:3.3=testCompileClasspath,testRuntimeClasspath org.opentest4j:opentest4j:1.3.0=testCompileClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.4=spotbugs -org.ow2.asm:asm-commons:9.4=spotbugs -org.ow2.asm:asm-commons:9.9=jacocoAnt -org.ow2.asm:asm-tree:9.4=spotbugs -org.ow2.asm:asm-tree:9.9=jacocoAnt -org.ow2.asm:asm-util:9.4=spotbugs -org.ow2.asm:asm:9.4=spotbugs -org.ow2.asm:asm:9.9=jacocoAnt +org.ow2.asm:asm-analysis:9.9=spotbugs +org.ow2.asm:asm-commons:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-tree:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-util:9.9=spotbugs +org.ow2.asm:asm:9.9=jacocoAnt,spotbugs org.slf4j:jcl-over-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath org.slf4j:jul-to-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath org.slf4j:log4j-over-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath org.slf4j:slf4j-api:1.7.32=testCompileClasspath,testRuntimeClasspath -org.slf4j:slf4j-api:2.0.0=spotbugs,spotbugsSlf4j -org.slf4j:slf4j-simple:2.0.0=spotbugsSlf4j -org.spockframework:spock-bom:2.4-M6-groovy-3.0=testCompileClasspath,testRuntimeClasspath -org.spockframework:spock-core:2.4-M6-groovy-3.0=testCompileClasspath,testRuntimeClasspath -org.testng:testng:7.5.1=testRuntimeClasspath -org.webjars:jquery:3.5.1=testRuntimeClasspath -org.xmlresolver:xmlresolver:4.4.3=spotbugs -xml-apis:xml-apis:1.4.01=spotbugs +org.slf4j:slf4j-api:2.0.17=spotbugs,spotbugsSlf4j +org.slf4j:slf4j-simple:2.0.17=spotbugsSlf4j +org.spockframework:spock-bom:2.4-groovy-3.0=testCompileClasspath,testRuntimeClasspath +org.spockframework:spock-core:2.4-groovy-3.0=testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-junit:1.2.1=testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-parser:1.2.0=testCompileClasspath,testRuntimeClasspath +org.xmlresolver:xmlresolver:5.3.3=spotbugs empty=annotationProcessor,runtimeClasspath,shadow,spotbugsPlugins,testAnnotationProcessor diff --git a/components/environment/src/main/java/datadog/environment/CommandLine.java b/components/environment/src/main/java/datadog/environment/CommandLine.java index e3b87e18b4b..79f78bef2af 100644 --- a/components/environment/src/main/java/datadog/environment/CommandLine.java +++ b/components/environment/src/main/java/datadog/environment/CommandLine.java @@ -28,7 +28,7 @@ class CommandLine { final String name = getCommandName(); final List arguments = getCommandArguments(); - @SuppressForbidden // split on single-character uses fast path + @SuppressForbidden // split on single-character uses a fast path private List findFullCommand() { String command = SystemProperties.getOrDefault(SUN_JAVA_COMMAND_PROPERTY, "").trim(); return command.isEmpty() ? emptyList() : Arrays.asList(command.split(" ")); diff --git a/components/environment/src/main/java/datadog/environment/JavaVirtualMachine.java b/components/environment/src/main/java/datadog/environment/JavaVirtualMachine.java index fa1bc0636a5..ed1ec0ae215 100644 --- a/components/environment/src/main/java/datadog/environment/JavaVirtualMachine.java +++ b/components/environment/src/main/java/datadog/environment/JavaVirtualMachine.java @@ -93,12 +93,27 @@ public static boolean isOracleJDK8() { && !runtime.name.contains("OpenJDK"); } + public static boolean isHotspot() { + String prop = SystemProperties.getOrDefault("java.vm.name", ""); + if (prop.isEmpty()) { + return false; + } + return prop.contains("OpenJDK") + || prop.contains("HotSpot") + || prop.contains("GraalVM") + || prop.contains("Dynamic Code Evolution"); + } + public static boolean isJ9() { return SystemProperties.getOrDefault("java.vm.name", "").contains("J9"); } + public static boolean isIbm() { + return runtime.vendor.contains("IBM"); + } + public static boolean isIbm8() { - return isJavaVersion(8) && runtime.vendor.contains("IBM"); + return isIbm() && isJavaVersion(8); } public static boolean isGraalVM() { diff --git a/components/environment/src/main/java/datadog/environment/JvmOptions.java b/components/environment/src/main/java/datadog/environment/JvmOptions.java index e78ff890c31..ef549bc4d73 100644 --- a/components/environment/src/main/java/datadog/environment/JvmOptions.java +++ b/components/environment/src/main/java/datadog/environment/JvmOptions.java @@ -24,7 +24,7 @@ class JvmOptions { final String[] PROCFS_CMDLINE = readProcFsCmdLine(); final List VM_OPTIONS = findVmOptions(); - @SuppressForbidden // split on single-character uses fast path + @SuppressForbidden // split on single-character uses a fast path private String[] readProcFsCmdLine() { if (isLinux()) { try { @@ -85,7 +85,7 @@ private List findVmOptions() { return ManagementFactory.getRuntimeMXBean().getInputArguments(); } catch (final Throwable t) { // Throws InvocationTargetException on modularized applications - // with non-opened java.management module + // with a non-opened java.management module System.err.println("WARNING: Unable to get VM args using managed beans"); } return emptyList(); @@ -95,9 +95,9 @@ private List findVmOptions() { // executable // Visible for testing List findVmOptionsFromProcFs(String[] procfsCmdline) { - // Create list of VM options + // Create the list of VM options List vmOptions = new ArrayList<>(); - // Look for first self-standing argument that is not prefixed with "-" or end of VM options + // Look for the first self-standing argument that is not prefixed with "-" or end of VM options // while simultaneously, collect all arguments in the VM options // Starts from 1 as 0 is the java command itself (or native-image) for (int index = 1; index < procfsCmdline.length; index++) { @@ -115,7 +115,7 @@ else if ("-jar".equals(argument) || !argument.startsWith("-")) { // End of VM options break; } - // Otherwise add as VM option + // Otherwise add as a VM option else { vmOptions.add(argument); } @@ -142,7 +142,7 @@ private static List getArgumentsFromFile(String argFile) { List args = new ArrayList<>(); try { for (String line : Files.readAllLines(path)) { - // Use default delimiters that matches argfiles separator specification + // Use default delimiters that match argfiles separator specification StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { args.add(tokenizer.nextToken()); @@ -207,13 +207,4 @@ static List parseOptions(String javaToolOptions) { } return options; } - - private static List split(String str, String delimiter) { - List parts = new ArrayList<>(); - StringTokenizer tokenizer = new StringTokenizer(str, delimiter); - while (tokenizer.hasMoreTokens()) { - parts.add(tokenizer.nextToken()); - } - return parts; - } } diff --git a/components/environment/src/main/java/datadog/environment/ThreadSupport.java b/components/environment/src/main/java/datadog/environment/ThreadSupport.java new file mode 100644 index 00000000000..861215ca837 --- /dev/null +++ b/components/environment/src/main/java/datadog/environment/ThreadSupport.java @@ -0,0 +1,142 @@ +package datadog.environment; + +import java.lang.invoke.MethodHandle; +import java.lang.invoke.MethodHandles; +import java.lang.invoke.MethodType; +import java.util.Optional; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; + +/** + * Helper class for working with {@link Thread}s. + * + *

Uses feature detection and provides static helpers to work with different versions of Java. + * + *

This class is designed to use {@link MethodHandle}s that constant propagate to minimize the + * overhead. + */ +public final class ThreadSupport { + static final MethodHandle THREAD_ID_MH = findThreadIdMethodHandle(); + static final MethodHandle IS_VIRTUAL_MH = findIsVirtualMethodHandle(); + static final MethodHandle NEW_VIRTUAL_THREAD_PER_TASK_EXECUTOR_MH = + findNewVirtualThreadPerTaskExecutorMethodHandle(); + + private ThreadSupport() {} + + /** + * Provides the best identifier available for the current {@link Thread}. Uses {@link + * Thread#threadId()} on 19+ or {@link Thread#getId()} on older JVMs. + * + * @return The best identifier available for the current {@link Thread}. + */ + public static long threadId() { + return threadId(Thread.currentThread()); + } + + /** + * Provides the best identifier available for the given {@link Thread}. Uses {@link + * Thread#threadId()} on 19+ or {@link Thread#getId()} on older JVMs. + * + * @return The best identifier available for the given {@link Thread}. + */ + public static long threadId(Thread thread) { + if (THREAD_ID_MH != null) { + try { + return (long) THREAD_ID_MH.invoke(thread); + } catch (Throwable ignored) { + } + } + return thread.getId(); + } + + /** + * Checks whether virtual threads are supported on this JVM. + * + * @return {@code true} if virtual threads are supported, {@code false} otherwise. + */ + public static boolean supportsVirtualThreads() { + return (IS_VIRTUAL_MH != null); + } + + /** + * Checks whether the current thread is a virtual thread. + * + * @return {@code true} if the current thread is a virtual thread, {@code false} otherwise. + */ + public static boolean isVirtual() { + // IS_VIRTUAL_MH will constant propagate -- then dead code eliminate -- and inline as needed + return IS_VIRTUAL_MH != null && isVirtual(Thread.currentThread()); + } + + /** + * Checks whether the given thread is a virtual thread. + * + * @param thread The thread to check. + * @return {@code true} if the given thread is virtual, {@code false} otherwise. + */ + public static boolean isVirtual(Thread thread) { + // IS_VIRTUAL_MH will constant propagate -- then dead code eliminate -- and inline as needed + if (IS_VIRTUAL_MH != null) { + try { + return (boolean) IS_VIRTUAL_MH.invoke(thread); + } catch (Throwable ignored) { + } + } + return false; + } + + /** + * Returns the virtual thread per task executor if available. + * + * @return The virtual thread per task executor if available wrapped into an {@link Optional}, or + * {@link Optional#empty()} otherwise. + */ + public static Optional newVirtualThreadPerTaskExecutor() { + if (NEW_VIRTUAL_THREAD_PER_TASK_EXECUTOR_MH != null) { + try { + ExecutorService executorService = + (ExecutorService) NEW_VIRTUAL_THREAD_PER_TASK_EXECUTOR_MH.invoke(); + return Optional.of(executorService); + } catch (Throwable ignored) { + } + } + return Optional.empty(); + } + + private static MethodHandle findThreadIdMethodHandle() { + if (JavaVirtualMachine.isJavaVersionAtLeast(19)) { + try { + return MethodHandles.lookup() + .findVirtual(Thread.class, "threadId", MethodType.methodType(long.class)); + } catch (Throwable ignored) { + return null; + } + } + return null; + } + + private static MethodHandle findIsVirtualMethodHandle() { + if (JavaVirtualMachine.isJavaVersionAtLeast(21)) { + try { + return MethodHandles.lookup() + .findVirtual(Thread.class, "isVirtual", MethodType.methodType(boolean.class)); + } catch (Throwable ignored) { + } + } + return null; + } + + private static MethodHandle findNewVirtualThreadPerTaskExecutorMethodHandle() { + if (JavaVirtualMachine.isJavaVersionAtLeast(21)) { + try { + return MethodHandles.lookup() + .findStatic( + Executors.class, + "newVirtualThreadPerTaskExecutor", + MethodType.methodType(ExecutorService.class)); + } catch (Throwable ignored) { + } + } + return null; + } +} diff --git a/components/environment/src/main/java/datadog/environment/ThreadUtils.java b/components/environment/src/main/java/datadog/environment/ThreadUtils.java deleted file mode 100644 index e07ccb44dce..00000000000 --- a/components/environment/src/main/java/datadog/environment/ThreadUtils.java +++ /dev/null @@ -1,85 +0,0 @@ -package datadog.environment; - -import java.lang.invoke.MethodHandle; -import java.lang.invoke.MethodHandles; -import java.lang.invoke.MethodType; - -/** - * Helper class for working with Threads - * - *

Uses feature detection and provides static helpers to work with different versions of Java - * - *

This class is designed to use MethodHandles that constant propagate to minimize the overhead - */ -public final class ThreadUtils { - static final MethodHandle H_IS_VIRTUAL = lookupIsVirtual(); - static final MethodHandle H_ID = lookupId(); - - private ThreadUtils() {} - - /** Provides the best id available for the Thread Uses threadId on 19+; getId on older JVMs */ - public static final long threadId(Thread thread) { - try { - return (long) H_ID.invoke(thread); - } catch (Throwable t) { - return 0L; - } - } - - /** Indicates whether virtual threads are supported on this JVM */ - public static final boolean supportsVirtualThreads() { - return (H_IS_VIRTUAL != null); - } - - /** Indicates if the current thread is a virtual thread */ - public static final boolean isCurrentThreadVirtual() { - // H_IS_VIRTUAL will constant propagate -- then dead code eliminate -- and inline as needed - try { - return (H_IS_VIRTUAL != null) && (boolean) H_IS_VIRTUAL.invoke(Thread.currentThread()); - } catch (Throwable t) { - return false; - } - } - - /** Indicates if the provided thread is a virtual thread */ - public static final boolean isVirtual(Thread thread) { - // H_IS_VIRTUAL will constant propagate -- then dead code eliminate -- and inline as needed - try { - return (H_IS_VIRTUAL != null) && (boolean) H_IS_VIRTUAL.invoke(thread); - } catch (Throwable t) { - return false; - } - } - - private static final MethodHandle lookupIsVirtual() { - try { - return MethodHandles.lookup() - .findVirtual(Thread.class, "isVirtual", MethodType.methodType(boolean.class)); - } catch (NoSuchMethodException | IllegalAccessException e) { - return null; - } - } - - private static final MethodHandle lookupId() { - MethodHandle threadIdHandle = lookupThreadId(); - return threadIdHandle != null ? threadIdHandle : lookupGetId(); - } - - private static final MethodHandle lookupThreadId() { - try { - return MethodHandles.lookup() - .findVirtual(Thread.class, "threadId", MethodType.methodType(long.class)); - } catch (NoSuchMethodException | IllegalAccessException e) { - return null; - } - } - - private static final MethodHandle lookupGetId() { - try { - return MethodHandles.lookup() - .findVirtual(Thread.class, "getId", MethodType.methodType(long.class)); - } catch (NoSuchMethodException | IllegalAccessException e) { - return null; - } - } -} diff --git a/components/environment/src/test/java/datadog/environment/EnvironmentVariablesTest.java b/components/environment/src/test/java/datadog/environment/EnvironmentVariablesTest.java index 3a9c2195f0d..98b0a0adf99 100644 --- a/components/environment/src/test/java/datadog/environment/EnvironmentVariablesTest.java +++ b/components/environment/src/test/java/datadog/environment/EnvironmentVariablesTest.java @@ -11,7 +11,7 @@ import org.junit.jupiter.api.Test; class EnvironmentVariablesTest { - private static final String EXISTING_ENV_VAR = "JAVA_8_HOME"; + private static final String EXISTING_ENV_VAR = "PATH"; private static final String MISSING_ENV_VAR = "UNDEFINED_ENV_VAR"; private static final String DEFAULT_VALUE = "DEFAULT"; diff --git a/components/environment/src/test/java/datadog/environment/JavaVirtualMachineTest.java b/components/environment/src/test/java/datadog/environment/JavaVirtualMachineTest.java index 8a8d1104e0d..7d540045205 100644 --- a/components/environment/src/test/java/datadog/environment/JavaVirtualMachineTest.java +++ b/components/environment/src/test/java/datadog/environment/JavaVirtualMachineTest.java @@ -113,6 +113,7 @@ void onlyOnGraalVm() { @EnabledOnJre(JAVA_8) void onlyOnIbm8() { assertFalse(JavaVirtualMachine.isGraalVM()); + assertTrue(JavaVirtualMachine.isIbm()); assertTrue(JavaVirtualMachine.isIbm8()); assertTrue(JavaVirtualMachine.isJ9()); assertFalse(JavaVirtualMachine.isOracleJDK8()); diff --git a/components/environment/src/test/java/datadog/environment/ThreadSupportTest.java b/components/environment/src/test/java/datadog/environment/ThreadSupportTest.java new file mode 100644 index 00000000000..be2032c5470 --- /dev/null +++ b/components/environment/src/test/java/datadog/environment/ThreadSupportTest.java @@ -0,0 +1,82 @@ +package datadog.environment; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.fail; +import static org.junit.jupiter.api.condition.JRE.JAVA_21; + +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.atomic.AtomicLong; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledOnJre; + +class ThreadSupportTest { + private static ExecutorService singleThreadExecutor; + private static ExecutorService newVirtualThreadPerTaskExecutor; + + @BeforeAll + static void beforeAll() { + singleThreadExecutor = Executors.newSingleThreadExecutor(); + newVirtualThreadPerTaskExecutor = ThreadSupport.newVirtualThreadPerTaskExecutor().orElse(null); + } + + @Test + public void testThreadId() throws InterruptedException { + AtomicLong threadId = new AtomicLong(); + Thread thread = new Thread(() -> threadId.set(ThreadSupport.threadId()), "foo"); + thread.start(); + try { + // always works on Thread's where getId isn't overridden by child class + assertEquals(thread.getId(), ThreadSupport.threadId(thread)); + } finally { + thread.join(); + } + assertEquals(thread.getId(), threadId.get()); + } + + @Test + void testSupportsVirtualThreads() { + assertEquals( + JavaVirtualMachine.isJavaVersionAtLeast(21), + ThreadSupport.supportsVirtualThreads(), + "expected virtual threads support status"); + } + + @Test + void testPlatformThread() { + assertVirtualThread(singleThreadExecutor, false); + } + + @Test + @EnabledOnJre(JAVA_21) + void testVirtualThread() { + assertVirtualThread(newVirtualThreadPerTaskExecutor, true); + } + + static void assertVirtualThread(ExecutorService executorService, boolean expected) { + Future futureCurrent = executorService.submit(() -> ThreadSupport.isVirtual()); + Future futureGiven = + executorService.submit( + () -> { + Thread thread = Thread.currentThread(); + return ThreadSupport.isVirtual(thread); + }); + try { + assertEquals(expected, futureCurrent.get(), "invalid current thread virtual status"); + assertEquals(expected, futureGiven.get(), "invalid given thread virtual status"); + } catch (Throwable e) { + fail("Can't get thread virtual status", e); + } + } + + @AfterAll + static void afterAll() { + singleThreadExecutor.shutdown(); + if (newVirtualThreadPerTaskExecutor != null) { + newVirtualThreadPerTaskExecutor.shutdown(); + } + } +} diff --git a/components/environment/src/test/java/datadog/environment/ThreadUtilsTest.java b/components/environment/src/test/java/datadog/environment/ThreadUtilsTest.java deleted file mode 100644 index 008897f6199..00000000000 --- a/components/environment/src/test/java/datadog/environment/ThreadUtilsTest.java +++ /dev/null @@ -1,107 +0,0 @@ -package datadog.environment; - -import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertFalse; -import static org.junit.jupiter.api.Assertions.assertTrue; -import static org.junit.jupiter.api.Assumptions.assumeTrue; - -import java.lang.invoke.MethodHandle; -import java.lang.invoke.MethodHandles; -import java.lang.invoke.MethodType; -import java.util.concurrent.ExecutionException; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; -import java.util.concurrent.atomic.AtomicBoolean; -import org.junit.jupiter.api.Test; - -public class ThreadUtilsTest { - @Test - public void threadId() throws InterruptedException { - Thread thread = new Thread("foo"); - thread.start(); - try { - // always works on Thread's where getId isn't overridden by child class - assertEquals(thread.getId(), ThreadUtils.threadId(thread)); - } finally { - thread.join(); - } - } - - @Test - public void supportsVirtualThreads() { - assertEquals( - JavaVersion.getRuntimeVersion().isAtLeast(21), ThreadUtils.supportsVirtualThreads()); - } - - @Test - public void isVirtualThread_false() throws InterruptedException { - Thread thread = new Thread("foo"); - thread.start(); - try { - assertFalse(ThreadUtils.isVirtual(thread)); - } finally { - thread.join(); - } - } - - @Test - public void isCurrentThreadVirtual_false() throws InterruptedException, ExecutionException { - ExecutorService executor = Executors.newSingleThreadExecutor(); - try { - assertFalse(executor.submit(() -> ThreadUtils.isCurrentThreadVirtual()).get()); - } finally { - executor.shutdown(); - } - } - - @Test - public void isVirtualThread_true() throws InterruptedException { - assumeTrue(ThreadUtils.supportsVirtualThreads()); - - Thread vThread = startVirtualThread(() -> {}); - try { - assertTrue(ThreadUtils.isVirtual(vThread)); - } finally { - vThread.join(); - } - } - - @Test - public void isCurrentThreadVirtual_true() throws InterruptedException { - assumeTrue(ThreadUtils.supportsVirtualThreads()); - - AtomicBoolean result = new AtomicBoolean(); - - Thread vThread = - startVirtualThread( - () -> { - result.set(ThreadUtils.isCurrentThreadVirtual()); - }); - - vThread.join(); - assertTrue(result.get()); - } - - /* - * Should only be called on JVMs that support virtual threads - */ - static final Thread startVirtualThread(Runnable runnable) { - MethodHandle h_startVThread; - try { - h_startVThread = - MethodHandles.lookup() - .findStatic( - Thread.class, - "startVirtualThread", - MethodType.methodType(Thread.class, Runnable.class)); - } catch (NoSuchMethodException | IllegalAccessException e) { - throw new IllegalStateException(e); - } - - try { - return (Thread) h_startVThread.invoke(runnable); - } catch (Throwable e) { - throw new IllegalStateException(e); - } - } -} diff --git a/components/http/http-api/build.gradle.kts b/components/http/http-api/build.gradle.kts new file mode 100644 index 00000000000..314579b64db --- /dev/null +++ b/components/http/http-api/build.gradle.kts @@ -0,0 +1,35 @@ +plugins { + `java-library` + `java-test-fixtures` +} + +apply(from = "$rootDir/gradle/java.gradle") + +description = "HTTP Client API" + +val minimumBranchCoverage by extra(0) // extra(0.7) -- need a library implementation +val minimumInstructionCoverage by extra(0) // extra(0.7) -- need a library implementation + +// Exclude interfaces for test coverage +val excludedClassesCoverage by extra( + listOf( + "datadog.http.client.HttpClient", + "datadog.http.client.HttpClient.Builder", + "datadog.http.client.HttpRequest", + "datadog.http.client.HttpRequest.Builder", + "datadog.http.client.HttpRequestBody", + "datadog.http.client.HttpRequestBody.MultipartBuilder", + "datadog.http.client.HttpRequestListener", + "datadog.http.client.HttpResponse", + "datadog.http.client.HttpUrl", + "datadog.http.client.HttpUrl.Builder", + ) +) + +dependencies { + // Add API implementations to test providers + // testRuntimeOnly(project(":components:http:http-lib-jdk")) + // testRuntimeOnly(project(":components:http:http-lib-okhttp")) + // Add MockServer for test fixtures + testFixturesImplementation("org.mock-server:mockserver-junit-jupiter-no-dependencies:5.14.0") +} diff --git a/components/http/http-api/gradle.lockfile b/components/http/http-api/gradle.lockfile new file mode 100644 index 00000000000..3a2aadfa4d8 --- /dev/null +++ b/components/http/http-api/gradle.lockfile @@ -0,0 +1,75 @@ +# This is a Gradle generated file for dependency locking. +# Manual edits can break the build and are not advised. +# This file is expected to be part of source control. +ch.qos.logback:logback-classic:1.2.13=testCompileClasspath,testRuntimeClasspath +ch.qos.logback:logback-core:1.2.13=testCompileClasspath,testRuntimeClasspath +com.github.javaparser:javaparser-core:3.25.6=codenarc +com.github.spotbugs:spotbugs-annotations:4.9.8=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath +com.github.spotbugs:spotbugs:4.9.8=spotbugs +com.github.stephenc.jcip:jcip-annotations:1.0-1=spotbugs +com.google.code.findbugs:jsr305:3.0.2=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath +com.google.code.gson:gson:2.13.2=spotbugs +com.google.errorprone:error_prone_annotations:2.41.0=spotbugs +com.thoughtworks.qdox:qdox:1.12.1=codenarc +commons-io:commons-io:2.20.0=spotbugs +de.thetaphi:forbiddenapis:3.10=compileClasspath +io.leangen.geantyref:geantyref:1.3.16=testRuntimeClasspath +jaxen:jaxen:2.0.0=spotbugs +net.bytebuddy:byte-buddy-agent:1.12.8=testRuntimeClasspath +net.bytebuddy:byte-buddy:1.12.8=testRuntimeClasspath +net.sf.saxon:Saxon-HE:12.9=spotbugs +org.apache.ant:ant-antlr:1.10.14=codenarc +org.apache.ant:ant-junit:1.10.14=codenarc +org.apache.bcel:bcel:6.11.0=spotbugs +org.apache.commons:commons-lang3:3.19.0=spotbugs +org.apache.commons:commons-text:1.14.0=spotbugs +org.apache.logging.log4j:log4j-api:2.25.2=spotbugs +org.apache.logging.log4j:log4j-core:2.25.2=spotbugs +org.apiguardian:apiguardian-api:1.1.2=testCompileClasspath +org.codehaus.groovy:groovy-ant:3.0.23=codenarc +org.codehaus.groovy:groovy-docgenerator:3.0.23=codenarc +org.codehaus.groovy:groovy-groovydoc:3.0.23=codenarc +org.codehaus.groovy:groovy-json:3.0.23=codenarc +org.codehaus.groovy:groovy-json:3.0.25=testCompileClasspath,testRuntimeClasspath +org.codehaus.groovy:groovy-templates:3.0.23=codenarc +org.codehaus.groovy:groovy-xml:3.0.23=codenarc +org.codehaus.groovy:groovy:3.0.23=codenarc +org.codehaus.groovy:groovy:3.0.25=testCompileClasspath,testRuntimeClasspath +org.codenarc:CodeNarc:3.7.0=codenarc +org.dom4j:dom4j:2.2.0=spotbugs +org.gmetrics:GMetrics:2.1.0=codenarc +org.hamcrest:hamcrest:3.0=testCompileClasspath,testRuntimeClasspath +org.jacoco:org.jacoco.agent:0.8.14=jacocoAgent,jacocoAnt +org.jacoco:org.jacoco.ant:0.8.14=jacocoAnt +org.jacoco:org.jacoco.core:0.8.14=jacocoAnt +org.jacoco:org.jacoco.report:0.8.14=jacocoAnt +org.junit.jupiter:junit-jupiter-api:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-engine:5.14.1=testRuntimeClasspath +org.junit.jupiter:junit-jupiter-params:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-commons:1.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-engine:1.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-launcher:1.14.1=testRuntimeClasspath +org.junit:junit-bom:5.14.0=spotbugs +org.junit:junit-bom:5.14.1=testCompileClasspath,testRuntimeClasspath +org.mock-server:mockserver-junit-jupiter-no-dependencies:5.14.0=testFixturesCompileClasspath,testFixturesRuntimeClasspath,testRuntimeClasspath +org.mockito:mockito-core:4.4.0=testRuntimeClasspath +org.objenesis:objenesis:3.3=testCompileClasspath,testRuntimeClasspath +org.opentest4j:opentest4j:1.3.0=testCompileClasspath,testRuntimeClasspath +org.ow2.asm:asm-analysis:9.9=spotbugs +org.ow2.asm:asm-commons:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-tree:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-util:9.9=spotbugs +org.ow2.asm:asm:9.9=jacocoAnt,spotbugs +org.slf4j:jcl-over-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath +org.slf4j:jul-to-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath +org.slf4j:log4j-over-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath +org.slf4j:slf4j-api:1.7.32=testCompileClasspath,testRuntimeClasspath +org.slf4j:slf4j-api:2.0.17=spotbugs,spotbugsSlf4j +org.slf4j:slf4j-simple:2.0.17=spotbugsSlf4j +org.spockframework:spock-bom:2.4-groovy-3.0=testCompileClasspath,testRuntimeClasspath +org.spockframework:spock-core:2.4-groovy-3.0=testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-junit:1.2.1=testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-parser:1.2.0=testCompileClasspath,testRuntimeClasspath +org.xmlresolver:xmlresolver:5.3.3=spotbugs +empty=annotationProcessor,runtimeClasspath,spotbugsPlugins,testAnnotationProcessor,testFixturesAnnotationProcessor diff --git a/components/http/http-api/src/main/java/datadog/http/client/HttpClient.java b/components/http/http-api/src/main/java/datadog/http/client/HttpClient.java new file mode 100644 index 00000000000..fb6a637cfdc --- /dev/null +++ b/components/http/http-api/src/main/java/datadog/http/client/HttpClient.java @@ -0,0 +1,113 @@ +package datadog.http.client; + +import java.io.File; +import java.io.IOException; +import java.net.Proxy; +import java.time.Duration; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.Executor; +import javax.annotation.Nullable; + +/** + * This interface is an abstraction for HTTP clients, providing request execution capabilities. This + * abstraction is implementation-agnostic and can be backed by third party libraries of the JDK + * itself. + * + *

HttpClient instances should be reused across requests for connection pooling. + */ +public interface HttpClient { + /** + * Executes an HTTP request synchronously and returns the response. The caller is responsible for + * closing the response. + * + * @param request the request to execute + * @return the HTTP response + * @throws IOException if an I/O error occurs + */ + HttpResponse execute(HttpRequest request) throws IOException; + + /** + * Executes an HTTP request asynchronously and returns a {@link CompletableFuture}. The caller is + * responsible for closing the response. + * + * @param request the request to execute + * @return a CompletableFuture that completes with the HTTP response + */ + CompletableFuture executeAsync(HttpRequest request); + + /** + * Creates a new {@link Builder} for constructing HTTP clients. + * + * @return a new http client builder + */ + static Builder newBuilder() { + return HttpProviders.get().newClientBuilder(); + } + + /** Builder for constructing {@link HttpClient} instances. */ + interface Builder { + /** + * Sets the client timeouts, including the connection. + * + * @param timeout the timeout duration + * @return this builder + */ + Builder connectTimeout(Duration timeout); + + /** + * Sets the proxy configuration. + * + * @param proxy the proxy to use + * @return this builder + */ + Builder proxy(Proxy proxy); + + /** + * Sets proxy authentication credentials. + * + * @param username the proxy username + * @param password the proxy password, or {@code null} to use an empty password + * @return this builder + */ + Builder proxyAuthenticator(String username, @Nullable String password); + + /** + * Configures the client to use a Unix domain socket. + * + * @param socketFile the Unix domain socket file + * @return this builder + */ + Builder unixDomainSocket(File socketFile); + + /** + * Configures the client to use a named pipe (Windows). + * + * @param pipeName the named pipe name + * @return this builder + */ + Builder namedPipe(String pipeName); + + /** + * Forces clear text (HTTP) connections, disabling TLS. + * + * @param clearText {@code true} to force HTTP, {@code false} to allow HTTPS + * @return this builder + */ + Builder clearText(boolean clearText); + + /** + * Sets a custom executor for executing async requests. + * + * @param executor the executor to use for async requests + * @return this builder + */ + Builder executor(Executor executor); + + /** + * Builds the {@link HttpClient} with the configured settings. + * + * @return the constructed HttpClient + */ + HttpClient build(); + } +} diff --git a/components/http/http-api/src/main/java/datadog/http/client/HttpProvider.java b/components/http/http-api/src/main/java/datadog/http/client/HttpProvider.java new file mode 100644 index 00000000000..4754ad5139f --- /dev/null +++ b/components/http/http-api/src/main/java/datadog/http/client/HttpProvider.java @@ -0,0 +1,28 @@ +package datadog.http.client; + +import java.net.URI; +import java.nio.ByteBuffer; +import java.util.List; + +/** Factory class providing the various HTTP client class implementations. */ +public abstract class HttpProvider { + public abstract HttpClient.Builder newClientBuilder(); + + public abstract HttpRequest.Builder newRequestBuilder(); + + public abstract HttpUrl.Builder newUrlBuilder(); + + public abstract HttpUrl httpUrlParse(String url); + + public abstract HttpUrl httpUrlFrom(URI uri); + + public abstract HttpRequestBody requestBodyOfString(String content); + + public abstract HttpRequestBody requestBodyOfBytes(byte[] bytes); + + public abstract HttpRequestBody requestBodyOfByteBuffers(List buffers); + + public abstract HttpRequestBody requestBodyGzip(HttpRequestBody body); + + public abstract HttpRequestBody.MultipartBuilder requestBodyMultipart(); +} diff --git a/components/http/http-api/src/main/java/datadog/http/client/HttpProviders.java b/components/http/http-api/src/main/java/datadog/http/client/HttpProviders.java new file mode 100644 index 00000000000..efa217c0dfc --- /dev/null +++ b/components/http/http-api/src/main/java/datadog/http/client/HttpProviders.java @@ -0,0 +1,74 @@ +package datadog.http.client; + +import de.thetaphi.forbiddenapis.SuppressForbidden; + +/** + * Static factory class for obtaining HTTP provider implementations. + * + *

This class provides a singleton access point to an {@link HttpProvider} instance, which serves + * as a factory for creating HTTP client components such as clients, requests, URLs, and request + * bodies. + * + *

The provider selection follows a hierarchical fallback strategy: - First attempts to load the + * JDK-based HTTP provider implementation - Falls back to the OkHttp-based provider if the JDK + * version is not available or incompatible - Can be forced into compatibility mode to skip the JDK + * provider and use OkHttp directly + * + *

The selected provider is cached after the first access for performance. This class is + * thread-safe and all methods can be safely called from multiple threads. + */ +public final class HttpProviders { + private static final String JDK_HTTP_PROVIDER_CLASS_NAME = + "datadog.http.client.jdk.JdkHttpProvider"; + private static final String OKHTTP_PROVIDER_CLASS_NAME = + "datadog.http.client.okhttp.OkHttpProvider"; + private static volatile boolean compatibilityMode = false; + private static HttpProvider provider; + + private HttpProviders() {} + + public static void forceCompatClient() { + // Skip if already in compat mode + if (compatibilityMode) { + return; + } + compatibilityMode = true; + provider = null; + } + + public static HttpProvider get() { + if (provider == null) { + provider = findProvider(); + } + return provider; + } + + @SuppressForbidden // Class#forName(String) used to dynamically load the http API implementation + private static HttpProvider findProvider() { + Class clazz = null; + // Load the default client class + if (!compatibilityMode) { + try { + clazz = Class.forName(JDK_HTTP_PROVIDER_CLASS_NAME); + } catch (ClassNotFoundException | UnsupportedClassVersionError ignored) { + compatibilityMode = true; + } + } + // If not loaded, load the compat client class + if (clazz == null) { + try { + clazz = Class.forName(OKHTTP_PROVIDER_CLASS_NAME); + } catch (ClassNotFoundException ignored) { + } + } + // If no class loaded, raise the illegal state + if (clazz == null) { + throw new IllegalStateException("No http client implementation found"); + } + try { + return (HttpProvider) clazz.getDeclaredConstructor().newInstance(); + } catch (ReflectiveOperationException e) { + throw new IllegalStateException("No http client implementation found", e); + } + } +} diff --git a/components/http/http-api/src/main/java/datadog/http/client/HttpRequest.java b/components/http/http-api/src/main/java/datadog/http/client/HttpRequest.java new file mode 100644 index 00000000000..b4649f4da27 --- /dev/null +++ b/components/http/http-api/src/main/java/datadog/http/client/HttpRequest.java @@ -0,0 +1,140 @@ +package datadog.http.client; + +import java.util.List; +import javax.annotation.Nullable; + +/** + * This interface is an abstraction for HTTP requests, providing access to URL, method, headers, and + * body. + */ +public interface HttpRequest { + /* Common headers names and values widely used in HTTP requests */ + String CONTENT_TYPE = "Content-Type"; + String APPLICATION_JSON = "application/json; charset=utf-8"; + + /** + * Returns the request URL. + * + * @return the HttpUrl + */ + HttpUrl url(); + + /** + * Returns the HTTP method ({@code GET}, {@code POST}, {@code PUT}, etc.). + * + * @return the method name + */ + String method(); + + /** + * Returns the first header value for the given name, or {@code null} if not present. + * + * @param name the header name + * @return the first header value, or {@code null} if not present + */ + @Nullable + String header(String name); + + /** + * Returns all header values for the given name. + * + * @param name the header name + * @return list of header values, an empty list if not present + */ + List headers(String name); + + /** + * Returns the request body, or {@code null} if this request has no body (e.g., GET requests). + * + * @return the request body, or {@code null} if this request has no body + */ + @Nullable + HttpRequestBody body(); + + /** + * Creates a new {@link Builder} for constructing HTTP requests. + * + * @return a new builder + */ + static Builder newBuilder() { + return HttpProviders.get().newRequestBuilder(); + } + + /** Builder for constructing {@link HttpRequest} instances. */ + interface Builder { + /** + * Sets the request URL. + * + * @param url the URL + * @return this builder + */ + Builder url(HttpUrl url); + + /** + * Sets the request URL from a {@link String}. + * + * @param url the URL string + * @return this builder + */ + Builder url(String url); + + /** + * Sets the request method to {@code GET}. This is the default method if other methods are not + * set. + * + * @return this builder + */ + Builder get(); + + /** + * Sets the request method to {@code POST} with the given body. + * + * @param body the request body + * @return this builder + */ + Builder post(HttpRequestBody body); + + /** + * Sets the request method to {@code PUT} with the given body. + * + * @param body the request body + * @return this builder + */ + Builder put(HttpRequestBody body); + + /** + * Sets a header, replacing any existing values for the same name. + * + * @param name the header name + * @param value the header value + * @return this builder + */ + Builder header(String name, String value); + + /** + * Adds a header without removing existing values for the same name. + * + * @param name the header name + * @param value the header value + * @return this builder + */ + Builder addHeader(String name, String value); + + /** + * Sets the request listener. + * + * @param listener the listener to notify of request events or {@code null} to remove any + * existing listener. + * @return this builder + */ + Builder listener(@Nullable HttpRequestListener listener); + + /** + * Builds the HttpRequest. + * + * @return the constructed HttpRequest + * @throws IllegalStateException if required fields (like url) are missing + */ + HttpRequest build(); + } +} diff --git a/components/http/http-api/src/main/java/datadog/http/client/HttpRequestBody.java b/components/http/http-api/src/main/java/datadog/http/client/HttpRequestBody.java new file mode 100644 index 00000000000..b46fc223771 --- /dev/null +++ b/components/http/http-api/src/main/java/datadog/http/client/HttpRequestBody.java @@ -0,0 +1,133 @@ +package datadog.http.client; + +import java.io.IOException; +import java.io.OutputStream; +import java.nio.ByteBuffer; +import java.util.List; +import java.util.Map; + +/** + * This interface is an abstraction for HTTP request bodies, providing content writing capabilities. + * It also offers static factory methods to build common request body types, including gzip + * compression and multipart/form-data. + */ +public interface HttpRequestBody { + /** + * Returns the content length in bytes, or {@code -1} if unknown (e.g., for gzipped content). + * + * @return the content length, or {@code -1} if unknown + */ + long contentLength(); + + /** + * Writes the body content to the given output stream. + * + * @param out the output stream to write to + * @throws IOException if an I/O error occurs + */ + void writeTo(OutputStream out) throws IOException; + + /** + * Creates a request body from a String using UTF-8 encoding. Content-Type should be set via + * request headers. + * + * @param content the string content + * @return a new {@link HttpRequestBody} + */ + static HttpRequestBody of(String content) { + return HttpProviders.get().requestBodyOfString(content); + } + + /** + * Creates a request body from raw bytes. Content-Type should be set via request headers. + * + * @param bytes the byte array content + * @return a new {@link HttpRequestBody} + */ + static HttpRequestBody of(byte[] bytes) { + return HttpProviders.get().requestBodyOfBytes(bytes); + } + + /** + * Creates a request body from a list of {@link ByteBuffer}s. Content-Type should be set via + * request headers. + * + * @param buffers the list of byte buffers content + * @return a new {@link HttpRequestBody} + */ + static HttpRequestBody of(List buffers) { + return HttpProviders.get().requestBodyOfByteBuffers(buffers); + } + + /** + * Wraps a request body with gzip compression. The body is compressed eagerly and the content + * length reflects the compressed size. Content-Encoding header should be set to "gzip" separately + * via request headers. + * + * @param body the body to compress + * @return a new gzip-compressed {@link HttpRequestBody} + */ + static HttpRequestBody gzip(HttpRequestBody body) { + return HttpProviders.get().requestBodyGzip(body); + } + + /** + * Creates a builder for multipart/form-data request bodies. + * + * @return a new {@link MultipartBuilder} + */ + static MultipartBuilder multipart() { + return HttpProviders.get().requestBodyMultipart(); + } + + /** + * Builder for creating multipart/form-data request bodies. Implements RFC 7578 + * multipart/form-data format. + */ + interface MultipartBuilder { + /** + * Adds a form data part with a text value. + * + * @param name the field name + * @param value the field value + * @return this builder + */ + MultipartBuilder addFormDataPart(String name, String value); + + /** + * Adds a form data part with a file attachment. + * + * @param name the field name + * @param filename the filename + * @param body the file content + * @return this builder + */ + MultipartBuilder addFormDataPart(String name, String filename, HttpRequestBody body); + + /** + * Adds a part with custom headers (advanced usage). Use this when you need full control over + * part headers. + * + * @param headers map of header name to value (e.g., {@code Content-Disposition}, {@code + * Content-Type}) + * @param body the part content + * @return this builder + */ + MultipartBuilder addPart(Map headers, HttpRequestBody body); + + /** + * Returns the {@code Content-Type} header value for this multipart body. Includes the boundary + * parameter required for parsing. Can be called before or after build(). + * + * @return the content type string (e.g., "multipart/form-data; boundary=...") + */ + String contentType(); + + /** + * Builds the multipart request body. + * + * @return the constructed {@link HttpRequestBody} + */ + HttpRequestBody build(); + } +} diff --git a/components/http/http-api/src/main/java/datadog/http/client/HttpRequestListener.java b/components/http/http-api/src/main/java/datadog/http/client/HttpRequestListener.java new file mode 100644 index 00000000000..4d86b99d890 --- /dev/null +++ b/components/http/http-api/src/main/java/datadog/http/client/HttpRequestListener.java @@ -0,0 +1,33 @@ +package datadog.http.client; + +import java.io.IOException; +import javax.annotation.Nullable; + +/** + * This interface represents a listener for HTTP request lifecycle events. Implementations can track + * request timing, log requests, or handle errors. + */ +public interface HttpRequestListener { + /** + * Called when a request is about to be sent. + * + * @param request the request being sent + */ + void onRequestStart(HttpRequest request); + + /** + * Called when a response is received successfully. + * + * @param request the request that was sent + * @param response the response received, or {@code null} if response body hasn't been read yet + */ + void onRequestEnd(HttpRequest request, @Nullable HttpResponse response); + + /** + * Called when a request fails with an exception. + * + * @param request the request that failed + * @param exception the exception that occurred + */ + void onRequestFailure(HttpRequest request, IOException exception); +} diff --git a/components/http/http-api/src/main/java/datadog/http/client/HttpResponse.java b/components/http/http-api/src/main/java/datadog/http/client/HttpResponse.java new file mode 100644 index 00000000000..09751b6244e --- /dev/null +++ b/components/http/http-api/src/main/java/datadog/http/client/HttpResponse.java @@ -0,0 +1,71 @@ +package datadog.http.client; + +import java.io.IOException; +import java.io.InputStream; +import java.util.List; +import java.util.Set; +import javax.annotation.Nullable; + +/** + * This interface is an abstraction for HTTP responses, providing access to status code, headers, + * and body. + * + *

HttpResponse instances must be closed after use to release resources. + */ +public interface HttpResponse { + /** + * Returns the HTTP status code. + * + * @return the status code (e.g., 200, 404, 500) + */ + int code(); + + /** + * Check whether the response code is in [200..300), indicating the request was successful. + * + * @return {@code true} if successful, {@code false} otherwise + */ + boolean isSuccessful(); + + /** + * Returns the first header value for the given name, or {@code null} if not present. Header names + * are case-insensitive. + * + * @param name the header name + * @return the first header value, or {@code null} if not present + */ + @Nullable + String header(String name); + + /** + * Returns all header values for the given name. Header names are case-insensitive. + * + * @param name the header name + * @return list of header values, an empty list if not present + */ + List headers(String name); + + /** + * Returns all header names in this response. Header names are returned in their canonical form. + * + * @return set of header names, empty if no headers present + */ + Set headerNames(); + + /** + * Returns the response body as an {@link InputStream}. The caller is responsible for closing the + * stream. + * + * @return the response body stream + */ + InputStream body(); + + /** + * Returns the response body as a {@link String} using {@code Content-Type} charset or UTF-8 if + * absent. + * + * @return the response body as a {@link String} + * @throws IOException if an I/O error occurs + */ + String bodyAsString() throws IOException; +} diff --git a/components/http/http-api/src/main/java/datadog/http/client/HttpUrl.java b/components/http/http-api/src/main/java/datadog/http/client/HttpUrl.java new file mode 100644 index 00000000000..4bdeb2ada1f --- /dev/null +++ b/components/http/http-api/src/main/java/datadog/http/client/HttpUrl.java @@ -0,0 +1,141 @@ +package datadog.http.client; + +import static java.util.Objects.requireNonNull; + +import java.net.URI; +import javax.annotation.Nullable; + +/** + * This interface is an abstraction for HTTP URLs, providing URL parsing, building, and manipulation + * capabilities. It also offers static factory methods to build URLs from JDK URIs using {@link + * #from(URI)}, or parse from strings using {@link #parse(String)}. + */ +public interface HttpUrl { + /** + * Returns the complete URL as a string. + * + * @return the URL string + */ + String url(); + + /** + * Returns the scheme (protocol) of this URL. + * + * @return the scheme (e.g., "http", "https") + */ + String scheme(); + + /** + * Returns the host of this URL. + * + * @return the host name or IP address + */ + String host(); + + /** + * Returns the port of this URL. Returns the default port for the scheme if not explicitly set (80 + * for http, 443 for https). + * + * @return the port number + */ + int port(); + + /** + * Resolves a relative URL against this URL. + * + * @param path the relative path to resolve + * @return a new {@link HttpUrl} with the resolved path + */ + HttpUrl resolve(String path); + + /** + * Returns a {@link Builder} to modify this URL. + * + * @return a new {@link Builder} based on this URL + */ + Builder newBuilder(); + + /** + * Parses a URL string into an {@link HttpUrl}. + * + * @param url the URL string to parse + * @return the parsed {@link HttpUrl} + * @throws IllegalArgumentException if the URL is malformed + */ + static HttpUrl parse(String url) throws IllegalArgumentException { + requireNonNull(url, "url"); + return HttpProviders.get().httpUrlParse(url); + } + + /** + * Creates an HttpUrl from an {@link URI}. + * + * @param uri the {@link URI} to get an {@link HttpUrl} from + * @return the {@link HttpUrl} related to the URI + */ + static HttpUrl from(URI uri) { + requireNonNull(uri, "uri"); + return HttpProviders.get().httpUrlFrom(uri); + } + + /** + * Creates a new {@link Builder}der for constructing URLs. + * + * @return a new {@link Builder} + */ + static Builder builder() { + return HttpProviders.get().newUrlBuilder(); + } + + /** Builder for constructing {@link HttpUrl} instances. */ + interface Builder { + /** + * Sets the scheme (protocol) for the URL. + * + * @param scheme the scheme (e.g., "http", "https") + * @return this builder + */ + Builder scheme(String scheme); + + /** + * Sets the host for the URL. + * + * @param host the host name or IP address + * @return this builder + */ + Builder host(String host); + + /** + * Sets the port for the URL. + * + * @param port the port number + * @return this builder + */ + Builder port(int port); + + /** + * Adds a path segment to the URL. + * + * @param segment the path segment to add + * @return this builder + */ + Builder addPathSegment(String segment); + + /** + * Adds a query parameter to the URL. + * + * @param name the parameter name + * @param value the parameter value + * @return this builder + */ + Builder addQueryParameter(String name, @Nullable String value); + + /** + * Builds the HttpUrl. + * + * @return the constructed HttpUrl + * @throws IllegalStateException if required fields are missing + */ + HttpUrl build(); + } +} diff --git a/components/http/http-api/src/main/java/datadog/http/client/package-info.java b/components/http/http-api/src/main/java/datadog/http/client/package-info.java new file mode 100644 index 00000000000..c56aab93967 --- /dev/null +++ b/components/http/http-api/src/main/java/datadog/http/client/package-info.java @@ -0,0 +1,4 @@ +@ParametersAreNonnullByDefault +package datadog.http.client; + +import javax.annotation.ParametersAreNonnullByDefault; diff --git a/components/http/http-api/src/test/java/datadog/http/client/HttpProvidersTest.java b/components/http/http-api/src/test/java/datadog/http/client/HttpProvidersTest.java new file mode 100644 index 00000000000..fcf12314b48 --- /dev/null +++ b/components/http/http-api/src/test/java/datadog/http/client/HttpProvidersTest.java @@ -0,0 +1,72 @@ +package datadog.http.client; + +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.condition.JRE.JAVA_10; +import static org.junit.jupiter.api.condition.JRE.JAVA_11; + +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Disabled; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledForJreRange; + +abstract class HttpProvidersTest { + abstract String getImplementationPackage(); + + @Test + void testProviderLookup() { + // Check runtime type + HttpProvider httpProvider = HttpProviders.get(); + assertType(httpProvider); + // Downgrade to compatibility mode + HttpProviders.forceCompatClient(); + httpProvider = HttpProviders.get(); + assertCompatType(httpProvider); + // Ensure downgrade is idempotent + HttpProviders.forceCompatClient(); + httpProvider = HttpProviders.get(); + assertCompatType(httpProvider); + } + + private void assertType(Object type) { + assertNotNull(type); + assertTrue(type.getClass().getName().startsWith(getImplementationPackage())); + } + + private void assertCompatType(Object type) { + assertNotNull(type); + assertTrue(type.getClass().getName().startsWith("datadog.http.client.okhttp")); + } + + @Disabled + @EnabledForJreRange(max = JAVA_10) + static class OkHttpProvidersForkedTest extends HttpProvidersTest { + @Override + String getImplementationPackage() { + return "datadog.http.client.okhttp"; + } + } + + @Disabled + @EnabledForJreRange(min = JAVA_11) + static class JdkHttpProvidersForkedTest extends HttpProvidersTest { + @Override + String getImplementationPackage() { + return "datadog.http.client.jdk"; + } + } + + @Disabled + @EnabledForJreRange(min = JAVA_11) + static class HttpProvidersCompatForkedTest extends HttpProvidersTest { + @BeforeAll + static void beforeAll() { + HttpProviders.forceCompatClient(); + } + + @Override + String getImplementationPackage() { + return "datadog.http.client.okhttp"; + } + } +} diff --git a/components/http/http-api/src/test/test/java/datadog/http/client/HttpProvidersTest.java b/components/http/http-api/src/test/test/java/datadog/http/client/HttpProvidersTest.java new file mode 100644 index 00000000000..d337299c1d6 --- /dev/null +++ b/components/http/http-api/src/test/test/java/datadog/http/client/HttpProvidersTest.java @@ -0,0 +1,159 @@ +package datadog.http.client; + +import static java.util.Collections.emptyList; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.condition.JRE.JAVA_10; +import static org.junit.jupiter.api.condition.JRE.JAVA_11; +import static org.junit.jupiter.params.provider.Arguments.arguments; + +import java.net.URI; +import java.util.function.Supplier; +import java.util.stream.Stream; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledForJreRange; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; + +abstract class HttpProvidersTest { + private static final String URL_EXAMPLE = "http://localhost"; + private static final String CONTENT_EXAMPLE = "content"; + + abstract String getImplementationPackage(); + + @ParameterizedTest(name = "[{index}] {0} builder") + @MethodSource("builders") + void testNewBuilder(String type, Supplier builderSupplier) { + Object builder = builderSupplier.get(); + assertType(builder); + } + + static Stream builders() { + return Stream.of( + arguments("client", (Supplier) HttpProviders::newClientBuilder), + arguments("request", (Supplier) HttpProviders::newRequestBuilder), + arguments("url", (Supplier) HttpProviders::newUrlBuilder)); + } + + @Test + void testHttpUrlParse() { + HttpUrl url = HttpProviders.httpUrlParse(URL_EXAMPLE); + assertType(url); + } + + @Test + void testHttpUrlParseInvalidUrl() { + // An invalid URL causes the underlying parse() to throw IllegalArgumentException, + // wrapped as InvocationTargetException. HttpProviders unwraps and re-throws it. + assertThrows( + IllegalArgumentException.class, () -> HttpProviders.httpUrlParse("not a valid url")); + } + + @Test + void testHttpUrlFromUri() { + HttpUrl url = HttpProviders.httpUrlFrom(URI.create(URL_EXAMPLE)); + assertType(url); + } + + @Test + void testRequestBodyOfString() { + HttpRequestBody body = HttpProviders.requestBodyOfString(CONTENT_EXAMPLE); + assertType(body); + } + + @Test + void testRequestBodyOfBytes() { + HttpRequestBody body = HttpProviders.requestBodyOfBytes(new byte[0]); + assertType(body); + } + + @Test + void testRequestBodyOfByteBuffers() { + HttpRequestBody body = HttpProviders.requestBodyOfByteBuffers(emptyList()); + assertType(body); + } + + @Test + void testRequestBodyGzip() { + HttpRequestBody body = + HttpProviders.requestBodyGzip(HttpProviders.requestBodyOfString(CONTENT_EXAMPLE)); + assertType(body); + } + + @Test + void testRequestBodyMultipart() { + HttpRequestBody.MultipartBuilder builder = HttpProviders.requestBodyMultipart(); + assertType(builder); + } + + @Test + void testCachedProviders() { + // First calls — populate all lazy-init caches + HttpProviders.newClientBuilder(); + HttpProviders.newRequestBuilder(); + HttpProviders.newUrlBuilder(); + HttpProviders.httpUrlParse(URL_EXAMPLE); + HttpProviders.httpUrlFrom(URI.create(URL_EXAMPLE)); + HttpProviders.requestBodyOfString(CONTENT_EXAMPLE); + HttpProviders.requestBodyOfBytes(new byte[0]); + HttpProviders.requestBodyOfByteBuffers(emptyList()); + HttpProviders.requestBodyGzip(HttpProviders.requestBodyOfString(CONTENT_EXAMPLE)); + HttpProviders.requestBodyMultipart(); + // Second calls — hit the non-null (cached) branch for every lazy field + assertNotNull(HttpProviders.newClientBuilder()); + assertNotNull(HttpProviders.newRequestBuilder()); + assertNotNull(HttpProviders.newUrlBuilder()); + assertNotNull(HttpProviders.httpUrlParse(URL_EXAMPLE)); + assertNotNull(HttpProviders.httpUrlFrom(URI.create(URL_EXAMPLE))); + assertNotNull(HttpProviders.requestBodyOfString(CONTENT_EXAMPLE)); + assertNotNull(HttpProviders.requestBodyOfBytes(new byte[0])); + assertNotNull(HttpProviders.requestBodyOfByteBuffers(emptyList())); + assertNotNull( + HttpProviders.requestBodyGzip(HttpProviders.requestBodyOfString(CONTENT_EXAMPLE))); + assertNotNull(HttpProviders.requestBodyMultipart()); + } + + private void assertType(Object builder) { + assertNotNull(builder); + assertTrue(builder.getClass().getName().startsWith(getImplementationPackage())); + } + + @EnabledForJreRange(max = JAVA_10) + static class OkHttpProvidersForkedTest extends HttpProvidersTest { + @Override + String getImplementationPackage() { + return "datadog.http.client.okhttp"; + } + } + + @EnabledForJreRange(min = JAVA_11) + static class JdkHttpProvidersForkedTest extends HttpProvidersTest { + @Override + String getImplementationPackage() { + return "datadog.http.client.jdk"; + } + } + + @EnabledForJreRange(min = JAVA_11) + static class HttpProvidersCompatForkedTest extends HttpProvidersTest { + @BeforeAll + static void beforeAll() { + HttpProviders.forceCompatClient(); + } + + @Override + String getImplementationPackage() { + return "datadog.http.client.okhttp"; + } + + @Test + void testForceCompatClientIsIdempotent() { + // compatibilityMode is already true — second call must hit early return (no NPE, no reset) + HttpProviders.forceCompatClient(); + assertNotNull(HttpProviders.newClientBuilder()); + } + } +} diff --git a/components/http/http-api/src/testFixtures/java/datadog/http/client/HttpClientAsyncTest.java b/components/http/http-api/src/testFixtures/java/datadog/http/client/HttpClientAsyncTest.java new file mode 100644 index 00000000000..30c2df16168 --- /dev/null +++ b/components/http/http-api/src/testFixtures/java/datadog/http/client/HttpClientAsyncTest.java @@ -0,0 +1,241 @@ +package datadog.http.client; + +import static java.util.concurrent.TimeUnit.SECONDS; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assertions.fail; +import static org.mockserver.model.HttpRequest.request; +import static org.mockserver.model.HttpResponse.response; + +import java.io.IOException; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicReference; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockserver.integration.ClientAndServer; +import org.mockserver.junit.jupiter.MockServerExtension; + +@ExtendWith(MockServerExtension.class) +public class HttpClientAsyncTest { + private static final int TIMEOUT_SECONDS = 5; + private ClientAndServer server; + private HttpClient client; + private String baseUrl; + + @BeforeEach + void setUp(ClientAndServer server) { + this.server = server; + this.client = HttpClient.newBuilder().build(); + this.baseUrl = "http://localhost:" + server.getPort(); + } + + @AfterEach + void tearDown() { + this.server.reset(); + } + + @Test + void testExecuteAsyncSuccess() throws Exception { + org.mockserver.model.HttpRequest expectedRequest = + request().withMethod("GET").withPath("/test"); + this.server.when(expectedRequest).respond(response().withStatusCode(200).withBody("success")); + + HttpUrl url = HttpUrl.parse(this.baseUrl + "/test"); + HttpRequest request = HttpRequest.newBuilder().url(url).get().build(); + + CompletableFuture future = this.client.executeAsync(request); + + HttpResponse response = future.get(TIMEOUT_SECONDS, SECONDS); + assertNotNull(response); + assertEquals(200, response.code()); + assertTrue(response.isSuccessful()); + assertEquals("success", response.bodyAsString()); + + this.server.verify(expectedRequest); + } + + @Test + void testExecuteAsyncHttpError() throws Exception { + org.mockserver.model.HttpRequest expectedRequest = + request().withMethod("GET").withPath("/notfound"); + this.server.when(expectedRequest).respond(response().withStatusCode(404)); + + HttpUrl url = HttpUrl.parse(this.baseUrl + "/notfound"); + HttpRequest request = HttpRequest.newBuilder().url(url).get().build(); + + CompletableFuture future = this.client.executeAsync(request); + + // HTTP errors (4xx, 5xx) should complete normally, not exceptionally + HttpResponse response = future.get(TIMEOUT_SECONDS, SECONDS); + assertNotNull(response); + assertEquals(404, response.code()); + + this.server.verify(expectedRequest); + } + + @Test + void testExecuteAsyncWithListener() throws Exception { + org.mockserver.model.HttpRequest expectedRequest = + request().withMethod("GET").withPath("/test"); + this.server.when(expectedRequest).respond(response().withStatusCode(200)); + + AtomicBoolean startCalled = new AtomicBoolean(false); + AtomicBoolean endCalled = new AtomicBoolean(false); + AtomicReference capturedResponse = new AtomicReference<>(); + CountDownLatch latch = new CountDownLatch(1); + + HttpUrl url = HttpUrl.parse(this.baseUrl + "/test"); + HttpRequest request = + HttpRequest.newBuilder() + .url(url) + .get() + .listener( + new HttpRequestListener() { + @Override + public void onRequestStart(HttpRequest request) { + startCalled.set(true); + } + + @Override + public void onRequestEnd(HttpRequest request, HttpResponse response) { + endCalled.set(true); + capturedResponse.set(response); + latch.countDown(); + } + + @Override + public void onRequestFailure(HttpRequest request, IOException exception) { + fail("Should not fail"); + } + }) + .build(); + + this.client.executeAsync(request); + + assertTrue(latch.await(TIMEOUT_SECONDS, SECONDS), "Listener should be called"); + assertTrue(startCalled.get(), "onRequestStart should be called"); + assertTrue(endCalled.get(), "onRequestEnd should be called"); + assertNotNull(capturedResponse.get()); + assertEquals(200, capturedResponse.get().code()); + + this.server.verify(expectedRequest); + } + + @Test + void testExecuteAsyncWithListenerOnFailure() throws Exception { + // Use an invalid port to cause connection failure + HttpUrl url = HttpUrl.parse("http://localhost:1/test"); + + AtomicBoolean startCalled = new AtomicBoolean(false); + AtomicBoolean failureCalled = new AtomicBoolean(false); + AtomicReference capturedException = new AtomicReference<>(); + CountDownLatch latch = new CountDownLatch(1); + + HttpRequest request = + HttpRequest.newBuilder() + .url(url) + .get() + .listener( + new HttpRequestListener() { + @Override + public void onRequestStart(HttpRequest request) { + startCalled.set(true); + } + + @Override + public void onRequestEnd(HttpRequest request, HttpResponse response) { + fail("Should not succeed"); + } + + @Override + public void onRequestFailure(HttpRequest request, IOException exception) { + failureCalled.set(true); + capturedException.set(exception); + latch.countDown(); + } + }) + .build(); + + CompletableFuture future = this.client.executeAsync(request); + + assertTrue(latch.await(TIMEOUT_SECONDS, SECONDS), "Listener should be called"); + assertTrue(startCalled.get(), "onRequestStart should be called"); + assertTrue(failureCalled.get(), "onRequestFailure should be called"); + assertNotNull(capturedException.get()); + + // The future should also complete exceptionally + try { + future.get(TIMEOUT_SECONDS, SECONDS); + fail("Future should complete exceptionally"); + } catch (ExecutionException e) { + assertInstanceOf(IOException.class, e.getCause()); + } + } + + @Test + void testExecuteAsyncComposition() throws Exception { + org.mockserver.model.HttpRequest expectedRequest = + request().withMethod("GET").withPath("/test"); + this.server.when(expectedRequest).respond(response().withStatusCode(200).withBody("42")); + + HttpUrl url = HttpUrl.parse(this.baseUrl + "/test"); + HttpRequest request = HttpRequest.newBuilder().url(url).get().build(); + + // Test thenApply composition + CompletableFuture future = + this.client + .executeAsync(request) + .thenApply( + response -> { + try { + return Integer.parseInt(response.bodyAsString().trim()); + } catch (IOException e) { + throw new RuntimeException(e); + } + }); + + Integer result = future.get(TIMEOUT_SECONDS, SECONDS); + assertEquals(42, result); + + this.server.verify(expectedRequest); + } + + @Test + void testExecuteAsyncMultipleRequests() throws Exception { + org.mockserver.model.HttpRequest expectedRequest1 = + request().withMethod("GET").withPath("/test1"); + org.mockserver.model.HttpRequest expectedRequest2 = + request().withMethod("GET").withPath("/test2"); + this.server + .when(expectedRequest1) + .respond(response().withStatusCode(200).withBody("response1")); + this.server + .when(expectedRequest2) + .respond(response().withStatusCode(200).withBody("response2")); + + HttpRequest request1 = + HttpRequest.newBuilder().url(HttpUrl.parse(this.baseUrl + "/test1")).get().build(); + HttpRequest request2 = + HttpRequest.newBuilder().url(HttpUrl.parse(this.baseUrl + "/test2")).get().build(); + + // Execute both requests concurrently + CompletableFuture future1 = this.client.executeAsync(request1); + CompletableFuture future2 = this.client.executeAsync(request2); + + // Wait for both + CompletableFuture.allOf(future1, future2).get(TIMEOUT_SECONDS, SECONDS); + + HttpResponse response1 = future1.get(); + HttpResponse response2 = future2.get(); + + assertEquals("response1", response1.bodyAsString()); + assertEquals("response2", response2.bodyAsString()); + } +} diff --git a/components/http/http-api/src/testFixtures/java/datadog/http/client/HttpClientTest.java b/components/http/http-api/src/testFixtures/java/datadog/http/client/HttpClientTest.java new file mode 100644 index 00000000000..811ed742723 --- /dev/null +++ b/components/http/http-api/src/testFixtures/java/datadog/http/client/HttpClientTest.java @@ -0,0 +1,200 @@ +package datadog.http.client; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockserver.model.HttpRequest.request; +import static org.mockserver.model.HttpResponse.response; + +import java.io.IOException; +import java.util.List; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockserver.integration.ClientAndServer; +import org.mockserver.junit.jupiter.MockServerExtension; + +@ExtendWith(MockServerExtension.class) +public class HttpClientTest { + private ClientAndServer server; + private HttpClient client; + private String baseUrl; + + @BeforeEach + void setUp(ClientAndServer server) { + this.server = server; + this.client = HttpClient.newBuilder().build(); + this.baseUrl = "http://localhost:" + server.getPort(); + } + + @AfterEach + void tearDown() { + this.server.reset(); + } + + @Test + void testGetRequest() throws IOException { + org.mockserver.model.HttpRequest expectedRequest = + request().withMethod("GET").withPath("/test"); + this.server.when(expectedRequest).respond(response()); + + HttpUrl url = HttpUrl.parse(this.baseUrl + "/test"); + HttpRequest request = HttpRequest.newBuilder().url(url).get().build(); + + HttpResponse response = this.client.execute(request); + + assertNotNull(response); + assertEquals(200, response.code()); + assertTrue(response.isSuccessful()); + + this.server.verify(expectedRequest); + } + + @Test + void testPostRequest() throws IOException { + String payload = "{\"key\":\"value\"}"; + org.mockserver.model.HttpRequest expectedRequest = + request() + .withMethod("POST") + .withPath("/test") + .withHeader("Content-Type", "application/json") + .withBody(payload); + this.server.when(expectedRequest).respond(response().withStatusCode(201)); + + HttpUrl url = HttpUrl.parse(this.baseUrl + "/test"); + HttpRequestBody body = HttpRequestBody.of(payload); + HttpRequest request = + HttpRequest.newBuilder() + .url(url) + .header("Content-Type", "application/json") + .post(body) + .build(); + + HttpResponse response = this.client.execute(request); + + assertNotNull(response); + assertEquals(201, response.code()); + assertTrue(response.isSuccessful()); + + this.server.verify(expectedRequest); + } + + @Test + void testPutRequest() throws IOException { + String payload = "{\"key\":\"value\"}"; + org.mockserver.model.HttpRequest expectedRequest = + request() + .withMethod("PUT") + .withPath("/test") + .withHeader("Content-Type", "application/json") + .withBody(payload); + this.server.when(expectedRequest).respond(response().withStatusCode(200)); + + HttpUrl url = HttpUrl.parse(this.baseUrl + "/test"); + HttpRequestBody body = HttpRequestBody.of(payload); + HttpRequest request = + HttpRequest.newBuilder() + .url(url) + .header("Content-Type", "application/json") + .put(body) + .build(); + + HttpResponse response = this.client.execute(request); + + assertNotNull(response); + assertEquals(200, response.code()); + assertTrue(response.isSuccessful()); + + this.server.verify(expectedRequest); + } + + @Test + void testErrorResponse() throws IOException { + org.mockserver.model.HttpRequest expectedRequest = + request().withMethod("GET").withPath("/missing"); + this.server.when(expectedRequest).respond(response().withStatusCode(404)); + + HttpUrl url = HttpUrl.parse(this.baseUrl + "/missing"); + HttpRequest request = HttpRequest.newBuilder().url(url).get().build(); + + HttpResponse response = this.client.execute(request); + + assertNotNull(response); + assertEquals(404, response.code()); + assertFalse(response.isSuccessful()); + + this.server.verify(expectedRequest); + } + + @Test + void testRequestHeaders() throws IOException { + org.mockserver.model.HttpRequest expectedRequest = + request() + .withMethod("GET") + .withPath("/test") + .withHeader("Accept", "text/plain") + .withHeader("X-Custom-Header", "custom-value1", "custom-value2", "custom-value3"); + this.server.when(expectedRequest).respond(response().withStatusCode(200)); + + HttpUrl url = HttpUrl.parse(this.baseUrl + "/test"); + HttpRequest request = + HttpRequest.newBuilder() + .url(url) + .get() + .header("Accept", "text/plain") + .addHeader("X-Custom-Header", "custom-value1") + .addHeader("X-Custom-Header", "custom-value2") + .addHeader("X-Custom-Header", "custom-value3") + .build(); + + HttpResponse response = this.client.execute(request); + + assertNotNull(response); + assertEquals(200, response.code()); + assertTrue(response.isSuccessful()); + + this.server.verify(expectedRequest); + } + + @Test + void testResponseHeaders() throws IOException { + org.mockserver.model.HttpRequest expectedRequest = + request().withMethod("GET").withPath("/test"); + org.mockserver.model.HttpResponse resultResponse = + response() + .withStatusCode(200) + .withHeader("Content-Type", "text/plain") + .withHeader("X-Custom-Header", "value1", "value2", "value3") + .withBody("test-response"); + this.server.when(expectedRequest).respond(resultResponse); + + HttpUrl url = HttpUrl.parse(this.baseUrl + "/test"); + HttpRequest request = HttpRequest.newBuilder().url(url).get().build(); + + HttpResponse response = this.client.execute(request); + + assertNotNull(response); + assertEquals(200, response.code()); + assertTrue(response.isSuccessful()); + assertEquals("text/plain", response.header("Content-Type")); + assertEquals("value1", response.header("X-Custom-Header")); + List customHeaderValues = response.headers("X-Custom-Header"); + assertEquals(3, customHeaderValues.size()); + assertEquals("value1", customHeaderValues.get(0)); + assertEquals("value2", customHeaderValues.get(1)); + assertEquals("value3", customHeaderValues.get(2)); + + this.server.verify(expectedRequest); + } + + @Test + void testNewBuilder() { + HttpClient.Builder builder = HttpClient.newBuilder(); + assertNotNull(builder); + + HttpClient client = builder.build(); + assertNotNull(client); + } +} diff --git a/components/http/http-api/src/testFixtures/java/datadog/http/client/HttpRequestBodyTest.java b/components/http/http-api/src/testFixtures/java/datadog/http/client/HttpRequestBodyTest.java new file mode 100644 index 00000000000..966175b070b --- /dev/null +++ b/components/http/http-api/src/testFixtures/java/datadog/http/client/HttpRequestBodyTest.java @@ -0,0 +1,155 @@ +package datadog.http.client; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.zip.GZIPInputStream; +import org.junit.jupiter.api.Test; + +public class HttpRequestBodyTest { + + // TODO Test empty string + // TODO Test empty byte array + // TODO Test empty ByteBuffer list + + @Test + void testNullString() { + assertThrows(NullPointerException.class, () -> HttpRequestBody.of((String) null)); + } + + @Test + void testNullBytes() { + assertThrows(NullPointerException.class, () -> HttpRequestBody.of((byte[]) null)); + } + + @Test + void testNullByteBuffer() { + assertThrows(NullPointerException.class, () -> HttpRequestBody.of((List) null)); + } + + @Test + void testMultipartBuilder() { + HttpRequestBody.MultipartBuilder builder = HttpRequestBody.multipart(); + assertNotNull(builder); + } + + @Test + void testMultipartContentType() { + HttpRequestBody.MultipartBuilder builder = HttpRequestBody.multipart(); + builder.addFormDataPart("name", "value"); + String contentType = builder.contentType(); + assertTrue(contentType.startsWith("multipart/form-data; boundary=")); + } + + @Test + void testMultipartAddFormDataPart() throws IOException { + HttpRequestBody.MultipartBuilder builder = HttpRequestBody.multipart(); + builder.addFormDataPart("name", "value"); + HttpRequestBody body = builder.build(); + + assertNotNull(body); + assertTrue(body.contentLength() > 0); + + ByteArrayOutputStream out = new ByteArrayOutputStream(); + body.writeTo(out); + String content = out.toString("UTF-8"); + assertTrue(content.contains("name=\"name\"")); + assertTrue(content.contains("value")); + } + + @Test + void testMultipartAddFormDataPartWithFile() throws IOException { + HttpRequestBody.MultipartBuilder builder = HttpRequestBody.multipart(); + HttpRequestBody fileBody = HttpRequestBody.of("file content"); + builder.addFormDataPart("file", "test.txt", fileBody); + HttpRequestBody body = builder.build(); + + assertNotNull(body); + assertTrue(body.contentLength() > 0); + + ByteArrayOutputStream out = new ByteArrayOutputStream(); + body.writeTo(out); + String content = out.toString("UTF-8"); + assertTrue(content.contains("name=\"file\"")); + assertTrue(content.contains("filename=\"test.txt\"")); + assertTrue(content.contains("file content")); + } + + @Test + void testMultipartAddPart() throws IOException { + HttpRequestBody.MultipartBuilder builder = HttpRequestBody.multipart(); + Map headers = new HashMap<>(); + headers.put("Content-Disposition", "form-data; name=\"custom\"; filename=\"data.bin\""); + HttpRequestBody partBody = HttpRequestBody.of("custom content"); + builder.addPart(headers, partBody); + HttpRequestBody body = builder.build(); + + assertNotNull(body); + assertTrue(body.contentLength() > 0); + + ByteArrayOutputStream out = new ByteArrayOutputStream(); + body.writeTo(out); + String content = out.toString("UTF-8"); + assertTrue(content.contains("name=\"custom\"")); + assertTrue(content.contains("custom content")); + } + + @Test + void testMultipartNullParams() { + HttpRequestBody.MultipartBuilder builder = HttpRequestBody.multipart(); + assertThrows(NullPointerException.class, () -> builder.addFormDataPart(null, "value")); + assertThrows(NullPointerException.class, () -> builder.addFormDataPart("name", null)); + + HttpRequestBody fileBody = HttpRequestBody.of("content"); + assertThrows( + NullPointerException.class, () -> builder.addFormDataPart(null, "file.txt", fileBody)); + assertThrows(NullPointerException.class, () -> builder.addFormDataPart("name", null, fileBody)); + assertThrows( + NullPointerException.class, () -> builder.addFormDataPart("name", "file.txt", null)); + + HttpRequestBody partBody = HttpRequestBody.of("content"); + Map headers = new HashMap<>(); + headers.put("Content-Disposition", "form-data; name=\"test\""); + assertThrows(NullPointerException.class, () -> builder.addPart(null, partBody)); + assertThrows(NullPointerException.class, () -> builder.addPart(headers, null)); + } + + @Test + void testGzipBody() throws IOException { + String content = "this is test content for gzip compression"; + HttpRequestBody originalBody = HttpRequestBody.of(content); + HttpRequestBody gzippedBody = HttpRequestBody.gzip(originalBody); + assertNotNull(gzippedBody); + // Content length is unknown since compression is done lazily + assertEquals(-1, gzippedBody.contentLength()); + // Dump zipped content to bytes + ByteArrayOutputStream compressedOut = new ByteArrayOutputStream(); + gzippedBody.writeTo(compressedOut); + byte[] compressedBytes = compressedOut.toByteArray(); + // Decompress and verify content matches the original content + try (GZIPInputStream gzipIn = new GZIPInputStream(new ByteArrayInputStream(compressedBytes)); + ByteArrayOutputStream decompressedOut = new ByteArrayOutputStream()) { + byte[] buffer = new byte[1024]; + int len; + while ((len = gzipIn.read(buffer)) != -1) { + decompressedOut.write(buffer, 0, len); + } + String decompressedContent = decompressedOut.toString("UTF-8"); + assertEquals(content, decompressedContent); + } + } + + @Test + void testGzipNullBody() { + assertThrows(NullPointerException.class, () -> HttpRequestBody.gzip(null)); + } +} diff --git a/components/http/http-api/src/testFixtures/java/datadog/http/client/HttpRequestTest.java b/components/http/http-api/src/testFixtures/java/datadog/http/client/HttpRequestTest.java new file mode 100644 index 00000000000..28b9efb0484 --- /dev/null +++ b/components/http/http-api/src/testFixtures/java/datadog/http/client/HttpRequestTest.java @@ -0,0 +1,142 @@ +package datadog.http.client; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.List; +import org.junit.jupiter.api.Test; + +public class HttpRequestTest { + @Test + void testGetRequest() { + HttpUrl url = HttpUrl.parse("http://localhost:8080/api"); + HttpRequest request = HttpRequest.newBuilder().url(url).get().build(); + + assertNotNull(request); + assertEquals(url, request.url()); + assertEquals("GET", request.method()); + } + + @Test + void testPostRequest() { + HttpUrl url = HttpUrl.parse("http://localhost:8080/api"); + String payload = "{\"key\":\"value\"}"; + HttpRequestBody body = HttpRequestBody.of(payload); + + HttpRequest request = HttpRequest.newBuilder().url(url).post(body).build(); + + assertNotNull(request); + assertEquals(url, request.url()); + assertEquals("POST", request.method()); + } + + @Test + void testPutRequest() { + HttpUrl url = HttpUrl.parse("http://localhost:8080/api"); + String payload = "{\"key\":\"value\"}"; + HttpRequestBody body = HttpRequestBody.of(payload); + + HttpRequest request = HttpRequest.newBuilder().url(url).put(body).build(); + + assertEquals("PUT", request.method()); + } + + @Test + void testWithoutMethod() { + HttpRequest request = HttpRequest.newBuilder().url("http://localhost:8080/test").build(); + assertEquals("GET", request.method()); + } + + @Test + void testRequestWithUrlString() { + HttpRequest request = HttpRequest.newBuilder().url("http://localhost:8080/test").get().build(); + + assertNotNull(request); + assertEquals("http://localhost:8080/test", request.url().url()); + } + + @Test + void testRequestWithSingleHeader() { + HttpRequest request = + HttpRequest.newBuilder() + .url("http://localhost:8080/test") + .header("Content-Type", "application/json") + .get() + .build(); + + assertEquals("application/json", request.header("Content-Type")); + } + + @Test + void testRequestWithMultipleHeaders() { + HttpRequest request = + HttpRequest.newBuilder() + .url("http://localhost:8080/test") + .header("Content-Type", "application/json") + .header("Accept", "application/json") + .addHeader("X-Custom-Header", "value1") + .addHeader("X-Custom-Header", "value2") + .get() + .build(); + + assertEquals("application/json", request.header("Content-Type")); + assertEquals("application/json", request.header("Accept")); + + List customHeaders = request.headers("X-Custom-Header"); + assertEquals(2, customHeaders.size()); + assertTrue(customHeaders.contains("value1")); + assertTrue(customHeaders.contains("value2")); + } + + @Test + void testWithoutUrl() { + assertThrows(IllegalStateException.class, () -> HttpRequest.newBuilder().get().build()); + } + + @Test + void testHeaderReplacement() { + HttpRequest request = + HttpRequest.newBuilder() + .url("http://localhost:8080/test") + .header("Content-Type", "text/plain") + .header("Content-Type", "application/json") + .get() + .build(); + + assertEquals("application/json", request.header("Content-Type")); + } + + @Test + void testMissingHeader() { + HttpRequest request = HttpRequest.newBuilder().url("http://localhost:8080/test").get().build(); + + assertNull(request.header("X-Missing")); + List missing = request.headers("X-Missing"); + assertNotNull(missing); + assertTrue(missing.isEmpty()); + } + + @Test + void testEmptyHeaderValue() { + HttpRequest request = + HttpRequest.newBuilder() + .url("http://localhost:8080/test") + .header("X-Empty-Header", "") + .get() + .build(); + + assertEquals("", request.header("X-Empty-Header")); + } + + @Test + void testNullHeader() { + HttpRequest.Builder builder = HttpRequest.newBuilder().url("http://localhost:8080/test"); + assertThrows(NullPointerException.class, () -> builder.header(null, "value")); + assertThrows(NullPointerException.class, () -> builder.header("X-Custom-Header", null)); + assertThrows(NullPointerException.class, () -> builder.addHeader(null, "value")); + assertThrows(NullPointerException.class, () -> builder.addHeader("X-Custom-Header", null)); + } +} diff --git a/components/http/http-api/src/testFixtures/java/datadog/http/client/HttpResponseTest.java b/components/http/http-api/src/testFixtures/java/datadog/http/client/HttpResponseTest.java new file mode 100644 index 00000000000..2bfde756dcc --- /dev/null +++ b/components/http/http-api/src/testFixtures/java/datadog/http/client/HttpResponseTest.java @@ -0,0 +1,136 @@ +package datadog.http.client; + +import static java.nio.charset.StandardCharsets.UTF_8; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockserver.model.HttpRequest.request; +import static org.mockserver.model.HttpResponse.response; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.util.Set; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockserver.integration.ClientAndServer; +import org.mockserver.junit.jupiter.MockServerExtension; + +@ExtendWith(MockServerExtension.class) +public class HttpResponseTest { + private ClientAndServer server; + private HttpClient client; + private String baseUrl; + + @BeforeEach + void setUp(ClientAndServer server) { + this.server = server; + this.client = HttpClient.newBuilder().build(); + this.baseUrl = "http://localhost:" + server.getPort(); + } + + @AfterEach + void tearDown() { + this.server.reset(); + } + + @Test + void testBody() throws IOException { + org.mockserver.model.HttpRequest expectedRequest = + request().withMethod("GET").withPath("/test"); + String responseBody = "content"; + this.server.when(expectedRequest).respond(response().withBody(responseBody)); + + HttpUrl url = HttpUrl.parse(this.baseUrl + "/test"); + HttpRequest request = HttpRequest.newBuilder().url(url).get().build(); + + HttpResponse response = this.client.execute(request); + + assertNotNull(response); + assertEquals(200, response.code()); + assertTrue(response.isSuccessful()); + try (InputStream body = response.body()) { + assertEquals(responseBody, readAll(body)); + } + } + + @Test + void testEmptyBody() throws IOException { + org.mockserver.model.HttpRequest expectedRequest = + request().withMethod("GET").withPath("/test"); + this.server.when(expectedRequest).respond(response()); + + HttpUrl url = HttpUrl.parse(this.baseUrl + "/test"); + HttpRequest request = HttpRequest.newBuilder().url(url).get().build(); + + HttpResponse response = this.client.execute(request); + + assertNotNull(response); + assertEquals(200, response.code()); + assertTrue(response.isSuccessful()); + try (InputStream body = response.body()) { + assertEquals("", readAll(body)); + } + } + + @Test + void testHeader() throws IOException { + org.mockserver.model.HttpRequest expectedRequest = + request().withMethod("GET").withPath("/test"); + org.mockserver.model.HttpResponse resultResponse = + response().withHeader("Content-Type", "text/plain"); + this.server.when(expectedRequest).respond(resultResponse); + + HttpUrl url = HttpUrl.parse(this.baseUrl + "/test"); + HttpRequest request = HttpRequest.newBuilder().url(url).get().build(); + + HttpResponse response = this.client.execute(request); + + // case-insensitive + assertEquals("text/plain", response.header("Content-Type")); + assertEquals("text/plain", response.header("content-type")); + assertEquals("text/plain", response.header("CONTENT-TYPE")); + // missing header + assertNull(response.header("X-Missing-Header")); + assertTrue(response.headers("X-Missing-Header").isEmpty()); + } + + @Test + void testHeaderNames() throws IOException { + org.mockserver.model.HttpRequest expectedRequest = + request().withMethod("GET").withPath("/test"); + org.mockserver.model.HttpResponse resultResponse = + response() + .withHeader("Content-Type", "application/json") + .withHeader("X-Custom-Header", "custom-value") + .withHeader("X-Another-Header", "another-value"); + this.server.when(expectedRequest).respond(resultResponse); + + HttpUrl url = HttpUrl.parse(this.baseUrl + "/test"); + HttpRequest request = HttpRequest.newBuilder().url(url).get().build(); + + HttpResponse response = this.client.execute(request); + + Set headerNames = response.headerNames(); + assertTrue(headerNames.contains("Content-Type")); + assertTrue(headerNames.contains("X-Custom-Header")); + assertTrue(headerNames.contains("X-Another-Header")); + } + + private String readAll(InputStream in) throws IOException { + BufferedReader reader = new BufferedReader(new InputStreamReader(in, UTF_8)); + StringBuilder sb = new StringBuilder(); + String line; + while ((line = reader.readLine()) != null) { + if (sb.length() > 0) { + sb.append('\n'); + } + sb.append(line); + } + return sb.toString(); + } +} diff --git a/components/http/http-api/src/testFixtures/java/datadog/http/client/HttpUrlTest.java b/components/http/http-api/src/testFixtures/java/datadog/http/client/HttpUrlTest.java new file mode 100644 index 00000000000..da4f6f1b513 --- /dev/null +++ b/components/http/http-api/src/testFixtures/java/datadog/http/client/HttpUrlTest.java @@ -0,0 +1,425 @@ +package datadog.http.client; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.net.URI; +import org.junit.jupiter.api.Test; + +// Mostly a bunch of generated tests to ensure similar behavior of the different implementations +public class HttpUrlTest { + + // ==================== parse() tests ==================== + + @Test + void testParseSimpleUrl() { + HttpUrl url = HttpUrl.parse("https://example.com"); + + assertNotNull(url); + assertEquals("https", url.scheme()); + assertEquals("example.com", url.host()); + assertEquals(443, url.port()); + } + + @Test + void testParseUrlWithPort() { + HttpUrl url = HttpUrl.parse("http://localhost:8080"); + + assertEquals("http", url.scheme()); + assertEquals("localhost", url.host()); + assertEquals(8080, url.port()); + } + + @Test + void testParseUrlWithPath() { + HttpUrl url = HttpUrl.parse("https://example.com/api/v1/users"); + + assertEquals("https", url.scheme()); + assertEquals("example.com", url.host()); + assertTrue(url.url().contains("/api/v1/users")); + } + + @Test + void testParseUrlWithQueryParameters() { + HttpUrl url = HttpUrl.parse("https://example.com/search?q=test&page=1"); + + assertTrue(url.url().contains("q=test")); + assertTrue(url.url().contains("page=1")); + } + + @Test + void testParseInvalidUrl() { + assertThrows(IllegalArgumentException.class, () -> HttpUrl.parse("not a valid url")); + } + + @Test + void testParseNullUrl() { + assertThrows(NullPointerException.class, () -> HttpUrl.parse(null)); + } + + // ==================== from(URI) tests ==================== + + @Test + void testFromUri() { + URI uri = URI.create("https://example.com:8443/path"); + HttpUrl url = HttpUrl.from(uri); + + assertNotNull(url); + assertEquals("https", url.scheme()); + assertEquals("example.com", url.host()); + assertEquals(8443, url.port()); + assertTrue(url.url().contains("/path")); + } + + @Test + void testFromUriNull() { + assertThrows(NullPointerException.class, () -> HttpUrl.from(null)); + } + + // ==================== scheme() tests ==================== + + @Test + void testSchemeHttp() { + HttpUrl url = HttpUrl.parse("http://example.com"); + assertEquals("http", url.scheme()); + } + + @Test + void testSchemeHttps() { + HttpUrl url = HttpUrl.parse("https://example.com"); + assertEquals("https", url.scheme()); + } + + // ==================== host() tests ==================== + + @Test + void testHostDomain() { + HttpUrl url = HttpUrl.parse("https://www.example.com"); + assertEquals("www.example.com", url.host()); + } + + @Test + void testHostLocalhost() { + HttpUrl url = HttpUrl.parse("http://localhost:8080"); + assertEquals("localhost", url.host()); + } + + @Test + void testHostIpAddress() { + HttpUrl url = HttpUrl.parse("http://192.168.1.1:8080"); + assertEquals("192.168.1.1", url.host()); + } + + // ==================== port() tests ==================== + + @Test + void testPortExplicit() { + HttpUrl url = HttpUrl.parse("https://example.com:8443"); + assertEquals(8443, url.port()); + } + + @Test + void testPortDefaultHttp() { + HttpUrl url = HttpUrl.parse("http://example.com"); + assertEquals(80, url.port()); + } + + @Test + void testPortDefaultHttps() { + HttpUrl url = HttpUrl.parse("https://example.com"); + assertEquals(443, url.port()); + } + + // ==================== resolve() tests ==================== + + @Test + void testResolveRelativePath() { + HttpUrl baseUrl = HttpUrl.parse("https://example.com/api"); + HttpUrl resolved = baseUrl.resolve("users"); + + assertTrue(resolved.url().contains("example.com")); + assertTrue(resolved.url().contains("users")); + } + + @Test + void testResolveAbsolutePath() { + HttpUrl baseUrl = HttpUrl.parse("https://example.com/api/v1"); + HttpUrl resolved = baseUrl.resolve("/v2/users"); + + assertTrue(resolved.url().contains("example.com")); + assertTrue(resolved.url().contains("/v2/users")); + assertFalse(resolved.url().contains("/api")); + } + + @Test + void testResolveWithQueryParameters() { + HttpUrl baseUrl = HttpUrl.parse("https://example.com/api"); + HttpUrl resolved = baseUrl.resolve("search?q=test"); + + assertTrue(resolved.url().contains("search")); + assertTrue(resolved.url().contains("q=test")); + } + + // ==================== newBuilder() tests ==================== + + @Test + void testNewBuilderPreservesUrl() { + HttpUrl original = HttpUrl.parse("https://example.com:8443/api"); + HttpUrl rebuilt = original.newBuilder().build(); + + assertEquals(original.scheme(), rebuilt.scheme()); + assertEquals(original.host(), rebuilt.host()); + assertEquals(original.port(), rebuilt.port()); + } + + @Test + void testNewBuilderAllowsModification() { + HttpUrl original = HttpUrl.parse("https://example.com/api"); + HttpUrl modified = original.newBuilder().addPathSegment("v2").build(); + + assertTrue(modified.url().contains("/api")); + assertTrue(modified.url().contains("v2")); + } + + // ==================== addPathSegment() tests ==================== + + @Test + void testAddPathSegmentSingle() { + HttpUrl url = + HttpUrl.builder().scheme("https").host("example.com").addPathSegment("api").build(); + + assertTrue(url.url().contains("/api")); + } + + @Test + void testAddPathSegmentMultiple() { + HttpUrl url = + HttpUrl.builder() + .scheme("https") + .host("example.com") + .addPathSegment("api") + .addPathSegment("v1") + .addPathSegment("users") + .build(); + + String urlString = url.url(); + assertTrue(urlString.contains("/api")); + assertTrue(urlString.contains("/v1")); + assertTrue(urlString.contains("/users")); + } + + @Test + void testAddPathSegmentWithPort() { + HttpUrl url = + HttpUrl.builder() + .scheme("https") + .host("example.com") + .port(8443) + .addPathSegment("api") + .build(); + + String urlString = url.url(); + assertTrue(urlString.contains(":8443")); + assertTrue(urlString.contains("/api")); + } + + // ==================== builder scheme/host/port tests ==================== + + @Test + void testBuilderSchemeHostPort() { + HttpUrl url = HttpUrl.builder().scheme("https").host("api.example.com").port(8443).build(); + + assertEquals("https", url.scheme()); + assertEquals("api.example.com", url.host()); + assertEquals(8443, url.port()); + } + + @Test + void testBuilderDefaultScheme() { + HttpUrl url = HttpUrl.builder().host("example.com").build(); + + assertEquals("http", url.scheme()); + } + + // ==================== addQueryParameter() tests ==================== + + @Test + void testAddQueryParameterSingle() { + HttpUrl url = + HttpUrl.builder() + .scheme("https") + .host("example.com") + .addQueryParameter("key", "value") + .build(); + + String urlString = url.url(); + // OkHttp adds trailing slash, JDK doesn't - both are valid + assertTrue( + urlString.matches("https://example\\.com/?\\?key=value"), + "Expected URL with query parameter, got: " + urlString); + } + + @Test + void testAddQueryParameterMultiple() { + HttpUrl url = + HttpUrl.builder() + .scheme("https") + .host("example.com") + .addQueryParameter("key1", "value1") + .addQueryParameter("key2", "value2") + .addQueryParameter("key3", "value3") + .build(); + + String urlString = url.url(); + // OkHttp adds trailing slash, JDK doesn't - both are valid + assertTrue( + urlString.matches("https://example\\.com/?\\?.*"), + "Expected URL with query parameters, got: " + urlString); + assertTrue(urlString.contains("key1=value1")); + assertTrue(urlString.contains("key2=value2")); + assertTrue(urlString.contains("key3=value3")); + assertTrue(urlString.contains("&")); + } + + @Test + void testAddQueryParameterWithNullValue() { + HttpUrl url = + HttpUrl.builder() + .scheme("https") + .host("example.com") + .addQueryParameter("flag", null) + .build(); + + String urlString = url.url(); + assertTrue(urlString.contains("flag")); + assertFalse(urlString.contains("=")); + assertFalse(urlString.contains("null")); + } + + @Test + void testAddQueryParameterWithEncoding() { + HttpUrl url = + HttpUrl.builder() + .scheme("https") + .host("example.com") + .addQueryParameter("message", "hello world") + .addQueryParameter("special", "a=b&c=d") + .build(); + + String urlString = url.url(); + // Values should be URL encoded - accept both + and %20 for space + assertTrue( + urlString.contains("message=hello+world") || urlString.contains("message=hello%20world"), + "Expected encoded space in URL, got: " + urlString); + assertTrue( + urlString.contains("special=a%3Db%26c%3Dd"), + "Expected encoded special chars in URL, got: " + urlString); + } + + @Test + void testAddQueryParameterWithPath() { + HttpUrl url = + HttpUrl.builder() + .scheme("https") + .host("example.com") + .addPathSegment("api") + .addPathSegment("v1") + .addQueryParameter("page", "1") + .addQueryParameter("limit", "10") + .build(); + + String urlString = url.url(); + assertTrue(urlString.contains("example.com/api/v1")); + assertTrue(urlString.contains("page=1")); + assertTrue(urlString.contains("limit=10")); + } + + @Test + void testAddQueryParameterFromExistingUrl() { + HttpUrl baseUrl = HttpUrl.parse("https://example.com/api"); + HttpUrl url = baseUrl.newBuilder().addQueryParameter("token", "abc123").build(); + + String urlString = url.url(); + assertTrue(urlString.contains("example.com/api")); + assertTrue(urlString.contains("token=abc123")); + } + + @Test + void testAddQueryParameterPreservesExistingQuery() { + HttpUrl baseUrl = HttpUrl.parse("https://example.com/api?existing=param"); + HttpUrl url = baseUrl.newBuilder().addQueryParameter("new", "value").build(); + + String urlString = url.url(); + assertTrue(urlString.contains("existing=param")); + assertTrue(urlString.contains("new=value")); + } + + @Test + void testAddQueryParameterEmptyValue() { + HttpUrl url = + HttpUrl.builder().scheme("https").host("example.com").addQueryParameter("key", "").build(); + + String urlString = url.url(); + assertTrue(urlString.contains("key=")); + } + + @Test + void testAddQueryParameterSpecialCharactersInName() { + HttpUrl url = + HttpUrl.builder() + .scheme("https") + .host("example.com") + .addQueryParameter("my-key", "value") + .addQueryParameter("my_key", "value2") + .build(); + + String urlString = url.url(); + assertTrue(urlString.contains("my-key=value")); + assertTrue(urlString.contains("my_key=value2")); + } + + @Test + void testAddQueryParameterWithPort() { + HttpUrl url = + HttpUrl.builder() + .scheme("https") + .host("example.com") + .port(8443) + .addQueryParameter("key", "value") + .build(); + + String urlString = url.url(); + assertTrue(urlString.contains(":8443")); + assertTrue(urlString.contains("key=value")); + } + + // ==================== url() tests ==================== + + @Test + void testUrlReturnsCompleteUrl() { + HttpUrl url = + HttpUrl.builder() + .scheme("https") + .host("example.com") + .port(8443) + .addPathSegment("api") + .addQueryParameter("key", "value") + .build(); + + String urlString = url.url(); + assertTrue(urlString.startsWith("https://")); + assertTrue(urlString.contains("example.com")); + assertTrue(urlString.contains(":8443")); + assertTrue(urlString.contains("/api")); + assertTrue(urlString.contains("key=value")); + } + + @Test + void testUrlMatchesToString() { + HttpUrl url = HttpUrl.parse("https://example.com/api"); + assertEquals(url.url(), url.toString()); + } +} diff --git a/components/json/build.gradle.kts b/components/json/build.gradle.kts index 4dca7fc3036..ce67f74ff29 100644 --- a/components/json/build.gradle.kts +++ b/components/json/build.gradle.kts @@ -5,5 +5,5 @@ plugins { apply(from = "$rootDir/gradle/java.gradle") jmh { - version = "1.28" + jmhVersion = libs.versions.jmh.get() } diff --git a/components/json/gradle.lockfile b/components/json/gradle.lockfile index 0f812828a68..7e798424d36 100644 --- a/components/json/gradle.lockfile +++ b/components/json/gradle.lockfile @@ -3,116 +3,79 @@ # This file is expected to be part of source control. ch.qos.logback:logback-classic:1.2.13=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath ch.qos.logback:logback-core:1.2.13=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -com.beust:jcommander:1.78=jmhRuntimeClasspath,testRuntimeClasspath -com.github.javaparser:javaparser-core:3.25.6=codenarc,jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.2.0=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.7.3=spotbugs -com.github.spotbugs:spotbugs:4.7.3=spotbugs +com.github.javaparser:javaparser-core:3.25.6=codenarc +com.github.spotbugs:spotbugs-annotations:4.9.8=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath +com.github.spotbugs:spotbugs:4.9.8=spotbugs +com.github.stephenc.jcip:jcip-annotations:1.0-1=spotbugs com.google.code.findbugs:jsr305:3.0.2=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -com.google.code.gson:gson:2.9.1=spotbugs -com.thoughtworks.qdox:qdox:1.12.1=codenarc,jmhRuntimeClasspath,testRuntimeClasspath -commons-codec:commons-codec:1.15=spotbugs -de.thetaphi:forbiddenapis:3.8=compileClasspath,jmhCompileClasspath -info.picocli:picocli:4.6.3=jmhRuntimeClasspath,testRuntimeClasspath +com.google.code.gson:gson:2.13.2=spotbugs +com.google.errorprone:error_prone_annotations:2.41.0=spotbugs +com.thoughtworks.qdox:qdox:1.12.1=codenarc +commons-io:commons-io:2.20.0=spotbugs +de.thetaphi:forbiddenapis:3.10=compileClasspath,jmhCompileClasspath io.leangen.geantyref:geantyref:1.3.16=jmhRuntimeClasspath,testRuntimeClasspath -jaxen:jaxen:1.2.0=spotbugs -jline:jline:2.14.6=jmhRuntimeClasspath,testRuntimeClasspath -junit:junit:4.13.2=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +jaxen:jaxen:2.0.0=spotbugs net.bytebuddy:byte-buddy-agent:1.12.8=jmhRuntimeClasspath,testRuntimeClasspath net.bytebuddy:byte-buddy:1.12.8=jmhRuntimeClasspath,testRuntimeClasspath -net.jcip:jcip-annotations:1.0=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath net.sf.jopt-simple:jopt-simple:5.0.4=jmh,jmhCompileClasspath,jmhRuntimeClasspath -net.sf.saxon:Saxon-HE:11.4=spotbugs +net.sf.saxon:Saxon-HE:12.9=spotbugs org.apache.ant:ant-antlr:1.10.14=codenarc -org.apache.ant:ant-antlr:1.10.15=jmhRuntimeClasspath,testRuntimeClasspath org.apache.ant:ant-junit:1.10.14=codenarc -org.apache.ant:ant-junit:1.10.15=jmhRuntimeClasspath,testRuntimeClasspath -org.apache.ant:ant-launcher:1.10.15=jmhRuntimeClasspath,testRuntimeClasspath -org.apache.ant:ant:1.10.15=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.apache.bcel:bcel:6.5.0=spotbugs -org.apache.commons:commons-lang3:3.12.0=spotbugs -org.apache.commons:commons-math3:3.2=jmh,jmhCompileClasspath,jmhRuntimeClasspath -org.apache.commons:commons-text:1.10.0=spotbugs -org.apache.httpcomponents.client5:httpclient5:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5-h2:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5:5.1.3=spotbugs -org.apache.logging.log4j:log4j-api:2.19.0=spotbugs -org.apache.logging.log4j:log4j-core:2.19.0=spotbugs +org.apache.bcel:bcel:6.11.0=spotbugs +org.apache.commons:commons-lang3:3.19.0=spotbugs +org.apache.commons:commons-math3:3.6.1=jmh,jmhCompileClasspath,jmhRuntimeClasspath +org.apache.commons:commons-text:1.14.0=spotbugs +org.apache.logging.log4j:log4j-api:2.25.2=spotbugs +org.apache.logging.log4j:log4j-core:2.25.2=spotbugs org.apiguardian:apiguardian-api:1.1.2=testCompileClasspath -org.codehaus.groovy:groovy-all:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-ant:3.0.23=codenarc -org.codehaus.groovy:groovy-ant:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-astbuilder:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-cli-picocli:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-console:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-datetime:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-docgenerator:3.0.23=codenarc -org.codehaus.groovy:groovy-docgenerator:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-groovydoc:3.0.23=codenarc -org.codehaus.groovy:groovy-groovydoc:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-groovysh:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jmx:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-json:3.0.23=codenarc -org.codehaus.groovy:groovy-json:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jsr223:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-macro:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-nio:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-servlet:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-sql:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-swing:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.codehaus.groovy:groovy-json:3.0.25=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-templates:3.0.23=codenarc -org.codehaus.groovy:groovy-templates:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test-junit5:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-testng:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-xml:3.0.23=codenarc -org.codehaus.groovy:groovy-xml:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy:3.0.23=codenarc -org.codehaus.groovy:groovy:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codenarc:CodeNarc:3.6.0=codenarc -org.dom4j:dom4j:2.1.3=spotbugs +org.codehaus.groovy:groovy:3.0.25=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.codenarc:CodeNarc:3.7.0=codenarc +org.dom4j:dom4j:2.2.0=spotbugs org.gmetrics:GMetrics:2.1.0=codenarc -org.hamcrest:hamcrest-core:1.3=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.hamcrest:hamcrest:3.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.jacoco:org.jacoco.agent:0.8.14=jacocoAgent,jacocoAnt org.jacoco:org.jacoco.ant:0.8.14=jacocoAnt org.jacoco:org.jacoco.core:0.8.14=jacocoAnt org.jacoco:org.jacoco.report:0.8.14=jacocoAnt -org.junit.jupiter:junit-jupiter-api:5.12.2=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-engine:5.12.2=jmhRuntimeClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-params:5.12.2=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter:5.12.2=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-commons:1.12.2=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-engine:1.12.2=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-launcher:1.12.2=jmhRuntimeClasspath,testRuntimeClasspath -org.junit:junit-bom:5.12.2=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit:junit-bom:5.9.1=spotbugs +org.junit.jupiter:junit-jupiter-api:5.14.1=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-engine:5.14.1=jmhRuntimeClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-params:5.14.1=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter:5.14.1=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-commons:1.14.1=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-engine:1.14.1=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-launcher:1.14.1=jmhRuntimeClasspath,testRuntimeClasspath +org.junit:junit-bom:5.14.0=spotbugs +org.junit:junit-bom:5.14.1=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.mockito:mockito-core:4.4.0=jmhRuntimeClasspath,testRuntimeClasspath org.objenesis:objenesis:3.3=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.openjdk.jmh:jmh-core:1.36=jmh,jmhCompileClasspath,jmhRuntimeClasspath -org.openjdk.jmh:jmh-generator-asm:1.36=jmh,jmhCompileClasspath,jmhRuntimeClasspath -org.openjdk.jmh:jmh-generator-bytecode:1.36=jmh,jmhCompileClasspath,jmhRuntimeClasspath -org.openjdk.jmh:jmh-generator-reflection:1.36=jmh,jmhCompileClasspath,jmhRuntimeClasspath +org.openjdk.jmh:jmh-core:1.37=jmh,jmhCompileClasspath,jmhRuntimeClasspath +org.openjdk.jmh:jmh-generator-asm:1.37=jmh,jmhCompileClasspath,jmhRuntimeClasspath +org.openjdk.jmh:jmh-generator-bytecode:1.37=jmh,jmhCompileClasspath,jmhRuntimeClasspath +org.openjdk.jmh:jmh-generator-reflection:1.37=jmh,jmhCompileClasspath,jmhRuntimeClasspath org.opentest4j:opentest4j:1.3.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.4=spotbugs -org.ow2.asm:asm-commons:9.4=spotbugs -org.ow2.asm:asm-commons:9.9=jacocoAnt -org.ow2.asm:asm-tree:9.4=spotbugs -org.ow2.asm:asm-tree:9.9=jacocoAnt -org.ow2.asm:asm-util:9.4=spotbugs +org.ow2.asm:asm-analysis:9.9=spotbugs +org.ow2.asm:asm-commons:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-tree:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-util:9.9=spotbugs org.ow2.asm:asm:9.0=jmh,jmhCompileClasspath,jmhRuntimeClasspath -org.ow2.asm:asm:9.4=spotbugs -org.ow2.asm:asm:9.9=jacocoAnt +org.ow2.asm:asm:9.9=jacocoAnt,spotbugs org.slf4j:jcl-over-slf4j:1.7.30=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.slf4j:jul-to-slf4j:1.7.30=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.slf4j:log4j-over-slf4j:1.7.30=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.slf4j:slf4j-api:1.7.32=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.slf4j:slf4j-api:2.0.0=spotbugs,spotbugsSlf4j -org.slf4j:slf4j-simple:2.0.0=spotbugsSlf4j -org.spockframework:spock-bom:2.4-M6-groovy-3.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.spockframework:spock-core:2.4-M6-groovy-3.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.testng:testng:7.5.1=jmhRuntimeClasspath,testRuntimeClasspath -org.webjars:jquery:3.5.1=jmhRuntimeClasspath,testRuntimeClasspath -org.xmlresolver:xmlresolver:4.4.3=spotbugs -xml-apis:xml-apis:1.4.01=spotbugs +org.slf4j:slf4j-api:2.0.17=spotbugs,spotbugsSlf4j +org.slf4j:slf4j-simple:2.0.17=spotbugsSlf4j +org.spockframework:spock-bom:2.4-groovy-3.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.spockframework:spock-core:2.4-groovy-3.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-junit:1.2.1=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-parser:1.2.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.xmlresolver:xmlresolver:5.3.3=spotbugs empty=annotationProcessor,jmhAnnotationProcessor,runtimeClasspath,spotbugsPlugins,testAnnotationProcessor diff --git a/components/json/src/test/groovy/datadog/json/JsonMapperTest.groovy b/components/json/src/test/groovy/datadog/json/JsonMapperTest.groovy deleted file mode 100644 index 43dd4dcb0aa..00000000000 --- a/components/json/src/test/groovy/datadog/json/JsonMapperTest.groovy +++ /dev/null @@ -1,170 +0,0 @@ -package datadog.json - -import spock.lang.Specification - -import static java.lang.Math.PI -import static java.util.Collections.emptyMap - -class JsonMapperTest extends Specification { - - def "test mapping to JSON object: #input"() { - setup: - def parsedExpected = input == null ? emptyMap() : input.clone() - parsedExpected.collect { - it -> { - if (it.value instanceof UnsupportedType) { - it.value = it.value.toString() - } else if (it.value instanceof Float) { - it.value = new Double(it.value) - } - - it - } - } - - when: - String json = JsonMapper.toJson((Map) input) - - then: - json == expected - - when: - def parsed = JsonMapper.fromJsonToMap(json) - - then: - if (input == null) { - parsed == [:] - } else { - parsed.size() == input.size() - input.each { - assert parsed.containsKey(it.key) - if (it.value instanceof UnsupportedType) { - assert parsed.get(it.key) == it.value.toString() - } else if (it.value instanceof Float) { - assert parsed.get(it.key) instanceof Double - assert (parsed.get(it.key) - it.value) < 0.001 - } else { - assert parsed.get(it.key) == it.value - } - } - } - - where: - input | expected - null | '{}' - new HashMap<>() | '{}' - ['key1': 'value1'] | '{"key1":"value1"}' - ['key1': 'value1', 'key2': 'value2'] | '{"key1":"value1","key2":"value2"}' - ['key1': 'va"lu"e1', 'ke"y2': 'value2'] | '{"key1":"va\\"lu\\"e1","ke\\"y2":"value2"}' - ['key1': null, 'key2': 'bar', 'key3': 3, 'key4': 3456789123L, 'key5': 3.142f, 'key6': PI, 'key7': true, 'key8': new UnsupportedType()] | '{"key1":null,"key2":"bar","key3":3,"key4":3456789123,"key5":3.142,"key6":3.141592653589793,"key7":true,"key8":"toString"}' - } - - private class UnsupportedType { - @Override - String toString() { - 'toString' - } - } - - def "test mapping to Map from empty JSON object"() { - when: - def parsed = JsonMapper.fromJsonToMap(json) - - then: - parsed == [:] - - where: - json << [null, 'null', '', '{}'] - } - - def "test mapping to Map from non-object JSON"() { - when: - JsonMapper.fromJsonToMap(json) - - then: - thrown(IOException) - - where: - json << ['1', '[1, 2]'] - } - - def "test mapping iterable to JSON array: #input"() { - when: - String json = JsonMapper.toJson(input as Collection) - - then: - json == expected - - when: - def parsed = JsonMapper.fromJsonToList(json) - - then: - parsed == (input?:[]) - - where: - input | expected - null | "[]" - new ArrayList<>() | "[]" - ['value1'] | "[\"value1\"]" - ['value1', 'value2'] | "[\"value1\",\"value2\"]" - ['va"lu"e1', 'value2'] | "[\"va\\\"lu\\\"e1\",\"value2\"]" - } - - def "test mapping array to JSON array: #input"() { - when: - String json = JsonMapper.toJson((String[]) input) - - then: - json == expected - - when: - def parsed = JsonMapper.fromJsonToList(json).toArray(new String[0]) - - then: - parsed == (String[]) (input?:[]) - - where: - input | expected - null | "[]" - [] | "[]" - ['value1'] | "[\"value1\"]" - ['value1', 'value2'] | "[\"value1\",\"value2\"]" - ['va"lu"e1', 'value2'] | "[\"va\\\"lu\\\"e1\",\"value2\"]" - } - - def "test mapping to List from empty JSON object"() { - when: - def parsed = JsonMapper.fromJsonToList(json) - - then: - parsed == [] - - where: - json << [null, 'null', '', '[]'] - } - - def "test mapping to JSON string: input"() { - when: - String escaped = JsonMapper.toJson((String) string) - - then: - escaped == expected - - where: - string | expected - null | "" - "" | "" - ((char) 4096).toString() | '"\\u1000"' - ((char) 256).toString() | '"\\u0100"' - ((char) 128).toString() | '"\\u0080"' - "\b" | '"\\b"' - "\t" | '"\\t"' - "\n" | '"\\n"' - "\f" | '"\\f"' - "\r" | '"\\r"' - '"' | '"\\\""' - '/' | '"\\/"' - '\\' | '"\\\\"' - "a" | '"a"' - } -} diff --git a/components/json/src/test/java/datadog/json/JsonMapperTest.java b/components/json/src/test/java/datadog/json/JsonMapperTest.java new file mode 100644 index 00000000000..926702881bc --- /dev/null +++ b/components/json/src/test/java/datadog/json/JsonMapperTest.java @@ -0,0 +1,173 @@ +package datadog.json; + +import static java.util.Collections.emptyList; +import static java.util.Collections.emptyMap; +import static org.junit.jupiter.api.Assertions.assertArrayEquals; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.params.provider.Arguments.arguments; + +import java.io.IOException; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Stream; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; +import org.junit.jupiter.params.provider.NullSource; +import org.junit.jupiter.params.provider.ValueSource; +import org.tabletest.junit.Scenario; +import org.tabletest.junit.TableTest; + +class JsonMapperTest { + @TableTest({ + "Scenario | Input | Expected ", + "null input | | '{}' ", + "empty map | [:] | '{}' ", + "single entry | [key1: value1] | '{\"key1\":\"value1\"}' ", + "two entries | [key1: value1, key2: value2] | '{\"key1\":\"value1\",\"key2\":\"value2\"}' ", + "quoted entries | [key1: va\"lu\"e1, ke\"y2: value2] | '{\"key1\":\"va\\\"lu\\\"e1\",\"ke\\\"y2\":\"value2\"}'" + }) + @ParameterizedTest(name = "test mapping to JSON object: {0}") + @MethodSource("testMappingToJsonObjectArguments") + void testMappingToJsonObject( + @Scenario String ignoredScenario, Map input, String expected) + throws IOException { + String json = JsonMapper.toJson(input); + assertEquals(expected, json); + + Map parsed = JsonMapper.fromJsonToMap(json); + if (input == null) { + assertEquals(emptyMap(), parsed); + } else { + assertEquals(input.size(), parsed.size()); + for (Map.Entry entry : input.entrySet()) { + String expectedKey = entry.getKey(); + Object expectedValue = entry.getValue(); + assertTrue(parsed.containsKey(expectedKey)); + Object parsedValue = parsed.get(expectedKey); + if (expectedValue instanceof UnsupportedType) { + assertEquals(expectedValue.toString(), parsedValue); + } else if (expectedValue instanceof Float) { + assertInstanceOf(Double.class, parsedValue); + assertEquals((Float) expectedValue, (Double) parsedValue, 0.001); + } else { + assertEquals(expectedValue, parsedValue); + } + } + } + } + + static Stream testMappingToJsonObjectArguments() { + Map complexMap = new LinkedHashMap<>(); + complexMap.put("key1", null); + complexMap.put("key2", "bar"); + complexMap.put("key3", 3); + complexMap.put("key4", 3456789123L); + complexMap.put("key5", 3.142f); + complexMap.put("key6", Math.PI); + complexMap.put("key7", true); + complexMap.put("key8", new UnsupportedType()); + + return Stream.of( + arguments( + "complex map", + complexMap, + "{\"key1\":null,\"key2\":\"bar\",\"key3\":3,\"key4\":3456789123,\"key5\":3.142,\"key6\":3.141592653589793,\"key7\":true,\"key8\":\"toString\"}")); + } + + @ParameterizedTest(name = "test mapping to Map from empty JSON object: {0}") + @NullSource + @ValueSource(strings = {"null", "", "{}"}) + void testMappingToMapFromEmptyJsonObject(String json) throws IOException { + Map parsed = JsonMapper.fromJsonToMap(json); + assertEquals(emptyMap(), parsed); + } + + @ParameterizedTest(name = "test mapping to Map from non-object JSON: {0}") + @ValueSource(strings = {"1", "[1, 2]"}) + void testMappingToMapFromNonObjectJson(String json) { + assertThrows(IOException.class, () -> JsonMapper.fromJsonToMap(json)); + } + + @TableTest({ + "Scenario | Input | Expected ", + "null input | | '[]' ", + "empty list | [] | '[]' ", + "single value | [value1] | '[\"value1\"]' ", + "two values | [value1, value2] | '[\"value1\",\"value2\"]' ", + "quoted values | [va\"lu\"e1, value2] | '[\"va\\\"lu\\\"e1\",\"value2\"]'" + }) + @ParameterizedTest(name = "test mapping iterable to JSON array: {0}") + void testMappingIterableToJsonArray(List input, String expected) throws IOException { + String json = JsonMapper.toJson(input); + assertEquals(expected, json); + + List parsed = JsonMapper.fromJsonToList(json); + assertEquals(input != null ? input : emptyList(), parsed); + } + + @TableTest({ + "Scenario | Input | Expected ", + "null input | | '[]' ", + "empty array | [] | '[]' ", + "single element | [value1] | '[\"value1\"]' ", + "two elements | [value1, value2] | '[\"value1\",\"value2\"]' ", + "escaped quotes | [va\"lu\"e1, value2] | '[\"va\\\"lu\\\"e1\",\"value2\"]'" + }) + @ParameterizedTest(name = "test mapping array to JSON array: {0}") + void testMappingArrayToJsonArray(String ignoredScenario, String[] input, String expected) + throws IOException { + String json = JsonMapper.toJson(input); + assertEquals(expected, json); + + String[] parsed = JsonMapper.fromJsonToList(json).toArray(new String[] {}); + assertArrayEquals(input != null ? input : new String[] {}, parsed); + } + + @ParameterizedTest(name = "test mapping to List from empty JSON object: {0}") + @NullSource + @ValueSource(strings = {"null", "", "[]"}) + void testMappingToListFromEmptyJsonObject(String json) throws IOException { + List parsed = JsonMapper.fromJsonToList(json); + assertEquals(emptyList(), parsed); + } + + @TableTest({ + "Scenario | input | expected ", + "null value | | '' ", + "empty string | '' | '' ", + "\\b | '\b' | '\"\\b\"'", + "\\t | '\t' | '\"\\t\"'", + "\\f | '\f' | '\"\\f\"'", + "a | 'a' | '\"a\"' ", + "/ | '/' | '\"\\/\"'" + }) + @ParameterizedTest(name = "test mapping to JSON string: {0}") + @MethodSource("testMappingToJsonStringArguments") + void testMappingToJsonString(@Scenario String ignoredScenario, String input, String expected) { + String json = JsonMapper.toJson(input); + assertEquals(expected, json); + } + + static Stream testMappingToJsonStringArguments() { + return Stream.of( + arguments("char #4096", String.valueOf((char) 4096), "\"\\u1000\""), + arguments("char #256", String.valueOf((char) 256), "\"\\u0100\""), + arguments("char #128", String.valueOf((char) 128), "\"\\u0080\""), + arguments("\\n", "\n", "\"\\n\""), + arguments("\\r", "\r", "\"\\r\""), + arguments("\"", "\"", "\"\\\"\""), + arguments("\\", "\\", "\"\\\\\"")); + } + + private static class UnsupportedType { + @Override + public String toString() { + return "toString"; + } + } +} diff --git a/components/json/src/test/java/datadog/json/JsonReaderTest.java b/components/json/src/test/java/datadog/json/JsonReaderTest.java index e716a625103..15a635c148a 100644 --- a/components/json/src/test/java/datadog/json/JsonReaderTest.java +++ b/components/json/src/test/java/datadog/json/JsonReaderTest.java @@ -215,7 +215,8 @@ void testStringEscaping() { assertEquals("\n", reader.nextString()); assertEquals("\r", reader.nextString()); assertEquals("\t", reader.nextString()); - assertEquals("É", reader.nextString()); + // Explicit escape for non-ASCII `É` to make test independent of container settings. + assertEquals("\u00C9", reader.nextString()); reader.endArray(); } catch (IOException e) { fail("Failed to read escaped JSON strings", e); diff --git a/components/native-loader/gradle.lockfile b/components/native-loader/gradle.lockfile index 0f0886b942b..e7ab997a313 100644 --- a/components/native-loader/gradle.lockfile +++ b/components/native-loader/gradle.lockfile @@ -3,109 +3,72 @@ # This file is expected to be part of source control. ch.qos.logback:logback-classic:1.2.13=testCompileClasspath,testRuntimeClasspath ch.qos.logback:logback-core:1.2.13=testCompileClasspath,testRuntimeClasspath -com.beust:jcommander:1.78=testRuntimeClasspath -com.github.javaparser:javaparser-core:3.25.6=codenarc,testCompileClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.2.0=compileClasspath,testCompileClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.7.3=spotbugs -com.github.spotbugs:spotbugs:4.7.3=spotbugs +com.github.javaparser:javaparser-core:3.25.6=codenarc +com.github.spotbugs:spotbugs-annotations:4.9.8=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath +com.github.spotbugs:spotbugs:4.9.8=spotbugs +com.github.stephenc.jcip:jcip-annotations:1.0-1=spotbugs com.google.code.findbugs:jsr305:3.0.2=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -com.google.code.gson:gson:2.9.1=spotbugs -com.thoughtworks.qdox:qdox:1.12.1=codenarc,testRuntimeClasspath -commons-codec:commons-codec:1.15=spotbugs -de.thetaphi:forbiddenapis:3.8=compileClasspath -info.picocli:picocli:4.6.3=testRuntimeClasspath +com.google.code.gson:gson:2.13.2=spotbugs +com.google.errorprone:error_prone_annotations:2.41.0=spotbugs +com.thoughtworks.qdox:qdox:1.12.1=codenarc +commons-io:commons-io:2.20.0=spotbugs +de.thetaphi:forbiddenapis:3.10=compileClasspath io.leangen.geantyref:geantyref:1.3.16=testRuntimeClasspath -jaxen:jaxen:1.2.0=spotbugs -jline:jline:2.14.6=testRuntimeClasspath -junit:junit:4.13.2=testCompileClasspath,testRuntimeClasspath +jaxen:jaxen:2.0.0=spotbugs net.bytebuddy:byte-buddy-agent:1.12.8=testRuntimeClasspath net.bytebuddy:byte-buddy:1.12.8=testRuntimeClasspath -net.jcip:jcip-annotations:1.0=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -net.sf.saxon:Saxon-HE:11.4=spotbugs +net.sf.saxon:Saxon-HE:12.9=spotbugs org.apache.ant:ant-antlr:1.10.14=codenarc -org.apache.ant:ant-antlr:1.10.15=testRuntimeClasspath org.apache.ant:ant-junit:1.10.14=codenarc -org.apache.ant:ant-junit:1.10.15=testRuntimeClasspath -org.apache.ant:ant-launcher:1.10.15=testRuntimeClasspath -org.apache.ant:ant:1.10.15=testCompileClasspath,testRuntimeClasspath -org.apache.bcel:bcel:6.5.0=spotbugs -org.apache.commons:commons-lang3:3.12.0=spotbugs -org.apache.commons:commons-text:1.10.0=spotbugs -org.apache.httpcomponents.client5:httpclient5:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5-h2:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5:5.1.3=spotbugs -org.apache.logging.log4j:log4j-api:2.19.0=spotbugs -org.apache.logging.log4j:log4j-core:2.19.0=spotbugs +org.apache.bcel:bcel:6.11.0=spotbugs +org.apache.commons:commons-lang3:3.19.0=spotbugs +org.apache.commons:commons-text:1.14.0=spotbugs +org.apache.logging.log4j:log4j-api:2.25.2=spotbugs +org.apache.logging.log4j:log4j-core:2.25.2=spotbugs org.apiguardian:apiguardian-api:1.1.2=testCompileClasspath -org.codehaus.groovy:groovy-all:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-ant:3.0.23=codenarc -org.codehaus.groovy:groovy-ant:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-astbuilder:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-cli-picocli:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-console:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-datetime:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-docgenerator:3.0.23=codenarc -org.codehaus.groovy:groovy-docgenerator:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-groovydoc:3.0.23=codenarc -org.codehaus.groovy:groovy-groovydoc:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-groovysh:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jmx:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-json:3.0.23=codenarc -org.codehaus.groovy:groovy-json:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jsr223:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-macro:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-nio:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-servlet:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-sql:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-swing:3.0.24=testCompileClasspath,testRuntimeClasspath +org.codehaus.groovy:groovy-json:3.0.25=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-templates:3.0.23=codenarc -org.codehaus.groovy:groovy-templates:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test-junit5:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-testng:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-xml:3.0.23=codenarc -org.codehaus.groovy:groovy-xml:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy:3.0.23=codenarc -org.codehaus.groovy:groovy:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codenarc:CodeNarc:3.6.0=codenarc -org.dom4j:dom4j:2.1.3=spotbugs +org.codehaus.groovy:groovy:3.0.25=testCompileClasspath,testRuntimeClasspath +org.codenarc:CodeNarc:3.7.0=codenarc +org.dom4j:dom4j:2.2.0=spotbugs org.gmetrics:GMetrics:2.1.0=codenarc -org.hamcrest:hamcrest-core:1.3=testCompileClasspath,testRuntimeClasspath org.hamcrest:hamcrest:3.0=testCompileClasspath,testRuntimeClasspath org.jacoco:org.jacoco.agent:0.8.14=jacocoAgent,jacocoAnt org.jacoco:org.jacoco.ant:0.8.14=jacocoAnt org.jacoco:org.jacoco.core:0.8.14=jacocoAnt org.jacoco:org.jacoco.report:0.8.14=jacocoAnt -org.junit.jupiter:junit-jupiter-api:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-engine:5.12.2=testRuntimeClasspath -org.junit.jupiter:junit-jupiter-params:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-commons:1.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-engine:1.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-launcher:1.12.2=testRuntimeClasspath -org.junit:junit-bom:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit:junit-bom:5.9.1=spotbugs +org.junit.jupiter:junit-jupiter-api:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-engine:5.14.1=testRuntimeClasspath +org.junit.jupiter:junit-jupiter-params:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-commons:1.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-engine:1.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-launcher:1.14.1=testRuntimeClasspath +org.junit:junit-bom:5.14.0=spotbugs +org.junit:junit-bom:5.14.1=testCompileClasspath,testRuntimeClasspath org.mockito:mockito-core:4.4.0=testRuntimeClasspath org.objenesis:objenesis:3.3=testCompileClasspath,testRuntimeClasspath org.opentest4j:opentest4j:1.3.0=testCompileClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.4=spotbugs -org.ow2.asm:asm-commons:9.4=spotbugs -org.ow2.asm:asm-commons:9.9=jacocoAnt -org.ow2.asm:asm-tree:9.4=spotbugs -org.ow2.asm:asm-tree:9.9=jacocoAnt -org.ow2.asm:asm-util:9.4=spotbugs -org.ow2.asm:asm:9.4=spotbugs -org.ow2.asm:asm:9.9=jacocoAnt +org.ow2.asm:asm-analysis:9.9=spotbugs +org.ow2.asm:asm-commons:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-tree:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-util:9.9=spotbugs +org.ow2.asm:asm:9.9=jacocoAnt,spotbugs org.slf4j:jcl-over-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath org.slf4j:jul-to-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath org.slf4j:log4j-over-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath org.slf4j:slf4j-api:1.7.32=testCompileClasspath,testRuntimeClasspath -org.slf4j:slf4j-api:2.0.0=spotbugs,spotbugsSlf4j -org.slf4j:slf4j-simple:2.0.0=spotbugsSlf4j -org.spockframework:spock-bom:2.4-M6-groovy-3.0=testCompileClasspath,testRuntimeClasspath -org.spockframework:spock-core:2.4-M6-groovy-3.0=testCompileClasspath,testRuntimeClasspath -org.testng:testng:7.5.1=testRuntimeClasspath -org.webjars:jquery:3.5.1=testRuntimeClasspath -org.xmlresolver:xmlresolver:4.4.3=spotbugs -xml-apis:xml-apis:1.4.01=spotbugs +org.slf4j:slf4j-api:2.0.17=spotbugs,spotbugsSlf4j +org.slf4j:slf4j-simple:2.0.17=spotbugsSlf4j +org.spockframework:spock-bom:2.4-groovy-3.0=testCompileClasspath,testRuntimeClasspath +org.spockframework:spock-core:2.4-groovy-3.0=testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-junit:1.2.1=testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-parser:1.2.0=testCompileClasspath,testRuntimeClasspath +org.xmlresolver:xmlresolver:5.3.3=spotbugs empty=annotationProcessor,runtimeClasspath,spotbugsPlugins,testAnnotationProcessor diff --git a/components/native-loader/src/main/java/datadog/nativeloader/ClassLoaderResourcePathLocator.java b/components/native-loader/src/main/java/datadog/nativeloader/ClassLoaderResourcePathLocator.java index 4563facfb81..1aef0ff3ca8 100644 --- a/components/native-loader/src/main/java/datadog/nativeloader/ClassLoaderResourcePathLocator.java +++ b/components/native-loader/src/main/java/datadog/nativeloader/ClassLoaderResourcePathLocator.java @@ -14,8 +14,9 @@ public ClassLoaderResourcePathLocator(final ClassLoader classLoader, final Strin } @Override - public URL locate(String component, String path) { - return this.classLoader.getResource(PathUtils.concatPath(component, this.baseResource, path)); + public URL locate(String optionalComponent, String path) { + return this.classLoader.getResource( + PathUtils.concatPath(optionalComponent, this.baseResource, path)); } @Override diff --git a/components/native-loader/src/main/java/datadog/nativeloader/CompositeLibraryLoadingListener.java b/components/native-loader/src/main/java/datadog/nativeloader/CompositeLibraryLoadingListener.java new file mode 100644 index 00000000000..6f87121b6d2 --- /dev/null +++ b/components/native-loader/src/main/java/datadog/nativeloader/CompositeLibraryLoadingListener.java @@ -0,0 +1,140 @@ +package datadog.nativeloader; + +import java.net.URL; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; + +final class CompositeLibraryLoadingListener extends SafeLibraryLoadingListener { + private final Collection listeners; + + CompositeLibraryLoadingListener(LibraryLoadingListener... listeners) { + this(Arrays.asList(listeners)); + } + + CompositeLibraryLoadingListener(Collection listeners) { + this.listeners = listeners; + } + + @Override + public boolean isNop() { + return this.listeners.isEmpty(); + } + + int size() { + return this.listeners.size(); + } + + @Override + public void onResolveDynamic( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + boolean isPreloaded, + URL optionalUrl) { + for (LibraryLoadingListener listener : this.listeners) { + try { + listener.onResolveDynamic( + platformSpec, optionalComponent, libName, isPreloaded, optionalUrl); + } catch (Throwable ignored) { + } + } + } + + @Override + public void onResolveDynamicFailure( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + Throwable optionalCause) { + for (LibraryLoadingListener listener : this.listeners) { + try { + listener.onResolveDynamicFailure(platformSpec, optionalComponent, libName, optionalCause); + } catch (Throwable ignored) { + } + } + } + + @Override + public void onLoad( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + boolean isPreloaded, + Path optionalLibPath) { + for (LibraryLoadingListener listener : this.listeners) { + try { + listener.onLoad(platformSpec, optionalComponent, libName, isPreloaded, optionalLibPath); + } catch (Throwable ignored) { + } + } + } + + @Override + public void onLoadFailure( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + Throwable optionalCause) { + for (LibraryLoadingListener listener : this.listeners) { + try { + listener.onLoadFailure(platformSpec, optionalComponent, libName, optionalCause); + } catch (Throwable ignored) { + } + } + } + + @Override + public void onTempFileCreated( + PlatformSpec platformSpec, String optionalComponent, String libName, Path tempFile) { + for (LibraryLoadingListener listener : this.listeners) { + try { + listener.onTempFileCreated(platformSpec, optionalComponent, libName, tempFile); + } catch (Throwable ignored) { + } + } + } + + @Override + public void onTempFileCreationFailure( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + Path tempDir, + String libExt, + Throwable optionalCause) { + for (LibraryLoadingListener listener : this.listeners) { + try { + listener.onTempFileCreationFailure( + platformSpec, optionalComponent, libName, tempDir, libExt, optionalCause); + } catch (Throwable ignored) { + } + } + } + + @Override + public void onTempFileCleanup( + PlatformSpec platformSpec, String optionalComponent, String libName, Path tempPath) { + for (LibraryLoadingListener listener : this.listeners) { + try { + listener.onTempFileCleanup(platformSpec, optionalComponent, libName, tempPath); + } catch (Throwable ignored) { + } + } + } + + @Override + public CompositeLibraryLoadingListener join(LibraryLoadingListener... listeners) { + ArrayList combinedListeners = + new ArrayList<>(this.listeners.size() + listeners.length); + combinedListeners.addAll(this.listeners); + combinedListeners.addAll(Arrays.asList(listeners)); + return new CompositeLibraryLoadingListener(combinedListeners); + } + + @Override + public String toString() { + return this.getClass().getSimpleName() + ":" + this.listeners.toString(); + } +} diff --git a/components/native-loader/src/main/java/datadog/nativeloader/FlatDirLibraryResolver.java b/components/native-loader/src/main/java/datadog/nativeloader/FlatDirLibraryResolver.java index 29b8f5e16c5..9730b8de7f5 100644 --- a/components/native-loader/src/main/java/datadog/nativeloader/FlatDirLibraryResolver.java +++ b/components/native-loader/src/main/java/datadog/nativeloader/FlatDirLibraryResolver.java @@ -13,7 +13,7 @@ private FlatDirLibraryResolver() {} @Override public final URL resolve( - PathLocator pathLocator, String component, PlatformSpec platformSpec, String libName) + PathLocator pathLocator, PlatformSpec platformSpec, String optionalComponent, String libName) throws Exception { PathLocatorHelper pathLocatorHelper = new PathLocatorHelper(libName, pathLocator); @@ -28,22 +28,22 @@ public final URL resolve( if (libcPath != null) { String specializedPath = regularPath + "-" + libcPath; - url = pathLocatorHelper.locate(component, specializedPath + "/" + libFileName); + url = pathLocatorHelper.locate(optionalComponent, specializedPath + "/" + libFileName); if (url != null) return url; } - url = pathLocatorHelper.locate(component, regularPath + "/" + libFileName); + url = pathLocatorHelper.locate(optionalComponent, regularPath + "/" + libFileName); if (url != null) return url; - url = pathLocatorHelper.locate(component, osPath + "/" + libFileName); + url = pathLocatorHelper.locate(optionalComponent, osPath + "/" + libFileName); if (url != null) return url; // fallback to searching at top-level, mostly concession to good out-of-box behavior // with java.library.path - url = pathLocatorHelper.locate(component, libFileName); + url = pathLocatorHelper.locate(optionalComponent, libFileName); if (url != null) return url; - if (component != null) { + if (optionalComponent != null) { url = pathLocatorHelper.locate(null, libFileName); if (url != null) return url; } diff --git a/components/native-loader/src/main/java/datadog/nativeloader/LibDirBasedPathLocator.java b/components/native-loader/src/main/java/datadog/nativeloader/LibDirBasedPathLocator.java index 5199ae0fc3a..2ed0528c161 100644 --- a/components/native-loader/src/main/java/datadog/nativeloader/LibDirBasedPathLocator.java +++ b/components/native-loader/src/main/java/datadog/nativeloader/LibDirBasedPathLocator.java @@ -14,8 +14,8 @@ public LibDirBasedPathLocator(File... libDirs) { } @Override - public URL locate(String component, String path) { - String fullPath = PathUtils.concatPath(component, path); + public URL locate(String optionalComponent, String path) { + String fullPath = PathUtils.concatPath(optionalComponent, path); for (File libDir : this.libDirs) { File libFile = new File(libDir, fullPath); diff --git a/components/native-loader/src/main/java/datadog/nativeloader/LibFile.java b/components/native-loader/src/main/java/datadog/nativeloader/LibFile.java index fb2e27d61d4..80a1957e5e8 100644 --- a/components/native-loader/src/main/java/datadog/nativeloader/LibFile.java +++ b/components/native-loader/src/main/java/datadog/nativeloader/LibFile.java @@ -16,66 +16,114 @@ public final class LibFile implements AutoCloseable { static final boolean NO_CLEAN_UP = false; static final boolean CLEAN_UP = true; - static final LibFile preloaded(String libName) { - return new LibFile(libName, null, NO_CLEAN_UP); + static final LibFile preloaded( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + SafeLibraryLoadingListener listeners) { + return new LibFile(platformSpec, optionalComponent, libName, null, NO_CLEAN_UP, listeners); } - static final LibFile fromFile(String libName, File file) { - return new LibFile(libName, file, NO_CLEAN_UP); + static final LibFile fromFile( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + File optionalFile, + SafeLibraryLoadingListener listeners) { + return new LibFile( + platformSpec, optionalComponent, libName, optionalFile, NO_CLEAN_UP, listeners); } - static final LibFile fromTempFile(String libName, File file) { - return new LibFile(libName, file, CLEAN_UP); + static final LibFile fromTempFile( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + File optionalFile, + SafeLibraryLoadingListener listeners) { + return new LibFile(platformSpec, optionalComponent, libName, optionalFile, CLEAN_UP, listeners); } + final PlatformSpec platformSpec; + final String optionalComponent; final String libName; - final File file; + final File optionalFile; final boolean needsCleanup; - LibFile(String libName, File file, boolean needsCleanup) { + final SafeLibraryLoadingListener listeners; + + LibFile( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + File optionalFile, + boolean needsCleanup, + SafeLibraryLoadingListener listeners) { + this.platformSpec = platformSpec; + this.optionalComponent = optionalComponent; this.libName = libName; - this.file = file; + this.optionalFile = optionalFile; this.needsCleanup = needsCleanup; + + this.listeners = listeners; } /** Indicates if this library was "preloaded" */ public boolean isPreloaded() { - return (this.file == null); + return (this.optionalFile == null); } /** Loads the underlying library into the JVM */ public void load() throws LibraryLoadException { - if (this.isPreloaded()) return; + boolean isPreloaded = this.isPreloaded(); + if (isPreloaded) { + this.listeners.onLoad( + this.platformSpec, this.optionalComponent, this.libName, isPreloaded, null); + return; + } try { Runtime.getRuntime().load(this.getAbsolutePath()); + + if (true) throw new RuntimeException("real load - worked?"); } catch (Throwable t) { + this.listeners.onLoadFailure(this.platformSpec, this.optionalComponent, this.libName, t); throw new LibraryLoadException(this.libName, t); } + + this.listeners.onLoad( + this.platformSpec, + this.optionalComponent, + this.libName, + isPreloaded, + this.optionalFile.toPath()); } /** Provides a File to the library -- returns null for pre-loaded libraries */ public final File toFile() { - return this.file; + return this.optionalFile; } /** Provides a Path to the library -- return null for pre-loaded libraries */ public final Path toPath() { - return this.file == null ? null : this.file.toPath(); + return this.optionalFile == null ? null : this.optionalFile.toPath(); } /** Provides the an absolute path to the library -- returns null for pre-loaded libraries */ public final String getAbsolutePath() { - return this.file == null ? null : this.file.getAbsolutePath(); + return this.optionalFile == null ? null : this.optionalFile.getAbsolutePath(); } - /** Schedules clean-up of underlying file -- if the file is a temp file */ + /** Schedules clean-up of underlying optionalFile -- if the file is a temp file */ @Override public void close() { if (this.needsCleanup) { - NativeLoader.delete(this.file); + boolean deleted = NativeLoader.delete(this.optionalFile); + if (deleted) { + this.listeners.onTempFileCleanup( + this.platformSpec, this.optionalComponent, this.libName, this.optionalFile.toPath()); + } } } } diff --git a/components/native-loader/src/main/java/datadog/nativeloader/LibraryLoadingListener.java b/components/native-loader/src/main/java/datadog/nativeloader/LibraryLoadingListener.java new file mode 100644 index 00000000000..8fa68a4feb0 --- /dev/null +++ b/components/native-loader/src/main/java/datadog/nativeloader/LibraryLoadingListener.java @@ -0,0 +1,78 @@ +package datadog.nativeloader; + +import java.net.URL; +import java.nio.file.Path; + +public interface LibraryLoadingListener { + /** + * Called when a dynamic library is resolved. This includes resolving a pre-loaded or already + * loaded library + * + *

If the library is pre-loaded optionalUrl will be null + */ + default void onResolveDynamic( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + boolean isPreloaded, + URL optionalUrl) {} + + /** + * Called when a dynamic library fails to resolve. This can occur because the library was not + * found -- or an exception occurred during resolution + */ + default void onResolveDynamicFailure( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + Throwable optionalCause) {} + + /** + * Called when a dynamic library loads successfully This includes loading a pre-loaded or already + * loaded library + */ + default void onLoad( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + boolean isPreloaded, + Path optionalLibPath) {} + + /** Called when a dynamic library fails to load */ + default void onLoadFailure( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + Throwable optionalCause) {} + + /** Called when a temp file is successfully created to hold the library */ + default void onTempFileCreated( + PlatformSpec platformSpec, String optionalComponent, String libName, Path tempFile) {} + + /** Called when a temp file could not be created */ + default void onTempFileCreationFailure( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + Path tempDir, + String libExt, + Throwable optionalCause) {} + + /** Called when a temp file is cleaned up */ + default void onTempFileCleanup( + PlatformSpec platformSpec, String optionalComponent, String libName, Path tempFile) {} +} + +/** + * "safe" listeners are used inside NativeLoader to avoid exceptions leaking out + * + *

The "safe" listeners are {@link CompositeLibraryLoadingListener} used to wrap regular + * listeners and {@link NopLibraryLoadingListener} used to optimize the nop case. + */ +abstract class SafeLibraryLoadingListener implements LibraryLoadingListener { + /** Used to create a new safe listener with the provided listeners append onto this one */ + public abstract SafeLibraryLoadingListener join(LibraryLoadingListener... listeners); + + /** Indicates if all listener operates are nops */ + public abstract boolean isNop(); +} diff --git a/components/native-loader/src/main/java/datadog/nativeloader/LibraryResolver.java b/components/native-loader/src/main/java/datadog/nativeloader/LibraryResolver.java index 4d28aae2794..66ab56c0169 100644 --- a/components/native-loader/src/main/java/datadog/nativeloader/LibraryResolver.java +++ b/components/native-loader/src/main/java/datadog/nativeloader/LibraryResolver.java @@ -15,6 +15,7 @@ default boolean isPreloaded(PlatformSpec platform, String libName) { return false; } - URL resolve(PathLocator pathLocator, String component, PlatformSpec platformSpec, String libName) + URL resolve( + PathLocator pathLocator, PlatformSpec platformSpec, String optionalComponent, String libName) throws Exception; } diff --git a/components/native-loader/src/main/java/datadog/nativeloader/LibraryResolvers.java b/components/native-loader/src/main/java/datadog/nativeloader/LibraryResolvers.java index 6ef8cec3c4b..bcbb434a3a1 100644 --- a/components/native-loader/src/main/java/datadog/nativeloader/LibraryResolvers.java +++ b/components/native-loader/src/main/java/datadog/nativeloader/LibraryResolvers.java @@ -27,9 +27,12 @@ public boolean isPreloaded(PlatformSpec platform, String libName) { @Override public URL resolve( - PathLocator pathLocator, String component, PlatformSpec platformSpec, String libName) + PathLocator pathLocator, + PlatformSpec platformSpec, + String optionalComponent, + String libName) throws Exception { - return baseResolver.resolve(pathLocator, component, platformSpec, libName); + return baseResolver.resolve(pathLocator, platformSpec, optionalComponent, libName); } }; } diff --git a/components/native-loader/src/main/java/datadog/nativeloader/NativeLoader.java b/components/native-loader/src/main/java/datadog/nativeloader/NativeLoader.java index 17af0b6f7ab..8a55eabe570 100644 --- a/components/native-loader/src/main/java/datadog/nativeloader/NativeLoader.java +++ b/components/native-loader/src/main/java/datadog/nativeloader/NativeLoader.java @@ -11,10 +11,13 @@ import java.nio.file.attribute.FileAttribute; import java.nio.file.attribute.PosixFilePermission; import java.nio.file.attribute.PosixFilePermissions; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; import java.util.Set; /** - * NativeLoader is intended as more feature rich replacement for calling {@link + * NativeLoader is intended as a more feature rich replacement for calling {@link * System#loadLibrary(String)} directly. NativeLoader can be used to find the corresponding platform * specific library using pluggable strategies -- for both path determination {@link * LibraryResolver} and path resolution {@link PathLocator} @@ -26,6 +29,7 @@ public static final class Builder { private String[] preloadedLibNames; private LibraryResolver libResolver; private PathLocator pathLocator; + private List listeners = new ArrayList<>(); Builder() {} @@ -138,6 +142,16 @@ public Builder tempDir(String tmpPath) { return this.tempDir(Paths.get(tmpPath)); } + public Builder addListener(LibraryLoadingListener listener) { + this.listeners.add(listener); + return this; + } + + public Builder addListeners(LibraryLoadingListener... listeners) { + this.listeners.addAll(Arrays.asList(listeners)); + return this; + } + /** Constructs and returns the {@link NativeLoader} */ public NativeLoader build() { return new NativeLoader(this); @@ -151,6 +165,12 @@ PathLocator pathLocator() { return (this.pathLocator == null) ? PathLocators.defaultPathLocator() : this.pathLocator; } + SafeLibraryLoadingListener listeners() { + return this.listeners.isEmpty() + ? NopLibraryLoadingListener.INSTANCE + : new CompositeLibraryLoadingListener(this.listeners); + } + LibraryResolver libResolver() { LibraryResolver baseResolver = (this.libResolver == null) ? LibraryResolvers.defaultLibraryResolver() : this.libResolver; @@ -169,18 +189,30 @@ public static final Builder builder() { return new Builder(); } + private static final URL NO_URL = null; + private static final Throwable NO_CAUSE = null; + private static final LibraryLoadingListener[] EMPTY_LISTENERS = {}; + private final PlatformSpec defaultPlatformSpec; private final LibraryResolver libResolver; private final PathLocator pathResolver; + private final SafeLibraryLoadingListener listeners; private final Path tempDir; private NativeLoader(Builder builder) { this.defaultPlatformSpec = builder.platformSpec(); this.libResolver = builder.libResolver(); this.pathResolver = builder.pathLocator(); + this.listeners = builder.listeners(); this.tempDir = builder.tempDir(); } + public boolean isPlatformSupported() { + if (this.defaultPlatformSpec.isUnknownOs()) return false; + if (this.defaultPlatformSpec.isUnknownArch()) return false; + return true; + } + /** Indicates if a library is considered "pre-loaded" */ public boolean isPreloaded(String libName) { return this.libResolver.isPreloaded(this.defaultPlatformSpec, libName); @@ -193,24 +225,47 @@ public boolean isPreloaded(PlatformSpec platformSpec, String libName) { /** Loads a library */ public void load(String libName) throws LibraryLoadException { - this.load(null, libName); + this.loadImpl(null, libName, EMPTY_LISTENERS); + } + + public void load(String libName, LibraryLoadingListener... scopedListeners) + throws LibraryLoadException { + this.loadImpl(null, libName, scopedListeners); } /** Loads a library associated with an associated component */ public void load(String component, String libName) throws LibraryLoadException { - try (LibFile libFile = this.resolveDynamic(component, libName)) { + this.loadImpl(component, libName, EMPTY_LISTENERS); + } + + public void load(String component, String libName, LibraryLoadingListener... scopedListeners) + throws LibraryLoadException { + this.loadImpl(component, libName, scopedListeners); + } + + private void loadImpl(String component, String libName, LibraryLoadingListener... scopedListeners) + throws LibraryLoadException { + + // scopedListeners are attached to the LibFile by resolveDynamicImpl + try (LibFile libFile = + this.resolveDynamicImpl(this.defaultPlatformSpec, component, libName, scopedListeners)) { libFile.load(); } } /** Resolves a library to a LibFile - creating a temporary file if necessary */ public LibFile resolveDynamic(String libName) throws LibraryLoadException { - return this.resolveDynamic((String) null, libName); + return this.resolveDynamicImpl(this.defaultPlatformSpec, null, libName, EMPTY_LISTENERS); + } + + public LibFile resolveDynamic(String libName, LibraryLoadingListener... scopedListeners) + throws LibraryLoadException { + return this.resolveDynamicImpl(this.defaultPlatformSpec, null, libName, scopedListeners); } /** Resolves a library with an associated component */ public LibFile resolveDynamic(String component, String libName) throws LibraryLoadException { - return this.resolveDynamic(component, this.defaultPlatformSpec, libName); + return this.resolveDynamicImpl(this.defaultPlatformSpec, component, libName, EMPTY_LISTENERS); } /** @@ -219,62 +274,104 @@ public LibFile resolveDynamic(String component, String libName) throws LibraryLo */ public LibFile resolveDynamic(PlatformSpec platformSpec, String libName) throws LibraryLoadException { - return this.resolveDynamic(null, platformSpec, libName); + return this.resolveDynamicImpl(platformSpec, null, libName, EMPTY_LISTENERS); + } + + public LibFile resolveDynamic( + PlatformSpec platformSpec, String libName, LibraryLoadingListener... scopedListeners) + throws LibraryLoadException { + return this.resolveDynamicImpl(platformSpec, null, libName, scopedListeners); } /** * Resolves a library with an associated component with a different {@link PlatformSpec} than the * default */ - public LibFile resolveDynamic(String component, PlatformSpec platformSpec, String libName) + public LibFile resolveDynamic(PlatformSpec platformSpec, String component, String libName) throws LibraryLoadException { + return this.resolveDynamicImpl(platformSpec, component, libName, EMPTY_LISTENERS); + } + + private LibFile resolveDynamicImpl( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + LibraryLoadingListener[] scopedListeners) + throws LibraryLoadException { + SafeLibraryLoadingListener allListeners = + (scopedListeners == null + || scopedListeners == EMPTY_LISTENERS + || scopedListeners.length == 0) + ? this.listeners + : this.listeners.join(scopedListeners); + if (platformSpec.isUnknownOs() || platformSpec.isUnknownArch()) { + allListeners.onResolveDynamicFailure(platformSpec, optionalComponent, libName, NO_CAUSE); throw new LibraryLoadException(libName, "Unsupported platform"); } - if (this.isPreloaded(platformSpec, libName)) { - return LibFile.preloaded(libName); + boolean isPreloaded = this.isPreloaded(platformSpec, libName); + if (isPreloaded) { + allListeners.onResolveDynamic(platformSpec, optionalComponent, libName, isPreloaded, NO_URL); + return LibFile.preloaded(platformSpec, optionalComponent, libName, allListeners); } URL url; try { - url = this.libResolver.resolve(this.pathResolver, component, platformSpec, libName); + url = this.libResolver.resolve(this.pathResolver, platformSpec, optionalComponent, libName); } catch (LibraryLoadException e) { // don't wrap if it is already a LibraryLoadException + allListeners.onResolveDynamicFailure(platformSpec, optionalComponent, libName, e.getCause()); throw e; } catch (Throwable t) { + allListeners.onResolveDynamicFailure(platformSpec, optionalComponent, libName, t); throw new LibraryLoadException(libName, t); } if (url == null) { + allListeners.onResolveDynamicFailure(platformSpec, optionalComponent, libName, NO_CAUSE); throw new LibraryLoadException(libName); } - return toLibFile(platformSpec, libName, url); - } - private LibFile toLibFile(PlatformSpec platformSpec, String libName, URL url) - throws LibraryLoadException { + // For listener purposes - at this point resolution completed successfully + // Although, the resolveDynamic method can still fail if we need a temp file and cannot create + // it + allListeners.onResolveDynamic(platformSpec, optionalComponent, libName, isPreloaded, url); + if (url.getProtocol().equals("file")) { - return LibFile.fromFile(libName, new File(url.getPath())); + return LibFile.fromFile( + platformSpec, optionalComponent, libName, new File(url.getPath()), allListeners); } else { - String libExt = PathUtils.dynamicLibExtension(platformSpec); - + Path tempFile; try { - Path tempFile = TempFileHelper.createTempFile(this.tempDir, libName, libExt); + tempFile = createTempFile(this.tempDir, platformSpec, libName, url); + allListeners.onTempFileCreated(platformSpec, optionalComponent, libName, tempFile); - try (InputStream in = url.openStream()) { - Files.copy(in, tempFile, StandardCopyOption.REPLACE_EXISTING); - } - - return LibFile.fromTempFile(libName, tempFile.toFile()); + return LibFile.fromTempFile( + platformSpec, optionalComponent, libName, tempFile.toFile(), allListeners); } catch (Throwable t) { + allListeners.onTempFileCreationFailure( + platformSpec, optionalComponent, libName, this.tempDir, libName, t); + throw new LibraryLoadException(libName, t); } } } - static void delete(File tempFile) { - TempFileHelper.delete(tempFile); + private static Path createTempFile( + Path tempDir, PlatformSpec platformSpec, String libName, URL url) throws IOException { + String libExt = PathUtils.dynamicLibExtension(platformSpec); + + Path tempFile = TempFileHelper.createTempFile(tempDir, libName, libExt); + try (InputStream in = url.openStream()) { + Files.copy(in, tempFile, StandardCopyOption.REPLACE_EXISTING); + } + + return tempFile; + } + + static boolean delete(File tempFile) { + return TempFileHelper.delete(tempFile); } static final class TempFileHelper { @@ -296,9 +393,10 @@ static Path createTempFile(Path tempDir, String libname, String libExt) } } - static void delete(File tempFile) { + static boolean delete(File tempFile) { boolean deleted = tempFile.delete(); if (!deleted) tempFile.deleteOnExit(); + return deleted; } } } diff --git a/components/native-loader/src/main/java/datadog/nativeloader/NestedDirLibraryResolver.java b/components/native-loader/src/main/java/datadog/nativeloader/NestedDirLibraryResolver.java index 966fd2c03e8..0137ca1d07e 100644 --- a/components/native-loader/src/main/java/datadog/nativeloader/NestedDirLibraryResolver.java +++ b/components/native-loader/src/main/java/datadog/nativeloader/NestedDirLibraryResolver.java @@ -11,7 +11,7 @@ final class NestedDirLibraryResolver implements LibraryResolver { @Override public final URL resolve( - PathLocator pathLocator, String component, PlatformSpec platformSpec, String libName) + PathLocator pathLocator, PlatformSpec platformSpec, String optionalComponent, String libName) throws Exception { PathLocatorHelper pathLocatorHelper = new PathLocatorHelper(libName, pathLocator); @@ -27,22 +27,22 @@ public final URL resolve( if (libcPath != null) { String specializedPath = regularPath + "/" + libcPath; - url = pathLocatorHelper.locate(component, specializedPath + "/" + libFileName); + url = pathLocatorHelper.locate(optionalComponent, specializedPath + "/" + libFileName); if (url != null) return url; } - url = pathLocatorHelper.locate(component, regularPath + "/" + libFileName); + url = pathLocatorHelper.locate(optionalComponent, regularPath + "/" + libFileName); if (url != null) return url; - url = pathLocatorHelper.locate(component, osPath + "/" + libFileName); + url = pathLocatorHelper.locate(optionalComponent, osPath + "/" + libFileName); if (url != null) return url; // fallback to searching at top-level, mostly concession to good out-of-box behavior // with java.library.path - url = pathLocatorHelper.locate(component, libFileName); + url = pathLocatorHelper.locate(optionalComponent, libFileName); if (url != null) return url; - if (component != null) { + if (optionalComponent != null) { url = pathLocatorHelper.locate(null, libFileName); if (url != null) return url; } diff --git a/components/native-loader/src/main/java/datadog/nativeloader/NopLibraryLoadingListener.java b/components/native-loader/src/main/java/datadog/nativeloader/NopLibraryLoadingListener.java new file mode 100644 index 00000000000..afec8dc1b1f --- /dev/null +++ b/components/native-loader/src/main/java/datadog/nativeloader/NopLibraryLoadingListener.java @@ -0,0 +1,19 @@ +package datadog.nativeloader; + +import java.util.Arrays; + +final class NopLibraryLoadingListener extends SafeLibraryLoadingListener { + static final NopLibraryLoadingListener INSTANCE = new NopLibraryLoadingListener(); + + private NopLibraryLoadingListener() {} + + @Override + public boolean isNop() { + return true; + } + + @Override + public SafeLibraryLoadingListener join(LibraryLoadingListener... listeners) { + return new CompositeLibraryLoadingListener(Arrays.asList(listeners)); + } +} diff --git a/components/native-loader/src/main/java/datadog/nativeloader/PathLocator.java b/components/native-loader/src/main/java/datadog/nativeloader/PathLocator.java index e7cc2e14f20..aae3c2e0b5a 100644 --- a/components/native-loader/src/main/java/datadog/nativeloader/PathLocator.java +++ b/components/native-loader/src/main/java/datadog/nativeloader/PathLocator.java @@ -14,5 +14,5 @@ public interface PathLocator { *

If the returned URL uses a non-file protocol, then {@link NativeLoader} will call {@link * URL#openStream()} and copy the contents to a temporary file */ - URL locate(String component, String path) throws Exception; + URL locate(String optionalComponent, String path) throws Exception; } diff --git a/components/native-loader/src/main/java/datadog/nativeloader/PathLocatorHelper.java b/components/native-loader/src/main/java/datadog/nativeloader/PathLocatorHelper.java index 38c976c5c86..8842d642afb 100644 --- a/components/native-loader/src/main/java/datadog/nativeloader/PathLocatorHelper.java +++ b/components/native-loader/src/main/java/datadog/nativeloader/PathLocatorHelper.java @@ -18,9 +18,9 @@ public PathLocatorHelper(String libName, PathLocator locator) { } @Override - public URL locate(String component, String path) { + public URL locate(String optionalComponent, String path) { try { - return this.locator.locate(component, path); + return this.locator.locate(optionalComponent, path); } catch (Throwable t) { if (this.firstCause == null) this.firstCause = t; return null; diff --git a/components/native-loader/src/test/java/datadog/nativeloader/CapturingPathLocator.java b/components/native-loader/src/test/java/datadog/nativeloader/CapturingPathLocator.java index 257e5ba9958..b002b2c144d 100644 --- a/components/native-loader/src/test/java/datadog/nativeloader/CapturingPathLocator.java +++ b/components/native-loader/src/test/java/datadog/nativeloader/CapturingPathLocator.java @@ -32,7 +32,7 @@ public static final void test( String comp = "comp"; CapturingPathLocator fullCaptureLocator = new CapturingPathLocator(Integer.MAX_VALUE); - resolver.resolve(fullCaptureLocator, comp, platformSpec, "test"); + resolver.resolve(fullCaptureLocator, platformSpec, comp, "test"); for (int i = 0; !fullCaptureLocator.isEmpty(); ++i) { if (i >= expectedPaths.length) { @@ -52,7 +52,7 @@ public static final void test( for (int i = 0; i < expectedPaths.length; ++i) { CapturingPathLocator fallbackLocator = new CapturingPathLocator(i); - resolver.resolve(fallbackLocator, comp, platformSpec, "test"); + resolver.resolve(fallbackLocator, platformSpec, comp, "test"); for (int j = 0; j <= i; ++j) { fallbackLocator.assertRequested(comp, expectedPaths[j]); @@ -62,7 +62,7 @@ public static final void test( if (withSkipCompFallback) { CapturingPathLocator fallbackLocator = new CapturingPathLocator(expectedPaths.length); - resolver.resolve(fallbackLocator, comp, platformSpec, "test"); + resolver.resolve(fallbackLocator, platformSpec, comp, "test"); for (int j = 0; j < expectedPaths.length; ++j) { fallbackLocator.assertRequested(comp, expectedPaths[j]); @@ -87,8 +87,8 @@ public CapturingPathLocator(int simulateNotFoundCount) { } @Override - public URL locate(String component, String path) { - this.locateRequests.addLast(new LocateRequest(component, path)); + public URL locate(String optionalComponent, String path) { + this.locateRequests.addLast(new LocateRequest(optionalComponent, path)); if (this.numRequests++ < this.simulateNotFoundCount) return null; try { diff --git a/components/native-loader/src/test/java/datadog/nativeloader/CompositeLibraryLoadingListenerTest.java b/components/native-loader/src/test/java/datadog/nativeloader/CompositeLibraryLoadingListenerTest.java new file mode 100644 index 00000000000..019f09c2138 --- /dev/null +++ b/components/native-loader/src/test/java/datadog/nativeloader/CompositeLibraryLoadingListenerTest.java @@ -0,0 +1,176 @@ +package datadog.nativeloader; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; + +import java.io.IOException; +import java.net.MalformedURLException; +import java.net.URL; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import org.junit.jupiter.api.Test; + +public final class CompositeLibraryLoadingListenerTest { + @Test + public void onResolveDynamic() throws MalformedURLException { + TestLibraryLoadingListener listener1 = + new TestLibraryLoadingListener().expectResolveDynamic("foo"); + + TestLibraryLoadingListener listener2 = listener1.copy(); + + listeners(listener1, listener2) + .onResolveDynamic( + PlatformSpec.defaultPlatformSpec(), null, "foo", false, new URL("http://localhost")); + + listener1.assertDone(); + listener2.assertDone(); + } + + @Test + public void onResolveDynamicFailure() { + TestLibraryLoadingListener listener1 = + new TestLibraryLoadingListener().expectResolveDynamicFailure("foo"); + + TestLibraryLoadingListener listener2 = listener1.copy(); + + listeners(listener1, listener2) + .onResolveDynamicFailure( + PlatformSpec.defaultPlatformSpec(), null, "foo", new Exception("foo")); + + listener1.assertDone(); + listener2.assertDone(); + } + + @Test + public void onLoad() { + TestLibraryLoadingListener listener1 = new TestLibraryLoadingListener().expectLoad("foo"); + + TestLibraryLoadingListener listener2 = listener1.copy(); + + listeners(listener1, listener2) + .onLoad(PlatformSpec.defaultPlatformSpec(), null, "foo", false, null); + + listener1.assertDone(); + listener2.assertDone(); + } + + @Test + public void onLoadFailure() { + TestLibraryLoadingListener listener1 = + new TestLibraryLoadingListener().expectLoadFailure("foo"); + + TestLibraryLoadingListener listener2 = listener1.copy(); + + listeners(listener1, listener2) + .onLoadFailure(PlatformSpec.defaultPlatformSpec(), null, "foo", null); + + listener1.assertDone(); + listener2.assertDone(); + } + + @Test + public void onTempFileCreated() { + TestLibraryLoadingListener listener1 = + new TestLibraryLoadingListener().expectTempFileCreated("foo"); + + TestLibraryLoadingListener listener2 = listener1.copy(); + + listeners(listener1, listener2) + .onTempFileCreated( + PlatformSpec.defaultPlatformSpec(), null, "foo", Paths.get("/tmp/foo.dll")); + + listener1.assertDone(); + listener2.assertDone(); + } + + @Test + public void onTempFileCreationFailure() { + TestLibraryLoadingListener listener1 = + new TestLibraryLoadingListener().expectTempFileCreationFailure("foo"); + + TestLibraryLoadingListener listener2 = listener1.copy(); + + listeners(listener1, listener2) + .onTempFileCreationFailure( + PlatformSpec.defaultPlatformSpec(), + null, + "foo", + Paths.get("/tmp"), + "dylib", + new IOException("perm")); + + listener1.assertDone(); + listener2.assertDone(); + } + + @Test + public void onTempFileCleanup() { + TestLibraryLoadingListener listener1 = + new TestLibraryLoadingListener().expectTempFileCleanup("foo"); + + TestLibraryLoadingListener listener2 = listener1.copy(); + + listeners(listener1, listener2) + .onTempFileCleanup( + PlatformSpec.defaultPlatformSpec(), null, "foo", Paths.get("/tmp/foo.dll")); + + listener1.assertDone(); + listener2.assertDone(); + } + + @Test + public void join() { + TestLibraryLoadingListener listener1 = new TestLibraryLoadingListener().expectLoad("foo"); + + CompositeLibraryLoadingListener composite = new CompositeLibraryLoadingListener(listener1); + assertEquals(1, composite.size()); + + TestLibraryLoadingListener listener2 = listener1.copy(); + + CompositeLibraryLoadingListener composite2 = composite.join(listener2); + assertEquals(2, composite2.size()); + + TestLibraryLoadingListener listener3 = listener1.copy(); + TestLibraryLoadingListener listener4 = listener1.copy(); + + CompositeLibraryLoadingListener finalComposite = composite2.join(listener3, listener4); + assertEquals(4, finalComposite.size()); + + finalComposite.onLoad( + PlatformSpec.defaultPlatformSpec(), null, "foo", false, Paths.get("/tmp/foo.dll")); + + listener1.assertDone(); + listener2.assertDone(); + listener3.assertDone(); + listener4.assertDone(); + } + + @Test + public void _toString() { + // just for coverage + assertNotNull(new CompositeLibraryLoadingListener().toString()); + } + + /* + * Constructs a composite listener that includes the provided listeners + * To test robustness... + * - adds in additional failing listeners + * - adds in additional default listeners + * - shuffles the order of the listeners + */ + static CompositeLibraryLoadingListener listeners(LibraryLoadingListener... listeners) { + List shuffledListeners = new ArrayList<>(listeners.length * 3); + shuffledListeners.addAll(Arrays.asList(listeners)); + + for (int i = 0; i < listeners.length; ++i) { + shuffledListeners.add(new LibraryLoadingListener() {}); + shuffledListeners.add(new ThrowingLibraryLoadingListener()); + } + + Collections.shuffle(shuffledListeners); + return new CompositeLibraryLoadingListener(shuffledListeners); + } +} diff --git a/components/native-loader/src/test/java/datadog/nativeloader/NativeLoaderBuilderTest.java b/components/native-loader/src/test/java/datadog/nativeloader/NativeLoaderBuilderTest.java index db04ea859e1..db5a25b4541 100644 --- a/components/native-loader/src/test/java/datadog/nativeloader/NativeLoaderBuilderTest.java +++ b/components/native-loader/src/test/java/datadog/nativeloader/NativeLoaderBuilderTest.java @@ -26,6 +26,46 @@ public void customPlatformSpec() { assertSame(platformSpec, builder.platformSpec()); } + @Test + public void defaultListeners() { + NativeLoader.Builder builder = NativeLoader.builder(); + + assertTrue(builder.listeners().isNop()); + } + + @Test + public void addListener() { + TestLibraryLoadingListener listener1 = new TestLibraryLoadingListener().expectLoad("foo"); + TestLibraryLoadingListener listener2 = listener1.copy(); + + NativeLoader.Builder builder = + NativeLoader.builder().addListener(listener1).addListener(listener2); + + SafeLibraryLoadingListener listener = builder.listeners(); + assertFalse(listener.isNop()); + + listener.onLoad(builder.platformSpec(), null, "foo", false, Paths.get("/tmp/foo.dylib")); + + listener1.assertDone(); + listener2.assertDone(); + } + + @Test + public void addListeners() { + TestLibraryLoadingListener listener1 = new TestLibraryLoadingListener().expectLoad("foo"); + TestLibraryLoadingListener listener2 = listener1.copy(); + + NativeLoader.Builder builder = NativeLoader.builder().addListeners(listener1, listener2); + + SafeLibraryLoadingListener listener = builder.listeners(); + assertFalse(listener.isNop()); + + listener.onLoad(builder.platformSpec(), null, "foo", false, Paths.get("/tmp/foo.dylib")); + + listener1.assertDone(); + listener2.assertDone(); + } + @Test public void defaultLibraryResolver() { NativeLoader.Builder builder = NativeLoader.builder(); diff --git a/components/native-loader/src/test/java/datadog/nativeloader/NativeLoaderTest.java b/components/native-loader/src/test/java/datadog/nativeloader/NativeLoaderTest.java index a122e88bea4..0a44b91f912 100644 --- a/components/native-loader/src/test/java/datadog/nativeloader/NativeLoaderTest.java +++ b/components/native-loader/src/test/java/datadog/nativeloader/NativeLoaderTest.java @@ -9,6 +9,7 @@ import static org.junit.jupiter.api.Assertions.assertNull; import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assumptions.assumeTrue; import java.io.File; import java.io.IOException; @@ -30,14 +31,14 @@ public class NativeLoaderTest { @Test public void preloaded() throws LibraryLoadException { - NativeLoader loader = NativeLoader.builder().preloaded("dne1", "dne2").build(); + NativeLoader loader = NativeLoader.builder().preloaded("preloaded1", "preloaded2").build(); - assertTrue(loader.isPreloaded("dne1")); - assertTrue(loader.isPreloaded("dne2")); + assertTrue(loader.isPreloaded("preloaded1")); + assertTrue(loader.isPreloaded("preloaded2")); - assertFalse(loader.isPreloaded("dne3")); + assertFalse(loader.isPreloaded("dne")); - try (LibFile lib = loader.resolveDynamic("dne1")) { + try (LibFile lib = loader.resolveDynamic("preloaded1")) { assertPreloaded(lib); // already considered loaded -- so this is a nop @@ -45,18 +46,83 @@ public void preloaded() throws LibraryLoadException { } // already considered loaded -- so this is a nop - loader.load("dne2"); + loader.load("preloaded2"); // not already loaded - so passes through to underlying resolver - assertThrows(LibraryLoadException.class, () -> loader.load("dne3")); + assertThrows(LibraryLoadException.class, () -> loader.load("dne")); + } + + @Test + public void preloaded_listenerSupport() throws LibraryLoadException { + TestLibraryLoadingListener sharedListener = new TestLibraryLoadingListener(); + + NativeLoader loader = + NativeLoader.builder() + .preloaded("preloaded1", "preloaded2") + .addListener(sharedListener) + .build(); + + // debatable - but no listener calls just for checking + assertTrue(loader.isPreloaded("preloaded1")); + assertTrue(loader.isPreloaded("preloaded2")); + + sharedListener.expectResolvePreloaded("preloaded1"); + sharedListener.expectLoadPreloaded("preloaded1"); + + TestLibraryLoadingListener scopedListener1 = + new TestLibraryLoadingListener() + .expectResolvePreloaded("preloaded1") + .expectLoadPreloaded("preloaded1"); + + try (LibFile lib = loader.resolveDynamic("preloaded1", scopedListener1)) { + lib.load(); + } + + sharedListener.assertDone(); + scopedListener1.assertDone(); + + sharedListener.expectResolvePreloaded("preloaded2"); + sharedListener.expectLoadPreloaded("preloaded2"); + + TestLibraryLoadingListener scopedListener2 = + new TestLibraryLoadingListener() + .expectResolvePreloaded("preloaded2") + .expectLoadPreloaded("preloaded2"); + + // load is just convenience for resolve & load + loader.load("preloaded2", scopedListener2); + + sharedListener.assertDone(); + scopedListener2.assertDone(); + + sharedListener.expectResolveDynamicFailure("dne"); + + TestLibraryLoadingListener scopedListener3 = + new TestLibraryLoadingListener().expectResolveDynamicFailure("dne"); + + // not already loaded - so passes through to underlying resolver + assertThrows(LibraryLoadException.class, () -> loader.load("dne", scopedListener3)); + + sharedListener.assertDone(); + scopedListener3.assertDone(); } @Test public void unsupportedPlatform() { + TestLibraryLoadingListener sharedListener = new TestLibraryLoadingListener(); + PlatformSpec unsupportedOsSpec = TestPlatformSpec.of(UNSUPPORTED_OS, AARCH64); - NativeLoader loader = NativeLoader.builder().platformSpec(unsupportedOsSpec).build(); + NativeLoader loader = + NativeLoader.builder().platformSpec(unsupportedOsSpec).addListener(sharedListener).build(); + + assertFalse(loader.isPlatformSupported()); + sharedListener.expectResolveDynamicFailure("dummy"); + + // short-circuit fails during resolution because os isn't supported assertThrows(LibraryLoadException.class, () -> loader.resolveDynamic("dummy")); + + sharedListener.assertDone(); } @Test @@ -64,17 +130,79 @@ public void unsupportArch() { PlatformSpec unsupportedOsSpec = TestPlatformSpec.of(LINUX, UNSUPPORTED_ARCH); NativeLoader loader = NativeLoader.builder().platformSpec(unsupportedOsSpec).build(); - assertThrows(LibraryLoadException.class, () -> loader.resolveDynamic("dummy")); + assertFalse(loader.isPlatformSupported()); + + TestLibraryLoadingListener scopedListener = + new TestLibraryLoadingListener().expectResolveDynamicFailure("dummy"); + + // short-circuit fails during resolution because arch isn't supported + assertThrows(LibraryLoadException.class, () -> loader.resolveDynamic("dummy", scopedListener)); + + scopedListener.assertDone(); } @Test public void loadFailure() throws LibraryLoadException { - NativeLoader loader = NativeLoader.builder().build(); + TestLibraryLoadingListener sharedListener = new TestLibraryLoadingListener(); + + NativeLoader loader = NativeLoader.builder().addListener(sharedListener).build(); + assumeTrue(loader.isPlatformSupported()); + + sharedListener.expectResolveDynamic("dummy"); + sharedListener.expectLoadFailure("dummy"); + + TestLibraryLoadingListener scopedListener = + new TestLibraryLoadingListener().expectResolveDynamic("dummy").expectLoadFailure("dummy"); // test libraries are just text files, so they shouldn't load & link properly // NativeLoader is supposed to wrap the loading failures, so that we // remember to handle them + + // on supported platforms, there is a dummy library file, so this will resolve but fail to load + // & link + assertThrows(LibraryLoadException.class, () -> loader.load("dummy", scopedListener)); + } + + @Test + public void resolutionFailure_in_LibraryResolver() { + Exception exception = new Exception("boom!"); + + NativeLoader loader = + NativeLoader.builder() + .libResolver( + (pathLocator, platformSpec, component, libName) -> { + throw exception; + }) + .build(); + + TestLibraryLoadingListener scopedListener = + new TestLibraryLoadingListener().expectResolveDynamicFailure("dummy", exception); + + assertThrows(LibraryLoadException.class, () -> loader.load("dummy", scopedListener)); + + scopedListener.assertDone(); + } + + @Test + public void resolutionFailure_in_PathLocator() { + TestLibraryLoadingListener sharedListener = new TestLibraryLoadingListener(); + + Exception exception = new Exception("boom!"); + + NativeLoader loader = + NativeLoader.builder() + .addListener(sharedListener) + .pathLocator( + (comp, path) -> { + throw exception; + }) + .build(); + + sharedListener.expectResolveDynamicFailure("dummy", exception); + assertThrows(LibraryLoadException.class, () -> loader.load("dummy")); + + sharedListener.assertDone(); } @Test @@ -92,24 +220,36 @@ public void fromDir() throws LibraryLoadException { @Test public void fromDir_override_windows() throws LibraryLoadException { - NativeLoader loader = NativeLoader.builder().fromDir("test-data").build(); + TestLibraryLoadingListener sharedListener = new TestLibraryLoadingListener(); + + NativeLoader loader = + NativeLoader.builder().fromDir("test-data").addListener(sharedListener).build(); + + sharedListener.expectResolveDynamic(TestPlatformSpec.windows(), "dummy"); try (LibFile lib = loader.resolveDynamic(TestPlatformSpec.windows(), "dummy")) { // loaded directly from directory, so no clean-up required assertRegularFile(lib); assertTrue(lib.getAbsolutePath().endsWith("dummy.dll")); } + + sharedListener.assertDone(); } @Test public void fromDir_override_mac() throws LibraryLoadException { NativeLoader loader = NativeLoader.builder().fromDir("test-data").build(); - try (LibFile lib = loader.resolveDynamic(TestPlatformSpec.mac(), "dummy")) { + TestLibraryLoadingListener scopedListener = + new TestLibraryLoadingListener().expectResolveDynamic(TestPlatformSpec.mac(), "dummy"); + + try (LibFile lib = loader.resolveDynamic(TestPlatformSpec.mac(), "dummy", scopedListener)) { // loaded directly from directory, so no clean-up required assertRegularFile(lib); assertTrue(lib.getAbsolutePath().endsWith("libdummy.dylib")); } + + scopedListener.assertDone(); } @Test @@ -135,17 +275,45 @@ public void fromDirList() throws LibraryLoadException { @Test public void fromDir_with_component() throws LibraryLoadException { - NativeLoader loader = NativeLoader.builder().fromDir("test-data").build(); + TestLibraryLoadingListener sharedListener = new TestLibraryLoadingListener(); + + NativeLoader loader = + NativeLoader.builder().fromDir("test-data").addListener(sharedListener).build(); + + sharedListener.expectResolveDynamic("comp1", "dummy"); try (LibFile lib = loader.resolveDynamic("comp1", "dummy")) { assertRegularFile(lib); assertTrue(lib.getAbsolutePath().contains("comp1")); } + sharedListener.assertDone(); + + sharedListener.expectResolveDynamic("comp2", "dummy"); + try (LibFile lib = loader.resolveDynamic("comp2", "dummy")) { assertRegularFile(lib); assertTrue(lib.getAbsolutePath().contains("comp2")); } + + sharedListener.assertDone(); + } + + @Test + public void fromDir_load_with_component() { + NativeLoader loader = NativeLoader.builder().fromDir("test-data").build(); + + // lib file is a dummy, so fails during loading and linking + assertThrows(LibraryLoadException.class, () -> loader.load("comp1", "dummy")); + + TestLibraryLoadingListener scopedListener2 = + new TestLibraryLoadingListener() + .expectResolveDynamic("comp2", "dummy") + .expectLoadFailure("comp2", "dummy"); + + assertThrows(LibraryLoadException.class, () -> loader.load("comp2", "dummy", scopedListener2)); + + scopedListener2.assertDone(); } @Test @@ -218,10 +386,47 @@ public void fromJarBackedClassLoader() throws IOException, LibraryLoadException try { try (URLClassLoader classLoader = createClassLoader(jar)) { NativeLoader loader = NativeLoader.builder().fromClassLoader(classLoader).build(); - try (LibFile lib = loader.resolveDynamic("dummy")) { + + TestLibraryLoadingListener scopedListener = + new TestLibraryLoadingListener() + .expectResolveDynamic("dummy") + .expectTempFileCreated("dummy") + .expectTempFileCleanup("dummy"); + + try (LibFile lib = loader.resolveDynamic("dummy", scopedListener)) { // loaded from a jar, so copied to temp file assertTempFile(lib); } + + scopedListener.assertDone(); + } + } finally { + deleteHelper(jar); + } + } + + @Test + public void fromJarBackedClassLoader_load_with_component() + throws IOException, LibraryLoadException { + Path jar = jar("test-data"); + try { + try (URLClassLoader classLoader = createClassLoader(jar)) { + NativeLoader loader = NativeLoader.builder().fromClassLoader(classLoader).build(); + + // lib file is a dummy, so fails during loading and linking + assertThrows(LibraryLoadException.class, () -> loader.load("comp1", "dummy")); + + TestLibraryLoadingListener scopedListener2 = + new TestLibraryLoadingListener() + .expectResolveDynamic("comp2", "dummy") + .expectTempFileCreated("comp2", "dummy") + .expectLoadFailure("comp2", "dummy") + .expectTempFileCleanup("comp2", "dummy"); + + assertThrows( + LibraryLoadException.class, () -> loader.load("comp2", "dummy", scopedListener2)); + + scopedListener2.assertDone(); } } finally { deleteHelper(jar); @@ -263,8 +468,16 @@ public void fromJarBackedClassLoader_with_unwritable_tempDir() NativeLoader loader = NativeLoader.builder().fromClassLoader(classLoader).tempDir(noWriteDir).build(); + TestLibraryLoadingListener scopedListener = + new TestLibraryLoadingListener() + .expectResolveDynamic("dummy") + .expectTempFileCreationFailure("dummy"); + // unable to resolve to a File because tempDir isn't writable - assertThrows(LibraryLoadException.class, () -> loader.resolveDynamic("dummy")); + assertThrows( + LibraryLoadException.class, () -> loader.resolveDynamic("dummy", scopedListener)); + + scopedListener.assertDone(); } finally { deleteHelper(noWriteDir); } diff --git a/components/native-loader/src/test/java/datadog/nativeloader/TestLibraryLoadingListener.java b/components/native-loader/src/test/java/datadog/nativeloader/TestLibraryLoadingListener.java new file mode 100644 index 00000000000..65af2e522a0 --- /dev/null +++ b/components/native-loader/src/test/java/datadog/nativeloader/TestLibraryLoadingListener.java @@ -0,0 +1,535 @@ +package datadog.nativeloader; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assertions.fail; + +import java.net.URL; +import java.nio.file.Path; +import java.util.Collections; +import java.util.LinkedList; + +public final class TestLibraryLoadingListener implements LibraryLoadingListener { + private final LinkedList checks; + + private Check failedCheck = null; + private Throwable failedCause = null; + + // By design, listeners are supposed to receive the underlying cause not a LibraryLoadException + // directly + static final ThrowableCheck NOT_LIB_LOAD_EXCEPTION = + (t) -> { + assertFalse( + t instanceof LibraryLoadException, + "LibraryLoadException - instead of underlying cause"); + }; + + public TestLibraryLoadingListener() { + this.checks = new LinkedList<>(); + } + + private TestLibraryLoadingListener(TestLibraryLoadingListener that) { + this.checks = new LinkedList<>(that.checks); + } + + public TestLibraryLoadingListener expectResolveDynamic(String expectedLibName) { + return this.expectResolveDynamic(new LibCheck(expectedLibName)); + } + + public TestLibraryLoadingListener expectResolveDynamic( + String expectedComponent, String expectedLibName) { + return this.expectResolveDynamic(new LibCheck(expectedComponent, expectedLibName)); + } + + public TestLibraryLoadingListener expectResolveDynamic( + PlatformSpec expectedPlatformSpec, String expectedLibName) { + return this.expectResolveDynamic(new LibCheck(expectedPlatformSpec, expectedLibName)); + } + + private TestLibraryLoadingListener expectResolveDynamic(LibCheck libCheck) { + return this.addCheck( + new Check("onResolveDynamic %s", libCheck) { + @Override + public void onResolveDynamic( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + boolean isPreloaded, + URL optionalUrl) { + libCheck.assertMatches(platformSpec, optionalComponent, libName); + } + }); + } + + public TestLibraryLoadingListener expectResolvePreloaded(String expectedLibName) { + return this.expectResolvePreloaded(new LibCheck(expectedLibName)); + } + + private TestLibraryLoadingListener expectResolvePreloaded(LibCheck libCheck) { + return this.addCheck( + new Check("onResolveDynamic:preloaded %s", libCheck) { + @Override + public void onResolveDynamic( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + boolean isPreloaded, + URL optionalUrl) { + libCheck.assertMatches(platformSpec, optionalComponent, libName); + assertTrue(isPreloaded); + } + }); + } + + public TestLibraryLoadingListener expectResolveDynamicFailure(String expectedLibName) { + return this.expectResolveDynamicFailure(new LibCheck(expectedLibName)); + } + + public TestLibraryLoadingListener expectResolveDynamicFailure( + String expectedLibName, Throwable expectedThrowable) { + return this.expectResolveDynamicFailure( + new LibCheck(expectedLibName), (t) -> assertSame(expectedThrowable, t)); + } + + private TestLibraryLoadingListener expectResolveDynamicFailure(LibCheck libCheck) { + return this.expectResolveDynamicFailure(libCheck, NOT_LIB_LOAD_EXCEPTION); + } + + private TestLibraryLoadingListener expectResolveDynamicFailure( + LibCheck libCheck, ThrowableCheck throwableCheck) { + return this.addCheck( + new Check("onResolveDynamicFailure %s", libCheck) { + @Override + public void onResolveDynamicFailure( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + Throwable optionalCause) { + libCheck.assertMatches(platformSpec, optionalComponent, libName); + throwableCheck.assertMatches(optionalCause); + } + }); + } + + public TestLibraryLoadingListener expectLoad(String expectedLibName) { + return this.expectLoad(new LibCheck(expectedLibName)); + } + + public TestLibraryLoadingListener expectLoad(String expectedComponent, String expectedLibName) { + return this.expectLoad(new LibCheck(expectedComponent, expectedLibName)); + } + + private TestLibraryLoadingListener expectLoad(LibCheck libCheck) { + return this.addCheck( + new Check("onLoad %s", libCheck) { + @Override + public void onLoad( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + boolean isPreloaded, + Path optionalLibPath) { + libCheck.assertMatches(platformSpec, optionalComponent, libName); + } + }); + } + + public TestLibraryLoadingListener expectLoadPreloaded(String expectedLibName) { + return this.expectLoadPreloaded(new LibCheck(expectedLibName)); + } + + private TestLibraryLoadingListener expectLoadPreloaded(LibCheck libCheck) { + return this.addCheck( + new Check("onLoad:preloaded %s", libCheck) { + @Override + public void onLoad( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + boolean isPreloaded, + Path optionalLibPath) { + libCheck.assertMatches(platformSpec, optionalComponent, libName); + assertTrue(isPreloaded); + } + }); + } + + public TestLibraryLoadingListener expectLoadFailure(String expectedLibName) { + return this.expectLoadFailure(new LibCheck(expectedLibName)); + } + + public TestLibraryLoadingListener expectLoadFailure( + String expectedComponent, String expectedLibName) { + return this.expectLoadFailure(new LibCheck(expectedComponent, expectedLibName)); + } + + private TestLibraryLoadingListener expectLoadFailure(LibCheck libCheck) { + return this.expectLoadFailure(libCheck, NOT_LIB_LOAD_EXCEPTION); + } + + private TestLibraryLoadingListener expectLoadFailure( + LibCheck libCheck, ThrowableCheck throwableCheck) { + return this.addCheck( + new Check("onLoadFailure %s", libCheck) { + @Override + public void onLoadFailure( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + Throwable optionalCause) { + libCheck.assertMatches(platformSpec, optionalComponent, libName); + throwableCheck.assertMatches(optionalCause); + } + }); + } + + public TestLibraryLoadingListener expectTempFileCreated(String expectedLibName) { + return this.expectTempFileCreated(new LibCheck(expectedLibName)); + } + + public TestLibraryLoadingListener expectTempFileCreated( + String expectedComponent, String expectedLibName) { + return this.expectTempFileCreated(new LibCheck(expectedComponent, expectedLibName)); + } + + private TestLibraryLoadingListener expectTempFileCreated(LibCheck libCheck) { + return this.addCheck( + new Check("onTempFileCreated %s", libCheck) { + @Override + public void onTempFileCreated( + PlatformSpec platformSpec, String optionalComponent, String libName, Path tempFile) { + libCheck.assertMatches(platformSpec, optionalComponent, libName); + assertNotNull(tempFile); + } + }); + } + + public TestLibraryLoadingListener expectTempFileCreationFailure(String expectedLibName) { + return this.expectTempFileCreationFailure(new LibCheck(expectedLibName)); + } + + private TestLibraryLoadingListener expectTempFileCreationFailure(LibCheck libCheck) { + return this.expectTempFileCreationFailure(libCheck, NOT_LIB_LOAD_EXCEPTION); + } + + private TestLibraryLoadingListener expectTempFileCreationFailure( + LibCheck libCheck, ThrowableCheck throwableCheck) { + return this.addCheck( + new Check("onTempFileCreationFailure %s", libCheck) { + @Override + public void onTempFileCreationFailure( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + Path tempDir, + String libExt, + Throwable optionalCause) { + libCheck.assertMatches(platformSpec, optionalComponent, libName); + assertNotNull(tempDir); + assertNotNull(libExt); + } + }); + } + + public TestLibraryLoadingListener expectTempFileCleanup(String expectedLibName) { + return this.expectTempFileCleanup(new LibCheck(expectedLibName)); + } + + public TestLibraryLoadingListener expectTempFileCleanup( + String expectedComponent, String expectedLibName) { + return this.expectTempFileCleanup(new LibCheck(expectedComponent, expectedLibName)); + } + + TestLibraryLoadingListener expectTempFileCleanup(LibCheck libCheck) { + return this.addCheck( + new Check("onTempFileCreationCleanup %s", libCheck) { + @Override + public void onTempFileCleanup( + PlatformSpec platformSpec, String optionalComponent, String libName, Path tempFile) { + libCheck.assertMatches(platformSpec, optionalComponent, libName); + assertNotNull(tempFile); + } + }); + } + + public TestLibraryLoadingListener copy() { + return new TestLibraryLoadingListener(this); + } + + public void assertDone() { + if (this.failedCheck != null) { + try { + fail("check failed: " + this.failedCheck, this.failedCause); + } catch (AssertionError e) { + e.initCause(this.failedCause); + + throw e; + } + } + + // written this way for better debugging + assertEquals(Collections.emptyList(), this.checks); + } + + @Override + public void onResolveDynamic( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + boolean isPreloaded, + URL optionalUrl) { + this.nextCheck( + check -> + check.onResolveDynamic( + platformSpec, optionalComponent, libName, isPreloaded, optionalUrl)); + } + + @Override + public void onResolveDynamicFailure( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + Throwable optionalCause) { + this.nextCheck( + check -> + check.onResolveDynamicFailure(platformSpec, optionalComponent, libName, optionalCause)); + } + + @Override + public void onLoad( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + boolean isPreloaded, + Path optionalLibPath) { + this.nextCheck( + check -> + check.onLoad(platformSpec, optionalComponent, libName, isPreloaded, optionalLibPath)); + } + + @Override + public void onLoadFailure( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + Throwable optionalCause) { + this.nextCheck( + check -> check.onLoadFailure(platformSpec, optionalComponent, libName, optionalCause)); + } + + @Override + public void onTempFileCreated( + PlatformSpec platformSpec, String optionalComponent, String libName, Path tempFile) { + this.nextCheck( + check -> check.onTempFileCreated(platformSpec, optionalComponent, libName, tempFile)); + } + + @Override + public void onTempFileCreationFailure( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + Path tempDir, + String libExt, + Throwable optionalCause) { + this.nextCheck( + check -> + check.onTempFileCreationFailure( + platformSpec, optionalComponent, libName, tempDir, libExt, optionalCause)); + } + + @Override + public void onTempFileCleanup( + PlatformSpec platformSpec, String optionalComponent, String libName, Path tempFile) { + this.nextCheck( + check -> check.onTempFileCleanup(platformSpec, optionalComponent, libName, tempFile)); + } + + public TestLibraryLoadingListener addCheck(Check check) { + this.checks.addLast(check); + return this; + } + + private void nextCheck(CheckInvocation invocation) { + Check nextCheck = this.checks.isEmpty() ? Check.NOTHING : this.checks.removeFirst(); + try { + invocation.invoke(nextCheck); + } catch (Throwable t) { + if (this.failedCheck == null) { + this.failedCheck = nextCheck; + this.failedCause = t; + } + } + } + + public abstract static class Check implements LibraryLoadingListener { + static final Check NOTHING = new Check("nothing") {}; + + private final String name; + + Check(String nameFormat, Object... nameArgs) { + this(String.format(nameFormat, nameArgs)); + } + + Check(String name) { + this.name = name; + } + + @Override + public void onLoad( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + boolean isPreloaded, + Path optionalLibPath) { + this.fallback( + "onLoad", platformSpec, optionalComponent, libName, isPreloaded, optionalLibPath); + } + + @Override + public void onLoadFailure( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + Throwable optionalCause) { + this.fallback("onLoadFailure", platformSpec, optionalComponent, libName, optionalCause); + } + + @Override + public void onResolveDynamic( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + boolean isPreloaded, + URL optionalUrl) { + this.fallback( + "onResolveDynamic", platformSpec, optionalComponent, libName, isPreloaded, optionalUrl); + } + + @Override + public void onResolveDynamicFailure( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + Throwable optionalCause) { + this.fallback( + "onResolveDynamicFailure", platformSpec, optionalComponent, libName, optionalCause); + } + + @Override + public void onTempFileCreated( + PlatformSpec platformSpec, String optionalComponent, String libName, Path tempFile) { + this.fallback("onTmepFileCreated", platformSpec, optionalComponent, libName, tempFile); + } + + @Override + public void onTempFileCreationFailure( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + Path tempDir, + String libExt, + Throwable optionalCause) { + this.fallback( + "onTempFileCreationFailure", + platformSpec, + optionalComponent, + libName, + tempDir, + libExt, + optionalCause); + } + + @Override + public void onTempFileCleanup( + PlatformSpec platformSpec, String optionalComponent, String libName, Path tempFile) { + this.fallback("onTempFileCleanup", platformSpec, optionalComponent, libName, tempFile); + } + + void fallback(String methodName, Object... args) { + fail("unxpected call: " + callToString(methodName, args) + " - expected: " + this.name); + } + + static final String callToString(String methodName, Object... args) { + StringBuilder builder = new StringBuilder(); + builder.append(methodName); + builder.append('('); + for (int i = 0; i < args.length; ++i) { + if (i != 0) builder.append(", "); + builder.append(args[i]); + } + builder.append(')'); + return builder.toString(); + } + + @Override + public String toString() { + return this.name; + } + } + + @FunctionalInterface + interface CheckInvocation { + void invoke(Check check); + } + + static final class LibCheck { + private final PlatformSpec expectedPlatformSpec; + private final String expectedComponent; + private final String expectedLibName; + + LibCheck(PlatformSpec expectedPlatformSpec, String expectedLibName) { + this(null, null, expectedLibName); + } + + LibCheck(String expectedComponent, String expectedLibName) { + this(null, expectedComponent, expectedLibName); + } + + LibCheck(String expectedLibName) { + this(null, null, expectedLibName); + } + + LibCheck(PlatformSpec expectedPlatformSpec, String expectedComponent, String expectedLibName) { + this.expectedPlatformSpec = expectedPlatformSpec; + this.expectedComponent = expectedComponent; + this.expectedLibName = expectedLibName; + } + + void assertMatches(PlatformSpec platformSpec, String optionalComponent, String libName) { + if (this.expectedPlatformSpec == null) { + // if no expectedPlatformSpec was provided -- just check that platformSpec is not null + assertNotNull(platformSpec); + } else { + assertEquals(this.expectedPlatformSpec, platformSpec); + } + + if (this.expectedComponent == null) { + // a null expectedComponent is treated as not expecting a component + assertNull(optionalComponent); + } else { + assertEquals(this.expectedComponent, optionalComponent); + } + + assertEquals(this.expectedLibName, libName); + } + + @Override + public String toString() { + if (this.expectedComponent == null) { + return this.expectedLibName; + } else { + return this.expectedComponent + "/" + this.expectedLibName; + } + } + } + + @FunctionalInterface + interface ThrowableCheck { + void assertMatches(Throwable t); + } +} diff --git a/components/native-loader/src/test/java/datadog/nativeloader/ThrowingLibraryLoadingListener.java b/components/native-loader/src/test/java/datadog/nativeloader/ThrowingLibraryLoadingListener.java new file mode 100644 index 00000000000..75f0a2ed6fc --- /dev/null +++ b/components/native-loader/src/test/java/datadog/nativeloader/ThrowingLibraryLoadingListener.java @@ -0,0 +1,71 @@ +package datadog.nativeloader; + +import java.net.URL; +import java.nio.file.Path; + +public class ThrowingLibraryLoadingListener implements LibraryLoadingListener { + @Override + public final void onLoad( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + boolean isPreloaded, + Path optionalLibPath) { + this.throwException("load"); + } + + @Override + public final void onLoadFailure( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + Throwable optionalCause) { + this.throwException("loadFailure"); + } + + @Override + public final void onResolveDynamic( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + boolean isPreloaded, + URL optionalUrl) { + this.throwException("resolveDynamic"); + } + + @Override + public final void onResolveDynamicFailure( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + Throwable optionalCause) { + this.throwException("resolveDynamicFailure"); + } + + @Override + public final void onTempFileCreated( + PlatformSpec platformSpec, String optionalComponent, String libName, Path tempFile) { + this.throwException("tempFileCreated"); + } + + @Override + public final void onTempFileCreationFailure( + PlatformSpec platformSpec, + String optionalComponent, + String libName, + Path tempDir, + String libExt, + Throwable optionalCause) { + this.throwException("tempFileCreationFailure"); + } + + @Override + public final void onTempFileCleanup( + PlatformSpec platformSpec, String optionalComponent, String libName, Path tempFile) { + this.throwException("tempFileCleanup"); + } + + void throwException(String event) { + throw new RuntimeException(event); + } +} diff --git a/components/yaml/build.gradle.kts b/components/yaml/build.gradle.kts deleted file mode 100644 index f2297262236..00000000000 --- a/components/yaml/build.gradle.kts +++ /dev/null @@ -1,9 +0,0 @@ -plugins { - `java-library` -} - -apply(from = "$rootDir/gradle/java.gradle") - -dependencies { - implementation("org.snakeyaml", "snakeyaml-engine", "2.9") -} diff --git a/components/yaml/gradle.lockfile b/components/yaml/gradle.lockfile deleted file mode 100644 index 15f3ac62247..00000000000 --- a/components/yaml/gradle.lockfile +++ /dev/null @@ -1,112 +0,0 @@ -# This is a Gradle generated file for dependency locking. -# Manual edits can break the build and are not advised. -# This file is expected to be part of source control. -ch.qos.logback:logback-classic:1.2.13=testCompileClasspath,testRuntimeClasspath -ch.qos.logback:logback-core:1.2.13=testCompileClasspath,testRuntimeClasspath -com.beust:jcommander:1.78=testRuntimeClasspath -com.github.javaparser:javaparser-core:3.25.6=codenarc,testCompileClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.2.0=compileClasspath,testCompileClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.7.3=spotbugs -com.github.spotbugs:spotbugs:4.7.3=spotbugs -com.google.code.findbugs:jsr305:3.0.2=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -com.google.code.gson:gson:2.9.1=spotbugs -com.thoughtworks.qdox:qdox:1.12.1=codenarc,testRuntimeClasspath -commons-codec:commons-codec:1.15=spotbugs -de.thetaphi:forbiddenapis:3.8=compileClasspath -info.picocli:picocli:4.6.3=testRuntimeClasspath -io.leangen.geantyref:geantyref:1.3.16=testRuntimeClasspath -jaxen:jaxen:1.2.0=spotbugs -jline:jline:2.14.6=testRuntimeClasspath -junit:junit:4.13.2=testCompileClasspath,testRuntimeClasspath -net.bytebuddy:byte-buddy-agent:1.12.8=testRuntimeClasspath -net.bytebuddy:byte-buddy:1.12.8=testRuntimeClasspath -net.jcip:jcip-annotations:1.0=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -net.sf.saxon:Saxon-HE:11.4=spotbugs -org.apache.ant:ant-antlr:1.10.14=codenarc -org.apache.ant:ant-antlr:1.10.15=testRuntimeClasspath -org.apache.ant:ant-junit:1.10.14=codenarc -org.apache.ant:ant-junit:1.10.15=testRuntimeClasspath -org.apache.ant:ant-launcher:1.10.15=testRuntimeClasspath -org.apache.ant:ant:1.10.15=testCompileClasspath,testRuntimeClasspath -org.apache.bcel:bcel:6.5.0=spotbugs -org.apache.commons:commons-lang3:3.12.0=spotbugs -org.apache.commons:commons-text:1.10.0=spotbugs -org.apache.httpcomponents.client5:httpclient5:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5-h2:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5:5.1.3=spotbugs -org.apache.logging.log4j:log4j-api:2.19.0=spotbugs -org.apache.logging.log4j:log4j-core:2.19.0=spotbugs -org.apiguardian:apiguardian-api:1.1.2=testCompileClasspath -org.codehaus.groovy:groovy-all:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-ant:3.0.23=codenarc -org.codehaus.groovy:groovy-ant:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-astbuilder:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-cli-picocli:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-console:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-datetime:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-docgenerator:3.0.23=codenarc -org.codehaus.groovy:groovy-docgenerator:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-groovydoc:3.0.23=codenarc -org.codehaus.groovy:groovy-groovydoc:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-groovysh:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jmx:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-json:3.0.23=codenarc -org.codehaus.groovy:groovy-json:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jsr223:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-macro:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-nio:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-servlet:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-sql:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-swing:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-templates:3.0.23=codenarc -org.codehaus.groovy:groovy-templates:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test-junit5:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-testng:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-xml:3.0.23=codenarc -org.codehaus.groovy:groovy-xml:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy:3.0.23=codenarc -org.codehaus.groovy:groovy:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codenarc:CodeNarc:3.6.0=codenarc -org.dom4j:dom4j:2.1.3=spotbugs -org.gmetrics:GMetrics:2.1.0=codenarc -org.hamcrest:hamcrest-core:1.3=testCompileClasspath,testRuntimeClasspath -org.hamcrest:hamcrest:3.0=testCompileClasspath,testRuntimeClasspath -org.jacoco:org.jacoco.agent:0.8.14=jacocoAgent,jacocoAnt -org.jacoco:org.jacoco.ant:0.8.14=jacocoAnt -org.jacoco:org.jacoco.core:0.8.14=jacocoAnt -org.jacoco:org.jacoco.report:0.8.14=jacocoAnt -org.junit.jupiter:junit-jupiter-api:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-engine:5.12.2=testRuntimeClasspath -org.junit.jupiter:junit-jupiter-params:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-commons:1.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-engine:1.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-launcher:1.12.2=testRuntimeClasspath -org.junit:junit-bom:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit:junit-bom:5.9.1=spotbugs -org.mockito:mockito-core:4.4.0=testRuntimeClasspath -org.objenesis:objenesis:3.3=testCompileClasspath,testRuntimeClasspath -org.opentest4j:opentest4j:1.3.0=testCompileClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.4=spotbugs -org.ow2.asm:asm-commons:9.4=spotbugs -org.ow2.asm:asm-commons:9.9=jacocoAnt -org.ow2.asm:asm-tree:9.4=spotbugs -org.ow2.asm:asm-tree:9.9=jacocoAnt -org.ow2.asm:asm-util:9.4=spotbugs -org.ow2.asm:asm:9.4=spotbugs -org.ow2.asm:asm:9.9=jacocoAnt -org.slf4j:jcl-over-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath -org.slf4j:jul-to-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath -org.slf4j:log4j-over-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath -org.slf4j:slf4j-api:1.7.32=testCompileClasspath,testRuntimeClasspath -org.slf4j:slf4j-api:2.0.0=spotbugs,spotbugsSlf4j -org.slf4j:slf4j-simple:2.0.0=spotbugsSlf4j -org.snakeyaml:snakeyaml-engine:2.9=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.spockframework:spock-bom:2.4-M6-groovy-3.0=testCompileClasspath,testRuntimeClasspath -org.spockframework:spock-core:2.4-M6-groovy-3.0=testCompileClasspath,testRuntimeClasspath -org.testng:testng:7.5.1=testRuntimeClasspath -org.webjars:jquery:3.5.1=testRuntimeClasspath -org.xmlresolver:xmlresolver:4.4.3=spotbugs -xml-apis:xml-apis:1.4.01=spotbugs -empty=annotationProcessor,spotbugsPlugins,testAnnotationProcessor diff --git a/dd-java-agent/agent-aiguard/gradle.lockfile b/dd-java-agent/agent-aiguard/gradle.lockfile index 30fd1190724..4f3827b2a8b 100644 --- a/dd-java-agent/agent-aiguard/gradle.lockfile +++ b/dd-java-agent/agent-aiguard/gradle.lockfile @@ -5,138 +5,102 @@ cafe.cryptography:curve25519-elisabeth:0.1.0=runtimeClasspath,testRuntimeClasspa cafe.cryptography:ed25519-elisabeth:0.1.0=runtimeClasspath,testRuntimeClasspath ch.qos.logback:logback-classic:1.2.13=testCompileClasspath,testRuntimeClasspath ch.qos.logback:logback-core:1.2.13=testCompileClasspath,testRuntimeClasspath -com.beust:jcommander:1.78=testRuntimeClasspath com.datadoghq.okhttp3:okhttp:3.12.15=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.datadoghq.okio:okio:1.17.6=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.datadoghq:dd-javac-plugin-client:0.2.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.datadoghq:java-dogstatsd-client:4.4.3=runtimeClasspath,testRuntimeClasspath +com.datadoghq:java-dogstatsd-client:4.4.5=runtimeClasspath,testRuntimeClasspath +com.datadoghq:sketches-java:0.8.3=runtimeClasspath,testRuntimeClasspath com.fasterxml.jackson.core:jackson-annotations:2.20=testCompileClasspath,testRuntimeClasspath com.fasterxml.jackson.core:jackson-core:2.20.0=testCompileClasspath,testRuntimeClasspath com.fasterxml.jackson.core:jackson-databind:2.20.0=testCompileClasspath,testRuntimeClasspath com.fasterxml.jackson:jackson-bom:2.20.0=testCompileClasspath,testRuntimeClasspath -com.github.javaparser:javaparser-core:3.25.6=codenarc,testCompileClasspath,testRuntimeClasspath -com.github.jnr:jffi:1.3.13=runtimeClasspath,testRuntimeClasspath +com.github.javaparser:javaparser-core:3.25.6=codenarc +com.github.jnr:jffi:1.3.14=runtimeClasspath,testRuntimeClasspath com.github.jnr:jnr-a64asm:1.0.0=runtimeClasspath,testRuntimeClasspath com.github.jnr:jnr-constants:0.10.4=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-enxio:0.32.17=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-ffi:2.2.16=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-posix:3.1.19=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-unixsocket:0.38.22=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-enxio:0.32.19=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-ffi:2.2.18=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-posix:3.1.21=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-unixsocket:0.38.24=runtimeClasspath,testRuntimeClasspath com.github.jnr:jnr-x86asm:1.0.2=runtimeClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.2.0=compileClasspath,testCompileClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.7.3=spotbugs -com.github.spotbugs:spotbugs:4.7.3=spotbugs +com.github.spotbugs:spotbugs-annotations:4.9.8=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath +com.github.spotbugs:spotbugs:4.9.8=spotbugs +com.github.stephenc.jcip:jcip-annotations:1.0-1=spotbugs com.google.code.findbugs:jsr305:3.0.2=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -com.google.code.gson:gson:2.9.1=spotbugs +com.google.code.gson:gson:2.13.2=spotbugs +com.google.errorprone:error_prone_annotations:2.41.0=spotbugs com.squareup.moshi:moshi:1.11.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.squareup.okio:okio:1.17.5=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.thoughtworks.qdox:qdox:1.12.1=codenarc,testRuntimeClasspath +com.thoughtworks.qdox:qdox:1.12.1=codenarc com.vaadin.external.google:android-json:0.0.20131108.vaadin1=testCompileClasspath,testRuntimeClasspath -commons-codec:commons-codec:1.15=spotbugs commons-fileupload:commons-fileupload:1.5=testCompileClasspath,testRuntimeClasspath commons-io:commons-io:2.11.0=testCompileClasspath,testRuntimeClasspath -de.thetaphi:forbiddenapis:3.8=compileClasspath -info.picocli:picocli:4.6.3=testRuntimeClasspath +commons-io:commons-io:2.20.0=spotbugs +de.thetaphi:forbiddenapis:3.10=compileClasspath,testCompileClasspath,testRuntimeClasspath io.leangen.geantyref:geantyref:1.3.16=testRuntimeClasspath -jaxen:jaxen:1.2.0=spotbugs -jline:jline:2.14.6=testRuntimeClasspath -junit:junit:4.13.2=testCompileClasspath,testRuntimeClasspath -net.bytebuddy:byte-buddy-agent:1.17.7=testCompileClasspath,testRuntimeClasspath -net.bytebuddy:byte-buddy:1.17.7=testCompileClasspath,testRuntimeClasspath -net.jcip:jcip-annotations:1.0=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -net.sf.saxon:Saxon-HE:11.4=spotbugs +jaxen:jaxen:2.0.0=spotbugs +net.bytebuddy:byte-buddy-agent:1.18.8=testCompileClasspath,testRuntimeClasspath +net.bytebuddy:byte-buddy:1.18.8=testCompileClasspath,testRuntimeClasspath +net.sf.saxon:Saxon-HE:12.9=spotbugs org.apache.ant:ant-antlr:1.10.14=codenarc -org.apache.ant:ant-antlr:1.10.15=testRuntimeClasspath org.apache.ant:ant-junit:1.10.14=codenarc -org.apache.ant:ant-junit:1.10.15=testRuntimeClasspath -org.apache.ant:ant-launcher:1.10.15=testRuntimeClasspath -org.apache.ant:ant:1.10.15=testCompileClasspath,testRuntimeClasspath -org.apache.bcel:bcel:6.5.0=spotbugs -org.apache.commons:commons-lang3:3.12.0=spotbugs -org.apache.commons:commons-text:1.10.0=spotbugs -org.apache.httpcomponents.client5:httpclient5:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5-h2:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5:5.1.3=spotbugs -org.apache.logging.log4j:log4j-api:2.19.0=spotbugs -org.apache.logging.log4j:log4j-core:2.19.0=spotbugs +org.apache.bcel:bcel:6.11.0=spotbugs +org.apache.commons:commons-lang3:3.19.0=spotbugs +org.apache.commons:commons-text:1.14.0=spotbugs +org.apache.logging.log4j:log4j-api:2.25.2=spotbugs +org.apache.logging.log4j:log4j-core:2.25.2=spotbugs org.apiguardian:apiguardian-api:1.1.2=testCompileClasspath -org.codehaus.groovy:groovy-all:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-ant:3.0.23=codenarc -org.codehaus.groovy:groovy-ant:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-astbuilder:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-cli-picocli:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-console:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-datetime:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-docgenerator:3.0.23=codenarc -org.codehaus.groovy:groovy-docgenerator:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-groovydoc:3.0.23=codenarc -org.codehaus.groovy:groovy-groovydoc:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-groovysh:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jmx:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-json:3.0.23=codenarc -org.codehaus.groovy:groovy-json:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jsr223:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-macro:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-nio:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-servlet:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-sql:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-swing:3.0.24=testCompileClasspath,testRuntimeClasspath +org.codehaus.groovy:groovy-json:3.0.25=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-templates:3.0.23=codenarc -org.codehaus.groovy:groovy-templates:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test-junit5:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-testng:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-xml:3.0.23=codenarc -org.codehaus.groovy:groovy-xml:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy:3.0.23=codenarc -org.codehaus.groovy:groovy:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codenarc:CodeNarc:3.6.0=codenarc -org.dom4j:dom4j:2.1.3=spotbugs +org.codehaus.groovy:groovy:3.0.25=testCompileClasspath,testRuntimeClasspath +org.codenarc:CodeNarc:3.7.0=codenarc +org.dom4j:dom4j:2.2.0=spotbugs org.gmetrics:GMetrics:2.1.0=codenarc -org.hamcrest:hamcrest-core:1.3=testCompileClasspath,testRuntimeClasspath org.hamcrest:hamcrest:3.0=testCompileClasspath,testRuntimeClasspath org.jacoco:org.jacoco.agent:0.8.14=jacocoAgent,jacocoAnt org.jacoco:org.jacoco.ant:0.8.14=jacocoAnt org.jacoco:org.jacoco.core:0.8.14=jacocoAnt org.jacoco:org.jacoco.report:0.8.14=jacocoAnt -org.junit.jupiter:junit-jupiter-api:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-engine:5.12.2=testRuntimeClasspath -org.junit.jupiter:junit-jupiter-params:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-commons:1.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-engine:1.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-launcher:1.12.2=testRuntimeClasspath -org.junit:junit-bom:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit:junit-bom:5.9.1=spotbugs +org.junit.jupiter:junit-jupiter-api:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-engine:5.14.1=testRuntimeClasspath +org.junit.jupiter:junit-jupiter-params:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-commons:1.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-engine:1.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-launcher:1.14.1=testRuntimeClasspath +org.junit:junit-bom:5.14.0=spotbugs +org.junit:junit-bom:5.14.1=testCompileClasspath,testRuntimeClasspath org.mockito:mockito-core:4.4.0=testRuntimeClasspath org.objenesis:objenesis:3.3=testCompileClasspath,testRuntimeClasspath org.opentest4j:opentest4j:1.3.0=testCompileClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.4=spotbugs -org.ow2.asm:asm-commons:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-commons:9.4=spotbugs -org.ow2.asm:asm-commons:9.9=jacocoAnt -org.ow2.asm:asm-tree:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-tree:9.4=spotbugs -org.ow2.asm:asm-tree:9.9=jacocoAnt -org.ow2.asm:asm-util:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-util:9.4=spotbugs -org.ow2.asm:asm:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm:9.4=spotbugs -org.ow2.asm:asm:9.9=jacocoAnt +org.ow2.asm:asm-analysis:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-analysis:9.9=spotbugs +org.ow2.asm:asm-commons:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-commons:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-tree:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-tree:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-util:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-util:9.9=spotbugs +org.ow2.asm:asm:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm:9.9=jacocoAnt,spotbugs org.skyscreamer:jsonassert:1.5.3=testCompileClasspath,testRuntimeClasspath org.slf4j:jcl-over-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath org.slf4j:jul-to-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath org.slf4j:log4j-over-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath org.slf4j:slf4j-api:1.7.30=compileClasspath,runtimeClasspath org.slf4j:slf4j-api:1.7.32=testCompileClasspath,testRuntimeClasspath -org.slf4j:slf4j-api:2.0.0=spotbugs,spotbugsSlf4j -org.slf4j:slf4j-simple:2.0.0=spotbugsSlf4j +org.slf4j:slf4j-api:2.0.17=spotbugs,spotbugsSlf4j +org.slf4j:slf4j-simple:2.0.17=spotbugsSlf4j org.snakeyaml:snakeyaml-engine:2.9=runtimeClasspath,testRuntimeClasspath -org.spockframework:spock-bom:2.4-M6-groovy-3.0=testCompileClasspath,testRuntimeClasspath -org.spockframework:spock-core:2.4-M6-groovy-3.0=testCompileClasspath,testRuntimeClasspath -org.testng:testng:7.5.1=testRuntimeClasspath -org.webjars:jquery:3.5.1=testRuntimeClasspath -org.xmlresolver:xmlresolver:4.4.3=spotbugs -xml-apis:xml-apis:1.4.01=spotbugs +org.spockframework:spock-bom:2.4-groovy-3.0=testCompileClasspath,testRuntimeClasspath +org.spockframework:spock-core:2.4-groovy-3.0=testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-junit:1.2.1=testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-parser:1.2.0=testCompileClasspath,testRuntimeClasspath +org.xmlresolver:xmlresolver:5.3.3=spotbugs empty=annotationProcessor,shadow,spotbugsPlugins,testAnnotationProcessor diff --git a/dd-java-agent/agent-aiguard/src/main/java/com/datadog/aiguard/AIGuardInternal.java b/dd-java-agent/agent-aiguard/src/main/java/com/datadog/aiguard/AIGuardInternal.java index fc2d2dafb64..2057c1a4532 100644 --- a/dd-java-agent/agent-aiguard/src/main/java/com/datadog/aiguard/AIGuardInternal.java +++ b/dd-java-agent/agent-aiguard/src/main/java/com/datadog/aiguard/AIGuardInternal.java @@ -4,7 +4,6 @@ import static datadog.trace.api.telemetry.WafMetricCollector.AIGuardTruncationType.CONTENT; import static datadog.trace.api.telemetry.WafMetricCollector.AIGuardTruncationType.MESSAGES; import static datadog.trace.util.Strings.isBlank; -import static java.util.Collections.singletonMap; import com.squareup.moshi.JsonAdapter; import com.squareup.moshi.JsonReader; @@ -17,6 +16,7 @@ import datadog.trace.api.aiguard.AIGuard.AIGuardAbortError; import datadog.trace.api.aiguard.AIGuard.AIGuardClientError; import datadog.trace.api.aiguard.AIGuard.Action; +import datadog.trace.api.aiguard.AIGuard.ContentPart; import datadog.trace.api.aiguard.AIGuard.Evaluation; import datadog.trace.api.aiguard.AIGuard.Message; import datadog.trace.api.aiguard.AIGuard.Options; @@ -28,6 +28,7 @@ import datadog.trace.bootstrap.instrumentation.api.AgentScope; import datadog.trace.bootstrap.instrumentation.api.AgentSpan; import datadog.trace.bootstrap.instrumentation.api.AgentTracer; +import datadog.trace.bootstrap.instrumentation.api.Tags; import java.io.IOException; import java.lang.annotation.Annotation; import java.lang.reflect.Type; @@ -64,12 +65,16 @@ public BadConfigurationException(final String message) { static final String SPAN_NAME = "ai_guard"; static final String TARGET_TAG = "ai_guard.target"; - static final String TOOL_TAG = "ai_guard.tool"; + static final String TOOL_TAG = "ai_guard.tool_name"; static final String ACTION_TAG = "ai_guard.action"; static final String REASON_TAG = "ai_guard.reason"; static final String BLOCKED_TAG = "ai_guard.blocked"; + static final String EVENT_TAG = "ai_guard.event"; static final String META_STRUCT_TAG = "ai_guard"; - static final String META_STRUCT_KEY = "messages"; + static final String META_STRUCT_MESSAGES = "messages"; + static final String META_STRUCT_CATEGORIES = "attack_categories"; + static final String META_STRUCT_SDS = "sds"; + static final String META_STRUCT_TAG_PROBS = "tag_probs"; public static void install() { final Config config = Config.get(); @@ -134,19 +139,39 @@ private static List messagesForMetaStruct(List messages) { final List result = new ArrayList<>(size); final int maxContent = config.getAiGuardMaxContentSize(); boolean contentTruncated = false; - for (int i = 0; i < size; i++) { - Message source = messages.get(i); - final String content = source.getContent(); - if (content != null && content.length() > maxContent) { - contentTruncated = true; - source = - new Message( - source.getRole(), - content.substring(0, maxContent), - source.getToolCalls(), - source.getToolCallId()); + for (int i = messages.size() - size; i < messages.size(); i++) { + final Message source = messages.get(i); + + List toolCalls = source.getToolCalls(); + if (toolCalls != null) { + toolCalls = new ArrayList<>(toolCalls); + } + + List contentParts = source.getContentParts(); + if (contentParts != null) { + final List truncatedParts = new ArrayList<>(contentParts.size()); + for (final ContentPart part : contentParts) { + if (part.getType() == ContentPart.Type.TEXT + && part.getText() != null + && part.getText().length() > maxContent) { + contentTruncated = true; + final String text = part.getText().substring(0, maxContent); + truncatedParts.add(ContentPart.text(text)); + } else { + truncatedParts.add(part); + } + } + + result.add( + new Message(source.getRole(), truncatedParts, toolCalls, source.getToolCallId())); + } else { + String content = source.getContent(); + if (content != null && content.length() > maxContent) { + contentTruncated = true; + content = content.substring(0, maxContent); + } + result.add(new Message(source.getRole(), content, toolCalls, source.getToolCallId())); } - result.add(source); } if (contentTruncated) { WafMetricCollector.get().aiGuardTruncated(CONTENT); @@ -182,6 +207,9 @@ private static String getToolName(final Message current, final List mes } private boolean isBlockingEnabled(final Options options, final Object isBlockingEnabled) { + if (isBlockingEnabled == null) { + return false; + } return options.block() && "true".equalsIgnoreCase(isBlockingEnabled.toString()); } @@ -197,6 +225,11 @@ public Evaluation evaluate(final List messages, final Options options) builder.asChildOf(parent.context()); } final AgentSpan span = builder.start(); + final AgentSpan localRootSpan = span.getLocalRootSpan(); + if (localRootSpan != null) { + localRootSpan.setTag(Tags.AI_GUARD_KEEP, true); + localRootSpan.setTag(EVENT_TAG, true); + } try (final AgentScope scope = tracer.activateSpan(span)) { final Message last = messages.get(messages.size() - 1); if (isToolCall(last)) { @@ -208,8 +241,8 @@ public Evaluation evaluate(final List messages, final Options options) } else { span.setTag(TARGET_TAG, "prompt"); } - final Map metaStruct = - singletonMap(META_STRUCT_KEY, messagesForMetaStruct(messages)); + final Map metaStruct = new HashMap<>(2); + metaStruct.put(META_STRUCT_MESSAGES, messagesForMetaStruct(messages)); span.setMetaStruct(META_STRUCT_TAG, metaStruct); final Request.Builder request = new Request.Builder() @@ -224,16 +257,33 @@ public Evaluation evaluate(final List messages, final Options options) } final Action action = Action.valueOf(actionStr); final String reason = (String) result.get("reason"); + @SuppressWarnings("unchecked") + final List tags = (List) result.get("tags"); + @SuppressWarnings("unchecked") + final List sdsFindings = (List) result.get("sds_findings"); + @SuppressWarnings("unchecked") + final Map tagProbs = (Map) result.get("tag_probs"); span.setTag(ACTION_TAG, action); - span.setTag(REASON_TAG, reason); + if (reason != null) { + span.setTag(REASON_TAG, reason); + } + if (tags != null && !tags.isEmpty()) { + metaStruct.put(META_STRUCT_CATEGORIES, tags); + } + if (tagProbs != null && !tagProbs.isEmpty()) { + metaStruct.put(META_STRUCT_TAG_PROBS, tagProbs); + } + if (sdsFindings != null && !sdsFindings.isEmpty()) { + metaStruct.put(META_STRUCT_SDS, sdsFindings); + } final boolean shouldBlock = isBlockingEnabled(options, result.get("is_blocking_enabled")) && action != Action.ALLOW; WafMetricCollector.get().aiGuardRequest(action, shouldBlock); if (shouldBlock) { span.setTag(BLOCKED_TAG, true); - throw new AIGuardAbortError(action, reason); + throw new AIGuardAbortError(action, reason, tags, tagProbs, sdsFindings); } - return new Evaluation(action, reason); + return new Evaluation(action, reason, tags, tagProbs, sdsFindings); } } catch (AIGuardAbortError e) { span.addThrowable(e); @@ -327,12 +377,45 @@ public Message fromJson(JsonReader reader) throws IOException { public void toJson(final JsonWriter writer, final Message value) throws IOException { writer.beginObject(); writeValue(writer, "role", value.getRole()); - writeValue(writer, "content", value.getContent()); + + if (value.getContentParts() != null) { + writeContentParts(writer, "content", value.getContentParts()); + } else { + writeValue(writer, "content", value.getContent()); + } + writeArray(writer, "tool_calls", value.getToolCalls()); writeValue(writer, "tool_call_id", value.getToolCallId()); writer.endObject(); } + private void writeContentParts( + final JsonWriter writer, final String name, final List contentParts) + throws IOException { + writer.name(name); + writer.beginArray(); + for (final ContentPart part : contentParts) { + writer.beginObject(); + + writer.name("type"); + writer.value(part.getType().toString()); + + if (part.getType() == ContentPart.Type.TEXT) { + writer.name("text"); + writer.value(part.getText()); + } else if (part.getType() == ContentPart.Type.IMAGE_URL) { + writer.name("image_url"); + writer.beginObject(); + writer.name("url"); + writer.value(part.getImageUrl().getUrl()); + writer.endObject(); + } + + writer.endObject(); + } + writer.endArray(); + } + private void writeValue(final JsonWriter writer, final String name, final Object value) throws IOException { if (value != null) { diff --git a/dd-java-agent/agent-aiguard/src/test/groovy/com/datadog/aiguard/AIGuardInternalTests.groovy b/dd-java-agent/agent-aiguard/src/test/groovy/com/datadog/aiguard/AIGuardInternalTests.groovy index 366b977a7a3..c017a2ac0ce 100644 --- a/dd-java-agent/agent-aiguard/src/test/groovy/com/datadog/aiguard/AIGuardInternalTests.groovy +++ b/dd-java-agent/agent-aiguard/src/test/groovy/com/datadog/aiguard/AIGuardInternalTests.groovy @@ -10,6 +10,7 @@ import datadog.trace.api.aiguard.AIGuard import datadog.trace.api.telemetry.WafMetricCollector import datadog.trace.bootstrap.instrumentation.api.AgentSpan import datadog.trace.bootstrap.instrumentation.api.AgentTracer +import datadog.trace.bootstrap.instrumentation.api.Tags import datadog.trace.test.util.DDSpecification import okhttp3.Call import okhttp3.HttpUrl @@ -72,12 +73,15 @@ class AIGuardInternalTests extends DDSpecification { protected static final PROMPT = TOOL_OUTPUT + [AIGuard.Message.message('assistant', '2 + 2 is 5'), AIGuard.Message.message('user', '')] protected AgentSpan span + protected AgentSpan localRootSpan void setup() { injectEnvConfig('SERVICE', 'ai_guard_test') injectEnvConfig('ENV', 'test') span = Mock(AgentSpan) + localRootSpan = Mock(AgentSpan) + span.getLocalRootSpan() >> localRootSpan final builder = Mock(AgentTracer.SpanBuilder) { start() >> span } @@ -157,13 +161,14 @@ class AIGuardInternalTests extends DDSpecification { Request request = null Throwable error = null AIGuard.Evaluation eval = null + Map receivedMeta = null final throwAbortError = suite.blocking && suite.action != ALLOW final call = Mock(Call) { execute() >> { return mockResponse( request, 200, - [data: [attributes: [action: suite.action, reason: suite.reason, is_blocking_enabled: suite.blocking]]] + [data: [attributes: [action: suite.action, reason: suite.reason, tags: suite.tags ?: [], tag_probs: suite.tagProbabilities ?: [:], is_blocking_enabled: suite.blocking]]] ) } } @@ -184,25 +189,37 @@ class AIGuardInternalTests extends DDSpecification { then: 1 * span.setTag(AIGuardInternal.TARGET_TAG, suite.target) + 1 * localRootSpan.setTag(Tags.AI_GUARD_KEEP, true) + 1 * localRootSpan.setTag(AIGuardInternal.EVENT_TAG, true) if (suite.target == 'tool') { 1 * span.setTag(AIGuardInternal.TOOL_TAG, 'calc') } 1 * span.setTag(AIGuardInternal.ACTION_TAG, suite.action) 1 * span.setTag(AIGuardInternal.REASON_TAG, suite.reason) - 1 * span.setMetaStruct(AIGuardInternal.META_STRUCT_TAG, [messages: suite.messages]) + 1 * span.setMetaStruct(AIGuardInternal.META_STRUCT_TAG, _ as Map) >> { + receivedMeta = it[1] as Map + return span + } if (throwAbortError) { 1 * span.addThrowable(_ as AIGuard.AIGuardAbortError) } + assertMeta(receivedMeta, suite) assertRequest(request, suite.messages) if (throwAbortError) { error instanceof AIGuard.AIGuardAbortError error.action == suite.action error.reason == suite.reason + error.tags == suite.tags + error.tagProbabilities == suite.tagProbabilities + error.sds == [] } else { error == null eval.action == suite.action eval.reason == suite.reason + eval.tags == suite.tags + eval.tagProbabilities == suite.tagProbabilities + eval.sds == [] } assertTelemetry('ai_guard.requests', "action:$suite.action", "block:$throwAbortError", 'error:false') @@ -210,13 +227,16 @@ class AIGuardInternalTests extends DDSpecification { suite << TestSuite.build() } - void 'test evaluate with API errors'() { + void 'test evaluate block defaults to remote is_blocking_enabled'() { given: - final errors = [[status: 400, title: 'Bad request']] - Request request = null + def request final call = Mock(Call) { execute() >> { - return mockResponse(request, 404, [errors: errors]) + return mockResponse( + request, + 200, + [data: [attributes: [action: 'DENY', reason: 'Nope', tags: ['deny_everything'], is_blocking_enabled: remoteBlocking]]] + ) } } final client = Mock(OkHttpClient) { @@ -227,6 +247,36 @@ class AIGuardInternalTests extends DDSpecification { } final aiguard = new AIGuardInternal(URL, HEADERS, client) + when: + Throwable error = null + AIGuard.Evaluation eval = null + try { + eval = aiguard.evaluate(TOOL_CALL, options) + } catch (Throwable e) { + error = e + } + + then: + if (shouldBlock) { + error instanceof AIGuard.AIGuardAbortError + error.action == DENY + } else { + error == null + eval.action == DENY + } + + where: + options | remoteBlocking | shouldBlock + AIGuard.Options.DEFAULT | true | true + AIGuard.Options.DEFAULT | false | false + new AIGuard.Options().block(false) | true | false + } + + void 'test evaluate with API errors'() { + given: + final errors = [[status: 400, title: 'Bad request']] + final aiguard = mockClient(404, [errors: errors]) + when: aiguard.evaluate(TOOL_CALL, AIGuard.Options.DEFAULT) @@ -239,19 +289,7 @@ class AIGuardInternalTests extends DDSpecification { void 'test evaluate with invalid JSON'() { given: - Request request = null - final call = Mock(Call) { - execute() >> { - return mockResponse(request, 200, [bad: 'This is an invalid response']) - } - } - final client = Mock(OkHttpClient) { - newCall(_ as Request) >> { - request = (Request) it[0] - return call - } - } - final aiguard = new AIGuardInternal(URL, HEADERS, client) + final aiguard = mockClient(200, [bad: 'This is an invalid response']) when: aiguard.evaluate(TOOL_CALL, AIGuard.Options.DEFAULT) @@ -264,19 +302,7 @@ class AIGuardInternalTests extends DDSpecification { void 'test evaluate with missing action'() { given: - Request request = null - final call = Mock(Call) { - execute() >> { - return mockResponse(request, 200, [data: [attributes: [reason: 'I miss something']]]) - } - } - final client = Mock(OkHttpClient) { - newCall(_ as Request) >> { - request = (Request) it[0] - return call - } - } - final aiguard = new AIGuardInternal(URL, HEADERS, client) + final aiguard = mockClient(200, [data: [attributes: [reason: 'I miss something']]]) when: aiguard.evaluate(TOOL_CALL, AIGuard.Options.DEFAULT) @@ -289,19 +315,7 @@ class AIGuardInternalTests extends DDSpecification { void 'test evaluate with non JSON response'() { given: - Request request = null - final call = Mock(Call) { - execute() >> { - return mockResponse(request, 200, [data: [attributes: [reason: 'I miss something']]]) - } - } - final client = Mock(OkHttpClient) { - newCall(_ as Request) >> { - request = (Request) it[0] - return call - } - } - final aiguard = new AIGuardInternal(URL, HEADERS, client) + final aiguard = mockClient(200, 'I am no JSON') when: aiguard.evaluate(TOOL_CALL, AIGuard.Options.DEFAULT) @@ -314,19 +328,7 @@ class AIGuardInternalTests extends DDSpecification { void 'test evaluate with empty response'() { given: - Request request = null - final call = Mock(Call) { - execute() >> { - return mockResponse(request, 200, null) - } - } - final client = Mock(OkHttpClient) { - newCall(_ as Request) >> { - request = (Request) it[0] - return call - } - } - final aiguard = new AIGuardInternal(URL, HEADERS, client) + final aiguard = mockClient(200, null) when: aiguard.evaluate(TOOL_CALL, AIGuard.Options.DEFAULT) @@ -340,19 +342,7 @@ class AIGuardInternalTests extends DDSpecification { void 'test message length truncation'() { given: final maxMessages = Config.get().getAiGuardMaxMessagesLength() - Request request = null - final call = Mock(Call) { - execute() >> { - return mockResponse(request, 200, [data: [attributes: [action: ALLOW, reason: 'It is fine']]]) - } - } - final client = Mock(OkHttpClient) { - newCall(_ as Request) >> { - request = (Request) it[0] - return call - } - } - final aiguard = new AIGuardInternal(URL, HEADERS, client) + final aiguard = mockClient(200, [data: [attributes: [action: 'ALLOW', reason: 'It is fine']]]) final messages = (0..maxMessages) .collect { AIGuard.Message.message('user', "This is a prompt: ${it}") } .toList() @@ -372,19 +362,7 @@ class AIGuardInternalTests extends DDSpecification { void 'test message content truncation'() { given: final maxContent = Config.get().getAiGuardMaxContentSize() - Request request = null - final call = Mock(Call) { - execute() >> { - return mockResponse(request, 200, [data: [attributes: [action: ALLOW, reason: 'It is fine']]]) - } - } - final client = Mock(OkHttpClient) { - newCall(_ as Request) >> { - request = (Request) it[0] - return call - } - } - final aiguard = new AIGuardInternal(URL, HEADERS, client) + final aiguard = mockClient(200, [data: [attributes: [action: 'ALLOW', reason: 'It is fine']]]) final message = AIGuard.Message.message("user", (0..maxContent).collect { 'A' }.join()) when: @@ -416,32 +394,149 @@ class AIGuardInternalTests extends DDSpecification { messages << [[], null] } + void 'test evaluate with sds findings'() { + given: + final sdsFindings = [ + [ + rule_display_name: 'Credit Card Number', + rule_tag: 'credit_card', + category: 'pii', + matched_text: '4111111111111111', + location: [start_index: 10, end_index_exclusive: 26, path: 'messages[0].content[0].text'] + ], + [ + rule_display_name: 'Social Security Number', + rule_tag: 'ssn', + category: 'pii', + matched_text: '123-45-6789', + location: [start_index: 30, end_index_exclusive: 41, path: 'messages[1].tool_calls[0].function.arguments'] + ] + ] + final aiguard = mockClient(200, [data: [attributes: [action: 'ALLOW', reason: 'It is fine', sds_findings: sdsFindings]]]) + Map receivedMeta + + when: + final result = aiguard.evaluate(PROMPT, AIGuard.Options.DEFAULT) + + then: + 1 * span.setMetaStruct(AIGuardInternal.META_STRUCT_TAG, _) >> { + receivedMeta = it[1] as Map + return span + } + receivedMeta.sds == sdsFindings + result.sds == sdsFindings + } + + void 'test evaluate with empty sds findings'() { + given: + final aiguard = mockClient(200, [data: [attributes: [action: 'ALLOW', reason: 'It is fine', sds_findings: sdsFindings]]]) + Map receivedMeta + + when: + final result = aiguard.evaluate(PROMPT, AIGuard.Options.DEFAULT) + + then: + 1 * span.setMetaStruct(AIGuardInternal.META_STRUCT_TAG, _) >> { + receivedMeta = it[1] as Map + return span + } + !receivedMeta.containsKey('sds') + result.sds == (sdsFindings ?: []) + + where: + sdsFindings << [null, []] + } + + void 'test evaluate with sds findings in abort error'() { + given: + final sdsFindings = [ + [ + rule_display_name: 'Credit Card Number', + rule_tag: 'credit_card', + category: 'pii', + matched_text: '4111111111111111', + location: [start_index: 10, end_index_exclusive: 26, path: 'messages[0].content[0].text'] + ] + ] + final aiguard = mockClient(200, [data: [attributes: [action: 'ABORT', reason: 'PII detected', tags: ['pii'], sds_findings: sdsFindings, is_blocking_enabled: true]]]) + + when: + aiguard.evaluate(PROMPT, new AIGuard.Options().block(true)) + + then: + final error = thrown(AIGuard.AIGuardAbortError) + error.sds == sdsFindings + } + void 'test missing tool name'() { given: + final aiguard = mockClient(200, [data: [attributes: [action: 'ALLOW', reason: 'Just do it']]]) + + when: + aiguard.evaluate([AIGuard.Message.tool('call_1', 'Content')], AIGuard.Options.DEFAULT) + + then: + 1 * span.setTag(AIGuardInternal.TARGET_TAG, 'tool') + 0 * span.setTag(AIGuardInternal.TOOL_TAG, _) + } + + void 'map requires even number of params'() { + when: + AIGuardInternal.mapOf('1', '2', '3') + + then: + thrown(IllegalArgumentException) + } + + void 'test message immutability'() { + given: + final aiguard = mockClient(200, [data: [attributes: [action: 'ALLOW', reason: 'Just do it']]]) + final messages = [ + new AIGuard.Message( + "assistant", + (String) null, + [AIGuard.ToolCall.toolCall('call_1', 'execute_shell', '{"cmd": "ls -lah"}')], + null + ) + ] + Map receivedMeta + + when: + aiguard.evaluate(messages, AIGuard.Options.DEFAULT) + + then: + 1 * span.finish() >> { + // modify the messages before serialization + messages.first().toolCalls.add( + AIGuard.ToolCall.toolCall('call_2', 'execute_shell', '{"cmd": "rm -rf"}') + ) + messages.add(AIGuard.Message.tool('call_1', 'dir1, dir2, dir3')) + } + 1 * span.setMetaStruct(AIGuardInternal.META_STRUCT_TAG, _ as Map) >> { + receivedMeta = it[1] as Map + return span + } + final metaStructMessages = receivedMeta.messages as List + metaStructMessages.size() != messages.size() + metaStructMessages.size() == 1 + metaStructMessages.first().toolCalls.size() != messages.first().toolCalls.size() + metaStructMessages.first().toolCalls.size() == 1 + } + + private AIGuardInternal mockClient(final int status, final Object response) { def request - final call = Mock(Call) { + final call = Stub(Call) { execute() >> { - return mockResponse( - request, - 200, - [data: [attributes: [action: 'ALLOW', reason: 'Just do it']]] - ) + return mockResponse(request, status, response) } } - final client = Mock(OkHttpClient) { + final client = Stub(OkHttpClient) { newCall(_ as Request) >> { request = (Request) it[0] return call } } - final aiguard = new AIGuardInternal(URL, HEADERS, client) - - when: - aiguard.evaluate([AIGuard.Message.tool('call_1', 'Content')], AIGuard.Options.DEFAULT) - - then: - 1 * span.setTag(AIGuardInternal.TARGET_TAG, 'tool') - 0 * span.setTag(AIGuardInternal.TOOL_TAG, _) + return new AIGuardInternal(URL, HEADERS, client) } private static assertTelemetry(final String metric, final String...tags) { @@ -459,6 +554,19 @@ class AIGuardInternalTests extends DDSpecification { return true } + private static assertMeta(final Map meta, final TestSuite suite) { + if (suite.tags) { + assert meta.attack_categories == suite.tags + } + if (suite.tagProbabilities) { + assert meta.tag_probs == suite.tagProbabilities + } + final receivedMessages = snakeCaseJson(meta.messages) + final expectedMessages = snakeCaseJson(suite.messages) + JSONAssert.assertEquals(expectedMessages, receivedMessages, JSONCompareMode.NON_EXTENSIBLE) + return true + } + private static assertRequest(final Request request, final List messages) { assert request.url() == URL assert request.method() == 'POST' @@ -494,17 +602,195 @@ class AIGuardInternalTests extends DDSpecification { .build() } + void 'test JSON serialization with text content parts'() { + given: + final aiguard = mockClient(200, [data: [attributes: [action: 'ALLOW', reason: 'Good']]]) + final messages = [AIGuard.Message.message('user', [AIGuard.ContentPart.text('Hello world')])] + + when: + aiguard.evaluate(messages, AIGuard.Options.DEFAULT) + + then: + 1 * span.setMetaStruct(AIGuardInternal.META_STRUCT_TAG, _) >> { + final meta = it[1] as Map + final receivedMessages = meta.messages as List + assert receivedMessages.size() == 1 + assert receivedMessages[0].contentParts.size() == 1 + assert receivedMessages[0].contentParts[0].type == AIGuard.ContentPart.Type.TEXT + assert receivedMessages[0].contentParts[0].text == 'Hello world' + return span + } + } + + void 'test JSON serialization with image_url content parts'() { + given: + final aiguard = mockClient(200, [data: [attributes: [action: 'ALLOW', reason: 'Good']]]) + final messages = [ + AIGuard.Message.message('user', [AIGuard.ContentPart.imageUrl('https://example.com/image.jpg')]) + ] + + when: + aiguard.evaluate(messages, AIGuard.Options.DEFAULT) + + then: + 1 * span.setMetaStruct(AIGuardInternal.META_STRUCT_TAG, _) >> { + final meta = it[1] as Map + final receivedMessages = meta.messages as List + assert receivedMessages.size() == 1 + assert receivedMessages[0].contentParts.size() == 1 + assert receivedMessages[0].contentParts[0].type == AIGuard.ContentPart.Type.IMAGE_URL + assert receivedMessages[0].contentParts[0].imageUrl.url == 'https://example.com/image.jpg' + return span + } + } + + void 'test JSON serialization with mixed content parts'() { + given: + final aiguard = mockClient(200, [data: [attributes: [action: 'ALLOW', reason: 'Good']]]) + final messages = [ + AIGuard.Message.message('user', [ + AIGuard.ContentPart.text('Describe this image:'), + AIGuard.ContentPart.imageUrl('https://example.com/image.jpg'), + AIGuard.ContentPart.text('What do you see?') + ]) + ] + + when: + aiguard.evaluate(messages, AIGuard.Options.DEFAULT) + + then: + 1 * span.setMetaStruct(AIGuardInternal.META_STRUCT_TAG, _) >> { + final meta = it[1] as Map + final receivedMessages = meta.messages as List + assert receivedMessages.size() == 1 + assert receivedMessages[0].contentParts.size() == 3 + assert receivedMessages[0].contentParts[0].type == AIGuard.ContentPart.Type.TEXT + assert receivedMessages[0].contentParts[0].text == 'Describe this image:' + assert receivedMessages[0].contentParts[1].type == AIGuard.ContentPart.Type.IMAGE_URL + assert receivedMessages[0].contentParts[1].imageUrl.url == 'https://example.com/image.jpg' + assert receivedMessages[0].contentParts[2].type == AIGuard.ContentPart.Type.TEXT + assert receivedMessages[0].contentParts[2].text == 'What do you see?' + return span + } + } + + void 'test content parts order is preserved'() { + given: + final aiguard = mockClient(200, [data: [attributes: [action: 'ALLOW', reason: 'Good']]]) + final parts = (0..4).collect { + it % 2 == 0 ? AIGuard.ContentPart.text("Text $it") : AIGuard.ContentPart.imageUrl("https://example.com/image${it}.jpg") + } + final messages = [AIGuard.Message.message('user', parts)] + + when: + aiguard.evaluate(messages, AIGuard.Options.DEFAULT) + + then: + 1 * span.setMetaStruct(AIGuardInternal.META_STRUCT_TAG, _) >> { + final meta = it[1] as Map + final receivedMessages = meta.messages as List + assert receivedMessages[0].contentParts.size() == 5 + (0..4).each { i -> + if (i % 2 == 0) { + assert receivedMessages[0].contentParts[i].type == AIGuard.ContentPart.Type.TEXT + assert receivedMessages[0].contentParts[i].text == "Text $i" + } else { + assert receivedMessages[0].contentParts[i].type == AIGuard.ContentPart.Type.IMAGE_URL + assert receivedMessages[0].contentParts[i].imageUrl.url == "https://example.com/image${i}.jpg" + } + } + return span + } + } + + void 'test content part text truncation'() { + given: + final maxContent = Config.get().getAiGuardMaxContentSize() + final aiguard = mockClient(200, [data: [attributes: [action: 'ALLOW', reason: 'Good']]]) + final longText = (0..maxContent).collect { 'A' }.join() + final messages = [ + AIGuard.Message.message('user', [AIGuard.ContentPart.text(longText), AIGuard.ContentPart.text('Short text')]) + ] + + when: + aiguard.evaluate(messages, AIGuard.Options.DEFAULT) + + then: + 1 * span.setMetaStruct(AIGuardInternal.META_STRUCT_TAG, _) >> { + final meta = it[1] as Map + final receivedMessages = meta.messages as List + assert receivedMessages[0].contentParts.size() == 2 + assert receivedMessages[0].contentParts[0].text.length() == maxContent + assert receivedMessages[0].contentParts[0].text.length() < longText.length() + assert receivedMessages[0].contentParts[1].text == 'Short text' + return span + } + assertTelemetry('ai_guard.truncated', 'type:content') + } + + void 'test content part image_url not truncated even with long data URI'() { + given: + final maxContent = Config.get().getAiGuardMaxContentSize() + final aiguard = mockClient(200, [data: [attributes: [action: 'ALLOW', reason: 'Good']]]) + // Create a very long data URI (longer than max content size) + final longDataUri = 'data:image/png;base64,' + (0..(maxContent + 1000)).collect { 'A' }.join() + final messages = [ + AIGuard.Message.message('user', [ + AIGuard.ContentPart.text('Image:'), + AIGuard.ContentPart.imageUrl(longDataUri) + ]) + ] + + when: + aiguard.evaluate(messages, AIGuard.Options.DEFAULT) + + then: + 1 * span.setMetaStruct(AIGuardInternal.META_STRUCT_TAG, _) >> { + final meta = it[1] as Map + final receivedMessages = meta.messages as List + assert receivedMessages[0].contentParts.size() == 2 + assert receivedMessages[0].contentParts[1].type == AIGuard.ContentPart.Type.IMAGE_URL + // Image URL should NOT be truncated + assert receivedMessages[0].contentParts[1].imageUrl.url == longDataUri + assert receivedMessages[0].contentParts[1].imageUrl.url.length() > maxContent + return span + } + } + + void 'test backward compatibility with string content'() { + given: + final aiguard = mockClient(200, [data: [attributes: [action: 'ALLOW', reason: 'Good']]]) + final messages = [AIGuard.Message.message('user', 'Hello world')] + + when: + aiguard.evaluate(messages, AIGuard.Options.DEFAULT) + + then: + 1 * span.setMetaStruct(AIGuardInternal.META_STRUCT_TAG, _) >> { + final meta = it[1] as Map + final receivedMessages = meta.messages as List + assert receivedMessages.size() == 1 + assert receivedMessages[0].content == 'Hello world' + assert receivedMessages[0].contentParts == null + return span + } + } + private static class TestSuite { private final AIGuard.Action action private final String reason + private final List tags + private final Map tagProbabilities private final boolean blocking private final String description private final String target private final List messages - TestSuite(AIGuard.Action action, String reason, boolean blocking, String description, String target, List messages) { + TestSuite(AIGuard.Action action, String reason, Map tagProbabilities, boolean blocking, String description, String target, List messages) { this.action = action this.reason = reason + this.tags = new ArrayList<>(tagProbabilities.keySet()) + this.tagProbabilities = tagProbabilities this.blocking = blocking this.description = description this.target = target @@ -512,7 +798,11 @@ class AIGuardInternalTests extends DDSpecification { } static List build() { - def actionValues = [[ALLOW, 'Go ahead'], [DENY, 'Nope'], [ABORT, 'Kill it with fire']] + def actionValues = [ + [ALLOW, 'Go ahead', [:]], + [DENY, 'Nope', ['deny_everything': 0.2D, 'test_deny': 0.8D]], + [ABORT, 'Kill it with fire', ['alarm_tag': 0.1D, 'abort_everything': 0.9D]] + ] def blockingValues = [true, false] def suiteValues = [ ['tool call', 'tool', TOOL_CALL], @@ -521,7 +811,7 @@ class AIGuardInternalTests extends DDSpecification { ] return combinations([actionValues, blockingValues, suiteValues] as Iterable) .collect { action, blocking, suite -> - new TestSuite(action[0], action[1], blocking, suite[0], suite[1], suite[2]) + new TestSuite(action[0], action[1], action[2], blocking, suite[0], suite[1], suite[2]) } } @@ -534,7 +824,8 @@ class AIGuardInternalTests extends DDSpecification { ", reason='" + reason + '\'' + ", blocking=" + blocking + ", target='" + target + '\'' + - ", messages=" + messages + + ", messages=" + messages.collect {it.content } + '\'' + + ", tags=" + tags + '}' } } diff --git a/dd-java-agent/agent-bootstrap/build.gradle b/dd-java-agent/agent-bootstrap/build.gradle index 1ee2d9a6cde..de0d326341e 100644 --- a/dd-java-agent/agent-bootstrap/build.gradle +++ b/dd-java-agent/agent-bootstrap/build.gradle @@ -1,5 +1,3 @@ -import de.thetaphi.forbiddenapis.gradle.CheckForbiddenApis - // The shadowJar of this project will be injected into the JVM's bootstrap classloader plugins { id 'com.gradleup.shadow' @@ -25,13 +23,13 @@ dependencies { api project(':dd-java-agent:agent-debugger:debugger-bootstrap') api project(':components:environment') api project(':components:json') + api project(':products:metrics:metrics-agent') + api libs.instrument.java api libs.slf4j // ^ Generally a bad idea for libraries, but we're shadowing. - api(variantOf(libs.instrumentjava, { classifier("all") })) { - transitive = false - } testImplementation project(':dd-java-agent:testing') + testImplementation group: 'com.google.guava', name: 'guava-testlib', version: '20.0' } // Must use Java 11 to build JFR enabled code - there is no JFR in OpenJDK 8 (revisit once JFR in Java 8 is available) @@ -54,10 +52,6 @@ tasks.named("jar", Jar) { from sourceSets.main_java11.output } -tasks.named("forbiddenApisMain_java11", CheckForbiddenApis) { - failOnMissingClasses = false -} - idea { module { jdkName = '11' @@ -70,10 +64,9 @@ jmh { } tasks.withType(Test).configureEach { - configureJvmArgs( + conditionalJvmArgs( it, JavaVersion.VERSION_16, ['--add-opens', 'java.base/java.net=ALL-UNNAMED'] // for HostNameResolverForkedTest ) - } diff --git a/dd-java-agent/agent-bootstrap/gradle.lockfile b/dd-java-agent/agent-bootstrap/gradle.lockfile index 55776a9516f..0ba47412aa0 100644 --- a/dd-java-agent/agent-bootstrap/gradle.lockfile +++ b/dd-java-agent/agent-bootstrap/gradle.lockfile @@ -5,125 +5,94 @@ cafe.cryptography:curve25519-elisabeth:0.1.0=jmhRuntimeClasspath,testRuntimeClas cafe.cryptography:ed25519-elisabeth:0.1.0=jmhRuntimeClasspath,testRuntimeClasspath ch.qos.logback:logback-classic:1.2.13=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath ch.qos.logback:logback-core:1.2.13=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -com.beust:jcommander:1.78=jmhRuntimeClasspath,testRuntimeClasspath com.blogspot.mydailyjava:weak-lock-free:0.17=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath com.datadoghq.okhttp3:okhttp:3.12.15=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath com.datadoghq.okio:okio:1.17.6=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -com.datadoghq:dd-instrument-java:0.0.2=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +com.datadoghq:dd-instrument-java:0.0.3=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.datadoghq:dd-javac-plugin-client:0.2.2=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,main_java11CompileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.datadoghq:java-dogstatsd-client:4.4.3=jmhRuntimeClasspath,testRuntimeClasspath +com.datadoghq:java-dogstatsd-client:4.4.5=jmhRuntimeClasspath,testRuntimeClasspath com.datadoghq:sketches-java:0.8.3=jmhRuntimeClasspath,testRuntimeClasspath -com.github.javaparser:javaparser-core:3.25.6=codenarc,jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -com.github.jnr:jffi:1.3.13=jmhRuntimeClasspath,testRuntimeClasspath +com.github.javaparser:javaparser-core:3.25.6=codenarc +com.github.jnr:jffi:1.3.14=jmhRuntimeClasspath,testRuntimeClasspath com.github.jnr:jnr-a64asm:1.0.0=jmhRuntimeClasspath,testRuntimeClasspath com.github.jnr:jnr-constants:0.10.4=jmhRuntimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-enxio:0.32.17=jmhRuntimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-ffi:2.2.16=jmhRuntimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-posix:3.1.19=jmhRuntimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-unixsocket:0.38.22=jmhRuntimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-enxio:0.32.19=jmhRuntimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-ffi:2.2.18=jmhRuntimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-posix:3.1.21=jmhRuntimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-unixsocket:0.38.24=jmhRuntimeClasspath,testRuntimeClasspath com.github.jnr:jnr-x86asm:1.0.2=jmhRuntimeClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.2.0=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.7.3=spotbugs -com.github.spotbugs:spotbugs:4.7.3=spotbugs +com.github.spotbugs:spotbugs-annotations:4.9.8=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath +com.github.spotbugs:spotbugs:4.9.8=spotbugs +com.github.stephenc.jcip:jcip-annotations:1.0-1=spotbugs com.google.code.findbugs:jsr305:3.0.2=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -com.google.code.gson:gson:2.9.1=spotbugs +com.google.code.gson:gson:2.13.2=spotbugs +com.google.errorprone:error_prone_annotations:2.0.12=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +com.google.errorprone:error_prone_annotations:2.41.0=spotbugs +com.google.guava:guava-testlib:20.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath com.google.guava:guava:20.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath com.google.re2j:re2j:1.7=jmhRuntimeClasspath,testRuntimeClasspath com.squareup.moshi:moshi:1.11.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath com.squareup.okhttp3:logging-interceptor:3.12.12=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath com.squareup.okhttp3:okhttp:3.12.12=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath com.squareup.okio:okio:1.17.5=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -com.thoughtworks.qdox:qdox:1.12.1=codenarc,jmhRuntimeClasspath,testRuntimeClasspath -commons-codec:commons-codec:1.15=spotbugs +com.thoughtworks.qdox:qdox:1.12.1=codenarc commons-fileupload:commons-fileupload:1.5=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath commons-io:commons-io:2.11.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -de.thetaphi:forbiddenapis:3.8=compileClasspath,jmhCompileClasspath -info.picocli:picocli:4.6.3=jmhRuntimeClasspath,testRuntimeClasspath +commons-io:commons-io:2.20.0=spotbugs +de.thetaphi:forbiddenapis:3.10=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath io.leangen.geantyref:geantyref:1.3.16=jmhRuntimeClasspath,testRuntimeClasspath -io.sqreen:libsqreen:17.2.0=jmhRuntimeClasspath,testRuntimeClasspath +io.sqreen:libsqreen:17.3.0=jmhRuntimeClasspath,testRuntimeClasspath javax.servlet:javax.servlet-api:3.1.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -jaxen:jaxen:1.2.0=spotbugs -jline:jline:2.14.6=jmhRuntimeClasspath,testRuntimeClasspath -junit:junit:4.13.2=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -net.bytebuddy:byte-buddy-agent:1.17.7=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -net.bytebuddy:byte-buddy:1.17.7=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +jaxen:jaxen:2.0.0=spotbugs +junit:junit:4.13.2=jmhRuntimeClasspath,testRuntimeClasspath +junit:junit:4.8.2=testCompileClasspath +net.bytebuddy:byte-buddy-agent:1.18.8=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +net.bytebuddy:byte-buddy:1.18.8=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath net.java.dev.jna:jna-platform:5.8.0=jmhRuntimeClasspath,testRuntimeClasspath net.java.dev.jna:jna:5.8.0=jmhRuntimeClasspath,testRuntimeClasspath -net.jcip:jcip-annotations:1.0=compileClasspath,jmhCompileClasspath,jmhRuntimeClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath net.sf.jopt-simple:jopt-simple:5.0.4=jmh,jmhCompileClasspath,jmhRuntimeClasspath -net.sf.saxon:Saxon-HE:11.4=spotbugs +net.sf.saxon:Saxon-HE:12.9=spotbugs org.apache.ant:ant-antlr:1.10.14=codenarc -org.apache.ant:ant-antlr:1.10.15=jmhRuntimeClasspath,testRuntimeClasspath org.apache.ant:ant-junit:1.10.14=codenarc -org.apache.ant:ant-junit:1.10.15=jmhRuntimeClasspath,testRuntimeClasspath -org.apache.ant:ant-launcher:1.10.15=jmhRuntimeClasspath,testRuntimeClasspath -org.apache.ant:ant:1.10.15=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.apache.bcel:bcel:6.5.0=spotbugs -org.apache.commons:commons-lang3:3.12.0=spotbugs +org.apache.bcel:bcel:6.11.0=spotbugs +org.apache.commons:commons-lang3:3.19.0=spotbugs org.apache.commons:commons-math3:3.6.1=jmh,jmhCompileClasspath,jmhRuntimeClasspath -org.apache.commons:commons-text:1.10.0=spotbugs -org.apache.httpcomponents.client5:httpclient5:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5-h2:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5:5.1.3=spotbugs -org.apache.logging.log4j:log4j-api:2.19.0=spotbugs -org.apache.logging.log4j:log4j-core:2.19.0=spotbugs +org.apache.commons:commons-text:1.14.0=spotbugs +org.apache.logging.log4j:log4j-api:2.25.2=spotbugs +org.apache.logging.log4j:log4j-core:2.25.2=spotbugs org.apiguardian:apiguardian-api:1.1.2=testCompileClasspath -org.codehaus.groovy:groovy-all:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-ant:3.0.23=codenarc -org.codehaus.groovy:groovy-ant:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-astbuilder:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-cli-picocli:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-console:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-datetime:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-docgenerator:3.0.23=codenarc -org.codehaus.groovy:groovy-docgenerator:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-groovydoc:3.0.23=codenarc -org.codehaus.groovy:groovy-groovydoc:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-groovysh:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jmx:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-json:3.0.23=codenarc -org.codehaus.groovy:groovy-json:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jsr223:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-macro:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-nio:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-servlet:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-sql:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-swing:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.codehaus.groovy:groovy-json:3.0.25=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-templates:3.0.23=codenarc -org.codehaus.groovy:groovy-templates:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test-junit5:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-testng:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-xml:3.0.23=codenarc -org.codehaus.groovy:groovy-xml:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy:3.0.23=codenarc -org.codehaus.groovy:groovy:3.0.24=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codenarc:CodeNarc:3.6.0=codenarc -org.dom4j:dom4j:2.1.3=spotbugs -org.eclipse.jetty:jetty-http:9.4.56.v20240826=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.eclipse.jetty:jetty-io:9.4.56.v20240826=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.eclipse.jetty:jetty-server:9.4.56.v20240826=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.eclipse.jetty:jetty-util:9.4.56.v20240826=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.codehaus.groovy:groovy:3.0.25=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.codenarc:CodeNarc:3.7.0=codenarc +org.dom4j:dom4j:2.2.0=spotbugs org.gmetrics:GMetrics:2.1.0=codenarc -org.hamcrest:hamcrest-core:1.3=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.hamcrest:hamcrest-core:1.3=jmhRuntimeClasspath,testRuntimeClasspath org.hamcrest:hamcrest:3.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.jacoco:org.jacoco.agent:0.8.14=jacocoAgent,jacocoAnt org.jacoco:org.jacoco.ant:0.8.14=jacocoAnt org.jacoco:org.jacoco.core:0.8.14=jacocoAnt org.jacoco:org.jacoco.report:0.8.14=jacocoAnt -org.jctools:jctools-core:3.3.0=jmhRuntimeClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-api:5.12.2=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-engine:5.12.2=jmhRuntimeClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-params:5.12.2=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter:5.12.2=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-commons:1.12.2=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-engine:1.12.2=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-launcher:1.12.2=jmhRuntimeClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-runner:1.12.2=jmhRuntimeClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-suite-api:1.12.2=jmhRuntimeClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-suite-commons:1.12.2=jmhRuntimeClasspath,testRuntimeClasspath -org.junit:junit-bom:5.12.2=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit:junit-bom:5.9.1=spotbugs +org.jctools:jctools-core-jdk11:4.0.6=jmhRuntimeClasspath,testRuntimeClasspath +org.jctools:jctools-core:4.0.6=jmhRuntimeClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-api:5.14.1=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-engine:5.14.1=jmhRuntimeClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-params:5.14.1=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter:5.14.1=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-commons:1.14.1=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-engine:1.14.1=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-launcher:1.14.1=jmhRuntimeClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-runner:1.14.1=jmhRuntimeClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-suite-api:1.14.1=jmhRuntimeClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-suite-commons:1.14.1=jmhRuntimeClasspath,testRuntimeClasspath +org.junit:junit-bom:5.14.0=spotbugs +org.junit:junit-bom:5.14.1=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.mockito:mockito-core:4.4.0=jmhRuntimeClasspath,testRuntimeClasspath org.objenesis:objenesis:3.3=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.openjdk.jmh:jmh-core:1.37=jmh,jmhCompileClasspath,jmhRuntimeClasspath @@ -131,32 +100,28 @@ org.openjdk.jmh:jmh-generator-asm:1.37=jmh,jmhCompileClasspath,jmhRuntimeClasspa org.openjdk.jmh:jmh-generator-bytecode:1.37=jmh,jmhCompileClasspath,jmhRuntimeClasspath org.openjdk.jmh:jmh-generator-reflection:1.37=jmh,jmhCompileClasspath,jmhRuntimeClasspath org.opentest4j:opentest4j:1.3.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.2=jmhRuntimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.4=spotbugs -org.ow2.asm:asm-commons:9.2=jmhRuntimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-commons:9.4=spotbugs -org.ow2.asm:asm-commons:9.9=jacocoAnt -org.ow2.asm:asm-tree:9.2=jmhRuntimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-tree:9.4=spotbugs -org.ow2.asm:asm-tree:9.9=jacocoAnt -org.ow2.asm:asm-util:9.2=jmhRuntimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-util:9.4=spotbugs +org.ow2.asm:asm-analysis:9.7.1=jmhRuntimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-analysis:9.9=spotbugs +org.ow2.asm:asm-commons:9.7.1=jmhRuntimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-commons:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-tree:9.7.1=jmhRuntimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-tree:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-util:9.7.1=jmhRuntimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-util:9.9=spotbugs org.ow2.asm:asm:9.0=jmh,jmhCompileClasspath -org.ow2.asm:asm:9.2=jmhRuntimeClasspath,testRuntimeClasspath -org.ow2.asm:asm:9.4=spotbugs -org.ow2.asm:asm:9.9=jacocoAnt +org.ow2.asm:asm:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm:9.9.1=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.slf4j:jcl-over-slf4j:1.7.30=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.slf4j:jul-to-slf4j:1.7.30=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.slf4j:log4j-over-slf4j:1.7.30=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath org.slf4j:slf4j-api:1.7.30=compileClasspath,jmhCompileClasspath,main_java11CompileClasspath,runtimeClasspath org.slf4j:slf4j-api:1.7.32=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.slf4j:slf4j-api:2.0.0=spotbugs,spotbugsSlf4j -org.slf4j:slf4j-simple:2.0.0=spotbugsSlf4j +org.slf4j:slf4j-api:2.0.17=spotbugs,spotbugsSlf4j +org.slf4j:slf4j-simple:2.0.17=spotbugsSlf4j org.snakeyaml:snakeyaml-engine:2.9=jmhRuntimeClasspath,runtimeClasspath,testRuntimeClasspath -org.spockframework:spock-bom:2.4-M6-groovy-3.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.spockframework:spock-core:2.4-M6-groovy-3.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath -org.testng:testng:7.5.1=jmhRuntimeClasspath,testRuntimeClasspath -org.webjars:jquery:3.5.1=jmhRuntimeClasspath,testRuntimeClasspath -org.xmlresolver:xmlresolver:4.4.3=spotbugs -xml-apis:xml-apis:1.4.01=spotbugs +org.spockframework:spock-bom:2.4-groovy-3.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.spockframework:spock-core:2.4-groovy-3.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-junit:1.2.1=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-parser:1.2.0=jmhRuntimeClasspath,testCompileClasspath,testRuntimeClasspath +org.xmlresolver:xmlresolver:5.3.3=spotbugs empty=annotationProcessor,jmhAnnotationProcessor,main_java11AnnotationProcessor,main_java11RuntimeClasspath,shadow,spotbugsPlugins,testAnnotationProcessor diff --git a/dd-java-agent/agent-bootstrap/src/jmh/java/datadog/trace/bootstrap/ExceptionLoggerBenchmark.java b/dd-java-agent/agent-bootstrap/src/jmh/java/datadog/trace/bootstrap/ExceptionLoggerBenchmark.java new file mode 100644 index 00000000000..807ac7c196c --- /dev/null +++ b/dd-java-agent/agent-bootstrap/src/jmh/java/datadog/trace/bootstrap/ExceptionLoggerBenchmark.java @@ -0,0 +1,28 @@ +package datadog.trace.bootstrap; + +import org.openjdk.jmh.annotations.Benchmark; +import org.openjdk.jmh.annotations.Fork; +import org.openjdk.jmh.annotations.Measurement; +import org.openjdk.jmh.annotations.Threads; +import org.openjdk.jmh.annotations.Warmup; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Benchmark showing impact of ExceptionLogger + * + *

NOTE: This benchmark exists to check the efficiency of retrieving the ExceptionLogger. + * Previously, this caused significant allocation. + */ +@Fork(2) +@Warmup(iterations = 2) +@Measurement(iterations = 5) +@Threads(8) +public class ExceptionLoggerBenchmark { + @Benchmark + public Logger getExceptionLogger() { + // This matches what happens in the bytecode weaving that defends against + // exception leaking out of instrumentation. + return LoggerFactory.getLogger(ExceptionLogger.class); + } +} diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Agent.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Agent.java index 551a4e7e527..b45e54b0b03 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Agent.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Agent.java @@ -2,6 +2,7 @@ import static datadog.environment.JavaVirtualMachine.isJavaVersionAtLeast; import static datadog.environment.JavaVirtualMachine.isOracleJDK8; +import static datadog.trace.api.Config.isExplicitlyDisabled; import static datadog.trace.api.ConfigDefaults.DEFAULT_STARTUP_LOGS_ENABLED; import static datadog.trace.api.config.GeneralConfig.DATA_JOBS_COMMAND_PATTERN; import static datadog.trace.api.config.GeneralConfig.DATA_JOBS_ENABLED; @@ -22,9 +23,10 @@ import datadog.environment.OperatingSystem; import datadog.environment.SystemProperties; import datadog.instrument.classinject.ClassInjector; +import datadog.instrument.utils.ClassLoaderValue; +import datadog.metrics.api.statsd.StatsDClientManager; import datadog.trace.api.Config; import datadog.trace.api.Platform; -import datadog.trace.api.StatsDClientManager; import datadog.trace.api.WithGlobalTracer; import datadog.trace.api.appsec.AppSecEventTracker; import datadog.trace.api.config.AppSecConfig; @@ -32,6 +34,7 @@ import datadog.trace.api.config.CrashTrackingConfig; import datadog.trace.api.config.CwsConfig; import datadog.trace.api.config.DebuggerConfig; +import datadog.trace.api.config.FeatureFlaggingConfig; import datadog.trace.api.config.GeneralConfig; import datadog.trace.api.config.IastConfig; import datadog.trace.api.config.JmxFetchConfig; @@ -45,6 +48,7 @@ import datadog.trace.api.gateway.SubscriptionService; import datadog.trace.api.git.EmbeddedGitInfoBuilder; import datadog.trace.api.git.GitInfoProvider; +import datadog.trace.api.intake.Intake; import datadog.trace.api.profiling.ProfilingEnablement; import datadog.trace.api.scopemanager.ScopeListener; import datadog.trace.bootstrap.benchmark.StaticEventLogger; @@ -57,6 +61,9 @@ import datadog.trace.util.AgentTaskScheduler; import datadog.trace.util.AgentThreadFactory.AgentThread; import datadog.trace.util.throwable.FatalAgentMisconfigurationError; +import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import java.lang.instrument.Instrumentation; import java.lang.reflect.InvocationTargetException; import java.lang.reflect.Method; @@ -100,6 +107,8 @@ public class Agent { private static final int DEFAULT_JMX_START_DELAY = 15; // seconds + private static final long CLASSLOADER_CLEAN_FREQUENCY_SECONDS = 30; + private static final Logger log; private enum AgentFeature { @@ -124,8 +133,10 @@ private enum AgentFeature { CODE_ORIGIN(TraceInstrumentationConfig.CODE_ORIGIN_FOR_SPANS_ENABLED, false), DATA_JOBS(GeneralConfig.DATA_JOBS_ENABLED, false), AGENTLESS_LOG_SUBMISSION(GeneralConfig.AGENTLESS_LOG_SUBMISSION_ENABLED, false), + APP_LOGS_COLLECTION(GeneralConfig.APP_LOGS_COLLECTION_ENABLED, false), LLMOBS(LlmObsConfig.LLMOBS_ENABLED, false), - LLMOBS_AGENTLESS(LlmObsConfig.LLMOBS_AGENTLESS_ENABLED, false); + LLMOBS_AGENTLESS(LlmObsConfig.LLMOBS_AGENTLESS_ENABLED, false), + FEATURE_FLAGGING(FeatureFlaggingConfig.FLAGGING_PROVIDER_ENABLED, false); private final String configKey; private final String systemProp; @@ -184,6 +195,8 @@ public boolean isEnabledByDefault() { private static boolean codeOriginEnabled = false; private static boolean distributedDebuggerEnabled = false; private static boolean agentlessLogSubmissionEnabled = false; + private static boolean appLogsCollectionEnabled = false; + private static boolean featureFlaggingEnabled = false; private static void safelySetContextClassLoader(ClassLoader classLoader) { try { @@ -199,6 +212,7 @@ private static void safelySetContextClassLoader(ClassLoader classLoader) { *

The Agent is considered to start successfully if Instrumentation can be activated. All other * pieces are considered optional. */ + @SuppressFBWarnings("AT_STALE_THREAD_WRITE_OF_PRIMITIVE") public static void start( final Object bootstrapInitTelemetry, final Instrumentation inst, @@ -267,7 +281,9 @@ public static void start( exceptionReplayEnabled = isFeatureEnabled(AgentFeature.EXCEPTION_REPLAY); codeOriginEnabled = isFeatureEnabled(AgentFeature.CODE_ORIGIN); agentlessLogSubmissionEnabled = isFeatureEnabled(AgentFeature.AGENTLESS_LOG_SUBMISSION); + appLogsCollectionEnabled = isFeatureEnabled(AgentFeature.APP_LOGS_COLLECTION); llmObsEnabled = isFeatureEnabled(AgentFeature.LLMOBS); + featureFlaggingEnabled = isFeatureEnabled(AgentFeature.FEATURE_FLAGGING); // setup writers when llmobs is enabled to accomodate apm and llmobs if (llmObsEnabled) { @@ -321,6 +337,7 @@ public static void start( startCrashTracking(); StaticEventLogger.end("crashtracking"); } + startDatadogAgent(initTelemetry, inst); final EnumSet libraries = detectLibraries(log); @@ -404,6 +421,15 @@ public static void start( StaticEventLogger.end("Profiling"); } + // This task removes stale ClassLoaderValue entries where the class-loader is gone + // It only runs a couple of times a minute since class-loaders are rarely unloaded + AgentTaskScheduler.get() + .scheduleAtFixedRate( + ClassLoaderValue::removeStaleEntries, + CLASSLOADER_CLEAN_FREQUENCY_SECONDS, + CLASSLOADER_CLEAN_FREQUENCY_SECONDS, + TimeUnit.SECONDS); + StaticEventLogger.end("Agent.start"); } @@ -456,6 +482,7 @@ private static void injectAgentArgsConfig(String agentArgs) { } } + @SuppressFBWarnings("AT_STALE_THREAD_WRITE_OF_PRIMITIVE") private static void configureCiVisibility(URL agentJarURL) { // Retro-compatibility for the old way to configure CI Visibility if ("true".equals(ddGetProperty("dd.integration.junit.enabled")) @@ -639,6 +666,7 @@ public InstallDatadogTracerCallback( throw new UndeclaredThrowableException(e); } + installDatadogMeter(initTelemetry); installDatadogTracer(initTelemetry, scoClass, sco); maybeInstallLogsIntake(scoClass, sco); maybeStartIast(instrumentation); @@ -662,6 +690,7 @@ public void execute() { maybeStartDebugger(instrumentation, scoClass, sco); maybeStartRemoteConfig(scoClass, sco); maybeStartAiGuard(); + maybeStartFeatureFlagging(scoClass, sco); if (telemetryEnabled) { startTelemetry(instrumentation, scoClass, sco); @@ -770,6 +799,29 @@ private static synchronized void startDatadogAgent( } } + private static synchronized void installDatadogMeter(InitializationTelemetry initTelemetry) { + if (AGENT_CLASSLOADER == null) { + throw new IllegalStateException("Datadog agent should have been started already"); + } + + StaticEventLogger.begin("AgentMeter"); + + try { + // Install AgentMeter, StatsDClient and Monitoring + final Class tracerInstallerClass = + AGENT_CLASSLOADER.loadClass("datadog.trace.agent.tooling.MeterInstaller"); + final Method installMeterMethod = tracerInstallerClass.getMethod("installMeter"); + installMeterMethod.invoke(null); + } catch (final FatalAgentMisconfigurationError ex) { + throw ex; + } catch (final Throwable ex) { + log.error("Throwable thrown while installing the Datadog meter", ex); + initTelemetry.onFatalError(ex); + } + + StaticEventLogger.end("AgentMeter"); + } + private static synchronized void installDatadogTracer( InitializationTelemetry initTelemetry, Class scoClass, Object sco) { if (AGENT_CLASSLOADER == null) { @@ -903,6 +955,15 @@ private static synchronized void registerDeadlockDetectionEvent() { private static synchronized void registerSmapEntryEvent() { log.debug("Initializing smap entry scraping"); + + // Load JFR Handlers class early, if present (it has been moved and renamed in JDK23+). + // This prevents a deadlock. See https://bugs.openjdk.org/browse/JDK-8371889. + try { + AGENT_CLASSLOADER.loadClass("jdk.jfr.events.Handlers"); + } catch (Exception e) { + // Ignore when the class is not found or anything else goes wrong. + } + try { final Class smapFactoryClass = AGENT_CLASSLOADER.loadClass( @@ -948,7 +1009,7 @@ private static synchronized void startJmxFetch() { private static StatsDClientManager statsDClientManager() throws Exception { final Class statsdClientManagerClass = - AGENT_CLASSLOADER.loadClass("datadog.communication.monitor.DDAgentStatsDClientManager"); + AGENT_CLASSLOADER.loadClass("datadog.metrics.impl.statsd.DDAgentStatsDClientManager"); final Method statsDClientManagerMethod = statsdClientManagerClass.getMethod("statsDClientManager"); return (StatsDClientManager) statsDClientManagerMethod.invoke(null); @@ -1083,16 +1144,34 @@ private static void maybeStartLLMObs(Instrumentation inst, Class scoClass, Ob } } + private static void maybeStartFeatureFlagging(final Class scoClass, final Object sco) { + if (featureFlaggingEnabled) { + StaticEventLogger.begin("Feature Flagging"); + + try { + final Class ffSysClass = + AGENT_CLASSLOADER.loadClass("com.datadog.featureflag.FeatureFlaggingSystem"); + final Method ffSysMethod = ffSysClass.getMethod("start", scoClass); + ffSysMethod.invoke(null, sco); + } catch (final Throwable e) { + log.warn("Not starting Feature Flagging subsystem", e); + } + + StaticEventLogger.end("Feature Flagging"); + } + } + private static void maybeInstallLogsIntake(Class scoClass, Object sco) { - if (agentlessLogSubmissionEnabled) { + if (agentlessLogSubmissionEnabled || appLogsCollectionEnabled) { StaticEventLogger.begin("Logs Intake"); try { final Class logsIntakeSystemClass = AGENT_CLASSLOADER.loadClass("datadog.trace.logging.intake.LogsIntakeSystem"); final Method logsIntakeInstallerMethod = - logsIntakeSystemClass.getMethod("install", scoClass); - logsIntakeInstallerMethod.invoke(null, sco); + logsIntakeSystemClass.getMethod("install", scoClass, Intake.class); + logsIntakeInstallerMethod.invoke( + null, sco, agentlessLogSubmissionEnabled ? Intake.LOGS : Intake.EVENT_PLATFORM); } catch (final Throwable e) { log.warn("Not installing Logs Intake subsystem", e); } @@ -1201,10 +1280,6 @@ private static boolean isCrashTrackingAutoconfigEnabled() { } private static void initializeCrashTracking(boolean delayed, boolean checkNative) { - if (JavaVirtualMachine.isJ9()) { - // TODO currently crash tracking is supported only for HotSpot based JVMs - return; - } log.debug("Initializing crashtracking"); try { Class clz = AGENT_CLASSLOADER.loadClass("datadog.crashtracking.Initializer"); @@ -1381,11 +1456,6 @@ && isExplicitlyDisabled(DebuggerConfig.DISTRIBUTED_DEBUGGER_ENABLED)) { startDebuggerAgent(inst, scoClass, sco); } - private static boolean isExplicitlyDisabled(String booleanKey) { - return Config.get().configProvider().isSet(booleanKey) - && !Config.get().configProvider().getBoolean(booleanKey); - } - private static synchronized void startDebuggerAgent( Instrumentation inst, Class scoClass, Object sco) { StaticEventLogger.begin("Debugger"); @@ -1396,8 +1466,8 @@ private static synchronized void startDebuggerAgent( final Class debuggerAgentClass = AGENT_CLASSLOADER.loadClass("com.datadog.debugger.agent.DebuggerAgent"); final Method debuggerInstallerMethod = - debuggerAgentClass.getMethod("run", Instrumentation.class, scoClass); - debuggerInstallerMethod.invoke(null, inst, sco); + debuggerAgentClass.getMethod("run", Config.class, Instrumentation.class, scoClass); + debuggerInstallerMethod.invoke(null, Config.get(), inst, sco); } catch (final Throwable ex) { log.error("Throwable thrown while starting debugger agent", ex); } finally { @@ -1409,9 +1479,15 @@ private static synchronized void startDebuggerAgent( private static void configureLogger() { setSystemPropertyDefault(SIMPLE_LOGGER_SHOW_DATE_TIME_PROPERTY, "true"); - setSystemPropertyDefault(SIMPLE_LOGGER_JSON_ENABLED_PROPERTY, "false"); - String simpleLoggerJsonEnabled = SystemProperties.get(SIMPLE_LOGGER_JSON_ENABLED_PROPERTY); - if (simpleLoggerJsonEnabled != null && simpleLoggerJsonEnabled.equalsIgnoreCase("true")) { + + String logFormatJson = ddGetProperty("dd.log.format.json"); + if (null != logFormatJson) { + setSystemPropertyDefault(SIMPLE_LOGGER_JSON_ENABLED_PROPERTY, logFormatJson); + } else { + setSystemPropertyDefault(SIMPLE_LOGGER_JSON_ENABLED_PROPERTY, "false"); + } + + if (Boolean.parseBoolean(SystemProperties.get(SIMPLE_LOGGER_JSON_ENABLED_PROPERTY))) { setSystemPropertyDefault( SIMPLE_LOGGER_DATE_TIME_FORMAT_PROPERTY, SIMPLE_LOGGER_DATE_TIME_FORMAT_JSON_DEFAULT); } else { @@ -1423,9 +1499,12 @@ private static void configureLogger() { if (isDebugMode()) { logLevel = "DEBUG"; } else { - logLevel = ddGetProperty("dd.log.level"); + logLevel = ddGetProperty("dd.trace.log.level"); if (null == logLevel) { - logLevel = EnvironmentVariables.get("OTEL_LOG_LEVEL"); + logLevel = ddGetProperty("dd.log.level"); + if (null == logLevel) { + logLevel = EnvironmentVariables.get("OTEL_LOG_LEVEL"); + } } } @@ -1475,7 +1554,9 @@ private static boolean isDebugMode() { return false; } - /** @return {@code true} if the agent feature is enabled */ + /** + * @return {@code true} if the agent feature is enabled + */ private static boolean isFeatureEnabled(AgentFeature feature) { // must be kept in sync with logic from Config! final String featureConfigKey = feature.getConfigKey(); @@ -1505,7 +1586,9 @@ private static boolean isFeatureEnabled(AgentFeature feature) { } } - /** @see datadog.trace.api.ProductActivation#fromString(String) */ + /** + * @see datadog.trace.api.ProductActivation#fromString(String) + */ private static boolean isFullyDisabled(final AgentFeature feature) { // must be kept in sync with logic from Config! final String featureConfigKey = feature.getConfigKey(); @@ -1543,7 +1626,9 @@ private static String getNullIfEmpty(final String value) { return value; } - /** @return configured JMX start delay in seconds */ + /** + * @return configured JMX start delay in seconds + */ private static int getJmxStartDelay() { String startDelay = ddGetProperty("dd.dogstatsd.start-delay"); if (startDelay == null) { diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/AgentJarIndex.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/AgentJarIndex.java index cedc773bbfb..5b0daa95995 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/AgentJarIndex.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/AgentJarIndex.java @@ -1,6 +1,6 @@ package datadog.trace.bootstrap; -import datadog.trace.util.ClassNameTrie; +import datadog.instrument.utils.ClassNameTrie; import java.io.BufferedInputStream; import java.io.BufferedOutputStream; import java.io.DataInputStream; diff --git a/dd-java-agent/agent-tooling/src/main/java/datadog/trace/agent/tooling/ClassHierarchyIterable.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/ClassHierarchyIterable.java similarity index 98% rename from dd-java-agent/agent-tooling/src/main/java/datadog/trace/agent/tooling/ClassHierarchyIterable.java rename to dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/ClassHierarchyIterable.java index b91c8c1cd11..903ac6a11d8 100644 --- a/dd-java-agent/agent-tooling/src/main/java/datadog/trace/agent/tooling/ClassHierarchyIterable.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/ClassHierarchyIterable.java @@ -1,4 +1,4 @@ -package datadog.trace.agent.tooling; +package datadog.trace.bootstrap; import java.util.ArrayDeque; import java.util.HashSet; diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Constants.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Constants.java index d046667fed5..c756d45a654 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Constants.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Constants.java @@ -21,6 +21,7 @@ public final class Constants { "datadog.yaml", "datadog.instrument", "datadog.appsec.api", + "datadog.metrics.api", "datadog.trace.api", "datadog.trace.bootstrap", "datadog.trace.config.inversion", diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/ExceptionLogger.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/ExceptionLogger.java index 8d3862189dc..7d54a515236 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/ExceptionLogger.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/ExceptionLogger.java @@ -8,6 +8,8 @@ * *

See datadog.trace.agent.tooling.ExceptionHandlers */ -public class ExceptionLogger { +public final class ExceptionLogger { public static final Logger LOGGER = LoggerFactory.getLogger(ExceptionLogger.class); + + private ExceptionLogger() {} } diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/InitializationTelemetry.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/InitializationTelemetry.java index d6cf0958ca9..e55ceb4f270 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/InitializationTelemetry.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/InitializationTelemetry.java @@ -97,7 +97,9 @@ static final class BootstrapProxy extends InitializationTelemetry { // DQH - Decided not to eager access MethodHandles, since exceptions are uncommon // However, MethodHandles are cached on lookup - /** @param bootstrapInitTelemetry - non-null BootstrapInitializationTelemetry */ + /** + * @param bootstrapInitTelemetry - non-null BootstrapInitializationTelemetry + */ BootstrapProxy(final Object bootstrapInitTelemetry) { this.bootstrapInitTelemetry = bootstrapInitTelemetry; } diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/InstanceStore.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/InstanceStore.java index 67612ac1eed..765f0df6a6c 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/InstanceStore.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/InstanceStore.java @@ -19,7 +19,9 @@ public final class InstanceStore { private static final ClassValue classInstanceStore = GenericClassValue.of(type -> new InstanceStore<>()); - /** @return global store of instances with the same common type */ + /** + * @return global store of instances with the same common type + */ @SuppressWarnings("unchecked") public static InstanceStore of(Class type) { return classInstanceStore.get(type); diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/PatchLogger.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/PatchLogger.java index 8f28ca27b69..eea5b4adbb7 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/PatchLogger.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/PatchLogger.java @@ -29,7 +29,7 @@ public static PatchLogger getAnonymousLogger(final String resourceBundleName) { return SAFE_LOGGER; } - protected PatchLogger(final String name, final String resourceBundleName) { + private PatchLogger(final String name, final String resourceBundleName) { // super(name, resourceBundleName); } diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/aot/PlaceholderTraceInterceptor.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/aot/PlaceholderTraceInterceptor.java new file mode 100644 index 00000000000..5b18ea5f075 --- /dev/null +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/aot/PlaceholderTraceInterceptor.java @@ -0,0 +1,23 @@ +package datadog.trace.bootstrap.aot; + +import datadog.trace.api.interceptor.MutableSpan; +import datadog.trace.api.interceptor.TraceInterceptor; +import java.util.Collection; + +/** Placeholder {@link TraceInterceptor} used to temporarily work around an AOT bug. */ +public final class PlaceholderTraceInterceptor implements TraceInterceptor { + public static final PlaceholderTraceInterceptor INSTANCE = new PlaceholderTraceInterceptor(); + + private PlaceholderTraceInterceptor() {} + + @Override + public Collection onTraceComplete( + Collection trace) { + return trace; // maintain original trace + } + + @Override + public int priority() { + return 0; + } +} diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/aot/TraceApiTransformer.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/aot/TraceApiTransformer.java new file mode 100644 index 00000000000..07c5984775e --- /dev/null +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/aot/TraceApiTransformer.java @@ -0,0 +1,156 @@ +package datadog.trace.bootstrap.aot; + +import static datadog.instrument.asm.Opcodes.ACC_ABSTRACT; +import static datadog.instrument.asm.Opcodes.ACC_NATIVE; +import static datadog.instrument.asm.Opcodes.ARETURN; +import static datadog.instrument.asm.Opcodes.ASM9; +import static datadog.instrument.asm.Opcodes.GETSTATIC; +import static datadog.instrument.asm.Opcodes.ICONST_1; +import static datadog.instrument.asm.Opcodes.INVOKEINTERFACE; +import static datadog.instrument.asm.Opcodes.POP; +import static datadog.instrument.asm.Opcodes.POP2; + +import datadog.instrument.asm.ClassReader; +import datadog.instrument.asm.ClassVisitor; +import datadog.instrument.asm.ClassWriter; +import datadog.instrument.asm.MethodVisitor; +import java.lang.instrument.ClassFileTransformer; +import java.security.ProtectionDomain; +import java.util.concurrent.atomic.AtomicBoolean; + +/** + * Workaround a potential AOT bug where {@link datadog.trace.api.interceptor.TraceInterceptor} is + * mistakenly restored from the system class-loader in production, even though it was visible from + * the boot class-loader during training, resulting in {@link LinkageError}s. + * + *

Any call to {@link datadog.trace.api.Tracer#addTraceInterceptor} from application code in the + * system class-loader appears to trigger this bug. The workaround is to replace these calls during + * training with opcodes that pop the tracer and argument, and push the expected return value. + * + *

Likewise, custom {@code TraceInterceptor} return values are replaced with simple placeholders + * from the boot class-path which are guaranteed not to trigger the AOT bug. + * + *

Note these transformations are not persisted, so in production the original code is used. + */ +final class TraceApiTransformer implements ClassFileTransformer { + private static final ClassLoader SYSTEM_CLASS_LOADER = ClassLoader.getSystemClassLoader(); + + static final String TRACE_INTERCEPTOR = "datadog/trace/api/interceptor/TraceInterceptor"; + + static final String PLACEHOLDER_TRACE_INTERCEPTOR = + "datadog/trace/bootstrap/aot/PlaceholderTraceInterceptor"; + + @Override + public byte[] transform( + ClassLoader loader, + String className, + Class classBeingRedefined, + ProtectionDomain pd, + byte[] bytecode) { + + // workaround only needed in the system class-loader + if (loader == SYSTEM_CLASS_LOADER) { + try { + ClassReader cr = new ClassReader(bytecode); + ClassWriter cw = new ClassWriter(cr, 0); + AtomicBoolean modified = new AtomicBoolean(); + cr.accept(new TraceInterceptorPatch(cw, modified), 0); + // only return something when we've modified the bytecode + if (modified.get()) { + return cw.toByteArray(); + } + } catch (Throwable ignore) { + // skip this class + } + } + return null; // tells the JVM to keep the original bytecode + } + + /** + * Patches certain references to {@code TraceInterceptor} to workaround AOT bug: + * + *

    + *
  • removes direct calls to {@code Tracer.addTraceInterceptor()} + *
  • replaces {@code TraceInterceptor} return values with placeholders + *
+ */ + static final class TraceInterceptorPatch extends ClassVisitor { + private final AtomicBoolean modified; + + TraceInterceptorPatch(ClassVisitor cv, AtomicBoolean modified) { + super(ASM9, cv); + this.modified = modified; + } + + @Override + public MethodVisitor visitMethod( + int access, String name, String descriptor, String signature, String[] exceptions) { + MethodVisitor mv = super.visitMethod(access, name, descriptor, signature, exceptions); + if ((access & (ACC_ABSTRACT | ACC_NATIVE)) == 0) { + if (descriptor.endsWith(")L" + TRACE_INTERCEPTOR + ";")) { + mv = new ReturnPatch(mv, modified); + } + return new InvokePatch(mv, modified); + } else { + return mv; // no need to patch abstract/native methods + } + } + } + + /** Removes direct calls to {@code Tracer.addTraceInterceptor()}. */ + static final class InvokePatch extends MethodVisitor { + private final AtomicBoolean modified; + + InvokePatch(MethodVisitor mv, AtomicBoolean modified) { + super(ASM9, mv); + this.modified = modified; + } + + @Override + public void visitMethodInsn( + int opcode, String owner, String name, String descriptor, boolean isInterface) { + if (INVOKEINTERFACE == opcode + && "datadog/trace/api/Tracer".equals(owner) + && "addTraceInterceptor".equals(name) + && ("(L" + TRACE_INTERCEPTOR + ";)Z").equals(descriptor)) { + // discard tracer and trace interceptor argument from call stack + mv.visitInsn(POP2); + // push substitute return value (true) + mv.visitInsn(ICONST_1); + // flag that we've modified the bytecode + modified.set(true); + } else { + super.visitMethodInsn(opcode, owner, name, descriptor, isInterface); + } + } + } + + /** Replaces custom {@code TraceInterceptor} return values with placeholders. */ + static final class ReturnPatch extends MethodVisitor { + private final AtomicBoolean modified; + + ReturnPatch(MethodVisitor mv, AtomicBoolean modified) { + super(ASM9, mv); + this.modified = modified; + } + + @Override + public void visitInsn(int opcode) { + if (ARETURN == opcode) { + // discard old return value from call stack + mv.visitInsn(POP); + // push our placeholder interceptor instead + mv.visitFieldInsn( + GETSTATIC, + PLACEHOLDER_TRACE_INTERCEPTOR, + "INSTANCE", + "L" + PLACEHOLDER_TRACE_INTERCEPTOR + ";"); + mv.visitInsn(ARETURN); + // flag that we've modified the bytecode + modified.set(true); + } else { + super.visitInsn(opcode); + } + } + } +} diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/aot/TrainingAgent.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/aot/TrainingAgent.java new file mode 100644 index 00000000000..07573bf6ab6 --- /dev/null +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/aot/TrainingAgent.java @@ -0,0 +1,19 @@ +package datadog.trace.bootstrap.aot; + +import java.lang.instrument.Instrumentation; +import java.net.URL; + +/** Prepares the agent for Ahead-of-Time training. */ +public final class TrainingAgent { + public static void start( + final Object bootstrapInitTelemetry, + final Instrumentation inst, + final URL agentJarURL, + final String agentArgs) { + + // apply TraceInterceptor LinkageError workaround + inst.addTransformer(new TraceApiTransformer()); + + // don't start services, they won't be cached as they use a custom classloader + } +} diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/blocking/BlockingActionHelper.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/blocking/BlockingActionHelper.java index fbcb1062eb1..fbe5e150367 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/blocking/BlockingActionHelper.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/blocking/BlockingActionHelper.java @@ -12,6 +12,7 @@ import java.io.FileNotFoundException; import java.io.IOException; import java.io.InputStream; +import java.nio.charset.StandardCharsets; import java.util.regex.Matcher; import java.util.regex.Pattern; import org.slf4j.Logger; @@ -35,7 +36,7 @@ public class BlockingActionHelper { public enum TemplateType { JSON, - HTML; + HTML } public static int getHttpCode(int actionHttpCode) { @@ -118,12 +119,26 @@ public static TemplateType determineTemplateType( } public static byte[] getTemplate(TemplateType type) { + return getTemplate(type, null); + } + + public static byte[] getTemplate(TemplateType type, String securityResponseId) { + byte[] template; if (type == TemplateType.JSON) { - return TEMPLATE_JSON; + template = TEMPLATE_JSON; } else if (type == TemplateType.HTML) { - return TEMPLATE_HTML; + template = TEMPLATE_HTML; + } else { + return null; } - return null; + + // Use empty string when securityResponseId is not present + String replacementValue = + (securityResponseId == null || securityResponseId.isEmpty()) ? "" : securityResponseId; + + String templateString = new String(template, StandardCharsets.UTF_8); + String replacedTemplate = templateString.replace("[security_response_id]", replacementValue); + return replacedTemplate.getBytes(StandardCharsets.UTF_8); } public static String getContentType(TemplateType type) { diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/api/Java8BytecodeBridge.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/api/Java8BytecodeBridge.java index 867473b5d30..6d3f43b5139 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/api/Java8BytecodeBridge.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/api/Java8BytecodeBridge.java @@ -10,27 +10,37 @@ * were not supported prior to Java 8 and will lead to a class verification error. */ public class Java8BytecodeBridge { - /** @see Context#root() */ + /** + * @see Context#root() + */ public static Context getRootContext() { return Context.root(); } - /** @see Context#current() */ + /** + * @see Context#current() + */ public static Context getCurrentContext() { return Context.current(); } - /** @see Context#from(Object) */ + /** + * @see Context#from(Object) + */ public static Context getContextFrom(Object carrier) { return Context.from(carrier); } - /** @see Context#detachFrom(Object) */ + /** + * @see Context#detachFrom(Object) + */ public static Context detachContextFrom(Object carrier) { return Context.detachFrom(carrier); } - /** @see AgentSpan#fromContext(Context) */ + /** + * @see AgentSpan#fromContext(Context) + */ public static AgentSpan spanFromContext(Context context) { return AgentSpan.fromContext(context); } diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/dbm/SharedDBCommenter.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/dbm/SharedDBCommenter.java new file mode 100644 index 00000000000..83604a017bd --- /dev/null +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/dbm/SharedDBCommenter.java @@ -0,0 +1,107 @@ +package datadog.trace.bootstrap.instrumentation.dbm; + +import static datadog.trace.bootstrap.instrumentation.api.AgentTracer.activeSpan; + +import datadog.trace.api.BaseHash; +import datadog.trace.api.Config; +import datadog.trace.bootstrap.instrumentation.api.AgentSpan; +import datadog.trace.bootstrap.instrumentation.api.Tags; +import java.io.UnsupportedEncodingException; +import java.net.URLEncoder; +import java.nio.charset.StandardCharsets; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** Shared database comment builder for generating trace context comments for SQL DBs and MongoDB */ +public class SharedDBCommenter { + private static final Logger log = LoggerFactory.getLogger(SharedDBCommenter.class); + private static final String UTF8 = StandardCharsets.UTF_8.toString(); + + private static final char EQUALS = '='; + private static final char COMMA = ','; + private static final char QUOTE = '\''; + + // Injected fields. When adding a new one, be sure to update this and the methods below. + private static final String PARENT_SERVICE = encode("ddps"); + private static final String DATABASE_SERVICE = encode("dddbs"); + private static final String DD_HOSTNAME = encode("ddh"); + private static final String DD_DB_NAME = encode("dddb"); + private static final String DD_PEER_SERVICE = "ddprs"; + private static final String DD_ENV = encode("dde"); + private static final String DD_VERSION = encode("ddpv"); + private static final String TRACEPARENT = encode("traceparent"); + private static final String DD_SERVICE_HASH = encode("ddsh"); + + // Used by SQLCommenter and MongoCommentInjector to avoid duplicate comment injection + public static boolean containsTraceComment(String commentContent) { + return commentContent.contains(PARENT_SERVICE + "=") + || commentContent.contains(DATABASE_SERVICE + "=") + || commentContent.contains(DD_HOSTNAME + "=") + || commentContent.contains(DD_DB_NAME + "=") + || commentContent.contains(DD_PEER_SERVICE + "=") + || commentContent.contains(DD_ENV + "=") + || commentContent.contains(DD_VERSION + "=") + || commentContent.contains(TRACEPARENT + "=") + || commentContent.contains(DD_SERVICE_HASH + "="); + } + + // Build database comment content without comment delimiters such as /* */ + public static String buildComment( + String dbService, String dbType, String hostname, String dbName, String traceParent) { + + Config config = Config.get(); + StringBuilder sb = new StringBuilder(); + + int initSize = 0; // No initial content for pure comment + append(sb, PARENT_SERVICE, config.getServiceName(), initSize); + append(sb, DATABASE_SERVICE, dbService, initSize); + append(sb, DD_HOSTNAME, hostname, initSize); + append(sb, DD_DB_NAME, dbName, initSize); + append(sb, DD_PEER_SERVICE, getPeerService(), initSize); + append(sb, DD_ENV, config.getEnv(), initSize); + append(sb, DD_VERSION, config.getVersion(), initSize); + append(sb, TRACEPARENT, traceParent, initSize); + + if (config.isDbmInjectSqlBaseHash() && config.isExperimentalPropagateProcessTagsEnabled()) { + append(sb, DD_SERVICE_HASH, BaseHash.getBaseHashStr(), initSize); + } + + return sb.length() > 0 ? sb.toString() : null; + } + + private static String getPeerService() { + AgentSpan span = activeSpan(); + Object peerService = null; + if (span != null) { + peerService = span.getTag(Tags.PEER_SERVICE); + } + return peerService != null ? peerService.toString() : null; + } + + private static String encode(String val) { + try { + return URLEncoder.encode(val, UTF8); + } catch (UnsupportedEncodingException exe) { + if (log.isDebugEnabled()) { + log.debug("exception thrown while encoding comment key {}", val, exe); + } + } + return val; + } + + private static void append(StringBuilder sb, String key, String value, int initSize) { + if (null == value || value.isEmpty()) { + return; + } + String encodedValue; + try { + encodedValue = URLEncoder.encode(value, UTF8); + } catch (UnsupportedEncodingException e) { + encodedValue = value; + } + if (sb.length() > initSize) { + sb.append(COMMA); + } + sb.append(key).append(EQUALS).append(QUOTE).append(encodedValue).append(QUOTE); + } +} diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/AsyncResultDecorator.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/AsyncResultDecorator.java index fdcf7c60f74..994cf8658a8 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/AsyncResultDecorator.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/AsyncResultDecorator.java @@ -11,17 +11,6 @@ */ public abstract class AsyncResultDecorator extends BaseDecorator { - private static final ClassValue EXTENSION_CLASS_VALUE = - new ClassValue() { - @Override - protected AsyncResultExtension computeValue(Class type) { - return AsyncResultExtensions.registered().stream() - .filter(extension -> extension.supports(type)) - .findFirst() - .orElse(null); - } - }; - /** * Look for asynchronous result and decorate it with span finisher. If the result is not * asynchronous, it will be return unmodified and span will be finished. @@ -33,12 +22,9 @@ protected AsyncResultExtension computeValue(Class type) { */ public Object wrapAsyncResultOrFinishSpan( final Object result, final Class methodReturnType, final AgentSpan span) { - AsyncResultExtension extension; - if (result != null && (extension = EXTENSION_CLASS_VALUE.get(methodReturnType)) != null) { - Object applied = extension.apply(result, span); - if (applied != null) { - return applied; - } + Object applied = AsyncResultExtensions.wrapAsyncResult(result, methodReturnType, span); + if (applied != null) { + return applied; } // If no extension was applied, immediately finish the span and return the original result span.finish(); diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/BaseDecorator.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/BaseDecorator.java index bb575d92080..676daefbe8a 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/BaseDecorator.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/BaseDecorator.java @@ -1,13 +1,13 @@ package datadog.trace.bootstrap.instrumentation.decorator; -import static datadog.trace.api.cache.RadixTreeCache.PORTS; -import static datadog.trace.api.cache.RadixTreeCache.UNSET_PORT; import static datadog.trace.bootstrap.instrumentation.java.net.HostNameResolver.hostName; +import datadog.context.Context; import datadog.context.ContextScope; import datadog.trace.api.Config; import datadog.trace.api.DDTags; import datadog.trace.api.Functions; +import datadog.trace.api.TagMap; import datadog.trace.api.cache.QualifiedClassNameCache; import datadog.trace.bootstrap.instrumentation.api.AgentScope; import datadog.trace.bootstrap.instrumentation.api.AgentSpan; @@ -22,6 +22,7 @@ import java.util.function.Function; public abstract class BaseDecorator { + protected static final int UNSET_PORT = 0; private static final QualifiedClassNameCache CLASS_NAMES = new QualifiedClassNameCache( @@ -40,17 +41,31 @@ public String apply(Class clazz) { Functions.PrefixJoin.of(".")); protected final boolean traceAnalyticsEnabled; - protected final Double traceAnalyticsSampleRate; + protected final double traceAnalyticsSampleRate; + + private final TagMap.Entry traceAnalyticsEntry; + + // Deliberately not volatile, reading null and repeating the calculation is safe + private TagMap.Entry cachedComponentEntry = null; + protected final CharSequence version; protected BaseDecorator() { final Config config = Config.get(); final String[] instrumentationNames = instrumentationNames(); + this.traceAnalyticsEnabled = instrumentationNames.length > 0 && config.isTraceAnalyticsIntegrationEnabled( traceAnalyticsDefault(), instrumentationNames); + version = config.getVersion(); + this.traceAnalyticsSampleRate = (double) config.getInstrumentationAnalyticsSampleRate(instrumentationNames); + + this.traceAnalyticsEntry = + this.traceAnalyticsEnabled + ? TagMap.Entry.create(DDTags.ANALYTICS_SAMPLE_RATE, traceAnalyticsSampleRate) + : null; } protected abstract String[] instrumentationNames(); @@ -59,6 +74,20 @@ protected BaseDecorator() { protected abstract CharSequence component(); + /** Caches the component TagMap.Entry, so it isn't recreated for every trace */ + protected final TagMap.Entry componentEntry() { + // DQH = Tried calling component() in the constructor, but that had issues with static + // field ordering. That was caught be an integration test, but I didn't want to risk + // breaking other integrations where the test is not as thorough. + + // This approach while more complicated doesn't have any field initialization ordering issues. + TagMap.Entry componentEntry = cachedComponentEntry; + if (componentEntry == null) { + cachedComponentEntry = componentEntry = TagMap.Entry.create(Tags.COMPONENT, component()); + } + return componentEntry; + } + protected boolean traceAnalyticsDefault() { return false; } @@ -67,17 +96,24 @@ public AgentSpan afterStart(final AgentSpan span) { if (spanType() != null) { span.setSpanType(spanType()); } + + span.setTag(componentEntry()); + + // DQH - Could retrieve the value from componentEntry and cast to avoid the virtual call, + // unclear which option is better here final CharSequence component = component(); span.setTag(Tags.COMPONENT, component); span.context().setIntegrationName(component); - if (traceAnalyticsEnabled) { - span.setMetric(DDTags.ANALYTICS_SAMPLE_RATE, traceAnalyticsSampleRate); + if (version != ""){ + span.setTag(Tags.DD_VERSION,version); } + // null handled by setMetric + span.setMetric(traceAnalyticsEntry); return span; } - public AgentScope beforeFinish(final AgentScope scope) { - beforeFinish(scope.span()); + public ContextScope beforeFinish(final ContextScope scope) { + beforeFinish(scope.context()); return scope; } @@ -85,8 +121,14 @@ public AgentSpan beforeFinish(final AgentSpan span) { return span; } + public Context beforeFinish(final Context context) { + return context; + } + public AgentScope onError(final AgentScope scope, final Throwable throwable) { - onError(scope.span(), throwable); + if (scope != null) { + onError(scope.span(), throwable); + } return scope; } @@ -95,7 +137,7 @@ public AgentSpan onError(final AgentSpan span, final Throwable throwable) { } public AgentSpan onError(final AgentSpan span, final Throwable throwable, byte errorPriority) { - if (throwable != null) { + if (throwable != null && span != null) { span.addThrowable( throwable instanceof ExecutionException ? throwable.getCause() : throwable, errorPriority); @@ -104,7 +146,9 @@ public AgentSpan onError(final AgentSpan span, final Throwable throwable, byte e } public ContextScope onError(final ContextScope scope, final Throwable throwable) { - onError(AgentSpan.fromContext(scope.context()), throwable); + if (scope != null) { + onError(AgentSpan.fromContext(scope.context()), throwable); + } return scope; } @@ -144,9 +188,8 @@ public AgentSpan setPeerPort(AgentSpan span, String port) { public AgentSpan setPeerPort(AgentSpan span, int port) { if (port > UNSET_PORT) { - span.setTag(Tags.PEER_PORT, PORTS.get(port)); + span.setTag(Tags.PEER_PORT, port); } - return span; } diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/ClientDecorator.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/ClientDecorator.java index 5c3dd0aaae6..99dec2dbc08 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/ClientDecorator.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/ClientDecorator.java @@ -1,22 +1,43 @@ package datadog.trace.bootstrap.instrumentation.decorator; +import datadog.trace.api.TagMap; import datadog.trace.bootstrap.instrumentation.api.AgentSpan; import datadog.trace.bootstrap.instrumentation.api.Tags; public abstract class ClientDecorator extends BaseDecorator { + // Deliberately not volatile, reading a stale null and creating an extra Entry is safe + private TagMap.Entry cachedSpanKindEntry = null; protected abstract String service(); + /** Caches span kind entry to reduce allocation */ + private final TagMap.Entry spanKindEntry() { + // DQH - I considered moving the creation of the TagMap.Entry into a ClientDecorator + // constructor, but that introduces a subtle ordering requirement. + + // If the spanKind method refers to a static that isn't yet initialized, + // then spanKind will return null when the Decorator singleton is being constructed. + + // Such an ordering problem did occur with similar changes in BaseDecorator, so I've + // decided to be cautious here, too. + TagMap.Entry kindEntry = cachedSpanKindEntry; + if (kindEntry == null) { + cachedSpanKindEntry = kindEntry = TagMap.Entry.create(Tags.SPAN_KIND, spanKind()); + } + return kindEntry; + } + protected String spanKind() { return Tags.SPAN_KIND_CLIENT; } @Override public AgentSpan afterStart(final AgentSpan span) { - if (service() != null) { - span.setServiceName(service()); + final String service = service(); + if (service != null) { + span.setServiceName(service, component()); } - span.setTag(Tags.SPAN_KIND, spanKind()); + span.setTag(spanKindEntry()); // Generate metrics for all client spans. span.setMeasured(true); diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/DatabaseClientDecorator.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/DatabaseClientDecorator.java index 9f88059611e..7336a059bdc 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/DatabaseClientDecorator.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/DatabaseClientDecorator.java @@ -1,6 +1,7 @@ package datadog.trace.bootstrap.instrumentation.decorator; import static datadog.trace.api.gateway.Events.EVENTS; +import static datadog.trace.bootstrap.instrumentation.api.ServiceNameSources.DB_CLIENT_SPLIT_BY_HOST; import static datadog.trace.bootstrap.instrumentation.api.Tags.DB_TYPE; import datadog.appsec.api.blocking.BlockingException; @@ -76,7 +77,7 @@ public AgentSpan onConnection(final AgentSpan span, final CONNECTION connection) span.setTag(Tags.PEER_HOSTNAME, hostName); if (Config.get().isDbClientSplitByHost()) { - span.setServiceName(hostName.toString()); + span.setServiceName(hostName.toString(), DB_CLIENT_SPLIT_BY_HOST); } } } @@ -88,7 +89,7 @@ protected AgentSpan onInstance(final AgentSpan span, final String dbInstance) { span.setTag(Tags.DB_INSTANCE, dbInstance); String serviceName = dbClientService(dbInstance); if (null != serviceName) { - span.setServiceName(serviceName); + span.setServiceName(serviceName, component()); } } return span; @@ -137,11 +138,7 @@ public void onRawStatement(AgentSpan span, String sql) { BlockResponseFunction brf = ctx.getBlockResponseFunction(); if (brf != null) { Flow.Action.RequestBlockingAction rba = (Flow.Action.RequestBlockingAction) action; - brf.tryCommitBlockingResponse( - ctx.getTraceSegment(), - rba.getStatusCode(), - rba.getBlockingContentType(), - rba.getExtraHeaders()); + brf.tryCommitBlockingResponse(ctx.getTraceSegment(), rba); } throw new BlockingException("Blocked request (for SQL query)"); } diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/HttpClientDecorator.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/HttpClientDecorator.java index 77e96360435..42214929571 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/HttpClientDecorator.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/HttpClientDecorator.java @@ -11,6 +11,8 @@ import datadog.trace.api.InstrumenterConfig; import datadog.trace.api.ProductActivation; import datadog.trace.api.appsec.HttpClientRequest; +import datadog.trace.api.datastreams.DataStreamsTransactionExtractor; +import datadog.trace.api.datastreams.DataStreamsTransactionTracker; import datadog.trace.api.gateway.BlockResponseFunction; import datadog.trace.api.gateway.Flow; import datadog.trace.api.gateway.RequestContext; @@ -36,6 +38,9 @@ public abstract class HttpClientDecorator extends UriBasedCli private static final Logger log = LoggerFactory.getLogger(HttpClientDecorator.class); + private static final String DATADOG_META_LANG_HEADER_NAME = "Datadog-Meta-Lang"; + private static final String DD_CLIENT_LIBRARY_LANGUAGE_HEADER_NAME = "DD-Client-Library-Language"; + private static final BitSet CLIENT_ERROR_STATUSES = Config.get().getHttpClientErrorStatuses(); private static final UTF8BytesString DEFAULT_RESOURCE_NAME = UTF8BytesString.create("/"); @@ -48,6 +53,15 @@ public abstract class HttpClientDecorator extends UriBasedCli protected abstract URI url(REQUEST request) throws URISyntaxException; + /** + * Returns {@code true} if the request was made by the Datadog agent itself. Such requests must + * not be traced to avoid self-tracing loops. + */ + public boolean isAgentRequest(final REQUEST request) { + return getRequestHeader(request, DATADOG_META_LANG_HEADER_NAME) != null + || getRequestHeader(request, DD_CLIENT_LIBRARY_LANGUAGE_HEADER_NAME) != null; + } + protected abstract int status(RESPONSE response); protected abstract String getRequestHeader(REQUEST request, String headerName); @@ -68,8 +82,25 @@ protected boolean shouldSetResourceName() { return true; } + private final DataStreamsTransactionTracker.TransactionSourceReader + DSM_TRANSACTION_SOURCE_READER = + (source, headerName) -> { + try { + return getRequestHeader((REQUEST) source, headerName); + } catch (Throwable ignored) { + return null; + } + }; + public AgentSpan onRequest(final AgentSpan span, final REQUEST request) { if (request != null) { + AgentTracer.get() + .getDataStreamsMonitoring() + .trackTransaction( + span, + DataStreamsTransactionExtractor.Type.HTTP_OUT_HEADERS, + request, + DSM_TRANSACTION_SOURCE_READER); String method = method(request); span.setTag(Tags.HTTP_METHOD, method); @@ -210,11 +241,7 @@ protected void onHttpClientRequest(final AgentSpan span, final String url) { BlockResponseFunction brf = ctx.getBlockResponseFunction(); if (brf != null) { Flow.Action.RequestBlockingAction rba = (Flow.Action.RequestBlockingAction) action; - brf.tryCommitBlockingResponse( - ctx.getTraceSegment(), - rba.getStatusCode(), - rba.getBlockingContentType(), - rba.getExtraHeaders()); + brf.tryCommitBlockingResponse(ctx.getTraceSegment(), rba); } throw new BlockingException("Blocked request (for SSRF attempt)"); } diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/HttpServerDecorator.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/HttpServerDecorator.java index 76163e9bb6f..eb0c19e397b 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/HttpServerDecorator.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/HttpServerDecorator.java @@ -13,6 +13,8 @@ import datadog.context.propagation.Propagators; import datadog.trace.api.Config; import datadog.trace.api.DDTags; +import datadog.trace.api.datastreams.DataStreamsTransactionExtractor; +import datadog.trace.api.datastreams.DataStreamsTransactionTracker; import datadog.trace.api.function.TriConsumer; import datadog.trace.api.function.TriFunction; import datadog.trace.api.gateway.BlockResponseFunction; @@ -39,6 +41,7 @@ import datadog.trace.bootstrap.instrumentation.decorator.http.ClientIpAddressResolver; import java.net.InetAddress; import java.util.BitSet; +import java.util.LinkedHashMap; import java.util.Locale; import java.util.Map; import java.util.concurrent.ExecutionException; @@ -54,9 +57,8 @@ public abstract class HttpServerDecorator SERVER_PATHWAY_EDGE_TAGS; + + static { + SERVER_PATHWAY_EDGE_TAGS = new LinkedHashMap<>(2); + // TODO: Refactor TagsProcessor to move it into a package that we can link the constants for. + SERVER_PATHWAY_EDGE_TAGS.put("direction", "in"); + SERVER_PATHWAY_EDGE_TAGS.put("type", "http"); + } + + private static final BitSet CLIENT_ERROR_STATUSES = Config.get().getHttpClientErrorStatuses(); + + private static final Boolean HTTP_ERROR_ENBALED = Config.get().isHttpErrorEnabled(); + private static final UTF8BytesString DEFAULT_RESOURCE_NAME = UTF8BytesString.create("/"); protected static final UTF8BytesString NOT_FOUND_RESOURCE_NAME = UTF8BytesString.create("404"); protected static final boolean SHOULD_SET_404_RESOURCE_NAME = @@ -79,6 +94,20 @@ public abstract class HttpServerDecorator 0 + ? instrumentationNames[0] + : DEFAULT_INSTRUMENTATION_NAME; + } + + protected final String primaryInstrumentationName() { + return primaryInstrumentationName; + } + protected abstract AgentPropagation.ContextVisitor getter(); protected abstract AgentPropagation.ContextVisitor responseGetter(); @@ -95,6 +124,14 @@ public abstract class HttpServerDecorator 0 - ? instrumentationNames[0] - : DEFAULT_INSTRUMENTATION_NAME; + String instrumentationName = primaryInstrumentationName(); AgentSpanContext extracted = getExtractedSpanContext(parentContext); // Call IG callbacks extracted = callIGCallbackStart(extracted); @@ -155,6 +188,14 @@ public Context startSpan(REQUEST_CARRIER carrier, Context parentContext) { extracted = startInferredProxySpan(parentContext, extracted); AgentSpan span = tracer().startSpan(instrumentationName, spanName(), extracted).setMeasured(true); + // Register service-entry span with inferred proxy span (if present) so that premature + // finish calls from child spans (e.g., Spring MVC handler) are deferred until the + // service-entry span finishes (after the response status is known). + registerServiceEntrySpanInInferredProxy(parentContext, span); + // Reset service name inherited from inferred proxy parent: the inferred span uses the + // gateway domain name as service name, but the service-entry span should identify + // the application (configured DD_SERVICE), not the upstream gateway. + resetServiceNameIfUnderInferredProxy(parentContext, span); // Apply RequestBlockingAction if any Flow flow = callIGCallbackRequestHeaders(span, carrier); if (flow.getAction() instanceof RequestBlockingAction) { @@ -174,6 +215,30 @@ protected AgentSpanContext startInferredProxySpan(Context context, AgentSpanCont return span.start(extracted); } + private void registerServiceEntrySpanInInferredProxy( + Context parentContext, AgentSpan serviceEntrySpan) { + InferredProxySpan inferredProxy = InferredProxySpan.fromContext(parentContext); + if (inferredProxy != null) { + inferredProxy.registerServiceEntrySpan(serviceEntrySpan); + } + } + + private void resetServiceNameIfUnderInferredProxy(Context parentContext, AgentSpan span) { + if (InferredProxySpan.fromContext(parentContext) != null) { + span.setServiceName(Config.get().getServiceName()); + } + } + + private final DataStreamsTransactionTracker.TransactionSourceReader + DSM_TRANSACTION_SOURCE_READER = + (source, headerName) -> { + try { + return getRequestHeader((REQUEST) source, headerName); + } catch (Throwable ignored) { + return null; + } + }; + public AgentSpan onRequest( final AgentSpan span, final CONNECTION connection, @@ -326,6 +391,13 @@ public AgentSpan onRequest( span.setRequestBlockingAction((RequestBlockingAction) flow.getAction()); } + AgentTracer.get() + .getDataStreamsMonitoring() + .trackTransaction( + span, + DataStreamsTransactionExtractor.Type.HTTP_IN_HEADERS, + request, + DSM_TRANSACTION_SOURCE_READER); return span; } @@ -350,6 +422,9 @@ protected BlockResponseFunction createBlockResponseFunction( public AgentSpan onResponseStatus(final AgentSpan span, final int status) { if (status > UNSET_STATUS) { span.setHttpStatusCode(status); + if (HTTP_ERROR_ENBALED && CLIENT_ERROR_STATUSES.get(status)) { + span.setError(true); + }else // explicitly set here because some other decorators might already set an error without // looking at the status code // XXX: the logic is questionable: span.error becomes equivalent to status 5xx, @@ -537,25 +612,37 @@ private Flow callIGCallbackURI( } @Override - public AgentSpan beforeFinish(AgentSpan span) { - // TODO Migrate beforeFinish to Context API - onRequestEndForInstrumentationGateway(span); + public Context beforeFinish(Context context) { + AgentSpan span = AgentSpan.fromContext(context); + if (span != null) { + onRequestEndForInstrumentationGateway(span); + } + // Close Serverless Gateway Inferred Span if any - // finishInferredProxySpan(context); - return super.beforeFinish(span); + finishInferredProxySpan(context); + + return super.beforeFinish(context); } protected void finishInferredProxySpan(Context context) { - InferredProxySpan span; - if ((span = InferredProxySpan.fromContext(context)) != null) { - span.finish(); + InferredProxySpan inferredProxySpan; + if ((inferredProxySpan = InferredProxySpan.fromContext(context)) != null) { + inferredProxySpan.finish(AgentSpan.fromContext(context)); } } private void onRequestEndForInstrumentationGateway(@Nonnull final AgentSpan span) { - if (span.getLocalRootSpan() != span) { + AgentSpan localRoot = span.getLocalRootSpan(); + + // Check if the local root is an inferred proxy span + boolean hasInferredProxyParent = + localRoot != span && localRoot.getTag("_dd.inferred_span") != null; + + // Only proceed if this is the root span OR if we have an inferred proxy parent + if (localRoot != span && !hasInferredProxyParent) { return; } + CallbackProvider cbp = tracer().getUniversalCallbackProvider(); RequestContext requestContext = span.getRequestContext(); if (cbp != null && requestContext != null) { diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/ServerDecorator.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/ServerDecorator.java index 8ee7e2f4f42..20b11038ffd 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/ServerDecorator.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/ServerDecorator.java @@ -1,15 +1,21 @@ package datadog.trace.bootstrap.instrumentation.decorator; import datadog.trace.api.DDTags; +import datadog.trace.api.TagMap; import datadog.trace.bootstrap.instrumentation.api.AgentSpan; import datadog.trace.bootstrap.instrumentation.api.Tags; public abstract class ServerDecorator extends BaseDecorator { + private static final TagMap.Entry SPAN_KIND_ENTRY = + TagMap.Entry.create(Tags.SPAN_KIND, Tags.SPAN_KIND_SERVER); + private static final TagMap.Entry LANG_ENTRY = + TagMap.Entry.create(DDTags.LANGUAGE_TAG_KEY, DDTags.LANGUAGE_TAG_VALUE); @Override public AgentSpan afterStart(final AgentSpan span) { - span.setTag(Tags.SPAN_KIND, Tags.SPAN_KIND_SERVER); - span.setTag(DDTags.LANGUAGE_TAG_KEY, DDTags.LANGUAGE_TAG_VALUE); + span.setTag(SPAN_KIND_ENTRY); + span.setTag(LANG_ENTRY); + return super.afterStart(span); } } diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/UriBasedClientDecorator.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/UriBasedClientDecorator.java index ca593f78e8e..0ab130eab19 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/UriBasedClientDecorator.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/decorator/UriBasedClientDecorator.java @@ -19,7 +19,7 @@ public void onURI(@Nonnull final AgentSpan span, @Nonnull final URI uri) { if (null != host && !host.isEmpty()) { span.setTag(Tags.PEER_HOSTNAME, host); if (Config.get().isHttpClientSplitByDomain() && host.charAt(0) >= 'A') { - span.setServiceName(host); + span.setServiceName(host, component()); } if (port > 0) { setPeerPort(span, port); diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/java/concurrent/AsyncResultExtensions.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/java/concurrent/AsyncResultExtensions.java index 1c33522a530..1127ba4455b 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/java/concurrent/AsyncResultExtensions.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/java/concurrent/AsyncResultExtensions.java @@ -16,6 +16,32 @@ public final class AsyncResultExtensions { private static final List EXTENSIONS = new CopyOnWriteArrayList<>(singletonList(new CompletableAsyncResultExtension())); + private static final ClassValue EXTENSION_CLASS_VALUE = + new ClassValue() { + @Override + protected AsyncResultExtension computeValue(Class type) { + return EXTENSIONS.stream() + .filter(extension -> extension.supports(type)) + .findFirst() + .orElse(null); + } + }; + + /** + * Wraps a supported async result so the span is finished when the async computation completes. + * + * @return the wrapped async result, or {@code null} if the result type is unsupported or no + * wrapping is applied + */ + public static Object wrapAsyncResult( + final Object result, final Class resultType, final AgentSpan span) { + AsyncResultExtension extension; + if (result != null && (extension = EXTENSION_CLASS_VALUE.get(resultType)) != null) { + return extension.apply(result, span); + } + return null; + } + /** * Registers an extension to add supported async types. * @@ -36,11 +62,6 @@ public static void register(AsyncResultExtension extension) { } } - /** Returns the list of currently registered extensions. */ - public static List registered() { - return EXTENSIONS; - } - static final class CompletableAsyncResultExtension implements AsyncResultExtension { @Override public boolean supports(Class result) { diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/java/concurrent/TPEHelper.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/java/concurrent/TPEHelper.java index 1f3eccab66b..12c4498fe6e 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/java/concurrent/TPEHelper.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/java/concurrent/TPEHelper.java @@ -101,6 +101,9 @@ public static AgentScope getAndClearThreadLocalScope(Runnable task) { return null; } AgentScope scope = threadLocalScope.get(); + // Intentionally use `.set(null)` instead of `.remove()` for performance reasons. + // For details see: https://github.com/DataDog/dd-trace-java/pull/9856#discussion_r2527729963 + // noinspection ThreadLocalSetWithNull threadLocalScope.set(null); return scope; } diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/java/lang/VirtualThreadHelper.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/java/lang/VirtualThreadHelper.java new file mode 100644 index 00000000000..69aa0dc8943 --- /dev/null +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/java/lang/VirtualThreadHelper.java @@ -0,0 +1,13 @@ +package datadog.trace.bootstrap.instrumentation.java.lang; + +/** This class is a helper for the java-lang-21.0 {@code VirtualThreadInstrumentation}. */ +public final class VirtualThreadHelper { + public static final String VIRTUAL_THREAD_CLASS_NAME = "java.lang.VirtualThread"; + + /** + * {@link VirtualThreadState} class name as string literal. This is mandatory for {@link + * datadog.trace.bootstrap.ContextStore} API call. + */ + public static final String VIRTUAL_THREAD_STATE_CLASS_NAME = + "datadog.trace.bootstrap.instrumentation.java.lang.VirtualThreadState"; +} diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/java/lang/VirtualThreadState.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/java/lang/VirtualThreadState.java new file mode 100644 index 00000000000..0a1eec7573c --- /dev/null +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/java/lang/VirtualThreadState.java @@ -0,0 +1,46 @@ +package datadog.trace.bootstrap.instrumentation.java.lang; + +import datadog.context.Context; +import datadog.trace.bootstrap.instrumentation.api.AgentScope.Continuation; + +/** + * This class holds the saved context and scope continuation for a virtual thread. + * + *

Used by java-lang-21.0 {@code VirtualThreadInstrumentation} to swap the entire scope stack on + * mount/unmount. + */ +public final class VirtualThreadState { + /** The virtual thread's saved context (scope stack snapshot). */ + private Context context; + + /** Prevents the enclosing context scope from completing before the virtual thread finishes. */ + private final Continuation continuation; + + /** The carrier thread's saved context, set between mount and unmount. */ + private Context previousContext; + + public VirtualThreadState(Context context, Continuation continuation) { + this.context = context; + this.continuation = continuation; + } + + /** Called on mount: swaps the virtual thread's context into the carrier thread. */ + public void onMount() { + this.previousContext = this.context.swap(); + } + + /** Called on unmount: restores the carrier thread's original context. */ + public void onUnmount() { + if (this.previousContext != null) { + this.context = this.previousContext.swap(); + this.previousContext = null; + } + } + + /** Called on termination: releases the trace continuation. */ + public void onTerminate() { + if (this.continuation != null) { + this.continuation.cancel(); + } + } +} diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/jdbc/DBInfo.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/jdbc/DBInfo.java index 9fb0fe5855c..6b802825b47 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/jdbc/DBInfo.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/jdbc/DBInfo.java @@ -1,9 +1,11 @@ package datadog.trace.bootstrap.instrumentation.jdbc; +import datadog.trace.util.HashingUtils; import java.util.Objects; -public class DBInfo { - public static DBInfo DEFAULT = new Builder().type("database").build(); +public final class DBInfo { + public static final DBInfo DEFAULT = new Builder().type("database").build(); + private final String type; private final String subtype; private final boolean fullPropagationSupport; @@ -15,7 +17,7 @@ public class DBInfo { private final Integer port; private final String warehouse; private final String schema; - private String poolName; + private volatile String poolName; DBInfo( String type, @@ -44,7 +46,7 @@ public class DBInfo { this.poolName = poolName; } - public static class Builder { + public static final class Builder { private String type; private String subtype; // most DBs do support full propagation (inserting trace ID in query comments), so we default to @@ -191,7 +193,7 @@ public String getInstance() { } public String getDb() { - return db; + return db != null ? db : instance; } public String getHost() { @@ -255,17 +257,18 @@ public boolean equals(Object o) { @Override public int hashCode() { - return Objects.hash( - type, - subtype, - fullPropagationSupport, - url, - user, - instance, - db, - host, - port, - warehouse, - schema); + int hash = 0; + hash = HashingUtils.addToHash(hash, type); + hash = HashingUtils.addToHash(hash, subtype); + hash = HashingUtils.addToHash(hash, fullPropagationSupport); + hash = HashingUtils.addToHash(hash, url); + hash = HashingUtils.addToHash(hash, user); + hash = HashingUtils.addToHash(hash, instance); + hash = HashingUtils.addToHash(hash, db); + hash = HashingUtils.addToHash(hash, host); + hash = HashingUtils.addToHash(hash, port); + hash = HashingUtils.addToHash(hash, warehouse); + hash = HashingUtils.addToHash(hash, schema); + return hash; } } diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/jdbc/DBQueryInfo.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/jdbc/DBQueryInfo.java index 71744b60741..4488ca8b161 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/jdbc/DBQueryInfo.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/jdbc/DBQueryInfo.java @@ -1,5 +1,6 @@ package datadog.trace.bootstrap.instrumentation.jdbc; +import datadog.trace.api.Config; import datadog.trace.api.cache.DDCache; import datadog.trace.api.cache.DDCaches; import datadog.trace.api.normalize.SQLNormalizer; @@ -7,6 +8,11 @@ import java.util.function.Function; import java.util.function.ToIntFunction; +import java.util.HashMap; +import java.util.Map; + +import static java.nio.charset.StandardCharsets.UTF_8; + public final class DBQueryInfo { private static final int COMBINED_SQL_LIMIT = 2 * 1024 * 1024; // characters @@ -26,9 +32,20 @@ public static DBQueryInfo ofPreparedStatement(String sql) { private final UTF8BytesString operation; private final UTF8BytesString sql; + private Map vals; + private UTF8BytesString originSql; + + public boolean SqlObfuscation = Config.get().getJdbcSqlObfuscation(); public DBQueryInfo(String sql) { this.sql = SQLNormalizer.normalize(sql); + + if (SqlObfuscation) { + this.originSql = UTF8BytesString.create(sql.getBytes(UTF_8)); + } else { + this.originSql = UTF8BytesString.EMPTY; + } + this.vals = new HashMap<>(); this.operation = UTF8BytesString.create(extractOperation(this.sql)); } @@ -36,13 +53,25 @@ public UTF8BytesString getOperation() { return operation; } + public Map getVals() { + return vals; + } + + public void setVal(int index, String val) { + vals.put(index, val); + } + public UTF8BytesString getSql() { return sql; } + int weight() { return sql.length(); } + public UTF8BytesString getOriginSql() { + return originSql; + } public static CharSequence extractOperation(CharSequence sql) { if (null == sql) { diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/jdbc/JDBCConnectionUrlParser.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/jdbc/JDBCConnectionUrlParser.java index 9f1099b6bfa..ce2c5bf77f4 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/jdbc/JDBCConnectionUrlParser.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/jdbc/JDBCConnectionUrlParser.java @@ -73,7 +73,7 @@ DBInfo.Builder doParse(final String jdbcUrl, final DBInfo.Builder builder) { String instanceName = null; final int hostIndex = jdbcUrl.indexOf("://"); - if (hostIndex <= 0) { + if (hostIndex <= 0 || jdbcUrl.length() == 3 + hostIndex) { return builder; } diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/jms/SessionState.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/jms/SessionState.java index 7142ac3a505..be2179818ea 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/jms/SessionState.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/jms/SessionState.java @@ -2,7 +2,6 @@ import datadog.trace.api.Config; import datadog.trace.bootstrap.instrumentation.api.AgentSpan; -import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; import java.util.ArrayDeque; import java.util.Collections; import java.util.Comparator; @@ -65,7 +64,6 @@ public int compare(Map.Entry o1, Map.Entry context; public static final InjectAdapter SETTER = new InjectAdapter(); @@ -39,20 +40,26 @@ public static ContextPayload from(final AgentSpan span) { } public static ContextPayload read(final ObjectInput oi) throws IOException { - try { - final Object object = oi.readObject(); - if (object instanceof Map) { - return new ContextPayload((Map) object); - } - } catch (final ClassCastException | ClassNotFoundException ex) { - log.debug("Error reading object", ex); + final int size = oi.readInt(); + if (size < 0 || size > MAX_CONTEXT_SIZE) { + log.debug("Dropping RMI context payload: size {} exceeds maximum {}", size, MAX_CONTEXT_SIZE); + return null; } - - return null; + final Map context = new HashMap<>(size * 2); + for (int i = 0; i < size; i++) { + final String key = oi.readUTF(); + final String value = oi.readUTF(); + context.put(key, value); + } + return new ContextPayload(context); } public void write(final ObjectOutput out) throws IOException { - out.writeObject(context); + out.writeInt(context.size()); + for (final Map.Entry entry : context.entrySet()) { + out.writeUTF(entry.getKey()); + out.writeUTF(entry.getValue()); + } } @ParametersAreNonnullByDefault diff --git a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/rmi/ContextPropagator.java b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/rmi/ContextPropagator.java index bd2add5f663..aa9a0f41ea9 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/rmi/ContextPropagator.java +++ b/dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/instrumentation/rmi/ContextPropagator.java @@ -20,7 +20,7 @@ public class ContextPropagator { private static final ObjID REGISTRY_ID = new ObjID(ObjID.REGISTRY_ID); // RMI object id used to identify DataDog instrumentation - public static final ObjID DD_CONTEXT_CALL_ID = new ObjID("Datadog.v1.context_call".hashCode()); + public static final ObjID DD_CONTEXT_CALL_ID = new ObjID("Datadog.v2.context_call".hashCode()); // Operation id used for checking context propagation is possible // RMI expects these operations to have negative identifier, as positive ones mean legacy @@ -62,7 +62,9 @@ private boolean checkIfContextCanBePassed( return result; } - /** @return {@code true} when no error happened during call */ + /** + * @return {@code true} when no error happened during call + */ private boolean syntheticCall( final Connection c, final ContextPayload payload, final int operationId) { final StreamRemoteCall shareContextCall = new StreamRemoteCall(c); diff --git a/dd-java-agent/agent-bootstrap/src/main/java11/datadog/trace/bootstrap/instrumentation/ffm/FFMNativeMethodDecorator.java b/dd-java-agent/agent-bootstrap/src/main/java11/datadog/trace/bootstrap/instrumentation/ffm/FFMNativeMethodDecorator.java new file mode 100644 index 00000000000..b7ce858c031 --- /dev/null +++ b/dd-java-agent/agent-bootstrap/src/main/java11/datadog/trace/bootstrap/instrumentation/ffm/FFMNativeMethodDecorator.java @@ -0,0 +1,216 @@ +package datadog.trace.bootstrap.instrumentation.ffm; + +import datadog.context.ContextScope; +import datadog.trace.api.InstrumenterConfig; +import datadog.trace.bootstrap.instrumentation.api.AgentSpan; +import datadog.trace.bootstrap.instrumentation.api.AgentTracer; +import datadog.trace.bootstrap.instrumentation.api.UTF8BytesString; +import datadog.trace.bootstrap.instrumentation.decorator.BaseDecorator; +import java.lang.invoke.MethodHandle; +import java.lang.invoke.MethodHandles; +import java.lang.invoke.MethodType; +import java.util.Set; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public final class FFMNativeMethodDecorator extends BaseDecorator { + private static final Logger LOGGER = LoggerFactory.getLogger(FFMNativeMethodDecorator.class); + private static final CharSequence TRACE_FFM = UTF8BytesString.create("trace-ffm"); + private static final CharSequence OPERATION_NAME = UTF8BytesString.create("trace.native"); + + private static final MethodHandle START_SPAN_MH = + safeFindStatic( + "startSpan", + MethodType.methodType(ContextScope.class, CharSequence.class, boolean.class)); + private static final MethodHandle END_SPAN_MH = + safeFindStatic( + "endSpan", + MethodType.methodType(Object.class, Throwable.class, ContextScope.class, Object.class)); + + public static final FFMNativeMethodDecorator DECORATE = new FFMNativeMethodDecorator(); + + private static MethodHandle safeFindStatic(String name, MethodType methodType) { + try { + return MethodHandles.lookup().findStatic(FFMNativeMethodDecorator.class, name, methodType); + } catch (Throwable t) { + LOGGER.debug("Cannot find method {} in NativeMethodHandleWrapper", name, t); + return null; + } + } + + public static MethodHandle wrap( + final MethodHandle original, final String libraryName, final String methodName) { + if (START_SPAN_MH == null || END_SPAN_MH == null) { + return original; + } + try { + MethodType originalType = original.type(); + boolean isVoid = originalType.returnType() == void.class; + + // We need the ContextScope to be visible to the finally block. + // Easiest way is to artificially prepend it to the target signature. + // The added parameter is ignored by the original handle. + // originalWithScope: (ContextScope, args...) -> R + MethodHandle originalWithScope = MethodHandles.dropArguments(original, 0, ContextScope.class); + + /* + * Build the cleanup handle used by MethodHandles.tryFinally. + * + * tryFinally has a strict calling convention: + * - void target -> cleanup(Throwable, ContextScope, args...) + * - non-void -> cleanup(Throwable, R, ContextScope, args...) + * + * END_SPAN_MH is (Throwable, ContextScope, Object) -> Object, + * so we need to reshape it to match what tryFinally expects. + */ + MethodHandle cleanup; + + if (isVoid) { + // No return value: bind `null` as the result argument. + MethodHandle endWithNull = MethodHandles.insertArguments(END_SPAN_MH, 2, (Object) null); + + // Make it accept the original arguments even though they are unused. + MethodHandle endDropped = + MethodHandles.dropArguments(endWithNull, 2, originalType.parameterList()); + + // tryFinally requires void return for void targets. + cleanup = endDropped.asType(endDropped.type().changeReturnType(void.class)); + + } else { + /* + * Non-void case: + * tryFinally will call cleanup as: + * (Throwable, returnValue, ContextScope, args...) + * + * END_SPAN_MH expects: + * (Throwable, ContextScope, result) + * + * So we first permute parameters to swap returnValue and ContextScope. + */ + MethodHandle endPermuted = + MethodHandles.permuteArguments( + END_SPAN_MH, + MethodType.methodType( + Object.class, Throwable.class, Object.class, ContextScope.class), + 0, + 2, + 1); + + // Accept original arguments (unused) after the required ones. + MethodHandle endDropped = + MethodHandles.dropArguments(endPermuted, 3, originalType.parameterList()); + + // Adapt return and result parameter types to match the original signature. + MethodType cleanupType = + endDropped + .type() + .changeParameterType(1, originalType.returnType()) + .changeReturnType(originalType.returnType()); + + cleanup = endDropped.asType(cleanupType); + } + + // Wrap the original in try/finally semantics. + // Resulting handle: + // (ContextScope, args...) -> R + MethodHandle withFinally = MethodHandles.tryFinally(originalWithScope, cleanup); + + // Precompute span metadata so we don't redo the lookup per invocation. + final CharSequence resourceName = resourceNameFor(libraryName, methodName); + final boolean methodMeasured = isMethodMeasured(libraryName, methodName); + + // Bind both arguments to startSpan. + // After binding: () -> ContextScope + MethodHandle boundStart = + MethodHandles.insertArguments(START_SPAN_MH, 0, resourceName, methodMeasured); + + // Make it look like it takes the same arguments as the original, + // even though they are ignored. + // (args...) -> ContextScope + MethodHandle startCombiner = + MethodHandles.dropArguments(boundStart, 0, originalType.parameterList()); + + /* + * foldArguments wires it all together: + * + * scope = startCombiner(args...) + * return withFinally(scope, args...) + * + * Final shape matches the original: + * (args...) -> R + */ + return MethodHandles.foldArguments(withFinally, startCombiner); + + } catch (Throwable t) { + LOGGER.debug( + "Cannot wrap method handle for library {} and method {}", libraryName, methodName, t); + return original; + } + } + + @SuppressWarnings("unused") + public static ContextScope startSpan(CharSequence resourceName, boolean methodMeasured) { + AgentSpan span = AgentTracer.startSpan(TRACE_FFM.toString(), OPERATION_NAME); + DECORATE.afterStart(span); + span.setResourceName(resourceName); + if (methodMeasured) { + span.setMeasured(true); + } + return AgentTracer.activateSpan(span); + } + + @SuppressWarnings("unused") + public static Object endSpan(Throwable t, ContextScope scope, Object result) { + try { + if (scope != null) { + final AgentSpan span = AgentSpan.fromContext(scope.context()); + scope.close(); + + if (span != null) { + if (t != null) { + DECORATE.onError(span, t); + span.addThrowable(t); + } + span.finish(); + } + } + } catch (Throwable ignored) { + + } + return result; + } + + public static boolean isMethodTraced(final String library, final String method) { + return matches(InstrumenterConfig.get().getTraceNativeMethods().get(library), method); + } + + public static boolean isMethodMeasured(final String library, final String method) { + return matches(InstrumenterConfig.get().getMeasureNativeMethods().get(library), method); + } + + public static CharSequence resourceNameFor(final String library, final String method) { + return UTF8BytesString.create(library + "." + method); + } + + private static boolean matches(final Set allows, final String method) { + if (allows == null) { + return false; + } + return allows.contains(method) || allows.contains("*"); + } + + @Override + protected String[] instrumentationNames() { + return new String[] {TRACE_FFM.toString()}; + } + + @Override + protected CharSequence spanType() { + return null; + } + + @Override + protected CharSequence component() { + return TRACE_FFM; + } +} diff --git a/dd-java-agent/agent-bootstrap/src/main/java11/datadog/trace/bootstrap/instrumentation/ffm/NativeLibraryHelper.java b/dd-java-agent/agent-bootstrap/src/main/java11/datadog/trace/bootstrap/instrumentation/ffm/NativeLibraryHelper.java new file mode 100644 index 00000000000..15033da489b --- /dev/null +++ b/dd-java-agent/agent-bootstrap/src/main/java11/datadog/trace/bootstrap/instrumentation/ffm/NativeLibraryHelper.java @@ -0,0 +1,35 @@ +package datadog.trace.bootstrap.instrumentation.ffm; + +import datadog.trace.api.Pair; +import java.io.File; +import java.util.Locale; +import java.util.concurrent.ConcurrentHashMap; + +public final class NativeLibraryHelper { + // this map is unlimited. However, the number of entries depends on the configured methods we want + // to trace. + private static final ConcurrentHashMap> SYMBOLS_MAP = + new ConcurrentHashMap<>(); + + private NativeLibraryHelper() {} + + public static void onSymbolLookup( + final String libraryName, final String symbol, final long address) { + if (libraryName != null && !libraryName.isEmpty()) { + if (FFMNativeMethodDecorator.isMethodTraced(libraryName, symbol)) { + SYMBOLS_MAP.put(address, Pair.of(libraryName, symbol)); + } + } + } + + public static Pair reverseResolveLibraryAndSymbol(long address) { + return SYMBOLS_MAP.get(address); + } + + public static String extractLibraryName(String fullPath) { + String libraryName = new File(fullPath).getName().toLowerCase(Locale.ROOT); + int dot = libraryName.lastIndexOf('.'); + libraryName = (dot > 0) ? libraryName.substring(0, dot) : libraryName; + return libraryName; + } +} diff --git a/dd-java-agent/agent-bootstrap/src/main/java11/datadog/trace/bootstrap/instrumentation/jfr/exceptions/ExceptionHistogram.java b/dd-java-agent/agent-bootstrap/src/main/java11/datadog/trace/bootstrap/instrumentation/jfr/exceptions/ExceptionHistogram.java index 07056ff7c4e..ccef08a4981 100644 --- a/dd-java-agent/agent-bootstrap/src/main/java11/datadog/trace/bootstrap/instrumentation/jfr/exceptions/ExceptionHistogram.java +++ b/dd-java-agent/agent-bootstrap/src/main/java11/datadog/trace/bootstrap/instrumentation/jfr/exceptions/ExceptionHistogram.java @@ -70,7 +70,16 @@ private boolean record(String typeName) { typeName = CLIPPED_ENTRY_TYPE_NAME; } - long count = histogram.computeIfAbsent(typeName, k -> new AtomicLong()).getAndIncrement(); +// long count = histogram.computeIfAbsent(typeName, k -> new AtomicLong()).getAndIncrement(); + + + AtomicLong atomicLong = histogram.get(typeName); + if (atomicLong==null){ + atomicLong = new AtomicLong(); + + } + long count = atomicLong.getAndIncrement(); + histogram.put(typeName,atomicLong); /* * This is supposed to signal that a particular exception type was seen the first time in a particular time span. diff --git a/dd-java-agent/agent-bootstrap/src/main/resources/META-INF/native-image/com.datadoghq/dd-java-agent/reflect-config.json b/dd-java-agent/agent-bootstrap/src/main/resources/META-INF/native-image/com.datadoghq/dd-java-agent/reflect-config.json index adf96f3c8fe..3225d6dd59f 100644 --- a/dd-java-agent/agent-bootstrap/src/main/resources/META-INF/native-image/com.datadoghq/dd-java-agent/reflect-config.json +++ b/dd-java-agent/agent-bootstrap/src/main/resources/META-INF/native-image/com.datadoghq/dd-java-agent/reflect-config.json @@ -70,6 +70,31 @@ {"name": "toJson"} ] }, + { + "name" : "datadog.trace.agent.core.datastreams.DataStreamsTransactionExtractors$DataStreamsTransactionExtractorAdapter", + "methods": [ + {"name": "fromJson"}, + {"name": "toJson"} + ] + }, + { + "name" : "datadog.trace.agent.core.datastreams.DataStreamsTransactionExtractors$DataStreamsTransactionExtractorImpl", + "allDeclaredConstructors" : true, + "allPublicConstructors" : true, + "allDeclaredFields" : true, + "allPublicFields" : true + }, + { + "name" : "datadog.trace.api.datastreams.DataStreamsTransactionExtractor$Type", + "allDeclaredFields" : true + }, + { + "name" : "datadog.trace.agent.core.datastreams.DataStreamsTransactionExtractors$DataStreamsTransactionExtractorsAdapter", + "methods": [ + {"name": "fromJson"}, + {"name": "toJson"} + ] + }, { "name" : "datadog.trace.agent.core.DDSpanLink$SpanLinkAdapter", "methods": [ @@ -181,5 +206,17 @@ "fields": [ {"name": "consumerIndex", "allowUnsafeAccess": true} ] + }, + { + "name": "com.datadoghq.profiler.BufferWriter8", + "methods": [ + {"name": "", "parameterTypes": []} + ] + }, + { + "name": "com.datadoghq.profiler.BufferWriter9", + "methods": [ + {"name": "", "parameterTypes": []} + ] } ] diff --git a/dd-java-agent/agent-bootstrap/src/main/resources/datadog/trace/bootstrap/blocking/template.html b/dd-java-agent/agent-bootstrap/src/main/resources/datadog/trace/bootstrap/blocking/template.html index b43edd96dd5..1c3bf791a6b 100644 --- a/dd-java-agent/agent-bootstrap/src/main/resources/datadog/trace/bootstrap/blocking/template.html +++ b/dd-java-agent/agent-bootstrap/src/main/resources/datadog/trace/bootstrap/blocking/template.html @@ -1 +1 @@ -You've been blocked

Sorry, you cannot access this page. Please contact the customer service team.

\ No newline at end of file +You've been blocked

Sorry, you cannot access this page. Please contact the customer service team.

Security Response ID: [security_response_id]

diff --git a/dd-java-agent/agent-bootstrap/src/main/resources/datadog/trace/bootstrap/blocking/template.json b/dd-java-agent/agent-bootstrap/src/main/resources/datadog/trace/bootstrap/blocking/template.json index 885d766c18f..8e8a49e6a81 100644 --- a/dd-java-agent/agent-bootstrap/src/main/resources/datadog/trace/bootstrap/blocking/template.json +++ b/dd-java-agent/agent-bootstrap/src/main/resources/datadog/trace/bootstrap/blocking/template.json @@ -1 +1 @@ -{"errors":[{"title":"You've been blocked","detail":"Sorry, you cannot access this page. Please contact the customer service team. Security provided by Datadog."}]} +{"errors":[{"title":"You've been blocked","detail":"Sorry, you cannot access this page. Please contact the customer service team. Security provided by Datadog."}],"security_response_id":"[security_response_id]"} diff --git a/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/blocking/BlockingActionHelperSpecification.groovy b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/blocking/BlockingActionHelperSpecification.groovy index 25cea31e886..64c89ef9f27 100644 --- a/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/blocking/BlockingActionHelperSpecification.groovy +++ b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/blocking/BlockingActionHelperSpecification.groovy @@ -67,104 +67,150 @@ class BlockingActionHelperSpecification extends DDSpecification { null | null } - void 'getTemplate returning default html template'() { + void 'getTemplate returning default #templateType template'() { expect: - new String(BlockingActionHelper.getTemplate(HTML), StandardCharsets.UTF_8) - .contains("You've been blocked") - } + new String(BlockingActionHelper.getTemplate(templateType), StandardCharsets.UTF_8) + .contains(expectedContent) - void 'getTemplate returning default JSON template'() { - expect: - new String(BlockingActionHelper.getTemplate(JSON), StandardCharsets.UTF_8) - .contains('"You\'ve been blocked"') + where: + templateType | expectedContent + HTML | "You've been blocked" + JSON | '"You\'ve been blocked"' } - void 'getTemplate returning custom html template'() { + void 'getTemplate returning custom #templateType template'() { setup: File tempDir = File.createTempDir('testTempDir-', '') Config config = Mock(Config) - File tempFile = new File(tempDir, 'template.html') - tempFile << 'My template' + File tempFile = new File(tempDir, fileName) + tempFile << templateContent when: BlockingActionHelper.reset(config) then: - 1 * config.getAppSecHttpBlockedTemplateHtml() >> tempFile.toString() - 1 * config.getAppSecHttpBlockedTemplateJson() >> null - new String(BlockingActionHelper.getTemplate(HTML), StandardCharsets.UTF_8) - .contains('My template') + 1 * config.getAppSecHttpBlockedTemplateHtml() >> (templateType == HTML ? tempFile.toString() : null) + 1 * config.getAppSecHttpBlockedTemplateJson() >> (templateType == JSON ? tempFile.toString() : null) + new String(BlockingActionHelper.getTemplate(templateType), StandardCharsets.UTF_8) + .contains(templateContent) cleanup: BlockingActionHelper.reset(Config.get()) tempDir.deleteDir() + + where: + templateType | fileName | templateContent + HTML | 'template.html' | 'My template' + JSON | 'template.json' | '{"foo":"bar"}' + } + + void 'getTemplate with null argument'() { + expect: + BlockingActionHelper.getTemplate(null) == null } - void 'getTemplate returning custom json template'() { + void 'will use default #templateType template if #reason'() { setup: - File tempDir = File.createTempDir('testTempDir-', '') Config config = Mock(Config) - File tempFile = new File(tempDir, 'template.json') - tempFile << '{"foo":"bar"}' + File tempDir = tempDirSetup ? File.createTempDir('testTempDir-', '') : null + File template = tempFile ? new File(tempDir, 'template') : null + if (template) { + template << 'a' * (500 * 1024 + 1) + } when: BlockingActionHelper.reset(config) then: - 1 * config.getAppSecHttpBlockedTemplateHtml() >> null - 1 * config.getAppSecHttpBlockedTemplateJson() >> tempFile.toString() - new String(BlockingActionHelper.getTemplate(JSON), StandardCharsets.UTF_8) - .contains('{"foo":"bar"}') + 1 * config.getAppSecHttpBlockedTemplateHtml() >> htmlConfigValue?.call(template) + 1 * config.getAppSecHttpBlockedTemplateJson() >> jsonConfigValue?.call(template) + new String(BlockingActionHelper.getTemplate(templateType), StandardCharsets.UTF_8) + .contains(expectedContent) cleanup: BlockingActionHelper.reset(Config.get()) - tempDir.deleteDir() - } + if (tempDir) { + tempDir.deleteDir() + } - void 'getTemplate with null argument'() { - expect: - BlockingActionHelper.getTemplate(null) == null + where: + templateType | reason | tempDirSetup | tempFile | htmlConfigValue | jsonConfigValue | expectedContent + HTML | 'custom file does not exist' | false | false | { _ -> '/bad/file.html' } | { _ -> '/bad/file.json' } | "You've been blocked" + JSON | 'custom file does not exist' | false | false | { _ -> '/bad/file.html' } | { _ -> '/bad/file.json' } | '"You\'ve been blocked' + HTML | 'custom file is too big' | true | true | { it -> it.toString() } | { it -> it.toString() } | "You've been blocked" + JSON | 'custom file is too big' | true | true | { it -> it.toString() } | { it -> it.toString() } | '"You\'ve been blocked' } - void 'will use default templates if custom files do not exist'() { - setup: - Config config = Mock(Config) + + void 'getTemplate with security_response_id replaces placeholder in #templateType template'() { + given: + def securityResponseId = '12345678-1234-1234-1234-123456789abc' when: - BlockingActionHelper.reset(config) + def template = BlockingActionHelper.getTemplate(templateType, securityResponseId) + def templateStr = new String(template, StandardCharsets.UTF_8) then: - 1 * config.getAppSecHttpBlockedTemplateHtml() >> '/bad/file.html' - 1 * config.getAppSecHttpBlockedTemplateJson() >> '/bad/file.json' - new String(BlockingActionHelper.getTemplate(HTML), StandardCharsets.UTF_8) - .contains("You've been blocked") - new String(BlockingActionHelper.getTemplate(JSON), StandardCharsets.UTF_8) - .contains('"You\'ve been blocked') + !templateStr.contains('[security_response_id]') + templateStr.contains(expectedContent.replace('[id]', securityResponseId)) - cleanup: - BlockingActionHelper.reset(Config.get()) + where: + templateType | expectedContent + HTML | 'Security Response ID: [id]' + JSON | '"security_response_id":"[id]"' + } + + void 'getTemplate without security_response_id uses empty string in #templateType template'() { + when: + def template = BlockingActionHelper.getTemplate(templateType, null) + def templateStr = new String(template, StandardCharsets.UTF_8) + + then: + !templateStr.contains('[security_response_id]') + expectedContents.every { content -> templateStr.contains(content) } + + where: + templateType | expectedContents + HTML | ['Security Response ID:'] + JSON | ['"security_response_id"', '""'] + } + + void 'getTemplate with empty security_response_id uses empty string'() { + when: + def htmlTemplate = BlockingActionHelper.getTemplate(HTML, '') + def jsonTemplate = BlockingActionHelper.getTemplate(JSON, '') + + then: + !new String(htmlTemplate, StandardCharsets.UTF_8).contains('[security_response_id]') + !new String(jsonTemplate, StandardCharsets.UTF_8).contains('[security_response_id]') } - void 'will use default templates if custom files are too big'() { + void 'getTemplate with security_response_id works with custom #templateType template'() { setup: - Config config = Mock(Config) File tempDir = File.createTempDir('testTempDir-', '') - File template = new File(tempDir, 'template') - template << 'a' * (500 * 1024 + 1) + Config config = Mock(Config) + File tempFile = new File(tempDir, fileName) + tempFile << templateContent + def securityResponseId = 'test-block-id-123' when: BlockingActionHelper.reset(config) + def template = BlockingActionHelper.getTemplate(templateType, securityResponseId) + def templateStr = new String(template, StandardCharsets.UTF_8) then: - 1 * config.getAppSecHttpBlockedTemplateHtml() >> template.toString() - 1 * config.getAppSecHttpBlockedTemplateJson() >> template.toString() - new String(BlockingActionHelper.getTemplate(HTML), StandardCharsets.UTF_8) - .contains("You've been blocked") - new String(BlockingActionHelper.getTemplate(JSON), StandardCharsets.UTF_8) - .contains('"You\'ve been blocked') + 1 * config.getAppSecHttpBlockedTemplateHtml() >> (templateType == HTML ? tempFile.toString() : null) + 1 * config.getAppSecHttpBlockedTemplateJson() >> (templateType == JSON ? tempFile.toString() : null) + templateStr.contains(expectedContent.replace('[id]', securityResponseId)) + !templateStr.contains('[security_response_id]') cleanup: BlockingActionHelper.reset(Config.get()) tempDir.deleteDir() + + where: + templateType | fileName | templateContent | expectedContent + HTML | 'template.html' | 'Custom template with security_response_id: [security_response_id]' | 'Custom template with security_response_id: [id]' + JSON | 'template.json' | '{"error":"blocked","id":"[security_response_id]"}' | '"error":"blocked","id":"[id]"' } } diff --git a/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/dbm/SharedDBCommenterForkedTest.groovy b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/dbm/SharedDBCommenterForkedTest.groovy new file mode 100644 index 00000000000..c655b5c1f97 --- /dev/null +++ b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/dbm/SharedDBCommenterForkedTest.groovy @@ -0,0 +1,142 @@ +package datadog.trace.bootstrap.instrumentation.dbm + +import spock.lang.Specification + +class SharedDBCommenterForkedTest extends Specification { + def setup() { + System.setProperty("dd.service.name", "test-service") + System.setProperty("dd.env", "test-env") + System.setProperty("dd.version", "1.0.0") + } + + def cleanup() { + System.clearProperty("dd.service.name") + System.clearProperty("dd.env") + System.clearProperty("dd.version") + } + + def "buildComment generates expected format for MongoDB"() { + when: + String comment = SharedDBCommenter.buildComment( + "my-db-service", "mongodb", "localhost", "testdb", null + ) + + then: + comment != null + comment.contains("ddps='test-service'") + comment.contains("dddbs='my-db-service'") + comment.contains("dde='test-env'") + comment.contains("ddpv='1.0.0'") + !comment.contains("traceparent") + } + + def "buildComment includes traceparent when provided"() { + when: + String traceParent = "00-1234567890123456789012345678901234-9876543210987654-01" + String comment = SharedDBCommenter.buildComment( + "my-db-service", "mongodb", "localhost", "testdb", traceParent + ) + + then: + comment != null + comment.contains("ddps='test-service'") + comment.contains("dddbs='my-db-service'") + comment.contains("traceparent='$traceParent'") + } + + def "buildComment handles null values gracefully"() { + when: + String comment = SharedDBCommenter.buildComment( + dbService, dbType, hostname, dbName, traceParent + ) + + then: + comment != null || expectedNull + + where: + dbService | dbType | hostname | dbName | traceParent | expectedNull + null | "mongodb" | "host" | "db" | null | false + "" | "mongodb" | "host" | "db" | null | false + "service" | "mongodb" | null | "db" | null | false + "service" | "mongodb" | "" | "db" | null | false + "service" | "mongodb" | "host" | null | null | false + "service" | "mongodb" | "host" | "" | null | false + } + + def "buildComment includes hostname when provided"() { + when: + String comment = SharedDBCommenter.buildComment( + "my-service", "mongodb", "prod-host", "mydb", null + ) + + then: + comment != null + comment.contains("ddh='prod-host'") + comment.contains("dddb='mydb'") + } + + def "containsTraceComment detects DD fields correctly"() { + when: + boolean hasComment = SharedDBCommenter.containsTraceComment(commentContent) + + then: + hasComment == expected + + where: + commentContent | expected + "ddps='service',dddbs='db'" | true + "dde='env',ddpv='1.0'" | true + "traceparent='00-123-456-01'" | true + "user comment" | false + "" | false + "some other comment with ddps but not the right format" | false + "ddps='test',dddbs='db',dde='env'" | true + "prefix ddps='service' suffix" | true + } + + def "buildComment escapes special characters"() { + when: + String comment = SharedDBCommenter.buildComment( + "service with spaces", "mongodb", "host'with'quotes", "db&name", null + ) + + then: + comment != null + comment.contains("dddbs='service+with+spaces'") + comment.contains("ddh='host%27with%27quotes'") + comment.contains("dddb='db%26name'") + } + + + def "buildComment works with different database types"() { + when: + String comment = SharedDBCommenter.buildComment( + "my-service", dbType, "localhost", "testdb", null + ) + + then: + comment != null + comment.contains("ddps='test-service'") + comment.contains("dddbs='my-service'") + + where: + dbType << ["mongodb", "mysql", "postgresql", "oracle"] + } + + def "buildComment format matches expected pattern"() { + when: + String comment = SharedDBCommenter.buildComment( + "test-db", "mongodb", "host", "db", "00-trace-span-01" + ) + + then: + comment != null + // Comment should be comma-separated key=value pairs + def parts = comment.split(",") + parts.size() >= 3 + parts.each { part -> + assert part.contains("=") + assert part.contains("'") + } + } +} diff --git a/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/BaseDecoratorTest.groovy b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/BaseDecoratorTest.groovy index b94b7013f39..354a9c6bc4f 100644 --- a/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/BaseDecoratorTest.groovy +++ b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/BaseDecoratorTest.groovy @@ -1,15 +1,20 @@ package datadog.trace.bootstrap.instrumentation.decorator - +import datadog.trace.api.TagMap import datadog.trace.bootstrap.instrumentation.api.AgentSpan import datadog.trace.bootstrap.instrumentation.api.AgentSpanContext import datadog.trace.bootstrap.instrumentation.api.ErrorPriorities import datadog.trace.bootstrap.instrumentation.api.Tags +import datadog.trace.config.inversion.ConfigHelper import datadog.trace.test.util.DDSpecification import spock.lang.Shared class BaseDecoratorTest extends DDSpecification { + def setupSpec() { + ConfigHelper.get().setConfigInversionStrict(ConfigHelper.StrictnessPolicy.TEST) + } + @Shared def decorator = newDecorator() @@ -25,13 +30,17 @@ class BaseDecoratorTest extends DDSpecification { then: 1 * span.setSpanType(decorator.spanType()) - 1 * span.setTag(Tags.COMPONENT, "test-component") + 1 * span.setTag(TagMap.Entry.create(Tags.COMPONENT, "test-component")) 1 * span.context() >> spanContext 1 * spanContext.setIntegrationName("test-component") + _ * span.setTag(_) _ * span.setTag(_, _) // Want to allow other calls from child implementations. + _ * span.setTag(_) _ * span.setMeasured(true) + _ * span.setMetric(_) _ * span.setMetric(_, _) - _ * span.setServiceName(_) + _ * span.setMetric(_) + _ * span.setServiceName(_, _) _ * span.setOperationName(_) _ * span.setSamplingPriority(_) 0 * _ diff --git a/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/ClientDecoratorTest.groovy b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/ClientDecoratorTest.groovy index 5153eeb09e0..3b40a04a9bb 100644 --- a/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/ClientDecoratorTest.groovy +++ b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/ClientDecoratorTest.groovy @@ -1,6 +1,7 @@ package datadog.trace.bootstrap.instrumentation.decorator import datadog.trace.api.DDTags +import datadog.trace.api.TagMap import datadog.trace.bootstrap.instrumentation.api.AgentSpan import datadog.trace.bootstrap.instrumentation.api.AgentSpanContext import datadog.trace.bootstrap.instrumentation.api.Tags @@ -19,16 +20,18 @@ class ClientDecoratorTest extends BaseDecoratorTest { then: if (serviceName != null) { - 1 * span.setServiceName(serviceName) + 1 * span.setServiceName(serviceName, "test-component") } 1 * span.setMeasured(true) - 1 * span.setTag(Tags.COMPONENT, "test-component") + 1 * span.setTag(TagMap.Entry.create(Tags.COMPONENT, "test-component")) 1 * span.context() >> spanContext 1 * spanContext.setIntegrationName("test-component") - 1 * span.setTag(Tags.SPAN_KIND, "client") + 1 * span.setTag(TagMap.Entry.create(Tags.SPAN_KIND, "client")) 1 * span.setSpanType(decorator.spanType()) - 1 * span.setMetric(DDTags.ANALYTICS_SAMPLE_RATE, 1.0) + 1 * span.setMetric(TagMap.Entry.create(DDTags.ANALYTICS_SAMPLE_RATE, 1.0)) + _ * span.setTag(_) _ * span.setTag(_, _) // Want to allow other calls from child implementations. + _ * span.setTag(_) _ * span.setServiceName(_) _ * span.setOperationName(_) 0 * _ diff --git a/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/DBTypeProcessingDatabaseClientDecoratorTest.groovy b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/DBTypeProcessingDatabaseClientDecoratorTest.groovy index 5e4e3d7d33f..85b9d6fd66a 100644 --- a/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/DBTypeProcessingDatabaseClientDecoratorTest.groovy +++ b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/DBTypeProcessingDatabaseClientDecoratorTest.groovy @@ -1,6 +1,7 @@ package datadog.trace.bootstrap.instrumentation.decorator import datadog.trace.api.DDTags +import datadog.trace.api.TagMap import datadog.trace.bootstrap.instrumentation.api.AgentSpan import datadog.trace.bootstrap.instrumentation.api.Tags import datadog.trace.bootstrap.instrumentation.api.UTF8BytesString @@ -20,13 +21,13 @@ class DBTypeProcessingDatabaseClientDecoratorTest extends ClientDecoratorTest { then: if (serviceName != null) { - 1 * span.setServiceName(serviceName) + 1 * span.setServiceName(serviceName, "test-component") } 1 * span.setMeasured(true) 1 * span.setTag(Tags.COMPONENT, "test-component") 1 * span.context() >> spanContext 1 * spanContext.setIntegrationName("test-component") - 1 * span.setTag(Tags.SPAN_KIND, "client") + 1 * span.setTag(TagMap.Entry.create(Tags.SPAN_KIND, "client")) 1 * span.setSpanType("test-type") 1 * span.setServiceName("test-db") 1 * span.setOperationName(UTF8BytesString.create("test-db.query")) diff --git a/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/DatabaseClientDecoratorTest.groovy b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/DatabaseClientDecoratorTest.groovy index e0ecce67b8e..164138cbe0f 100644 --- a/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/DatabaseClientDecoratorTest.groovy +++ b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/DatabaseClientDecoratorTest.groovy @@ -1,6 +1,7 @@ package datadog.trace.bootstrap.instrumentation.decorator import datadog.trace.api.DDTags +import datadog.trace.api.TagMap import datadog.trace.bootstrap.instrumentation.api.AgentSpan import datadog.trace.bootstrap.instrumentation.api.AgentSpanContext import datadog.trace.bootstrap.instrumentation.api.Tags @@ -23,15 +24,15 @@ class DatabaseClientDecoratorTest extends ClientDecoratorTest { then: if (serviceName != null) { - 1 * span.setServiceName(serviceName) + 1 * span.setServiceName(serviceName, "test-component") } 1 * span.setMeasured(true) - 1 * span.setTag(Tags.COMPONENT, "test-component") + 1 * span.setTag(TagMap.Entry.create(Tags.COMPONENT, "test-component")) 1 * span.context() >> spanContext 1 * spanContext.setIntegrationName("test-component") - 1 * span.setTag(Tags.SPAN_KIND, "client") + 1 * span.setTag(TagMap.Entry.create(Tags.SPAN_KIND, "client")) 1 * span.setSpanType("test-type") - 1 * span.setMetric(DDTags.ANALYTICS_SAMPLE_RATE, 1.0) + 1 * span.setMetric(TagMap.Entry.create(DDTags.ANALYTICS_SAMPLE_RATE, 1.0)) 0 * _ where: @@ -58,11 +59,11 @@ class DatabaseClientDecoratorTest extends ClientDecoratorTest { 1 * span.setTag(Tags.PEER_HOSTNAME, session.hostname) } if (instanceTypeSuffix && renameByInstance && session.instance) { - 1 * span.setServiceName(session.instance + "-" + decorator.dbType()) + 1 * span.setServiceName(session.instance + "-" + decorator.dbType(), _) } else if (renameByInstance && session.instance) { - 1 * span.setServiceName(session.instance) + 1 * span.setServiceName(session.instance, _) } else if (renameByHost) { - 1 * span.setServiceName(session.hostname) + 1 * span.setServiceName(session.hostname, _) } } 0 * _ diff --git a/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/HttpClientDecoratorTest.groovy b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/HttpClientDecoratorTest.groovy index bd98bc53226..1bc83457bd0 100644 --- a/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/HttpClientDecoratorTest.groovy +++ b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/HttpClientDecoratorTest.groovy @@ -72,7 +72,7 @@ class HttpClientDecoratorTest extends ClientDecoratorTest { 1 * span.setTag(Tags.PEER_PORT, req.url.port) 1 * span.setResourceName({ it as String == req.method.toUpperCase() + " " + req.path }, ResourceNamePriorities.HTTP_PATH_NORMALIZER) if (renameService) { - 1 * span.setServiceName(req.url.host) + 1 * span.setServiceName(req.url.host, _) } 1 * span.traceConfig() >> AgentTracer.traceConfig() } @@ -147,7 +147,7 @@ class HttpClientDecoratorTest extends ClientDecoratorTest { then: if (expectedServiceName) { - 1 * span.setServiceName(expectedServiceName) + 1 * span.setServiceName(expectedServiceName, _) } if (url != null) { 1 * span.setResourceName(_, _) diff --git a/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/HttpServerDecoratorTest.groovy b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/HttpServerDecoratorTest.groovy index 7c3efb1d48c..c56efc5f014 100644 --- a/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/HttpServerDecoratorTest.groovy +++ b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/HttpServerDecoratorTest.groovy @@ -468,6 +468,11 @@ class HttpServerDecoratorTest extends ServerDecoratorTest { protected int status(Map m) { return m.status == null ? 0 : m.status } + + @Override + protected String getRequestHeader(Map map, String key) { + return map.getOrDefault(key, null) + } } } diff --git a/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/ServerDecoratorTest.groovy b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/ServerDecoratorTest.groovy index 9eeb1b19e77..ae41a1f523b 100644 --- a/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/ServerDecoratorTest.groovy +++ b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/decorator/ServerDecoratorTest.groovy @@ -1,6 +1,6 @@ package datadog.trace.bootstrap.instrumentation.decorator - +import datadog.trace.api.TagMap import datadog.trace.bootstrap.instrumentation.api.AgentSpan import datadog.trace.bootstrap.instrumentation.api.AgentSpanContext @@ -22,14 +22,16 @@ class ServerDecoratorTest extends BaseDecoratorTest { decorator.afterStart(span) then: - 1 * span.setTag(LANGUAGE_TAG_KEY, LANGUAGE_TAG_VALUE) - 1 * span.setTag(COMPONENT, "test-component") + 1 * span.setTag(TagMap.Entry.create(LANGUAGE_TAG_KEY, LANGUAGE_TAG_VALUE)) + 1 * span.setTag(TagMap.Entry.create(COMPONENT, "test-component")) 1 * span.context() >> spanContext 1 * spanContext.setIntegrationName("test-component") - 1 * span.setTag(SPAN_KIND, "server") + 1 * span.setTag(TagMap.Entry.create(SPAN_KIND, "server")) 1 * span.setSpanType(decorator.spanType()) if (decorator.traceAnalyticsEnabled) { - 1 * span.setMetric(ANALYTICS_SAMPLE_RATE, 1.0) + 1 * span.setMetric(TagMap.Entry.create(ANALYTICS_SAMPLE_RATE, 1.0)) + } else { + 1 * span.setMetric(null) } 0 * _ } diff --git a/dd-java-agent/agent-tooling/src/test/groovy/datadog/trace/agent/tooling/log/UnionMapTest.groovy b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/log/UnionMapTest.groovy similarity index 98% rename from dd-java-agent/agent-tooling/src/test/groovy/datadog/trace/agent/tooling/log/UnionMapTest.groovy rename to dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/log/UnionMapTest.groovy index 4ee00025ed6..6f40b1900ec 100644 --- a/dd-java-agent/agent-tooling/src/test/groovy/datadog/trace/agent/tooling/log/UnionMapTest.groovy +++ b/dd-java-agent/agent-bootstrap/src/test/groovy/datadog/trace/bootstrap/instrumentation/log/UnionMapTest.groovy @@ -1,4 +1,4 @@ -package datadog.trace.agent.tooling.log +package datadog.trace.bootstrap.instrumentation.log import com.google.common.collect.testing.MapTestSuiteBuilder diff --git a/dd-java-agent/agent-builder/build.gradle b/dd-java-agent/agent-builder/build.gradle deleted file mode 100644 index b18f70658d7..00000000000 --- a/dd-java-agent/agent-builder/build.gradle +++ /dev/null @@ -1,44 +0,0 @@ -apply from: "$rootDir/gradle/java.gradle" -apply plugin: "idea" - -minimumBranchCoverage = 0.6 -excludedClassesCoverage += ['datadog.trace.agent.tooling.*'] - -dependencies { - api project(':dd-java-agent:agent-tooling') - - implementation project(':dd-java-agent:agent-otel:otel-tooling') - - testImplementation project(':dd-java-agent:testing') -} - -// Use Java 11 to build a delegating ClassFileTransformer that understands Java modules -sourceSets { - "main_java11" { - java.srcDirs "${project.projectDir}/src/main/java11" - } -} - -tasks.named("compileMain_java11Java", JavaCompile) { - configureCompiler(it, 11, JavaVersion.VERSION_1_8) -} - -dependencies { - main_java11CompileOnly project(':dd-java-agent:agent-tooling') - main_java11CompileOnly sourceSets.main.output - runtimeOnly sourceSets.main_java11.output -} - -tasks.named("jar", Jar) { - from sourceSets.main_java11.output -} - -tasks.named("forbiddenApisMain_java11") { - failOnMissingClasses = false -} - -idea { - module { - jdkName = '11' - } -} diff --git a/dd-java-agent/agent-builder/gradle.lockfile b/dd-java-agent/agent-builder/gradle.lockfile deleted file mode 100644 index d1e26b993e8..00000000000 --- a/dd-java-agent/agent-builder/gradle.lockfile +++ /dev/null @@ -1,155 +0,0 @@ -# This is a Gradle generated file for dependency locking. -# Manual edits can break the build and are not advised. -# This file is expected to be part of source control. -cafe.cryptography:curve25519-elisabeth:0.1.0=runtimeClasspath,testRuntimeClasspath -cafe.cryptography:ed25519-elisabeth:0.1.0=runtimeClasspath,testRuntimeClasspath -ch.qos.logback:logback-classic:1.2.13=testCompileClasspath,testRuntimeClasspath -ch.qos.logback:logback-core:1.2.13=testCompileClasspath,testRuntimeClasspath -com.beust:jcommander:1.78=testRuntimeClasspath -com.blogspot.mydailyjava:weak-lock-free:0.17=compileClasspath,main_java11CompileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.datadoghq.okhttp3:okhttp:3.12.15=compileClasspath,main_java11CompileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.datadoghq.okio:okio:1.17.6=compileClasspath,main_java11CompileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.datadoghq:dd-instrument-java:0.0.2=compileClasspath,main_java11CompileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.datadoghq:dd-javac-plugin-client:0.2.2=compileClasspath,main_java11CompileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.datadoghq:java-dogstatsd-client:4.4.3=runtimeClasspath,testRuntimeClasspath -com.datadoghq:sketches-java:0.8.3=runtimeClasspath,testRuntimeClasspath -com.github.javaparser:javaparser-core:3.25.6=codenarc,testCompileClasspath,testRuntimeClasspath -com.github.jnr:jffi:1.3.13=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-a64asm:1.0.0=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-constants:0.10.4=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-enxio:0.32.17=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-ffi:2.2.16=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-posix:3.1.19=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-unixsocket:0.38.22=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-x86asm:1.0.2=runtimeClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.2.0=compileClasspath,testCompileClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.7.3=spotbugs -com.github.spotbugs:spotbugs:4.7.3=spotbugs -com.google.code.findbugs:jsr305:3.0.2=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -com.google.code.gson:gson:2.9.1=spotbugs -com.google.guava:guava:20.0=testCompileClasspath,testRuntimeClasspath -com.google.re2j:re2j:1.7=runtimeClasspath,testRuntimeClasspath -com.squareup.moshi:moshi:1.11.0=compileClasspath,main_java11CompileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.squareup.okhttp3:logging-interceptor:3.12.12=testCompileClasspath,testRuntimeClasspath -com.squareup.okhttp3:okhttp:3.12.12=testCompileClasspath,testRuntimeClasspath -com.squareup.okio:okio:1.17.5=compileClasspath,main_java11CompileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.thoughtworks.qdox:qdox:1.12.1=codenarc,testRuntimeClasspath -commons-codec:commons-codec:1.15=spotbugs -commons-fileupload:commons-fileupload:1.5=testCompileClasspath,testRuntimeClasspath -commons-io:commons-io:2.11.0=testCompileClasspath,testRuntimeClasspath -de.thetaphi:forbiddenapis:3.8=compileClasspath -info.picocli:picocli:4.6.3=testRuntimeClasspath -io.leangen.geantyref:geantyref:1.3.16=testRuntimeClasspath -io.sqreen:libsqreen:17.2.0=testRuntimeClasspath -javax.servlet:javax.servlet-api:3.1.0=testCompileClasspath,testRuntimeClasspath -jaxen:jaxen:1.2.0=spotbugs -jline:jline:2.14.6=testRuntimeClasspath -junit:junit:4.13.2=testCompileClasspath,testRuntimeClasspath -net.bytebuddy:byte-buddy-agent:1.17.7=compileClasspath,main_java11CompileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -net.bytebuddy:byte-buddy:1.17.7=compileClasspath,main_java11CompileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -net.java.dev.jna:jna-platform:5.8.0=runtimeClasspath,testRuntimeClasspath -net.java.dev.jna:jna:5.8.0=runtimeClasspath,testRuntimeClasspath -net.jcip:jcip-annotations:1.0=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -net.sf.saxon:Saxon-HE:11.4=spotbugs -org.apache.ant:ant-antlr:1.10.14=codenarc -org.apache.ant:ant-antlr:1.10.15=testRuntimeClasspath -org.apache.ant:ant-junit:1.10.14=codenarc -org.apache.ant:ant-junit:1.10.15=testRuntimeClasspath -org.apache.ant:ant-launcher:1.10.15=testRuntimeClasspath -org.apache.ant:ant:1.10.15=testCompileClasspath,testRuntimeClasspath -org.apache.bcel:bcel:6.5.0=spotbugs -org.apache.commons:commons-lang3:3.12.0=spotbugs -org.apache.commons:commons-text:1.10.0=spotbugs -org.apache.httpcomponents.client5:httpclient5:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5-h2:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5:5.1.3=spotbugs -org.apache.logging.log4j:log4j-api:2.19.0=spotbugs -org.apache.logging.log4j:log4j-core:2.19.0=spotbugs -org.apiguardian:apiguardian-api:1.1.2=testCompileClasspath -org.codehaus.groovy:groovy-all:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-ant:3.0.23=codenarc -org.codehaus.groovy:groovy-ant:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-astbuilder:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-cli-picocli:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-console:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-datetime:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-docgenerator:3.0.23=codenarc -org.codehaus.groovy:groovy-docgenerator:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-groovydoc:3.0.23=codenarc -org.codehaus.groovy:groovy-groovydoc:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-groovysh:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jmx:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-json:3.0.23=codenarc -org.codehaus.groovy:groovy-json:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jsr223:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-macro:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-nio:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-servlet:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-sql:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-swing:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-templates:3.0.23=codenarc -org.codehaus.groovy:groovy-templates:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test-junit5:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-testng:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-xml:3.0.23=codenarc -org.codehaus.groovy:groovy-xml:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy:3.0.23=codenarc -org.codehaus.groovy:groovy:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codenarc:CodeNarc:3.6.0=codenarc -org.dom4j:dom4j:2.1.3=spotbugs -org.eclipse.jetty:jetty-http:9.4.56.v20240826=testCompileClasspath,testRuntimeClasspath -org.eclipse.jetty:jetty-io:9.4.56.v20240826=testCompileClasspath,testRuntimeClasspath -org.eclipse.jetty:jetty-server:9.4.56.v20240826=testCompileClasspath,testRuntimeClasspath -org.eclipse.jetty:jetty-util:9.4.56.v20240826=testCompileClasspath,testRuntimeClasspath -org.gmetrics:GMetrics:2.1.0=codenarc -org.hamcrest:hamcrest-core:1.3=testCompileClasspath,testRuntimeClasspath -org.hamcrest:hamcrest:3.0=testCompileClasspath,testRuntimeClasspath -org.jacoco:org.jacoco.agent:0.8.14=jacocoAgent,jacocoAnt -org.jacoco:org.jacoco.ant:0.8.14=jacocoAnt -org.jacoco:org.jacoco.core:0.8.14=jacocoAnt -org.jacoco:org.jacoco.report:0.8.14=jacocoAnt -org.jctools:jctools-core:3.3.0=runtimeClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-api:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-engine:5.12.2=testRuntimeClasspath -org.junit.jupiter:junit-jupiter-params:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-commons:1.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-engine:1.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-launcher:1.12.2=testRuntimeClasspath -org.junit.platform:junit-platform-runner:1.12.2=testRuntimeClasspath -org.junit.platform:junit-platform-suite-api:1.12.2=testRuntimeClasspath -org.junit.platform:junit-platform-suite-commons:1.12.2=testRuntimeClasspath -org.junit:junit-bom:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit:junit-bom:5.9.1=spotbugs -org.mockito:mockito-core:4.4.0=testRuntimeClasspath -org.objenesis:objenesis:3.3=testCompileClasspath,testRuntimeClasspath -org.opentest4j:opentest4j:1.3.0=testCompileClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.4=spotbugs -org.ow2.asm:asm-commons:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-commons:9.4=spotbugs -org.ow2.asm:asm-commons:9.9=jacocoAnt -org.ow2.asm:asm-tree:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-tree:9.4=spotbugs -org.ow2.asm:asm-tree:9.9=jacocoAnt -org.ow2.asm:asm-util:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-util:9.4=spotbugs -org.ow2.asm:asm:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm:9.4=spotbugs -org.ow2.asm:asm:9.9=jacocoAnt -org.slf4j:jcl-over-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath -org.slf4j:jul-to-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath -org.slf4j:log4j-over-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath -org.slf4j:slf4j-api:1.7.30=compileClasspath,main_java11CompileClasspath,runtimeClasspath -org.slf4j:slf4j-api:1.7.32=testCompileClasspath,testRuntimeClasspath -org.slf4j:slf4j-api:2.0.0=spotbugs,spotbugsSlf4j -org.slf4j:slf4j-simple:2.0.0=spotbugsSlf4j -org.snakeyaml:snakeyaml-engine:2.9=runtimeClasspath,testRuntimeClasspath -org.spockframework:spock-bom:2.4-M6-groovy-3.0=testCompileClasspath,testRuntimeClasspath -org.spockframework:spock-core:2.4-M6-groovy-3.0=testCompileClasspath,testRuntimeClasspath -org.testng:testng:7.5.1=testRuntimeClasspath -org.webjars:jquery:3.5.1=testRuntimeClasspath -org.xmlresolver:xmlresolver:4.4.3=spotbugs -xml-apis:xml-apis:1.4.01=spotbugs -empty=annotationProcessor,main_java11AnnotationProcessor,main_java11RuntimeClasspath,spotbugsPlugins,testAnnotationProcessor diff --git a/dd-java-agent/agent-ci-visibility/build.gradle b/dd-java-agent/agent-ci-visibility/build.gradle index e0dd263bbd3..323553b8c03 100644 --- a/dd-java-agent/agent-ci-visibility/build.gradle +++ b/dd-java-agent/agent-ci-visibility/build.gradle @@ -4,7 +4,7 @@ import org.jetbrains.kotlin.gradle.dsl.KotlinVersion plugins { id 'com.gradleup.shadow' - id 'org.jetbrains.kotlin.jvm' version libs.versions.kotlin.plugin + id 'org.jetbrains.kotlin.jvm' } apply from: "$rootDir/gradle/java.gradle" @@ -19,6 +19,7 @@ dependencies { api libs.slf4j implementation libs.bundles.asm + implementation libs.instrument.java implementation group: 'org.jacoco', name: 'org.jacoco.core', version: '0.8.14' implementation group: 'org.jacoco', name: 'org.jacoco.report', version: '0.8.14' @@ -36,6 +37,7 @@ dependencies { testImplementation group: 'org.skyscreamer', name: 'jsonassert', version: '1.5.1' testImplementation group: 'org.freemarker', name: 'freemarker', version: '2.3.31' testImplementation group: 'org.msgpack', name: 'jackson-dataformat-msgpack', version: '0.9.6' + testImplementation libs.bundles.mockito } tasks.named("shadowJar", ShadowJar) { diff --git a/dd-java-agent/agent-ci-visibility/civisibility-instrumentation-test-fixtures/build.gradle b/dd-java-agent/agent-ci-visibility/civisibility-instrumentation-test-fixtures/build.gradle index dec74075fae..01f83345bfe 100644 --- a/dd-java-agent/agent-ci-visibility/civisibility-instrumentation-test-fixtures/build.gradle +++ b/dd-java-agent/agent-ci-visibility/civisibility-instrumentation-test-fixtures/build.gradle @@ -4,5 +4,8 @@ apply from: "$rootDir/gradle/version.gradle" dependencies { api project(':dd-java-agent:instrumentation-testing') api project(':dd-java-agent:agent-ci-visibility:civisibility-test-fixtures') + + compileOnly(libs.bundles.groovy) + compileOnly(libs.bundles.spock) } diff --git a/dd-java-agent/agent-ci-visibility/civisibility-instrumentation-test-fixtures/gradle.lockfile b/dd-java-agent/agent-ci-visibility/civisibility-instrumentation-test-fixtures/gradle.lockfile index 3e8bac579ae..77b5710aa81 100644 --- a/dd-java-agent/agent-ci-visibility/civisibility-instrumentation-test-fixtures/gradle.lockfile +++ b/dd-java-agent/agent-ci-visibility/civisibility-instrumentation-test-fixtures/gradle.lockfile @@ -5,32 +5,32 @@ cafe.cryptography:curve25519-elisabeth:0.1.0=runtimeClasspath,testRuntimeClasspa cafe.cryptography:ed25519-elisabeth:0.1.0=runtimeClasspath,testRuntimeClasspath ch.qos.logback:logback-classic:1.2.13=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath ch.qos.logback:logback-core:1.2.13=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.beust:jcommander:1.78=runtimeClasspath,testRuntimeClasspath com.blogspot.mydailyjava:weak-lock-free:0.17=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.datadoghq.okhttp3:okhttp:3.12.15=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.datadoghq.okio:okio:1.17.6=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.datadoghq:dd-instrument-java:0.0.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +com.datadoghq:dd-instrument-java:0.0.3=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.datadoghq:dd-javac-plugin-client:0.2.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.datadoghq:java-dogstatsd-client:4.4.3=runtimeClasspath,testRuntimeClasspath +com.datadoghq:java-dogstatsd-client:4.4.5=runtimeClasspath,testRuntimeClasspath com.datadoghq:sketches-java:0.8.3=runtimeClasspath,testRuntimeClasspath com.fasterxml.jackson.core:jackson-annotations:2.20=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.fasterxml.jackson.core:jackson-core:2.20.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.fasterxml.jackson.core:jackson-databind:2.20.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.fasterxml.jackson:jackson-bom:2.20.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.github.javaparser:javaparser-core:3.25.6=codenarc,compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.github.jnr:jffi:1.3.13=runtimeClasspath,testRuntimeClasspath +com.github.javaparser:javaparser-core:3.25.6=codenarc +com.github.jnr:jffi:1.3.14=runtimeClasspath,testRuntimeClasspath com.github.jnr:jnr-a64asm:1.0.0=runtimeClasspath,testRuntimeClasspath com.github.jnr:jnr-constants:0.10.4=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-enxio:0.32.17=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-ffi:2.2.16=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-posix:3.1.19=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-unixsocket:0.38.22=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-enxio:0.32.19=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-ffi:2.2.18=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-posix:3.1.21=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-unixsocket:0.38.24=runtimeClasspath,testRuntimeClasspath com.github.jnr:jnr-x86asm:1.0.2=runtimeClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.2.0=compileClasspath,testCompileClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.7.3=spotbugs -com.github.spotbugs:spotbugs:4.7.3=spotbugs +com.github.spotbugs:spotbugs-annotations:4.9.8=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath +com.github.spotbugs:spotbugs:4.9.8=spotbugs +com.github.stephenc.jcip:jcip-annotations:1.0-1=spotbugs com.google.code.findbugs:jsr305:3.0.2=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -com.google.code.gson:gson:2.9.1=spotbugs +com.google.code.gson:gson:2.13.2=spotbugs +com.google.errorprone:error_prone_annotations:2.41.0=spotbugs com.google.guava:guava:20.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.google.re2j:re2j:1.7=runtimeClasspath,testRuntimeClasspath com.jayway.jsonpath:json-path:2.8.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath @@ -38,128 +38,93 @@ com.squareup.moshi:moshi:1.11.0=compileClasspath,runtimeClasspath,testCompileCla com.squareup.okhttp3:logging-interceptor:3.12.12=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.squareup.okhttp3:okhttp:3.12.12=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.squareup.okio:okio:1.17.5=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.thoughtworks.qdox:qdox:1.12.1=codenarc,runtimeClasspath,testRuntimeClasspath +com.thoughtworks.qdox:qdox:1.12.1=codenarc com.vaadin.external.google:android-json:0.0.20131108.vaadin1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -commons-codec:commons-codec:1.15=spotbugs commons-fileupload:commons-fileupload:1.5=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath commons-io:commons-io:2.11.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -de.thetaphi:forbiddenapis:3.8=compileClasspath -info.picocli:picocli:4.6.3=runtimeClasspath,testRuntimeClasspath -io.leangen.geantyref:geantyref:1.3.16=runtimeClasspath,testRuntimeClasspath -io.sqreen:libsqreen:17.2.0=runtimeClasspath,testRuntimeClasspath +commons-io:commons-io:2.20.0=spotbugs +de.thetaphi:forbiddenapis:3.10=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.leangen.geantyref:geantyref:1.3.16=testRuntimeClasspath +io.sqreen:libsqreen:17.3.0=runtimeClasspath,testRuntimeClasspath javax.servlet:javax.servlet-api:3.1.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -jaxen:jaxen:1.2.0=spotbugs -jline:jline:2.14.6=runtimeClasspath,testRuntimeClasspath -junit:junit:4.13.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -net.bytebuddy:byte-buddy-agent:1.17.7=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -net.bytebuddy:byte-buddy:1.17.7=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +jaxen:jaxen:2.0.0=spotbugs +junit:junit:4.13.2=runtimeClasspath,testRuntimeClasspath +net.bytebuddy:byte-buddy-agent:1.18.8=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +net.bytebuddy:byte-buddy:1.18.8=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath net.java.dev.jna:jna-platform:5.8.0=runtimeClasspath,testRuntimeClasspath net.java.dev.jna:jna:5.8.0=runtimeClasspath,testRuntimeClasspath -net.jcip:jcip-annotations:1.0=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath net.minidev:accessors-smart:2.4.9=runtimeClasspath,testRuntimeClasspath net.minidev:json-smart:2.4.10=runtimeClasspath,testRuntimeClasspath -net.sf.saxon:Saxon-HE:11.4=spotbugs +net.sf.saxon:Saxon-HE:12.9=spotbugs org.apache.ant:ant-antlr:1.10.14=codenarc -org.apache.ant:ant-antlr:1.10.15=runtimeClasspath,testRuntimeClasspath org.apache.ant:ant-junit:1.10.14=codenarc -org.apache.ant:ant-junit:1.10.15=runtimeClasspath,testRuntimeClasspath -org.apache.ant:ant-launcher:1.10.15=runtimeClasspath,testRuntimeClasspath -org.apache.ant:ant:1.10.15=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.apache.bcel:bcel:6.5.0=spotbugs -org.apache.commons:commons-lang3:3.12.0=spotbugs -org.apache.commons:commons-text:1.10.0=spotbugs -org.apache.httpcomponents.client5:httpclient5:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5-h2:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5:5.1.3=spotbugs -org.apache.logging.log4j:log4j-api:2.19.0=spotbugs -org.apache.logging.log4j:log4j-core:2.19.0=spotbugs +org.apache.bcel:bcel:6.11.0=spotbugs +org.apache.commons:commons-lang3:3.19.0=spotbugs +org.apache.commons:commons-text:1.14.0=spotbugs +org.apache.logging.log4j:log4j-api:2.25.2=spotbugs +org.apache.logging.log4j:log4j-core:2.25.2=spotbugs org.apiguardian:apiguardian-api:1.1.2=compileClasspath,testCompileClasspath -org.codehaus.groovy:groovy-all:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-ant:3.0.23=codenarc -org.codehaus.groovy:groovy-ant:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-astbuilder:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-cli-picocli:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-console:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-datetime:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-docgenerator:3.0.23=codenarc -org.codehaus.groovy:groovy-docgenerator:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-groovydoc:3.0.23=codenarc -org.codehaus.groovy:groovy-groovydoc:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-groovysh:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jmx:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-json:3.0.23=codenarc -org.codehaus.groovy:groovy-json:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jsr223:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-macro:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-nio:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-servlet:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-sql:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-swing:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +org.codehaus.groovy:groovy-json:3.0.25=compileClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-templates:3.0.23=codenarc -org.codehaus.groovy:groovy-templates:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test-junit5:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-testng:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-xml:3.0.23=codenarc -org.codehaus.groovy:groovy-xml:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy:3.0.23=codenarc -org.codehaus.groovy:groovy:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codenarc:CodeNarc:3.6.0=codenarc -org.dom4j:dom4j:2.1.3=spotbugs -org.eclipse.jetty:jetty-http:9.4.56.v20240826=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.eclipse.jetty:jetty-io:9.4.56.v20240826=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.eclipse.jetty:jetty-server:9.4.56.v20240826=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.eclipse.jetty:jetty-util:9.4.56.v20240826=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +org.codehaus.groovy:groovy:3.0.25=compileClasspath,testCompileClasspath,testRuntimeClasspath +org.codenarc:CodeNarc:3.7.0=codenarc +org.dom4j:dom4j:2.2.0=spotbugs org.freemarker:freemarker:2.3.31=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.gmetrics:GMetrics:2.1.0=codenarc -org.hamcrest:hamcrest-core:1.3=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.hamcrest:hamcrest:3.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +org.hamcrest:hamcrest-core:1.3=runtimeClasspath,testRuntimeClasspath +org.hamcrest:hamcrest:3.0=compileClasspath,testCompileClasspath,testRuntimeClasspath org.jacoco:org.jacoco.agent:0.8.14=jacocoAgent,jacocoAnt org.jacoco:org.jacoco.ant:0.8.14=jacocoAnt org.jacoco:org.jacoco.core:0.8.14=jacocoAnt,runtimeClasspath,testRuntimeClasspath org.jacoco:org.jacoco.report:0.8.14=jacocoAnt,runtimeClasspath,testRuntimeClasspath -org.jctools:jctools-core:3.3.0=runtimeClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-api:5.12.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-engine:5.12.2=runtimeClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-params:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-commons:1.12.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-engine:1.12.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-launcher:1.12.2=runtimeClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-runner:1.12.2=runtimeClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-suite-api:1.12.2=runtimeClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-suite-commons:1.12.2=runtimeClasspath,testRuntimeClasspath -org.junit:junit-bom:5.12.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit:junit-bom:5.9.1=spotbugs +org.jctools:jctools-core-jdk11:4.0.6=runtimeClasspath,testRuntimeClasspath +org.jctools:jctools-core:4.0.6=runtimeClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-api:5.14.1=runtimeClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-engine:5.14.1=runtimeClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-params:5.14.1=runtimeClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter:5.14.1=runtimeClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-commons:1.14.1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-engine:1.14.1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-launcher:1.14.1=runtimeClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-runner:1.14.1=runtimeClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-suite-api:1.14.1=runtimeClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-suite-commons:1.14.1=runtimeClasspath,testRuntimeClasspath +org.junit:junit-bom:5.14.0=spotbugs +org.junit:junit-bom:5.14.1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.mockito:mockito-core:4.4.0=testRuntimeClasspath org.msgpack:jackson-dataformat-msgpack:0.9.6=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.msgpack:msgpack-core:0.9.6=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.objenesis:objenesis:3.3=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +org.objenesis:objenesis:3.3=compileClasspath,testCompileClasspath,testRuntimeClasspath org.opentest4j:opentest4j:1.3.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.4=spotbugs -org.ow2.asm:asm-commons:9.4=spotbugs -org.ow2.asm:asm-commons:9.9=jacocoAnt,runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-tree:9.4=spotbugs -org.ow2.asm:asm-tree:9.9=jacocoAnt,runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-util:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-util:9.4=spotbugs -org.ow2.asm:asm:9.4=spotbugs -org.ow2.asm:asm:9.9=jacocoAnt,runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-analysis:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-analysis:9.9=spotbugs +org.ow2.asm:asm-commons:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-commons:9.9.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-tree:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-tree:9.9.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-util:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-util:9.9=spotbugs +org.ow2.asm:asm:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm:9.9.1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.skyscreamer:jsonassert:1.5.1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.slf4j:jcl-over-slf4j:1.7.30=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.slf4j:jul-to-slf4j:1.7.30=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.slf4j:log4j-over-slf4j:1.7.30=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.slf4j:slf4j-api:1.7.32=compileClasspath,testCompileClasspath org.slf4j:slf4j-api:1.7.36=runtimeClasspath,testRuntimeClasspath -org.slf4j:slf4j-api:2.0.0=spotbugs,spotbugsSlf4j -org.slf4j:slf4j-simple:2.0.0=spotbugsSlf4j +org.slf4j:slf4j-api:2.0.17=spotbugs,spotbugsSlf4j +org.slf4j:slf4j-simple:2.0.17=spotbugsSlf4j org.snakeyaml:snakeyaml-engine:2.9=runtimeClasspath,testRuntimeClasspath -org.spockframework:spock-bom:2.4-M6-groovy-3.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.spockframework:spock-core:2.4-M6-groovy-3.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.testng:testng:7.5.1=runtimeClasspath,testRuntimeClasspath -org.webjars:jquery:3.5.1=runtimeClasspath,testRuntimeClasspath -org.xmlresolver:xmlresolver:4.4.3=spotbugs +org.spockframework:spock-bom:2.4-groovy-3.0=compileClasspath,testCompileClasspath,testRuntimeClasspath +org.spockframework:spock-core:2.4-groovy-3.0=compileClasspath,testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-junit:1.2.1=testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-parser:1.2.0=testCompileClasspath,testRuntimeClasspath +org.xmlresolver:xmlresolver:5.3.3=spotbugs org.xmlunit:xmlunit-core:2.10.3=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -xml-apis:xml-apis:1.4.01=spotbugs empty=annotationProcessor,spotbugsPlugins,testAnnotationProcessor diff --git a/dd-java-agent/agent-ci-visibility/civisibility-instrumentation-test-fixtures/src/main/groovy/datadog/trace/civisibility/CiVisibilityInstrumentationTest.groovy b/dd-java-agent/agent-ci-visibility/civisibility-instrumentation-test-fixtures/src/main/groovy/datadog/trace/civisibility/CiVisibilityInstrumentationTest.groovy index 4697abe5549..e71a8a5d8c9 100644 --- a/dd-java-agent/agent-ci-visibility/civisibility-instrumentation-test-fixtures/src/main/groovy/datadog/trace/civisibility/CiVisibilityInstrumentationTest.groovy +++ b/dd-java-agent/agent-ci-visibility/civisibility-instrumentation-test-fixtures/src/main/groovy/datadog/trace/civisibility/CiVisibilityInstrumentationTest.groovy @@ -128,9 +128,9 @@ abstract class CiVisibilityInstrumentationTest extends InstrumentationSpecificat def metricCollector = Stub(CiVisibilityMetricCollectorImpl) def sourcePathResolver = Stub(SourcePathResolver) - sourcePathResolver.getSourcePath(_ as Class) >> DUMMY_SOURCE_PATH - sourcePathResolver.getResourcePath(_ as String) >> { - String path -> path + sourcePathResolver.getSourcePaths(_ as Class) >> [DUMMY_SOURCE_PATH] + sourcePathResolver.getResourcePaths(_ as String) >> { + String path -> [path] } def codeowners = Stub(Codeowners) @@ -248,7 +248,8 @@ abstract class CiVisibilityInstrumentationTest extends InstrumentationSpecificat settings.quarantinedTests, settings.disabledTests, settings.attemptToFixTests, - settings.diff) + settings.diff, + ConfigurationErrors.NONE) } } diff --git a/dd-java-agent/agent-ci-visibility/civisibility-test-fixtures/build.gradle b/dd-java-agent/agent-ci-visibility/civisibility-test-fixtures/build.gradle index 9dae3654dd3..2e2d7a6c16a 100644 --- a/dd-java-agent/agent-ci-visibility/civisibility-test-fixtures/build.gradle +++ b/dd-java-agent/agent-ci-visibility/civisibility-test-fixtures/build.gradle @@ -11,5 +11,9 @@ dependencies { api libs.jackson.databind api group: 'org.msgpack', name: 'jackson-dataformat-msgpack', version: '0.9.6' api group: 'org.xmlunit', name: 'xmlunit-core', version: '2.10.3' + + compileOnly(libs.junit.jupiter) + compileOnly(libs.bundles.groovy) + compileOnly(libs.bundles.spock) } diff --git a/dd-java-agent/agent-ci-visibility/civisibility-test-fixtures/gradle.lockfile b/dd-java-agent/agent-ci-visibility/civisibility-test-fixtures/gradle.lockfile index 3e8bac579ae..7f232d926ab 100644 --- a/dd-java-agent/agent-ci-visibility/civisibility-test-fixtures/gradle.lockfile +++ b/dd-java-agent/agent-ci-visibility/civisibility-test-fixtures/gradle.lockfile @@ -5,32 +5,32 @@ cafe.cryptography:curve25519-elisabeth:0.1.0=runtimeClasspath,testRuntimeClasspa cafe.cryptography:ed25519-elisabeth:0.1.0=runtimeClasspath,testRuntimeClasspath ch.qos.logback:logback-classic:1.2.13=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath ch.qos.logback:logback-core:1.2.13=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.beust:jcommander:1.78=runtimeClasspath,testRuntimeClasspath com.blogspot.mydailyjava:weak-lock-free:0.17=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.datadoghq.okhttp3:okhttp:3.12.15=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.datadoghq.okio:okio:1.17.6=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.datadoghq:dd-instrument-java:0.0.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +com.datadoghq:dd-instrument-java:0.0.3=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.datadoghq:dd-javac-plugin-client:0.2.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.datadoghq:java-dogstatsd-client:4.4.3=runtimeClasspath,testRuntimeClasspath +com.datadoghq:java-dogstatsd-client:4.4.5=runtimeClasspath,testRuntimeClasspath com.datadoghq:sketches-java:0.8.3=runtimeClasspath,testRuntimeClasspath com.fasterxml.jackson.core:jackson-annotations:2.20=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.fasterxml.jackson.core:jackson-core:2.20.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.fasterxml.jackson.core:jackson-databind:2.20.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.fasterxml.jackson:jackson-bom:2.20.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.github.javaparser:javaparser-core:3.25.6=codenarc,compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.github.jnr:jffi:1.3.13=runtimeClasspath,testRuntimeClasspath +com.github.javaparser:javaparser-core:3.25.6=codenarc +com.github.jnr:jffi:1.3.14=runtimeClasspath,testRuntimeClasspath com.github.jnr:jnr-a64asm:1.0.0=runtimeClasspath,testRuntimeClasspath com.github.jnr:jnr-constants:0.10.4=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-enxio:0.32.17=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-ffi:2.2.16=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-posix:3.1.19=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-unixsocket:0.38.22=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-enxio:0.32.19=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-ffi:2.2.18=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-posix:3.1.21=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-unixsocket:0.38.24=runtimeClasspath,testRuntimeClasspath com.github.jnr:jnr-x86asm:1.0.2=runtimeClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.2.0=compileClasspath,testCompileClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.7.3=spotbugs -com.github.spotbugs:spotbugs:4.7.3=spotbugs +com.github.spotbugs:spotbugs-annotations:4.9.8=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath +com.github.spotbugs:spotbugs:4.9.8=spotbugs +com.github.stephenc.jcip:jcip-annotations:1.0-1=spotbugs com.google.code.findbugs:jsr305:3.0.2=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -com.google.code.gson:gson:2.9.1=spotbugs +com.google.code.gson:gson:2.13.2=spotbugs +com.google.errorprone:error_prone_annotations:2.41.0=spotbugs com.google.guava:guava:20.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.google.re2j:re2j:1.7=runtimeClasspath,testRuntimeClasspath com.jayway.jsonpath:json-path:2.8.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath @@ -38,128 +38,93 @@ com.squareup.moshi:moshi:1.11.0=compileClasspath,runtimeClasspath,testCompileCla com.squareup.okhttp3:logging-interceptor:3.12.12=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.squareup.okhttp3:okhttp:3.12.12=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.squareup.okio:okio:1.17.5=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.thoughtworks.qdox:qdox:1.12.1=codenarc,runtimeClasspath,testRuntimeClasspath +com.thoughtworks.qdox:qdox:1.12.1=codenarc com.vaadin.external.google:android-json:0.0.20131108.vaadin1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -commons-codec:commons-codec:1.15=spotbugs commons-fileupload:commons-fileupload:1.5=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath commons-io:commons-io:2.11.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -de.thetaphi:forbiddenapis:3.8=compileClasspath -info.picocli:picocli:4.6.3=runtimeClasspath,testRuntimeClasspath -io.leangen.geantyref:geantyref:1.3.16=runtimeClasspath,testRuntimeClasspath -io.sqreen:libsqreen:17.2.0=runtimeClasspath,testRuntimeClasspath +commons-io:commons-io:2.20.0=spotbugs +de.thetaphi:forbiddenapis:3.10=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +io.leangen.geantyref:geantyref:1.3.16=testRuntimeClasspath +io.sqreen:libsqreen:17.3.0=runtimeClasspath,testRuntimeClasspath javax.servlet:javax.servlet-api:3.1.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -jaxen:jaxen:1.2.0=spotbugs -jline:jline:2.14.6=runtimeClasspath,testRuntimeClasspath -junit:junit:4.13.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -net.bytebuddy:byte-buddy-agent:1.17.7=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -net.bytebuddy:byte-buddy:1.17.7=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +jaxen:jaxen:2.0.0=spotbugs +junit:junit:4.13.2=runtimeClasspath,testRuntimeClasspath +net.bytebuddy:byte-buddy-agent:1.18.8=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +net.bytebuddy:byte-buddy:1.18.8=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath net.java.dev.jna:jna-platform:5.8.0=runtimeClasspath,testRuntimeClasspath net.java.dev.jna:jna:5.8.0=runtimeClasspath,testRuntimeClasspath -net.jcip:jcip-annotations:1.0=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath net.minidev:accessors-smart:2.4.9=runtimeClasspath,testRuntimeClasspath net.minidev:json-smart:2.4.10=runtimeClasspath,testRuntimeClasspath -net.sf.saxon:Saxon-HE:11.4=spotbugs +net.sf.saxon:Saxon-HE:12.9=spotbugs org.apache.ant:ant-antlr:1.10.14=codenarc -org.apache.ant:ant-antlr:1.10.15=runtimeClasspath,testRuntimeClasspath org.apache.ant:ant-junit:1.10.14=codenarc -org.apache.ant:ant-junit:1.10.15=runtimeClasspath,testRuntimeClasspath -org.apache.ant:ant-launcher:1.10.15=runtimeClasspath,testRuntimeClasspath -org.apache.ant:ant:1.10.15=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.apache.bcel:bcel:6.5.0=spotbugs -org.apache.commons:commons-lang3:3.12.0=spotbugs -org.apache.commons:commons-text:1.10.0=spotbugs -org.apache.httpcomponents.client5:httpclient5:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5-h2:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5:5.1.3=spotbugs -org.apache.logging.log4j:log4j-api:2.19.0=spotbugs -org.apache.logging.log4j:log4j-core:2.19.0=spotbugs +org.apache.bcel:bcel:6.11.0=spotbugs +org.apache.commons:commons-lang3:3.19.0=spotbugs +org.apache.commons:commons-text:1.14.0=spotbugs +org.apache.logging.log4j:log4j-api:2.25.2=spotbugs +org.apache.logging.log4j:log4j-core:2.25.2=spotbugs org.apiguardian:apiguardian-api:1.1.2=compileClasspath,testCompileClasspath -org.codehaus.groovy:groovy-all:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-ant:3.0.23=codenarc -org.codehaus.groovy:groovy-ant:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-astbuilder:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-cli-picocli:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-console:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-datetime:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-docgenerator:3.0.23=codenarc -org.codehaus.groovy:groovy-docgenerator:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-groovydoc:3.0.23=codenarc -org.codehaus.groovy:groovy-groovydoc:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-groovysh:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jmx:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-json:3.0.23=codenarc -org.codehaus.groovy:groovy-json:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jsr223:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-macro:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-nio:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-servlet:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-sql:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-swing:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +org.codehaus.groovy:groovy-json:3.0.25=compileClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-templates:3.0.23=codenarc -org.codehaus.groovy:groovy-templates:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test-junit5:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-testng:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-xml:3.0.23=codenarc -org.codehaus.groovy:groovy-xml:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy:3.0.23=codenarc -org.codehaus.groovy:groovy:3.0.24=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.codenarc:CodeNarc:3.6.0=codenarc -org.dom4j:dom4j:2.1.3=spotbugs -org.eclipse.jetty:jetty-http:9.4.56.v20240826=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.eclipse.jetty:jetty-io:9.4.56.v20240826=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.eclipse.jetty:jetty-server:9.4.56.v20240826=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.eclipse.jetty:jetty-util:9.4.56.v20240826=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +org.codehaus.groovy:groovy:3.0.25=compileClasspath,testCompileClasspath,testRuntimeClasspath +org.codenarc:CodeNarc:3.7.0=codenarc +org.dom4j:dom4j:2.2.0=spotbugs org.freemarker:freemarker:2.3.31=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.gmetrics:GMetrics:2.1.0=codenarc -org.hamcrest:hamcrest-core:1.3=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.hamcrest:hamcrest:3.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +org.hamcrest:hamcrest-core:1.3=runtimeClasspath,testRuntimeClasspath +org.hamcrest:hamcrest:3.0=compileClasspath,testCompileClasspath,testRuntimeClasspath org.jacoco:org.jacoco.agent:0.8.14=jacocoAgent,jacocoAnt org.jacoco:org.jacoco.ant:0.8.14=jacocoAnt org.jacoco:org.jacoco.core:0.8.14=jacocoAnt,runtimeClasspath,testRuntimeClasspath org.jacoco:org.jacoco.report:0.8.14=jacocoAnt,runtimeClasspath,testRuntimeClasspath -org.jctools:jctools-core:3.3.0=runtimeClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-api:5.12.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-engine:5.12.2=runtimeClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-params:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-commons:1.12.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-engine:1.12.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-launcher:1.12.2=runtimeClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-runner:1.12.2=runtimeClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-suite-api:1.12.2=runtimeClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-suite-commons:1.12.2=runtimeClasspath,testRuntimeClasspath -org.junit:junit-bom:5.12.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.junit:junit-bom:5.9.1=spotbugs +org.jctools:jctools-core-jdk11:4.0.6=runtimeClasspath,testRuntimeClasspath +org.jctools:jctools-core:4.0.6=runtimeClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-api:5.14.1=compileClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-engine:5.14.1=testRuntimeClasspath +org.junit.jupiter:junit-jupiter-params:5.14.1=compileClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter:5.14.1=compileClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-commons:1.14.1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-engine:1.14.1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-launcher:1.14.1=runtimeClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-runner:1.14.1=runtimeClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-suite-api:1.14.1=runtimeClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-suite-commons:1.14.1=runtimeClasspath,testRuntimeClasspath +org.junit:junit-bom:5.14.0=spotbugs +org.junit:junit-bom:5.14.1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.mockito:mockito-core:4.4.0=testRuntimeClasspath org.msgpack:jackson-dataformat-msgpack:0.9.6=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.msgpack:msgpack-core:0.9.6=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.objenesis:objenesis:3.3=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +org.objenesis:objenesis:3.3=compileClasspath,testCompileClasspath,testRuntimeClasspath org.opentest4j:opentest4j:1.3.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.4=spotbugs -org.ow2.asm:asm-commons:9.4=spotbugs -org.ow2.asm:asm-commons:9.9=jacocoAnt,runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-tree:9.4=spotbugs -org.ow2.asm:asm-tree:9.9=jacocoAnt,runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-util:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-util:9.4=spotbugs -org.ow2.asm:asm:9.4=spotbugs -org.ow2.asm:asm:9.9=jacocoAnt,runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-analysis:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-analysis:9.9=spotbugs +org.ow2.asm:asm-commons:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-commons:9.9.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-tree:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-tree:9.9.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-util:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-util:9.9=spotbugs +org.ow2.asm:asm:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm:9.9.1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.skyscreamer:jsonassert:1.5.1=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.slf4j:jcl-over-slf4j:1.7.30=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.slf4j:jul-to-slf4j:1.7.30=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.slf4j:log4j-over-slf4j:1.7.30=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath org.slf4j:slf4j-api:1.7.32=compileClasspath,testCompileClasspath org.slf4j:slf4j-api:1.7.36=runtimeClasspath,testRuntimeClasspath -org.slf4j:slf4j-api:2.0.0=spotbugs,spotbugsSlf4j -org.slf4j:slf4j-simple:2.0.0=spotbugsSlf4j +org.slf4j:slf4j-api:2.0.17=spotbugs,spotbugsSlf4j +org.slf4j:slf4j-simple:2.0.17=spotbugsSlf4j org.snakeyaml:snakeyaml-engine:2.9=runtimeClasspath,testRuntimeClasspath -org.spockframework:spock-bom:2.4-M6-groovy-3.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.spockframework:spock-core:2.4-M6-groovy-3.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -org.testng:testng:7.5.1=runtimeClasspath,testRuntimeClasspath -org.webjars:jquery:3.5.1=runtimeClasspath,testRuntimeClasspath -org.xmlresolver:xmlresolver:4.4.3=spotbugs +org.spockframework:spock-bom:2.4-groovy-3.0=compileClasspath,testCompileClasspath,testRuntimeClasspath +org.spockframework:spock-core:2.4-groovy-3.0=compileClasspath,testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-junit:1.2.1=testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-parser:1.2.0=testCompileClasspath,testRuntimeClasspath +org.xmlresolver:xmlresolver:5.3.3=spotbugs org.xmlunit:xmlunit-core:2.10.3=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -xml-apis:xml-apis:1.4.01=spotbugs empty=annotationProcessor,spotbugsPlugins,testAnnotationProcessor diff --git a/dd-java-agent/agent-ci-visibility/civisibility-test-fixtures/src/main/groovy/datadog/trace/civisibility/CiVisibilitySmokeTest.groovy b/dd-java-agent/agent-ci-visibility/civisibility-test-fixtures/src/main/groovy/datadog/trace/civisibility/CiVisibilitySmokeTest.groovy index 0957ec215a8..0eaeb2e755a 100644 --- a/dd-java-agent/agent-ci-visibility/civisibility-test-fixtures/src/main/groovy/datadog/trace/civisibility/CiVisibilitySmokeTest.groovy +++ b/dd-java-agent/agent-ci-visibility/civisibility-test-fixtures/src/main/groovy/datadog/trace/civisibility/CiVisibilitySmokeTest.groovy @@ -4,13 +4,18 @@ import datadog.trace.api.Config import datadog.trace.api.civisibility.config.TestFQN import datadog.trace.api.config.CiVisibilityConfig import datadog.trace.api.config.GeneralConfig +import datadog.trace.api.config.TraceInstrumentationConfig +import datadog.trace.api.config.TracerConfig +import java.nio.file.Paths import spock.lang.Specification -import spock.util.environment.Jvm +import spock.lang.TempDir + +import java.nio.file.Path import static datadog.trace.util.ConfigStrings.propertyNameToSystemPropertyName abstract class CiVisibilitySmokeTest extends Specification { - static final List SMOKE_IGNORED_TAGS = ["content.meta.['_dd.integration']"] + static final List SMOKE_IGNORED_TAGS = ["content.meta.['_dd.integration']", "content.meta.['_dd.svc_src']"] protected static final String AGENT_JAR = System.getProperty("datadog.smoketest.agent.shadowJar.path") protected static final String TEST_ENVIRONMENT_NAME = "integration-test" @@ -19,11 +24,24 @@ abstract class CiVisibilitySmokeTest extends Specification { private static final Map DEFAULT_TRACER_CONFIG = defaultJvmArguments() + @TempDir + protected Path prefsDir + protected static String buildJavaHome() { - if (Jvm.current.isJava8()) { - return System.getenv("JAVA_8_HOME") + def javaHome = System.getProperty("java.home") + def javacPath = Paths.get(javaHome, "bin", "javac").toFile() + if (javacPath.exists()) { + return javaHome + } + // In CI for JDK 8, java.home may point to the JRE directory (e.g., /usr/lib/jvm/8/jre) + // The JDK with javac is in the parent directory + def parentDir = new File(javaHome).getParentFile() + def parentJavacPath = new File(parentDir, Paths.get("bin", "javac").toString()) + if (parentJavacPath.exists()) { + return parentDir.getAbsolutePath() } - return System.getenv("JAVA_" + Jvm.current.getJavaSpecificationVersion() + "_HOME") + // Fallback to java.home and let callers handle the error if javac is not found + return javaHome } protected static String javaPath() { @@ -44,8 +62,10 @@ abstract class CiVisibilitySmokeTest extends Specification { argMap.put(CiVisibilityConfig.CIVISIBILITY_AGENTLESS_ENABLED, "true") argMap.put(CiVisibilityConfig.CIVISIBILITY_CIPROVIDER_INTEGRATION_ENABLED, "false") argMap.put(CiVisibilityConfig.CIVISIBILITY_GIT_UPLOAD_ENABLED, "false") + argMap.put(CiVisibilityConfig.CIVISIBILITY_GIT_CLIENT_ENABLED, "false") argMap.put(CiVisibilityConfig.CIVISIBILITY_FLAKY_RETRY_ONLY_KNOWN_FLAKES, "true") argMap.put(CiVisibilityConfig.CIVISIBILITY_COMPILER_PLUGIN_VERSION, JAVAC_PLUGIN_VERSION) + argMap.put(TraceInstrumentationConfig.CODE_ORIGIN_FOR_SPANS_ENABLED, "false") return argMap } @@ -53,6 +73,7 @@ abstract class CiVisibilitySmokeTest extends Specification { Map argMap = new HashMap<>(DEFAULT_TRACER_CONFIG) argMap.put(CiVisibilityConfig.CIVISIBILITY_AGENTLESS_URL, mockBackendIntakeUrl) argMap.put(CiVisibilityConfig.CIVISIBILITY_INTAKE_AGENTLESS_URL, mockBackendIntakeUrl) + argMap.put(TracerConfig.TRACE_AGENT_URL, mockBackendIntakeUrl) argMap.putAll(additionalArgs) if (serviceName != null) { @@ -63,7 +84,10 @@ abstract class CiVisibilitySmokeTest extends Specification { } protected List buildJvmArguments(String mockBackendIntakeUrl, String serviceName, Map additionalArgs) { - List arguments = [] + List arguments = ["-Xms256m", "-Xmx256m"] + + arguments += preventJulPrefsFileLock() + Map argMap = buildJvmArgMap(mockBackendIntakeUrl, serviceName, additionalArgs) // for convenience when debugging locally @@ -80,6 +104,28 @@ abstract class CiVisibilitySmokeTest extends Specification { return arguments } + /** + * Trick to prevent jul Prefs file lock issue on forked processes, in particular in CI which + * runs on Linux and have competing processes trying to write to it, including the Gradle daemon. + * + * 

+   * Couldn't flush user prefs: java.util.prefs.BackingStoreException: Couldn't get file lock.
+   *
+ * + * Note, some tests can setup arguments on spec level, so `prefsDir` will be `null` during + * `setupSpec()`. + */ + protected String preventJulPrefsFileLock() { + String prefsPath = (prefsDir ?: tempUserPrefsPath()).toAbsolutePath() + return "-Djava.util.prefs.userRoot=$prefsPath".toString() + } + + private static Path tempUserPrefsPath() { + String uniqueId = "${System.currentTimeMillis()}_${System.nanoTime()}_${Thread.currentThread().id}" + Path prefsPath = Paths.get(System.getProperty("java.io.tmpdir"), "gradle-test-userPrefs", uniqueId) + return prefsPath + } + protected verifyEventsAndCoverages(String projectName, String toolchain, String toolchainVersion, List> events, List> coverages, List additionalDynamicTags = []) { def additionalReplacements = ["content.meta.['test.toolchain']": "$toolchain:$toolchainVersion"] diff --git a/dd-java-agent/agent-ci-visibility/civisibility-test-fixtures/src/main/groovy/datadog/trace/civisibility/CiVisibilityTestUtils.groovy b/dd-java-agent/agent-ci-visibility/civisibility-test-fixtures/src/main/groovy/datadog/trace/civisibility/CiVisibilityTestUtils.groovy index a77570a2721..c12ae929675 100644 --- a/dd-java-agent/agent-ci-visibility/civisibility-test-fixtures/src/main/groovy/datadog/trace/civisibility/CiVisibilityTestUtils.groovy +++ b/dd-java-agent/agent-ci-visibility/civisibility-test-fixtures/src/main/groovy/datadog/trace/civisibility/CiVisibilityTestUtils.groovy @@ -65,7 +65,7 @@ abstract class CiVisibilityTestUtils { // ignored tags on assertion and fixture build static final List IGNORED_TAGS = LibraryCapability.values().toList().stream().map(c -> "content.meta.['${c.asTag()}']").collect(Collectors.toList()) + - ["content.meta.['_dd.integration']"] + ["content.meta.['_dd.integration']", "content.meta.['_dd.svc_src']"] static final List COVERAGE_DYNAMIC_PATHS = [path("test_session_id"), path("test_suite_id"), path("span_id"),] @@ -188,7 +188,9 @@ abstract class CiVisibilityTestUtils { } static List getTestIdentifiers(List> events) { - events.sort(Comparator.comparing { it['content']['start'] as Long }) + events.sort(Comparator.comparing { + it['content']['start'] as Long + }) def testIdentifiers = [] for (Map event : events) { if (event['content']['meta']['test.name']) { @@ -275,7 +277,6 @@ abstract class CiVisibilityTestUtils { StringWriter coveragesOut = new StringWriter() coveragesTemplate.process(replacements, coveragesOut) return coveragesOut.toString() - } catch (Exception e) { throw new RuntimeException("Could not get Freemarker template " + templatePath + "; replacements map: " + replacements + "; replacements source: " + replacementsSource, e) } @@ -304,7 +305,6 @@ abstract class CiVisibilityTestUtils { return label.forTemplateKey(dynamicPath.rawPath) }) } - } return JSON_MAPPER .writeValueAsString(objects) diff --git a/dd-java-agent/agent-ci-visibility/gradle.lockfile b/dd-java-agent/agent-ci-visibility/gradle.lockfile index 511b239db13..4cc47dbe841 100644 --- a/dd-java-agent/agent-ci-visibility/gradle.lockfile +++ b/dd-java-agent/agent-ci-visibility/gradle.lockfile @@ -5,14 +5,13 @@ cafe.cryptography:curve25519-elisabeth:0.1.0=runtimeClasspath,testRuntimeClasspa cafe.cryptography:ed25519-elisabeth:0.1.0=runtimeClasspath,testRuntimeClasspath ch.qos.logback:logback-classic:1.2.13=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath ch.qos.logback:logback-core:1.2.13=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -com.beust:jcommander:1.78=testRuntimeClasspath com.blogspot.mydailyjava:weak-lock-free:0.17=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath com.datadoghq.okhttp3:okhttp:3.12.15=compileClasspath,implementationDependenciesMetadata,runtimeClasspath,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath com.datadoghq.okio:okio:1.17.6=compileClasspath,implementationDependenciesMetadata,runtimeClasspath,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -com.datadoghq:dd-instrument-java:0.0.2=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +com.datadoghq:dd-instrument-java:0.0.3=compileClasspath,implementationDependenciesMetadata,runtimeClasspath,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath com.datadoghq:dd-javac-plugin-client:0.2.2=compileClasspath,implementationDependenciesMetadata,runtimeClasspath,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -com.datadoghq:java-dogstatsd-client:4.4.3=runtimeClasspath,testRuntimeClasspath -com.datadoghq:sketches-java:0.8.3=testRuntimeClasspath +com.datadoghq:java-dogstatsd-client:4.4.5=runtimeClasspath,testRuntimeClasspath +com.datadoghq:sketches-java:0.8.3=runtimeClasspath,testRuntimeClasspath com.eed3si9n:shaded-jawn-parser_2.13:1.3.2=zinc com.eed3si9n:shaded-scalajson_2.13:1.0.0-M4=zinc com.eed3si9n:sjson-new-core_2.13:0.10.1=zinc @@ -21,20 +20,21 @@ com.fasterxml.jackson.core:jackson-annotations:2.15.2=testCompileClasspath,testI com.fasterxml.jackson.core:jackson-core:2.15.2=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath com.fasterxml.jackson.core:jackson-databind:2.15.2=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath com.fasterxml.jackson:jackson-bom:2.15.2=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -com.github.javaparser:javaparser-core:3.25.6=codenarc,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -com.github.jnr:jffi:1.3.13=runtimeClasspath,testRuntimeClasspath +com.github.javaparser:javaparser-core:3.25.6=codenarc +com.github.jnr:jffi:1.3.14=runtimeClasspath,testRuntimeClasspath com.github.jnr:jnr-a64asm:1.0.0=runtimeClasspath,testRuntimeClasspath com.github.jnr:jnr-constants:0.10.4=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-enxio:0.32.17=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-ffi:2.2.16=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-posix:3.1.19=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-unixsocket:0.38.22=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-enxio:0.32.19=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-ffi:2.2.18=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-posix:3.1.21=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-unixsocket:0.38.24=runtimeClasspath,testRuntimeClasspath com.github.jnr:jnr-x86asm:1.0.2=runtimeClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.2.0=compileClasspath,compileOnlyDependenciesMetadata,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.7.3=spotbugs -com.github.spotbugs:spotbugs:4.7.3=spotbugs +com.github.spotbugs:spotbugs-annotations:4.9.8=compileClasspath,compileOnlyDependenciesMetadata,spotbugs,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +com.github.spotbugs:spotbugs:4.9.8=spotbugs +com.github.stephenc.jcip:jcip-annotations:1.0-1=spotbugs com.google.code.findbugs:jsr305:3.0.2=compileClasspath,compileOnlyDependenciesMetadata,spotbugs,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -com.google.code.gson:gson:2.9.1=spotbugs +com.google.code.gson:gson:2.13.2=spotbugs +com.google.errorprone:error_prone_annotations:2.41.0=spotbugs com.google.guava:guava:18.0=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath com.google.jimfs:jimfs:1.1=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath com.google.re2j:re2j:1.7=testRuntimeClasspath @@ -44,142 +44,110 @@ com.squareup.okhttp3:logging-interceptor:3.12.12=testCompileClasspath,testImplem com.squareup.okhttp3:okhttp:3.12.12=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath com.squareup.okio:okio:1.17.5=compileClasspath,implementationDependenciesMetadata,runtimeClasspath,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath com.swoval:file-tree-views:2.1.12=zinc -com.thoughtworks.qdox:qdox:1.12.1=codenarc,testRuntimeClasspath +com.thoughtworks.qdox:qdox:1.12.1=codenarc com.vaadin.external.google:android-json:0.0.20131108.vaadin1=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -commons-codec:commons-codec:1.15=spotbugs commons-fileupload:commons-fileupload:1.5=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath commons-io:commons-io:2.11.0=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -de.thetaphi:forbiddenapis:3.8=compileClasspath,compileOnlyDependenciesMetadata -info.picocli:picocli:4.6.3=testRuntimeClasspath +commons-io:commons-io:2.20.0=spotbugs +de.thetaphi:forbiddenapis:3.10=compileClasspath,compileOnlyDependenciesMetadata,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath io.github.java-diff-utils:java-diff-utils:4.12=zinc io.leangen.geantyref:geantyref:1.3.16=testRuntimeClasspath -io.sqreen:libsqreen:17.2.0=testRuntimeClasspath +io.sqreen:libsqreen:17.3.0=testRuntimeClasspath javax.servlet:javax.servlet-api:3.1.0=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -jaxen:jaxen:1.2.0=spotbugs -jline:jline:2.14.6=testRuntimeClasspath -junit:junit:4.13.2=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -net.bytebuddy:byte-buddy-agent:1.17.7=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -net.bytebuddy:byte-buddy:1.17.7=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +jaxen:jaxen:2.0.0=spotbugs +junit:junit:4.13.2=testRuntimeClasspath +net.bytebuddy:byte-buddy-agent:1.18.8=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +net.bytebuddy:byte-buddy:1.18.8=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath net.java.dev.jna:jna-platform:5.8.0=testRuntimeClasspath net.java.dev.jna:jna:5.14.0=zinc net.java.dev.jna:jna:5.8.0=testRuntimeClasspath -net.jcip:jcip-annotations:1.0=compileClasspath,compileOnlyDependenciesMetadata,spotbugs,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath net.openhft:zero-allocation-hashing:0.16=zinc -net.sf.saxon:Saxon-HE:11.4=spotbugs +net.sf.saxon:Saxon-HE:12.9=spotbugs org.apache.ant:ant-antlr:1.10.14=codenarc -org.apache.ant:ant-antlr:1.10.15=testRuntimeClasspath org.apache.ant:ant-junit:1.10.14=codenarc -org.apache.ant:ant-junit:1.10.15=testRuntimeClasspath -org.apache.ant:ant-launcher:1.10.15=testRuntimeClasspath -org.apache.ant:ant:1.10.15=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.apache.bcel:bcel:6.5.0=spotbugs -org.apache.commons:commons-lang3:3.12.0=spotbugs -org.apache.commons:commons-text:1.10.0=spotbugs -org.apache.httpcomponents.client5:httpclient5:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5-h2:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5:5.1.3=spotbugs +org.apache.bcel:bcel:6.11.0=spotbugs +org.apache.commons:commons-lang3:3.19.0=spotbugs +org.apache.commons:commons-text:1.14.0=spotbugs org.apache.logging.log4j:log4j-api:2.17.1=zinc -org.apache.logging.log4j:log4j-api:2.19.0=spotbugs +org.apache.logging.log4j:log4j-api:2.25.2=spotbugs org.apache.logging.log4j:log4j-core:2.17.1=zinc -org.apache.logging.log4j:log4j-core:2.19.0=spotbugs +org.apache.logging.log4j:log4j-core:2.25.2=spotbugs org.apiguardian:apiguardian-api:1.1.2=testCompileClasspath,testImplementationDependenciesMetadata -org.codehaus.groovy:groovy-all:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath org.codehaus.groovy:groovy-ant:3.0.23=codenarc -org.codehaus.groovy:groovy-ant:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.codehaus.groovy:groovy-astbuilder:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.codehaus.groovy:groovy-cli-picocli:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.codehaus.groovy:groovy-console:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.codehaus.groovy:groovy-datetime:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath org.codehaus.groovy:groovy-docgenerator:3.0.23=codenarc -org.codehaus.groovy:groovy-docgenerator:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath org.codehaus.groovy:groovy-groovydoc:3.0.23=codenarc -org.codehaus.groovy:groovy-groovydoc:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.codehaus.groovy:groovy-groovysh:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.codehaus.groovy:groovy-jmx:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath org.codehaus.groovy:groovy-json:3.0.23=codenarc -org.codehaus.groovy:groovy-json:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.codehaus.groovy:groovy-jsr223:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.codehaus.groovy:groovy-macro:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.codehaus.groovy:groovy-nio:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.codehaus.groovy:groovy-servlet:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.codehaus.groovy:groovy-sql:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.codehaus.groovy:groovy-swing:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +org.codehaus.groovy:groovy-json:3.0.25=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath org.codehaus.groovy:groovy-templates:3.0.23=codenarc -org.codehaus.groovy:groovy-templates:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.codehaus.groovy:groovy-test-junit5:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.codehaus.groovy:groovy-test:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.codehaus.groovy:groovy-testng:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath org.codehaus.groovy:groovy-xml:3.0.23=codenarc -org.codehaus.groovy:groovy-xml:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath org.codehaus.groovy:groovy:3.0.23=codenarc -org.codehaus.groovy:groovy:3.0.24=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.codenarc:CodeNarc:3.6.0=codenarc -org.dom4j:dom4j:2.1.3=spotbugs -org.eclipse.jetty:jetty-http:9.4.56.v20240826=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.eclipse.jetty:jetty-io:9.4.56.v20240826=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.eclipse.jetty:jetty-server:9.4.56.v20240826=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.eclipse.jetty:jetty-util:9.4.56.v20240826=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +org.codehaus.groovy:groovy:3.0.25=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +org.codenarc:CodeNarc:3.7.0=codenarc +org.dom4j:dom4j:2.2.0=spotbugs org.freemarker:freemarker:2.3.31=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath org.fusesource.jansi:jansi:2.4.0=zinc org.gmetrics:GMetrics:2.1.0=codenarc -org.hamcrest:hamcrest-core:1.3=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +org.hamcrest:hamcrest-core:1.3=testRuntimeClasspath org.hamcrest:hamcrest:3.0=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath org.jacoco:org.jacoco.agent:0.8.14=jacocoAgent,jacocoAnt org.jacoco:org.jacoco.ant:0.8.14=jacocoAnt org.jacoco:org.jacoco.core:0.8.14=compileClasspath,implementationDependenciesMetadata,jacocoAnt,runtimeClasspath,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath org.jacoco:org.jacoco.report:0.8.14=compileClasspath,implementationDependenciesMetadata,jacocoAnt,runtimeClasspath,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.jctools:jctools-core:3.3.0=testRuntimeClasspath +org.jctools:jctools-core-jdk11:4.0.6=testRuntimeClasspath +org.jctools:jctools-core:4.0.6=testRuntimeClasspath org.jetbrains.intellij.deps:trove4j:1.0.20200330=kotlinBuildToolsApiClasspath,kotlinCompilerClasspath,kotlinKlibCommonizerClasspath -org.jetbrains.kotlin:kotlin-build-common:1.9.24=kotlinBuildToolsApiClasspath -org.jetbrains.kotlin:kotlin-build-tools-api:1.9.24=kotlinBuildToolsApiClasspath -org.jetbrains.kotlin:kotlin-build-tools-impl:1.9.24=kotlinBuildToolsApiClasspath -org.jetbrains.kotlin:kotlin-compiler-embeddable:1.9.24=kotlinBuildToolsApiClasspath,kotlinCompilerClasspath,kotlinKlibCommonizerClasspath -org.jetbrains.kotlin:kotlin-compiler-runner:1.9.24=kotlinBuildToolsApiClasspath -org.jetbrains.kotlin:kotlin-daemon-client:1.9.24=kotlinBuildToolsApiClasspath -org.jetbrains.kotlin:kotlin-daemon-embeddable:1.9.24=kotlinBuildToolsApiClasspath,kotlinCompilerClasspath,kotlinKlibCommonizerClasspath -org.jetbrains.kotlin:kotlin-klib-commonizer-embeddable:1.9.24=kotlinKlibCommonizerClasspath +org.jetbrains.kotlin:kotlin-build-common:2.0.21=kotlinBuildToolsApiClasspath +org.jetbrains.kotlin:kotlin-build-tools-api:2.0.21=kotlinBuildToolsApiClasspath +org.jetbrains.kotlin:kotlin-build-tools-impl:2.0.21=kotlinBuildToolsApiClasspath +org.jetbrains.kotlin:kotlin-compiler-embeddable:2.0.21=kotlinBuildToolsApiClasspath,kotlinCompilerClasspath,kotlinKlibCommonizerClasspath +org.jetbrains.kotlin:kotlin-compiler-runner:2.0.21=kotlinBuildToolsApiClasspath +org.jetbrains.kotlin:kotlin-daemon-client:2.0.21=kotlinBuildToolsApiClasspath +org.jetbrains.kotlin:kotlin-daemon-embeddable:2.0.21=kotlinBuildToolsApiClasspath,kotlinCompilerClasspath,kotlinKlibCommonizerClasspath +org.jetbrains.kotlin:kotlin-klib-commonizer-embeddable:2.0.21=kotlinKlibCommonizerClasspath +org.jetbrains.kotlin:kotlin-native-prebuilt:2.0.21=kotlinNativeBundleConfiguration org.jetbrains.kotlin:kotlin-reflect:1.6.10=kotlinBuildToolsApiClasspath,kotlinCompilerClasspath,kotlinKlibCommonizerClasspath -org.jetbrains.kotlin:kotlin-script-runtime:1.9.24=kotlinBuildToolsApiClasspath,kotlinCompilerClasspath,kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest,kotlinKlibCommonizerClasspath -org.jetbrains.kotlin:kotlin-scripting-common:1.9.24=kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest -org.jetbrains.kotlin:kotlin-scripting-compiler-embeddable:1.9.24=kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest -org.jetbrains.kotlin:kotlin-scripting-compiler-impl-embeddable:1.9.24=kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest -org.jetbrains.kotlin:kotlin-scripting-jvm:1.9.24=kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest +org.jetbrains.kotlin:kotlin-script-runtime:2.0.21=kotlinBuildToolsApiClasspath,kotlinCompilerClasspath,kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest,kotlinKlibCommonizerClasspath +org.jetbrains.kotlin:kotlin-scripting-common:2.0.21=kotlinBuildToolsApiClasspath,kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest +org.jetbrains.kotlin:kotlin-scripting-compiler-embeddable:2.0.21=kotlinBuildToolsApiClasspath,kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest +org.jetbrains.kotlin:kotlin-scripting-compiler-impl-embeddable:2.0.21=kotlinBuildToolsApiClasspath,kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest +org.jetbrains.kotlin:kotlin-scripting-jvm:2.0.21=kotlinBuildToolsApiClasspath,kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest org.jetbrains.kotlin:kotlin-stdlib-common:1.6.21=compileClasspath,compileOnlyDependenciesMetadata,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath org.jetbrains.kotlin:kotlin-stdlib:1.6.21=compileClasspath,compileOnlyDependenciesMetadata,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.jetbrains.kotlin:kotlin-stdlib:1.9.24=kotlinBuildToolsApiClasspath,kotlinCompilerClasspath,kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest,kotlinKlibCommonizerClasspath -org.jetbrains.kotlinx:kotlinx-coroutines-core-jvm:1.5.0=kotlinBuildToolsApiClasspath +org.jetbrains.kotlin:kotlin-stdlib:2.0.21=kotlinBuildToolsApiClasspath,kotlinCompilerClasspath,kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest,kotlinKlibCommonizerClasspath +org.jetbrains.kotlinx:kotlinx-coroutines-core-jvm:1.6.4=kotlinBuildToolsApiClasspath,kotlinCompilerClasspath,kotlinKlibCommonizerClasspath org.jetbrains:annotations:13.0=compileClasspath,compileOnlyDependenciesMetadata,kotlinBuildToolsApiClasspath,kotlinCompilerClasspath,kotlinCompilerPluginClasspathMain,kotlinCompilerPluginClasspathTest,kotlinKlibCommonizerClasspath,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath org.jline:jline-native:3.27.1=zinc org.jline:jline-terminal-jni:3.27.1=zinc org.jline:jline-terminal:3.27.1=zinc org.jline:jline:3.26.3=zinc -org.junit.jupiter:junit-jupiter-api:5.12.2=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-engine:5.12.2=testRuntimeClasspath -org.junit.jupiter:junit-jupiter-params:5.12.2=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.junit.jupiter:junit-jupiter:5.12.2=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.junit.platform:junit-platform-commons:1.12.2=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.junit.platform:junit-platform-engine:1.12.2=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.junit.platform:junit-platform-launcher:1.12.2=testRuntimeClasspath -org.junit.platform:junit-platform-runner:1.12.2=testRuntimeClasspath -org.junit.platform:junit-platform-suite-api:1.12.2=testRuntimeClasspath -org.junit.platform:junit-platform-suite-commons:1.12.2=testRuntimeClasspath -org.junit:junit-bom:5.12.2=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.junit:junit-bom:5.9.1=spotbugs -org.mockito:mockito-core:4.4.0=testRuntimeClasspath +org.junit.jupiter:junit-jupiter-api:5.14.1=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-engine:5.14.1=testRuntimeClasspath +org.junit.jupiter:junit-jupiter-params:5.14.1=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +org.junit.jupiter:junit-jupiter:5.14.1=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +org.junit.platform:junit-platform-commons:1.14.1=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +org.junit.platform:junit-platform-engine:1.14.1=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +org.junit.platform:junit-platform-launcher:1.14.1=testRuntimeClasspath +org.junit.platform:junit-platform-runner:1.14.1=testRuntimeClasspath +org.junit.platform:junit-platform-suite-api:1.14.1=testRuntimeClasspath +org.junit.platform:junit-platform-suite-commons:1.14.1=testRuntimeClasspath +org.junit:junit-bom:5.14.0=spotbugs +org.junit:junit-bom:5.14.1=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +org.mockito:mockito-core:4.4.0=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +org.mockito:mockito-junit-jupiter:4.4.0=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath org.msgpack:jackson-dataformat-msgpack:0.9.6=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath org.msgpack:msgpack-core:0.9.6=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath org.objenesis:objenesis:3.3=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath org.opentest4j:opentest4j:1.3.0=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.4=spotbugs -org.ow2.asm:asm-commons:9.4=spotbugs -org.ow2.asm:asm-commons:9.9=compileClasspath,implementationDependenciesMetadata,jacocoAnt,runtimeClasspath,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.ow2.asm:asm-tree:9.4=spotbugs -org.ow2.asm:asm-tree:9.9=compileClasspath,implementationDependenciesMetadata,jacocoAnt,runtimeClasspath,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.ow2.asm:asm-util:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-util:9.4=spotbugs -org.ow2.asm:asm:9.4=spotbugs -org.ow2.asm:asm:9.9=compileClasspath,implementationDependenciesMetadata,jacocoAnt,runtimeClasspath,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +org.ow2.asm:asm-analysis:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-analysis:9.9=spotbugs +org.ow2.asm:asm-commons:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-commons:9.9.1=compileClasspath,implementationDependenciesMetadata,runtimeClasspath,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +org.ow2.asm:asm-tree:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-tree:9.9.1=compileClasspath,implementationDependenciesMetadata,runtimeClasspath,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +org.ow2.asm:asm-util:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-util:9.9=spotbugs +org.ow2.asm:asm:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm:9.9.1=compileClasspath,implementationDependenciesMetadata,runtimeClasspath,testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath org.scala-lang.modules:scala-parallel-collections_2.13:0.2.0=zinc org.scala-lang.modules:scala-parser-combinators_2.13:1.1.2=zinc org.scala-lang.modules:scala-xml_2.13:2.3.0=zinc @@ -214,13 +182,12 @@ org.slf4j:jul-to-slf4j:1.7.30=testCompileClasspath,testImplementationDependencie org.slf4j:log4j-over-slf4j:1.7.30=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath org.slf4j:slf4j-api:1.7.30=apiDependenciesMetadata,compileClasspath,implementationDependenciesMetadata,runtimeClasspath org.slf4j:slf4j-api:1.7.32=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.slf4j:slf4j-api:2.0.0=spotbugs,spotbugsSlf4j -org.slf4j:slf4j-simple:2.0.0=spotbugsSlf4j +org.slf4j:slf4j-api:2.0.17=spotbugs,spotbugsSlf4j +org.slf4j:slf4j-simple:2.0.17=spotbugsSlf4j org.snakeyaml:snakeyaml-engine:2.9=runtimeClasspath,testRuntimeClasspath -org.spockframework:spock-bom:2.4-M6-groovy-3.0=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.spockframework:spock-core:2.4-M6-groovy-3.0=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath -org.testng:testng:7.5.1=testRuntimeClasspath -org.webjars:jquery:3.5.1=testRuntimeClasspath -org.xmlresolver:xmlresolver:4.4.3=spotbugs -xml-apis:xml-apis:1.4.01=spotbugs -empty=annotationProcessor,intransitiveDependenciesMetadata,kotlinCompilerPluginClasspath,kotlinNativeCompilerPluginClasspath,kotlinScriptDef,kotlinScriptDefExtensions,scalaCompilerPlugins,scalaToolchainRuntimeClasspath,shadow,spotbugsPlugins,testAnnotationProcessor,testApiDependenciesMetadata,testCompileOnlyDependenciesMetadata,testIntransitiveDependenciesMetadata,testKotlinScriptDef,testKotlinScriptDefExtensions +org.spockframework:spock-bom:2.4-groovy-3.0=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +org.spockframework:spock-core:2.4-groovy-3.0=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +org.tabletest:tabletest-junit:1.2.1=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +org.tabletest:tabletest-parser:1.2.0=testCompileClasspath,testImplementationDependenciesMetadata,testRuntimeClasspath +org.xmlresolver:xmlresolver:5.3.3=spotbugs +empty=annotationProcessor,intransitiveDependenciesMetadata,kotlinCompilerPluginClasspath,kotlinNativeCompilerPluginClasspath,kotlinScriptDefExtensions,scalaCompilerPlugins,scalaToolchainRuntimeClasspath,shadow,spotbugsPlugins,testAnnotationProcessor,testApiDependenciesMetadata,testCompileOnlyDependenciesMetadata,testIntransitiveDependenciesMetadata,testKotlinScriptDefExtensions diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/CiVisibilityRepoServices.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/CiVisibilityRepoServices.java index 57afe1d8e0b..bdbf2148c10 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/CiVisibilityRepoServices.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/CiVisibilityRepoServices.java @@ -159,10 +159,15 @@ private static PullRequestInfo buildUserPullRequestInfo( String targetSha = environment.get(Constants.DDCI_PULL_REQUEST_TARGET_SHA); String sourceSha = environment.get(Constants.DDCI_PULL_REQUEST_SOURCE_SHA); String mergeBase = null; - try { - mergeBase = gitClient.getMergeBase(targetSha, sourceSha); - } catch (Exception ignored) { + + if (!Constants.DDCI_LEGACY_KIND.equals(environment.get(Constants.DDCI_REQUEST_KIND))) { + // legacy mode doesn't set a valid target sha to compute the merge base + try { + mergeBase = gitClient.getMergeBase(targetSha, sourceSha); + } catch (Exception ignored) { + } } + PullRequestInfo ddCiInfo = new PullRequestInfo( null, diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/CiVisibilitySystem.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/CiVisibilitySystem.java index 7ebb99c36a3..246ebe2b336 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/CiVisibilitySystem.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/CiVisibilitySystem.java @@ -18,6 +18,7 @@ import datadog.trace.api.git.GitInfoProvider; import datadog.trace.bootstrap.ContextStore; import datadog.trace.bootstrap.instrumentation.api.AgentTracer; +import datadog.trace.civisibility.compiler.CompilerModuleExporter; import datadog.trace.civisibility.config.ExecutionSettings; import datadog.trace.civisibility.config.JvmInfo; import datadog.trace.civisibility.coverage.file.instrumentation.CoverageClassTransformer; @@ -75,6 +76,10 @@ public static void start(Instrumentation inst, SharedCommunicationObjects sco) { sco.createRemaining(config); + if (config.isCiVisibilityCompilerPluginAutoConfigurationEnabled()) { + inst.addTransformer(new CompilerModuleExporter(inst)); + } + CiVisibilityMetricCollector metricCollector = config.isCiVisibilityTelemetryEnabled() ? new CiVisibilityMetricCollectorImpl() diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/Constants.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/Constants.java index d662f9b5e58..80ce365480b 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/Constants.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/Constants.java @@ -22,4 +22,14 @@ public interface Constants { * the necessary data is not exposed by the CI provider */ String DDCI_PULL_REQUEST_TARGET_SHA = "DDCI_PULL_REQUEST_TARGET_SHA"; + + /** + * Env var containing the DDCI mode used. When the legacy mode is used, + * DDCI_PULL_REQUEST_TARGET_SHA won't contain the expected value, but the previous base on the + * push event. + */ + String DDCI_REQUEST_KIND = "DDCI_REQUEST_KIND"; + + // Legacy mode identifier + String DDCI_LEGACY_KIND = "REQUEST_KIND_LEGACY_REQUEST"; } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/ci/CIInfo.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/ci/CIInfo.java index 8387207d8ed..090e366d35f 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/ci/CIInfo.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/ci/CIInfo.java @@ -232,7 +232,9 @@ private String sanitizeWorkspace(String workspace) { : (realCiWorkspace.substring(0, realCiWorkspace.length() - 1)); } - /** @return Workspace path without the trailing separator */ + /** + * @return Workspace path without the trailing separator + */ public String getCiWorkspace() { return ciWorkspace; } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/ci/GithubActionsInfo.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/ci/GithubActionsInfo.java index 67ed88bad56..856e280b933 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/ci/GithubActionsInfo.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/ci/GithubActionsInfo.java @@ -14,15 +14,25 @@ import datadog.trace.api.git.GitInfo; import datadog.trace.civisibility.ci.env.CiEnvironment; import datadog.trace.util.Strings; +import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; +import java.io.IOException; import java.nio.charset.StandardCharsets; +import java.nio.file.DirectoryStream; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; import java.util.Map; +import java.util.regex.Matcher; +import java.util.regex.Pattern; import javax.annotation.Nonnull; +import javax.annotation.Nullable; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +@SuppressFBWarnings( + value = "DMI_HARDCODED_ABSOLUTE_FILENAME", + justification = + "The GitHub Actions runner diagnostics directories have well-known absolute paths for Linux runners") class GithubActionsInfo implements CIProviderInfo { private static final Logger LOGGER = LoggerFactory.getLogger(GithubActionsInfo.class); @@ -43,11 +53,26 @@ class GithubActionsInfo implements CIProviderInfo { public static final String GHACTIONS_JOB = "GITHUB_JOB"; public static final String GITHUB_BASE_REF = "GITHUB_BASE_REF"; public static final String GITHUB_EVENT_PATH = "GITHUB_EVENT_PATH"; + public static final String GHACTIONS_JOB_CHECK_RUN_ID = "JOB_CHECK_RUN_ID"; + + static final String GHA_DIAGNOSTICS_DIR = "/home/runner/actions-runner/_diag"; + static final String GHA_DIAGNOSTICS_DIR_CACHED = "/home/runner/actions-runner/cached/_diag"; + private static final Pattern CHECK_RUN_ID_PATTERN = + Pattern.compile("\"k\"\\s*:\\s*\"check_run_id\"\\s*,\\s*\"v\"\\s*:\\s*(\\d+(?:\\.\\d+)?)"); private final CiEnvironment environment; + private final Path diagnosticsDir; + private final Path diagnosticsDirCached; GithubActionsInfo(CiEnvironment environment) { + this(environment, Paths.get(GHA_DIAGNOSTICS_DIR), Paths.get(GHA_DIAGNOSTICS_DIR_CACHED)); + } + + // for testing purposes + GithubActionsInfo(CiEnvironment environment, Path diagnosticsDir, Path diagnosticsDirCached) { this.environment = environment; + this.diagnosticsDir = diagnosticsDir; + this.diagnosticsDirCached = diagnosticsDirCached; } @Override @@ -63,26 +88,35 @@ public GitInfo buildCIGitInfo() { @Override public CIInfo buildCIInfo() { + String serverUrl = filterSensitiveInfo(environment.get(GHACTIONS_URL)); + String repository = environment.get(GHACTIONS_REPOSITORY); + String pipelineId = environment.get(GHACTIONS_PIPELINE_ID); + String commit = environment.get(GHACTIONS_SHA); + final String pipelineUrl = buildPipelineUrl( - filterSensitiveInfo(environment.get(GHACTIONS_URL)), - environment.get(GHACTIONS_REPOSITORY), - environment.get(GHACTIONS_PIPELINE_ID), - environment.get(GHACTIONS_PIPELINE_RETRY)); - final String jobUrl = - buildJobUrl( - filterSensitiveInfo(environment.get(GHACTIONS_URL)), - environment.get(GHACTIONS_REPOSITORY), - environment.get(GHACTIONS_SHA)); + serverUrl, repository, pipelineId, environment.get(GHACTIONS_PIPELINE_RETRY)); + + // Try to get numeric job ID for better job URL + String numericJobId = getNumericJobId(); + String jobId; + String jobUrl; + if (numericJobId != null) { + jobId = numericJobId; + jobUrl = buildJobUrlWithNumericId(serverUrl, repository, pipelineId, numericJobId); + } else { + jobId = environment.get(GHACTIONS_JOB); + jobUrl = buildJobUrl(serverUrl, repository, commit); + } CIInfo.Builder builder = CIInfo.builder(environment); return builder .ciProviderName(GHACTIONS_PROVIDER_NAME) - .ciPipelineId(environment.get(GHACTIONS_PIPELINE_ID)) + .ciPipelineId(pipelineId) .ciPipelineName(environment.get(GHACTIONS_PIPELINE_NAME)) .ciPipelineNumber(environment.get(GHACTIONS_PIPELINE_NUMBER)) .ciPipelineUrl(pipelineUrl) - .ciJobId(environment.get(GHACTIONS_JOB)) + .ciJobId(jobId) .ciJobName(environment.get(GHACTIONS_JOB)) .ciJobUrl(jobUrl) .ciWorkspace(expandTilde(environment.get(GHACTIONS_WORKSPACE_PATH))) @@ -186,6 +220,94 @@ private String buildJobUrl(final String host, final String repo, final String co return String.format("%s/%s/commit/%s/checks", host, repo, commit); } + private String buildJobUrlWithNumericId( + final String host, final String repo, final String pipelineId, final String jobId) { + return String.format("%s/%s/actions/runs/%s/job/%s", host, repo, pipelineId, jobId); + } + + /** + * Gets the numeric job ID for GitHub Actions. + * + *

First checks the JOB_CHECK_RUN_ID environment variable. If not present, falls back to + * parsing GitHub Actions diagnostics files (Worker_*.log) in the runner's diagnostics directory. + * + * @return the numeric job ID, or null if not found + */ + @Nullable + private String getNumericJobId() { + // First, check if the numeric job ID is provided via environment variable + String jobId = environment.get(GHACTIONS_JOB_CHECK_RUN_ID); + if (Strings.isNotBlank(jobId)) { + return jobId; + } + + // Fall back to parsing diagnostics files + jobId = parseJobIdFromDirectory(diagnosticsDir); + if (Strings.isNotBlank(jobId)) { + return jobId; + } + + return parseJobIdFromDirectory(diagnosticsDirCached); + } + + @Nullable + private String parseJobIdFromDirectory(Path directory) { + if (!Files.isDirectory(directory)) { + return null; + } + + try (DirectoryStream stream = Files.newDirectoryStream(directory, "Worker_*.log")) { + // Find the most recent file by filename (contains timestamp like + // Worker_20260130-154110-utc.log) + Path mostRecentLog = null; + for (Path workerLog : stream) { + if (mostRecentLog == null + || workerLog.getFileName().toString().compareTo(mostRecentLog.getFileName().toString()) + > 0) { + mostRecentLog = workerLog; + } + } + + if (mostRecentLog == null) { + return null; + } + + String content = new String(Files.readAllBytes(mostRecentLog), StandardCharsets.UTF_8); + return parseCheckRunIdFromContent(content); + } catch (IOException e) { + LOGGER.debug("Error reading diagnostics directory: {}", directory, e); + return null; + } + } + + /** + * Extracts the last check_run_id value from the file content. + * + *

The JSON structure in worker logs contains: {"k":"check_run_id","v":12345.0} + * + *

Uses the last match because a single worker file might contain multiple jobs' data, and the + * most recent job entry appears last. + * + * @param content the file content to parse + * @return the check_run_id as a string, or null if not found + */ + @Nullable + String parseCheckRunIdFromContent(String content) { + Matcher matcher = CHECK_RUN_ID_PATTERN.matcher(content); + String lastMatch = null; + while (matcher.find()) { + String value = matcher.group(1); + // Strip decimal part if present + int pointIdx = value.indexOf('.'); + if (pointIdx != -1) { + lastMatch = value.substring(0, pointIdx); + } else { + lastMatch = value; + } + } + return lastMatch; + } + @Override public Provider getProvider() { return Provider.GITHUBACTIONS; diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/codeowners/Codeowners.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/codeowners/Codeowners.java index 15bcd2d911e..3ac7aead31f 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/codeowners/Codeowners.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/codeowners/Codeowners.java @@ -13,6 +13,8 @@ public interface Codeowners { @Nullable Collection getOwners(@Nonnull String path); - /** @return {@code true} if {@code CODEOWNERS} file could be located and parsed */ + /** + * @return {@code true} if {@code CODEOWNERS} file could be located and parsed + */ boolean exist(); } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/compiler/CompilerModuleExporter.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/compiler/CompilerModuleExporter.java new file mode 100644 index 00000000000..31344f9b298 --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/compiler/CompilerModuleExporter.java @@ -0,0 +1,65 @@ +package datadog.trace.civisibility.compiler; + +import datadog.trace.util.JDK9ModuleAccess; +import java.lang.instrument.ClassFileTransformer; +import java.lang.instrument.Instrumentation; +import java.security.ProtectionDomain; +import java.util.concurrent.ConcurrentHashMap; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Exports jdk.compiler internal packages to the classloader that loads dd-javac-plugin. + * + *

On JDK 16+ (strong encapsulation), dd-javac-plugin's CompilerModuleOpener uses burningwave to + * export these packages. On JDK 26+, burningwave fails due to Unsafe restrictions (JEP 471/498). + * This transformer intercepts dd-javac-plugin class loading and does the export using + * Instrumentation.redefineModule() instead. + * + *

Each Maven compilation step (compile, testCompile) may use a different classloader, so we + * track which classloaders have already been exported to and re-export for new ones. + */ +public class CompilerModuleExporter implements ClassFileTransformer { + + private static final Logger LOGGER = LoggerFactory.getLogger(CompilerModuleExporter.class); + + private static final String COMPILER_PLUGIN_CLASS_PREFIX = "datadog/compiler/"; + private static final String[] COMPILER_PACKAGES = { + "com.sun.tools.javac.api", + "com.sun.tools.javac.code", + "com.sun.tools.javac.comp", + "com.sun.tools.javac.tree", + "com.sun.tools.javac.util" + }; + + private final Instrumentation inst; + private final ConcurrentHashMap exportedClassLoaders = + new ConcurrentHashMap<>(); + + public CompilerModuleExporter(Instrumentation inst) { + this.inst = inst; + } + + @Override + public byte[] transform( + ClassLoader loader, + String className, + Class classBeingRedefined, + ProtectionDomain protectionDomain, + byte[] classfileBuffer) { + if (loader != null && className != null && className.startsWith(COMPILER_PLUGIN_CLASS_PREFIX)) { + exportedClassLoaders.computeIfAbsent(loader, this::exportJdkCompilerModule); + } + return null; // no bytecode modification + } + + private Boolean exportJdkCompilerModule(ClassLoader loader) { + try { + JDK9ModuleAccess.exportModuleToUnnamedModule(inst, "jdk.compiler", COMPILER_PACKAGES, loader); + LOGGER.debug("Exported jdk.compiler to classloader {}", loader); + } catch (Throwable e) { + LOGGER.debug("Could not export jdk.compiler packages for compiler plugin", e); + } + return Boolean.TRUE; + } +} diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/CiVisibilitySettings.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/CiVisibilitySettings.java index 05aeca8a8d5..d7502ccec48 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/CiVisibilitySettings.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/CiVisibilitySettings.java @@ -21,7 +21,24 @@ public class CiVisibilitySettings { false, EarlyFlakeDetectionSettings.DEFAULT, TestManagementSettings.DEFAULT, - null); + null, + false); + + public static final CiVisibilitySettings SETTINGS_REQUEST_ERROR = + new CiVisibilitySettings( + false, + false, + false, + false, + false, + false, + false, + false, + false, + EarlyFlakeDetectionSettings.DEFAULT, + TestManagementSettings.DEFAULT, + null, + true); private final boolean itrEnabled; private final boolean codeCoverage; @@ -35,6 +52,7 @@ public class CiVisibilitySettings { private final EarlyFlakeDetectionSettings earlyFlakeDetectionSettings; private final TestManagementSettings testManagementSettings; @Nullable private final String defaultBranch; + private final boolean settingsRequestError; CiVisibilitySettings( boolean itrEnabled, @@ -48,7 +66,8 @@ public class CiVisibilitySettings { boolean failedTestReplayEnabled, EarlyFlakeDetectionSettings earlyFlakeDetectionSettings, TestManagementSettings testManagementSettings, - @Nullable String defaultBranch) { + @Nullable String defaultBranch, + boolean settingsRequestError) { this.itrEnabled = itrEnabled; this.codeCoverage = codeCoverage; this.testsSkipping = testsSkipping; @@ -61,6 +80,7 @@ public class CiVisibilitySettings { this.earlyFlakeDetectionSettings = earlyFlakeDetectionSettings; this.testManagementSettings = testManagementSettings; this.defaultBranch = defaultBranch; + this.settingsRequestError = settingsRequestError; } public boolean isItrEnabled() { @@ -112,6 +132,10 @@ public String getDefaultBranch() { return defaultBranch; } + public boolean isSettingsRequestError() { + return settingsRequestError; + } + @Override public boolean equals(Object o) { if (this == o) { @@ -132,7 +156,8 @@ public boolean equals(Object o) { && failedTestReplayEnabled == that.failedTestReplayEnabled && Objects.equals(earlyFlakeDetectionSettings, that.earlyFlakeDetectionSettings) && Objects.equals(testManagementSettings, that.testManagementSettings) - && Objects.equals(defaultBranch, that.defaultBranch); + && Objects.equals(defaultBranch, that.defaultBranch) + && settingsRequestError == that.settingsRequestError; } @Override @@ -149,7 +174,8 @@ public int hashCode() { failedTestReplayEnabled, earlyFlakeDetectionSettings, testManagementSettings, - defaultBranch); + defaultBranch, + settingsRequestError); } public interface Factory { @@ -180,7 +206,8 @@ public CiVisibilitySettings fromJson(Map json) { (Map) json.get("early_flake_detection")), TestManagementSettings.JsonAdapter.INSTANCE.fromJson( (Map) json.get("test_management")), - getString(json, "default_branch", null)); + getString(json, "default_branch", null), + false); // Correctly deserialized settings response is never considered as "errored" } private static boolean getBoolean( diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/ConfigurationApiImpl.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/ConfigurationApiImpl.java index d3bc8345e0b..79bc4899b30 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/ConfigurationApiImpl.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/ConfigurationApiImpl.java @@ -65,6 +65,7 @@ public class ConfigurationApiImpl implements ConfigurationApi { private final JsonAdapter> requestAdapter; private final JsonAdapter> settingsResponseAdapter; private final JsonAdapter> testIdentifiersResponseAdapter; + private final JsonAdapter> knownTestsRequestAdapter; private final JsonAdapter> testFullNamesResponseAdapter; private final JsonAdapter> testManagementRequestAdapter; private final JsonAdapter> testManagementTestsResponseAdapter; @@ -104,6 +105,11 @@ public ConfigurationApiImpl(BackendApi backendApi, CiVisibilityMetricCollector m ConfigurationApiImpl.class, MultiEnvelopeDto.class, TestIdentifierJson.class); testIdentifiersResponseAdapter = moshi.adapter(testIdentifiersResponseType); + ParameterizedType knownTestsRequestType = + Types.newParameterizedTypeWithOwner( + ConfigurationApiImpl.class, EnvelopeDto.class, KnownTestsRequestDto.class); + knownTestsRequestAdapter = moshi.adapter(knownTestsRequestType); + ParameterizedType testFullNamesResponseType = Types.newParameterizedTypeWithOwner( ConfigurationApiImpl.class, EnvelopeDto.class, KnownTestsDto.class); @@ -269,28 +275,89 @@ public Map> getKnownTestsByModule(TracerEnvironment .responseBytes(CiVisibilityDistributionMetric.KNOWN_TESTS_RESPONSE_BYTES) .build(); - String uuid = uuidGenerator.get(); - EnvelopeDto request = - new EnvelopeDto<>(new DataDto<>(uuid, "ci_app_libraries_tests_request", tracerEnvironment)); - String json = requestAdapter.toJson(request); - RequestBody requestBody = RequestBody.create(JSON, json); - KnownTestsDto knownTests = - backendApi.post( - KNOWN_TESTS_URI, - requestBody, - is -> - testFullNamesResponseAdapter.fromJson(Okio.buffer(Okio.source(is))).data.attributes, - telemetryListener, - false); + // Aggregate tests map across all pages: module -> suite -> tests + Map>> aggregateTests = new HashMap<>(); + String pageState = null; + int pageNumber = 0; + + do { + pageNumber += 1; + LOGGER.debug( + "Fetching known tests page #{}{}", pageNumber, pageState != null ? " with cursor" : ""); + String uuid = uuidGenerator.get(); + KnownTestsRequestDto requestDto = new KnownTestsRequestDto(tracerEnvironment, pageState); + EnvelopeDto request = + new EnvelopeDto<>(new DataDto<>(uuid, "ci_app_libraries_tests_request", requestDto)); + String json = knownTestsRequestAdapter.toJson(request); + RequestBody requestBody = RequestBody.create(JSON, json); + KnownTestsDto knownTests = + backendApi.post( + KNOWN_TESTS_URI, + requestBody, + is -> + testFullNamesResponseAdapter.fromJson(Okio.buffer(Okio.source(is))) + .data + .attributes, + telemetryListener, + false); + + // Merge page's tests into aggregate + mergeKnownTests(aggregateTests, knownTests.tests); + + Integer pageSize = knownTests.getPageSize(); + if (pageSize != null) { + LOGGER.debug("Received page #{} of size {} for known tests", pageNumber, pageSize); + } else { + LOGGER.debug("Received page #{} for known tests", pageNumber); + } + + // Get cursor for next page (if any) + if (knownTests.hasNextPage()) { + pageState = knownTests.getNextPageCursor(); + } else { + pageState = null; + } + } while (pageState != null); + + LOGGER.debug("Finished fetching known tests after {} page(s)", pageNumber); - return parseTestIdentifiers(knownTests); + return parseTestIdentifiers(aggregateTests); } - private Map> parseTestIdentifiers(KnownTestsDto knownTests) { + private void mergeKnownTests( + Map>> aggregate, + Map>> page) { + if (page == null) { + return; + } + for (Map.Entry>> moduleEntry : page.entrySet()) { + String moduleName = moduleEntry.getKey(); + Map> pageSuites = moduleEntry.getValue(); + + Map> aggregateSuites = + aggregate.computeIfAbsent(moduleName, k -> new HashMap<>()); + + for (Map.Entry> suiteEntry : pageSuites.entrySet()) { + String suiteName = suiteEntry.getKey(); + List pageTests = suiteEntry.getValue(); + + aggregateSuites.merge( + suiteName, + pageTests, + (existingTests, newTests) -> { + existingTests.addAll(newTests); + return existingTests; + }); + } + } + } + + private Map> parseTestIdentifiers( + Map>> testsMap) { int knownTestsCount = 0; Map> testIdentifiers = new HashMap<>(); - for (Map.Entry>> e : knownTests.tests.entrySet()) { + for (Map.Entry>> e : testsMap.entrySet()) { String moduleName = e.getKey(); Map> testsBySuiteName = e.getValue(); @@ -341,7 +408,8 @@ public Map>> getTestManagementTests tracerEnvironment.getRepositoryUrl(), commitMessage, tracerEnvironment.getConfigurations().getTestBundle(), - commitSha))); + commitSha, + tracerEnvironment.getBranch()))); String json = testManagementRequestAdapter.toJson(request); RequestBody requestBody = RequestBody.create(JSON, json); TestManagementTestsDto testManagementTestsDto = @@ -495,8 +563,68 @@ public Map toJson(MetaDto metaDto) { private static final class KnownTestsDto { private final Map>> tests; - private KnownTestsDto(Map>> tests) { + @Json(name = "page_info") + private final PageInfoResponse pageInfo; + + private KnownTestsDto(Map>> tests, PageInfoResponse pageInfo) { this.tests = tests; + this.pageInfo = pageInfo; + } + + public boolean hasNextPage() { + return pageInfo != null && pageInfo.hasNext; + } + + @Nullable + public Integer getPageSize() { + return pageInfo != null ? pageInfo.size : null; + } + + @Nullable + public String getNextPageCursor() { + return pageInfo != null ? pageInfo.cursor : null; + } + } + + private static final class PageInfoResponse { + private final String cursor; + private final int size; + + @Json(name = "has_next") + private final boolean hasNext; + + private PageInfoResponse(String cursor, int size, boolean hasNext) { + this.cursor = cursor; + this.size = size; + this.hasNext = hasNext; + } + } + + private static final class KnownTestsRequestDto { + @Json(name = "repository_url") + private final String repositoryUrl; + + private final String service; + private final String env; + + @Json(name = "page_info") + private final PageInfoRequest pageInfo; + + private KnownTestsRequestDto(TracerEnvironment tracerEnvironment, @Nullable String pageState) { + this.repositoryUrl = tracerEnvironment.getRepositoryUrl(); + this.service = tracerEnvironment.getService(); + this.env = tracerEnvironment.getEnv(); + this.pageInfo = new PageInfoRequest(pageState); + } + + private static final class PageInfoRequest { + @Json(name = "page_state") + @Nullable + private final String pageState; + + private PageInfoRequest(@Nullable String pageState) { + this.pageState = pageState; + } } } @@ -509,13 +637,15 @@ private static final class TestManagementDto { private final String module; private final String sha; + private final String branch; private TestManagementDto( - String repositoryUrl, String commitMessage, String module, String sha) { + String repositoryUrl, String commitMessage, String module, String sha, String branch) { this.repositoryUrl = repositoryUrl; this.commitMessage = commitMessage; this.module = module; this.sha = sha; + this.branch = branch; } } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/ConfigurationErrors.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/ConfigurationErrors.java new file mode 100644 index 00000000000..f488468f2f4 --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/ConfigurationErrors.java @@ -0,0 +1,106 @@ +package datadog.trace.civisibility.config; + +import datadog.trace.api.DDTags; +import datadog.trace.bootstrap.instrumentation.api.AgentSpan; +import datadog.trace.civisibility.ipc.serialization.Serializer; +import java.nio.ByteBuffer; +import java.util.Objects; + +/** Tracks which CI Visibility backend requests failed. */ +public class ConfigurationErrors { + + public static final ConfigurationErrors NONE = + new ConfigurationErrors(false, false, false, false, false); + + private static final int SETTINGS_FLAG = 1; + private static final int SKIPPABLE_TESTS_FLAG = 2; + private static final int FLAKY_TESTS_FLAG = 4; + private static final int KNOWN_TESTS_FLAG = 8; + private static final int TEST_MANAGEMENT_TESTS_FLAG = 16; + + private final boolean settings; + private final boolean skippableTests; + private final boolean flakyTests; + private final boolean knownTests; + private final boolean testManagementTests; + + public ConfigurationErrors( + boolean settings, + boolean skippableTests, + boolean flakyTests, + boolean knownTests, + boolean testManagementTests) { + this.settings = settings; + this.skippableTests = skippableTests; + this.flakyTests = flakyTests; + this.knownTests = knownTests; + this.testManagementTests = testManagementTests; + } + + public boolean hasAny() { + return settings || skippableTests || flakyTests || knownTests || testManagementTests; + } + + public void applyTags(AgentSpan span) { + if (settings) { + span.setTag(DDTags.CI_LIBRARY_CONFIGURATION_ERROR_SETTINGS, true); + } + if (skippableTests) { + span.setTag(DDTags.CI_LIBRARY_CONFIGURATION_ERROR_SKIPPABLE_TESTS, true); + } + if (flakyTests) { + span.setTag(DDTags.CI_LIBRARY_CONFIGURATION_ERROR_FLAKY_TESTS, true); + } + if (knownTests) { + span.setTag(DDTags.CI_LIBRARY_CONFIGURATION_ERROR_KNOWN_TESTS, true); + } + if (testManagementTests) { + span.setTag(DDTags.CI_LIBRARY_CONFIGURATION_ERROR_TEST_MANAGEMENT_TESTS, true); + } + } + + public static void serialize(Serializer s, ConfigurationErrors errors) { + byte flags = + (byte) + ((errors.settings ? SETTINGS_FLAG : 0) + | (errors.skippableTests ? SKIPPABLE_TESTS_FLAG : 0) + | (errors.flakyTests ? FLAKY_TESTS_FLAG : 0) + | (errors.knownTests ? KNOWN_TESTS_FLAG : 0) + | (errors.testManagementTests ? TEST_MANAGEMENT_TESTS_FLAG : 0)); + s.write(flags); + } + + public static ConfigurationErrors deserialize(ByteBuffer buffer) { + byte flags = Serializer.readByte(buffer); + if (flags == 0) { + return NONE; + } + return new ConfigurationErrors( + (flags & SETTINGS_FLAG) != 0, + (flags & SKIPPABLE_TESTS_FLAG) != 0, + (flags & FLAKY_TESTS_FLAG) != 0, + (flags & KNOWN_TESTS_FLAG) != 0, + (flags & TEST_MANAGEMENT_TESTS_FLAG) != 0); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + ConfigurationErrors that = (ConfigurationErrors) o; + return settings == that.settings + && skippableTests == that.skippableTests + && flakyTests == that.flakyTests + && knownTests == that.knownTests + && testManagementTests == that.testManagementTests; + } + + @Override + public int hashCode() { + return Objects.hash(settings, skippableTests, flakyTests, knownTests, testManagementTests); + } +} diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/ExecutionSettings.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/ExecutionSettings.java index ba312cf8fa4..36e48401132 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/ExecutionSettings.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/ExecutionSettings.java @@ -38,7 +38,30 @@ public class ExecutionSettings { Collections.emptyList(), Collections.emptyList(), Collections.emptyList(), - LineDiff.EMPTY); + LineDiff.EMPTY, + ConfigurationErrors.NONE); + + public static final ExecutionSettings SETTINGS_REQUEST_ERROR = + new ExecutionSettings( + false, + false, + false, + false, + false, + false, + false, + EarlyFlakeDetectionSettings.DEFAULT, + TestManagementSettings.DEFAULT, + null, + Collections.emptyMap(), + Collections.emptyMap(), + null, + null, + Collections.emptyList(), + Collections.emptyList(), + Collections.emptyList(), + LineDiff.EMPTY, + new ConfigurationErrors(true, false, false, false, false)); private final boolean itrEnabled; private final boolean codeCoverageEnabled; @@ -55,6 +78,7 @@ public class ExecutionSettings { @Nonnull private final Map testSettings; @Nonnull private final Map settingsCount; @Nonnull private final Diff pullRequestDiff; + @Nonnull private final ConfigurationErrors configurationErrors; public ExecutionSettings( boolean itrEnabled, @@ -74,7 +98,8 @@ public ExecutionSettings( @Nonnull Collection quarantinedTests, @Nonnull Collection disabledTests, @Nonnull Collection attemptToFixTests, - @Nonnull Diff pullRequestDiff) { + @Nonnull Diff pullRequestDiff, + @Nonnull ConfigurationErrors configurationErrors) { this.itrEnabled = itrEnabled; this.codeCoverageEnabled = codeCoverageEnabled; this.testSkippingEnabled = testSkippingEnabled; @@ -88,6 +113,7 @@ public ExecutionSettings( this.skippableTests = skippableTests; this.skippableTestsCoverage = skippableTestsCoverage; this.pullRequestDiff = pullRequestDiff; + this.configurationErrors = configurationErrors; testSettings = new HashMap<>(); if (flakyTests != null) { @@ -127,7 +153,8 @@ private ExecutionSettings( @Nonnull Map skippableTestsCoverage, @Nonnull Map testSettings, @Nonnull EnumMap settingsCount, - @Nonnull Diff pullRequestDiff) { + @Nonnull Diff pullRequestDiff, + @Nonnull ConfigurationErrors configurationErrors) { this.itrEnabled = itrEnabled; this.codeCoverageEnabled = codeCoverageEnabled; this.testSkippingEnabled = testSkippingEnabled; @@ -143,6 +170,7 @@ private ExecutionSettings( this.testSettings = testSettings; this.settingsCount = settingsCount; this.pullRequestDiff = pullRequestDiff; + this.configurationErrors = configurationErrors; } /** @@ -249,6 +277,11 @@ public Diff getPullRequestDiff() { return pullRequestDiff; } + @Nonnull + public ConfigurationErrors getConfigurationErrors() { + return configurationErrors; + } + @Override public boolean equals(Object o) { if (this == o) { @@ -272,7 +305,8 @@ public boolean equals(Object o) { && Objects.equals(skippableTestsCoverage, that.skippableTestsCoverage) && Objects.equals(testSettings, that.testSettings) && Objects.equals(settingsCount, that.settingsCount) - && Objects.equals(pullRequestDiff, that.pullRequestDiff); + && Objects.equals(pullRequestDiff, that.pullRequestDiff) + && Objects.equals(configurationErrors, that.configurationErrors); } @Override @@ -292,7 +326,8 @@ public int hashCode() { skippableTestsCoverage, testSettings, settingsCount, - pullRequestDiff); + pullRequestDiff, + configurationErrors); } public static class Serializer { @@ -324,6 +359,8 @@ public static ByteBuffer serialize(ExecutionSettings settings) { | (settings.failedTestReplayEnabled ? FAILED_TEST_REPLAY_ENABLED_FLAG : 0)); s.write(flags); + ConfigurationErrors.serialize(s, settings.configurationErrors); + EarlyFlakeDetectionSettings.Serializer.serialize(s, settings.earlyFlakeDetectionSettings); TestManagementSettings.Serializer.serialize(s, settings.testManagementSettings); @@ -364,6 +401,8 @@ public static ExecutionSettings deserialize(ByteBuffer buffer) { (flags & CODE_COVERAGE_REPORT_UPLOAD_ENABLED_FLAG) != 0; boolean failedTestReplayEnabled = (flags & FAILED_TEST_REPLAY_ENABLED_FLAG) != 0; + ConfigurationErrors configurationErrors = ConfigurationErrors.deserialize(buffer); + EarlyFlakeDetectionSettings earlyFlakeDetectionSettings = EarlyFlakeDetectionSettings.Serializer.deserialize(buffer); @@ -414,7 +453,8 @@ public static ExecutionSettings deserialize(ByteBuffer buffer) { skippableTestsCoverage, testSettings, settingsCount, - diff); + diff, + configurationErrors); } } } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/ExecutionSettingsFactoryImpl.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/ExecutionSettingsFactoryImpl.java index 5fcc776edce..46ecb133ff3 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/ExecutionSettingsFactoryImpl.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/ExecutionSettingsFactoryImpl.java @@ -27,6 +27,7 @@ import java.util.concurrent.Future; import java.util.concurrent.ThreadFactory; import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.function.Function; import javax.annotation.Nonnull; import javax.annotation.Nullable; @@ -73,7 +74,9 @@ public ExecutionSettingsFactoryImpl( this.repositoryRoot = repositoryRoot; } - /** @return Executions settings by module name */ + /** + * @return Executions settings by module name + */ public Map create(@Nonnull JvmInfo jvmInfo) { TracerEnvironment tracerEnvironment = buildTracerEnvironment(jvmInfo, null); return create(tracerEnvironment); @@ -129,11 +132,11 @@ private Map create(TracerEnvironment tracerEnvironmen } catch (InterruptedException e) { Thread.currentThread().interrupt(); LOGGER.error("Interrupted while creating execution settings"); - return Collections.singletonMap(DEFAULT_SETTINGS, ExecutionSettings.EMPTY); + return Collections.singletonMap(DEFAULT_SETTINGS, ExecutionSettings.SETTINGS_REQUEST_ERROR); } catch (ExecutionException e) { LOGGER.error("Error while creating execution settings", e); - return Collections.singletonMap(DEFAULT_SETTINGS, ExecutionSettings.EMPTY); + return Collections.singletonMap(DEFAULT_SETTINGS, ExecutionSettings.SETTINGS_REQUEST_ERROR); } finally { settingsExecutor.shutdownNow(); @@ -144,6 +147,10 @@ private Map create(TracerEnvironment tracerEnvironmen private Map doCreate( TracerEnvironment tracerEnvironment, CiVisibilitySettings settings, ExecutorService executor) throws InterruptedException, ExecutionException { + if (settings.isSettingsRequestError()) { + return Collections.singletonMap(DEFAULT_SETTINGS, ExecutionSettings.SETTINGS_REQUEST_ERROR); + } + boolean itrEnabled = isFeatureEnabled( settings, CiVisibilitySettings::isItrEnabled, Config::isCiVisibilityItrEnabled); @@ -217,17 +224,28 @@ private Map doCreate( codeCoverageReportUpload, failedTestReplayEnabled); + AtomicBoolean skippableTestsError = new AtomicBoolean(); + AtomicBoolean flakyTestsError = new AtomicBoolean(); + AtomicBoolean knownTestsError = new AtomicBoolean(); + AtomicBoolean testManagementTestsError = new AtomicBoolean(); + Future skippableTestsFuture = - executor.submit(() -> getSkippableTests(tracerEnvironment, itrEnabled)); + executor.submit( + () -> getSkippableTests(tracerEnvironment, itrEnabled, skippableTestsError)); Future>> flakyTestsFuture = - executor.submit(() -> getFlakyTestsByModule(tracerEnvironment, flakyTestRetriesEnabled)); + executor.submit( + () -> + getFlakyTestsByModule(tracerEnvironment, flakyTestRetriesEnabled, flakyTestsError)); Future>> knownTestsFuture = - executor.submit(() -> getKnownTestsByModule(tracerEnvironment, knownTestsRequest)); + executor.submit( + () -> getKnownTestsByModule(tracerEnvironment, knownTestsRequest, knownTestsError)); Future>>> testManagementTestsFuture = executor.submit( () -> getTestManagementTestsByModule( - tracerEnvironment, testManagementSettings.isEnabled())); + tracerEnvironment, + testManagementSettings.isEnabled(), + testManagementTestsError)); Future pullRequestDiffFuture = executor.submit( () -> getPullRequestDiff(impactedTestsEnabled, settings.getDefaultBranch())); @@ -248,6 +266,14 @@ private Map doCreate( Diff pullRequestDiff = pullRequestDiffFuture.get(); + ConfigurationErrors configurationErrors = + new ConfigurationErrors( + false, + skippableTestsError.get(), + flakyTestsError.get(), + knownTestsError.get(), + testManagementTestsError.get()); + Map settingsByModule = new HashMap<>(); Set moduleNames = getModuleNames( @@ -285,7 +311,8 @@ private Map doCreate( quarantinedTestsByModule.getOrDefault(moduleName, Collections.emptyList()), disabledTestsByModule.getOrDefault(moduleName, Collections.emptyList()), attemptToFixTestsByModule.getOrDefault(moduleName, Collections.emptyList()), - pullRequestDiff)); + pullRequestDiff, + configurationErrors)); } return settingsByModule; } @@ -305,8 +332,8 @@ private CiVisibilitySettings getCiVisibilitySettings(TracerEnvironment tracerEnv } } catch (Exception e) { - LOGGER.warn("Error while obtaining CI Visibility settings", e); - return CiVisibilitySettings.DEFAULT; + LOGGER.error("Error while obtaining CI Visibility settings", e); + return CiVisibilitySettings.SETTINGS_REQUEST_ERROR; } } @@ -339,7 +366,7 @@ private TestManagementSettings getTestManagementSettings(CiVisibilitySettings se @Nonnull private SkippableTests getSkippableTests( - TracerEnvironment tracerEnvironment, boolean itrEnabled) { + TracerEnvironment tracerEnvironment, boolean itrEnabled, AtomicBoolean errorFlag) { if (!itrEnabled || repositoryRoot == null) { return SkippableTests.EMPTY; } @@ -368,16 +395,20 @@ private SkippableTests getSkippableTests( } catch (InterruptedException e) { Thread.currentThread().interrupt(); LOGGER.error("Interrupted while waiting for git data upload", e); + errorFlag.set(true); return SkippableTests.EMPTY; } catch (Exception e) { LOGGER.error("Could not obtain list of skippable tests, will proceed without skipping", e); + errorFlag.set(true); return SkippableTests.EMPTY; } } @Nullable private Map> getFlakyTestsByModule( - TracerEnvironment tracerEnvironment, boolean flakyTestRetriesEnabled) { + TracerEnvironment tracerEnvironment, + boolean flakyTestRetriesEnabled, + AtomicBoolean errorFlag) { if (!(flakyTestRetriesEnabled && config.isCiVisibilityFlakyRetryOnlyKnownFlakes()) && !CIConstants.FAIL_FAST_TEST_ORDER.equalsIgnoreCase(config.getCiVisibilityTestOrder())) { return null; @@ -386,13 +417,14 @@ private Map> getFlakyTestsByModule( return configurationApi.getFlakyTestsByModule(tracerEnvironment); } catch (Exception e) { LOGGER.error("Could not obtain list of flaky tests", e); + errorFlag.set(true); return null; } } @Nullable private Map> getKnownTestsByModule( - TracerEnvironment tracerEnvironment, boolean knownTestsRequest) { + TracerEnvironment tracerEnvironment, boolean knownTestsRequest, AtomicBoolean errorFlag) { if (!knownTestsRequest) { return null; } @@ -401,13 +433,16 @@ private Map> getKnownTestsByModule( } catch (Exception e) { LOGGER.error("Could not obtain list of known tests", e); + errorFlag.set(true); return null; } } @Nullable private Map>> getTestManagementTestsByModule( - TracerEnvironment tracerEnvironment, boolean testManagementTestsRequest) { + TracerEnvironment tracerEnvironment, + boolean testManagementTestsRequest, + AtomicBoolean errorFlag) { if (!testManagementTestsRequest) { return Collections.emptyMap(); } @@ -424,6 +459,7 @@ private Map>> getTestManagementTest } } catch (Exception e) { LOGGER.error("Could not obtain list of test management tests", e); + errorFlag.set(true); return Collections.emptyMap(); } } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/TestManagementSettings.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/TestManagementSettings.java index fe4f82fd445..6581d478e17 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/TestManagementSettings.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/config/TestManagementSettings.java @@ -2,8 +2,6 @@ import com.squareup.moshi.FromJson; import java.nio.ByteBuffer; -import java.util.Collections; -import java.util.List; import java.util.Map; import java.util.Objects; @@ -27,10 +25,6 @@ public int getAttemptToFixRetries() { return attemptToFixRetries; } - public List getAttemptToFixExecutions() { - return Collections.singletonList(new ExecutionsByDuration(Long.MAX_VALUE, attemptToFixRetries)); - } - @Override public boolean equals(Object o) { if (this == o) { diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/ConcurrentCoverageStore.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/ConcurrentCoverageStore.java index 8edcaa0adb8..3a68c61f40d 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/ConcurrentCoverageStore.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/ConcurrentCoverageStore.java @@ -4,12 +4,11 @@ import datadog.trace.api.civisibility.coverage.CoverageProbes; import datadog.trace.api.civisibility.coverage.CoverageStore; import datadog.trace.api.civisibility.coverage.TestReport; -import datadog.trace.civisibility.source.SourceResolutionException; import java.util.Collection; import java.util.Map; import java.util.concurrent.ConcurrentHashMap; import java.util.function.Function; -import org.jetbrains.annotations.Nullable; +import javax.annotation.Nullable; /** A store that keeps track of coverage probes allocated for multiple threads. */ public abstract class ConcurrentCoverageStore implements CoverageStore { @@ -37,18 +36,13 @@ private T create(Thread thread) { @Override public boolean report(DDTraceId testSessionId, Long testSuiteId, long testSpanId) { - try { - report = report(testSessionId, testSuiteId, testSpanId, probes.values()); - return report != null && report.isNotEmpty(); - } catch (SourceResolutionException e) { - return false; - } + report = report(testSessionId, testSuiteId, testSpanId, probes.values()); + return report != null && report.isNotEmpty(); } @Nullable protected abstract TestReport report( - DDTraceId testSessionId, Long testSuiteId, long testSpanId, Collection probes) - throws SourceResolutionException; + DDTraceId testSessionId, Long testSuiteId, long testSpanId, Collection probes); @Nullable @Override diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/file/FileCoverageStore.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/file/FileCoverageStore.java index 0399ccbf6e2..6de1354e7c7 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/file/FileCoverageStore.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/file/FileCoverageStore.java @@ -11,7 +11,6 @@ import datadog.trace.api.civisibility.telemetry.tag.CoverageErrorType; import datadog.trace.civisibility.coverage.ConcurrentCoverageStore; import datadog.trace.civisibility.source.SourcePathResolver; -import datadog.trace.civisibility.source.SourceResolutionException; import java.util.ArrayList; import java.util.Collection; import java.util.Collections; @@ -20,7 +19,7 @@ import java.util.List; import java.util.Set; import java.util.function.Function; -import org.jetbrains.annotations.Nullable; +import javax.annotation.Nullable; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -47,61 +46,54 @@ private FileCoverageStore( @Nullable @Override protected TestReport report( - DDTraceId testSessionId, Long testSuiteId, long testSpanId, Collection probes) - throws SourceResolutionException { - try { - Set> combinedClasses = Collections.newSetFromMap(new IdentityHashMap<>()); - Collection combinedNonCodeResources = new HashSet<>(); - - for (FileProbes probe : probes) { - combinedClasses.addAll(probe.getCoveredClasses()); - combinedNonCodeResources.addAll(probe.getNonCodeResources()); - } + DDTraceId testSessionId, Long testSuiteId, long testSpanId, Collection probes) { + Set> combinedClasses = Collections.newSetFromMap(new IdentityHashMap<>()); + Collection combinedNonCodeResources = new HashSet<>(); - if (combinedClasses.isEmpty() && combinedNonCodeResources.isEmpty()) { - return null; - } + for (FileProbes probe : probes) { + combinedClasses.addAll(probe.getCoveredClasses()); + combinedNonCodeResources.addAll(probe.getNonCodeResources()); + } - Set coveredPaths = set(combinedClasses.size() + combinedNonCodeResources.size()); - for (Class clazz : combinedClasses) { - String sourcePath = sourcePathResolver.getSourcePath(clazz); - if (sourcePath == null) { - log.debug( - "Skipping coverage reporting for {} because source path could not be determined", - clazz); - metrics.add(CiVisibilityCountMetric.CODE_COVERAGE_ERRORS, 1, CoverageErrorType.PATH); - continue; - } - coveredPaths.add(sourcePath); - } + if (combinedClasses.isEmpty() && combinedNonCodeResources.isEmpty()) { + return null; + } - for (String nonCodeResource : combinedNonCodeResources) { - String resourcePath = sourcePathResolver.getResourcePath(nonCodeResource); - if (resourcePath == null) { - log.debug( - "Skipping coverage reporting for {} because resource path could not be determined", - nonCodeResource); - metrics.add(CiVisibilityCountMetric.CODE_COVERAGE_ERRORS, 1, CoverageErrorType.PATH); - continue; - } - coveredPaths.add(resourcePath); + Set coveredPaths = set(combinedClasses.size() + combinedNonCodeResources.size()); + for (Class clazz : combinedClasses) { + Collection sourcePaths = sourcePathResolver.getSourcePaths(clazz); + if (sourcePaths.isEmpty()) { + log.debug( + "Skipping coverage reporting for {} because source path could not be determined", + clazz); + metrics.add(CiVisibilityCountMetric.CODE_COVERAGE_ERRORS, 1, CoverageErrorType.PATH); + continue; } + coveredPaths.addAll(sourcePaths); + } - List fileEntries = new ArrayList<>(coveredPaths.size()); - for (String path : coveredPaths) { - fileEntries.add(new TestReportFileEntry(path, null)); + for (String nonCodeResource : combinedNonCodeResources) { + Collection resourcePaths = sourcePathResolver.getResourcePaths(nonCodeResource); + if (resourcePaths.isEmpty()) { + log.debug( + "Skipping coverage reporting for {} because resource path could not be determined", + nonCodeResource); + metrics.add(CiVisibilityCountMetric.CODE_COVERAGE_ERRORS, 1, CoverageErrorType.PATH); + continue; } + coveredPaths.addAll(resourcePaths); + } - TestReport report = new TestReport(testSessionId, testSuiteId, testSpanId, fileEntries); - metrics.add( - CiVisibilityDistributionMetric.CODE_COVERAGE_FILES, - report.getTestReportFileEntries().size()); - return report; - - } catch (Exception e) { - metrics.add(CiVisibilityCountMetric.CODE_COVERAGE_ERRORS, 1); - throw e; + List fileEntries = new ArrayList<>(coveredPaths.size()); + for (String path : coveredPaths) { + fileEntries.add(new TestReportFileEntry(path, null)); } + + TestReport report = new TestReport(testSessionId, testSuiteId, testSpanId, fileEntries); + metrics.add( + CiVisibilityDistributionMetric.CODE_COVERAGE_FILES, + report.getTestReportFileEntries().size()); + return report; } private static Set set(int size) { diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/line/LineCoverageStore.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/line/LineCoverageStore.java index 651e0acd5dc..647f28ca181 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/line/LineCoverageStore.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/line/LineCoverageStore.java @@ -11,7 +11,6 @@ import datadog.trace.api.civisibility.telemetry.tag.CoverageErrorType; import datadog.trace.civisibility.coverage.ConcurrentCoverageStore; import datadog.trace.civisibility.source.SourcePathResolver; -import datadog.trace.civisibility.source.SourceResolutionException; import datadog.trace.civisibility.source.Utils; import java.io.InputStream; import java.util.ArrayList; @@ -24,9 +23,9 @@ import java.util.Map; import java.util.concurrent.ConcurrentHashMap; import java.util.function.Function; +import javax.annotation.Nullable; import org.jacoco.core.analysis.Analyzer; import org.jacoco.core.data.ExecutionDataStore; -import org.jetbrains.annotations.Nullable; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -53,88 +52,84 @@ private LineCoverageStore( @Nullable @Override protected TestReport report( - DDTraceId testSessionId, Long testSuiteId, long testSpanId, Collection probes) - throws SourceResolutionException { - try { - Map, ExecutionDataAdapter> combinedExecutionData = new IdentityHashMap<>(); - Collection combinedNonCodeResources = new HashSet<>(); - - for (LineProbes probe : probes) { - for (Map.Entry, ExecutionDataAdapter> e : probe.getExecutionData().entrySet()) { - combinedExecutionData.merge(e.getKey(), e.getValue(), ExecutionDataAdapter::merge); - } - combinedNonCodeResources.addAll(probe.getNonCodeResources()); - } + DDTraceId testSessionId, Long testSuiteId, long testSpanId, Collection probes) { + Map, ExecutionDataAdapter> combinedExecutionData = new IdentityHashMap<>(); + Collection combinedNonCodeResources = new HashSet<>(); - if (combinedExecutionData.isEmpty() && combinedNonCodeResources.isEmpty()) { - return null; + for (LineProbes probe : probes) { + for (Map.Entry, ExecutionDataAdapter> e : probe.getExecutionData().entrySet()) { + combinedExecutionData.merge(e.getKey(), e.getValue(), ExecutionDataAdapter::merge); } + combinedNonCodeResources.addAll(probe.getNonCodeResources()); + } - Map coveredLinesBySourcePath = new HashMap<>(); - for (Map.Entry, ExecutionDataAdapter> e : combinedExecutionData.entrySet()) { - ExecutionDataAdapter executionDataAdapter = e.getValue(); - String className = executionDataAdapter.getClassName(); - - Class clazz = e.getKey(); - String sourcePath = sourcePathResolver.getSourcePath(clazz); - if (sourcePath == null) { - log.debug( - "Skipping coverage reporting for {} because source path could not be determined", - className); - metrics.add(CiVisibilityCountMetric.CODE_COVERAGE_ERRORS, 1, CoverageErrorType.PATH); - continue; - } - - try (InputStream is = Utils.getClassStream(clazz)) { - BitSet coveredLines = - coveredLinesBySourcePath.computeIfAbsent(sourcePath, key -> new BitSet()); - ExecutionDataStore store = new ExecutionDataStore(); - store.put(executionDataAdapter.toExecutionData()); - - // TODO optimize this part to avoid parsing - // the same class multiple times for different test cases - Analyzer analyzer = new Analyzer(store, new SourceAnalyzer(coveredLines)); - analyzer.analyzeClass(is, null); - - } catch (Exception exception) { - log.debug( - "Skipping coverage reporting for {} ({}) because of error", - className, - sourcePath, - exception); - metrics.add(CiVisibilityCountMetric.CODE_COVERAGE_ERRORS, 1); - } - } + if (combinedExecutionData.isEmpty() && combinedNonCodeResources.isEmpty()) { + return null; + } - List fileEntries = new ArrayList<>(coveredLinesBySourcePath.size()); - for (Map.Entry e : coveredLinesBySourcePath.entrySet()) { - String sourcePath = e.getKey(); - BitSet coveredLines = e.getValue(); - fileEntries.add(new TestReportFileEntry(sourcePath, coveredLines)); + Map coveredLinesBySourcePath = new HashMap<>(); + for (Map.Entry, ExecutionDataAdapter> e : combinedExecutionData.entrySet()) { + ExecutionDataAdapter executionDataAdapter = e.getValue(); + String className = executionDataAdapter.getClassName(); + + Class clazz = e.getKey(); + Collection sourcePaths = sourcePathResolver.getSourcePaths(clazz); + if (sourcePaths.size() != 1) { + log.debug( + "Skipping coverage reporting for {} because source path could not be determined", + className); + metrics.add(CiVisibilityCountMetric.CODE_COVERAGE_ERRORS, 1, CoverageErrorType.PATH); + continue; } - - for (String nonCodeResource : combinedNonCodeResources) { - String resourcePath = sourcePathResolver.getResourcePath(nonCodeResource); - if (resourcePath == null) { - log.debug( - "Skipping coverage reporting for {} because resource path could not be determined", - nonCodeResource); - metrics.add(CiVisibilityCountMetric.CODE_COVERAGE_ERRORS, 1, CoverageErrorType.PATH); - continue; - } - fileEntries.add(new TestReportFileEntry(resourcePath, null)); + String sourcePath = sourcePaths.iterator().next(); + + try (InputStream is = Utils.getClassStream(clazz)) { + BitSet coveredLines = + coveredLinesBySourcePath.computeIfAbsent(sourcePath, key -> new BitSet()); + ExecutionDataStore store = new ExecutionDataStore(); + store.put(executionDataAdapter.toExecutionData()); + + // TODO optimize this part to avoid parsing + // the same class multiple times for different test cases + Analyzer analyzer = new Analyzer(store, new SourceAnalyzer(coveredLines)); + analyzer.analyzeClass(is, null); + + } catch (Exception exception) { + log.debug( + "Skipping coverage reporting for {} ({}) because of error", + className, + sourcePath, + exception); + metrics.add(CiVisibilityCountMetric.CODE_COVERAGE_ERRORS, 1); } + } - TestReport report = new TestReport(testSessionId, testSuiteId, testSpanId, fileEntries); - metrics.add( - CiVisibilityDistributionMetric.CODE_COVERAGE_FILES, - report.getTestReportFileEntries().size()); - return report; + List fileEntries = new ArrayList<>(coveredLinesBySourcePath.size()); + for (Map.Entry e : coveredLinesBySourcePath.entrySet()) { + String sourcePath = e.getKey(); + BitSet coveredLines = e.getValue(); + fileEntries.add(new TestReportFileEntry(sourcePath, coveredLines)); + } - } catch (Exception e) { - metrics.add(CiVisibilityCountMetric.CODE_COVERAGE_ERRORS, 1); - throw e; + for (String nonCodeResource : combinedNonCodeResources) { + Collection resourcePaths = sourcePathResolver.getResourcePaths(nonCodeResource); + if (resourcePaths.isEmpty()) { + log.debug( + "Skipping coverage reporting for {} because resource path could not be determined", + nonCodeResource); + metrics.add(CiVisibilityCountMetric.CODE_COVERAGE_ERRORS, 1, CoverageErrorType.PATH); + continue; + } + for (String resourcePath : resourcePaths) { + fileEntries.add(new TestReportFileEntry(resourcePath, null)); + } } + + TestReport report = new TestReport(testSessionId, testSuiteId, testSpanId, fileEntries); + metrics.add( + CiVisibilityDistributionMetric.CODE_COVERAGE_FILES, + report.getTestReportFileEntries().size()); + return report; } public static final class Factory implements CoverageStore.Factory { diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/percentage/CoverageCalculator.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/percentage/CoverageCalculator.java new file mode 100644 index 00000000000..c7fcaa3668d --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/percentage/CoverageCalculator.java @@ -0,0 +1,21 @@ +package datadog.trace.civisibility.coverage.percentage; + +import datadog.trace.api.civisibility.domain.BuildModuleLayout; +import datadog.trace.civisibility.config.ExecutionSettings; +import javax.annotation.Nullable; + +/** Calculates percentage of executable lines that are covered with tests. */ +public interface CoverageCalculator { + @Nullable + Long calculateCoveragePercentage(); + + interface Factory { + T sessionCoverage(long sessionId); + + T moduleCoverage( + long moduleId, + @Nullable BuildModuleLayout moduleLayout, + ExecutionSettings executionSettings, + T sessionCoverage); + } +} diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/report/JacocoCoverageProcessor.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/report/JacocoCoverageProcessor.java index db630801e84..0514dde7ab1 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/report/JacocoCoverageProcessor.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/report/JacocoCoverageProcessor.java @@ -9,7 +9,6 @@ import datadog.trace.civisibility.ipc.ModuleCoverageDataJacoco; import datadog.trace.civisibility.ipc.SignalResponse; import datadog.trace.civisibility.ipc.SignalType; -import datadog.trace.civisibility.source.SourceResolutionException; import datadog.trace.civisibility.source.index.RepoIndex; import datadog.trace.civisibility.source.index.RepoIndexProvider; import datadog.trace.util.Strings; @@ -355,19 +354,15 @@ private RepoIndexFileLocator(RepoIndex repoIndex, @Nonnull String repoRoot) { @Override protected InputStream getSourceStream(String path) throws IOException { - try { - String relativePath = repoIndex.getSourcePath(path); - if (relativePath == null) { - return null; - } - String absolutePath = - repoRoot + (!repoRoot.endsWith(File.separator) ? File.separator : "") + relativePath; - return new BufferedInputStream(Files.newInputStream(Paths.get(absolutePath))); - - } catch (SourceResolutionException e) { - LOGGER.debug("Could not resolve source for path {}", path, e); + Collection relativePaths = repoIndex.getSourcePaths(path); + if (relativePaths.size() != 1) { + LOGGER.debug("Could not resolve source for path {}", path); return null; } + String relativePath = relativePaths.iterator().next(); + String absolutePath = + repoRoot + (!repoRoot.endsWith(File.separator) ? File.separator : "") + relativePath; + return new BufferedInputStream(Files.newInputStream(Paths.get(absolutePath))); } } @@ -437,19 +432,16 @@ private long mergeAndUploadCoverageReport(IBundleCoverage coverageBundle) { String fileName = sourceFile.getName(); String pathRelativeToSourceRoot = (Strings.isNotBlank(packageName) ? packageName + "/" : "") + fileName; - String pathRelativeToIndexRoot; - try { - pathRelativeToIndexRoot = repoIndex.getSourcePath(pathRelativeToSourceRoot); - } catch (SourceResolutionException e) { - LOGGER.debug("Could not resolve source for path {}", pathRelativeToSourceRoot, e); - continue; - } + Collection pathsRelativeToIndexRoot = + repoIndex.getSourcePaths(pathRelativeToSourceRoot); - if (pathRelativeToIndexRoot == null) { + if (pathsRelativeToIndexRoot.size() != 1) { LOGGER.debug("Could not resolve source for path {}", pathRelativeToSourceRoot); continue; } + String pathRelativeToIndexRoot = pathsRelativeToIndexRoot.iterator().next(); + LinesCoverage linesCoverage = getLinesCoverage(sourceFile); // backendCoverageData contains data for all modules in the repo, // but coverageBundle bundle only has source files that are relevant for the given module, diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/report/NoOpCoverageProcessor.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/report/NoOpCoverageProcessor.java index 90929e33784..2dd5c4db3c3 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/report/NoOpCoverageProcessor.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/coverage/report/NoOpCoverageProcessor.java @@ -2,7 +2,7 @@ import datadog.trace.api.civisibility.domain.BuildModuleLayout; import datadog.trace.civisibility.config.ExecutionSettings; -import org.jetbrains.annotations.Nullable; +import javax.annotation.Nullable; public class NoOpCoverageProcessor implements CoverageProcessor { diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/TestImpl.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/TestImpl.java index 4bf97d50e6b..e1e87f67765 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/TestImpl.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/TestImpl.java @@ -40,10 +40,10 @@ import datadog.trace.bootstrap.instrumentation.api.TagContext; import datadog.trace.bootstrap.instrumentation.api.Tags; import datadog.trace.civisibility.codeowners.Codeowners; +import datadog.trace.civisibility.config.ConfigurationErrors; import datadog.trace.civisibility.decorator.TestDecorator; import datadog.trace.civisibility.source.LinesResolver; import datadog.trace.civisibility.source.SourcePathResolver; -import datadog.trace.civisibility.source.SourceResolutionException; import datadog.trace.civisibility.test.ExecutionResults; import java.lang.reflect.Method; import java.util.Collection; @@ -90,6 +90,7 @@ public TestImpl( Codeowners codeowners, CoverageStore.Factory coverageStoreFactory, ExecutionResults executionResults, + @Nonnull ConfigurationErrors configurationErrors, @Nonnull Collection capabilities, Consumer onSpanFinish) { this.instrumentation = instrumentation; @@ -157,6 +158,8 @@ public TestImpl( testDecorator.afterStart(span); + configurationErrors.applyTags(span); + metricCollector.add(CiVisibilityCountMetric.EVENT_CREATED, 1, instrumentation, EventType.TEST); if (instrumentationType == InstrumentationType.MANUAL_API) { @@ -175,17 +178,13 @@ private void populateSourceDataTags( return; } - String sourcePath; - try { - sourcePath = sourcePathResolver.getSourcePath(testClass); - if (sourcePath == null || sourcePath.isEmpty()) { - return; - } - } catch (SourceResolutionException e) { - log.debug("Could not populate source path for {}", testClass, e); + Collection sourcePaths = sourcePathResolver.getSourcePaths(testClass); + if (sourcePaths.size() != 1) { + log.debug("Could not populate source path for {}", testClass); return; } + String sourcePath = sourcePaths.iterator().next(); span.setTag(Tags.TEST_SOURCE_FILE, sourcePath); if (testMethod != null) { diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/TestSuiteImpl.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/TestSuiteImpl.java index c29fa175229..ed853e9576e 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/TestSuiteImpl.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/TestSuiteImpl.java @@ -19,10 +19,10 @@ import datadog.trace.bootstrap.instrumentation.api.InternalSpanTypes; import datadog.trace.bootstrap.instrumentation.api.Tags; import datadog.trace.civisibility.codeowners.Codeowners; +import datadog.trace.civisibility.config.ConfigurationErrors; import datadog.trace.civisibility.decorator.TestDecorator; import datadog.trace.civisibility.source.LinesResolver; import datadog.trace.civisibility.source.SourcePathResolver; -import datadog.trace.civisibility.source.SourceResolutionException; import datadog.trace.civisibility.test.ExecutionResults; import java.lang.reflect.Method; import java.util.Collection; @@ -53,6 +53,7 @@ public class TestSuiteImpl implements DDTestSuite { private final CoverageStore.Factory coverageStoreFactory; private final ExecutionResults executionResults; private final boolean parallelized; + private final ConfigurationErrors configurationErrors; private final Collection capabilities; private final Consumer onSpanFinish; private final SpanTagsPropagator tagsPropagator; @@ -75,6 +76,7 @@ public TestSuiteImpl( LinesResolver linesResolver, CoverageStore.Factory coverageStoreFactory, ExecutionResults executionResults, + @Nonnull ConfigurationErrors configurationErrors, @Nonnull Collection capabilities, Consumer onSpanFinish) { this.moduleSpanContext = moduleSpanContext; @@ -92,6 +94,7 @@ public TestSuiteImpl( this.linesResolver = linesResolver; this.coverageStoreFactory = coverageStoreFactory; this.executionResults = executionResults; + this.configurationErrors = configurationErrors; this.capabilities = capabilities; this.onSpanFinish = onSpanFinish; @@ -131,6 +134,8 @@ public TestSuiteImpl( testDecorator.afterStart(span); + configurationErrors.applyTags(span); + if (!parallelized) { activateSpanWithoutScope(span); } @@ -151,17 +156,13 @@ private void populateSourceDataTags( return; } - String sourcePath; - try { - sourcePath = sourcePathResolver.getSourcePath(testClass); - if (sourcePath == null || sourcePath.isEmpty()) { - return; - } - } catch (SourceResolutionException e) { - log.debug("Could not populate source path for {}", testClass, e); + Collection sourcePaths = sourcePathResolver.getSourcePaths(testClass); + if (sourcePaths.size() != 1) { + log.debug("Could not populate source path for {}", testClass); return; } + String sourcePath = sourcePaths.iterator().next(); span.setTag(Tags.TEST_SOURCE_FILE, sourcePath); LinesResolver.Lines testClassLines = linesResolver.getClassLines(testClass); @@ -264,6 +265,7 @@ public TestImpl testStart( codeowners, coverageStoreFactory, executionResults, + configurationErrors, capabilities, tagsPropagator::propagateStatus); } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/buildsystem/BuildSystemModuleImpl.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/buildsystem/BuildSystemModuleImpl.java index 049a22e580f..48915cdb479 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/buildsystem/BuildSystemModuleImpl.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/buildsystem/BuildSystemModuleImpl.java @@ -110,6 +110,8 @@ public BuildSystemModuleImpl( sessionSettings)); setTag(Tags.TEST_COMMAND, startCommand); + + executionSettings.getConfigurationErrors().applyTags(span); } @ParametersAreNonnullByDefault diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/buildsystem/BuildSystemSessionImpl.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/buildsystem/BuildSystemSessionImpl.java index 7b5361a8d0b..de116acf05c 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/buildsystem/BuildSystemSessionImpl.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/buildsystem/BuildSystemSessionImpl.java @@ -194,7 +194,13 @@ private void onModuleFinish(AgentSpan moduleSpan) { TagMergeSpec.of(Tags.TEST_ITR_TESTS_SKIPPING_COUNT, Long::sum), TagMergeSpec.of(DDTags.CI_ITR_TESTS_SKIPPED, Boolean::logicalOr), TagMergeSpec.of(Tags.TEST_TEST_MANAGEMENT_ENABLED, Boolean::logicalOr), - TagMergeSpec.of(DDTags.TEST_HAS_FAILED_TEST_REPLAY, Boolean::logicalOr)); + TagMergeSpec.of(DDTags.TEST_HAS_FAILED_TEST_REPLAY, Boolean::logicalOr), + TagMergeSpec.of(DDTags.CI_LIBRARY_CONFIGURATION_ERROR_SETTINGS, Boolean::logicalOr), + TagMergeSpec.of(DDTags.CI_LIBRARY_CONFIGURATION_ERROR_SKIPPABLE_TESTS, Boolean::logicalOr), + TagMergeSpec.of(DDTags.CI_LIBRARY_CONFIGURATION_ERROR_FLAKY_TESTS, Boolean::logicalOr), + TagMergeSpec.of(DDTags.CI_LIBRARY_CONFIGURATION_ERROR_KNOWN_TESTS, Boolean::logicalOr), + TagMergeSpec.of( + DDTags.CI_LIBRARY_CONFIGURATION_ERROR_TEST_MANAGEMENT_TESTS, Boolean::logicalOr)); } @Override diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/buildsystem/ProxyTestModule.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/buildsystem/ProxyTestModule.java index a101c16f18f..43c4e8748cb 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/buildsystem/ProxyTestModule.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/buildsystem/ProxyTestModule.java @@ -212,6 +212,7 @@ public TestSuiteImpl testSuiteStart( linesResolver, coverageStoreFactory, executionResults, + executionStrategy.getExecutionSettings().getConfigurationErrors(), capabilities, this::propagateTestFrameworkData); } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/headless/HeadlessTestModule.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/headless/HeadlessTestModule.java index d2409dd07eb..142c2abd46b 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/headless/HeadlessTestModule.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/headless/HeadlessTestModule.java @@ -124,6 +124,8 @@ public int executionPriority(@Nullable TestIdentifier test, @Nonnull TestSourceD @Override public void end(@Nullable Long endTime) { ExecutionSettings executionSettings = executionStrategy.getExecutionSettings(); + executionSettings.getConfigurationErrors().applyTags(span); + if (executionSettings.isCodeCoverageEnabled()) { setTag(Tags.TEST_CODE_COVERAGE_ENABLED, true); } @@ -185,6 +187,7 @@ public TestSuiteImpl testSuiteStart( linesResolver, coverageStoreFactory, executionResults, + executionStrategy.getExecutionSettings().getConfigurationErrors(), capabilities, tagsPropagator::propagateCiVisibilityTags); } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/headless/HeadlessTestSession.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/headless/HeadlessTestSession.java index 9e04a31ab27..1024104361c 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/headless/HeadlessTestSession.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/headless/HeadlessTestSession.java @@ -94,7 +94,13 @@ private void onModuleFinish(AgentSpan moduleSpan) { TagMergeSpec.of(Tags.TEST_EARLY_FLAKE_ABORT_REASON), TagMergeSpec.of(DDTags.CI_ITR_TESTS_SKIPPED), TagMergeSpec.of(Tags.TEST_TEST_MANAGEMENT_ENABLED), - TagMergeSpec.of(DDTags.TEST_HAS_FAILED_TEST_REPLAY)); + TagMergeSpec.of(DDTags.TEST_HAS_FAILED_TEST_REPLAY), + TagMergeSpec.of(DDTags.CI_LIBRARY_CONFIGURATION_ERROR_SETTINGS, Boolean::logicalOr), + TagMergeSpec.of(DDTags.CI_LIBRARY_CONFIGURATION_ERROR_SKIPPABLE_TESTS, Boolean::logicalOr), + TagMergeSpec.of(DDTags.CI_LIBRARY_CONFIGURATION_ERROR_FLAKY_TESTS, Boolean::logicalOr), + TagMergeSpec.of(DDTags.CI_LIBRARY_CONFIGURATION_ERROR_KNOWN_TESTS, Boolean::logicalOr), + TagMergeSpec.of( + DDTags.CI_LIBRARY_CONFIGURATION_ERROR_TEST_MANAGEMENT_TESTS, Boolean::logicalOr)); } @Override diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/manualapi/ManualApiTestModule.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/manualapi/ManualApiTestModule.java index 22dbe30994a..3ec4196401f 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/manualapi/ManualApiTestModule.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/manualapi/ManualApiTestModule.java @@ -7,7 +7,9 @@ import datadog.trace.api.civisibility.telemetry.tag.TestFrameworkInstrumentation; import datadog.trace.bootstrap.instrumentation.api.AgentSpan; import datadog.trace.bootstrap.instrumentation.api.AgentSpanContext; +import datadog.trace.bootstrap.instrumentation.api.Tags; import datadog.trace.civisibility.codeowners.Codeowners; +import datadog.trace.civisibility.config.ConfigurationErrors; import datadog.trace.civisibility.decorator.TestDecorator; import datadog.trace.civisibility.domain.AbstractTestModule; import datadog.trace.civisibility.domain.InstrumentationType; @@ -56,30 +58,37 @@ public ManualApiTestModule( } @Override - public TestSuiteImpl testSuiteStart( + public ManualApiTestSuite testSuiteStart( String testSuiteName, @Nullable Class testClass, @Nullable Long startTime, boolean parallelized) { - return new TestSuiteImpl( - span.context(), - moduleName, - testSuiteName, - null, - testClass, - startTime, - parallelized, - InstrumentationType.MANUAL_API, - TestFrameworkInstrumentation.OTHER, - config, - metricCollector, - testDecorator, - sourcePathResolver, - codeowners, - linesResolver, - coverageStoreFactory, - executionResults, - Collections.emptyList(), - tagsPropagator::propagateCiVisibilityTags); + TestSuiteImpl suite = + new TestSuiteImpl( + span.context(), + moduleName, + testSuiteName, + null, + testClass, + startTime, + parallelized, + InstrumentationType.MANUAL_API, + TestFrameworkInstrumentation.OTHER, // for metric purposes, framework is OTHER + config, + metricCollector, + testDecorator, + sourcePathResolver, + codeowners, + linesResolver, + coverageStoreFactory, + executionResults, + ConfigurationErrors.NONE, + Collections.emptyList(), + tagsPropagator::propagateCiVisibilityTags); + + String frameworkName = testDecorator.component().toString(); + suite.setTag(Tags.TEST_FRAMEWORK, frameworkName); + + return new ManualApiTestSuite(suite, frameworkName); } } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/manualapi/ManualApiTestSuite.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/manualapi/ManualApiTestSuite.java new file mode 100644 index 00000000000..312ce6c8cc9 --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/domain/manualapi/ManualApiTestSuite.java @@ -0,0 +1,51 @@ +package datadog.trace.civisibility.domain.manualapi; + +import datadog.trace.api.civisibility.DDTest; +import datadog.trace.api.civisibility.DDTestSuite; +import datadog.trace.bootstrap.instrumentation.api.Tags; +import datadog.trace.civisibility.domain.TestImpl; +import datadog.trace.civisibility.domain.TestSuiteImpl; +import java.lang.reflect.Method; +import javax.annotation.Nullable; + +/** + * Test suite that was created using manual API ({@link + * datadog.trace.api.civisibility.CIVisibility}). + */ +public class ManualApiTestSuite implements DDTestSuite { + + private final TestSuiteImpl delegate; + private final String frameworkName; + + public ManualApiTestSuite(TestSuiteImpl delegate, String frameworkName) { + this.delegate = delegate; + this.frameworkName = frameworkName; + } + + @Override + public void setTag(String key, Object value) { + delegate.setTag(key, value); + } + + @Override + public void setErrorInfo(Throwable error) { + delegate.setErrorInfo(error); + } + + @Override + public void setSkipReason(String skipReason) { + delegate.setSkipReason(skipReason); + } + + @Override + public void end(@Nullable Long endTime) { + delegate.end(endTime); + } + + @Override + public DDTest testStart(String testName, @Nullable Method testMethod, @Nullable Long startTime) { + TestImpl test = delegate.testStart(testName, testMethod, startTime); + test.setTag(Tags.TEST_FRAMEWORK, frameworkName); + return test; + } +} diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/events/NoOpTestEventsHandler.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/events/NoOpTestEventsHandler.java index 0660bff05f2..cc9a88830a6 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/events/NoOpTestEventsHandler.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/events/NoOpTestEventsHandler.java @@ -6,8 +6,8 @@ import datadog.trace.api.civisibility.config.TestIdentifier; import datadog.trace.api.civisibility.config.TestSourceData; import datadog.trace.api.civisibility.events.TestEventsHandler; -import datadog.trace.api.civisibility.execution.TestExecutionHistory; import datadog.trace.api.civisibility.execution.TestExecutionPolicy; +import datadog.trace.api.civisibility.execution.TestExecutionTracker; import datadog.trace.api.civisibility.telemetry.tag.SkipReason; import datadog.trace.api.civisibility.telemetry.tag.TestFrameworkInstrumentation; import datadog.trace.bootstrap.ContextStore; @@ -59,7 +59,7 @@ public void onTestStart( @Nullable Collection categories, @Nonnull TestSourceData testSourceData, @Nullable Long startTime, - @Nullable TestExecutionHistory testExecutionHistory) { + @Nullable TestExecutionTracker testExecutionTracker) { // do nothing } @@ -75,7 +75,7 @@ public void onTestFailure(TestKey descriptor, @Nullable Throwable throwable) { @Override public void onTestFinish( - TestKey descriptor, @Nullable Long endTime, @Nullable TestExecutionHistory executionHistory) { + TestKey descriptor, @Nullable Long endTime, @Nullable TestExecutionTracker executionTracker) { // do nothing } @@ -90,7 +90,7 @@ public void onTestIgnore( @Nullable Collection categories, @Nonnull TestSourceData testSourceData, @Nullable String reason, - @Nullable TestExecutionHistory testExecutionHistory) { + @Nullable TestExecutionTracker testExecutionTracker) { // do nothing } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/events/TestEventsHandlerImpl.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/events/TestEventsHandlerImpl.java index 5a0b4070d57..26b083f0c17 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/events/TestEventsHandlerImpl.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/events/TestEventsHandlerImpl.java @@ -8,8 +8,9 @@ import datadog.trace.api.civisibility.config.TestIdentifier; import datadog.trace.api.civisibility.config.TestSourceData; import datadog.trace.api.civisibility.events.TestEventsHandler; -import datadog.trace.api.civisibility.execution.TestExecutionHistory; +import datadog.trace.api.civisibility.execution.ExecutionAggregation; import datadog.trace.api.civisibility.execution.TestExecutionPolicy; +import datadog.trace.api.civisibility.execution.TestExecutionTracker; import datadog.trace.api.civisibility.execution.TestStatus; import datadog.trace.api.civisibility.telemetry.CiVisibilityCountMetric; import datadog.trace.api.civisibility.telemetry.CiVisibilityMetricCollector; @@ -142,7 +143,7 @@ public void onTestStart( final @Nullable Collection categories, final @Nonnull TestSourceData testSourceData, final @Nullable Long startTime, - final @Nullable TestExecutionHistory testExecutionHistory) { + final @Nullable TestExecutionTracker testExecutionTracker) { if (skipTrace(testSourceData.getTestClass())) { return; } @@ -180,8 +181,8 @@ public void onTestStart( test.setTag(Tags.TEST_TEST_MANAGEMENT_IS_ATTEMPT_TO_FIX, true); } - if (testExecutionHistory != null) { - test.getContext().set(TestExecutionHistory.class, testExecutionHistory); + if (testExecutionTracker != null) { + test.getContext().set(TestExecutionTracker.class, testExecutionTracker); } if (testFramework != null) { @@ -243,18 +244,18 @@ public void onTestFailure(TestKey descriptor, @Nullable Throwable throwable) { public void onTestFinish( TestKey descriptor, @Nullable Long endTime, - @Nullable TestExecutionHistory testExecutionHistory) { + @Nullable TestExecutionTracker testExecutionTracker) { TestImpl test = inProgressTests.remove(descriptor); if (test == null) { log.debug("Ignoring finish event, could not find test {}", descriptor); return; } - if (testExecutionHistory != null) { - TestStatus testStatus = test.getStatus(); - TestExecutionHistory.ExecutionOutcome outcome = - testExecutionHistory.registerExecution( - testStatus != null ? testStatus : TestStatus.skip, test.getDuration(endTime)); + TestStatus testStatus = test.getStatus() != null ? test.getStatus() : TestStatus.skip; + + if (testExecutionTracker != null) { + TestExecutionTracker.ExecutionOutcome outcome = + testExecutionTracker.registerExecution(testStatus, test.getDuration(endTime)); RetryReason retryReason = outcome.retryReason(); if (retryReason != null) { @@ -262,17 +263,25 @@ public void onTestFinish( test.setTag(Tags.TEST_RETRY_REASON, retryReason); } - if (outcome.failedAllRetries()) { - test.setTag(Tags.TEST_HAS_FAILED_ALL_RETRIES, true); + if (outcome.failureSuppressed()) { + test.setTag(Tags.TEST_FAILURE_SUPPRESSED, true); } - if (outcome.lastExecution() && testModule.isAttemptToFix(test.getIdentifier())) { - test.setTag(Tags.TEST_TEST_MANAGEMENT_ATTEMPT_TO_FIX_PASSED, outcome.succeededAllRetries()); - } + if (outcome.lastExecution()) { + test.setTag(Tags.TEST_FINAL_STATUS, outcome.finalStatus()); - if (outcome.failureSuppressed()) { - test.setTag(Tags.TEST_FAILURE_SUPPRESSED, true); + if (retryReason != null && outcome.aggregation() == ExecutionAggregation.ONLY_FAILED) { + test.setTag(Tags.TEST_HAS_FAILED_ALL_RETRIES, true); + } + + if (testModule.isAttemptToFix(test.getIdentifier())) { + test.setTag( + Tags.TEST_TEST_MANAGEMENT_ATTEMPT_TO_FIX_PASSED, + outcome.aggregation() == ExecutionAggregation.ONLY_PASSED); + } } + } else { + test.setTag(Tags.TEST_FINAL_STATUS, testStatus); } test.end(endTime); @@ -289,7 +298,7 @@ public void onTestIgnore( final @Nullable Collection categories, @Nonnull TestSourceData testSourceData, final @Nullable String reason, - @Nullable TestExecutionHistory testExecutionHistory) { + @Nullable TestExecutionTracker testExecutionTracker) { onTestStart( suiteDescriptor, testDescriptor, @@ -300,9 +309,9 @@ public void onTestIgnore( categories, testSourceData, null, - testExecutionHistory); + testExecutionTracker); onTestSkip(testDescriptor, reason); - onTestFinish(testDescriptor, null, testExecutionHistory); + onTestFinish(testDescriptor, null, testExecutionTracker); } @Override diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/AttemptToFix.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/AttemptToFix.java new file mode 100644 index 00000000000..37783399817 --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/AttemptToFix.java @@ -0,0 +1,76 @@ +package datadog.trace.civisibility.execution; + +import datadog.trace.api.civisibility.execution.ExecutionAggregation; +import datadog.trace.api.civisibility.execution.TestExecutionPolicy; +import datadog.trace.api.civisibility.execution.TestStatus; +import datadog.trace.api.civisibility.telemetry.tag.RetryReason; + +/** + * Execution policy for the Attempt to Fix feature. Runs a test case up to N times. Stops retrying + * as soon as a failure is observed, since a single failure proves the fix did not work. + */ +public class AttemptToFix implements TestExecutionPolicy { + + private final int maxExecutions; + private final boolean suppressFailures; + private int executions; + private ExecutionAggregation results; + private TestStatus lastStatus; + + public AttemptToFix(int maxExecutions, boolean suppressFailures) { + this.maxExecutions = maxExecutions; + this.suppressFailures = suppressFailures; + this.executions = 0; + this.results = ExecutionAggregation.NONE; + } + + @Override + public ExecutionOutcome registerExecution(TestStatus status, long durationMillis) { + lastStatus = status; + ++executions; + results = results.withExecution(status); + + boolean lastExecution = !retriesLeft(); + boolean retry = executions > 1; + boolean failureSuppressed = status == TestStatus.fail && suppressFailures; + TestStatus finalStatus = null; + if (lastExecution) { + if (results == ExecutionAggregation.ONLY_PASSED || suppressFailures) { + finalStatus = TestStatus.pass; + } else { + finalStatus = TestStatus.fail; + } + } + + return new ExecutionOutcomeImpl( + failureSuppressed, + lastExecution, + results, + retry ? RetryReason.attemptToFix : null, + finalStatus); + } + + private boolean retriesLeft() { + // stop retrying if the test was skipped, max executions reached, + // or a failure was observed (the fix didn't work) + return lastStatus != TestStatus.skip + && executions < maxExecutions + && results != ExecutionAggregation.ONLY_FAILED + && results != ExecutionAggregation.MIXED; + } + + @Override + public boolean applicable() { + return retriesLeft(); + } + + @Override + public boolean suppressFailures() { + return suppressFailures; + } + + @Override + public boolean failedTestReplayApplicable() { + return false; + } +} diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/AutoTestRetry.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/AutoTestRetry.java new file mode 100644 index 00000000000..9dabc20c7c3 --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/AutoTestRetry.java @@ -0,0 +1,79 @@ +package datadog.trace.civisibility.execution; + +import datadog.trace.api.civisibility.execution.ExecutionAggregation; +import datadog.trace.api.civisibility.execution.TestExecutionPolicy; +import datadog.trace.api.civisibility.execution.TestStatus; +import datadog.trace.api.civisibility.telemetry.tag.RetryReason; +import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; +import java.util.concurrent.atomic.AtomicInteger; + +/** + * Execution policy for Auto Test Retries (ATR). Retries a test case if it failed, up to a maximum + * number of times. Stops retrying as soon as the test passes. + */ +@SuppressFBWarnings( + value = {"AT_NONATOMIC_OPERATIONS_ON_SHARED_VARIABLE"}, + justification = + "TestExecutionPolicy instances are confined to a single thread and are not meant to be thread-safe") +public class AutoTestRetry implements TestExecutionPolicy { + + private final int maxExecutions; + private final boolean suppressFailures; + private int executions; + private ExecutionAggregation results; + + /** Total retry counter that is shared by all auto test retry policies */ + private final AtomicInteger totalRetryCount; + + public AutoTestRetry(int maxExecutions, boolean suppressFailures, AtomicInteger totalRetryCount) { + this.maxExecutions = maxExecutions; + this.suppressFailures = suppressFailures; + this.totalRetryCount = totalRetryCount; + this.executions = 0; + this.results = ExecutionAggregation.NONE; + } + + @Override + public ExecutionOutcome registerExecution(TestStatus status, long durationMillis) { + ++executions; + results = results.withExecution(status); + if (executions > 1) { + totalRetryCount.incrementAndGet(); + } + + boolean lastExecution = !retriesLeft(); + boolean retry = executions > 1; // first execution is not a retry + boolean failureSuppressed = status == TestStatus.fail && (!lastExecution || suppressFailures); + TestStatus finalStatus = null; + if (lastExecution) { + // final status is always the last status reported (or pass if a failure is suppressed) + finalStatus = failureSuppressed ? TestStatus.pass : status; + } + + return new ExecutionOutcomeImpl( + failureSuppressed, lastExecution, results, retry ? RetryReason.atr : null, finalStatus); + } + + private boolean retriesLeft() { + return executions < maxExecutions + && results != ExecutionAggregation.ONLY_PASSED + && results != ExecutionAggregation.MIXED; + } + + @Override + public boolean applicable() { + return retriesLeft(); + } + + @Override + public boolean suppressFailures() { + // do not suppress failures for last execution (unless flag to suppress all failures is set); + // the +1 is because this method is called _before_ subsequent execution is registered + return executions + 1 < maxExecutions || suppressFailures; + } + + @Override + public boolean failedTestReplayApplicable() { + return true; + } +} diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/EarlyFlakeDetection.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/EarlyFlakeDetection.java new file mode 100644 index 00000000000..9e0819e0ca0 --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/EarlyFlakeDetection.java @@ -0,0 +1,96 @@ +package datadog.trace.civisibility.execution; + +import datadog.trace.api.civisibility.execution.ExecutionAggregation; +import datadog.trace.api.civisibility.execution.TestExecutionPolicy; +import datadog.trace.api.civisibility.execution.TestStatus; +import datadog.trace.api.civisibility.telemetry.tag.RetryReason; +import datadog.trace.civisibility.config.ExecutionsByDuration; +import java.util.List; + +/** + * Execution policy for Early Flake Detection. Runs a new or modified test case multiple times to + * determine if it is flaky. The number of executions depends on test duration. Stops retrying once + * flakiness is detected (mixed pass/fail results). + */ +public class EarlyFlakeDetection implements TestExecutionPolicy { + + private final boolean suppressFailures; + private final List executionsByDuration; + private int executions; + private int maxExecutions; + private ExecutionAggregation results; + private TestStatus lastStatus; + + public EarlyFlakeDetection( + List executionsByDuration, boolean suppressFailures) { + this.suppressFailures = suppressFailures; + this.executionsByDuration = executionsByDuration; + this.executions = 0; + this.maxExecutions = getExecutions(0); + this.results = ExecutionAggregation.NONE; + } + + @Override + public ExecutionOutcome registerExecution(TestStatus status, long durationMillis) { + lastStatus = status; + ++executions; + results = results.withExecution(status); + int maxExecutionsForGivenDuration = getExecutions(durationMillis); + maxExecutions = Math.min(maxExecutions, maxExecutionsForGivenDuration); + + boolean lastExecution = !retriesLeft(); + boolean retry = executions > 1; + boolean failureSuppressed = status == TestStatus.fail && suppressFailures; + TestStatus finalStatus = null; + if (lastExecution) { + if (results == ExecutionAggregation.ONLY_PASSED || suppressFailures) { + finalStatus = TestStatus.pass; + } else { + finalStatus = TestStatus.fail; + } + } + + return new ExecutionOutcomeImpl( + failureSuppressed, lastExecution, results, retry ? RetryReason.efd : null, finalStatus); + } + + private boolean retriesLeft() { + // skipped tests should not be retried and stop once flakiness is detected + return lastStatus != TestStatus.skip + && executions < maxExecutions + && results != ExecutionAggregation.MIXED; + } + + @Override + public boolean applicable() { + return retriesLeft(); + } + + @Override + public boolean suppressFailures() { + return suppressFailures; + } + + @Override + public boolean propagateFailure() { + // used to bypass TestNG's RetryAnalyzer, which made the framework session status depend on + // result order: + // pass + fail -> fail (correct) + // fail + pass -> pass (incorrect) + return !suppressFailures && results == ExecutionAggregation.MIXED; + } + + private int getExecutions(long durationMillis) { + for (ExecutionsByDuration e : executionsByDuration) { + if (durationMillis <= e.getDurationMillis()) { + return e.getExecutions(); + } + } + return 0; + } + + @Override + public boolean failedTestReplayApplicable() { + return false; + } +} diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/ExecutionOutcomeImpl.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/ExecutionOutcomeImpl.java index 47b0fe76722..a599e9aab43 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/ExecutionOutcomeImpl.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/ExecutionOutcomeImpl.java @@ -1,28 +1,30 @@ package datadog.trace.civisibility.execution; -import datadog.trace.api.civisibility.execution.TestExecutionHistory; +import datadog.trace.api.civisibility.execution.ExecutionAggregation; +import datadog.trace.api.civisibility.execution.TestExecutionTracker; +import datadog.trace.api.civisibility.execution.TestStatus; import datadog.trace.api.civisibility.telemetry.tag.RetryReason; -import org.jetbrains.annotations.Nullable; +import javax.annotation.Nullable; -class ExecutionOutcomeImpl implements TestExecutionHistory.ExecutionOutcome { +class ExecutionOutcomeImpl implements TestExecutionTracker.ExecutionOutcome { private final boolean failureSuppressed; private final boolean lastExecution; - private final boolean failedAllRetries; - private final boolean succeededAllRetries; + private final ExecutionAggregation aggregation; private final RetryReason retryReason; + private final TestStatus finalStatus; ExecutionOutcomeImpl( boolean failureSuppressed, boolean lastExecution, - boolean failedAllRetries, - boolean succeededAllRetries, - RetryReason retryReason) { + ExecutionAggregation aggregation, + RetryReason retryReason, + TestStatus finalStatus) { this.failureSuppressed = failureSuppressed; this.lastExecution = lastExecution; - this.failedAllRetries = failedAllRetries; - this.succeededAllRetries = succeededAllRetries; + this.aggregation = aggregation; this.retryReason = retryReason; + this.finalStatus = finalStatus; } @Override @@ -36,18 +38,19 @@ public boolean lastExecution() { } @Override - public boolean failedAllRetries() { - return failedAllRetries; + public ExecutionAggregation aggregation() { + return aggregation; } + @Nullable @Override - public boolean succeededAllRetries() { - return succeededAllRetries; + public RetryReason retryReason() { + return retryReason; } @Nullable @Override - public RetryReason retryReason() { - return retryReason; + public TestStatus finalStatus() { + return finalStatus; } } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/Quarantine.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/Quarantine.java new file mode 100644 index 00000000000..f6616c8b576 --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/Quarantine.java @@ -0,0 +1,40 @@ +package datadog.trace.civisibility.execution; + +import datadog.trace.api.civisibility.execution.ExecutionAggregation; +import datadog.trace.api.civisibility.execution.TestExecutionPolicy; +import datadog.trace.api.civisibility.execution.TestStatus; + +/** + * Execution policy for quarantined tests. Runs a test case once. If it fails, suppresses the + * failure so that the build status is not affected. + */ +public class Quarantine implements TestExecutionPolicy { + + private boolean testExecuted; + + @Override + public ExecutionOutcome registerExecution(TestStatus status, long durationMillis) { + testExecuted = true; + return new ExecutionOutcomeImpl( + status == TestStatus.fail, + testExecuted, + ExecutionAggregation.NONE.withExecution(status), + null, + status == TestStatus.fail ? TestStatus.pass : status); + } + + @Override + public boolean applicable() { + return !testExecuted; + } + + @Override + public boolean suppressFailures() { + return true; + } + + @Override + public boolean failedTestReplayApplicable() { + return false; + } +} diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/Regular.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/Regular.java index b0c53f09958..56e2c9e2e4d 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/Regular.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/Regular.java @@ -1,10 +1,8 @@ package datadog.trace.civisibility.execution; -import datadog.trace.api.civisibility.execution.TestExecutionHistory; +import datadog.trace.api.civisibility.execution.ExecutionAggregation; import datadog.trace.api.civisibility.execution.TestExecutionPolicy; import datadog.trace.api.civisibility.execution.TestStatus; -import datadog.trace.api.civisibility.telemetry.tag.RetryReason; -import javax.annotation.Nullable; /** Regular test case execution with no alterations. */ public class Regular implements TestExecutionPolicy { @@ -15,7 +13,8 @@ private Regular() {} @Override public ExecutionOutcome registerExecution(TestStatus status, long durationMillis) { - return RegularExecutionOutcome.INSTANCE; + return new ExecutionOutcomeImpl( + false, true, ExecutionAggregation.NONE.withExecution(status), null, status); } @Override @@ -28,40 +27,6 @@ public boolean suppressFailures() { return false; } - private static final class RegularExecutionOutcome - implements TestExecutionHistory.ExecutionOutcome { - - static final TestExecutionHistory.ExecutionOutcome INSTANCE = new RegularExecutionOutcome(); - - private RegularExecutionOutcome() {} - - @Override - public boolean failureSuppressed() { - return false; - } - - @Override - public boolean lastExecution() { - return false; - } - - @Override - public boolean failedAllRetries() { - return false; - } - - @Override - public boolean succeededAllRetries() { - return false; - } - - @Nullable - @Override - public RetryReason retryReason() { - return null; - } - } - @Override public boolean failedTestReplayApplicable() { return false; diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/RetryUntilSuccessful.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/RetryUntilSuccessful.java deleted file mode 100644 index e26fef213f0..00000000000 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/RetryUntilSuccessful.java +++ /dev/null @@ -1,67 +0,0 @@ -package datadog.trace.civisibility.execution; - -import datadog.trace.api.civisibility.execution.TestExecutionPolicy; -import datadog.trace.api.civisibility.execution.TestStatus; -import datadog.trace.api.civisibility.telemetry.tag.RetryReason; -import java.util.concurrent.atomic.AtomicInteger; - -/** Retries a test case if it failed, up to a maximum number of times. */ -public class RetryUntilSuccessful implements TestExecutionPolicy { - - private final int maxExecutions; - private final boolean suppressFailures; - private int executions; - private boolean successfulExecutionSeen; - - /** Total retry counter that is shared by all retry until successful policies (currently ATR) */ - private final AtomicInteger totalRetryCount; - - public RetryUntilSuccessful( - int maxExecutions, boolean suppressFailures, AtomicInteger totalRetryCount) { - this.maxExecutions = maxExecutions; - this.suppressFailures = suppressFailures; - this.totalRetryCount = totalRetryCount; - this.executions = 0; - } - - @Override - public ExecutionOutcome registerExecution(TestStatus status, long durationMillis) { - ++executions; - successfulExecutionSeen |= (status != TestStatus.fail); - if (executions > 1) { - totalRetryCount.incrementAndGet(); - } - - boolean lastExecution = !retriesLeft(); - boolean retry = executions > 1; // first execution is not a retry - return new ExecutionOutcomeImpl( - status == TestStatus.fail && (!lastExecution || suppressFailures), - lastExecution, - lastExecution && !successfulExecutionSeen, - false, - retry ? RetryReason.atr : null); - } - - private boolean retriesLeft() { - return !successfulExecutionSeen && executions < maxExecutions; - } - - @Override - public boolean applicable() { - // executions must always be registered, therefore consider it applicable as long as there are - // retries left - return retriesLeft(); - } - - @Override - public boolean suppressFailures() { - // do not suppress failures for last execution (unless flag to suppress all failures is set); - // the +1 is because this method is called _before_ subsequent execution is registered - return executions + 1 < maxExecutions || suppressFailures; - } - - @Override - public boolean failedTestReplayApplicable() { - return true; - } -} diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/RunNTimes.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/RunNTimes.java deleted file mode 100644 index a303fecd6f2..00000000000 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/RunNTimes.java +++ /dev/null @@ -1,82 +0,0 @@ -package datadog.trace.civisibility.execution; - -import datadog.trace.api.civisibility.execution.TestExecutionPolicy; -import datadog.trace.api.civisibility.execution.TestStatus; -import datadog.trace.api.civisibility.telemetry.tag.RetryReason; -import datadog.trace.civisibility.config.ExecutionsByDuration; -import java.util.List; - -/** Runs a test case N times (N depends on test duration) regardless of success or failure. */ -public class RunNTimes implements TestExecutionPolicy { - - private final boolean suppressFailures; - private final List executionsByDuration; - private int executions; - private int maxExecutions; - private int successfulExecutionsSeen; - private final RetryReason retryReason; - private TestStatus lastStatus; - - public RunNTimes( - List executionsByDuration, - boolean suppressFailures, - RetryReason retryReason) { - this.suppressFailures = suppressFailures; - this.executionsByDuration = executionsByDuration; - this.executions = 0; - this.maxExecutions = getExecutions(0); - this.successfulExecutionsSeen = 0; - this.retryReason = retryReason; - } - - @Override - public ExecutionOutcome registerExecution(TestStatus status, long durationMillis) { - lastStatus = status; - ++executions; - if (status != TestStatus.fail) { - ++successfulExecutionsSeen; - } - int maxExecutionsForGivenDuration = getExecutions(durationMillis); - maxExecutions = Math.min(maxExecutions, maxExecutionsForGivenDuration); - - boolean lastExecution = !retriesLeft(); - boolean retry = executions > 1; // first execution is not a retry - return new ExecutionOutcomeImpl( - status == TestStatus.fail && suppressFailures(), - lastExecution, - lastExecution && successfulExecutionsSeen == 0, - lastExecution && successfulExecutionsSeen == executions, - retry ? retryReason : null); - } - - private boolean retriesLeft() { - // skipped tests (either by the framework or DD) should not be retried - return lastStatus != TestStatus.skip && executions < maxExecutions; - } - - @Override - public boolean applicable() { - // executions must always be registered, therefore consider it applicable as long as there are - // retries left - return retriesLeft(); - } - - @Override - public boolean suppressFailures() { - return suppressFailures; - } - - private int getExecutions(long durationMillis) { - for (ExecutionsByDuration e : executionsByDuration) { - if (durationMillis <= e.getDurationMillis()) { - return e.getExecutions(); - } - } - return 0; - } - - @Override - public boolean failedTestReplayApplicable() { - return false; - } -} diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/RunOnceIgnoreOutcome.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/RunOnceIgnoreOutcome.java deleted file mode 100644 index a41ca781ec2..00000000000 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/execution/RunOnceIgnoreOutcome.java +++ /dev/null @@ -1,34 +0,0 @@ -package datadog.trace.civisibility.execution; - -import datadog.trace.api.civisibility.execution.TestExecutionPolicy; -import datadog.trace.api.civisibility.execution.TestStatus; - -/** - * Runs a test case once. If it fails - suppresses the failure so that the build status is not - * affected. - */ -public class RunOnceIgnoreOutcome implements TestExecutionPolicy { - - private boolean testExecuted; - - @Override - public ExecutionOutcome registerExecution(TestStatus status, long durationMillis) { - testExecuted = true; - return new ExecutionOutcomeImpl(status == TestStatus.fail, testExecuted, false, false, null); - } - - @Override - public boolean applicable() { - return !testExecuted; - } - - @Override - public boolean suppressFailures() { - return true; - } - - @Override - public boolean failedTestReplayApplicable() { - return false; - } -} diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/git/tree/GitDiffParser.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/git/tree/GitDiffParser.java index 53ba6cbae29..374e3a75f6b 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/git/tree/GitDiffParser.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/git/tree/GitDiffParser.java @@ -16,7 +16,7 @@ public class GitDiffParser { private static final Pattern CHANGED_FILE_PATTERN = - Pattern.compile("^diff --git a/(?.+) b/(?.+)$"); + Pattern.compile("^diff --git (?.+) (?.+)$"); private static final Pattern CHANGED_LINES_PATTERN = Pattern.compile("^@@ -\\d+(,\\d+)? \\+(?\\d+)(,(?\\d+))? @@"); diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/git/tree/ShellGitClient.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/git/tree/ShellGitClient.java index f353b05c0b9..8f63a088b28 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/git/tree/ShellGitClient.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/git/tree/ShellGitClient.java @@ -52,6 +52,7 @@ public class ShellGitClient implements GitClient { private final String latestCommitsSince; private final int latestCommitsLimit; private final ShellCommandExecutor commandExecutor; + private final String safeDirectoryOption; /** * Creates a new git client @@ -71,10 +72,50 @@ public class ShellGitClient implements GitClient { int latestCommitsLimit, long timeoutMillis) { this.metricCollector = metricCollector; - this.repoRoot = repoRoot; this.latestCommitsSince = latestCommitsSince; this.latestCommitsLimit = latestCommitsLimit; - commandExecutor = new ShellCommandExecutor(new File(repoRoot), timeoutMillis); + + this.repoRoot = findGitRepositoryRoot(new File(repoRoot)); + this.safeDirectoryOption = "safe.directory=" + this.repoRoot; + commandExecutor = new ShellCommandExecutor(new File(this.repoRoot), timeoutMillis); + } + + /** + * Finds the Git repository root by traversing upward from the given directory looking for a .git + * directory or file (for worktrees). + * + * @param startDir The directory to start searching from + * @return The absolute path to the repository root, or the original path if no .git is found + */ + static String findGitRepositoryRoot(File startDir) { + File current = startDir.getAbsoluteFile(); + LOGGER.debug("Finding git repository root for {}", current.getPath()); + while (current != null) { + File gitDir = new File(current, ".git"); + if (gitDir.exists()) { + String repoRootFound = current.getPath(); + LOGGER.debug("Git repository root found as {}", repoRootFound); + return repoRootFound; + } + current = current.getParentFile(); + } + LOGGER.debug("No .git found for repository root, defaulting to original starting directory"); + return startDir.getAbsolutePath(); + } + + /** + * Builds a git command with the {@code safe.directory} option. + * + * @param gitArgs The git command arguments (everything after "git") + * @return The complete command array including "git", "-c", "safe.directory=...", and the args + */ + private String[] buildGitCommand(String... gitArgs) { + String[] command = new String[gitArgs.length + 3]; + command[0] = "git"; + command[1] = "-c"; + command[2] = safeDirectoryOption; + System.arraycopy(gitArgs, 0, command, 3, gitArgs.length); + return command; } /** @@ -93,7 +134,8 @@ public boolean isShallow() throws IOException, TimeoutException, InterruptedExce () -> { String output = commandExecutor - .executeCommand(IOUtils::readFully, "git", "rev-parse", "--is-shallow-repository") + .executeCommand( + IOUtils::readFully, buildGitCommand("rev-parse", "--is-shallow-repository")) .trim(); return Boolean.parseBoolean(output); }); @@ -119,7 +161,7 @@ public String getUpstreamBranchSha() throws IOException, TimeoutException, Inter Command.OTHER, () -> commandExecutor - .executeCommand(IOUtils::readFully, "git", "rev-parse", "@{upstream}") + .executeCommand(IOUtils::readFully, buildGitCommand("rev-parse", "@{upstream}")) .trim()); } @@ -147,24 +189,24 @@ public void unshallow(@Nullable String remoteCommitReference) String commitSha = getSha(remoteCommitReference); commandExecutor.executeCommand( ShellCommandExecutor.OutputParser.IGNORE, - "git", - "fetch", - "--update-shallow", - "--filter=blob:none", - "--recurse-submodules=no", - String.format("--shallow-since='%s'", latestCommitsSince), - remote, - commitSha); + buildGitCommand( + "fetch", + "--update-shallow", + "--filter=blob:none", + "--recurse-submodules=no", + String.format("--shallow-since='%s'", latestCommitsSince), + remote, + commitSha)); } else { commandExecutor.executeCommand( ShellCommandExecutor.OutputParser.IGNORE, - "git", - "fetch", - "--update-shallow", - "--filter=blob:none", - "--recurse-submodules=no", - String.format("--shallow-since='%s'", latestCommitsSince), - remote); + buildGitCommand( + "fetch", + "--update-shallow", + "--filter=blob:none", + "--recurse-submodules=no", + String.format("--shallow-since='%s'", latestCommitsSince), + remote)); } return (Void) null; @@ -187,7 +229,8 @@ public String getGitFolder() throws IOException, TimeoutException, InterruptedEx Command.OTHER, () -> commandExecutor - .executeCommand(IOUtils::readFully, "git", "rev-parse", "--absolute-git-dir") + .executeCommand( + IOUtils::readFully, buildGitCommand("rev-parse", "--absolute-git-dir")) .trim()); } @@ -207,7 +250,7 @@ public String getRepoRoot() throws IOException, TimeoutException, InterruptedExc Command.OTHER, () -> commandExecutor - .executeCommand(IOUtils::readFully, "git", "rev-parse", "--show-toplevel") + .executeCommand(IOUtils::readFully, buildGitCommand("rev-parse", "--show-toplevel")) .trim()); } @@ -233,7 +276,8 @@ public String getRemoteUrl(String remoteName) () -> commandExecutor .executeCommand( - IOUtils::readFully, "git", "config", "--get", "remote." + remoteName + ".url") + IOUtils::readFully, + buildGitCommand("config", "--get", "remote." + remoteName + ".url")) .trim()); } @@ -253,7 +297,7 @@ public String getCurrentBranch() throws IOException, TimeoutException, Interrupt Command.GET_BRANCH, () -> commandExecutor - .executeCommand(IOUtils::readFully, "git", "branch", "--show-current") + .executeCommand(IOUtils::readFully, buildGitCommand("branch", "--show-current")) .trim()); } @@ -279,7 +323,7 @@ public List getTags(String commit) () -> { try { return commandExecutor.executeCommand( - IOUtils::readLines, "git", "describe", "--tags", "--exact-match", commit); + IOUtils::readLines, buildGitCommand("describe", "--tags", "--exact-match", commit)); } catch (ShellCommandExecutor.ShellCommandFailedException e) { // if provided commit is not tagged, // command will fail because "--exact-match" is specified @@ -309,7 +353,7 @@ public String getSha(String reference) Command.OTHER, () -> commandExecutor - .executeCommand(IOUtils::readFully, "git", "rev-parse", reference) + .executeCommand(IOUtils::readFully, buildGitCommand("rev-parse", reference)) .trim()); } @@ -325,10 +369,7 @@ private boolean isCommitPresent(String commitReference) try { commandExecutor.executeCommand( ShellCommandExecutor.OutputParser.IGNORE, - "git", - "cat-file", - "-e", - commitReference + "^{commit}"); + buildGitCommand("cat-file", "-e", commitReference + "^{commit}")); return true; } catch (ShellCommandExecutor.ShellCommandFailedException ignored) { return false; @@ -348,13 +389,13 @@ private void fetchCommit(String remoteCommitReference) String remote = getRemoteName(); commandExecutor.executeCommand( ShellCommandExecutor.OutputParser.IGNORE, - "git", - "fetch", - "--filter=blob:none", - "--recurse-submodules=no", - "--no-write-fetch-head", - remote, - remoteCommitReference); + buildGitCommand( + "fetch", + "--filter=blob:none", + "--recurse-submodules=no", + "--no-write-fetch-head", + remote, + remoteCommitReference)); return (Void) null; }); @@ -393,11 +434,11 @@ public CommitInfo getCommitInfo(String commit, boolean fetchIfNotPresent) commandExecutor .executeCommand( IOUtils::readFully, - "git", - "show", - commit, - "-s", - "--format=%H\",\"%an\",\"%ae\",\"%aI\",\"%cn\",\"%ce\",\"%cI\",\"%B") + buildGitCommand( + "show", + commit, + "-s", + "--format=%H\",\"%an\",\"%ae\",\"%aI\",\"%cn\",\"%ce\",\"%cI\",\"%B")) .trim(); } catch (ShellCommandExecutor.ShellCommandFailedException e) { LOGGER.error("Failed to fetch commit info", e); @@ -436,12 +477,12 @@ public List getLatestCommits() () -> commandExecutor.executeCommand( IOUtils::readLines, - "git", - "log", - "--format=%H", - "-n", - String.valueOf(latestCommitsLimit), - String.format("--since='%s'", latestCommitsSince))); + buildGitCommand( + "log", + "--format=%H", + "-n", + String.valueOf(latestCommitsLimit), + String.format("--since='%s'", latestCommitsSince)))); } /** @@ -464,23 +505,22 @@ public List getObjects( return executeCommand( Command.GET_OBJECTS, () -> { - String[] command = new String[6 + commitsToSkip.size() + commitsToInclude.size()]; - command[0] = "git"; - command[1] = "rev-list"; - command[2] = "--objects"; - command[3] = "--no-object-names"; - command[4] = "--filter=blob:none"; - command[5] = String.format("--since='%s'", latestCommitsSince); - - int count = 6; + String[] gitArgs = new String[5 + commitsToSkip.size() + commitsToInclude.size()]; + gitArgs[0] = "rev-list"; + gitArgs[1] = "--objects"; + gitArgs[2] = "--no-object-names"; + gitArgs[3] = "--filter=blob:none"; + gitArgs[4] = String.format("--since='%s'", latestCommitsSince); + + int count = 5; for (String commitToSkip : commitsToSkip) { - command[count++] = "^" + commitToSkip; + gitArgs[count++] = "^" + commitToSkip; } for (String commitToInclude : commitsToInclude) { - command[count++] = commitToInclude; + gitArgs[count++] = commitToInclude; } - return commandExecutor.executeCommand(IOUtils::readLines, command); + return commandExecutor.executeCommand(IOUtils::readLines, buildGitCommand(gitArgs)); }); } @@ -510,11 +550,7 @@ public Path createPackFiles(List objectHashes) commandExecutor.executeCommand( ShellCommandExecutor.OutputParser.IGNORE, input, - "git", - "pack-objects", - "--compression=9", - "--max-pack-size=3m", - path); + buildGitCommand("pack-objects", "--compression=9", "--max-pack-size=3m", path)); return tempDirectory; }); } @@ -587,11 +623,8 @@ String getRemoteName() throws IOException, InterruptedException, TimeoutExceptio commandExecutor .executeCommand( IOUtils::readFully, - "git", - "rev-parse", - "--abbrev-ref", - "--symbolic-full-name", - "@{upstream}") + buildGitCommand( + "rev-parse", "--abbrev-ref", "--symbolic-full-name", "@{upstream}")) .trim(); int slashIdx = remote.indexOf('/'); @@ -602,7 +635,8 @@ String getRemoteName() throws IOException, InterruptedException, TimeoutExceptio // fallback to first remote if no upstream try { - List remotes = commandExecutor.executeCommand(IOUtils::readLines, "git", "remote"); + List remotes = + commandExecutor.executeCommand(IOUtils::readLines, buildGitCommand("remote")); return remotes.get(0); } catch (ShellCommandExecutor.ShellCommandFailedException e) { LOGGER.debug("Error getting remotes", e); @@ -658,11 +692,11 @@ void tryFetchingIfNotFoundLocally(String branch, String remoteName) // check if branch exists locally as a remote ref commandExecutor.executeCommand( ShellCommandExecutor.OutputParser.IGNORE, - "git", - "show-ref", - "--verify", - "--quiet", - "refs/remotes/" + remoteName + "/" + shortBranchName); + buildGitCommand( + "show-ref", + "--verify", + "--quiet", + "refs/remotes/" + remoteName + "/" + shortBranchName)); LOGGER.debug("Branch {}/{} exists locally, skipping fetch", remoteName, shortBranchName); return; } catch (ShellCommandExecutor.ShellCommandFailedException e) { @@ -676,7 +710,8 @@ void tryFetchingIfNotFoundLocally(String branch, String remoteName) remoteHeads = commandExecutor .executeCommand( - IOUtils::readFully, "git", "ls-remote", "--heads", remoteName, shortBranchName) + IOUtils::readFully, + buildGitCommand("ls-remote", "--heads", remoteName, shortBranchName)) .trim(); } catch (ShellCommandExecutor.ShellCommandFailedException ignored) { } @@ -691,12 +726,7 @@ void tryFetchingIfNotFoundLocally(String branch, String remoteName) try { commandExecutor.executeCommand( ShellCommandExecutor.OutputParser.IGNORE, - "git", - "fetch", - "--depth", - "1", - remoteName, - shortBranchName); + buildGitCommand("fetch", "--depth", "1", remoteName, shortBranchName)); } catch (ShellCommandExecutor.ShellCommandFailedException e) { LOGGER.debug("Branch {}/{} couldn't be fetched from remote", remoteName, shortBranchName, e); } @@ -710,10 +740,8 @@ List getBaseBranchCandidates(@Nullable String defaultBranch, String remo List branches = commandExecutor.executeCommand( IOUtils::readLines, - "git", - "for-each-ref", - "--format=%(refname:short)", - "refs/remotes/" + remoteName); + buildGitCommand( + "for-each-ref", "--format=%(refname:short)", "refs/remotes/" + remoteName)); for (String branch : branches) { if (isBaseLikeBranch(branch, remoteName) || branchesEquals(branch, defaultBranch, remoteName)) { @@ -763,11 +791,8 @@ String detectDefaultBranch(String remoteName) commandExecutor .executeCommand( IOUtils::readFully, - "git", - "symbolic-ref", - "--quiet", - "--short", - "refs/remotes/" + remoteName + "/HEAD") + buildGitCommand( + "symbolic-ref", "--quiet", "--short", "refs/remotes/" + remoteName + "/HEAD")) .trim(); if (Strings.isNotBlank(defaultRef)) { return removeRemotePrefix(defaultRef, remoteName); @@ -781,11 +806,8 @@ String detectDefaultBranch(String remoteName) try { commandExecutor.executeCommand( ShellCommandExecutor.OutputParser.IGNORE, - "git", - "show-ref", - "--verify", - "--quiet", - "refs/remotes/" + remoteName + "/" + branch); + buildGitCommand( + "show-ref", "--verify", "--quiet", "refs/remotes/" + remoteName + "/" + branch)); LOGGER.debug("Found fallback default branch: {}", branch); return branch; } catch (ShellCommandExecutor.ShellCommandFailedException ignored) { @@ -843,11 +865,8 @@ List computeBranchMetrics(List candidates, String sour commandExecutor .executeCommand( IOUtils::readFully, - "git", - "rev-list", - "--left-right", - "--count", - candidate + "..." + sourceBranch) + buildGitCommand( + "rev-list", "--left-right", "--count", candidate + "..." + sourceBranch)) .trim(); String[] counts = WHITESPACE_PATTERN.split(countsResult); @@ -893,7 +912,7 @@ public String getMergeBase(@Nullable String base, @Nullable String source) } try { return commandExecutor - .executeCommand(IOUtils::readFully, "git", "merge-base", base, source) + .executeCommand(IOUtils::readFully, buildGitCommand("merge-base", base, source)) .trim(); } catch (ShellCommandExecutor.ShellCommandFailedException e) { LOGGER.debug("Error calculating common ancestor for {} and {}", base, source, e); @@ -925,35 +944,21 @@ public LineDiff getGitDiff(String baseCommit, String targetCommit) () -> commandExecutor.executeCommand( GitDiffParser::parse, - "git", - "diff", - "-U0", - "--word-diff=porcelain", - baseCommit, - targetCommit)); + buildGitCommand( + "diff", + "-U0", + "--word-diff=porcelain", + "--no-prefix", + baseCommit, + targetCommit))); } else { return executeCommand( Command.DIFF, () -> commandExecutor.executeCommand( - GitDiffParser::parse, "git", "diff", "-U0", "--word-diff=porcelain", baseCommit)); - } - } - - private void makeRepoRootSafeDirectory() { - // Some CI envs check out the repo as a different user than the one running the command - // This will avoid the "dubious ownership" error - try { - commandExecutor.executeCommand( - ShellCommandExecutor.OutputParser.IGNORE, - "git", - "config", - "--global", - "--add", - "safe.directory", - repoRoot); - } catch (IOException | TimeoutException | InterruptedException e) { - LOGGER.debug("Failed to add safe directory", e); + GitDiffParser::parse, + buildGitCommand( + "diff", "-U0", "--word-diff=porcelain", "--no-prefix", baseCommit))); } } @@ -1002,11 +1007,8 @@ public Factory(Config config, CiVisibilityMetricCollector metricCollector) { public GitClient create(@Nullable String repoRoot) { long commandTimeoutMillis = config.getCiVisibilityGitCommandTimeoutMillis(); if (repoRoot != null && GitUtils.isValidPath(repoRoot)) { - ShellGitClient client = - new ShellGitClient( - metricCollector, repoRoot, "1 month ago", 1000, commandTimeoutMillis); - client.makeRepoRootSafeDirectory(); - return client; + return new ShellGitClient( + metricCollector, repoRoot, "1 month ago", 1000, commandTimeoutMillis); } else { LOGGER.debug("Could not determine repository root, using no-op git client"); return NoOpGitClient.INSTANCE; diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/BestEffortSourcePathResolver.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/BestEffortSourcePathResolver.java index 3302f5c4723..814104cc3a9 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/BestEffortSourcePathResolver.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/BestEffortSourcePathResolver.java @@ -1,5 +1,7 @@ package datadog.trace.civisibility.source; +import java.util.Collection; +import java.util.Collections; import javax.annotation.Nonnull; import javax.annotation.Nullable; @@ -11,24 +13,22 @@ public BestEffortSourcePathResolver(SourcePathResolver... delegates) { this.delegates = delegates; } - @Nullable @Override - public String getSourcePath(@Nonnull Class c) throws SourceResolutionException { + public Collection getSourcePaths(@Nonnull Class c) { for (SourcePathResolver delegate : delegates) { - String sourcePath = delegate.getSourcePath(c); - if (sourcePath != null) { - return sourcePath; + Collection sourcePaths = delegate.getSourcePaths(c); + if (!sourcePaths.isEmpty()) { + return sourcePaths; } } - return null; + return Collections.emptyList(); } - @Nullable @Override - public String getResourcePath(@Nullable String relativePath) throws SourceResolutionException { + public Collection getResourcePaths(@Nullable String relativePath) { for (SourcePathResolver delegate : delegates) { - String resourcePath = delegate.getResourcePath(relativePath); - if (resourcePath != null) { + Collection resourcePath = delegate.getResourcePaths(relativePath); + if (!resourcePath.isEmpty()) { return resourcePath; } } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/CompilerAidedSourcePathResolver.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/CompilerAidedSourcePathResolver.java index c2aa8831780..35cef73d1d6 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/CompilerAidedSourcePathResolver.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/CompilerAidedSourcePathResolver.java @@ -2,6 +2,8 @@ import datadog.compiler.utils.CompilerUtils; import java.io.File; +import java.util.Collection; +import java.util.Collections; import javax.annotation.Nonnull; import javax.annotation.Nullable; @@ -13,20 +15,19 @@ public CompilerAidedSourcePathResolver(String repoRoot) { this.repoRoot = repoRoot.endsWith(File.separator) ? repoRoot : repoRoot + File.separator; } - @Nullable + @Nonnull @Override - public String getSourcePath(@Nonnull Class c) { + public Collection getSourcePaths(@Nonnull Class c) { String absoluteSourcePath = CompilerUtils.getSourcePath(c); if (absoluteSourcePath != null && absoluteSourcePath.startsWith(repoRoot)) { - return absoluteSourcePath.substring(repoRoot.length()); + return Collections.singletonList(absoluteSourcePath.substring(repoRoot.length())); } else { - return null; + return Collections.emptyList(); } } - @Nullable @Override - public String getResourcePath(String relativePath) { - return null; + public @Nullable Collection getResourcePaths(String relativePath) { + return Collections.emptyList(); } } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/NoOpSourcePathResolver.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/NoOpSourcePathResolver.java index a7b9e18f7bf..fb77c9cd87f 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/NoOpSourcePathResolver.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/NoOpSourcePathResolver.java @@ -1,5 +1,7 @@ package datadog.trace.civisibility.source; +import java.util.Collection; +import java.util.Collections; import javax.annotation.Nonnull; import javax.annotation.Nullable; @@ -7,15 +9,13 @@ public class NoOpSourcePathResolver implements SourcePathResolver { public static final SourcePathResolver INSTANCE = new NoOpSourcePathResolver(); - @Nullable @Override - public String getSourcePath(@Nonnull Class c) { - return null; + public Collection getSourcePaths(@Nonnull Class c) { + return Collections.emptyList(); } - @Nullable @Override - public String getResourcePath(@Nullable String relativePath) { - return null; + public Collection getResourcePaths(@Nullable String relativePath) { + return Collections.emptyList(); } } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/SourcePathResolver.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/SourcePathResolver.java index e76ed076c5f..e12705ba6b5 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/SourcePathResolver.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/SourcePathResolver.java @@ -1,20 +1,21 @@ package datadog.trace.civisibility.source; +import java.util.Collection; import javax.annotation.Nonnull; import javax.annotation.Nullable; public interface SourcePathResolver { + /** - * @return path to the source file corresponding to the provided class, relative to repository - * root. {@code null} is returned if the path could not be resolved + * @return paths to the source files corresponding to the provided class, relative to repository + * root. Returns all candidate paths when multiple matches exist (e.g. duplicate trie keys in + * repo index approach). Empty collection is returned if no paths could be resolved. */ - @Nullable - String getSourcePath(@Nonnull Class c) throws SourceResolutionException; + Collection getSourcePaths(@Nonnull Class c); /** * @param relativePath Path to a resource in current run's repository, relative to a resource root - * @return Path relative to repository root + * @return Candidate paths relative to repository root */ - @Nullable - String getResourcePath(@Nullable String relativePath) throws SourceResolutionException; + Collection getResourcePaths(@Nullable String relativePath); } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/SourceResolutionException.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/SourceResolutionException.java deleted file mode 100644 index 657b46d5031..00000000000 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/SourceResolutionException.java +++ /dev/null @@ -1,8 +0,0 @@ -package datadog.trace.civisibility.source; - -public class SourceResolutionException extends Exception { - - public SourceResolutionException(String message) { - super(message); - } -} diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/index/PackageResolver.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/index/PackageResolver.java index 82ec4ea641d..f8e239dfc04 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/index/PackageResolver.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/index/PackageResolver.java @@ -5,7 +5,9 @@ import javax.annotation.Nullable; public interface PackageResolver { - /** @return the package path or null if the file is in the default package */ + /** + * @return the package path or null if the file is in the default package + */ @Nullable Path getPackage(Path sourceFile) throws IOException; } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/index/RepoIndex.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/index/RepoIndex.java index b0250a7afd4..86f47c55eed 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/index/RepoIndex.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/index/RepoIndex.java @@ -1,11 +1,10 @@ package datadog.trace.civisibility.source.index; +import datadog.instrument.utils.ClassNameTrie; import datadog.trace.api.Config; import datadog.trace.api.civisibility.domain.Language; import datadog.trace.civisibility.ipc.serialization.Serializer; -import datadog.trace.civisibility.source.SourceResolutionException; import datadog.trace.civisibility.source.Utils; -import datadog.trace.util.ClassNameTrie; import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; import java.io.DataInputStream; @@ -16,6 +15,7 @@ import java.util.Collection; import java.util.Collections; import java.util.List; +import java.util.Map; import java.util.Objects; import javax.annotation.Nonnull; import javax.annotation.Nullable; @@ -27,20 +27,20 @@ public class RepoIndex { static final RepoIndex EMPTY = new RepoIndex( ClassNameTrie.Builder.EMPTY_TRIE, - Collections.emptyList(), + Collections.emptyMap(), Collections.emptyList(), Collections.emptyList()); private static final Logger log = LoggerFactory.getLogger(RepoIndex.class); private final ClassNameTrie trie; - private final Collection duplicateTrieKeys; + private final Map> duplicateTrieKeys; private final List sourceRoots; private final List rootPackages; RepoIndex( ClassNameTrie trie, - Collection duplicateTrieKeys, + Map> duplicateTrieKeys, List sourceRoots, List rootPackages) { this.trie = trie; @@ -53,11 +53,10 @@ public List getRootPackages() { return rootPackages; } - @Nullable - public String getSourcePath(@Nonnull Class c) throws SourceResolutionException { + public Collection getSourcePaths(@Nonnull Class c) { String topLevelClassName = Utils.stripNestedClassNames(c.getName()); - String sourcePath = doGetSourcePath(topLevelClassName); - return sourcePath != null ? sourcePath : getFallbackSourcePath(c); + Collection sourcePaths = doGetAllSourcePaths(topLevelClassName); + return !sourcePaths.isEmpty() ? sourcePaths : getFallbackSourcePaths(c); } /** @@ -65,55 +64,51 @@ public String getSourcePath(@Nonnull Class c) throws SourceResolutionExceptio * name does not necessarily correspond to the source file name, so source file name needs to be * retrieved from the bytecode. */ - @Nullable - private String getFallbackSourcePath(@Nonnull Class c) throws SourceResolutionException { + private Collection getFallbackSourcePaths(@Nonnull Class c) { try { String fileName = Utils.getFileName(c); if (fileName == null) { log.debug("Could not retrieve file name for class {}", c.getName()); - return null; + return Collections.emptyList(); } String fileNameWithoutExtension = Utils.stripExtension(fileName); Package classPackage = c.getPackage(); String packageName = classPackage != null ? classPackage.getName() : ""; String key = packageName + '.' + fileNameWithoutExtension; - return doGetSourcePath(key); + return doGetAllSourcePaths(key); } catch (IOException e) { log.error("Error while trying to retrieve file name for class {}", c.getName(), e); - return null; + return Collections.emptyList(); } } - @Nullable - public String getSourcePath(@Nullable String pathRelativeToSourceRoot) - throws SourceResolutionException { + public Collection getSourcePaths(@Nullable String pathRelativeToSourceRoot) { if (pathRelativeToSourceRoot == null) { - return null; + return Collections.emptyList(); } String key = Utils.toTrieKey(pathRelativeToSourceRoot); - return doGetSourcePath(key); + return doGetAllSourcePaths(key); } - @Nullable - private String doGetSourcePath(String key) throws SourceResolutionException { - if (Config.get().isCiVisibilityRepoIndexDuplicateKeyCheckEnabled()) { - if (!duplicateTrieKeys.isEmpty() && duplicateTrieKeys.contains(key)) { - throw new SourceResolutionException("There are multiple repo index entries for " + key); - } + private Collection doGetAllSourcePaths(String key) { + if (Config.get().isCiVisibilityRepoIndexDuplicateKeyCheckEnabled() + && !duplicateTrieKeys.isEmpty() + && duplicateTrieKeys.containsKey(key)) { + List paths = duplicateTrieKeys.get(key); + log.debug( + "Duplicate trie key {} resolved to {} candidate paths: {}", key, paths.size(), paths); + return paths; } int sourceRootIdx = trie.apply(key); if (sourceRootIdx < 0) { log.debug("Could not find source root for {}", key); - return null; + return Collections.emptyList(); } SourceRoot sourceRoot = sourceRoots.get(sourceRootIdx); - return sourceRoot.relativePath - + File.separatorChar - + key.replace('.', File.separatorChar) - + sourceRoot.language.getExtension(); + return Collections.singletonList(sourceRoot.resolveSourcePath(key)); } public ByteBuffer serialize() { @@ -129,7 +124,7 @@ public ByteBuffer serialize() { Serializer s = new Serializer(); s.write(serializedTrie); - s.write(duplicateTrieKeys); + s.write(duplicateTrieKeys, Serializer::write, (ser, paths) -> ser.write(paths)); s.write(sourceRoots, SourceRoot::serialize); s.write(rootPackages); return s.flush(); @@ -152,7 +147,8 @@ public static RepoIndex deserialize(ByteBuffer buffer) { } } - Collection duplicateTrieKeys = Serializer.readSet(buffer, Serializer::readString); + Map> duplicateTrieKeys = + Serializer.readMap(buffer, Serializer::readString, Serializer::readStringList); List sourceRoots = Serializer.readList(buffer, SourceRoot::deserialize); List rootPackages = Serializer.readStringList(buffer); return new RepoIndex(trie, duplicateTrieKeys, sourceRoots, rootPackages); @@ -169,6 +165,14 @@ static final class SourceRoot { this.language = language; } + /** Resolves a trie key (dot-separated) to a full source path relative to the source root. */ + String resolveSourcePath(String trieKey) { + return relativePath + + File.separatorChar + + trieKey.replace('.', File.separatorChar) + + language.getExtension(); + } + static void serialize(Serializer s, SourceRoot sourceRoot) { s.write(sourceRoot.relativePath); s.write(sourceRoot.language.ordinal()); diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/index/RepoIndexBuilder.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/index/RepoIndexBuilder.java index 23b93ec5580..2c223c34f90 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/index/RepoIndexBuilder.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/index/RepoIndexBuilder.java @@ -1,9 +1,9 @@ package datadog.trace.civisibility.source.index; +import datadog.instrument.utils.ClassNameTrie; import datadog.trace.api.Config; import datadog.trace.api.civisibility.domain.Language; import datadog.trace.civisibility.source.Utils; -import datadog.trace.util.ClassNameTrie; import java.io.IOException; import java.nio.file.FileSystem; import java.nio.file.FileVisitOption; @@ -12,11 +12,11 @@ import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.attribute.BasicFileAttributes; +import java.util.ArrayList; import java.util.Arrays; -import java.util.Collection; import java.util.EnumSet; import java.util.HashMap; -import java.util.HashSet; +import java.util.List; import java.util.Map; import java.util.concurrent.atomic.AtomicInteger; import javax.annotation.Nonnull; @@ -97,8 +97,8 @@ private static final class RepoIndexingFileVisitor implements FileVisitor private final PackageResolver packageResolver; private final ResourceResolver resourceResolver; private final ClassNameTrie.Builder trieBuilder; - private final Map trieKeyToPath; - private final Collection duplicateTrieKeys; + private final Map trieKeyToSourceRootIdx; + private final Map> duplicateSourceRootIndices; private final Map sourceRoots; private final PackageTree packageTree; private final RepoIndexingStats indexingStats; @@ -115,8 +115,8 @@ private RepoIndexingFileVisitor( this.resourceResolver = resourceResolver; this.repoRoot = repoRoot; trieBuilder = new ClassNameTrie.Builder(); - trieKeyToPath = new HashMap<>(); - duplicateTrieKeys = new HashSet<>(); + trieKeyToSourceRootIdx = new HashMap<>(); + duplicateSourceRootIndices = new HashMap<>(); sourceRoots = new HashMap<>(); packageTree = new PackageTree(config); indexingStats = new RepoIndexingStats(); @@ -124,6 +124,7 @@ private RepoIndexingFileVisitor( followSymlinks = config.isCiVisibilityRepoIndexFollowSymlinks(); } + @Nonnull @Override public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) { if (Files.isSymbolicLink(dir)) { @@ -151,6 +152,7 @@ private static Path readSymbolicLink(Path path) { } } + @Nonnull @Override public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) { indexingStats.filesVisited++; @@ -177,10 +179,18 @@ public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) { String key = Utils.toTrieKey(relativePath); trieBuilder.put(key, sourceRootIdx); - String existingPath = trieKeyToPath.put(key, file.toString()); - if (existingPath != null) { - log.debug("Duplicate repo index key: {} - {}", existingPath, file); - duplicateTrieKeys.add(key); + Integer existingSourceRootIdx = trieKeyToSourceRootIdx.put(key, sourceRootIdx); + if (existingSourceRootIdx != null) { + log.debug("Duplicate repo index key: {}", key); + duplicateSourceRootIndices + .computeIfAbsent( + key, + k -> { + List indices = new ArrayList<>(); + indices.add(existingSourceRootIdx); // Initialize with original source root + return indices; + }) + .add(sourceRootIdx); } } } @@ -220,6 +230,7 @@ private Path getNonCodeSourceRoot(Path file) throws IOException { return resourceResolver.getResourceRoot(file); } + @Nonnull @Override public FileVisitResult visitFileFailed(Path file, IOException exc) { if (exc != null) { @@ -228,6 +239,7 @@ public FileVisitResult visitFileFailed(Path file, IOException exc) { return FileVisitResult.CONTINUE; } + @Nonnull @Override public FileVisitResult postVisitDirectory(Path dir, IOException exc) { if (exc != null) { @@ -242,8 +254,23 @@ public RepoIndex getIndex() { roots[e.getValue()] = e.getKey(); } + Map> duplicateTrieKeyPaths = + new HashMap<>(duplicateSourceRootIndices.size() * 4 / 3); + for (Map.Entry> entry : duplicateSourceRootIndices.entrySet()) { + String key = entry.getKey(); + List paths = new ArrayList<>(entry.getValue().size()); + for (int idx : entry.getValue()) { + RepoIndex.SourceRoot sr = roots[idx]; + paths.add(sr.resolveSourcePath(key)); + } + duplicateTrieKeyPaths.put(key, paths); + } + return new RepoIndex( - trieBuilder.buildTrie(), duplicateTrieKeys, Arrays.asList(roots), packageTree.asList()); + trieBuilder.buildTrie(), + duplicateTrieKeyPaths, + Arrays.asList(roots), + packageTree.asList()); } } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/index/RepoIndexSourcePathResolver.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/index/RepoIndexSourcePathResolver.java index f58de66307f..52ae4db3109 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/index/RepoIndexSourcePathResolver.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/source/index/RepoIndexSourcePathResolver.java @@ -1,7 +1,7 @@ package datadog.trace.civisibility.source.index; import datadog.trace.civisibility.source.SourcePathResolver; -import datadog.trace.civisibility.source.SourceResolutionException; +import java.util.Collection; import javax.annotation.Nonnull; import javax.annotation.Nullable; @@ -13,15 +13,13 @@ public RepoIndexSourcePathResolver(RepoIndexProvider indexProvider) { this.indexProvider = indexProvider; } - @Nullable @Override - public String getSourcePath(@Nonnull Class c) throws SourceResolutionException { - return indexProvider.getIndex().getSourcePath(c); + public Collection getSourcePaths(@Nonnull Class c) { + return indexProvider.getIndex().getSourcePaths(c); } - @Nullable @Override - public String getResourcePath(@Nullable String relativePath) throws SourceResolutionException { - return indexProvider.getIndex().getSourcePath(relativePath); + public Collection getResourcePaths(@Nullable String relativePath) { + return indexProvider.getIndex().getSourcePaths(relativePath); } } diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/telemetry/CiVisibilityMetricCollectorImpl.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/telemetry/CiVisibilityMetricCollectorImpl.java index e9cf22caa6c..68902e2def8 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/telemetry/CiVisibilityMetricCollectorImpl.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/telemetry/CiVisibilityMetricCollectorImpl.java @@ -26,6 +26,7 @@ public class CiVisibilityMetricCollectorImpl implements CiVisibilityMetricCollec private final BlockingQueue rawMetricsQueue; private final BlockingQueue rawDistributionPointsQueue; private final AtomicLongArray counters; + /** * Cards are used to avoid iterating over the entire {@link * CiVisibilityMetricCollectorImpl#counters} array every time {@link diff --git a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/test/ExecutionStrategy.java b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/test/ExecutionStrategy.java index 7cdacf572fc..77fe7f354f9 100644 --- a/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/test/ExecutionStrategy.java +++ b/dd-java-agent/agent-ci-visibility/src/main/java/datadog/trace/civisibility/test/ExecutionStrategy.java @@ -6,16 +6,16 @@ import datadog.trace.api.civisibility.config.TestMetadata; import datadog.trace.api.civisibility.config.TestSourceData; import datadog.trace.api.civisibility.execution.TestExecutionPolicy; -import datadog.trace.api.civisibility.telemetry.tag.RetryReason; import datadog.trace.api.civisibility.telemetry.tag.SkipReason; import datadog.trace.civisibility.config.EarlyFlakeDetectionSettings; import datadog.trace.civisibility.config.ExecutionSettings; import datadog.trace.civisibility.config.TestManagementSettings; import datadog.trace.civisibility.config.TestSetting; +import datadog.trace.civisibility.execution.AttemptToFix; +import datadog.trace.civisibility.execution.AutoTestRetry; +import datadog.trace.civisibility.execution.EarlyFlakeDetection; +import datadog.trace.civisibility.execution.Quarantine; import datadog.trace.civisibility.execution.Regular; -import datadog.trace.civisibility.execution.RetryUntilSuccessful; -import datadog.trace.civisibility.execution.RunNTimes; -import datadog.trace.civisibility.execution.RunOnceIgnoreOutcome; import datadog.trace.civisibility.source.LinesResolver; import datadog.trace.civisibility.source.SourcePathResolver; import java.lang.reflect.Method; @@ -126,31 +126,29 @@ public TestExecutionPolicy executionPolicy( } if (isAttemptToFix(test)) { - return new RunNTimes( - executionSettings.getTestManagementSettings().getAttemptToFixExecutions(), - isQuarantined(test) || isDisabled(test), - RetryReason.attemptToFix); + return new AttemptToFix( + executionSettings.getTestManagementSettings().getAttemptToFixRetries(), + isQuarantined(test) || isDisabled(test)); } if (isEFDApplicable(test, testSource, testTags)) { // check-then-act with "earlyFlakeDetectionsUsed" is not atomic here, // but we don't care if we go "a bit" over the limit, it does not have to be precise earlyFlakeDetectionsUsed.incrementAndGet(); - return new RunNTimes( + return new EarlyFlakeDetection( executionSettings.getEarlyFlakeDetectionSettings().getExecutionsByDuration(), - isQuarantined(test), - RetryReason.efd); + isQuarantined(test)); } if (isAutoRetryApplicable(test)) { // check-then-act with "autoRetriesUsed" is not atomic here, // but we don't care if we go "a bit" over the limit, it does not have to be precise - return new RetryUntilSuccessful( + return new AutoTestRetry( config.getCiVisibilityFlakyRetryCount(), isQuarantined(test), autoRetriesUsed); } if (isQuarantined(test)) { - return new RunOnceIgnoreOutcome(); + return new Quarantine(); } return Regular.INSTANCE; @@ -199,10 +197,11 @@ public boolean isModified(@Nonnull TestSourceData testSourceData) { return false; } try { - String sourcePath = sourcePathResolver.getSourcePath(testClass); - if (sourcePath == null) { + Collection sourcePaths = sourcePathResolver.getSourcePaths(testClass); + if (sourcePaths.size() != 1) { return false; } + String sourcePath = sourcePaths.iterator().next(); LinesResolver.Lines lines = getLines(testSourceData.getTestMethod()); return executionSettings diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/ci/GithubActionsInfoTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/ci/GithubActionsInfoTest.groovy index 912b5201040..4563ca9ecc2 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/ci/GithubActionsInfoTest.groovy +++ b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/ci/GithubActionsInfoTest.groovy @@ -1,11 +1,17 @@ package datadog.trace.civisibility.ci import datadog.trace.civisibility.ci.env.CiEnvironmentImpl +import spock.lang.TempDir +import java.nio.file.Files +import java.nio.file.Path import java.nio.file.Paths class GithubActionsInfoTest extends CITagsProviderTest { + @TempDir + Path temporaryFolder + @Override String getProviderName() { return GithubActionsInfo.GHACTIONS_PROVIDER_NAME @@ -46,4 +52,176 @@ class GithubActionsInfoTest extends CITagsProviderTest { pullRequestInfo.getHeadCommit().getSha() == "df289512a51123083a8e6931dd6f57bb3883d4c4" pullRequestInfo.getPullRequestNumber() == "1" } + + def "test parseCheckRunIdFromContent extracts ID from content"() { + setup: + def info = new GithubActionsInfo(new CiEnvironmentImpl([:]), temporaryFolder, temporaryFolder) + + expect: + info.parseCheckRunIdFromContent(content) == expected + + where: + content | expected + // Basic format with decimal + '{"k":"check_run_id","v":55411116365.0}' | "55411116365" + // Format with spaces + '{"k" : "check_run_id" , "v" : 55411116365.0}' | "55411116365" + // Integer format (no decimal) + '{"k":"check_run_id","v":55411116365}' | "55411116365" + // Multiple entries - should return the last one + '{"k":"check_run_id","v":111.0}\n{"k":"check_run_id","v":222.0}' | "222" + // Content with surrounding text + '[2025-01-15 INFO] {"k":"check_run_id","v":12345.0} some other text' | "12345" + // No match + 'some random content without check_run_id' | null + // Empty content + '' | null + // Malformed JSON (no match) + '{"k":"check_run_id"}' | null + } + + def "test job ID from environment variable takes precedence"() { + setup: + env.set(GithubActionsInfo.GHACTIONS, "true") + env.set(GithubActionsInfo.GHACTIONS_URL, "https://github.com") + env.set(GithubActionsInfo.GHACTIONS_REPOSITORY, "owner/repo") + env.set(GithubActionsInfo.GHACTIONS_PIPELINE_ID, "12345") + env.set(GithubActionsInfo.GHACTIONS_JOB, "build") + env.set(GithubActionsInfo.GHACTIONS_SHA, "abc123") + env.set(GithubActionsInfo.GHACTIONS_JOB_CHECK_RUN_ID, "99999") + + // Create a diagnostics directory with a worker file (should be ignored) + def diagDir = Files.createDirectories(temporaryFolder.resolve("diag")) + def workerFile = diagDir.resolve("Worker_20250115-123456-utc.log") + Files.write(workerFile, '{"k":"check_run_id","v":11111.0}'.getBytes()) + + when: + def info = new GithubActionsInfo(new CiEnvironmentImpl(env.getAll()), diagDir, temporaryFolder) + def ciInfo = info.buildCIInfo() + + then: + ciInfo.getCiJobId() == "99999" + ciInfo.getCiJobUrl() == "https://github.com/owner/repo/actions/runs/12345/job/99999" + ciInfo.getCiJobName() == "build" + } + + def "test job ID from diagnostics file when env var not set"() { + setup: + env.set(GithubActionsInfo.GHACTIONS, "true") + env.set(GithubActionsInfo.GHACTIONS_URL, "https://github.com") + env.set(GithubActionsInfo.GHACTIONS_REPOSITORY, "owner/repo") + env.set(GithubActionsInfo.GHACTIONS_PIPELINE_ID, "12345") + env.set(GithubActionsInfo.GHACTIONS_JOB, "build") + env.set(GithubActionsInfo.GHACTIONS_SHA, "abc123") + // No JOB_CHECK_RUN_ID env var set + + // Create a diagnostics directory with a worker file + def diagDir = Files.createDirectories(temporaryFolder.resolve("diag")) + def workerFile = diagDir.resolve("Worker_20250115-123456-utc.log") + Files.write(workerFile, '[2025-01-15 INFO] {"job":{"d":[{"k":"check_run_id","v":77777.0}]}}'.getBytes()) + + when: + def info = new GithubActionsInfo(new CiEnvironmentImpl(env.getAll()), diagDir, temporaryFolder) + def ciInfo = info.buildCIInfo() + + then: + ciInfo.getCiJobId() == "77777" + ciInfo.getCiJobUrl() == "https://github.com/owner/repo/actions/runs/12345/job/77777" + ciInfo.getCiJobName() == "build" + } + + def "test fallback to cached diagnostics directory"() { + setup: + env.set(GithubActionsInfo.GHACTIONS, "true") + env.set(GithubActionsInfo.GHACTIONS_URL, "https://github.com") + env.set(GithubActionsInfo.GHACTIONS_REPOSITORY, "owner/repo") + env.set(GithubActionsInfo.GHACTIONS_PIPELINE_ID, "12345") + env.set(GithubActionsInfo.GHACTIONS_JOB, "build") + env.set(GithubActionsInfo.GHACTIONS_SHA, "abc123") + + // Create only the cached diagnostics directory (primary doesn't exist) + def cachedDiagDir = Files.createDirectories(temporaryFolder.resolve("cached_diag")) + def workerFile = cachedDiagDir.resolve("Worker_20250115-123456-utc.log") + Files.write(workerFile, '{"k":"check_run_id","v":88888.0}'.getBytes()) + + // Use a non-existent path for primary diag dir + def nonExistentDir = temporaryFolder.resolve("non_existent") + + when: + def info = new GithubActionsInfo(new CiEnvironmentImpl(env.getAll()), nonExistentDir, cachedDiagDir) + def ciInfo = info.buildCIInfo() + + then: + ciInfo.getCiJobId() == "88888" + ciInfo.getCiJobUrl() == "https://github.com/owner/repo/actions/runs/12345/job/88888" + } + + def "test fallback to commit-based URL when no job ID available"() { + setup: + env.set(GithubActionsInfo.GHACTIONS, "true") + env.set(GithubActionsInfo.GHACTIONS_URL, "https://github.com") + env.set(GithubActionsInfo.GHACTIONS_REPOSITORY, "owner/repo") + env.set(GithubActionsInfo.GHACTIONS_PIPELINE_ID, "12345") + env.set(GithubActionsInfo.GHACTIONS_JOB, "build") + env.set(GithubActionsInfo.GHACTIONS_SHA, "abc123def456") + // No JOB_CHECK_RUN_ID env var and no diagnostics files + + // Use non-existent directories + def nonExistentDir1 = temporaryFolder.resolve("non_existent1") + def nonExistentDir2 = temporaryFolder.resolve("non_existent2") + + when: + def info = new GithubActionsInfo(new CiEnvironmentImpl(env.getAll()), nonExistentDir1, nonExistentDir2) + def ciInfo = info.buildCIInfo() + + then: + ciInfo.getCiJobId() == "build" + ciInfo.getCiJobUrl() == "https://github.com/owner/repo/commit/abc123def456/checks" + ciInfo.getCiJobName() == "build" + } + + def "test empty diagnostics directory"() { + setup: + env.set(GithubActionsInfo.GHACTIONS, "true") + env.set(GithubActionsInfo.GHACTIONS_URL, "https://github.com") + env.set(GithubActionsInfo.GHACTIONS_REPOSITORY, "owner/repo") + env.set(GithubActionsInfo.GHACTIONS_PIPELINE_ID, "12345") + env.set(GithubActionsInfo.GHACTIONS_JOB, "build") + env.set(GithubActionsInfo.GHACTIONS_SHA, "abc123") + + // Create empty diagnostics directory (no worker files) + def diagDir = Files.createDirectories(temporaryFolder.resolve("diag")) + + when: + def info = new GithubActionsInfo(new CiEnvironmentImpl(env.getAll()), diagDir, temporaryFolder) + def ciInfo = info.buildCIInfo() + + then: + // Falls back to GITHUB_JOB for job ID + ciInfo.getCiJobId() == "build" + ciInfo.getCiJobUrl() == "https://github.com/owner/repo/commit/abc123/checks" + } + + def "test worker file without check_run_id"() { + setup: + env.set(GithubActionsInfo.GHACTIONS, "true") + env.set(GithubActionsInfo.GHACTIONS_URL, "https://github.com") + env.set(GithubActionsInfo.GHACTIONS_REPOSITORY, "owner/repo") + env.set(GithubActionsInfo.GHACTIONS_PIPELINE_ID, "12345") + env.set(GithubActionsInfo.GHACTIONS_JOB, "build") + env.set(GithubActionsInfo.GHACTIONS_SHA, "abc123") + + // Create diagnostics directory with worker file that lacks check_run_id + def diagDir = Files.createDirectories(temporaryFolder.resolve("diag")) + def workerFile = diagDir.resolve("Worker_20250115-123456-utc.log") + Files.write(workerFile, '[2025-01-15 INFO] Some log content without the expected JSON'.getBytes()) + + when: + def info = new GithubActionsInfo(new CiEnvironmentImpl(env.getAll()), diagDir, temporaryFolder) + def ciInfo = info.buildCIInfo() + + then: + ciInfo.getCiJobId() == "build" + ciInfo.getCiJobUrl() == "https://github.com/owner/repo/commit/abc123/checks" + } } diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/config/ConfigurationApiImplTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/config/ConfigurationApiImplTest.groovy index 505b4bdcda4..d37abda8f52 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/config/ConfigurationApiImplTest.groovy +++ b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/config/ConfigurationApiImplTest.groovy @@ -16,6 +16,7 @@ import freemarker.core.InvalidReferenceException import freemarker.template.Template import freemarker.template.TemplateException import freemarker.template.TemplateExceptionHandler +import java.util.concurrent.atomic.AtomicInteger import okhttp3.HttpUrl import okhttp3.OkHttpClient import org.apache.commons.io.IOUtils @@ -59,10 +60,10 @@ class ConfigurationApiImplTest extends Specification { where: agentless | compression | expectedSettings - false | false | new CiVisibilitySettings(false, false, false, false, false, false, false, false, false, EarlyFlakeDetectionSettings.DEFAULT, TestManagementSettings.DEFAULT, null) - false | true | new CiVisibilitySettings(true, true, true, true, true, true, true, true, true, EarlyFlakeDetectionSettings.DEFAULT, TestManagementSettings.DEFAULT, "main") - true | false | new CiVisibilitySettings(false, true, false, true, false, true, false, false, true, new EarlyFlakeDetectionSettings(true, [new ExecutionsByDuration(1000, 3)], 10), new TestManagementSettings(true, 10), "master") - true | true | new CiVisibilitySettings(false, false, true, true, false, false, true, true, false, new EarlyFlakeDetectionSettings(true, [new ExecutionsByDuration(5000, 3), new ExecutionsByDuration(120000, 2)], 10), new TestManagementSettings(true, 20), "prod") + false | false | new CiVisibilitySettings(false, false, false, false, false, false, false, false, false, EarlyFlakeDetectionSettings.DEFAULT, TestManagementSettings.DEFAULT, null, false) + false | true | new CiVisibilitySettings(true, true, true, true, true, true, true, true, true, EarlyFlakeDetectionSettings.DEFAULT, TestManagementSettings.DEFAULT, "main", false) + true | false | new CiVisibilitySettings(false, true, false, true, false, true, false, false, true, new EarlyFlakeDetectionSettings(true, [new ExecutionsByDuration(1000, 3)], 10), new TestManagementSettings(true, 10), "master", false) + true | true | new CiVisibilitySettings(false, false, true, true, false, false, true, true, false, new EarlyFlakeDetectionSettings(true, [new ExecutionsByDuration(5000, 3), new ExecutionsByDuration(120000, 2)], 10), new TestManagementSettings(true, 20), "prod", false) } def "test skippable tests request"() { @@ -146,11 +147,12 @@ class ConfigurationApiImplTest extends Specification { def "test known tests request"() { given: def tracerEnvironment = givenTracerEnvironment() + def pageInfo = [:] def intakeServer = givenBackendEndpoint( "/api/v2/ci/libraries/tests", "/datadog/trace/civisibility/config/known-tests-request.ftl", - [uid: REQUEST_UID, tracerEnvironment: tracerEnvironment], + [uid: REQUEST_UID, tracerEnvironment: tracerEnvironment, pageInfo: pageInfo], "/datadog/trace/civisibility/config/known-tests-response.ftl", [:] ) @@ -184,6 +186,61 @@ class ConfigurationApiImplTest extends Specification { intakeServer.close() } + def "test known tests request with pagination"() { + given: + def tracerEnvironment = givenTracerEnvironment() + def requestCount = new AtomicInteger(0) + + // Define page responses + def page1 = '{"data":{"id":"page1","type":"ci_app_libraries_tests","attributes":{"tests":{"module-a":{"suite-a":["test-1","test-2"]}},"page_info":{"size":2,"has_next":true,"cursor":"cursor-page-2"}}}}' + def page2 = '{"data":{"id":"page2","type":"ci_app_libraries_tests","attributes":{"tests":{"module-b":{"suite-b":["test-3","test-4"]}},"page_info":{"size":2,"has_next":true,"cursor":"cursor-page-3"}}}}' + def page3 = '{"data":{"id":"page3","type":"ci_app_libraries_tests","attributes":{"tests":{"module-c":{"suite-c":["test-5","test-6"]}},"page_info":{"size":2,"has_next":false}}}}' + def responses = [page1, page2, page3] + + def intakeServer = httpServer { + handlers { + prefix("/api/v2/ci/libraries/tests") { + def currentRequest = requestCount.incrementAndGet() + def responseBody = responses[currentRequest - 1].bytes + + def response = response + def header = request.getHeader("Accept-Encoding") + def gzipSupported = header != null && header.contains("gzip") + if (gzipSupported) { + response.addHeader("Content-Encoding", "gzip") + responseBody = gzip(responseBody) + } + + response.status(200).send(responseBody) + } + } + } + def configurationApi = givenConfigurationApi(intakeServer) + + when: + def knownTests = configurationApi.getKnownTestsByModule(tracerEnvironment) + + for (Map.Entry> e : knownTests.entrySet()) { + def sortedTests = new ArrayList<>(e.value) + Collections.sort(sortedTests, Comparator.comparing(TestFQN::getSuite).thenComparing((Function)TestFQN::getName)) + e.value = sortedTests + } + + then: + // Verify 3 requests were made (3 pages) + requestCount.get() == 3 + + // Verify merged results from all 3 pages + knownTests == [ + "module-a": [new TestFQN("suite-a", "test-1"), new TestFQN("suite-a", "test-2")], + "module-b": [new TestFQN("suite-b", "test-3"), new TestFQN("suite-b", "test-4")], + "module-c": [new TestFQN("suite-c", "test-5"), new TestFQN("suite-c", "test-6")] + ] + + cleanup: + intakeServer.close() + } + def "test test management tests request"() { given: def tracerEnvironment = givenTracerEnvironment() diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/config/ExecutionSettingsTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/config/ExecutionSettingsTest.groovy index 423ff5b0ecf..e98a9bcadc6 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/config/ExecutionSettingsTest.groovy +++ b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/config/ExecutionSettingsTest.groovy @@ -38,7 +38,8 @@ class ExecutionSettingsTest extends DDSpecification { new HashSet<>([]), new HashSet<>([]), new HashSet<>([]), - LineDiff.EMPTY), + LineDiff.EMPTY, + ConfigurationErrors.NONE), new ExecutionSettings( true, @@ -58,7 +59,8 @@ class ExecutionSettingsTest extends DDSpecification { new HashSet<>([new TestFQN("suite", "quarantined")]), new HashSet<>([new TestFQN("suite", "disabled")]), new HashSet<>([new TestFQN("suite", "attemptToFix")]), - new LineDiff(["path": lines()]) + new LineDiff(["path": lines()]), + ConfigurationErrors.NONE ), new ExecutionSettings( @@ -84,6 +86,7 @@ class ExecutionSettingsTest extends DDSpecification { new HashSet<>([new TestFQN("suite", "disabled"), new TestFQN("another", "another-disabled")]), new HashSet<>([new TestFQN("suite", "attemptToFix"), new TestFQN("another", "another-attemptToFix")]), new LineDiff(["path": lines(1, 2, 3)]), + new ConfigurationErrors(false, true, false, true, false) ), new ExecutionSettings( @@ -109,6 +112,7 @@ class ExecutionSettingsTest extends DDSpecification { new HashSet<>([new TestFQN("suite", "disabled"), new TestFQN("another", "another-disabled")]), new HashSet<>([new TestFQN("suite", "attemptToFix"), new TestFQN("another", "another-attemptToFix")]), new LineDiff(["path": lines(1, 2, 3), "path-b": lines(1, 2, 128, 257, 999)]), + new ConfigurationErrors(true, true, true, true, true) ), ] } diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/domain/TestImplTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/domain/TestImplTest.groovy index 56b1a8ab640..5c1d7801e55 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/domain/TestImplTest.groovy +++ b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/domain/TestImplTest.groovy @@ -13,6 +13,7 @@ import datadog.trace.api.civisibility.telemetry.tag.TestFrameworkInstrumentation import datadog.trace.bootstrap.instrumentation.api.AgentSpanContext import datadog.trace.bootstrap.instrumentation.api.AgentTracer import datadog.trace.civisibility.codeowners.NoCodeowners +import datadog.trace.civisibility.config.ConfigurationErrors import datadog.trace.civisibility.decorator.TestDecoratorImpl import datadog.trace.civisibility.source.LinesResolver import datadog.trace.civisibility.source.NoOpSourcePathResolver @@ -141,6 +142,7 @@ class TestImplTest extends SpanWriterTest { codeowners, coverageStoreFactory, executionResults, + ConfigurationErrors.NONE, libraryCapabilities, SpanTagsPropagator.NOOP_PROPAGATOR ) diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/domain/TestSuiteImplTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/domain/TestSuiteImplTest.groovy index c7819669a85..3059fe6e8f2 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/domain/TestSuiteImplTest.groovy +++ b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/domain/TestSuiteImplTest.groovy @@ -10,6 +10,7 @@ import datadog.trace.api.civisibility.telemetry.tag.TestFrameworkInstrumentation import datadog.trace.bootstrap.instrumentation.api.AgentSpanContext import datadog.trace.bootstrap.instrumentation.api.Tags import datadog.trace.civisibility.codeowners.NoCodeowners +import datadog.trace.civisibility.config.ConfigurationErrors import datadog.trace.civisibility.decorator.TestDecoratorImpl import datadog.trace.civisibility.source.LinesResolver import datadog.trace.civisibility.source.SourcePathResolver @@ -61,7 +62,7 @@ class TestSuiteImplTest extends SpanWriterTest { linesResolver.getClassLines(MyClass) >> classLines def resolver = Stub(SourcePathResolver) - resolver.getSourcePath(MyClass) >> "MyClass.java" + resolver.getSourcePaths(MyClass) >> ["MyClass.java"] def codeowners = Stub(NoCodeowners) codeowners.getOwners("MyClass.java") >> ["@global-owner1", "@global-owner2"] @@ -84,6 +85,7 @@ class TestSuiteImplTest extends SpanWriterTest { linesResolver, coverageStoreFactory, executionResults, + ConfigurationErrors.NONE, [], SpanTagsPropagator.NOOP_PROPAGATOR ) diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/domain/manualapi/ManualApiTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/domain/manualapi/ManualApiTest.groovy new file mode 100644 index 00000000000..51962ea5704 --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/domain/manualapi/ManualApiTest.groovy @@ -0,0 +1,66 @@ +package datadog.trace.civisibility.domain.manualapi + +import datadog.trace.api.Config +import datadog.trace.api.DDSpanTypes +import datadog.trace.api.civisibility.coverage.CoverageStore +import datadog.trace.api.civisibility.telemetry.CiVisibilityMetricCollector +import datadog.trace.api.civisibility.telemetry.tag.Provider +import datadog.trace.bootstrap.instrumentation.api.Tags +import datadog.trace.civisibility.codeowners.Codeowners +import datadog.trace.civisibility.decorator.TestDecoratorImpl +import datadog.trace.civisibility.domain.SpanWriterTest +import datadog.trace.civisibility.source.LinesResolver +import datadog.trace.civisibility.source.SourcePathResolver + +class ManualApiTest extends SpanWriterTest { + + def "test framework tag is set on suite, test, module and session with component info"() { + setup: + def component = "my-custom-framework" + def session = givenAManualApiSession(component) + def module = session.testModuleStart("module-name", null) + def suite = module.testSuiteStart("suite-name", null, null, false) + def test = suite.testStart("test-name", null, null) + + when: + test.end(null) + suite.end(null) + module.end(null) + session.end(null) + + then: + def traces = TEST_WRITER.toList() + traces.size() == 2 + + def allSpans = traces.flatten() + def sessionSpan = allSpans.find { it.spanType == DDSpanTypes.TEST_SESSION_END } + def moduleSpan = allSpans.find { it.spanType == DDSpanTypes.TEST_MODULE_END } + def suiteSpan = allSpans.find { it.spanType == DDSpanTypes.TEST_SUITE_END } + def testSpan = allSpans.find { it.spanType == DDSpanTypes.TEST } + + sessionSpan != null + moduleSpan != null + suiteSpan != null + testSpan != null + + sessionSpan.tags[Tags.TEST_FRAMEWORK] == component + moduleSpan.tags[Tags.TEST_FRAMEWORK] == component + suiteSpan.tags[Tags.TEST_FRAMEWORK] == component + testSpan.tags[Tags.TEST_FRAMEWORK] == component + } + + private ManualApiTestSession givenAManualApiSession(String component) { + new ManualApiTestSession( + "project-name", + null, + Provider.UNSUPPORTED, + Stub(Config), + Stub(CiVisibilityMetricCollector), + new TestDecoratorImpl(component, "session-name", "test-command", [:]), + Stub(SourcePathResolver), + Stub(Codeowners), + Stub(LinesResolver), + Stub(CoverageStore.Factory) + ) + } +} diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/AttemptToFixTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/AttemptToFixTest.groovy new file mode 100644 index 00000000000..1337a069541 --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/AttemptToFixTest.groovy @@ -0,0 +1,74 @@ +package datadog.trace.civisibility.execution + +import datadog.trace.api.civisibility.execution.ExecutionAggregation +import datadog.trace.api.civisibility.execution.TestStatus +import datadog.trace.api.civisibility.telemetry.tag.RetryReason +import spock.lang.Specification + +class AttemptToFixTest extends Specification { + + def "test attempt to fix exits on failure"() { + setup: + def executionPolicy = new AttemptToFix(3, false) + + when: + def outcome = executionPolicy.registerExecution(TestStatus.fail, 0) + + then: + outcome.retryReason() == null + outcome.lastExecution() + !outcome.failureSuppressed() + outcome.aggregation() == ExecutionAggregation.ONLY_FAILED + outcome.finalStatus() == TestStatus.fail + } + + def "test attempt to fix succeeded all executions"() { + setup: + def executionPolicy = new AttemptToFix(3, false) + + when: + def outcome = executionPolicy.registerExecution(TestStatus.pass, 0) + + then: + outcome.retryReason() == null + !outcome.lastExecution() + !outcome.failureSuppressed() + outcome.aggregation() == ExecutionAggregation.ONLY_PASSED + outcome.finalStatus() == null + + when: + def outcome2 = executionPolicy.registerExecution(TestStatus.pass, 0) + + then: + outcome2.retryReason() == RetryReason.attemptToFix + !outcome2.lastExecution() + !outcome2.failureSuppressed() + outcome2.aggregation() == ExecutionAggregation.ONLY_PASSED + outcome2.finalStatus() == null + + when: + def outcome3 = executionPolicy.registerExecution(TestStatus.pass, 0) + + then: + outcome3.retryReason() == RetryReason.attemptToFix + outcome3.lastExecution() + !outcome3.failureSuppressed() + outcome3.aggregation() == ExecutionAggregation.ONLY_PASSED + outcome3.finalStatus() == TestStatus.pass + } + + def "test attempt to fix suppresses failures when quarantined or disabled"() { + setup: + def executionPolicy = new AttemptToFix(3, true) + + when: + def outcome = executionPolicy.registerExecution(TestStatus.fail, 0) + + then: + outcome.retryReason() == null + outcome.lastExecution() + outcome.failureSuppressed() + outcome.aggregation() == ExecutionAggregation.ONLY_FAILED + outcome.finalStatus() == TestStatus.pass + } +} diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/AutoTestRetryTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/AutoTestRetryTest.groovy new file mode 100644 index 00000000000..ca9409529ea --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/AutoTestRetryTest.groovy @@ -0,0 +1,121 @@ +package datadog.trace.civisibility.execution + +import datadog.trace.api.civisibility.execution.ExecutionAggregation +import datadog.trace.api.civisibility.execution.TestStatus +import datadog.trace.api.civisibility.telemetry.tag.RetryReason +import spock.lang.Specification + +import java.util.concurrent.atomic.AtomicInteger + +class AutoTestRetryTest extends Specification { + + def "test retry until successful"() { + setup: + def executionPolicy = new AutoTestRetry(3, false, new AtomicInteger()) + + when: + def outcome = executionPolicy.registerExecution(TestStatus.fail, 0) + + then: + outcome.retryReason() == null + !outcome.lastExecution() + outcome.failureSuppressed() + outcome.aggregation() == ExecutionAggregation.ONLY_FAILED + outcome.finalStatus() == null + + when: + def outcome2 = executionPolicy.registerExecution(TestStatus.pass, 0) + + then: + outcome2.retryReason() == RetryReason.atr + outcome2.lastExecution() + !outcome2.failureSuppressed() + outcome2.aggregation() == ExecutionAggregation.MIXED + outcome2.finalStatus() == TestStatus.pass + } + + def "test fail all retries"() { + setup: + def executionPolicy = new AutoTestRetry(3, false, new AtomicInteger()) + + when: + def outcome = executionPolicy.registerExecution(TestStatus.fail, 0) + + then: + outcome.retryReason() == null + !outcome.lastExecution() + outcome.failureSuppressed() + outcome.aggregation() == ExecutionAggregation.ONLY_FAILED + outcome.finalStatus() == null + + when: + def outcome2 = executionPolicy.registerExecution(TestStatus.fail, 0) + + then: + outcome2.retryReason() == RetryReason.atr + !outcome2.lastExecution() + outcome2.failureSuppressed() + outcome2.aggregation() == ExecutionAggregation.ONLY_FAILED + outcome2.finalStatus() == null + + when: + def outcome3 = executionPolicy.registerExecution(TestStatus.fail, 0) + + then: + outcome3.retryReason() == RetryReason.atr + outcome3.lastExecution() + !outcome3.failureSuppressed() + outcome3.aggregation() == ExecutionAggregation.ONLY_FAILED + outcome3.finalStatus() == TestStatus.fail + } + + def "test succeed on first try"() { + setup: + def executionPolicy = new AutoTestRetry(3, false, new AtomicInteger()) + + when: + def outcome = executionPolicy.registerExecution(TestStatus.pass, 0) + + then: + outcome.retryReason() == null + outcome.lastExecution() + !outcome.failureSuppressed() + outcome.aggregation() == ExecutionAggregation.ONLY_PASSED + outcome.finalStatus() == TestStatus.pass + } + + def "test suppress failures"() { + setup: + def executionPolicy = new AutoTestRetry(3, true, new AtomicInteger()) + + when: + def outcome = executionPolicy.registerExecution(TestStatus.fail, 0) + + then: + outcome.retryReason() == null + !outcome.lastExecution() + outcome.failureSuppressed() + outcome.aggregation() == ExecutionAggregation.ONLY_FAILED + outcome.finalStatus() == null + + when: + def outcome2 = executionPolicy.registerExecution(TestStatus.fail, 0) + + then: + outcome2.retryReason() == RetryReason.atr + !outcome2.lastExecution() + outcome2.failureSuppressed() + outcome2.aggregation() == ExecutionAggregation.ONLY_FAILED + outcome2.finalStatus() == null + + when: + def outcome3 = executionPolicy.registerExecution(TestStatus.fail, 0) + + then: + outcome3.retryReason() == RetryReason.atr + outcome3.lastExecution() + outcome3.failureSuppressed() + outcome3.aggregation() == ExecutionAggregation.ONLY_FAILED + outcome3.finalStatus() == TestStatus.pass + } +} diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/EarlyFlakeDetectionTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/EarlyFlakeDetectionTest.groovy new file mode 100644 index 00000000000..7addf0e724a --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/EarlyFlakeDetectionTest.groovy @@ -0,0 +1,122 @@ +package datadog.trace.civisibility.execution + +import datadog.trace.api.civisibility.execution.ExecutionAggregation +import datadog.trace.api.civisibility.execution.TestStatus +import datadog.trace.api.civisibility.telemetry.tag.RetryReason +import datadog.trace.civisibility.config.ExecutionsByDuration +import spock.lang.Specification + +class EarlyFlakeDetectionTest extends Specification { + + def "test EFD exits on flake"() { + setup: + def executionPolicy = new EarlyFlakeDetection([new ExecutionsByDuration(Long.MAX_VALUE, 3)], false) + + when: + def outcome = executionPolicy.registerExecution(TestStatus.fail, 0) + + then: + outcome.retryReason() == null + !outcome.lastExecution() + !outcome.failureSuppressed() + outcome.aggregation() == ExecutionAggregation.ONLY_FAILED + outcome.finalStatus() == null + + when: + def outcome2 = executionPolicy.registerExecution(TestStatus.pass, 0) + + then: + outcome2.retryReason() == RetryReason.efd + outcome2.lastExecution() + !outcome2.failureSuppressed() + outcome2.aggregation() == ExecutionAggregation.MIXED + outcome2.finalStatus() == TestStatus.fail + } + + def "test EFD failed all executions"() { + setup: + def executionPolicy = new EarlyFlakeDetection([new ExecutionsByDuration(Long.MAX_VALUE, 3)], false) + + when: + def outcome = executionPolicy.registerExecution(TestStatus.fail, 0) + + then: + outcome.retryReason() == null + !outcome.lastExecution() + !outcome.failureSuppressed() + outcome.aggregation() == ExecutionAggregation.ONLY_FAILED + outcome.finalStatus() == null + + when: + def outcome2 = executionPolicy.registerExecution(TestStatus.fail, 0) + + then: + outcome2.retryReason() == RetryReason.efd + !outcome2.lastExecution() + !outcome2.failureSuppressed() + outcome2.aggregation() == ExecutionAggregation.ONLY_FAILED + outcome2.finalStatus() == null + + when: + def outcome3 = executionPolicy.registerExecution(TestStatus.fail, 0) + + then: + outcome3.retryReason() == RetryReason.efd + outcome3.lastExecution() + !outcome3.failureSuppressed() + outcome3.aggregation() == ExecutionAggregation.ONLY_FAILED + outcome3.finalStatus() == TestStatus.fail + } + + def "test EFD succeeded all executions"() { + setup: + def executionPolicy = new EarlyFlakeDetection([new ExecutionsByDuration(Long.MAX_VALUE, 3)], false) + + when: + def outcome = executionPolicy.registerExecution(TestStatus.pass, 0) + + then: + outcome.retryReason() == null + !outcome.lastExecution() + !outcome.failureSuppressed() + outcome.aggregation() == ExecutionAggregation.ONLY_PASSED + outcome.finalStatus() == null + + when: + def outcome2 = executionPolicy.registerExecution(TestStatus.pass, 0) + + then: + outcome2.retryReason() == RetryReason.efd + !outcome2.lastExecution() + !outcome2.failureSuppressed() + outcome2.aggregation() == ExecutionAggregation.ONLY_PASSED + outcome2.finalStatus() == null + + when: + def outcome3 = executionPolicy.registerExecution(TestStatus.pass, 0) + + then: + outcome3.retryReason() == RetryReason.efd + outcome3.lastExecution() + !outcome3.failureSuppressed() + outcome3.aggregation() == ExecutionAggregation.ONLY_PASSED + outcome3.finalStatus() == TestStatus.pass + } + + def "test EFD adaptive retry count"() { + when: + def executionPolicy = new EarlyFlakeDetection([new ExecutionsByDuration(100, 3), new ExecutionsByDuration(Long.MAX_VALUE, 1)], false) + + then: + !executionPolicy.registerExecution(TestStatus.fail, 0).lastExecution() + !executionPolicy.registerExecution(TestStatus.fail, 0).lastExecution() + executionPolicy.registerExecution(TestStatus.fail, 0).lastExecution() + + when: + executionPolicy = new EarlyFlakeDetection([new ExecutionsByDuration(100, 3), new ExecutionsByDuration(Long.MAX_VALUE, 1)], false) + + then: + !executionPolicy.registerExecution(TestStatus.fail, 0).lastExecution() + executionPolicy.registerExecution(TestStatus.fail, 101).lastExecution() // exceed duration, go from "3 retries" bracket to "1 retry" bracket + } +} diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/QuarantineTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/QuarantineTest.groovy new file mode 100644 index 00000000000..a1e9894ca71 --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/QuarantineTest.groovy @@ -0,0 +1,38 @@ +package datadog.trace.civisibility.execution + +import datadog.trace.api.civisibility.execution.ExecutionAggregation +import datadog.trace.api.civisibility.execution.TestStatus +import spock.lang.Specification + +class QuarantineTest extends Specification { + + def "test quarantine passed"() { + setup: + def executionPolicy = new Quarantine() + + when: + def outcome = executionPolicy.registerExecution(TestStatus.pass, 0) + + then: + outcome.retryReason() == null + outcome.lastExecution() + !outcome.failureSuppressed() + outcome.aggregation() == ExecutionAggregation.ONLY_PASSED + outcome.finalStatus() == TestStatus.pass + } + + def "test quarantine failed"() { + setup: + def executionPolicy = new Quarantine() + + when: + def outcome = executionPolicy.registerExecution(TestStatus.fail, 0) + + then: + outcome.retryReason() == null + outcome.lastExecution() + outcome.failureSuppressed() + outcome.aggregation() == ExecutionAggregation.ONLY_FAILED + outcome.finalStatus() == TestStatus.pass + } +} diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/RegularExecutionTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/RegularExecutionTest.groovy index 9c614363ee1..60998961e30 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/RegularExecutionTest.groovy +++ b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/RegularExecutionTest.groovy @@ -1,5 +1,6 @@ package datadog.trace.civisibility.execution +import datadog.trace.api.civisibility.execution.ExecutionAggregation import datadog.trace.api.civisibility.execution.TestStatus import spock.lang.Specification @@ -10,13 +11,19 @@ class RegularExecutionTest extends Specification { def executionPolicy = Regular.INSTANCE when: - def outcome = executionPolicy.registerExecution(TestStatus.pass, 0) + def outcome = executionPolicy.registerExecution(status, 0) then: outcome.retryReason() == null - !outcome.lastExecution() + outcome.lastExecution() !outcome.failureSuppressed() - !outcome.failedAllRetries() - !outcome.succeededAllRetries() + outcome.aggregation() == expectedResults + outcome.finalStatus() == status + + where: + status | expectedResults + TestStatus.pass | ExecutionAggregation.ONLY_PASSED + TestStatus.fail | ExecutionAggregation.ONLY_FAILED + TestStatus.skip | ExecutionAggregation.ONLY_PASSED } } diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/RetryUntilSuccessfulTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/RetryUntilSuccessfulTest.groovy deleted file mode 100644 index f27c79c5895..00000000000 --- a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/RetryUntilSuccessfulTest.groovy +++ /dev/null @@ -1,155 +0,0 @@ -package datadog.trace.civisibility.execution - -import datadog.trace.api.civisibility.execution.TestStatus -import datadog.trace.api.civisibility.telemetry.tag.RetryReason -import spock.lang.Specification - -import java.util.concurrent.atomic.AtomicInteger - -class RetryUntilSuccessfulTest extends Specification { - - def "test retry until successful"() { - setup: - def executionPolicy = new RetryUntilSuccessful(3, false, new AtomicInteger()) - - when: - def outcome = executionPolicy.registerExecution(TestStatus.fail, 0) - - then: - outcome.retryReason() == null - !outcome.lastExecution() - outcome.failureSuppressed() - !outcome.failedAllRetries() - !outcome.succeededAllRetries() - - when: - def outcome2 = executionPolicy.registerExecution(TestStatus.pass, 0) - - then: - outcome2.retryReason() == RetryReason.atr - outcome2.lastExecution() - !outcome2.failureSuppressed() - !outcome2.failedAllRetries() - !outcome2.succeededAllRetries() - } - - def "test fail all retries"() { - setup: - def executionPolicy = new RetryUntilSuccessful(3, false, new AtomicInteger()) - - when: - def outcome = executionPolicy.registerExecution(TestStatus.fail, 0) - - then: - outcome.retryReason() == null - !outcome.lastExecution() - outcome.failureSuppressed() - !outcome.failedAllRetries() - !outcome.succeededAllRetries() - - when: - def outcome2 = executionPolicy.registerExecution(TestStatus.fail, 0) - - then: - outcome2.retryReason() == RetryReason.atr - !outcome2.lastExecution() - outcome2.failureSuppressed() - !outcome2.failedAllRetries() - !outcome2.succeededAllRetries() - - when: - def outcome3 = executionPolicy.registerExecution(TestStatus.fail, 0) - - then: - outcome3.retryReason() == RetryReason.atr - outcome3.lastExecution() - !outcome3.failureSuppressed() - outcome3.failedAllRetries() - !outcome2.succeededAllRetries() - } - - def "test succeed on last try"() { - setup: - def executionPolicy = new RetryUntilSuccessful(3, false, new AtomicInteger()) - - when: - def outcome = executionPolicy.registerExecution(TestStatus.fail, 0) - - then: - outcome.retryReason() == null - !outcome.lastExecution() - outcome.failureSuppressed() - !outcome.failedAllRetries() - !outcome.succeededAllRetries() - - when: - def outcome2 = executionPolicy.registerExecution(TestStatus.fail, 0) - - then: - outcome2.retryReason() == RetryReason.atr - !outcome2.lastExecution() - outcome2.failureSuppressed() - !outcome2.failedAllRetries() - !outcome2.succeededAllRetries() - - when: - def outcome3 = executionPolicy.registerExecution(TestStatus.pass, 0) - - then: - outcome3.retryReason() == RetryReason.atr - outcome3.lastExecution() - !outcome3.failureSuppressed() - !outcome3.failedAllRetries() - !outcome2.succeededAllRetries() - } - - def "test succeed on first try"() { - setup: - def executionPolicy = new RetryUntilSuccessful(3, false, new AtomicInteger()) - - when: - def outcome = executionPolicy.registerExecution(TestStatus.pass, 0) - - then: - outcome.retryReason() == null - outcome.lastExecution() - !outcome.failureSuppressed() - !outcome.failedAllRetries() - !outcome.succeededAllRetries() - } - - def "test suppress failures"() { - setup: - def executionPolicy = new RetryUntilSuccessful(3, true, new AtomicInteger()) - - when: - def outcome = executionPolicy.registerExecution(TestStatus.fail, 0) - - then: - outcome.retryReason() == null - !outcome.lastExecution() - outcome.failureSuppressed() - !outcome.failedAllRetries() - !outcome.succeededAllRetries() - - when: - def outcome2 = executionPolicy.registerExecution(TestStatus.fail, 0) - - then: - outcome2.retryReason() == RetryReason.atr - !outcome2.lastExecution() - outcome2.failureSuppressed() - !outcome2.failedAllRetries() - !outcome2.succeededAllRetries() - - when: - def outcome3 = executionPolicy.registerExecution(TestStatus.fail, 0) - - then: - outcome3.retryReason() == RetryReason.atr - outcome3.lastExecution() - outcome3.failureSuppressed() - outcome3.failedAllRetries() - !outcome2.succeededAllRetries() - } -} diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/RunNTimesTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/RunNTimesTest.groovy deleted file mode 100644 index c08f9e85798..00000000000 --- a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/RunNTimesTest.groovy +++ /dev/null @@ -1,166 +0,0 @@ -package datadog.trace.civisibility.execution - -import datadog.trace.api.civisibility.execution.TestStatus -import datadog.trace.api.civisibility.telemetry.tag.RetryReason -import datadog.trace.civisibility.config.ExecutionsByDuration -import spock.lang.Specification - -class RunNTimesTest extends Specification { - - def "test run N times"() { - setup: - def executionPolicy = new RunNTimes([new ExecutionsByDuration(Long.MAX_VALUE, 3)], false, RetryReason.efd) - - when: - def outcome = executionPolicy.registerExecution(TestStatus.fail, 0) - - then: - outcome.retryReason() == null - !outcome.lastExecution() - !outcome.failureSuppressed() - !outcome.failedAllRetries() - !outcome.succeededAllRetries() - - when: - def outcome2 = executionPolicy.registerExecution(TestStatus.pass, 0) - - then: - outcome2.retryReason() == RetryReason.efd - !outcome2.lastExecution() - !outcome2.failureSuppressed() - !outcome2.failedAllRetries() - !outcome2.succeededAllRetries() - - when: - def outcome3 = executionPolicy.registerExecution(TestStatus.fail, 0) - - then: - outcome3.retryReason() == RetryReason.efd - outcome3.lastExecution() - !outcome3.failureSuppressed() - !outcome3.failedAllRetries() - !outcome3.succeededAllRetries() - } - - def "test failed all retries"() { - setup: - def executionPolicy = new RunNTimes([new ExecutionsByDuration(Long.MAX_VALUE, 3)], false, RetryReason.efd) - - when: - def outcome = executionPolicy.registerExecution(TestStatus.fail, 0) - - then: - outcome.retryReason() == null - !outcome.lastExecution() - !outcome.failureSuppressed() - !outcome.failedAllRetries() - !outcome.succeededAllRetries() - - when: - def outcome2 = executionPolicy.registerExecution(TestStatus.fail, 0) - - then: - outcome2.retryReason() == RetryReason.efd - !outcome2.lastExecution() - !outcome2.failureSuppressed() - !outcome2.failedAllRetries() - !outcome2.succeededAllRetries() - - when: - def outcome3 = executionPolicy.registerExecution(TestStatus.fail, 0) - - then: - outcome3.retryReason() == RetryReason.efd - outcome3.lastExecution() - !outcome3.failureSuppressed() - outcome3.failedAllRetries() - !outcome3.succeededAllRetries() - } - - def "test succeeded all retries"() { - setup: - def executionPolicy = new RunNTimes([new ExecutionsByDuration(Long.MAX_VALUE, 3)], false, RetryReason.efd) - - when: - def outcome = executionPolicy.registerExecution(TestStatus.pass, 0) - - then: - outcome.retryReason() == null - !outcome.lastExecution() - !outcome.failureSuppressed() - !outcome.failedAllRetries() - !outcome.succeededAllRetries() - - when: - def outcome2 = executionPolicy.registerExecution(TestStatus.pass, 0) - - then: - outcome2.retryReason() == RetryReason.efd - !outcome2.lastExecution() - !outcome2.failureSuppressed() - !outcome2.failedAllRetries() - !outcome2.succeededAllRetries() - - when: - def outcome3 = executionPolicy.registerExecution(TestStatus.pass, 0) - - then: - outcome3.retryReason() == RetryReason.efd - outcome3.lastExecution() - !outcome3.failureSuppressed() - !outcome3.failedAllRetries() - outcome3.succeededAllRetries() - } - - def "test suppress failures"() { - setup: - def executionPolicy = new RunNTimes([new ExecutionsByDuration(Long.MAX_VALUE, 3)], true, RetryReason.attemptToFix) - - when: - def outcome = executionPolicy.registerExecution(TestStatus.fail, 0) - - then: - outcome.retryReason() == null - !outcome.lastExecution() - outcome.failureSuppressed() - !outcome.failedAllRetries() - !outcome.succeededAllRetries() - - when: - def outcome2 = executionPolicy.registerExecution(TestStatus.fail, 0) - - then: - outcome2.retryReason() == RetryReason.attemptToFix - !outcome2.lastExecution() - outcome2.failureSuppressed() - !outcome2.failedAllRetries() - !outcome2.succeededAllRetries() - - when: - def outcome3 = executionPolicy.registerExecution(TestStatus.fail, 0) - - then: - outcome3.retryReason() == RetryReason.attemptToFix - outcome3.lastExecution() - outcome3.failureSuppressed() - outcome3.failedAllRetries() - !outcome3.succeededAllRetries() - } - - def "test adaptive retry count"() { - when: - def executionPolicy = new RunNTimes([new ExecutionsByDuration(100, 3), new ExecutionsByDuration(Long.MAX_VALUE, 1)], true, RetryReason.efd) - - then: - !executionPolicy.registerExecution(TestStatus.fail, 0).lastExecution() - !executionPolicy.registerExecution(TestStatus.fail, 0).lastExecution() - executionPolicy.registerExecution(TestStatus.fail, 0).lastExecution() - - when: - executionPolicy = new RunNTimes([new ExecutionsByDuration(100, 3), new ExecutionsByDuration(Long.MAX_VALUE, 1)], true, RetryReason.efd) - - then: - !executionPolicy.registerExecution(TestStatus.fail, 0).lastExecution() - executionPolicy.registerExecution(TestStatus.fail, 101).lastExecution() // exceed duration, go from "3 retries" bracket to "1 retry" bracket - } -} diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/RunOnceIgnoreOutcomeTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/RunOnceIgnoreOutcomeTest.groovy deleted file mode 100644 index 766301f9d92..00000000000 --- a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/execution/RunOnceIgnoreOutcomeTest.groovy +++ /dev/null @@ -1,37 +0,0 @@ -package datadog.trace.civisibility.execution - -import datadog.trace.api.civisibility.execution.TestStatus -import spock.lang.Specification - -class RunOnceIgnoreOutcomeTest extends Specification { - - def "test run once ignore outcome"() { - setup: - def executionPolicy = new RunOnceIgnoreOutcome() - - when: - def outcome = executionPolicy.registerExecution(TestStatus.pass, 0) - - then: - outcome.retryReason() == null - outcome.lastExecution() - !outcome.failureSuppressed() - !outcome.failedAllRetries() - !outcome.succeededAllRetries() - } - - def "test run once ignore outcome failed"() { - setup: - def executionPolicy = new RunOnceIgnoreOutcome() - - when: - def outcome = executionPolicy.registerExecution(TestStatus.fail, 0) - - then: - outcome.retryReason() == null - outcome.lastExecution() - outcome.failureSuppressed() - !outcome.failedAllRetries() - !outcome.succeededAllRetries() - } -} diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/git/tree/GitClientTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/git/tree/GitClientTest.groovy index 479c3ce922c..024a2bd7437 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/git/tree/GitClientTest.groovy +++ b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/git/tree/GitClientTest.groovy @@ -23,6 +23,56 @@ class GitClientTest extends Specification { @TempDir private Path tempDir + def "test find git repo root with .git directory"() { + given: + givenGitRepo() + + when: + def repoRoot = ShellGitClient.findGitRepositoryRoot(tempDir.toFile()) + + then: + repoRoot == tempDir.toAbsolutePath().toString() + + when: + def subDir = tempDir.resolve("subdir") + Files.createDirectories(subDir) + repoRoot = ShellGitClient.findGitRepositoryRoot(subDir.toFile()) + + then: + repoRoot == tempDir.toAbsolutePath().toString() + } + + def "test find git repo root with .git file (worktree)"() { + given: + givenGitWorktree("ci/git/worktree") + + when: + def repoRoot = ShellGitClient.findGitRepositoryRoot(tempDir.toFile()) + + then: + repoRoot == tempDir.toAbsolutePath().toString() + + when: + def subDir = tempDir.resolve("subdir") + Files.createDirectories(subDir) + repoRoot = ShellGitClient.findGitRepositoryRoot(subDir.toFile()) + + then: + repoRoot == tempDir.toAbsolutePath().toString() + } + + def "test find git repo root defaults to original when no .git"() { + given: + def dirWithNoGit = tempDir.resolve("no_git_here") + Files.createDirectories(dirWithNoGit) + + when: + def repoRoot = ShellGitClient.findGitRepositoryRoot(dirWithNoGit.toFile()) + + then: + repoRoot == dirWithNoGit.toAbsolutePath().toString() + } + def "test is not shallow"() { given: givenGitRepo() @@ -37,10 +87,10 @@ class GitClientTest extends Specification { def "test is shallow"() { given: - givenGitRepo("ci/git/shallow/git") + givenGitRepos(["ci/git/shallow_with_origin/origin", "ci/git/shallow_with_origin/repo"]) when: - def gitClient = givenGitClient() + def gitClient = givenGitClient("repo") def shallow = gitClient.isShallow() then: @@ -61,22 +111,22 @@ class GitClientTest extends Specification { def "test get upstream branch SHA"() { given: - givenGitRepo("ci/git/shallow/git") + givenGitRepos(["ci/git/shallow_with_origin/origin", "ci/git/shallow_with_origin/repo"]) when: - def gitClient = givenGitClient() + def gitClient = givenGitClient("repo") def upstreamBranch = gitClient.getUpstreamBranchSha() then: - upstreamBranch == "98b944cc44f18bfb78e3021de2999cdcda8efdf6" + upstreamBranch == "c76ef954d23f8fdb42dcf2fe956d6af5a31fe7bd" } def "test unshallow: sha-#remoteSha"() { given: - givenGitRepo("ci/git/shallow/git") + givenGitRepos(["ci/git/shallow_with_origin/origin", "ci/git/shallow_with_origin/repo"]) when: - def gitClient = givenGitClient() + def gitClient = givenGitClient("repo") def shallow = gitClient.isShallow() def commits = gitClient.getLatestCommits() @@ -159,11 +209,11 @@ class GitClientTest extends Specification { def "test get commit info with fetching"() { given: - givenGitRepo("ci/git/shallow/git") + givenGitRepos(["ci/git/shallow_with_origin/origin", "ci/git/shallow_with_origin/repo"]) when: - def commit = "f4377e97f10c2d58696192b170b2fef2a8464b04" - def gitClient = givenGitClient() + def commit = "6e55a15a35ad46f74e4203dd42f7797173a6edcb" + def gitClient = givenGitClient("repo") def commitInfo = gitClient.getCommitInfo(commit, false) then: @@ -174,13 +224,13 @@ class GitClientTest extends Specification { then: commitInfo.sha == commit - commitInfo.author.name == "sullis" - commitInfo.author.email == "github@seansullivan.com" - commitInfo.author.iso8601Date == "2023-05-30T07:07:35-07:00" - commitInfo.committer.name == "GitHub" - commitInfo.committer.email == "noreply@github.com" - commitInfo.committer.iso8601Date == "2023-05-30T07:07:35-07:00" - commitInfo.fullMessage == "brotli4j 1.12.0 (#1592)" + commitInfo.author.name == "Test Author" + commitInfo.author.email == "test-author@example.com" + commitInfo.author.iso8601Date == "2026-03-12T17:02:46+01:00" + commitInfo.committer.name == "Test Author" + commitInfo.committer.email == "test-author@example.com" + commitInfo.committer.iso8601Date == "2026-03-12T17:02:46+01:00" + commitInfo.fullMessage == "Commit message 0" } def "test get latest commits"() { @@ -397,7 +447,7 @@ class GitClientTest extends Specification { sortedBranches == expectedOrder where: - metrics | expectedOrder + metrics | expectedOrder [ new ShellGitClient.BaseBranchMetric("main", 10, 2), new ShellGitClient.BaseBranchMetric("master", 15, 1), @@ -406,7 +456,7 @@ class GitClientTest extends Specification { new ShellGitClient.BaseBranchMetric("main", 10, 2), new ShellGitClient.BaseBranchMetric("master", 15, 2), new ShellGitClient.BaseBranchMetric("origin/main", 5, 2)] | ["main", "origin/main", "master"] - [] | [] + [] | [] } def "test get base branch sha: #testcaseName"() { @@ -429,6 +479,13 @@ class GitClientTest extends Specification { givenGitRepo("ci/git/with_pack/git") } + private void givenGitWorktree(String resourceName) { + // Worktree has a .git file (not directory) that points to the actual git dir + def gitFile = Paths.get(getClass().getClassLoader().getResource(resourceName + "/git").toURI()) + def tempGitFile = tempDir.resolve(GIT_FOLDER) + Files.copy(gitFile, tempGitFile) + } + private void givenGitRepo(String resourceName) { def gitFolder = Paths.get(getClass().getClassLoader().getResource(resourceName).toURI()) def tempGitFolder = tempDir.resolve(GIT_FOLDER) diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/source/BestEffortSourcePathResolverTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/source/BestEffortSourcePathResolverTest.groovy index 23ae4fd48e5..4568b0e6104 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/source/BestEffortSourcePathResolverTest.groovy +++ b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/source/BestEffortSourcePathResolverTest.groovy @@ -5,54 +5,54 @@ import spock.lang.Specification class BestEffortSourcePathResolverTest extends Specification { - def "test get source info from delegate"() { + def "test get source paths from first delegate"() { setup: - def expectedPath = "source/path/TestClass.java" + def expectedPaths = ["source/path/TestClass.java"] def delegate = Stub(SourcePathResolver) def secondDelegate = Stub(SourcePathResolver) def resolver = new BestEffortSourcePathResolver(delegate, secondDelegate) - delegate.getSourcePath(TestClass) >> expectedPath - secondDelegate.getSourcePath(TestClass) >> null + delegate.getSourcePaths(TestClass) >> expectedPaths + secondDelegate.getSourcePaths(TestClass) >> [] when: - def path = resolver.getSourcePath(TestClass) + def paths = resolver.getSourcePaths(TestClass) then: - path == expectedPath + paths == expectedPaths } - def "test get source info from second delegate"() { + def "test get source paths from second delegate when first returns empty"() { setup: - def expectedPath = "source/path/TestClass.java" + def expectedPaths = ["debug/path/TestClass.java", "release/path/TestClass.java"] def delegate = Stub(SourcePathResolver) def secondDelegate = Stub(SourcePathResolver) def resolver = new BestEffortSourcePathResolver(delegate, secondDelegate) - delegate.getSourcePath(TestClass) >> null - secondDelegate.getSourcePath(TestClass) >> expectedPath + delegate.getSourcePaths(TestClass) >> [] + secondDelegate.getSourcePaths(TestClass) >> expectedPaths when: - def path = resolver.getSourcePath(TestClass) + def paths = resolver.getSourcePaths(TestClass) then: - path == expectedPath + paths == expectedPaths } - def "test failed to get info from both delegates"() { + def "test failed to get source paths from both delegates"() { setup: def delegate = Stub(SourcePathResolver) def secondDelegate = Stub(SourcePathResolver) def resolver = new BestEffortSourcePathResolver(delegate, secondDelegate) - delegate.getSourcePath(TestClass) >> null - secondDelegate.getSourcePath(TestClass) >> null + delegate.getSourcePaths(TestClass) >> [] + secondDelegate.getSourcePaths(TestClass) >> [] when: - def path = resolver.getSourcePath(TestClass) + def paths = resolver.getSourcePaths(TestClass) then: - path == null + paths.isEmpty() } private static final class TestClass {} diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/source/CompilerAidedSourcePathResolverTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/source/CompilerAidedSourcePathResolverTest.groovy index 77b563fd96c..c1d516b2256 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/source/CompilerAidedSourcePathResolverTest.groovy +++ b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/source/CompilerAidedSourcePathResolverTest.groovy @@ -14,16 +14,17 @@ class CompilerAidedSourcePathResolverTest extends Specification { def sourcePathResolver = new CompilerAidedSourcePathResolver(REPO_ROOT) when: - def path = sourcePathResolver.getSourcePath(clazz) + def path = sourcePathResolver.getSourcePaths(clazz) then: - path == expectedPath + path.size() == expectedPath.size() + path.containsAll(expectedPath) where: clazz | expectedPath - AClassWithNoSourceInfoInjected | null - AClassWithSourceInfoInjected | "path/to/AClassWithSourceInfoInjected.java" - AClassWithSourceOutsideRepository | null + AClassWithNoSourceInfoInjected | [] + AClassWithSourceInfoInjected | ["path/to/AClassWithSourceInfoInjected.java"] + AClassWithSourceOutsideRepository | [] } private static final class AClassWithNoSourceInfoInjected {} diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/source/index/RepoIndexSourcePathResolverTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/source/index/RepoIndexSourcePathResolverTest.groovy index 108c0020065..df61ba11170 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/source/index/RepoIndexSourcePathResolverTest.groovy +++ b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/source/index/RepoIndexSourcePathResolverTest.groovy @@ -4,7 +4,6 @@ import com.google.common.jimfs.Configuration import com.google.common.jimfs.Jimfs import datadog.trace.api.Config import datadog.trace.api.civisibility.domain.Language -import datadog.trace.civisibility.source.SourceResolutionException import groovy.transform.PackageScope import spock.lang.Specification @@ -25,9 +24,11 @@ class RepoIndexSourcePathResolverTest extends Specification { when: def sourcePathResolver = new RepoIndexSourcePathResolver(new RepoIndexBuilder(config, repoRoot, packageResolver, resourceResolver, fileSystem)) + def sourcePaths = sourcePathResolver.getSourcePaths(RepoIndexSourcePathResolverTest) then: - sourcePathResolver.getSourcePath(RepoIndexSourcePathResolverTest) == expectedSourcePath + sourcePaths.size() == 1 + sourcePaths.contains(expectedSourcePath) } def "test source path resolution for inner class"() { @@ -36,9 +37,11 @@ class RepoIndexSourcePathResolverTest extends Specification { when: def sourcePathResolver = new RepoIndexSourcePathResolver(new RepoIndexBuilder(config, repoRoot, packageResolver, resourceResolver, fileSystem)) + def sourcePaths = sourcePathResolver.getSourcePaths(InnerClass) then: - sourcePathResolver.getSourcePath(InnerClass) == expectedSourcePath + sourcePaths.size() == 1 + sourcePaths.contains(expectedSourcePath) } def "test source path resolution for nested inner class"() { @@ -47,9 +50,11 @@ class RepoIndexSourcePathResolverTest extends Specification { when: def sourcePathResolver = new RepoIndexSourcePathResolver(new RepoIndexBuilder(config, repoRoot, packageResolver, resourceResolver, fileSystem)) + def sourcePaths = sourcePathResolver.getSourcePaths(InnerClass.NestedInnerClass) then: - sourcePathResolver.getSourcePath(InnerClass.NestedInnerClass) == expectedSourcePath + sourcePaths.size() == 1 + sourcePaths.contains(expectedSourcePath) } def "test source path resolution for anonymous class"() { @@ -61,9 +66,11 @@ class RepoIndexSourcePathResolverTest extends Specification { def r = new Runnable() { void run() {} } + def sourcePaths = sourcePathResolver.getSourcePaths(r.getClass()) then: - sourcePathResolver.getSourcePath(r.getClass()) == expectedSourcePath + sourcePaths.size() == 1 + sourcePaths.contains(expectedSourcePath) } def "test source path resolution for package-private class"() { @@ -72,9 +79,11 @@ class RepoIndexSourcePathResolverTest extends Specification { when: def sourcePathResolver = new RepoIndexSourcePathResolver(new RepoIndexBuilder(config, repoRoot, packageResolver, resourceResolver, fileSystem)) + def sourcePaths = sourcePathResolver.getSourcePaths(PackagePrivateClass) then: - sourcePathResolver.getSourcePath(PackagePrivateClass) == expectedSourcePath + sourcePaths.size() == 1 + sourcePaths.contains(expectedSourcePath) } def "test source path resolution for class nested into package-private class"() { @@ -83,9 +92,11 @@ class RepoIndexSourcePathResolverTest extends Specification { when: def sourcePathResolver = new RepoIndexSourcePathResolver(new RepoIndexBuilder(config, repoRoot, packageResolver, resourceResolver, fileSystem)) + def sourcePaths = sourcePathResolver.getSourcePaths(PackagePrivateClass.NestedIntoPackagePrivateClass) then: - sourcePathResolver.getSourcePath(PackagePrivateClass.NestedIntoPackagePrivateClass) == expectedSourcePath + sourcePaths.size() == 1 + sourcePaths.contains(expectedSourcePath) } def "test source path resolution for non-java class whose file name is different from class name"() { @@ -94,9 +105,11 @@ class RepoIndexSourcePathResolverTest extends Specification { when: def sourcePathResolver = new RepoIndexSourcePathResolver(new RepoIndexBuilder(config, repoRoot, packageResolver, resourceResolver, fileSystem)) + def sourcePaths = sourcePathResolver.getSourcePaths(PublicClassWhoseNameDoesNotCorrespondToFileName) then: - sourcePathResolver.getSourcePath(PublicClassWhoseNameDoesNotCorrespondToFileName) == expectedSourcePath + sourcePaths.size() == 1 + sourcePaths.contains(expectedSourcePath) } def "test source path for non-indexed class"() { @@ -106,7 +119,7 @@ class RepoIndexSourcePathResolverTest extends Specification { def sourcePathResolver = new RepoIndexSourcePathResolver(new RepoIndexBuilder(config, repoRoot, packageResolver, resourceResolver, fileSystem)) then: - sourcePathResolver.getSourcePath(RepoIndexSourcePathResolver) == null + sourcePathResolver.getSourcePaths(RepoIndexSourcePathResolver).isEmpty() } def "test source path for non-indexed package-private class"() { @@ -116,7 +129,7 @@ class RepoIndexSourcePathResolverTest extends Specification { def sourcePathResolver = new RepoIndexSourcePathResolver(new RepoIndexBuilder(config, repoRoot, packageResolver, resourceResolver, fileSystem)) then: - sourcePathResolver.getSourcePath(PackagePrivateClass) == null + sourcePathResolver.getSourcePaths(PackagePrivateClass).isEmpty() } def "test file-indexing failure"() { @@ -131,7 +144,7 @@ class RepoIndexSourcePathResolverTest extends Specification { def sourcePathResolver = new RepoIndexSourcePathResolver(new RepoIndexBuilder(config, repoRoot, packageResolver, resourceResolver, fileSystem)) then: - sourcePathResolver.getSourcePath(RepoIndexSourcePathResolverTest) == null + sourcePathResolver.getSourcePaths(RepoIndexSourcePathResolverTest).isEmpty() } def "test source path resolution for repo with multiple files"() { @@ -143,25 +156,32 @@ class RepoIndexSourcePathResolverTest extends Specification { when: def sourcePathResolver = new RepoIndexSourcePathResolver(new RepoIndexBuilder(config, repoRoot, packageResolver, resourceResolver, fileSystem)) + def sourcePathsOne = sourcePathResolver.getSourcePaths(RepoIndexSourcePathResolverTest) + def sourcePathsTwo = sourcePathResolver.getSourcePaths(InnerClass) + def sourcePathsThree = sourcePathResolver.getSourcePaths(RepoIndexSourcePathResolver) then: - sourcePathResolver.getSourcePath(RepoIndexSourcePathResolverTest) == expectedSourcePathOne - sourcePathResolver.getSourcePath(InnerClass) == expectedSourcePathOne - sourcePathResolver.getSourcePath(RepoIndexSourcePathResolver) == expectedSourcePathTwo + sourcePathsOne.size() == 1 + sourcePathsOne.contains(expectedSourcePathOne) + sourcePathsTwo.size() == 1 + sourcePathsTwo.contains(expectedSourcePathOne) + sourcePathsThree.size() == 1 + sourcePathsThree.contains(expectedSourcePathTwo) } - def "test trying to resolve a duplicate key throws exception"() { + def "test trying to resolve a duplicate key returns both candidates"() { setup: - givenSourceFile(RepoIndexSourcePathResolverTest, repoRoot + "/src/java") - givenSourceFile(RepoIndexSourcePathResolverTest, repoRoot + "/src/scala") + def expectedJavaPath = givenSourceFile(RepoIndexSourcePathResolverTest, repoRoot + "/src/java") + def expectedScalaPath = givenSourceFile(RepoIndexSourcePathResolverTest, repoRoot + "/src/scala") def sourcePathResolver = new RepoIndexSourcePathResolver(new RepoIndexBuilder(config, repoRoot, packageResolver, resourceResolver, fileSystem)) when: - sourcePathResolver.getSourcePath(RepoIndexSourcePathResolverTest) + def sourcePaths = sourcePathResolver.getSourcePaths(RepoIndexSourcePathResolverTest) then: - thrown SourceResolutionException + sourcePaths.size() == 2 + sourcePaths.containsAll([expectedJavaPath, expectedScalaPath]) } private String givenSourceFile(Class c, String sourceRoot, Language language = Language.GROOVY) { diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/source/index/RepoIndexTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/source/index/RepoIndexTest.groovy index d8b884a53f4..70da9c5780a 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/source/index/RepoIndexTest.groovy +++ b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/source/index/RepoIndexTest.groovy @@ -1,8 +1,7 @@ package datadog.trace.civisibility.source.index +import datadog.instrument.utils.ClassNameTrie import datadog.trace.api.civisibility.domain.Language -import datadog.trace.civisibility.source.SourceResolutionException -import datadog.trace.util.ClassNameTrie import spock.lang.Specification class RepoIndexTest extends Specification { @@ -21,26 +20,95 @@ class RepoIndexTest extends Specification { new RepoIndex.SourceRoot("myClassSourceRoot", Language.GROOVY), new RepoIndex.SourceRoot("myOtherClassSourceRoot", Language.GROOVY)) - def repoIndex = new RepoIndex(trie, Collections.emptyList(), sourceRoots, Collections.emptyList()) + def repoIndex = new RepoIndex(trie, Collections.emptyMap(), sourceRoots, Collections.emptyList()) when: def serialized = repoIndex.serialize() def deserialized = RepoIndex.deserialize(serialized) then: - deserialized.getSourcePath(RepoIndexTest) == "myClassSourceRoot/" + myClassName.replace('.' as char, File.separatorChar) + Language.GROOVY.extension - deserialized.getSourcePath(RepoIndexSourcePathResolverTest) == "myOtherClassSourceRoot/" + myOtherClassName.replace('.' as char, File.separatorChar) + Language.GROOVY.extension + deserialized.getSourcePaths(RepoIndexTest).size() == 1 + deserialized.getSourcePaths(RepoIndexTest) .contains("myClassSourceRoot/" + myClassName.replace('.' as char, File.separatorChar) + Language.GROOVY.extension) + deserialized.getSourcePaths(RepoIndexSourcePathResolverTest).size() == 1 + deserialized.getSourcePaths(RepoIndexSourcePathResolverTest).contains("myOtherClassSourceRoot/" + myOtherClassName.replace('.' as char, File.separatorChar) + Language.GROOVY.extension) } - def "test trying to resolve a duplicate key throws exception"() { + def "test serialization and deserialization with duplicate keys"() { given: - def duplicateKeys = [RepoIndexTest.name] - def repoIndex = new RepoIndex(new ClassNameTrie.Builder().buildTrie(), duplicateKeys, Collections.emptyList(), Collections.emptyList()) + def myClassName = RepoIndexTest.name + + def trieBuilder = new ClassNameTrie.Builder() + trieBuilder.put(myClassName, 0) + def trie = trieBuilder.buildTrie() + + def sourceRoots = Arrays.asList( + new RepoIndex.SourceRoot("sourceRoot1", Language.GROOVY), + new RepoIndex.SourceRoot("sourceRoot2", Language.GROOVY)) + + def duplicateKeys = [(myClassName): [ + "sourceRoot1/" + myClassName.replace('.' as char, File.separatorChar) + Language.GROOVY.extension, + "sourceRoot2/" + myClassName.replace('.' as char, File.separatorChar) + Language.GROOVY.extension + ]] + + def repoIndex = new RepoIndex(trie, duplicateKeys, sourceRoots, Collections.emptyList()) + + when: + def serialized = repoIndex.serialize() + def deserialized = RepoIndex.deserialize(serialized) + + then: + def paths = deserialized.getSourcePaths(RepoIndexTest) + paths.size() == 2 + paths.containsAll([ + "sourceRoot1/" + myClassName.replace('.' as char, File.separatorChar) + Language.GROOVY.extension, + "sourceRoot2/" + myClassName.replace('.' as char, File.separatorChar) + Language.GROOVY.extension + ]) + } + + def "test getSourcePaths returns all paths for duplicate key"() { + given: + def myClassName = RepoIndexTest.name + + def trieBuilder = new ClassNameTrie.Builder() + trieBuilder.put(myClassName, 0) + def trie = trieBuilder.buildTrie() + + def sourceRoots = Arrays.asList( + new RepoIndex.SourceRoot("debug", Language.GROOVY), + new RepoIndex.SourceRoot("release", Language.GROOVY)) + + def expectedPath1 = "debug/" + myClassName.replace('.' as char, File.separatorChar) + Language.GROOVY.extension + def expectedPath2 = "release/" + myClassName.replace('.' as char, File.separatorChar) + Language.GROOVY.extension + def duplicateKeys = [(myClassName): [expectedPath1, expectedPath2]] + + def repoIndex = new RepoIndex(trie, duplicateKeys, sourceRoots, Collections.emptyList()) + + when: + def paths = repoIndex.getSourcePaths(RepoIndexTest) + + then: + paths.size() == 2 + paths.containsAll([expectedPath1, expectedPath2]) + } + + def "test getSourcePaths returns single path for non-duplicate key"() { + given: + def myClassName = RepoIndexTest.name + + def trieBuilder = new ClassNameTrie.Builder() + trieBuilder.put(myClassName, 0) + def trie = trieBuilder.buildTrie() + + def sourceRoots = Arrays.asList( + new RepoIndex.SourceRoot("src/main/groovy", Language.GROOVY)) + + def repoIndex = new RepoIndex(trie, Collections.emptyMap(), sourceRoots, Collections.emptyList()) when: - repoIndex.getSourcePath(RepoIndexTest) + def paths = repoIndex.getSourcePaths(RepoIndexTest) then: - thrown SourceResolutionException + paths.size() == 1 + paths.first() == "src/main/groovy/" + myClassName.replace('.' as char, File.separatorChar) + Language.GROOVY.extension } } diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/telemetry/CiVisibilityMetricCollectorTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/telemetry/CiVisibilityMetricCollectorTest.groovy index 9fa8c47209b..11cb162befa 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/telemetry/CiVisibilityMetricCollectorTest.groovy +++ b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/telemetry/CiVisibilityMetricCollectorTest.groovy @@ -149,19 +149,20 @@ class CiVisibilityMetricCollectorTest extends Specification { } /** - * This test enumerates all possible metric+tags variants, - * then tries submitting all possible variant pairs (combinations of 2 different metric+tags). + * This test enumerates a few different tag combinations for every metric, + * then submits a metric count for each one. * The goal is to ensure that index calculation logic and card-marking are done right. */ - def "test submission of all possible count metric pairs"() { + def "test submission of different count metric pairs"() { setup: List possibleMetrics = [] for (CiVisibilityCountMetric metric : CiVisibilityCountMetric.values()) { def metricTags = metric.getTags() - int cartesianProductSizeLimit = 2000 // limiting the number of combinations to avoid OOM/timeout - for (TagValue[] tags : cartesianProduct(metricTags, cartesianProductSizeLimit)) { // iterate over combinations of metric tags + int cartesianProductSizeLimit = 20 // limiting the number of combinations to avoid OOM/timeout + for (TagValue[] tags : cartesianProduct(metricTags, cartesianProductSizeLimit)) { + // iterate over combinations of metric tags possibleMetrics += new PossibleMetric(metric, tags) } } @@ -231,5 +232,4 @@ class CiVisibilityMetricCollectorTest extends Specification { this.tags = tags } } - } diff --git a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/test/ExecutionStrategyTest.groovy b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/test/ExecutionStrategyTest.groovy index b1455fd8118..e01e57315fc 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/test/ExecutionStrategyTest.groovy +++ b/dd-java-agent/agent-ci-visibility/src/test/groovy/datadog/trace/civisibility/test/ExecutionStrategyTest.groovy @@ -10,9 +10,8 @@ import datadog.trace.api.civisibility.telemetry.tag.RetryReason import datadog.trace.api.civisibility.telemetry.tag.SkipReason import datadog.trace.civisibility.config.EarlyFlakeDetectionSettings import datadog.trace.civisibility.config.ExecutionSettings -import datadog.trace.civisibility.config.ExecutionsByDuration import datadog.trace.civisibility.config.TestManagementSettings -import datadog.trace.civisibility.execution.RunNTimes +import datadog.trace.civisibility.execution.AttemptToFix import datadog.trace.civisibility.source.LinesResolver import datadog.trace.civisibility.source.SourcePathResolver import spock.lang.Specification @@ -76,7 +75,7 @@ class ExecutionStrategyTest extends Specification { def strategy = givenAnExecutionStrategy(executionSettings) expect: - strategy.executionPolicy(testID, TestSourceData.UNKNOWN, []).class == RunNTimes + strategy.executionPolicy(testID, TestSourceData.UNKNOWN, []).class == AttemptToFix } def "test attempt to fix + efd"() { @@ -86,7 +85,7 @@ class ExecutionStrategyTest extends Specification { def testManagementSettings = Stub(TestManagementSettings) testManagementSettings.isEnabled() >> true - testManagementSettings.getAttemptToFixExecutions() >> Collections.singletonList(new ExecutionsByDuration(Long.MAX_VALUE, 20)) + testManagementSettings.getAttemptToFixRetries() >> 20 def earlyFlakeDetectionSettings = Stub(EarlyFlakeDetectionSettings) earlyFlakeDetectionSettings.isEnabled() >> true @@ -103,7 +102,7 @@ class ExecutionStrategyTest extends Specification { def policy = strategy.executionPolicy(testID, TestSourceData.UNKNOWN, []) then: - policy.class == RunNTimes + policy.class == AttemptToFix when: def outcome = policy.registerExecution(TestStatus.pass, 0) diff --git a/dd-java-agent/agent-ci-visibility/src/test/java/datadog/trace/civisibility/coverage/file/FileCoverageStoreTest.java b/dd-java-agent/agent-ci-visibility/src/test/java/datadog/trace/civisibility/coverage/file/FileCoverageStoreTest.java new file mode 100644 index 00000000000..fef10b98639 --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/java/datadog/trace/civisibility/coverage/file/FileCoverageStoreTest.java @@ -0,0 +1,97 @@ +package datadog.trace.civisibility.coverage.file; + +import static java.util.Arrays.asList; +import static java.util.Collections.emptyList; +import static java.util.Collections.singletonList; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +import datadog.trace.api.DDTraceId; +import datadog.trace.api.civisibility.coverage.CoverageStore; +import datadog.trace.api.civisibility.coverage.TestReport; +import datadog.trace.api.civisibility.telemetry.CiVisibilityMetricCollector; +import datadog.trace.civisibility.source.SourcePathResolver; +import org.junit.jupiter.api.Test; + +class FileCoverageStoreTest { + + private static final class ResolvableClassA {} + + private static final class DuplicateKeyClass {} + + private static final class ResolvableClassC {} + + @Test + void duplicateKeyClassReturnsAllCandidatePathsInCoverageReport() { + CiVisibilityMetricCollector metrics = mock(CiVisibilityMetricCollector.class); + SourcePathResolver sourcePathResolver = mock(SourcePathResolver.class); + when(sourcePathResolver.getSourcePaths(ResolvableClassA.class)) + .thenReturn(singletonList("src/main/java/com/example/ClassA.java")); + when(sourcePathResolver.getSourcePaths(DuplicateKeyClass.class)) + .thenReturn( + asList( + "src/debug/java/com/example/DuplicateKeyClass.java", + "src/release/java/com/example/DuplicateKeyClass.java")); + when(sourcePathResolver.getSourcePaths(ResolvableClassC.class)) + .thenReturn(singletonList("src/main/java/com/example/ClassC.java")); + + CoverageStore store = new FileCoverageStore.Factory(metrics, sourcePathResolver).create(null); + store.getProbes().record(ResolvableClassA.class); + store.getProbes().record(DuplicateKeyClass.class); + store.getProbes().record(ResolvableClassC.class); + + boolean result = store.report(DDTraceId.ONE, 1L, 1L); + + assertTrue(result); + TestReport report = store.getReport(); + assertNotNull(report); + assertEquals(4, report.getTestReportFileEntries().size()); + } + + @Test + void coverageReportSucceedsForNonDuplicateClasses() { + CiVisibilityMetricCollector metrics = mock(CiVisibilityMetricCollector.class); + SourcePathResolver sourcePathResolver = mock(SourcePathResolver.class); + when(sourcePathResolver.getSourcePaths(ResolvableClassA.class)) + .thenReturn(singletonList("src/main/java/com/example/ClassA.java")); + when(sourcePathResolver.getSourcePaths(ResolvableClassC.class)) + .thenReturn(singletonList("src/main/java/com/example/ClassC.java")); + + CoverageStore store = new FileCoverageStore.Factory(metrics, sourcePathResolver).create(null); + store.getProbes().record(ResolvableClassA.class); + store.getProbes().record(ResolvableClassC.class); + + boolean result = store.report(DDTraceId.ONE, 1L, 1L); + + assertTrue(result); + TestReport report = store.getReport(); + assertNotNull(report); + assertEquals(2, report.getTestReportFileEntries().size()); + } + + @Test + void emptySourcePathsForOneClassDoesNotKillCoverageReport() { + CiVisibilityMetricCollector metrics = mock(CiVisibilityMetricCollector.class); + SourcePathResolver sourcePathResolver = mock(SourcePathResolver.class); + when(sourcePathResolver.getSourcePaths(ResolvableClassA.class)) + .thenReturn(singletonList("src/main/java/com/example/ClassA.java")); + when(sourcePathResolver.getSourcePaths(DuplicateKeyClass.class)).thenReturn(emptyList()); + when(sourcePathResolver.getSourcePaths(ResolvableClassC.class)) + .thenReturn(singletonList("src/main/java/com/example/ClassC.java")); + + CoverageStore store = new FileCoverageStore.Factory(metrics, sourcePathResolver).create(null); + store.getProbes().record(ResolvableClassA.class); + store.getProbes().record(DuplicateKeyClass.class); + store.getProbes().record(ResolvableClassC.class); + + boolean result = store.report(DDTraceId.ONE, 1L, 1L); + + assertTrue(result); + TestReport report = store.getReport(); + assertNotNull(report); + assertEquals(2, report.getTestReportFileEntries().size()); + } +} diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/dummy.txt b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/dummy.txt deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/config b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/config deleted file mode 100644 index f145b18c5c0..00000000000 --- a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/config +++ /dev/null @@ -1,13 +0,0 @@ -[core] - repositoryformatversion = 0 - filemode = true - bare = false - logallrefupdates = true - ignorecase = true - precomposeunicode = true -[remote "origin"] - url = https://github.com/Netflix/zuul.git - fetch = +refs/heads/master:refs/remotes/origin/master -[branch "master"] - remote = origin - merge = refs/heads/master diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/description b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/description deleted file mode 100644 index 498b267a8c7..00000000000 --- a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/description +++ /dev/null @@ -1 +0,0 @@ -Unnamed repository; edit this file 'description' to name the repository. diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/index b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/index deleted file mode 100644 index 7e85fb0abe6..00000000000 Binary files a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/index and /dev/null differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/info/exclude b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/info/exclude deleted file mode 100644 index a5196d1be8f..00000000000 --- a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/info/exclude +++ /dev/null @@ -1,6 +0,0 @@ -# git ls-files --others --exclude-from=.git/info/exclude -# Lines that start with '#' are comments. -# For a project mostly in C, the following would be a good set of -# exclude patterns (uncomment them if you want to use them): -# *.[oa] -# *~ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/logs/HEAD b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/logs/HEAD deleted file mode 100644 index 9527e7281e1..00000000000 --- a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/logs/HEAD +++ /dev/null @@ -1 +0,0 @@ -0000000000000000000000000000000000000000 98b944cc44f18bfb78e3021de2999cdcda8efdf6 Nikita Tkachenko 1687280379 +0200 clone: from github.com:Netflix/zuul.git diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/logs/refs/heads/master b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/logs/refs/heads/master deleted file mode 100644 index 9527e7281e1..00000000000 --- a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/logs/refs/heads/master +++ /dev/null @@ -1 +0,0 @@ -0000000000000000000000000000000000000000 98b944cc44f18bfb78e3021de2999cdcda8efdf6 Nikita Tkachenko 1687280379 +0200 clone: from github.com:Netflix/zuul.git diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/logs/refs/remotes/origin/HEAD b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/logs/refs/remotes/origin/HEAD deleted file mode 100644 index 9527e7281e1..00000000000 --- a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/logs/refs/remotes/origin/HEAD +++ /dev/null @@ -1 +0,0 @@ -0000000000000000000000000000000000000000 98b944cc44f18bfb78e3021de2999cdcda8efdf6 Nikita Tkachenko 1687280379 +0200 clone: from github.com:Netflix/zuul.git diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/objects/pack/pack-5d8e32f4728b0c4a1ce9a1df8abf396036e42acb.idx b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/objects/pack/pack-5d8e32f4728b0c4a1ce9a1df8abf396036e42acb.idx deleted file mode 100644 index 893dbbdd5e3..00000000000 Binary files a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/objects/pack/pack-5d8e32f4728b0c4a1ce9a1df8abf396036e42acb.idx and /dev/null differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/objects/pack/pack-5d8e32f4728b0c4a1ce9a1df8abf396036e42acb.pack b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/objects/pack/pack-5d8e32f4728b0c4a1ce9a1df8abf396036e42acb.pack deleted file mode 100644 index 31240eda865..00000000000 Binary files a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/objects/pack/pack-5d8e32f4728b0c4a1ce9a1df8abf396036e42acb.pack and /dev/null differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/packed-refs b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/packed-refs deleted file mode 100644 index 76b054de75a..00000000000 --- a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/packed-refs +++ /dev/null @@ -1,2 +0,0 @@ -# pack-refs with: peeled fully-peeled sorted -98b944cc44f18bfb78e3021de2999cdcda8efdf6 refs/remotes/origin/master diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/refs/heads/master b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/refs/heads/master deleted file mode 100644 index e9cca1fc9e8..00000000000 --- a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/refs/heads/master +++ /dev/null @@ -1 +0,0 @@ -98b944cc44f18bfb78e3021de2999cdcda8efdf6 diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/refs/remotes/origin/HEAD b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/refs/remotes/origin/HEAD deleted file mode 100644 index 6efe28fff83..00000000000 --- a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/refs/remotes/origin/HEAD +++ /dev/null @@ -1 +0,0 @@ -ref: refs/remotes/origin/master diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/shallow b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/shallow deleted file mode 100644 index e9cca1fc9e8..00000000000 --- a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/shallow +++ /dev/null @@ -1 +0,0 @@ -98b944cc44f18bfb78e3021de2999cdcda8efdf6 diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/build_script.txt b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/build_script.txt new file mode 100644 index 00000000000..9522bb95bdd --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/build_script.txt @@ -0,0 +1,79 @@ +#!/bin/bash +set -e + +# IMPORTANT: always use relative paths, as tests move the folders to temp dirs +base_path="/tmp/shallow_with_origin/" +origin_path="origin" +repo_path="repo" + +rm -rf $base_path +mkdir -p $base_path +cd $base_path + +# create bare origin +mkdir -p $origin_path +cd $origin_path && git init --bare --initial-branch=master +cd .. + +# create working copy to populate origin with commits +mkdir -p work && cd work +git init --initial-branch=master +git remote add origin "../$origin_path" +git config user.email "test-author@example.com" +git config user.name "Test Author" + +# create 10 commits +for i in $(seq 0 9); do + echo "content $i" >> "file${i}.txt" + git add . + git commit -m "Commit message ${i}" +done + +git push origin master +cd .. + +# create shallow clone from origin (use file:// URL so --depth works, fix config after) +git clone --depth 1 "file://$base_path/$origin_path" $repo_path + +# print commit SHAs for reference +echo "=== COMMIT SHAs ===" +cd work +git log --format="%H %s" --reverse +first_commit=$(git log --format="%H" --reverse | head -1) +echo "=== FIRST COMMIT (not in shallow clone): $first_commit ===" +cd .. + +echo "=== SHALLOW CLONE HEAD ===" +cd $repo_path +git log --format="%H %s" +echo "=== IS SHALLOW ===" +git rev-parse --is-shallow-repository +cd .. + +# cleanup origin (remove unnecessary files) +(cd $origin_path && rm -rf hooks info logs COMMIT_EDITMSG description) + +# cleanup working repo (shallow clone) - rename .git to git +(cd $repo_path && rm -rf .git/hooks .git/info .git/logs .git/COMMIT_EDITMSG .git/description .git/index && mv .git git) + +# update the repo config to use relative path (git clone uses absolute) +cd $repo_path/git +# Replace the absolute URL with relative path +python3 -c " +import re +with open('config', 'r') as f: + content = f.read() +content = re.sub(r'url = .*/origin', 'url = ../origin', content) +with open('config', 'w') as f: + f.write(content) +" +cd ../.. + +echo "=== ORIGIN CONFIG ===" +cat $origin_path/config +echo "" +echo "=== REPO CONFIG ===" +cat $repo_path/git/config +echo "" +echo "=== DONE ===" +echo "Resources at: $base_path" diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/HEAD b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/HEAD similarity index 100% rename from dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow/git/HEAD rename to dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/HEAD diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/config b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/config new file mode 100644 index 00000000000..e6da231579b --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/config @@ -0,0 +1,6 @@ +[core] + repositoryformatversion = 0 + filemode = true + bare = true + ignorecase = true + precomposeunicode = true diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/03/5f9b742ebf552ed87f003d4944480bfea6ba99 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/03/5f9b742ebf552ed87f003d4944480bfea6ba99 new file mode 100644 index 00000000000..28f856e0422 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/03/5f9b742ebf552ed87f003d4944480bfea6ba99 differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/04/91fb935b55dd8540bb73107cf3bb8e87295f38 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/04/91fb935b55dd8540bb73107cf3bb8e87295f38 new file mode 100644 index 00000000000..68efbd02c53 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/04/91fb935b55dd8540bb73107cf3bb8e87295f38 differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/06/62c64ab1577a242b69147e4db7b0944aea7c59 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/06/62c64ab1577a242b69147e4db7b0944aea7c59 new file mode 100644 index 00000000000..8b6201f3449 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/06/62c64ab1577a242b69147e4db7b0944aea7c59 differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/1a/d0869257423218471478b0e2c64182dc62ea66 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/1a/d0869257423218471478b0e2c64182dc62ea66 new file mode 100644 index 00000000000..2b72e32b289 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/1a/d0869257423218471478b0e2c64182dc62ea66 differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/23/e08071b1db93bf70e59f586d3241036d7676fc b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/23/e08071b1db93bf70e59f586d3241036d7676fc new file mode 100644 index 00000000000..0f999c22c8a Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/23/e08071b1db93bf70e59f586d3241036d7676fc differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/2e/995eb933f07ca377b8dd8a093bf80a83c257a2 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/2e/995eb933f07ca377b8dd8a093bf80a83c257a2 new file mode 100644 index 00000000000..14dcfa63745 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/2e/995eb933f07ca377b8dd8a093bf80a83c257a2 differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/37/5380121cbe5dc44c75f873b8c7ee1638bb20e5 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/37/5380121cbe5dc44c75f873b8c7ee1638bb20e5 new file mode 100644 index 00000000000..4854c672680 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/37/5380121cbe5dc44c75f873b8c7ee1638bb20e5 differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/48/d2c5157523fda29176a438843267660ca6f339 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/48/d2c5157523fda29176a438843267660ca6f339 new file mode 100644 index 00000000000..24f56e549fb Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/48/d2c5157523fda29176a438843267660ca6f339 differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/49/99f131df6d18a74e842bae0c275941f7831249 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/49/99f131df6d18a74e842bae0c275941f7831249 new file mode 100644 index 00000000000..2d02c84406c --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/49/99f131df6d18a74e842bae0c275941f7831249 @@ -0,0 +1 @@ +x+)JMU044d040031QHI5+(a8}dmr[WwEm,ȋFRdVSkѓ./=h^"#"_hrgGԂK. \ No newline at end of file diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/4d/0dd1f1bd0c254172b401d87109a9c234d86ba9 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/4d/0dd1f1bd0c254172b401d87109a9c234d86ba9 new file mode 100644 index 00000000000..27ab85bbaa6 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/4d/0dd1f1bd0c254172b401d87109a9c234d86ba9 differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/4d/4bc1c77f63bd5429cb0ab4d74cb8729570d293 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/4d/4bc1c77f63bd5429cb0ab4d74cb8729570d293 new file mode 100644 index 00000000000..dd33a633876 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/4d/4bc1c77f63bd5429cb0ab4d74cb8729570d293 differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/5b/7a80d8f027b2d78b2f8a665fac7bac810bcbd5 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/5b/7a80d8f027b2d78b2f8a665fac7bac810bcbd5 new file mode 100644 index 00000000000..f91235fffb1 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/5b/7a80d8f027b2d78b2f8a665fac7bac810bcbd5 differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/62/1f347db1d56ee2876551b8afdb73f6f5cd9b5b b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/62/1f347db1d56ee2876551b8afdb73f6f5cd9b5b new file mode 100644 index 00000000000..21c4167572a Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/62/1f347db1d56ee2876551b8afdb73f6f5cd9b5b differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/6c/11c899f73a3954cf6b5a17ebc8633eafebbea5 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/6c/11c899f73a3954cf6b5a17ebc8633eafebbea5 new file mode 100644 index 00000000000..6e766720016 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/6c/11c899f73a3954cf6b5a17ebc8633eafebbea5 differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/6e/55a15a35ad46f74e4203dd42f7797173a6edcb b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/6e/55a15a35ad46f74e4203dd42f7797173a6edcb new file mode 100644 index 00000000000..be0442b1f45 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/6e/55a15a35ad46f74e4203dd42f7797173a6edcb differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/70/26c29355f94d1d121cce9f16ac88db570023cd b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/70/26c29355f94d1d121cce9f16ac88db570023cd new file mode 100644 index 00000000000..aa00a6b728e --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/70/26c29355f94d1d121cce9f16ac88db570023cd @@ -0,0 +1 @@ +xM0{}R[oXH $mį/l=uyFj4JI?L7)j\QD9Ť%g$B3ΘQIb-4E1%DTܬjn qÑ(qɀ|s5N~OR5M'߰:(y`b-l0_E;J\EQn}_]\StQ{?91xAl/V} \ No newline at end of file diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/75/db99094e57d9f02f7eca73dd08f93ce2e302ba b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/75/db99094e57d9f02f7eca73dd08f93ce2e302ba new file mode 100644 index 00000000000..2dc0f778554 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/75/db99094e57d9f02f7eca73dd08f93ce2e302ba differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/8d/c57bcc0891edd46d2e69890fad7a96bd837dca b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/8d/c57bcc0891edd46d2e69890fad7a96bd837dca new file mode 100644 index 00000000000..d81f5a97f3f Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/8d/c57bcc0891edd46d2e69890fad7a96bd837dca differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/99/976b0393834a1238c2fa74cd798222f718b2b6 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/99/976b0393834a1238c2fa74cd798222f718b2b6 new file mode 100644 index 00000000000..cdf283a966d Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/99/976b0393834a1238c2fa74cd798222f718b2b6 differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/a0/054e492840f572e48a3cb791d2e083afaf08f6 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/a0/054e492840f572e48a3cb791d2e083afaf08f6 new file mode 100644 index 00000000000..caa4f79235a Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/a0/054e492840f572e48a3cb791d2e083afaf08f6 differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/aa/c30baac47ad1ab57c40da29bf39895bddec2d8 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/aa/c30baac47ad1ab57c40da29bf39895bddec2d8 new file mode 100644 index 00000000000..f08bdb298c3 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/aa/c30baac47ad1ab57c40da29bf39895bddec2d8 differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/b1/32e6aa2561ac807452786c9a3c98a6de99f718 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/b1/32e6aa2561ac807452786c9a3c98a6de99f718 new file mode 100644 index 00000000000..8044427c9f5 --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/b1/32e6aa2561ac807452786c9a3c98a6de99f718 @@ -0,0 +1,2 @@ +xM0{=i[v@IH IS~}asSGg4Fm]{h,O}D8[j93Ÿ +Tp.:D bQè,* +, !Bŵv0S܏ӣM݋g_!6MJ)k xOS, J`07$ʑ)6cւ&q&vHAXuPQLWuQMwi.¶Cx[4e6&C`w#dD7k-3|wfa+O)~̭v[U[uϜy_nnkg~/te/q/q?:pQ˥8(?!ӥ0 \ No newline at end of file diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/c7/6ef954d23f8fdb42dcf2fe956d6af5a31fe7bd b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/c7/6ef954d23f8fdb42dcf2fe956d6af5a31fe7bd new file mode 100644 index 00000000000..c58b40d4c49 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/c7/6ef954d23f8fdb42dcf2fe956d6af5a31fe7bd differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/cb/d03493c413b61eb5ab8a5a4eb17712286e5d8b b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/cb/d03493c413b61eb5ab8a5a4eb17712286e5d8b new file mode 100644 index 00000000000..75137775197 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/cb/d03493c413b61eb5ab8a5a4eb17712286e5d8b differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/ce/637da508bf67e5249a0721129fca5e7f0ea7eb b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/ce/637da508bf67e5249a0721129fca5e7f0ea7eb new file mode 100644 index 00000000000..2ba5d447263 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/ce/637da508bf67e5249a0721129fca5e7f0ea7eb differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/d0/b00a988cc19e4882df65d586311a361e77ed82 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/d0/b00a988cc19e4882df65d586311a361e77ed82 new file mode 100644 index 00000000000..7f6fa69ceb9 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/d0/b00a988cc19e4882df65d586311a361e77ed82 differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/d7/c9aa1754730abbb4748607f685795968351ea8 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/d7/c9aa1754730abbb4748607f685795968351ea8 new file mode 100644 index 00000000000..12bbae587a4 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/d7/c9aa1754730abbb4748607f685795968351ea8 differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/ec/7443c854237077ac2bbca423aad88ca670ecfa b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/ec/7443c854237077ac2bbca423aad88ca670ecfa new file mode 100644 index 00000000000..c3cc5bca18b --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/ec/7443c854237077ac2bbca423aad88ca670ecfa @@ -0,0 +1,3 @@ +xMo@{_(~VcRʒXh +bp> 54;h^FS]4Ah-DURs^3b$湴RWTaؔH:.Jd3!sdI0,IY84PqELsq}ioZ A)qkN jx*#d (TN(E[~N?^Ԯ=>4ǺYX\ .(ϤSlMzp;ܶGbj-gA˄C,eg[4^ȚY6#t\o~ +Ebfx/_}itdtsbCL3\Jgpr@/q4tΤ~ \ No newline at end of file diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/f1/f36270ff97895ac0f75837c262cfcb3c21cc2f b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/f1/f36270ff97895ac0f75837c262cfcb3c21cc2f new file mode 100644 index 00000000000..b6ed3797c90 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/f1/f36270ff97895ac0f75837c262cfcb3c21cc2f differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/f8/d3a988b4b6abe60fee48ecccad69efa7cf2639 b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/f8/d3a988b4b6abe60fee48ecccad69efa7cf2639 new file mode 100644 index 00000000000..9b36b062fc5 Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/objects/f8/d3a988b4b6abe60fee48ecccad69efa7cf2639 differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/refs/heads/master b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/refs/heads/master new file mode 100644 index 00000000000..2853a7db1bc --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/origin/refs/heads/master @@ -0,0 +1 @@ +c76ef954d23f8fdb42dcf2fe956d6af5a31fe7bd diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/HEAD b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/HEAD new file mode 100644 index 00000000000..cb089cd89a7 --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/HEAD @@ -0,0 +1 @@ +ref: refs/heads/master diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/config b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/config new file mode 100644 index 00000000000..01969e97449 --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/config @@ -0,0 +1,13 @@ +[core] + repositoryformatversion = 0 + filemode = true + bare = false + logallrefupdates = true + ignorecase = true + precomposeunicode = true +[remote "origin"] + url = ../origin + fetch = +refs/heads/master:refs/remotes/origin/master +[branch "master"] + remote = origin + merge = refs/heads/master diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/objects/pack/pack-6716438cb31b79ac3770f7b655f049426173d7d4.idx b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/objects/pack/pack-6716438cb31b79ac3770f7b655f049426173d7d4.idx new file mode 100644 index 00000000000..392a0da80da Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/objects/pack/pack-6716438cb31b79ac3770f7b655f049426173d7d4.idx differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/objects/pack/pack-6716438cb31b79ac3770f7b655f049426173d7d4.pack b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/objects/pack/pack-6716438cb31b79ac3770f7b655f049426173d7d4.pack new file mode 100644 index 00000000000..8d39837e16b Binary files /dev/null and b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/objects/pack/pack-6716438cb31b79ac3770f7b655f049426173d7d4.pack differ diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/packed-refs b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/packed-refs new file mode 100644 index 00000000000..f8d9d96116e --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/packed-refs @@ -0,0 +1,2 @@ +# pack-refs with: peeled fully-peeled sorted +c76ef954d23f8fdb42dcf2fe956d6af5a31fe7bd refs/remotes/origin/master diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/refs/heads/master b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/refs/heads/master new file mode 100644 index 00000000000..2853a7db1bc --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/refs/heads/master @@ -0,0 +1 @@ +c76ef954d23f8fdb42dcf2fe956d6af5a31fe7bd diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/shallow b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/shallow new file mode 100644 index 00000000000..2853a7db1bc --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/shallow_with_origin/repo/git/shallow @@ -0,0 +1 @@ +c76ef954d23f8fdb42dcf2fe956d6af5a31fe7bd diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/worktree/git b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/worktree/git new file mode 100644 index 00000000000..9b0610cfaac --- /dev/null +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/git/worktree/git @@ -0,0 +1 @@ +gitdir: /some/path/.git/worktrees/something diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/github.json b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/github.json index 2695a190a70..284ee3173da 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/resources/ci/github.json +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/ci/github.json @@ -11,13 +11,14 @@ "GITHUB_SERVER_URL": "https://ghenterprise.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", "GITHUB_WORKFLOW": "ghactions-pipeline-name", - "GITHUB_WORKSPACE": "/foo/bar" + "GITHUB_WORKSPACE": "/foo/bar", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://ghenterprise.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://ghenterprise.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://ghenterprise.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -41,13 +42,14 @@ "GITHUB_SERVER_URL": "https://github.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", "GITHUB_WORKFLOW": "ghactions-pipeline-name", - "GITHUB_WORKSPACE": "/foo/bar" + "GITHUB_WORKSPACE": "/foo/bar", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -71,13 +73,14 @@ "GITHUB_SERVER_URL": "https://github.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", "GITHUB_WORKFLOW": "ghactions-pipeline-name", - "GITHUB_WORKSPACE": "foo/bar" + "GITHUB_WORKSPACE": "foo/bar", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -101,13 +104,14 @@ "GITHUB_SERVER_URL": "https://github.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", "GITHUB_WORKFLOW": "ghactions-pipeline-name", - "GITHUB_WORKSPACE": "/foo/bar~" + "GITHUB_WORKSPACE": "/foo/bar~", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -131,13 +135,14 @@ "GITHUB_SERVER_URL": "https://github.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", "GITHUB_WORKFLOW": "ghactions-pipeline-name", - "GITHUB_WORKSPACE": "/foo/~/bar" + "GITHUB_WORKSPACE": "/foo/~/bar", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -163,13 +168,14 @@ "GITHUB_WORKFLOW": "ghactions-pipeline-name", "GITHUB_WORKSPACE": "~/foo/bar", "HOME": "/not-my-home", + "JOB_CHECK_RUN_ID": "123456", "USERPROFILE": "/not-my-home" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -195,13 +201,14 @@ "GITHUB_WORKFLOW": "ghactions-pipeline-name", "GITHUB_WORKSPACE": "~foo/bar", "HOME": "/not-my-home", + "JOB_CHECK_RUN_ID": "123456", "USERPROFILE": "/not-my-home" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -227,13 +234,14 @@ "GITHUB_WORKFLOW": "ghactions-pipeline-name", "GITHUB_WORKSPACE": "~", "HOME": "/not-my-home", + "JOB_CHECK_RUN_ID": "123456", "USERPROFILE": "/not-my-home" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -257,13 +265,14 @@ "GITHUB_SERVER_URL": "https://github.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", "GITHUB_WORKFLOW": "ghactions-pipeline-name", - "GITHUB_WORKSPACE": "/foo/bar" + "GITHUB_WORKSPACE": "/foo/bar", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -287,13 +296,14 @@ "GITHUB_SERVER_URL": "https://github.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", "GITHUB_WORKFLOW": "ghactions-pipeline-name", - "GITHUB_WORKSPACE": "/foo/bar" + "GITHUB_WORKSPACE": "/foo/bar", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -317,13 +327,14 @@ "GITHUB_SERVER_URL": "https://github.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", "GITHUB_WORKFLOW": "ghactions-pipeline-name", - "GITHUB_WORKSPACE": "/foo/bar" + "GITHUB_WORKSPACE": "/foo/bar", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -347,13 +358,14 @@ "GITHUB_SERVER_URL": "https://github.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", "GITHUB_WORKFLOW": "ghactions-pipeline-name", - "GITHUB_WORKSPACE": "/foo/bar" + "GITHUB_WORKSPACE": "/foo/bar", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -377,13 +389,14 @@ "GITHUB_SERVER_URL": "https://github.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", "GITHUB_WORKFLOW": "ghactions-pipeline-name", - "GITHUB_WORKSPACE": "/foo/bar" + "GITHUB_WORKSPACE": "/foo/bar", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -408,13 +421,14 @@ "GITHUB_SERVER_URL": "https://github.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", "GITHUB_WORKFLOW": "ghactions-pipeline-name", - "GITHUB_WORKSPACE": "/foo/bar" + "GITHUB_WORKSPACE": "/foo/bar", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -439,13 +453,14 @@ "GITHUB_SERVER_URL": "https://github.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", "GITHUB_WORKFLOW": "ghactions-pipeline-name", - "GITHUB_WORKSPACE": "/foo/bar" + "GITHUB_WORKSPACE": "/foo/bar", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -470,13 +485,14 @@ "GITHUB_SERVER_URL": "https://github.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", "GITHUB_WORKFLOW": "ghactions-pipeline-name", - "GITHUB_WORKSPACE": "/foo/bar" + "GITHUB_WORKSPACE": "/foo/bar", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -509,13 +525,14 @@ "GITHUB_SERVER_URL": "https://github.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", "GITHUB_WORKFLOW": "ghactions-pipeline-name", - "GITHUB_WORKSPACE": "foo/bar" + "GITHUB_WORKSPACE": "foo/bar", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -555,13 +572,14 @@ "GITHUB_SERVER_URL": "https://github.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", "GITHUB_WORKFLOW": "ghactions-pipeline-name", - "GITHUB_WORKSPACE": "foo/bar" + "GITHUB_WORKSPACE": "foo/bar", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -589,13 +607,14 @@ "GITHUB_RUN_NUMBER": "ghactions-pipeline-number", "GITHUB_SERVER_URL": "https://github.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", - "GITHUB_WORKFLOW": "ghactions-pipeline-name" + "GITHUB_WORKFLOW": "ghactions-pipeline-name", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -615,13 +634,14 @@ "GITHUB_RUN_NUMBER": "ghactions-pipeline-number", "GITHUB_SERVER_URL": "https://user:password@github.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", - "GITHUB_WORKFLOW": "ghactions-pipeline-name" + "GITHUB_WORKFLOW": "ghactions-pipeline-name", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -641,13 +661,14 @@ "GITHUB_RUN_NUMBER": "ghactions-pipeline-number", "GITHUB_SERVER_URL": "https://user@github.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", - "GITHUB_WORKFLOW": "ghactions-pipeline-name" + "GITHUB_WORKFLOW": "ghactions-pipeline-name", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -667,13 +688,14 @@ "GITHUB_RUN_NUMBER": "ghactions-pipeline-number", "GITHUB_SERVER_URL": "https://user:password@github.com:1234", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", - "GITHUB_WORKFLOW": "ghactions-pipeline-name" + "GITHUB_WORKFLOW": "ghactions-pipeline-name", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com:1234\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com:1234/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com:1234/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -693,13 +715,14 @@ "GITHUB_RUN_NUMBER": "ghactions-pipeline-number", "GITHUB_SERVER_URL": "https://user:password@1.1.1.1", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", - "GITHUB_WORKFLOW": "ghactions-pipeline-name" + "GITHUB_WORKFLOW": "ghactions-pipeline-name", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://1.1.1.1\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://1.1.1.1/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://1.1.1.1/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -719,13 +742,14 @@ "GITHUB_RUN_NUMBER": "ghactions-pipeline-number", "GITHUB_SERVER_URL": "https://user:password@1.1.1.1:1234", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", - "GITHUB_WORKFLOW": "ghactions-pipeline-name" + "GITHUB_WORKFLOW": "ghactions-pipeline-name", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://1.1.1.1:1234\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://1.1.1.1:1234/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://1.1.1.1:1234/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", @@ -746,13 +770,14 @@ "GITHUB_RUN_NUMBER": "ghactions-pipeline-number", "GITHUB_SERVER_URL": "https://github.com", "GITHUB_SHA": "b9f0fb3fdbb94c9d24b2c75b49663122a529e123", - "GITHUB_WORKFLOW": "ghactions-pipeline-name" + "GITHUB_WORKFLOW": "ghactions-pipeline-name", + "JOB_CHECK_RUN_ID": "123456" }, { "_dd.ci.env_vars": "{\"GITHUB_SERVER_URL\":\"https://github.com\",\"GITHUB_REPOSITORY\":\"ghactions-repo\",\"GITHUB_RUN_ID\":\"ghactions-pipeline-id\",\"GITHUB_RUN_ATTEMPT\":\"ghactions-run-attempt\"}", - "ci.job.id": "github-job-name", + "ci.job.id": "123456", "ci.job.name": "github-job-name", - "ci.job.url": "https://github.com/ghactions-repo/commit/b9f0fb3fdbb94c9d24b2c75b49663122a529e123/checks", + "ci.job.url": "https://github.com/ghactions-repo/actions/runs/ghactions-pipeline-id/job/123456", "ci.pipeline.id": "ghactions-pipeline-id", "ci.pipeline.name": "ghactions-pipeline-name", "ci.pipeline.number": "ghactions-pipeline-number", diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/config/known-tests-request.ftl b/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/config/known-tests-request.ftl index 4b9593e0a43..7686905896a 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/config/known-tests-request.ftl +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/config/known-tests-request.ftl @@ -3,26 +3,11 @@ "type" : "ci_app_libraries_tests_request", "id" : "${uid}", "attributes": { + "repository_url": "${tracerEnvironment.repositoryUrl}", "service" : "${tracerEnvironment.service}", "env" : "${tracerEnvironment.env}", - "repository_url": "${tracerEnvironment.repositoryUrl}", - "branch" : "${tracerEnvironment.branch}", - "sha" : "${tracerEnvironment.sha}", - "test_level" : "${tracerEnvironment.testLevel}", - "configurations": { - "os.platform" : "${tracerEnvironment.configurations.osPlatform}", - "os.architecture" : "${tracerEnvironment.configurations.osArchitecture}", - "os.arch" : "${tracerEnvironment.configurations.osArchitecture}", - "os.version" : "${tracerEnvironment.configurations.osVersion}", - "runtime.name" : "${tracerEnvironment.configurations.runtimeName}", - "runtime.version" : "${tracerEnvironment.configurations.runtimeVersion}", - "runtime.vendor" : "${tracerEnvironment.configurations.runtimeVendor}", - "runtime.architecture": "${tracerEnvironment.configurations.runtimeArchitecture}", - "custom" : { - <#list tracerEnvironment.configurations.custom as customTag, customValue> - "${customTag}": "${customValue}"<#if customTag?has_next>, - - } + "page_info": {<#if pageInfo.pageState??> + "page_state": "${pageInfo.pageState}" } } } diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/config/known-tests-response.ftl b/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/config/known-tests-response.ftl index 98c22839499..41b4370f686 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/config/known-tests-response.ftl +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/config/known-tests-response.ftl @@ -20,7 +20,12 @@ "test-name-2" ] } - } + }<#if pageInfo??>, + "page_info": { + "size": ${pageInfo.size}, + "has_next": ${pageInfo.hasNext?c}<#if pageInfo.cursor??>, + "cursor": "${pageInfo.cursor}" + } } } } diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/config/test-management-tests-request.ftl b/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/config/test-management-tests-request.ftl index b6ae9190501..829ef1a97e1 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/config/test-management-tests-request.ftl +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/config/test-management-tests-request.ftl @@ -5,8 +5,8 @@ "attributes": { "repository_url" : "${tracerEnvironment.repositoryUrl}", "commit_message" : "${tracerEnvironment.commitMessage}", - "sha" : "${tracerEnvironment.sha}" - } + "sha" : "${tracerEnvironment.sha}", + "branch" : "${tracerEnvironment.branch}" } } } diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/git/tree/git-diff.txt b/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/git/tree/git-diff.txt index d79dc73c99a..36a9fd554f0 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/git/tree/git-diff.txt +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/git/tree/git-diff.txt @@ -1,16 +1,16 @@ -diff --git a/java/maven-junit4/pom.xml b/java/maven-junit4/pom.xml +diff --git java/maven-junit4/pom.xml java/maven-junit4/pom.xml index 6d73cda..2a1f220 100644 ---- a/java/maven-junit4/pom.xml -+++ b/java/maven-junit4/pom.xml +--- java/maven-junit4/pom.xml ++++ java/maven-junit4/pom.xml @@ -10 +10 @@ -java-maven-junit4 +java-maven-junit4-test-project ~ -diff --git a/java/maven-junit5/pom.xml b/java/maven-junit5/pom.xml +diff --git java/maven-junit5/pom.xml java/maven-junit5/pom.xml index 7b92d64..834a61c 100644 ---- a/java/maven-junit5/pom.xml -+++ b/java/maven-junit5/pom.xml +--- java/maven-junit5/pom.xml ++++ java/maven-junit5/pom.xml @@ -14 +14 @@ -module-b diff --git a/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/git/tree/larger-git-diff.txt b/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/git/tree/larger-git-diff.txt index 41e227c58eb..c628d1dfaaa 100644 --- a/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/git/tree/larger-git-diff.txt +++ b/dd-java-agent/agent-ci-visibility/src/test/resources/datadog/trace/civisibility/git/tree/larger-git-diff.txt @@ -1,16 +1,16 @@ -diff --git a/java/maven-junit4/pom.xml b/java/maven-junit4/pom.xml +diff --git java/maven-junit4/pom.xml java/maven-junit4/pom.xml index 6d73cda..2a1f220 100644 ---- a/java/maven-junit4/pom.xml -+++ b/java/maven-junit4/pom.xml +--- java/maven-junit4/pom.xml ++++ java/maven-junit4/pom.xml @@ -10 +10 @@ -java-maven-junit4 +java-maven-junit4-test-project ~ -diff --git a/java/maven-junit5/module-a/pom.xml b/java/maven-junit5/module-a/pom.xml +diff --git java/maven-junit5/module-a/pom.xml java/maven-junit5/module-a/pom.xml index 29a3a73..4567037 100644 ---- a/java/maven-junit5/module-a/pom.xml -+++ b/java/maven-junit5/module-a/pom.xml +--- java/maven-junit5/module-a/pom.xml ++++ java/maven-junit5/module-a/pom.xml @@ -8,3 +8,3 @@ -com.datadog.ci.test @@ -53,10 +53,10 @@ index 29a3a73..4567037 100644 @@ -34 +40 @@ ~ -diff --git a/java/maven-junit5/module-b/pom.xml b/java/maven-junit5/module-b/pom.xml +diff --git java/maven-junit5/module-b/pom.xml java/maven-junit5/module-b/pom.xml deleted file mode 100644 index f18dd09..0000000 ---- a/java/maven-junit5/module-b/pom.xml +--- java/maven-junit5/module-b/pom.xml +++ /dev/null @@ -1,17 +0,0 @@ - @@ -91,10 +91,10 @@ index f18dd09..0000000 ~ - ~ -diff --git a/java/maven-junit5/pom.xml b/java/maven-junit5/pom.xml +diff --git java/maven-junit5/pom.xml java/maven-junit5/pom.xml index 7b92d64..834a61c 100644 ---- a/java/maven-junit5/pom.xml -+++ b/java/maven-junit5/pom.xml +--- java/maven-junit5/pom.xml ++++ java/maven-junit5/pom.xml @@ -14 +14 @@ -module-b diff --git a/dd-java-agent/agent-crashtracking/build.gradle b/dd-java-agent/agent-crashtracking/build.gradle index 752ba182ac4..af93daaeb0a 100644 --- a/dd-java-agent/agent-crashtracking/build.gradle +++ b/dd-java-agent/agent-crashtracking/build.gradle @@ -14,7 +14,9 @@ dependencies { implementation libs.slf4j implementation project(':communication') implementation project(':internal-api') + implementation project(':products:metrics:metrics-lib') implementation project(':utils:container-utils') + implementation project(':utils:queue-utils') implementation project(':utils:version-utils') implementation project(path: ':dd-java-agent:ddprof-lib', configuration: 'shadow') @@ -23,7 +25,10 @@ dependencies { testImplementation libs.bundles.junit5 testImplementation libs.bundles.mockito + testImplementation libs.assertj.core + testImplementation libs.json.unit.assertj testImplementation libs.jackson.databind + testImplementation libs.testcontainers testImplementation group: 'com.squareup.okhttp3', name: 'mockwebserver', version: libs.versions.okhttp.legacy.get() } diff --git a/dd-java-agent/agent-crashtracking/gradle.lockfile b/dd-java-agent/agent-crashtracking/gradle.lockfile index 5b4d09a43c4..2761bba8443 100644 --- a/dd-java-agent/agent-crashtracking/gradle.lockfile +++ b/dd-java-agent/agent-crashtracking/gradle.lockfile @@ -5,137 +5,120 @@ cafe.cryptography:curve25519-elisabeth:0.1.0=runtimeClasspath,testRuntimeClasspa cafe.cryptography:ed25519-elisabeth:0.1.0=runtimeClasspath,testRuntimeClasspath ch.qos.logback:logback-classic:1.2.13=testCompileClasspath,testRuntimeClasspath ch.qos.logback:logback-core:1.2.13=testCompileClasspath,testRuntimeClasspath -com.beust:jcommander:1.78=testRuntimeClasspath com.datadoghq.okhttp3:okhttp:3.12.15=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.datadoghq.okio:okio:1.17.6=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.datadoghq:dd-javac-plugin-client:0.2.2=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.datadoghq:java-dogstatsd-client:4.4.3=runtimeClasspath,testRuntimeClasspath +com.datadoghq:java-dogstatsd-client:4.4.5=runtimeClasspath,testRuntimeClasspath +com.datadoghq:sketches-java:0.8.3=runtimeClasspath,testRuntimeClasspath com.fasterxml.jackson.core:jackson-annotations:2.20=testCompileClasspath,testRuntimeClasspath com.fasterxml.jackson.core:jackson-core:2.20.0=testCompileClasspath,testRuntimeClasspath com.fasterxml.jackson.core:jackson-databind:2.20.0=testCompileClasspath,testRuntimeClasspath com.fasterxml.jackson:jackson-bom:2.20.0=testCompileClasspath,testRuntimeClasspath -com.github.javaparser:javaparser-core:3.25.6=codenarc,testCompileClasspath,testRuntimeClasspath -com.github.jnr:jffi:1.3.13=runtimeClasspath,testRuntimeClasspath +com.github.docker-java:docker-java-api:3.4.2=testCompileClasspath,testRuntimeClasspath +com.github.docker-java:docker-java-transport-zerodep:3.4.2=testCompileClasspath,testRuntimeClasspath +com.github.docker-java:docker-java-transport:3.4.2=testCompileClasspath,testRuntimeClasspath +com.github.javaparser:javaparser-core:3.25.6=codenarc +com.github.jnr:jffi:1.3.14=runtimeClasspath,testRuntimeClasspath com.github.jnr:jnr-a64asm:1.0.0=runtimeClasspath,testRuntimeClasspath com.github.jnr:jnr-constants:0.10.4=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-enxio:0.32.17=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-ffi:2.2.16=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-posix:3.1.19=runtimeClasspath,testRuntimeClasspath -com.github.jnr:jnr-unixsocket:0.38.22=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-enxio:0.32.19=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-ffi:2.2.18=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-posix:3.1.21=runtimeClasspath,testRuntimeClasspath +com.github.jnr:jnr-unixsocket:0.38.24=runtimeClasspath,testRuntimeClasspath com.github.jnr:jnr-x86asm:1.0.2=runtimeClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.2.0=compileClasspath,testCompileClasspath,testRuntimeClasspath -com.github.spotbugs:spotbugs-annotations:4.7.3=spotbugs -com.github.spotbugs:spotbugs:4.7.3=spotbugs +com.github.spotbugs:spotbugs-annotations:4.9.8=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath +com.github.spotbugs:spotbugs:4.9.8=spotbugs +com.github.stephenc.jcip:jcip-annotations:1.0-1=spotbugs com.google.code.findbugs:jsr305:3.0.2=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -com.google.code.gson:gson:2.9.1=spotbugs +com.google.code.gson:gson:2.13.2=spotbugs +com.google.errorprone:error_prone_annotations:2.41.0=spotbugs +com.jayway.jsonpath:json-path:2.8.0=testCompileClasspath,testRuntimeClasspath com.squareup.moshi:moshi:1.11.0=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath com.squareup.okhttp3:mockwebserver:3.12.12=testCompileClasspath,testRuntimeClasspath com.squareup.okhttp3:okhttp:3.12.12=testCompileClasspath,testRuntimeClasspath com.squareup.okio:okio:1.17.5=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath -com.thoughtworks.qdox:qdox:1.12.1=codenarc,testRuntimeClasspath -commons-codec:commons-codec:1.15=spotbugs -de.thetaphi:forbiddenapis:3.8=compileClasspath -info.picocli:picocli:4.6.3=testRuntimeClasspath +com.thoughtworks.qdox:qdox:1.12.1=codenarc +commons-io:commons-io:2.20.0=spotbugs +de.thetaphi:forbiddenapis:3.10=compileClasspath io.leangen.geantyref:geantyref:1.3.16=testRuntimeClasspath -jaxen:jaxen:1.2.0=spotbugs -jline:jline:2.14.6=testRuntimeClasspath +jaxen:jaxen:2.0.0=spotbugs junit:junit:4.13.2=testCompileClasspath,testRuntimeClasspath -net.bytebuddy:byte-buddy-agent:1.17.7=testCompileClasspath,testRuntimeClasspath -net.bytebuddy:byte-buddy:1.17.7=testCompileClasspath,testRuntimeClasspath -net.jcip:jcip-annotations:1.0=compileClasspath,spotbugs,testCompileClasspath,testRuntimeClasspath -net.sf.saxon:Saxon-HE:11.4=spotbugs +net.bytebuddy:byte-buddy-agent:1.18.8=testCompileClasspath,testRuntimeClasspath +net.bytebuddy:byte-buddy:1.18.8=testCompileClasspath,testRuntimeClasspath +net.java.dev.jna:jna:5.13.0=testCompileClasspath,testRuntimeClasspath +net.javacrumbs.json-unit:json-unit-assertj:2.40.1=testCompileClasspath,testRuntimeClasspath +net.javacrumbs.json-unit:json-unit-core:2.40.1=testCompileClasspath,testRuntimeClasspath +net.javacrumbs.json-unit:json-unit-json-path:2.40.1=testCompileClasspath,testRuntimeClasspath +net.minidev:accessors-smart:2.4.9=testRuntimeClasspath +net.minidev:json-smart:2.4.10=testRuntimeClasspath +net.sf.saxon:Saxon-HE:12.9=spotbugs org.apache.ant:ant-antlr:1.10.14=codenarc -org.apache.ant:ant-antlr:1.10.15=testRuntimeClasspath org.apache.ant:ant-junit:1.10.14=codenarc -org.apache.ant:ant-junit:1.10.15=testRuntimeClasspath -org.apache.ant:ant-launcher:1.10.15=testRuntimeClasspath -org.apache.ant:ant:1.10.15=testCompileClasspath,testRuntimeClasspath -org.apache.bcel:bcel:6.5.0=spotbugs -org.apache.commons:commons-lang3:3.12.0=spotbugs -org.apache.commons:commons-text:1.10.0=spotbugs -org.apache.httpcomponents.client5:httpclient5:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5-h2:5.1.3=spotbugs -org.apache.httpcomponents.core5:httpcore5:5.1.3=spotbugs -org.apache.logging.log4j:log4j-api:2.19.0=spotbugs -org.apache.logging.log4j:log4j-core:2.19.0=spotbugs +org.apache.bcel:bcel:6.11.0=spotbugs +org.apache.commons:commons-compress:1.24.0=testCompileClasspath,testRuntimeClasspath +org.apache.commons:commons-lang3:3.19.0=spotbugs +org.apache.commons:commons-text:1.14.0=spotbugs +org.apache.logging.log4j:log4j-api:2.25.2=spotbugs +org.apache.logging.log4j:log4j-core:2.25.2=spotbugs org.apiguardian:apiguardian-api:1.1.2=testCompileClasspath -org.codehaus.groovy:groovy-all:3.0.24=testCompileClasspath,testRuntimeClasspath +org.assertj:assertj-core:3.27.7=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-ant:3.0.23=codenarc -org.codehaus.groovy:groovy-ant:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-astbuilder:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-cli-picocli:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-console:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-datetime:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-docgenerator:3.0.23=codenarc -org.codehaus.groovy:groovy-docgenerator:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-groovydoc:3.0.23=codenarc -org.codehaus.groovy:groovy-groovydoc:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-groovysh:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jmx:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-json:3.0.23=codenarc -org.codehaus.groovy:groovy-json:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-jsr223:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-macro:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-nio:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-servlet:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-sql:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-swing:3.0.24=testCompileClasspath,testRuntimeClasspath +org.codehaus.groovy:groovy-json:3.0.25=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-templates:3.0.23=codenarc -org.codehaus.groovy:groovy-templates:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test-junit5:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-test:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codehaus.groovy:groovy-testng:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy-xml:3.0.23=codenarc -org.codehaus.groovy:groovy-xml:3.0.24=testCompileClasspath,testRuntimeClasspath org.codehaus.groovy:groovy:3.0.23=codenarc -org.codehaus.groovy:groovy:3.0.24=testCompileClasspath,testRuntimeClasspath -org.codenarc:CodeNarc:3.6.0=codenarc -org.dom4j:dom4j:2.1.3=spotbugs +org.codehaus.groovy:groovy:3.0.25=testCompileClasspath,testRuntimeClasspath +org.codenarc:CodeNarc:3.7.0=codenarc +org.dom4j:dom4j:2.2.0=spotbugs org.gmetrics:GMetrics:2.1.0=codenarc -org.hamcrest:hamcrest-core:1.3=testCompileClasspath,testRuntimeClasspath +org.hamcrest:hamcrest-core:2.2=testCompileClasspath,testRuntimeClasspath org.hamcrest:hamcrest:3.0=testCompileClasspath,testRuntimeClasspath org.jacoco:org.jacoco.agent:0.8.14=jacocoAgent,jacocoAnt org.jacoco:org.jacoco.ant:0.8.14=jacocoAnt org.jacoco:org.jacoco.core:0.8.14=jacocoAnt org.jacoco:org.jacoco.report:0.8.14=jacocoAnt -org.junit.jupiter:junit-jupiter-api:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter-engine:5.12.2=testRuntimeClasspath -org.junit.jupiter:junit-jupiter-params:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.jupiter:junit-jupiter:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-commons:1.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-engine:1.12.2=testCompileClasspath,testRuntimeClasspath -org.junit.platform:junit-platform-launcher:1.12.2=testRuntimeClasspath -org.junit:junit-bom:5.12.2=testCompileClasspath,testRuntimeClasspath -org.junit:junit-bom:5.9.1=spotbugs +org.jctools:jctools-core-jdk11:4.0.6=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +org.jctools:jctools-core:4.0.6=compileClasspath,runtimeClasspath,testCompileClasspath,testRuntimeClasspath +org.jetbrains:annotations:17.0.0=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-api:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter-engine:5.14.1=testRuntimeClasspath +org.junit.jupiter:junit-jupiter-params:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.jupiter:junit-jupiter:5.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-commons:1.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-engine:1.14.1=testCompileClasspath,testRuntimeClasspath +org.junit.platform:junit-platform-launcher:1.14.1=testRuntimeClasspath +org.junit:junit-bom:5.14.0=spotbugs +org.junit:junit-bom:5.14.1=testCompileClasspath,testRuntimeClasspath org.mockito:mockito-core:4.4.0=testCompileClasspath,testRuntimeClasspath org.mockito:mockito-junit-jupiter:4.4.0=testCompileClasspath,testRuntimeClasspath org.objenesis:objenesis:3.3=testCompileClasspath,testRuntimeClasspath org.opentest4j:opentest4j:1.3.0=testCompileClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-analysis:9.4=spotbugs -org.ow2.asm:asm-commons:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-commons:9.4=spotbugs -org.ow2.asm:asm-commons:9.9=jacocoAnt -org.ow2.asm:asm-tree:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-tree:9.4=spotbugs -org.ow2.asm:asm-tree:9.9=jacocoAnt -org.ow2.asm:asm-util:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm-util:9.4=spotbugs -org.ow2.asm:asm:9.2=runtimeClasspath,testRuntimeClasspath -org.ow2.asm:asm:9.4=spotbugs -org.ow2.asm:asm:9.9=jacocoAnt +org.ow2.asm:asm-analysis:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-analysis:9.9=spotbugs +org.ow2.asm:asm-commons:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-commons:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-tree:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-tree:9.9=jacocoAnt,spotbugs +org.ow2.asm:asm-util:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm-util:9.9=spotbugs +org.ow2.asm:asm:9.7.1=runtimeClasspath,testRuntimeClasspath +org.ow2.asm:asm:9.9=jacocoAnt,spotbugs +org.rnorth.duct-tape:duct-tape:1.0.8=testCompileClasspath,testRuntimeClasspath org.slf4j:jcl-over-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath org.slf4j:jul-to-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath org.slf4j:log4j-over-slf4j:1.7.30=testCompileClasspath,testRuntimeClasspath org.slf4j:slf4j-api:1.7.30=compileClasspath,runtimeClasspath -org.slf4j:slf4j-api:1.7.32=testCompileClasspath,testRuntimeClasspath -org.slf4j:slf4j-api:2.0.0=spotbugs,spotbugsSlf4j -org.slf4j:slf4j-simple:2.0.0=spotbugsSlf4j +org.slf4j:slf4j-api:1.7.36=testCompileClasspath,testRuntimeClasspath +org.slf4j:slf4j-api:2.0.17=spotbugs,spotbugsSlf4j +org.slf4j:slf4j-simple:2.0.17=spotbugsSlf4j org.snakeyaml:snakeyaml-engine:2.9=runtimeClasspath,testRuntimeClasspath -org.spockframework:spock-bom:2.4-M6-groovy-3.0=testCompileClasspath,testRuntimeClasspath -org.spockframework:spock-core:2.4-M6-groovy-3.0=testCompileClasspath,testRuntimeClasspath -org.testng:testng:7.5.1=testRuntimeClasspath -org.webjars:jquery:3.5.1=testRuntimeClasspath -org.xmlresolver:xmlresolver:4.4.3=spotbugs -xml-apis:xml-apis:1.4.01=spotbugs +org.spockframework:spock-bom:2.4-groovy-3.0=testCompileClasspath,testRuntimeClasspath +org.spockframework:spock-core:2.4-groovy-3.0=testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-junit:1.2.1=testCompileClasspath,testRuntimeClasspath +org.tabletest:tabletest-parser:1.2.0=testCompileClasspath,testRuntimeClasspath +org.testcontainers:testcontainers:1.21.4=testCompileClasspath,testRuntimeClasspath +org.xmlresolver:xmlresolver:5.3.3=spotbugs empty=annotationProcessor,spotbugsPlugins,testAnnotationProcessor diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/ConfigManager.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/ConfigManager.java index 8a4f681cc4f..02c10016d83 100644 --- a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/ConfigManager.java +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/ConfigManager.java @@ -12,9 +12,13 @@ import datadog.trace.util.RandomUtils; import java.io.BufferedReader; import java.io.BufferedWriter; +import java.io.File; +import java.io.FileInputStream; +import java.io.FileOutputStream; import java.io.IOException; -import java.nio.file.Files; -import java.nio.file.Path; +import java.io.InputStreamReader; +import java.io.OutputStreamWriter; +import java.nio.charset.StandardCharsets; import java.util.regex.Pattern; import java.util.stream.Collectors; import javax.annotation.Nullable; @@ -33,6 +37,9 @@ public static class StoredConfig { final String processTags; final String runtimeId; final String reportUUID; + final boolean agentless; + final boolean sendToErrorTracking; + final boolean extendedInfoEnabled; StoredConfig( String reportUUID, @@ -41,7 +48,10 @@ public static class StoredConfig { String version, String tags, String processTags, - String runtimeId) { + String runtimeId, + boolean agentless, + boolean sendToErrorTracking, + boolean extendedInfoEnabled) { this.service = service; this.env = env; this.version = version; @@ -49,6 +59,13 @@ public static class StoredConfig { this.processTags = processTags; this.runtimeId = runtimeId; this.reportUUID = reportUUID; + this.agentless = agentless; + this.sendToErrorTracking = sendToErrorTracking; + this.extendedInfoEnabled = extendedInfoEnabled; + } + + public CrashUploaderSettings toCrashUploaderSettings() { + return new CrashUploaderSettings(extendedInfoEnabled); } public static class Builder { @@ -59,6 +76,9 @@ public static class Builder { String processTags; String runtimeId; String reportUUID; + boolean agentless; + boolean sendToErrorTracking; + boolean extendedInfoEnabled; public Builder(Config config) { // get sane defaults @@ -67,6 +87,9 @@ public Builder(Config config) { this.version = config.getVersion(); this.runtimeId = config.getRuntimeId(); this.reportUUID = RandomUtils.randomUUID().toString(); + this.agentless = config.isCrashTrackingAgentless(); + this.sendToErrorTracking = config.isCrashTrackingErrorsIntakeEnabled(); + this.extendedInfoEnabled = config.isCrashTrackingExtendedInfoEnabled(); } public Builder service(String service) { @@ -99,6 +122,21 @@ public Builder runtimeId(String runtimeId) { return this; } + public Builder sendToErrorTracking(boolean sendToErrorTracking) { + this.sendToErrorTracking = sendToErrorTracking; + return this; + } + + public Builder agentless(boolean agentless) { + this.agentless = agentless; + return this; + } + + public Builder extendedInfoEnabled(boolean extendedInfoEnabled) { + this.extendedInfoEnabled = extendedInfoEnabled; + return this; + } + // @VisibleForTesting Builder reportUUID(String reportUUID) { this.reportUUID = reportUUID; @@ -106,15 +144,25 @@ Builder reportUUID(String reportUUID) { } public StoredConfig build() { - return new StoredConfig(reportUUID, service, env, version, tags, processTags, runtimeId); + return new StoredConfig( + reportUUID, + service, + env, + version, + tags, + processTags, + runtimeId, + agentless, + sendToErrorTracking, + extendedInfoEnabled); } } } private ConfigManager() {} - private static String getBaseName(Path path) { - String filename = path.getFileName().toString(); + private static String getBaseName(File file) { + String filename = file.getName(); int dotIndex = filename.lastIndexOf('.'); if (dotIndex == -1) { return filename; @@ -122,8 +170,10 @@ private static String getBaseName(Path path) { return filename.substring(0, dotIndex); } - private static String getMergedTagsForSerialization(Config config) { + // @VisibleForTesting + static String getMergedTagsForSerialization(Config config) { return config.getMergedCrashTrackingTags().entrySet().stream() + .filter(e -> e.getValue() != null) .map(e -> e.getKey() + ":" + e.getValue()) .collect(Collectors.joining(",")); } @@ -139,18 +189,20 @@ private static void writeEntry(BufferedWriter writer, CharSequence key, CharSequ writer.newLine(); } - public static void writeConfigToPath(Path scriptPath, String... additionalEntries) { - String cfgFileName = getBaseName(scriptPath) + PID_PREFIX + PidHelper.getPid() + ".cfg"; - Path cfgPath = scriptPath.resolveSibling(cfgFileName); - writeConfigToFile(Config.get(), cfgPath, additionalEntries); + public static void writeConfigToPath(File scriptFile, String... additionalEntries) { + String cfgFileName = getBaseName(scriptFile) + PID_PREFIX + PidHelper.getPid() + ".cfg"; + File cfgFile = new File(scriptFile.getParentFile(), cfgFileName); + writeConfigToFile(Config.get(), cfgFile, additionalEntries); } // @VisibleForTesting - static void writeConfigToFile(Config config, Path cfgPath, String... additionalEntries) { + static void writeConfigToFile(Config config, File cfgFile, String... additionalEntries) { final WellKnownTags wellKnownTags = config.getWellKnownTags(); - LOGGER.debug("Writing config file: {}", cfgPath); - try (BufferedWriter bw = Files.newBufferedWriter(cfgPath)) { + LOGGER.debug("Writing config file: {}", cfgFile); + try (BufferedWriter bw = + new BufferedWriter( + new OutputStreamWriter(new FileOutputStream(cfgFile), StandardCharsets.UTF_8))) { for (int i = 0; i < additionalEntries.length; i += 2) { writeEntry(bw, additionalEntries[i], additionalEntries[i + 1]); } @@ -161,33 +213,31 @@ static void writeConfigToFile(Config config, Path cfgPath, String... additionalE writeEntry(bw, "process_tags", ProcessTags.getTagsForSerialization()); writeEntry(bw, "runtime_id", wellKnownTags.getRuntimeId()); writeEntry(bw, "java_home", SystemProperties.get("java.home")); + writeEntry(bw, "agentless", Boolean.toString(config.isCrashTrackingAgentless())); + writeEntry(bw, "upload_to_et", Boolean.toString(config.isCrashTrackingErrorsIntakeEnabled())); + writeEntry( + bw, "extended_info", Boolean.toString(config.isCrashTrackingExtendedInfoEnabled())); Runtime.getRuntime() .addShutdownHook( new Thread( AGENT_THREAD_GROUP, () -> { - try { - LOGGER.debug("Deleting config file: {}", cfgPath); - Files.deleteIfExists(cfgPath); - } catch (IOException e) { - LOGGER.warn(SEND_TELEMETRY, "Failed deleting config file: {}", cfgPath, e); - } + LOGGER.debug("Deleting config file: {}", cfgFile); + cfgFile.delete(); })); - LOGGER.debug("Config file written: {}", cfgPath); + LOGGER.debug("Config file written: {}", cfgFile); } catch (IOException e) { - LOGGER.warn(SEND_TELEMETRY, "Failed writing config file: {}", cfgPath); - try { - Files.deleteIfExists(cfgPath); - } catch (IOException ignored) { - // ignore - } + LOGGER.warn(SEND_TELEMETRY, "Failed writing config file: {}", cfgFile); + cfgFile.delete(); // best-effort cleanup; failure is acceptable here } } @Nullable - public static StoredConfig readConfig(Config config, Path scriptPath) { - try (final BufferedReader reader = Files.newBufferedReader(scriptPath)) { + public static StoredConfig readConfig(Config config, File scriptFile) { + try (final BufferedReader reader = + new BufferedReader( + new InputStreamReader(new FileInputStream(scriptFile), StandardCharsets.UTF_8))) { final StoredConfig.Builder cfgBuilder = new StoredConfig.Builder(config); String line; while ((line = reader.readLine()) != null) { @@ -218,6 +268,15 @@ public static StoredConfig readConfig(Config config, Path scriptPath) { case "runtime_id": cfgBuilder.runtimeId(value); break; + case "agentless": + cfgBuilder.agentless(Boolean.parseBoolean(value)); + break; + case "upload_to_et": + cfgBuilder.sendToErrorTracking(Boolean.parseBoolean(value)); + break; + case "extended_info": + cfgBuilder.extendedInfoEnabled(Boolean.parseBoolean(value)); + break; default: // ignore break; @@ -225,7 +284,7 @@ public static StoredConfig readConfig(Config config, Path scriptPath) { } return cfgBuilder.build(); } catch (Throwable t) { - LOGGER.error("Failed to read config file: {}", scriptPath, t); + LOGGER.error("Failed to read config file: {}", scriptFile, t); } return null; } diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/CrashLogParser.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/CrashLogParser.java index 20ce9f4c9a5..02136fa2ebb 100644 --- a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/CrashLogParser.java +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/CrashLogParser.java @@ -2,9 +2,49 @@ import datadog.crashtracking.dto.CrashLog; import datadog.crashtracking.parsers.HotspotCrashLogParser; +import datadog.crashtracking.parsers.J9JavacoreParser; public final class CrashLogParser { + + /** J9 javacore files start with section markers like "0SECTION" */ + private static final String J9_SECTION_MARKER = "0SECTION"; + + /** J9 javacore TITLE section identifier */ + private static final String J9_TITLE_MARKER = "TITLE"; + + /** Parse a HotSpot crash log (hs_err_pidXXX.log format). */ public static CrashLog fromHotspotCrashLog(String uuid, String logText) { return new HotspotCrashLogParser().parse(uuid, logText); } + + /** Parse a J9/OpenJ9 javacore dump file. */ + public static CrashLog fromJ9Javacore(String uuid, String javacoreContent) { + return new J9JavacoreParser().parse(uuid, javacoreContent); + } + + /** + * Auto-detect crash log format and parse accordingly. + * + *

Detection is based on format-specific markers: + * + *

    + *
  • J9 javacore: Contains "0SECTION" and "TITLE" markers + *
  • HotSpot hs_err: Default fallback + *
+ */ + public static CrashLog parse(String uuid, String content) { + if (isJ9Javacore(content)) { + return fromJ9Javacore(uuid, content); + } + return fromHotspotCrashLog(uuid, content); + } + + /** Check if the content appears to be a J9 javacore file. */ + static boolean isJ9Javacore(String content) { + if (content == null || content.isEmpty()) { + return false; + } + // J9 javacores have a distinctive format with 0SECTION markers + return content.contains(J9_SECTION_MARKER) && content.contains(J9_TITLE_MARKER); + } } diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/CrashUploader.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/CrashUploader.java index 179a99a199a..a8090459e6d 100644 --- a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/CrashUploader.java +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/CrashUploader.java @@ -7,37 +7,49 @@ import static datadog.trace.api.config.CrashTrackingConfig.CRASH_TRACKING_UPLOAD_TIMEOUT; import static datadog.trace.api.config.CrashTrackingConfig.CRASH_TRACKING_UPLOAD_TIMEOUT_DEFAULT; import static datadog.trace.api.telemetry.LogCollector.SEND_TELEMETRY; +import static datadog.trace.util.AgentThreadFactory.AgentThread.CRASHTRACKING_HTTP_DISPATCHER; import static datadog.trace.util.TraceUtils.normalizeServiceName; import static datadog.trace.util.TraceUtils.normalizeTagValue; +import static java.time.format.DateTimeFormatter.ISO_OFFSET_DATE_TIME; +import static java.util.concurrent.TimeUnit.MILLISECONDS; +import static java.util.concurrent.TimeUnit.SECONDS; import com.squareup.moshi.JsonWriter; import datadog.common.container.ContainerInfo; import datadog.common.version.VersionInfo; import datadog.communication.http.OkHttpUtils; import datadog.crashtracking.dto.CrashLog; +import datadog.crashtracking.dto.ErrorData; +import datadog.crashtracking.dto.OSInfo; import datadog.environment.SystemProperties; import datadog.trace.api.Config; import datadog.trace.api.DDTags; import datadog.trace.bootstrap.config.provider.ConfigProvider; +import datadog.trace.util.AgentThreadFactory; import datadog.trace.util.PidHelper; import de.thetaphi.forbiddenapis.SuppressForbidden; import edu.umd.cs.findbugs.annotations.NonNull; import java.io.*; import java.nio.charset.Charset; -import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Path; import java.time.Instant; +import java.time.ZonedDateTime; import java.util.Arrays; import java.util.HashMap; import java.util.Map; import java.util.Scanner; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.SynchronousQueue; +import java.util.concurrent.ThreadPoolExecutor; import java.util.concurrent.TimeUnit; import java.util.regex.Matcher; import java.util.regex.Pattern; import java.util.stream.Collectors; import javax.annotation.Nonnull; import okhttp3.Call; +import okhttp3.Callback; +import okhttp3.Dispatcher; import okhttp3.HttpUrl; import okhttp3.MediaType; import okhttp3.OkHttpClient; @@ -56,6 +68,8 @@ public final class CrashUploader { static final String HEADER_DD_EVP_ORIGIN = "DD-EVP-ORIGIN"; static final String JAVA_TRACING_LIBRARY = "dd-trace-java"; static final String HEADER_DD_EVP_ORIGIN_VERSION = "DD-EVP-ORIGIN-VERSION"; + static final String HEADER_DD_EVP_SUBDOMAIN = "X-Datadog-EVP-Subdomain"; + static final String ERROR_TRACKING_INTAKE = "error-tracking-intake"; static final String HEADER_DD_TELEMETRY_API_VERSION = "DD-Telemetry-API-Version"; static final String TELEMETRY_API_VERSION = "v2"; static final String HEADER_DD_TELEMETRY_REQUEST_TYPE = "DD-Telemetry-Request-Type"; @@ -64,13 +78,51 @@ public final class CrashUploader { private static final MediaType APPLICATION_JSON = MediaType.get("application/json; charset=utf-8"); + private static final class CallResult implements Callback { + private final String kind; // for logging + + private CallResult(String kind) { + this.kind = kind; + } + + @Override + public void onFailure(Call call, IOException e) { + log.error("Failed to upload {} to {}, got exception", kind, call.request().url(), e); + } + + @Override + public void onResponse(Call call, Response response) throws IOException { + if (response.isSuccessful()) { + log.info( + "Successfully uploaded the {} to {}, code = {} \"{}\"", + kind, + call.request().url(), + response.code(), + response.message()); + } else { + log.error( + "Failed to upload {} to {}, code = {} \"{}\", body = \"{}\"", + kind, + call.request().url(), + response.code(), + response.message(), + response.body() != null ? response.body().string().trim() : ""); + } + } + } + private final Config config; private final ConfigManager.StoredConfig storedConfig; + private final CrashUploaderSettings uploaderSettings; - private final OkHttpClient telemetryClient; private final HttpUrl telemetryUrl; + private final HttpUrl errorTrackingUrl; + private final OkHttpClient uploadClient; + private final Dispatcher dispatcher; + private final ExecutorService executor; private final boolean agentless; private final String tags; + private final long timeout; public CrashUploader(@Nonnull final ConfigManager.StoredConfig storedConfig) { this(Config.get(), storedConfig); @@ -80,9 +132,20 @@ public CrashUploader(@Nonnull final ConfigManager.StoredConfig storedConfig) { @NonNull final Config config, @Nonnull final ConfigManager.StoredConfig storedConfig) { this.config = config; this.storedConfig = storedConfig; - - telemetryUrl = HttpUrl.get(config.getFinalCrashTrackingTelemetryUrl()); - agentless = config.isCrashTrackingAgentless(); + this.uploaderSettings = storedConfig.toCrashUploaderSettings(); + this.telemetryUrl = HttpUrl.get(config.getFinalCrashTrackingTelemetryUrl()); + this.errorTrackingUrl = HttpUrl.get(config.getFinalCrashTrackingErrorTrackingUrl()); + this.agentless = config.isCrashTrackingAgentless(); + // This is the same thing OkHttp Dispatcher is doing except thread naming and daemonization + this.executor = + new ThreadPoolExecutor( + 0, + 4, + 60, + TimeUnit.SECONDS, + new SynchronousQueue<>(), + new AgentThreadFactory(CRASHTRACKING_HTTP_DISPATCHER)); + this.dispatcher = new Dispatcher(executor); final StringBuilder tagsBuilder = new StringBuilder(storedConfig.tags != null ? storedConfig.tags : ""); @@ -98,23 +161,34 @@ public CrashUploader(@Nonnull final ConfigManager.StoredConfig storedConfig) { ConfigProvider configProvider = config.configProvider(); - telemetryClient = + this.timeout = + SECONDS.toMillis( + configProvider.getInteger( + CRASH_TRACKING_UPLOAD_TIMEOUT, CRASH_TRACKING_UPLOAD_TIMEOUT_DEFAULT)); + + uploadClient = OkHttpUtils.buildHttpClient( config, - null, /* dispatcher */ - telemetryUrl, + dispatcher, /* dispatcher */ + telemetryUrl, // will be overridden in each request true, /* retryOnConnectionFailure */ - null, /* maxRunningRequests */ + 4, /* maxRunningRequests */ // not having one request blocking the others configProvider.getString(CRASH_TRACKING_PROXY_HOST), configProvider.getInteger(CRASH_TRACKING_PROXY_PORT), configProvider.getString(CRASH_TRACKING_PROXY_USERNAME), configProvider.getString(CRASH_TRACKING_PROXY_PASSWORD), - TimeUnit.SECONDS.toMillis( - configProvider.getInteger( - CRASH_TRACKING_UPLOAD_TIMEOUT, CRASH_TRACKING_UPLOAD_TIMEOUT_DEFAULT))); + timeout); } public void notifyCrashStarted(String error) { + sendPingToTelemetry(error); + if (storedConfig.sendToErrorTracking) { + sendPingToErrorTracking(error); + } + } + + // @VisibleForTesting + void sendPingToTelemetry(String error) { // send a ping message to the telemetry to notify that the crash report started try (Buffer buf = new Buffer(); JsonWriter writer = JsonWriter.of(buf)) { @@ -128,27 +202,93 @@ public void notifyCrashStarted(String error) { "Crashtracker crash ping: " + (error != null ? error : "crash processing started")); writer.endObject(); handleCall(makeTelemetryRequest(makeTelemetryRequestBody(buf.readUtf8(), true)), "ping"); - } catch (Throwable t) { - log.error("Failed to send crash ping", t); + log.error("Failed to prepare the telemetry crash ping payload", t); } } - public void upload(@Nonnull Path file) throws IOException { - String uuid = storedConfig.reportUUID; - uploadToLogs(file); - uploadToTelemetry(file, uuid); + // @VisibleForTesting + void sendPingToErrorTracking(String error) { + try { + final CrashLog ping = + new CrashLog( + storedConfig.reportUUID, + false, + ZonedDateTime.now().format(ISO_OFFSET_DATE_TIME), + new ErrorData( + null, + "Crashtracker crash ping: " + + (error != null ? error : "crash processing started"), + null), + null, + OSInfo.current(), + null, + null, + "1.0"); + handleCall(makeErrorTrackingRequest(makeErrorTrackingRequestBody(ping, true)), "ping"); + } catch (Throwable t) { + log.error("Failed to prepare the error tracking crash ping payload", t); + } } @SuppressForbidden - boolean uploadToLogs(@Nonnull Path file) { + public void upload(@Nonnull Path file) { + String fileContent; + try { + fileContent = new String(Files.readAllBytes(file), Charset.defaultCharset()); + } catch (Throwable t) { + log.error("Failed to collect information about the crash", t); + return; // cannot proceed further + } + + try { + uploadToLogs(fileContent, System.out); + } catch (Throwable t) { + log.error("Unable to print the error crash as a log message", t); + } try { - uploadToLogs(new String(Files.readAllBytes(file), StandardCharsets.UTF_8), System.out); - } catch (IOException e) { - log.error("Failed to upload crash file: {}", file, e); - return false; + remoteUpload(fileContent, true, storedConfig.sendToErrorTracking); + } finally { + uploadClient.dispatcher().cancelAll(); + } + } + + // @VisibleForTesting + void remoteUpload( + @Nonnull String fileContent, boolean sendToTelemetry, boolean sendToErrorTracking) { + final String uuid = storedConfig.reportUUID; + try { + // Auto-detect crash log format (HotSpot hs_err or J9 javacore) + CrashLog crashLog = CrashLogParser.parse(uuid, fileContent); + if (sendToTelemetry) { + uploadToTelemetry(crashLog); + } + if (sendToErrorTracking) { + uploadToErrorTracking(crashLog); + } + } catch (Throwable t) { + log.error("Error while before sending remotely the crash report with uuid {}", uuid, t); + } + int remaining; + long deadline = MILLISECONDS.toNanos(timeout) + System.nanoTime(); + // container that crashed + while ((remaining = dispatcher.queuedCallsCount() + dispatcher.runningCallsCount()) > 0 + && deadline > System.nanoTime()) { + try { + Thread.sleep(100); // good enough for this purpose even if we overflow + } catch (InterruptedException ie) { + Thread.currentThread().interrupt(); + } + } + if (remaining > 0) { + dispatcher.cancelAll(); + uploadClient.connectionPool().evictAll(); + log.error( + SEND_TELEMETRY, + "Failed to fully send the crash report with UUID {}. Still {} calls remaining", + uuid, + remaining); } - return true; } void uploadToLogs(@Nonnull String message, @Nonnull PrintStream out) throws IOException { @@ -172,7 +312,6 @@ void uploadToLogs(@Nonnull String message, @Nonnull PrintStream out) throws IOEx writer.endObject(); writer.endObject(); } - out.println(buf.readByteString().utf8()); } } @@ -182,7 +321,7 @@ void uploadToLogs(@Nonnull String message, @Nonnull PrintStream out) throws IOEx static String extractErrorKind(String fileContent) { Matcher matcher = ERROR_MESSAGE_PATTERN.matcher(fileContent); if (!matcher.find()) { - System.err.println("No match found for error.kind"); + log.error("No match found for error.kind"); return null; } @@ -214,7 +353,7 @@ static String extractErrorKind(String fileContent) { static String extractErrorMessage(String fileContent) { Matcher matcher = ERROR_MESSAGE_PATTERN.matcher(fileContent); if (!matcher.find()) { - System.err.println("No match found for error.message"); + log.error("No match found for error.message"); return null; } return Arrays.stream(matcher.group().split(System.lineSeparator())) @@ -255,20 +394,12 @@ private String extractErrorStackTrace(String fileContent, boolean redact) { return ""; } - boolean uploadToTelemetry(@Nonnull Path file, String uuid) { + void uploadToTelemetry(@Nonnull CrashLog crashLog) { try { - String content = new String(Files.readAllBytes(file), Charset.defaultCharset()); - CrashLog crashLog = CrashLogParser.fromHotspotCrashLog(uuid, content); - if (crashLog == null) { - log.error(SEND_TELEMETRY, "Failed to parse crash log with uuid {} ", uuid); - return false; - } handleCall(makeTelemetryRequest(makeTelemetryRequestBody(crashLog.toJson(), false)), "crash"); } catch (Throwable t) { - log.error("Failed to upload crash file: {}", file, t); - return false; + log.error("Failed to make a telemetry request", t); } - return true; } private Call makeTelemetryRequest(@Nonnull RequestBody requestBody) throws IOException { @@ -285,7 +416,7 @@ private Call makeTelemetryRequest(@Nonnull RequestBody requestBody) throws IOExc headers.put(HEADER_DD_TELEMETRY_API_VERSION, TELEMETRY_API_VERSION); headers.put(HEADER_DD_TELEMETRY_REQUEST_TYPE, TELEMETRY_REQUEST_TYPE); - return telemetryClient.newCall( + return uploadClient.newCall( OkHttpUtils.prepareRequest(telemetryUrl, headers, config, agentless) .post(requestBody) .build()); @@ -349,6 +480,182 @@ private RequestBody makeTelemetryRequestBody(@Nonnull String payload, boolean is } } + void uploadToErrorTracking(@Nonnull CrashLog crashLog) { + try { + handleCall(makeErrorTrackingRequest(makeErrorTrackingRequestBody(crashLog, false)), "crash"); + } catch (Throwable t) { + log.error("Failed to make a error tracking request", t); + } + } + + private Call makeErrorTrackingRequest(@Nonnull RequestBody requestBody) throws IOException { + final Map headers = new HashMap<>(); + // Set chunked transfer + MediaType contentType = requestBody.contentType(); + if (contentType != null) { + headers.put("Content-Type", contentType.toString()); + } + headers.put("Content-Length", Long.toString(requestBody.contentLength())); + headers.put("Transfer-Encoding", "chunked"); + headers.put(HEADER_DD_EVP_ORIGIN, JAVA_TRACING_LIBRARY); + headers.put(HEADER_DD_EVP_ORIGIN_VERSION, VersionInfo.VERSION); + if (!agentless) { + headers.put(HEADER_DD_EVP_SUBDOMAIN, ERROR_TRACKING_INTAKE); + } + + return uploadClient.newCall( + OkHttpUtils.prepareRequest(errorTrackingUrl, headers, config, agentless) + .post(requestBody) + .build()); + } + + private RequestBody makeErrorTrackingRequestBody(@Nonnull CrashLog payload, boolean isPing) + throws IOException { + try (Buffer buf = new Buffer()) { + try (JsonWriter writer = JsonWriter.of(buf)) { + writer.beginObject(); + writer.name("timestamp").value(payload.timestamp); + writer.name("ddsource").value("crashtracker"); + // tags + writer.name("ddtags").value(tagsForErrorTracking(payload.uuid, isPing, payload.incomplete)); + // error payload + if (payload.error != null) { + writer.name("error"); + writer.beginObject(); + if (!isPing) { + writer.name("is_crash").value(true); + } + writer.name("type").value(payload.error.kind); + writer.name("message").value(payload.error.message); + if (uploaderSettings.isExtendedInfoEnabled() && payload.error.threadName != null) { + writer.name("thread_name").value(payload.error.threadName); + } + writer.name("source_type").value("Crashtracking"); + if (payload.error.stack != null) { + writer.name("stack"); + // flat write an already serialized json object + payload.error.stack.writeAsJson(writer); + } + writer.endObject(); + } + // signal info + if (payload.sigInfo != null) { + writer.name("sig_info"); + writer.beginObject(); + if (payload.sigInfo.address != null) { + writer.name("si_addr").value(payload.sigInfo.address); + } + if (payload.sigInfo.name != null) { + writer.name("si_signo_human_readable").value(payload.sigInfo.name); + writer.name("si_signo").value(payload.sigInfo.number); + } + if (payload.sigInfo.action != null) { + writer.name("si_code").value(payload.sigInfo.code); + writer.name("si_code_human_readable").value(payload.sigInfo.action); + } + if (payload.sigInfo.pid != null) { + writer.name("si_pid").value(payload.sigInfo.pid); + } + if (payload.sigInfo.uid != null) { + writer.name("si_uid").value(payload.sigInfo.uid); + } + writer.endObject(); + } + + // os info + if (payload.osInfo != null) { + writer.name("os_info"); + writer.beginObject(); + writer.name("architecture").value(payload.osInfo.architecture); + writer.name("bitness").value(payload.osInfo.bitness); + writer.name("os_type").value(payload.osInfo.osType); + writer + .name("version") + .value( + SystemProperties.get( + "os.version")); // this has been restructured under OsInfo so taking raw here + writer.endObject(); + } + // experimental + if (payload.experimental != null + && (payload.experimental.ucontext != null + || payload.experimental.runtimeArgs != null)) { + writer.name("experimental"); + writer.beginObject(); + if (payload.experimental.ucontext != null) { + writer.name("ucontext"); + writer.beginObject(); + for (Map.Entry entry : payload.experimental.ucontext.entrySet()) { + writer.name(entry.getKey()).value(entry.getValue()); + } + writer.endObject(); + } + if (uploaderSettings.isExtendedInfoEnabled() + && payload.experimental.runtimeArgs != null) { + writer.name("runtime_args"); + writer.beginArray(); + for (String arg : payload.experimental.runtimeArgs) { + writer.value(arg); + } + writer.endArray(); + } + writer.endObject(); + } + // files (e.g. /proc/self/maps or dynamic_libraries) + if (uploaderSettings.isExtendedInfoEnabled() && payload.files != null) { + writer.name("files"); + writer.beginObject(); + writer.name(payload.files.name); + writer.beginArray(); + for (String fileLine : payload.files.lines) { + writer.value(fileLine); + } + writer.endArray(); + writer.endObject(); + } + writer.endObject(); + } + return RequestBody.create(APPLICATION_JSON, buf.readByteString()); + } + } + + private String tagsForErrorTracking(String uuid, boolean isPing, boolean incomplete) { + final StringBuilder tags = new StringBuilder(); + if (storedConfig.tags != null) { + // normally it does not happen + tags.append(storedConfig.tags); + } else { + tags.append("service:") + .append(normalizeServiceName(storedConfig.service)); // ensure the service name is there + } + if (isPing) { + tags.append(",").append("is_crash_ping:true"); + } else { + tags.append(",").append("is_crash:true"); + if (incomplete) { + tags.append(",").append("incomplete:true"); + } + } + tags.append(",").append("data_schema_version:1.0"); + tags.append(",").append("language_name:jvm"); + tags.append(",") + .append("language_version:") + .append(normalizeTagValue(SystemProperties.getOrDefault("java.version", "unknown"))); + tags.append(",") + .append("runtime_version:") + .append( + normalizeTagValue(SystemProperties.getOrDefault("java.runtime.version", "unknown"))); + tags.append(",") + .append("runtime_vendor:") + .append(normalizeTagValue(SystemProperties.getOrDefault("java.vendor", "unknown"))); + tags.append(",") + .append("runtime_name:") + .append(normalizeTagValue(SystemProperties.getOrDefault("java.runtime.name", "unknown"))); + tags.append(",").append("tracer_version:").append(normalizeTagValue(VersionInfo.VERSION)); + tags.append(",").append("uuid:").append(uuid); + return (tags.toString()); + } + private String tagsForPing(String uuid) { final StringBuilder tags = new StringBuilder("is_crash_ping:true"); tags.append(",").append("language_name:jvm"); @@ -356,40 +663,22 @@ private String tagsForPing(String uuid) { tags.append(",") .append("language_version:") .append(normalizeTagValue(SystemProperties.getOrDefault("java.version", "unknown"))); + tags.append(",") + .append("runtime_version:") + .append( + normalizeTagValue(SystemProperties.getOrDefault("java.runtime.version", "unknown"))); + tags.append(",") + .append("runtime_vendor:") + .append(normalizeTagValue(SystemProperties.getOrDefault("java.vendor", "unknown"))); + tags.append(",") + .append("runtime_name:") + .append(normalizeTagValue(SystemProperties.getOrDefault("java.runtime.name", "unknown"))); tags.append(",").append("tracer_version:").append(normalizeTagValue(VersionInfo.VERSION)); tags.append(",").append("uuid:").append(uuid); return (tags.toString()); } private void handleCall(final Call call, String kind) { - try (Response response = call.execute()) { - handleSuccess(call, response, kind); - } catch (Throwable t) { - handleFailure(t, kind); - } - } - - private void handleSuccess(final Call call, final Response response, String kind) - throws IOException { - if (response.isSuccessful()) { - log.info( - "Successfully uploaded the crash {} to {}, code = {} \"{}\"", - kind, - call.request().url(), - response.code(), - response.message()); - } else { - log.error( - "Failed to upload crash {} to {}, code = {} \"{}\", body = \"{}\"", - kind, - call.request().url(), - response.code(), - response.message(), - response.body() != null ? response.body().string().trim() : ""); - } - } - - private void handleFailure(final Throwable exception, String kind) { - log.error("Failed to upload crash {}, got exception", kind, exception); + call.enqueue(new CallResult(kind)); } } diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/CrashUploaderScriptInitializer.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/CrashUploaderScriptInitializer.java index fdfad88bb7f..0e810694be1 100644 --- a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/CrashUploaderScriptInitializer.java +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/CrashUploaderScriptInitializer.java @@ -2,13 +2,9 @@ import static datadog.crashtracking.ConfigManager.writeConfigToPath; import static datadog.crashtracking.Initializer.LOG; -import static datadog.crashtracking.Initializer.RWXRWXRWX; -import static datadog.crashtracking.Initializer.R_XR_XR_X; import static datadog.crashtracking.Initializer.findAgentJar; import static datadog.crashtracking.Initializer.getCrashUploaderTemplate; import static datadog.trace.api.telemetry.LogCollector.SEND_TELEMETRY; -import static java.nio.file.attribute.PosixFilePermissions.asFileAttribute; -import static java.nio.file.attribute.PosixFilePermissions.fromString; import static java.util.Locale.ROOT; import datadog.environment.SystemProperties; @@ -16,13 +12,13 @@ import datadog.trace.util.Strings; import java.io.BufferedReader; import java.io.BufferedWriter; +import java.io.File; +import java.io.FileOutputStream; import java.io.IOException; import java.io.InputStream; import java.io.InputStreamReader; -import java.nio.file.FileAlreadyExistsException; -import java.nio.file.Files; -import java.nio.file.Path; -import java.nio.file.Paths; +import java.io.OutputStreamWriter; +import java.nio.charset.StandardCharsets; public final class CrashUploaderScriptInitializer { private static final String SETUP_FAILURE_MESSAGE = "Crash tracking will not work properly."; @@ -31,6 +27,11 @@ private CrashUploaderScriptInitializer() {} // @VisibleForTests static void initialize(String onErrorVal, String onErrorFile) { + initialize(onErrorVal, onErrorFile, null); + } + + // @VisibleForTests + static void initialize(String onErrorVal, String onErrorFile, String javacorePath) { if (onErrorVal == null || onErrorVal.isEmpty()) { LOG.debug( SEND_TELEMETRY, "'-XX:OnError' argument was not provided. Crash tracking is disabled."); @@ -49,67 +50,78 @@ static void initialize(String onErrorVal, String onErrorFile) { return; } - Path scriptPath = Paths.get(onErrorVal.replace(" %p", "")); + File scriptFile = new File(onErrorVal.replace(" %p", "")); boolean isDDCrashUploader = - scriptPath.getFileName().toString().toLowerCase(ROOT).contains("dd_crash_uploader"); - if (isDDCrashUploader && !copyCrashUploaderScript(scriptPath, onErrorFile, agentJar)) { + scriptFile.getName().toLowerCase(ROOT).contains("dd_crash_uploader"); + if (isDDCrashUploader && !copyCrashUploaderScript(scriptFile, onErrorFile, agentJar)) { return; } - writeConfigToPath(scriptPath, "agent", agentJar, "hs_err", onErrorFile); + if (javacorePath != null && !javacorePath.isEmpty()) { + writeConfigToPath(scriptFile, "agent", agentJar, "javacore_path", javacorePath); + } else { + writeConfigToPath(scriptFile, "agent", agentJar, "hs_err", onErrorFile); + } } private static boolean copyCrashUploaderScript( - Path scriptPath, String onErrorFile, String agentJar) { - Path scriptDirectory = scriptPath.getParent(); - try { - Files.createDirectories(scriptDirectory, asFileAttribute(fromString(RWXRWXRWX))); - } catch (UnsupportedOperationException e) { - LOG.warn( - SEND_TELEMETRY, - "Unsupported permissions '" + RWXRWXRWX + "' for {}. " + SETUP_FAILURE_MESSAGE, - scriptDirectory); - return false; - } catch (FileAlreadyExistsException ignored) { - // can be safely ignored; if the folder exists we will just reuse it - if (!Files.isWritable(scriptDirectory)) { + File scriptFile, String onErrorFile, String agentJar) { + File scriptDirectory = scriptFile.getParentFile(); + if (!scriptDirectory.exists()) { + if (!scriptDirectory.mkdirs()) { LOG.warn( - SEND_TELEMETRY, "Read only directory {}. " + SETUP_FAILURE_MESSAGE, scriptDirectory); + SEND_TELEMETRY, + "Failed to create writable crash tracking script folder {}. " + SETUP_FAILURE_MESSAGE, + scriptDirectory); return false; } - } catch (IOException e) { - LOG.warn( - SEND_TELEMETRY, - "Failed to create writable crash tracking script folder {}. " + SETUP_FAILURE_MESSAGE, - scriptDirectory); + boolean permissionFailure = false; + permissionFailure |= !scriptDirectory.setReadable(true, false); + permissionFailure |= !scriptDirectory.setWritable(true, false); + permissionFailure |= !scriptDirectory.setExecutable(true, false); + if (permissionFailure) { + LOG.warn( + SEND_TELEMETRY, + "Failed to set permissions on crash tracking script folder {}. {}", + scriptDirectory, + SETUP_FAILURE_MESSAGE); + } + } + if (!scriptDirectory.canWrite()) { + LOG.warn(SEND_TELEMETRY, "Read only directory {}. " + SETUP_FAILURE_MESSAGE, scriptDirectory); return false; } try { - LOG.debug("Writing crash uploader script: {}", scriptPath); - writeCrashUploaderScript(getCrashUploaderTemplate(), scriptPath, agentJar, onErrorFile); + LOG.debug("Writing crash uploader script: {}", scriptFile); + writeCrashUploaderScript(getCrashUploaderTemplate(), scriptFile, agentJar, onErrorFile); } catch (IOException e) { LOG.warn( SEND_TELEMETRY, "Failed to copy crash tracking script {}. " + SETUP_FAILURE_MESSAGE, - scriptPath); + scriptFile); return false; } return true; } private static void writeCrashUploaderScript( - InputStream template, Path scriptPath, String execClass, String crashFile) + InputStream template, File scriptFile, String execClass, String crashFile) throws IOException { - if (!Files.exists(scriptPath)) { + if (!scriptFile.exists()) { try (BufferedReader br = new BufferedReader(new InputStreamReader(template)); - BufferedWriter bw = Files.newBufferedWriter(scriptPath)) { + BufferedWriter bw = + new BufferedWriter( + new OutputStreamWriter( + new FileOutputStream(scriptFile), StandardCharsets.UTF_8))) { String line; while ((line = br.readLine()) != null) { bw.write(template(line, execClass, crashFile)); bw.newLine(); } } - Files.setPosixFilePermissions(scriptPath, fromString(R_XR_XR_X)); + scriptFile.setReadable(true, false); + scriptFile.setWritable(false, false); + scriptFile.setExecutable(true, false); } } diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/CrashUploaderSettings.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/CrashUploaderSettings.java new file mode 100644 index 00000000000..15a79834f50 --- /dev/null +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/CrashUploaderSettings.java @@ -0,0 +1,14 @@ +package datadog.crashtracking; + +/** Immutable settings that control what data {@link CrashUploader} includes in uploaded reports. */ +public final class CrashUploaderSettings { + final boolean extendedInfoEnabled; + + CrashUploaderSettings(boolean extendedInfoEnabled) { + this.extendedInfoEnabled = extendedInfoEnabled; + } + + boolean isExtendedInfoEnabled() { + return extendedInfoEnabled; + } +} diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/Initializer.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/Initializer.java index 964d3a0b2ec..0aeca69ed00 100644 --- a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/Initializer.java +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/Initializer.java @@ -1,32 +1,29 @@ package datadog.crashtracking; import static datadog.trace.api.telemetry.LogCollector.SEND_TELEMETRY; -import static java.util.Comparator.reverseOrder; import static java.util.Locale.ROOT; import com.datadoghq.profiler.JVMAccess; import com.sun.management.HotSpotDiagnosticMXBean; +import datadog.environment.JavaVirtualMachine; import datadog.environment.OperatingSystem; +import datadog.environment.SystemProperties; import datadog.libs.ddprof.DdprofLibraryLoader; +import datadog.trace.api.Platform; import datadog.trace.util.TempLocationManager; -import java.io.IOException; +import java.io.File; import java.io.InputStream; import java.lang.management.ManagementFactory; import java.net.URL; -import java.nio.file.Files; -import java.nio.file.Path; -import java.nio.file.Paths; +import java.util.Arrays; +import java.util.List; import java.util.StringTokenizer; -import java.util.function.Predicate; -import java.util.stream.Stream; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public final class Initializer { static final Logger LOG = LoggerFactory.getLogger(Initializer.class); static final String PID_PREFIX = "_pid"; - static final String RWXRWXRWX = "rwxrwxrwx"; - static final String R_XR_XR_X = "r-xr-xr-x"; private interface FlagAccess { String getValue(String flagName); @@ -74,9 +71,15 @@ public boolean setValue(String flagName, String value) { } public static boolean initialize(boolean forceJmx) { + // J9/OpenJ9 requires different initialization path + if (JavaVirtualMachine.isJ9()) { + return initializeJ9(); + } + try { FlagAccess access = null; - if (forceJmx) { + // Native images don't support the native ddprof library, use JMX instead + if (forceJmx || Platform.isNativeImage()) { access = new JMXFlagAccess(ManagementFactory.getPlatformMXBean(HotSpotDiagnosticMXBean.class)); } else { @@ -85,9 +88,8 @@ public static boolean initialize(boolean forceJmx) { if (reasonNotLoaded != null) { LOG.debug( SEND_TELEMETRY, - "Failed to load JVM access library: " - + jvmAccessHolder.getReasonNotLoaded().getMessage() - + ". Crash tracking will need to rely on user provided JVM arguments."); + "Failed to load JVM access library: {}. Crash tracking will need to rely on user provided JVM arguments.", + jvmAccessHolder.getReasonNotLoaded().getMessage()); return false; } else { JVMAccess.Flags flags = jvmAccessHolder.getComponent().flags(); @@ -104,6 +106,148 @@ public static boolean initialize(boolean forceJmx) { return false; } + /** + * Initialize crash tracking for J9/OpenJ9 JVMs. + * + *

Unlike HotSpot, J9's -Xdump option cannot be modified at runtime. This method: + * + *

    + *
  1. Checks if -Xdump:tool is already configured + *
  2. Deploys the crash uploader script if configured + *
  3. Logs instructions for manual configuration if not configured + *
+ * + * @return true if -Xdump is properly configured, false otherwise + */ + private static boolean initializeJ9() { + try { + // Check if -Xdump:tool is already configured via JVM arguments + boolean xdumpConfigured = isXdumpToolConfigured(); + // Get custom javacore path if configured + String javacorePath = getJ9JavacorePath(); + if (javacorePath == null || javacorePath.isEmpty()) { + // OpenJ9 defaults javacore output to the JVM working directory. Persist that location in + // the uploader config so the crash script does not need to guess from its own cwd. + javacorePath = SystemProperties.get("user.dir"); + } + + if (xdumpConfigured) { + LOG.debug("J9 crash tracking: -Xdump:tool already configured, crash uploads enabled"); + // Use the path from the -Xdump:tool arg when available (allows callers to specify a known + // path via -Xdump:tool:events=gpf+abort,exec=\ %pid), falling back to the default + // TempLocationManager path when the path cannot be extracted. + String extractedPath = extractJ9ScriptPathFromXdumpArg(); + String scriptPath = extractedPath != null ? extractedPath : getJ9CrashUploaderScriptPath(); + // Initialize the crash uploader script and config manager + CrashUploaderScriptInitializer.initialize(scriptPath, null, javacorePath); + // Also set up OOME notifier script + String oomeScript = getScript("dd_oome_notifier"); + OOMENotifierScriptInitializer.initialize(oomeScript); + return true; + } else { + String scriptPath = getJ9CrashUploaderScriptPath(); + // Log instructions for manual configuration + LOG.info("J9 JVM detected. To enable crash tracking, add this JVM argument at startup:"); + LOG.info(" -Xdump:tool:events=gpf+abort,exec={}\\ %pid", scriptPath); + LOG.info( + "Crash tracking will not be active until this argument is added and JVM is restarted."); + return false; + } + } catch (Throwable t) { + logInitializationError( + "Unexpected exception while initializing J9 crash tracking. Crash tracking will not work.", + t); + } + return false; + } + + /** + * Extract the crash uploader script path from the {@code -Xdump:tool} JVM argument. + * + *

Looks for a JVM argument of the form {@code + * -Xdump:tool:events=...,exec=/path/to/dd_crash_uploader.sh\ %pid} and returns the script path + * portion (before the {@code \ %pid} argument separator). + * + * @return the script path, or {@code null} if not found or not extractable + */ + private static String extractJ9ScriptPathFromXdumpArg() { + List vmArgs = JavaVirtualMachine.getVmOptions(); + for (String arg : vmArgs) { + if (arg.startsWith("-Xdump:tool") && arg.contains("dd_crash_uploader")) { + int execIdx = arg.indexOf("exec="); + if (execIdx >= 0) { + String execVal = arg.substring(execIdx + 5); + // Separator between command and args: plain space, or "\ " (backslash + space) as + // suggested by the Initializer's log hint. Check plain space first since that is the + // form that actually works when the shell splits the exec string into tokens. + int spaceIdx = execVal.indexOf(' '); + if (spaceIdx >= 0) { + String candidate = execVal.substring(0, spaceIdx); + // Strip a trailing backslash left over from the "\ %pid" notation + return candidate.endsWith("\\") + ? candidate.substring(0, candidate.length() - 1) + : candidate; + } + return execVal; + } + } + } + return null; + } + + /** + * Get the custom javacore file path from -Xdump:java:file=... JVM argument. + * + * @return the custom javacore path, or null if not configured + */ + private static String getJ9JavacorePath() { + List vmArgs = JavaVirtualMachine.getVmOptions(); + for (String arg : vmArgs) { + if (arg.startsWith("-Xdump:java:file=") || arg.startsWith("-Xdump:java+heap:file=")) { + int fileIdx = arg.indexOf("file="); + if (fileIdx >= 0) { + String path = arg.substring(fileIdx + 5); + // Handle comma-separated options: -Xdump:java:file=/path,request=exclusive + int commaIdx = path.indexOf(','); + if (commaIdx > 0) { + path = path.substring(0, commaIdx); + } + return path; + } + } + } + return null; + } + + /** + * Check if -Xdump:tool is configured with our crash uploader script. + * + *

Looks for JVM arguments matching: -Xdump:tool:events=...,exec=...dd_crash_uploader... + */ + private static boolean isXdumpToolConfigured() { + List vmArgs = JavaVirtualMachine.getVmOptions(); + for (String arg : vmArgs) { + if (arg.startsWith("-Xdump:tool") && arg.contains("dd_crash_uploader")) { + return true; + } + } + return false; + } + + /** + * Get the path where the crash uploader script should be deployed for J9. + * + *

Note: The actual script deployment is handled by {@link CrashUploaderScriptInitializer} when + * initialize() is called with this path. + * + * @return the full path for the crash uploader script + */ + private static String getJ9CrashUploaderScriptPath() { + String scriptFileName = getScriptFileName("dd_crash_uploader"); + String tempDir = TempLocationManager.getInstance().getTempDir().toString(); + return tempDir + File.separator + scriptFileName; + } + static InputStream getCrashUploaderTemplate() { String name = OperatingSystem.isWindows() ? "upload_crash.bat" : "upload_crash.sh"; return CrashUploader.class.getResourceAsStream(name); @@ -129,19 +273,11 @@ static String findAgentJar() { else if (selfClass.startsWith("file:")) { int idx = selfClass.lastIndexOf("dd-java-agent"); if (idx > -1) { - Path libsPath = Paths.get(selfClass.substring(5, idx + 13), "build", "libs"); - try (Stream files = Files.walk(libsPath)) { - Predicate isJarFile = - p -> p.getFileName().toString().toLowerCase(ROOT).endsWith(".jar"); - agentPath = - files - .sorted(reverseOrder()) - .filter(isJarFile) - .findFirst() - .map(Path::toString) - .orElse(null); - } catch (IOException ignored) { - // Ignore failure to get agent path + File libsDir = new File(selfClass.substring(5, idx + 13), "build/libs"); + File[] jars = libsDir.listFiles(f -> f.getName().toLowerCase(ROOT).endsWith(".jar")); + if (jars != null && jars.length > 0) { + Arrays.sort(jars, (a, b) -> b.getName().compareTo(a.getName())); + agentPath = jars[0].getAbsolutePath(); } } } @@ -230,7 +366,8 @@ private static void initializeCrashUploader(FlagAccess flags) { if (!rslt && LOG.isDebugEnabled()) { LOG.debug( SEND_TELEMETRY, - "Unable to set OnError flag to " + onErrorVal + ". Crash-tracking may not work."); + "Unable to set OnError flag to {}. Crash-tracking may not work.", + onErrorVal); } CrashUploaderScriptInitializer.initialize(uploadScript, onErrorFile); @@ -269,9 +406,8 @@ private static void initializeOOMENotifier(FlagAccess flags) { if (!rslt && LOG.isDebugEnabled()) { LOG.debug( SEND_TELEMETRY, - "Unable to set OnOutOfMemoryError flag to " - + onOutOfMemoryVal - + ". OOME tracking may not work."); + "Unable to set OnOutOfMemoryError flag to {}. OOME tracking may not work.", + onOutOfMemoryVal); } OOMENotifierScriptInitializer.initialize(notifierScript); @@ -298,7 +434,8 @@ private static void logInitializationError(String msg, Throwable t) { } else { LOG.warn( SEND_TELEMETRY, - msg + " [{}] (Change the logging level to debug to see the full stacktrace)", + "{} [{}] (Change the logging level to debug to see the full stacktrace)", + msg, t.getMessage()); } } diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/OOMENotifier.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/OOMENotifier.java index 4e8486c1c39..0013c4062dd 100644 --- a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/OOMENotifier.java +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/OOMENotifier.java @@ -1,8 +1,8 @@ package datadog.crashtracking; -import static datadog.communication.monitor.DDAgentStatsDClientManager.statsDClientManager; +import static datadog.metrics.impl.statsd.DDAgentStatsDClientManager.statsDClientManager; -import datadog.trace.api.StatsDClient; +import datadog.metrics.api.statsd.StatsDClient; import de.thetaphi.forbiddenapis.SuppressForbidden; import java.util.concurrent.locks.LockSupport; import org.slf4j.Logger; diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/OOMENotifierScriptInitializer.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/OOMENotifierScriptInitializer.java index 668dfecd612..c558d3a83a5 100644 --- a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/OOMENotifierScriptInitializer.java +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/OOMENotifierScriptInitializer.java @@ -2,26 +2,17 @@ import static datadog.crashtracking.ConfigManager.writeConfigToPath; import static datadog.crashtracking.Initializer.LOG; -import static datadog.crashtracking.Initializer.RWXRWXRWX; -import static datadog.crashtracking.Initializer.R_XR_XR_X; import static datadog.crashtracking.Initializer.findAgentJar; import static datadog.crashtracking.Initializer.getOomeNotifierTemplate; import static datadog.crashtracking.Initializer.getScriptPathFromArg; import static datadog.crashtracking.Initializer.pidFromSpecialFileName; import static datadog.trace.api.telemetry.LogCollector.SEND_TELEMETRY; -import static java.nio.file.FileVisitResult.CONTINUE; -import static java.nio.file.attribute.PosixFilePermissions.asFileAttribute; -import static java.nio.file.attribute.PosixFilePermissions.fromString; import datadog.trace.util.PidHelper; +import java.io.File; +import java.io.FileOutputStream; import java.io.IOException; -import java.nio.file.FileAlreadyExistsException; -import java.nio.file.FileVisitResult; -import java.nio.file.FileVisitor; -import java.nio.file.Files; -import java.nio.file.Path; -import java.nio.file.Paths; -import java.nio.file.attribute.BasicFileAttributes; +import java.io.InputStream; import java.util.Set; public final class OOMENotifierScriptInitializer { @@ -37,8 +28,8 @@ static void initialize(String onOutOfMemoryVal) { "'-XX:OnOutOfMemoryError' argument was not provided. OOME tracking is disabled."); return; } - Path scriptPath = getOOMEScriptPath(onOutOfMemoryVal); - if (scriptPath == null) { + File scriptFile = getOOMEScriptFile(onOutOfMemoryVal); + if (scriptFile == null) { LOG.error( SEND_TELEMETRY, "OOME notifier script value ({}) does not follow the expected format: /dd_oome_notifier.(sh|bat) %p. OOME tracking is disabled.", @@ -52,123 +43,99 @@ static void initialize(String onOutOfMemoryVal) { "Unable to locate the agent jar. OOME notification will not work properly."); return; } - if (!copyOOMEscript(scriptPath)) { + if (!copyOOMEscript(scriptFile)) { return; } - writeConfigToPath(scriptPath, "agent", agentJar); + writeConfigToPath(scriptFile, "agent", agentJar); } - private static Path getOOMEScriptPath(String onOutOfMemoryVal) { + private static File getOOMEScriptFile(String onOutOfMemoryVal) { String path = getScriptPathFromArg(onOutOfMemoryVal, OOME_NOTIFIER_SCRIPT_PREFIX); - return path == null ? null : Paths.get(path); + return path == null ? null : new File(path); } - private static boolean copyOOMEscript(Path scriptPath) { - Path scriptDirectory = scriptPath.getParent(); + private static boolean copyOOMEscript(File scriptFile) { + File scriptDirectory = scriptFile.getParentFile(); // cleanup all stale process-specific generated files in the parent folder of the given OOME // notifier script - ScriptCleanupVisitor.run(scriptDirectory); - - try { - if (Files.exists(scriptDirectory)) { - // can be safely ignored; if the folder exists we will just reuse it - if (!Files.isWritable(scriptDirectory)) { - LOG.warn( - SEND_TELEMETRY, - "Read only directory {}. OOME notification will not work properly.", - scriptDirectory); - return false; - } - } else { - Files.createDirectories(scriptDirectory, asFileAttribute(fromString(RWXRWXRWX))); + runScriptCleanup(scriptDirectory); + + if (scriptDirectory.exists()) { + // can be safely ignored; if the folder exists we will just reuse it + if (!scriptDirectory.canWrite()) { + LOG.warn( + SEND_TELEMETRY, + "Read only directory {}. OOME notification will not work properly.", + scriptDirectory); + return false; } - } catch (UnsupportedOperationException e) { - LOG.warn( - SEND_TELEMETRY, - "Unsupported permissions '{" - + RWXRWXRWX - + "' for {}. OOME notification will not work properly.", - scriptDirectory); - return false; - } catch (FileAlreadyExistsException ignored) { - LOG.warn(SEND_TELEMETRY, "Path {} already exists and is not a directory.", scriptDirectory); - return false; - } catch (IOException e) { - LOG.warn( - SEND_TELEMETRY, - "Failed to create writable OOME script folder {}. OOME notification will not work properly.", - scriptDirectory); - return false; + } else { + if (!scriptDirectory.mkdirs()) { + LOG.warn( + SEND_TELEMETRY, + "Failed to create writable OOME script folder {}. OOME notification will not work properly.", + scriptDirectory); + return false; + } + scriptDirectory.setReadable(true, false); + scriptDirectory.setWritable(true, false); + scriptDirectory.setExecutable(true, false); } try { // do not overwrite existing - if (!Files.exists(scriptPath)) { - Files.copy(getOomeNotifierTemplate(), scriptPath); + if (!scriptFile.exists()) { + copyStream(getOomeNotifierTemplate(), scriptFile); } - Files.setPosixFilePermissions(scriptPath, fromString(R_XR_XR_X)); + scriptFile.setReadable(true, false); + scriptFile.setWritable(false, false); + scriptFile.setExecutable(true, false); } catch (IOException e) { LOG.warn( SEND_TELEMETRY, "Failed to copy OOME script {}. OOME notification will not work properly.", - scriptPath); + scriptFile); return false; } return true; } - private static class ScriptCleanupVisitor implements FileVisitor { - private Set pidSet; - - static void run(Path dir) { - try { - if (Files.exists(dir)) { - Files.walkFileTree(dir, new ScriptCleanupVisitor()); - } - } catch (IOException e) { - if (LOG.isDebugEnabled()) { - LOG.info("Failed cleaning up process specific files in {}", dir, e); - } else { - LOG.info("Failed cleaning up process specific files in {}: {}", dir, e.toString()); - } + private static void copyStream(InputStream in, File dest) throws IOException { + try (InputStream src = in; + FileOutputStream out = new FileOutputStream(dest)) { + byte[] buf = new byte[4096]; + int n; + while ((n = src.read(buf)) >= 0) { + out.write(buf, 0, n); } } + } - private ScriptCleanupVisitor() {} - - @Override - public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) { - return CONTINUE; + private static void runScriptCleanup(File dir) { + if (!dir.exists()) { + return; } - - @Override - public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException { - String fileName = file.getFileName().toString(); - String pid = pidFromSpecialFileName(fileName); + File[] files = dir.listFiles(); + if (files == null) { + return; + } + Set pidSet = null; + for (File file : files) { + if (!file.isFile()) { + continue; + } + String pid = pidFromSpecialFileName(file.getName()); if (pid != null && !pid.equals(PidHelper.getPid())) { - if (this.pidSet == null) { - // if pidSet is not initialized, initialize it - // this will fork jps to get the list of Java PIDs - this.pidSet = PidHelper.getJavaPids(); + if (pidSet == null) { + // lazy init: forks jps to get the list of running Java PIDs + pidSet = PidHelper.getJavaPids(); } - if (!this.pidSet.contains(pid)) { + if (!pidSet.contains(pid)) { LOG.debug("Cleaning process specific file {}", file); - Files.delete(file); + file.delete(); } } - return CONTINUE; - } - - @Override - public FileVisitResult visitFileFailed(Path file, IOException exc) { - LOG.debug("Failed to delete file {}", file, exc); - return CONTINUE; - } - - @Override - public FileVisitResult postVisitDirectory(Path dir, IOException exc) { - return CONTINUE; } } } diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/buildid/BuildIdCollector.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/buildid/BuildIdCollector.java new file mode 100644 index 00000000000..cc55dfe5c29 --- /dev/null +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/buildid/BuildIdCollector.java @@ -0,0 +1,194 @@ +package datadog.crashtracking.buildid; + +import static datadog.crashtracking.buildid.BuildInfo.EMPTY; +import static java.util.concurrent.TimeUnit.MILLISECONDS; +import static java.util.concurrent.TimeUnit.SECONDS; + +import datadog.common.queue.Queues; +import datadog.trace.util.AgentTaskScheduler; +import java.nio.file.Path; +import java.util.HashSet; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.locks.LockSupport; +import org.jctools.queues.MessagePassingQueue; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Collects build IDs from library files asynchronously. + * + *

Threading Model

+ * + * This class follows a single-producer, single-consumer (SPSC) threading model: + * + *
    + *
  • Producer (single-threaded): The crash parsing flow calls {@link + * #resolveBuildId(Path)} to enqueue libraries for processing. This method is guaranteed to be + * called from a single thread. + *
  • Consumer (single-threaded): The background {@link Collector} thread processes the + * work queue. Only one collector instance is ever started, enforced by {@code + * collecting.compareAndSet(false, true)}. + *
+ * + *

Synchronization Strategy

+ * + *
    + *
  • workQueue: An SPSC (Single Producer Single Consumer) queue - thread-safe for one + * producer and one consumer without additional synchronization. + *
  • processed: A plain {@link HashSet} - safe because it's only accessed from the + * producer thread (crash parsing flow). + *
  • libraryBuildInfo: A {@link ConcurrentHashMap} - accessed from both producer + * (removal) and consumer (insertion) threads, requires concurrent access. + *
  • collecting: An {@link AtomicBoolean} - coordinates lifecycle and ensures exactly one + * collector is started. + *
+ */ +public class BuildIdCollector { + static final Logger LOGGER = LoggerFactory.getLogger(BuildIdCollector.class); + + /** Thread-safe map: accessed by both producer and consumer threads. */ + private final Map libraryBuildInfo = new ConcurrentHashMap<>(); + + /** Tracks processed filenames. Only accessed from producer thread - no synchronization needed. */ + private final Set processed = new HashSet<>(); + + /** Ensures exactly one collector thread is started. */ + private final AtomicBoolean collecting = new AtomicBoolean(false); + + /** SPSC queue: one producer (crash parsing), one consumer (collector thread). */ + private final MessagePassingQueue workQueue = Queues.spscArrayQueue(Short.MAX_VALUE); + + /** Signals when collection is complete. */ + private final CountDownLatch latch = new CountDownLatch(1); + + /** + * Consumer thread that processes the work queue and extracts build IDs. + * + *

Threading: Runs in a single background thread. Only one instance is ever created, + * guaranteed by the {@code collecting.compareAndSet(false, true)} check in {@link + * #resolveBuildId(Path)}. + * + *

Polls the {@code workQueue} until either: + * + *

    + *
  • The deadline is reached, or + *
  • The {@code collecting} flag is set to false (via {@link #awaitCollectionDone(int)}) and + * the queue is empty + *
+ */ + class Collector implements Runnable { + private final BuildIdExtractor extractor = BuildIdExtractor.create(); + private final long deadline; + + Collector(long timeout, TimeUnit unit) { + this.deadline = unit.toNanos(timeout) + System.nanoTime(); + } + + @Override + public void run() { + while (System.nanoTime() <= deadline) { + final Path path = workQueue.poll(); + if (path == null) { + if (!collecting.get()) { + break; + } + LockSupport.parkNanos(MILLISECONDS.toNanos(50)); + continue; + } + final String fileName = path.getFileName().toString(); + LOGGER.debug("Resolving build id for {} against {}", fileName, path); + final String buildId = extractor.extractBuildId(path); + if (buildId != null) { + LOGGER.debug("Found build id {} for library {}", buildId, fileName); + libraryBuildInfo.put( + fileName, new BuildInfo(buildId, extractor.buildIdType(), extractor.fileType())); + } + } + latch.countDown(); + } + } + + /** + * Registers a library filename as needing build ID resolution. + * + *

Called from producer thread (crash parsing flow) before collection starts. + * + * @param filename the library filename to track + */ + public void addUnprocessedLibrary(String filename) { + if (!collecting.get()) { + libraryBuildInfo.putIfAbsent(filename, EMPTY); + } + } + + /** + * Enqueues a library path for build ID extraction. + * + *

Threading: This method is called exclusively from the producer thread (crash parsing + * flow). It starts the collector thread on first invocation and enqueues work items. + * + *

The {@code processed} set is only accessed here (producer thread), so no synchronization is + * needed for it. + * + * @param path the path to the library file + */ + public void resolveBuildId(Path path) { + if (collecting.compareAndSet(false, true)) { + AgentTaskScheduler.get().execute(new Collector(5, SECONDS)); + } + final String filename = path.getFileName().toString(); + if (!processed.add(filename)) { + return; + } + if (libraryBuildInfo.remove(filename) == null) { + // the library is not present in the collected ones part of the stackframe + LOGGER.debug( + "Skipping build id resolution for {} as it was not added to unprocessed", filename); + + } else if (!workQueue.offer(path)) { + LOGGER.warn( + "Could not resolve the build id for library {} because the processing queue is full", + path); + } + } + + /** + * Signals that no more work will be enqueued and waits for collection to complete. + * + *

Called from producer thread to stop collection and wait for the collector to finish + * processing the queue. + * + * @param timeoutSeconds maximum time to wait for collection to complete + */ + public void awaitCollectionDone(final int timeoutSeconds) { + if (!collecting.compareAndSet(true, false)) { + return; + } + try { + if (!latch.await(timeoutSeconds, SECONDS)) { + LOGGER.warn("Build id collection incomplete."); + } + } catch (InterruptedException ie) { + Thread.currentThread().interrupt(); + LOGGER.warn("Interrupted while waiting for build id collection to finish"); + } + } + + /** + * Retrieves the build information for a library. + * + *

This method can be called from any thread after collection is complete. The {@link + * ConcurrentHashMap} ensures thread-safe reads. + * + * @param filename the library filename + * @return the build information, or null if not found + */ + public BuildInfo getBuildInfo(String filename) { + return libraryBuildInfo.get(filename); + } +} diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/buildid/BuildIdExtractor.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/buildid/BuildIdExtractor.java new file mode 100644 index 00000000000..859a62b10f9 --- /dev/null +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/buildid/BuildIdExtractor.java @@ -0,0 +1,42 @@ +package datadog.crashtracking.buildid; + +import datadog.environment.OperatingSystem; +import java.nio.file.Path; + +/** + * Interface for extracting build IDs from native library binaries. Build IDs help identify exact + * library versions for symbolization of native stack traces. + */ +public interface BuildIdExtractor { + /** + * Extracts build ID from a binary file. + * + * @param file Path to the library file + * @return Build ID as hex string, or null if not found or on error + */ + String extractBuildId(Path file); + + /** + * @return the file type this extractor operates for. + */ + BuildInfo.FileType fileType(); + + /** + * @return the build id type this extractor is able to provide. + */ + BuildInfo.BuildIdType buildIdType(); + + /** + * Factory method that returns appropriate extractor for the platform. + * + * @return Platform-specific build ID extractor + */ + static BuildIdExtractor create() { + if (OperatingSystem.isLinux()) { + return new ElfBuildIdExtractor(); + } else if (OperatingSystem.isWindows()) { + return new PeBuildIdExtractor(); + } + return new NoOpBuildIdExtractor(); + } +} diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/buildid/BuildInfo.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/buildid/BuildInfo.java new file mode 100644 index 00000000000..fe4ffd41477 --- /dev/null +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/buildid/BuildInfo.java @@ -0,0 +1,25 @@ +package datadog.crashtracking.buildid; + +public class BuildInfo { + public enum BuildIdType { + GNU, // for ELF + PDB // for DLL PE + } + + public enum FileType { + ELF, + PE, + } + + static final BuildInfo EMPTY = new BuildInfo(null, null, null); + + public final String buildId; + public final BuildIdType buildIdType; + public final FileType fileType; + + public BuildInfo(final String buildId, final BuildIdType buildIdType, final FileType fileType) { + this.buildId = buildId; + this.buildIdType = buildIdType; + this.fileType = fileType; + } +} diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/buildid/ElfBuildIdExtractor.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/buildid/ElfBuildIdExtractor.java new file mode 100644 index 00000000000..3f65a326dcb --- /dev/null +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/buildid/ElfBuildIdExtractor.java @@ -0,0 +1,213 @@ +package datadog.crashtracking.buildid; + +import java.io.IOException; +import java.io.RandomAccessFile; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.charset.StandardCharsets; +import java.nio.file.Path; +import java.util.Arrays; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Extracts build IDs from ELF (Executable and Linkable Format) binaries. Supports both 32-bit and + * 64-bit ELF files with little-endian and big-endian byte ordering. + */ +public class ElfBuildIdExtractor implements BuildIdExtractor { + private static final Logger log = LoggerFactory.getLogger(ElfBuildIdExtractor.class); + + // ELF magic: 0x7f 'E' 'L' 'F' + private static final byte[] ELF_MAGIC = {0x7f, 0x45, 0x4c, 0x46}; + + // ELF header constants + private static final int ELFCLASS32 = 1; + private static final int ELFCLASS64 = 2; + + private static final int ELFDATA2LSB = 1; // Little endian + private static final int ELFDATA2MSB = 2; // Big endian + + // Program header constants + private static final int PT_NOTE = 4; + + // Note header constants + private static final int NT_GNU_BUILD_ID = 3; + private static final byte[] GNU_NOTE_NAME = "GNU\0".getBytes(StandardCharsets.UTF_8); + + @Override + public String extractBuildId(Path file) { + try (RandomAccessFile raf = new RandomAccessFile(file.toFile(), "r")) { + // 1. Read and verify ELF magic (first 4 bytes) + byte[] magic = new byte[4]; + if (raf.read(magic) != 4 || !Arrays.equals(magic, ELF_MAGIC)) { + log.debug("Not an ELF file: {}", file); + return null; + } + + // 2. Determine file class (32 or 64 bit) + int elfClass = raf.read(); + boolean is64Bit = (elfClass == ELFCLASS64); + if (elfClass != ELFCLASS32 && elfClass != ELFCLASS64) { + log.debug("Invalid ELF class in file: {}", file); + return null; + } + + // 3. Determine endianness + int elfData = raf.read(); + boolean isLittleEndian = (elfData == ELFDATA2LSB); + if (elfData != ELFDATA2LSB && elfData != ELFDATA2MSB) { + log.debug("Invalid ELF data encoding in file: {}", file); + return null; + } + + // 4. Read ELF header to get program header table offset and size + raf.seek(is64Bit ? 32 : 28); // Offset to e_phoff + long phoff = is64Bit ? readLong(raf, isLittleEndian) : readInt(raf, isLittleEndian); + + raf.seek(is64Bit ? 54 : 42); // Offset to e_phnum + int phnum = readShort(raf, isLittleEndian); + + // 5. Iterate through program headers to find PT_NOTE segments + for (int i = 0; i < phnum; i++) { + long phEntryOffset = phoff + ((long) i * (is64Bit ? 56 : 32)); + raf.seek(phEntryOffset); + + int pType = (int) readInt(raf, isLittleEndian); + if (pType != PT_NOTE) { + continue; + } + + // Read PT_NOTE segment offset and size + long pOffset, pSize; + if (is64Bit) { + raf.seek(phEntryOffset + 8); + pOffset = readLong(raf, isLittleEndian); + raf.seek(phEntryOffset + 32); + pSize = readLong(raf, isLittleEndian); + } else { + raf.seek(phEntryOffset + 4); + pOffset = readInt(raf, isLittleEndian); + raf.seek(phEntryOffset + 16); + pSize = readInt(raf, isLittleEndian); + } + + // 6. Parse notes in this PT_NOTE segment + String buildId = parseNoteSegment(raf, pOffset, pSize, isLittleEndian); + if (buildId != null) { + return buildId; + } + } + + log.debug("No build ID found in ELF file: {}", file); + return null; + + } catch (IOException | SecurityException e) { + log.debug("Failed to extract ELF build ID from {}: {}", file, e.getMessage()); + return null; + } catch (Throwable t) { + log.debug("Unexpected error extracting ELF build ID from {}: {}", file, t.getMessage()); + return null; + } + } + + @Override + public BuildInfo.FileType fileType() { + return BuildInfo.FileType.ELF; + } + + @Override + public BuildInfo.BuildIdType buildIdType() { + return BuildInfo.BuildIdType.GNU; + } + + private String parseNoteSegment( + RandomAccessFile raf, long offset, long size, boolean isLittleEndian) throws IOException { + raf.seek(offset); + long end = offset + size; + + while (raf.getFilePointer() < end) { + long currentPos = raf.getFilePointer(); + + // Ensure we don't read beyond segment + if (currentPos + 12 > end) { + break; + } + + int namesz = (int) readInt(raf, isLittleEndian); + int descsz = (int) readInt(raf, isLittleEndian); + int type = (int) readInt(raf, isLittleEndian); + + // Align to 4-byte boundary + int nameLen = (namesz + 3) & ~3; + int descLen = (descsz + 3) & ~3; + + // Bounds check + if (currentPos + 12 + nameLen + descLen > end) { + break; + } + + // Read note name + byte[] name = new byte[namesz]; + if (raf.read(name) != namesz) { + throw new IOException("Failed to read note name"); + } + int skipped = raf.skipBytes(nameLen - namesz); + if (skipped != nameLen - namesz) { + throw new IOException("Failed to skip padding after note name"); + } + + // Check if this is the GNU build ID note + if (type == NT_GNU_BUILD_ID && Arrays.equals(name, GNU_NOTE_NAME)) { + // Read build ID + byte[] buildIdBytes = new byte[descsz]; + if (raf.read(buildIdBytes) != descsz) { + throw new IOException("Failed to read build ID"); + } + + // Convert to hex string + StringBuilder hex = new StringBuilder(descsz * 2); + for (byte b : buildIdBytes) { + hex.append(String.format("%02x", b & 0xff)); + } + return hex.toString(); + } else { + // Skip descriptor + skipped = raf.skipBytes(descLen); + if (skipped != descLen) { + throw new IOException("Failed to skip descriptor"); + } + } + } + return null; + } + + private int readShort(RandomAccessFile raf, boolean isLittleEndian) throws IOException { + byte[] bytes = new byte[2]; + if (raf.read(bytes) != 2) { + throw new IOException("Failed to read short"); + } + ByteBuffer buf = ByteBuffer.wrap(bytes); + buf.order(isLittleEndian ? ByteOrder.LITTLE_ENDIAN : ByteOrder.BIG_ENDIAN); + return buf.getShort() & 0xFFFF; + } + + private long readInt(RandomAccessFile raf, boolean isLittleEndian) throws IOException { + byte[] bytes = new byte[4]; + if (raf.read(bytes) != 4) { + throw new IOException("Failed to read int"); + } + ByteBuffer buf = ByteBuffer.wrap(bytes); + buf.order(isLittleEndian ? ByteOrder.LITTLE_ENDIAN : ByteOrder.BIG_ENDIAN); + return buf.getInt() & 0xFFFFFFFFL; + } + + private long readLong(RandomAccessFile raf, boolean isLittleEndian) throws IOException { + byte[] bytes = new byte[8]; + if (raf.read(bytes) != 8) { + throw new IOException("Failed to read long"); + } + ByteBuffer buf = ByteBuffer.wrap(bytes); + buf.order(isLittleEndian ? ByteOrder.LITTLE_ENDIAN : ByteOrder.BIG_ENDIAN); + return buf.getLong(); + } +} diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/buildid/NoOpBuildIdExtractor.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/buildid/NoOpBuildIdExtractor.java new file mode 100644 index 00000000000..71bf8e8c31f --- /dev/null +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/buildid/NoOpBuildIdExtractor.java @@ -0,0 +1,24 @@ +package datadog.crashtracking.buildid; + +import java.nio.file.Path; + +/** + * No-op build ID extractor for unsupported platforms. Always returns null to indicate build IDs are + * not available. + */ +public class NoOpBuildIdExtractor implements BuildIdExtractor { + @Override + public String extractBuildId(Path file) { + return null; // No build ID on this platform + } + + @Override + public BuildInfo.FileType fileType() { + return null; + } + + @Override + public BuildInfo.BuildIdType buildIdType() { + return null; + } +} diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/buildid/PeBuildIdExtractor.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/buildid/PeBuildIdExtractor.java new file mode 100644 index 00000000000..19a7afed4d5 --- /dev/null +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/buildid/PeBuildIdExtractor.java @@ -0,0 +1,314 @@ +package datadog.crashtracking.buildid; + +import java.io.IOException; +import java.io.RandomAccessFile; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.file.Path; +import java.util.Arrays; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Extracts build IDs from PE (Portable Executable) binaries used on Windows. Uses the GUID and Age + * from the PDB70 CodeView debug information. + */ +public class PeBuildIdExtractor implements BuildIdExtractor { + private static final Logger log = LoggerFactory.getLogger(PeBuildIdExtractor.class); + + // DOS header magic: 'M' 'Z' + private static final byte[] MZ_MAGIC = {0x4d, 0x5a}; + + // PE signature: 'P' 'E' 0x00 0x00 + private static final byte[] PE_SIGNATURE = {0x50, 0x45, 0x00, 0x00}; + + // PDB70 CodeView signature: 'R' 'S' 'D' 'S' + private static final int PDB70_SIGNATURE = 0x53445352; + + // Debug directory type for CodeView + private static final int IMAGE_DEBUG_TYPE_CODEVIEW = 2; + + // PE32 magic + private static final short PE32_MAGIC = 0x10b; + + // PE32+ (64-bit) magic + private static final short PE32_PLUS_MAGIC = 0x20b; + + @Override + public String extractBuildId(Path file) { + try (RandomAccessFile raf = new RandomAccessFile(file.toFile(), "r")) { + long size = raf.length(); + + // 1. Verify DOS header magic (first 2 bytes) + byte[] magic = new byte[2]; + if (raf.read(magic) != 2 || !Arrays.equals(magic, MZ_MAGIC)) { + log.debug("Not a PE file (missing MZ magic): {}", file); + return null; + } + + // 2. Read PE header offset from DOS header (at offset 0x3C) + raf.seek(0x3C); + byte[] offsetBytes = new byte[4]; + if (raf.read(offsetBytes) != 4) { + log.debug("Failed to read PE header offset from: {}", file); + return null; + } + ByteBuffer buf = ByteBuffer.wrap(offsetBytes); + buf.order(ByteOrder.LITTLE_ENDIAN); // PE is always little-endian + long peOffset = buf.getInt() & 0xFFFFFFFFL; + + // Bounds check + if (peOffset > size - 4) { + log.debug("Invalid PE header offset in file: {}", file); + return null; + } + + // 3. Verify PE signature + raf.seek(peOffset); + byte[] peSig = new byte[4]; + if (raf.read(peSig) != 4 || !Arrays.equals(peSig, PE_SIGNATURE)) { + log.debug("Invalid PE signature in file: {}", file); + return null; + } + + // 4. Read COFF header + // COFF header starts right after PE signature + // Layout: + // +0: Machine (2 bytes) + // +2: NumberOfSections (2 bytes) + // +4: TimeDateStamp (4 bytes) + // +8: PointerToSymbolTable (4 bytes) + // +12: NumberOfSymbols (4 bytes) + // +16: SizeOfOptionalHeader (2 bytes) + // +18: Characteristics (2 bytes) + long coffHeaderOffset = peOffset + 4; + + if (coffHeaderOffset + 20 > size) { + log.debug("Invalid COFF header position in file: {}", file); + return null; + } + + raf.seek(coffHeaderOffset + 16); + byte[] sizeBytes = new byte[2]; + if (raf.read(sizeBytes) != 2) { + log.debug("Failed to read SizeOfOptionalHeader from: {}", file); + return null; + } + buf = ByteBuffer.wrap(sizeBytes); + buf.order(ByteOrder.LITTLE_ENDIAN); + int sizeOfOptionalHeader = buf.getShort() & 0xFFFF; + + if (sizeOfOptionalHeader == 0) { + log.debug("No optional header in PE file: {}", file); + return null; + } + + // 5. Read Optional Header to get Debug Directory RVA and size + // Optional header starts right after COFF header (20 bytes after PE signature) + long optionalHeaderOffset = coffHeaderOffset + 20; + + if (optionalHeaderOffset + 2 > size) { + log.debug("Invalid optional header position in file: {}", file); + return null; + } + + raf.seek(optionalHeaderOffset); + byte[] magicBytes = new byte[2]; + if (raf.read(magicBytes) != 2) { + log.debug("Failed to read optional header magic from: {}", file); + return null; + } + buf = ByteBuffer.wrap(magicBytes); + buf.order(ByteOrder.LITTLE_ENDIAN); + short optionalMagic = buf.getShort(); + + boolean is64Bit = (optionalMagic == PE32_PLUS_MAGIC); + + // Debug directory is in the data directories array + // For PE32: offset 96 from optional header start + // For PE32+: offset 112 from optional header start + long debugDirOffset = optionalHeaderOffset + (is64Bit ? 112 : 96) + (6 * 8); // 6th entry + // Each data directory entry is 8 bytes: RVA (4) + Size (4) + + if (debugDirOffset + 8 > size) { + log.debug("Invalid debug directory offset in file: {}", file); + return null; + } + + raf.seek(debugDirOffset); + byte[] debugDirData = new byte[8]; + if (raf.read(debugDirData) != 8) { + log.debug("Failed to read debug directory data from: {}", file); + return null; + } + buf = ByteBuffer.wrap(debugDirData); + buf.order(ByteOrder.LITTLE_ENDIAN); + long debugDirRva = buf.getInt() & 0xFFFFFFFFL; + long debugDirSize = buf.getInt() & 0xFFFFFFFFL; + + if (debugDirRva == 0 || debugDirSize == 0) { + log.debug("No debug directory in PE file: {}", file); + return null; + } + + // 6. Convert RVA to file offset by reading section headers + long sectionHeadersOffset = optionalHeaderOffset + sizeOfOptionalHeader; + long debugDirFileOffset = + rvaToFileOffset(raf, sectionHeadersOffset, coffHeaderOffset, debugDirRva, size); + + if (debugDirFileOffset == -1) { + log.debug("Failed to convert debug directory RVA to file offset: {}", file); + return null; + } + + // 7. Parse debug directory entries to find CodeView entry + int numEntries = (int) (debugDirSize / 28); // Each debug directory entry is 28 bytes + for (int i = 0; i < numEntries; i++) { + long entryOffset = debugDirFileOffset + (i * 28); + if (entryOffset + 28 > size) { + continue; + } + + raf.seek(entryOffset); + byte[] entryData = new byte[28]; + if (raf.read(entryData) != 28) { + continue; + } + + buf = ByteBuffer.wrap(entryData); + buf.order(ByteOrder.LITTLE_ENDIAN); + + buf.getInt(); // Characteristics + buf.getInt(); // TimeDateStamp + buf.getShort(); // MajorVersion + buf.getShort(); // MinorVersion + int type = buf.getInt(); + int dataSize = buf.getInt(); + long dataRva = buf.getInt() & 0xFFFFFFFFL; + long dataFilePointer = buf.getInt() & 0xFFFFFFFFL; + + if (type == IMAGE_DEBUG_TYPE_CODEVIEW && dataSize > 0) { + // Found CodeView entry, read PDB70 structure + if (dataFilePointer + dataSize > size) { + log.debug("Invalid CodeView data pointer in file: {}", file); + continue; + } + + raf.seek(dataFilePointer); + byte[] cvData = new byte[Math.min(dataSize, 24)]; // PDB70 header is 24 bytes + if (raf.read(cvData) != cvData.length) { + continue; + } + + buf = ByteBuffer.wrap(cvData); + buf.order(ByteOrder.LITTLE_ENDIAN); + + int signature = buf.getInt(); + if (signature != PDB70_SIGNATURE) { + continue; + } + + // Read GUID (16 bytes) + long guidData1 = buf.getInt() & 0xFFFFFFFFL; + int guidData2 = buf.getShort() & 0xFFFF; + int guidData3 = buf.getShort() & 0xFFFF; + byte[] guidData4 = new byte[8]; + buf.get(guidData4); + + // Read Age (4 bytes) + long age = buf.getInt() & 0xFFFFFFFFL; + + // Format as GUID + Age in hex (like dotnet tracer) + return formatBuildId(guidData1, guidData2, guidData3, guidData4, age); + } + } + + log.debug("No CodeView debug information found in PE file: {}", file); + return null; + + } catch (IOException | SecurityException e) { + log.debug("Failed to extract PE build ID from {}: {}", file, e.getMessage()); + return null; + } catch (Throwable t) { + log.debug("Unexpected error extracting PE build ID from {}: {}", file, t.getMessage()); + return null; + } + } + + private long rvaToFileOffset( + RandomAccessFile raf, + long sectionHeadersOffset, + long coffHeaderOffset, + long rva, + long fileSize) + throws IOException { + // Read number of sections from COFF header + raf.seek(coffHeaderOffset + 2); + byte[] numSectionsBytes = new byte[2]; + if (raf.read(numSectionsBytes) != 2) { + return -1; + } + ByteBuffer buf = ByteBuffer.wrap(numSectionsBytes); + buf.order(ByteOrder.LITTLE_ENDIAN); + int numSections = buf.getShort() & 0xFFFF; + + // Each section header is 40 bytes + for (int i = 0; i < numSections; i++) { + long sectionOffset = sectionHeadersOffset + (i * 40); + if (sectionOffset + 40 > fileSize) { + continue; + } + + raf.seek(sectionOffset + 8); // Skip name (8 bytes) + byte[] sectionData = new byte[16]; + if (raf.read(sectionData) != 16) { + continue; + } + + buf = ByteBuffer.wrap(sectionData); + buf.order(ByteOrder.LITTLE_ENDIAN); + + long virtualSize = buf.getInt() & 0xFFFFFFFFL; + long virtualAddress = buf.getInt() & 0xFFFFFFFFL; + @SuppressWarnings("unused") + long sizeOfRawData = buf.getInt() & 0xFFFFFFFFL; + long pointerToRawData = buf.getInt() & 0xFFFFFFFFL; + + // Check if RVA falls within this section + if (rva >= virtualAddress && rva < virtualAddress + virtualSize) { + return pointerToRawData + (rva - virtualAddress); + } + } + + return -1; + } + + private String formatBuildId( + long guidData1, int guidData2, int guidData3, byte[] guidData4, long age) { + // Format: GUID (uppercase, without dashes) + Age (lowercase hex) + return String.format( + "%08X%04X%04X%02X%02X%02X%02X%02X%02X%02X%02X%x", + guidData1, + guidData2, + guidData3, + guidData4[0] & 0xFF, + guidData4[1] & 0xFF, + guidData4[2] & 0xFF, + guidData4[3] & 0xFF, + guidData4[4] & 0xFF, + guidData4[5] & 0xFF, + guidData4[6] & 0xFF, + guidData4[7] & 0xFF, + age); + } + + @Override + public BuildInfo.FileType fileType() { + return BuildInfo.FileType.PE; + } + + @Override + public BuildInfo.BuildIdType buildIdType() { + return BuildInfo.BuildIdType.PDB; + } +} diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/CrashLog.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/CrashLog.java index 0dbefd4ad8b..b0508d9ab45 100644 --- a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/CrashLog.java +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/CrashLog.java @@ -10,10 +10,10 @@ public final class CrashLog { private static final int VERSION = 0; - private static final JsonAdapter ADAPTER; + public static final JsonAdapter ADAPTER; static { - Moshi moshi = new Moshi.Builder().add(new SemanticVersion.SemanticVersionAdapter()).build(); + Moshi moshi = new Moshi.Builder().add(new DynamicLibs.JsonAdapter()).build(); ADAPTER = moshi.adapter(CrashLog.class); } @@ -36,6 +36,17 @@ public final class CrashLog { @Json(name = "version_id") public final int version = VERSION; + @Json(name = "sig_info") + public final SigInfo sigInfo; + + public final Experimental experimental; + + /** + * Useful files for triage and debugging (e.g. {@code /proc/self/maps}, {@code + * dynamic_libraries}). + */ + public final DynamicLibs files; + public CrashLog( String uuid, boolean incomplete, @@ -44,7 +55,34 @@ public CrashLog( Metadata metadata, OSInfo osInfo, ProcInfo procInfo, + SigInfo sigInfo, String dataSchemaVersion) { + this( + uuid, + incomplete, + timestamp, + error, + metadata, + osInfo, + procInfo, + sigInfo, + dataSchemaVersion, + null, + null); + } + + public CrashLog( + String uuid, + boolean incomplete, + String timestamp, + ErrorData error, + Metadata metadata, + OSInfo osInfo, + ProcInfo procInfo, + SigInfo sigInfo, + String dataSchemaVersion, + Experimental experimental, + DynamicLibs files) { this.uuid = uuid != null ? uuid : RandomUtils.randomUUID().toString(); this.incomplete = incomplete; this.timestamp = timestamp; @@ -52,7 +90,10 @@ public CrashLog( this.metadata = metadata; this.osInfo = osInfo; this.procInfo = procInfo; + this.sigInfo = sigInfo; this.dataSchemaVersion = dataSchemaVersion; + this.experimental = experimental; + this.files = files; } public String toJson() { @@ -79,13 +120,27 @@ public boolean equals(Object o) { && Objects.equals(metadata, crashLog.metadata) && Objects.equals(osInfo, crashLog.osInfo) && Objects.equals(procInfo, crashLog.procInfo) - && Objects.equals(dataSchemaVersion, crashLog.dataSchemaVersion); + && Objects.equals(sigInfo, crashLog.sigInfo) + && Objects.equals(dataSchemaVersion, crashLog.dataSchemaVersion) + && Objects.equals(experimental, crashLog.experimental) + && Objects.equals(files, crashLog.files); } @Override public int hashCode() { return Objects.hash( - uuid, timestamp, incomplete, error, metadata, osInfo, procInfo, version, dataSchemaVersion); + uuid, + timestamp, + incomplete, + error, + metadata, + osInfo, + procInfo, + sigInfo, + version, + dataSchemaVersion, + experimental, + files); } public boolean equalsForTest(Object o) { @@ -103,6 +158,9 @@ public boolean equalsForTest(Object o) { && Objects.equals(timestamp, crashLog.timestamp) && Objects.equals(error, crashLog.error) && Objects.equals(procInfo, crashLog.procInfo) - && Objects.equals(dataSchemaVersion, crashLog.dataSchemaVersion); + && Objects.equals(sigInfo, crashLog.sigInfo) + && Objects.equals(dataSchemaVersion, crashLog.dataSchemaVersion) + && Objects.equals(experimental, crashLog.experimental) + && Objects.equals(files, crashLog.files); } } diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/DynamicLibs.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/DynamicLibs.java new file mode 100644 index 00000000000..05807513f8d --- /dev/null +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/DynamicLibs.java @@ -0,0 +1,30 @@ +package datadog.crashtracking.dto; + +import com.squareup.moshi.FromJson; +import com.squareup.moshi.ToJson; +import java.util.Collections; +import java.util.List; +import java.util.Map; + +public class DynamicLibs { + public final String name; + public final List lines; + + public DynamicLibs(String name, List lines) { + this.name = name; + this.lines = lines; + } + + public static class JsonAdapter { + @ToJson + Map> toJson(DynamicLibs dynamicLibs) { + return Collections.singletonMap(dynamicLibs.name, dynamicLibs.lines); + } + + @FromJson + DynamicLibs fromJson(Map> map) { + Map.Entry> entry = map.entrySet().iterator().next(); + return new DynamicLibs(entry.getKey(), entry.getValue()); + } + } +} diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/ErrorData.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/ErrorData.java index 9ccb9a96e0d..55984a47fee 100644 --- a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/ErrorData.java +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/ErrorData.java @@ -10,14 +10,22 @@ public final class ErrorData { public final String kind; public final String message; + @Json(name = "thread_name") + public final String threadName; + @Json(name = "source_type") - public final String sourceType = "crashtracking"; + public final String sourceType = "Crashtracking"; public final StackTrace stack; public ErrorData(String kind, String message, StackTrace stack) { + this(kind, message, null, stack); + } + + public ErrorData(String kind, String message, String threadName, StackTrace stack) { this.kind = kind; this.message = message; + this.threadName = threadName; this.stack = stack; } @@ -33,12 +41,13 @@ public boolean equals(Object o) { return isCrash == errorData.isCrash && Objects.equals(kind, errorData.kind) && Objects.equals(message, errorData.message) + && Objects.equals(threadName, errorData.threadName) && Objects.equals(sourceType, errorData.sourceType) && Objects.equals(stack, errorData.stack); } @Override public int hashCode() { - return Objects.hash(isCrash, kind, message, sourceType, stack); + return Objects.hash(isCrash, kind, message, threadName, sourceType, stack); } } diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/Experimental.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/Experimental.java new file mode 100644 index 00000000000..914c6640dd4 --- /dev/null +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/Experimental.java @@ -0,0 +1,34 @@ +package datadog.crashtracking.dto; + +import com.squareup.moshi.Json; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +public final class Experimental { + public final Map ucontext; + + @Json(name = "runtime_args") + public final List runtimeArgs; + + public Experimental(Map ucontext) { + this(ucontext, null); + } + + public Experimental(Map ucontext, List runtimeArgs) { + this.ucontext = ucontext; + this.runtimeArgs = runtimeArgs; + } + + @Override + public boolean equals(Object o) { + if (!(o instanceof Experimental)) return false; + Experimental that = (Experimental) o; + return Objects.equals(ucontext, that.ucontext) && Objects.equals(runtimeArgs, that.runtimeArgs); + } + + @Override + public int hashCode() { + return Objects.hash(ucontext, runtimeArgs); + } +} diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/OSInfo.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/OSInfo.java index 793ec27aa4f..9a666270dc2 100644 --- a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/OSInfo.java +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/OSInfo.java @@ -1,6 +1,7 @@ package datadog.crashtracking.dto; import com.squareup.moshi.Json; +import datadog.environment.SystemProperties; import java.util.Objects; public final class OSInfo { @@ -10,9 +11,9 @@ public final class OSInfo { @Json(name = "os_type") public final String osType; - public final SemanticVersion version; + public final String version; - public OSInfo(String architecture, String bitness, String osType, SemanticVersion version) { + public OSInfo(String architecture, String bitness, String osType, String version) { this.architecture = architecture; this.bitness = bitness; this.osType = osType; @@ -38,4 +39,15 @@ public boolean equals(Object o) { public int hashCode() { return Objects.hash(architecture, bitness, osType, version); } + + public static OSInfo current() { + String rawBitness = SystemProperties.get("sun.arch.data.model"); + String bitness = rawBitness != null ? rawBitness + "-bit" : null; + String osName = SystemProperties.get("os.name"); + if (osName != null && osName.startsWith("Mac OS")) { + osName = "Mac OS"; + } + return new OSInfo( + SystemProperties.get("os.arch"), bitness, osName, SystemProperties.get("os.version")); + } } diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/ProcInfo.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/ProcInfo.java index 36dd8f5b3b3..22b5b822f17 100644 --- a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/ProcInfo.java +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/ProcInfo.java @@ -1,11 +1,9 @@ package datadog.crashtracking.dto; -import java.util.Objects; - public final class ProcInfo { - public final String pid; + public final int pid; - public ProcInfo(String pid) { + public ProcInfo(int pid) { this.pid = pid; } @@ -18,11 +16,11 @@ public boolean equals(Object o) { return false; } ProcInfo procInfo = (ProcInfo) o; - return Objects.equals(pid, procInfo.pid); + return pid == procInfo.pid; } @Override public int hashCode() { - return Objects.hashCode(pid); + return pid; } } diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/SemanticVersion.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/SemanticVersion.java deleted file mode 100644 index 2ac34dc2ecc..00000000000 --- a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/SemanticVersion.java +++ /dev/null @@ -1,101 +0,0 @@ -package datadog.crashtracking.dto; - -import com.squareup.moshi.FromJson; -import com.squareup.moshi.JsonReader; -import com.squareup.moshi.JsonWriter; -import com.squareup.moshi.ToJson; -import java.io.IOException; -import java.util.Objects; -import java.util.regex.Pattern; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -public final class SemanticVersion { - private static final Pattern NUMERIC_SPLITTER = Pattern.compile("[^0-9]+"); - private static final Logger LOGGER = LoggerFactory.getLogger(SemanticVersion.class.getName()); - - public static final class SemanticVersionAdapter { - - @ToJson - public void toJson(JsonWriter writer, SemanticVersion version) throws IOException { - writer.beginObject(); - writer.name("Semantic"); - writer.beginArray(); - writer.value(version.major); - writer.value(version.minor); - writer.value(version.patch); - writer.endArray(); - writer.endObject(); - } - - @FromJson - public SemanticVersion fromJson(JsonReader reader) throws IOException { - reader.beginObject(); - String name = reader.nextName(); - if (!"Semantic".equals(name)) { - throw new IOException("Expected 'Semantic' key"); - } - reader.beginArray(); - int major = reader.nextInt(); - int minor = reader.nextInt(); - int patch = reader.nextInt(); - reader.endArray(); - reader.endObject(); - return new SemanticVersion(major, minor, patch); - } - } - - public final int major; - public final int minor; - public final int patch; - - public SemanticVersion(int major, int minor, int patch) { - this.major = major; - this.minor = minor; - this.patch = patch; - } - - public static SemanticVersion of(String version) { - String[] parts = NUMERIC_SPLITTER.split(version, 4); - if (parts.length == 0) { - LOGGER.error("Invalid version string {} ", version); - return new SemanticVersion(0, 0, 0); // have a sane default - } else if (parts.length == 2) { - return new SemanticVersion(safeParseInteger(parts[0]), safeParseInteger(parts[1]), 0); - } else if (parts.length == 1) { - return new SemanticVersion(safeParseInteger(parts[0]), 0, 0); - } - return new SemanticVersion( - safeParseInteger(parts[0]), safeParseInteger(parts[1]), safeParseInteger(parts[2])); - } - - private static int safeParseInteger(String value) { - try { - return Integer.parseInt(value); - } catch (Throwable ignored) { - return 0; - } - } - - @Override - public boolean equals(Object o) { - if (this == o) { - return true; - } - if (o == null || getClass() != o.getClass()) { - return false; - } - SemanticVersion that = (SemanticVersion) o; - return major == that.major && minor == that.minor && patch == that.patch; - } - - @Override - public int hashCode() { - return Objects.hash(major, minor, patch); - } - - @Override - public String toString() { - return "SemanticVersion{" + "major=" + major + ", minor=" + minor + ", patch=" + patch + '}'; - } -} diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/SigInfo.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/SigInfo.java new file mode 100644 index 00000000000..dcb0ebb2265 --- /dev/null +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/SigInfo.java @@ -0,0 +1,62 @@ +package datadog.crashtracking.dto; + +import com.squareup.moshi.Json; +import java.util.Objects; + +public class SigInfo { + @Json(name = "si_signo") + public final Integer number; + + @Json(name = "si_code") + public final Integer code; + + @Json(name = "si_signo_human_readable") + public final String name; + + @Json(name = "si_code_human_readable") + public final String action; + + @Json(name = "si_addr") + public final String address; + + @Json(name = "si_pid") + public final Integer pid; + + @Json(name = "si_uid") + public final Integer uid; + + public SigInfo( + Integer number, + String name, + Integer code, + String action, + String address, + Integer pid, + Integer uid) { + this.number = number; + this.name = name; + this.address = address; + this.code = code; + this.action = action; + this.pid = pid; + this.uid = uid; + } + + @Override + public boolean equals(Object o) { + if (!(o instanceof SigInfo)) return false; + SigInfo sigInfo = (SigInfo) o; + return Objects.equals(number, sigInfo.number) + && Objects.equals(name, sigInfo.name) + && Objects.equals(address, sigInfo.address) + && Objects.equals(code, sigInfo.code) + && Objects.equals(action, sigInfo.action) + && Objects.equals(pid, sigInfo.pid) + && Objects.equals(uid, sigInfo.uid); + } + + @Override + public int hashCode() { + return Objects.hash(number, name, address, code, action, pid, uid); + } +} diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/StackFrame.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/StackFrame.java index e4232936935..a8bcdf69889 100644 --- a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/StackFrame.java +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/StackFrame.java @@ -1,16 +1,57 @@ package datadog.crashtracking.dto; +import com.squareup.moshi.Json; +import datadog.crashtracking.buildid.BuildInfo; import java.util.Objects; public final class StackFrame { - public final String file; + + public final String path; public final Integer line; public final String function; - public StackFrame(String file, Integer line, String function) { - this.file = file; + @Json(name = "type") + public final String frameType; + + @Json(name = "build_id") + public final String buildId; + + @Json(name = "build_id_type") + public final BuildInfo.BuildIdType buildIdType; + + @Json(name = "file_type") + public final BuildInfo.FileType fileType; + + @Json(name = "ip") + public final String ip; + + @Json(name = "symbol_address") + public final String symbolAddress; + + @Json(name = "relative_address") + public String relativeAddress; + + public StackFrame( + String path, + Integer line, + String function, + String frameType, + String buildId, + BuildInfo.BuildIdType buildIdType, + BuildInfo.FileType fileType, + String ip, + String symbolAddress, + String relativeAddress) { + this.path = path; this.line = line; this.function = function; + this.frameType = frameType; + this.buildId = buildId; + this.buildIdType = buildIdType; + this.fileType = fileType; + this.ip = ip; + this.symbolAddress = symbolAddress; + this.relativeAddress = relativeAddress; } @Override @@ -22,13 +63,30 @@ public boolean equals(Object o) { return false; } StackFrame that = (StackFrame) o; - return Objects.equals(file, that.file) + return Objects.equals(path, that.path) && Objects.equals(line, that.line) - && Objects.equals(function, that.function); + && Objects.equals(function, that.function) + && Objects.equals(frameType, that.frameType) + && Objects.equals(buildId, that.buildId) + && buildIdType == that.buildIdType + && fileType == that.fileType + && Objects.equals(ip, that.ip) + && Objects.equals(symbolAddress, that.symbolAddress) + && Objects.equals(relativeAddress, that.relativeAddress); } @Override public int hashCode() { - return Objects.hash(file, line, function); + return Objects.hash( + path, + line, + function, + frameType, + buildId, + buildIdType, + fileType, + ip, + symbolAddress, + relativeAddress); } } diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/StackTrace.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/StackTrace.java index 23a12ca2c69..9467e01fcba 100644 --- a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/StackTrace.java +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/dto/StackTrace.java @@ -1,9 +1,15 @@ package datadog.crashtracking.dto; +import com.squareup.moshi.JsonAdapter; +import com.squareup.moshi.JsonWriter; +import com.squareup.moshi.Moshi; +import java.io.IOException; import java.util.Arrays; import java.util.Objects; public final class StackTrace { + private static final JsonAdapter ADAPTER = + new Moshi.Builder().build().adapter(StackTrace.class); private static final String FORMAT = "CrashTrackerV1"; public final String format = FORMAT; @@ -29,4 +35,8 @@ public boolean equals(Object o) { public int hashCode() { return Objects.hash(format, Arrays.hashCode(frames)); } + + public void writeAsJson(final JsonWriter writer) throws IOException { + ADAPTER.toJson(writer, this); + } } diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/parsers/HotspotCrashLogParser.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/parsers/HotspotCrashLogParser.java index a1e21144608..ff89656ab26 100644 --- a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/parsers/HotspotCrashLogParser.java +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/parsers/HotspotCrashLogParser.java @@ -3,29 +3,57 @@ import static java.time.format.DateTimeFormatter.ISO_OFFSET_DATE_TIME; import datadog.common.version.VersionInfo; +import datadog.crashtracking.buildid.BuildIdCollector; +import datadog.crashtracking.buildid.BuildInfo; import datadog.crashtracking.dto.CrashLog; +import datadog.crashtracking.dto.DynamicLibs; import datadog.crashtracking.dto.ErrorData; +import datadog.crashtracking.dto.Experimental; import datadog.crashtracking.dto.Metadata; import datadog.crashtracking.dto.OSInfo; import datadog.crashtracking.dto.ProcInfo; -import datadog.crashtracking.dto.SemanticVersion; +import datadog.crashtracking.dto.SigInfo; import datadog.crashtracking.dto.StackFrame; import datadog.crashtracking.dto.StackTrace; -import datadog.environment.SystemProperties; +import java.nio.file.InvalidPathException; +import java.nio.file.Path; +import java.nio.file.Paths; import java.time.OffsetDateTime; import java.time.ZonedDateTime; import java.time.format.DateTimeFormatter; import java.time.format.DateTimeParseException; import java.util.ArrayList; +import java.util.LinkedHashMap; import java.util.List; import java.util.Locale; +import java.util.Map; +import java.util.regex.Matcher; import java.util.regex.Pattern; +/** + * Parser for HotSpot JVM fatal error logs ({@code hs_err_pidNNN.log}). + * + *

The log is parsed using a linear state machine that mirrors the deterministic section order + * emitted by {@code VMError::report()} in HotSpot. The section order is fixed for a given platform + * but differs across OS/CPU combinations. + * + *

If an early sentinel line is absent (e.g. {@code "Native frames:"} is missing because the JVM + * crashed before producing a stack), the state machine will not advance past {@code THREAD} state + * and subsequent sections such as {@code siginfo} and registers will be silently skipped. The + * resulting {@link datadog.crashtracking.dto.CrashLog} will be marked {@code incomplete}. + */ public final class HotspotCrashLogParser { + private static final String HOTSPOT_JVM_ARGS_PREFIX = "jvm_args:"; private static final DateTimeFormatter ZONED_DATE_TIME_FORMATTER = DateTimeFormatter.ofPattern("EEE MMM ppd HH:mm:ss yyyy zzz", Locale.getDefault()); private static final DateTimeFormatter OFFSET_DATE_TIME_FORMATTER = DateTimeFormatter.ofPattern("EEE MMM ppd HH:mm:ss yyyy X", Locale.getDefault()); + private static final String OOM_MARKER = "OutOfMemory encountered: "; + + // all lowercased + private static final String[] KNOWN_LIBRARY_NAMES = {"libjavaprofiler", "libddwaf", "libsqreen"}; + + private final BuildIdCollector buildIdCollector; enum State { NEW, @@ -34,38 +62,146 @@ enum State { SUMMARY, THREAD, STACKTRACE, + REGISTERS, + PROCESS, + VM_ARGUMENTS, + DYNAMIC_LIBRARIES, + SYSTEM, DONE } private State state = State.NEW; + public HotspotCrashLogParser() { + this.buildIdCollector = new BuildIdCollector(); + } + private static final Pattern PLUS_SPLITTER = Pattern.compile("\\+"); private static final Pattern SPACE_SPLITTER = Pattern.compile("\\s+"); private static final Pattern NEWLINE_SPLITTER = Pattern.compile("\n"); + // Groups: 1=si_signo, 2=signal name, 3=si_code, 4=si_code name, + // 5=si_addr (null for SI_USER), 6=si_pid (null for si_addr), 7=si_uid (null for si_addr) + private static final Pattern SIGINFO_PARSER = + Pattern.compile( + "siginfo:\\s+si_signo:\\s+(\\d+)\\s+\\((\\w+)\\),\\s+si_code:\\s+(\\d+)\\s+\\(([^)]+)\\),\\s+" + + "(?:si_addr:\\s+(0x[0-9a-fA-F]+)|si_pid:\\s+(\\d+),\\s+si_uid:\\s+(\\d+))"); + private static final Pattern DYNAMIC_LIBS_PATH_PARSER = + Pattern.compile("^(?:0x)?[0-9a-fA-F]+(?:-[0-9a-fA-F]+)?\\s+(?:[^\\s/\\[]+\\s+)*(.*)$"); + // Matches register entries like: + // * RAX=0x..., R8 =0x..., TRAPNO=0x... (x86-64) + // * R0=0x..., R30=0x... (Linux aarch64) + // * x0=0x..., fp=0x..., lr=0x..., sp=0x..., pc=0x... (macOS aarch64) + // Note that register formatting varies by platform, the JVM crash handler can emit one or four + // per line. + private static final Pattern REGISTER_ENTRY_PARSER = + Pattern.compile("([A-Za-z][A-Za-z0-9]*)\\s*=\\s*(0x[0-9a-fA-F]+)"); + // Used for the REGISTERS-state exit condition only: the register name must start the line + // (after optional whitespace). This prevents lines like "Top of Stack: (sp=0x...)" and + // "Instructions: (pc=0x...)" from being mistaken for register entries by REGISTER_ENTRY_PARSER's + // find(), which would otherwise match the lowercase "sp"/"pc" tokens embedded in those lines. + private static final Pattern REGISTER_LINE_START = + Pattern.compile("^\\s*[A-Za-z][A-Za-z0-9]*\\s*=\\s*0x"); + private static final Pattern COMPILED_JAVA_ADDRESS_PARSER = + Pattern.compile("@\\s+(0x[0-9a-fA-F]+)\\s+\\[(0x[0-9a-fA-F]+)\\+(0x[0-9a-fA-F]+)\\]"); + + // HotSpot crash logs encode the execution kind in the first column of each frame line. + // Source references: + // JDK 8: + // https://github.com/openjdk/jdk8u/blob/73c9c6bcd062196cbebc4d9f22b13d2e20a14f98/hotspot/src/share/vm/runtime/frame.cpp#L710-L724 + // JDK 11: + // https://github.com/openjdk/jdk11u/blob/970d6cf491a55fd6ab98ec3f449c13a58633078a/src/hotspot/share/runtime/frame.cpp#L647-L662 + // JDK 25: + // https://github.com/openjdk/jdk25u/blob/2fe611a2a3386d097f636c15bd4d396a82dc695e/src/hotspot/share/runtime/frame.cpp#L652-L666 + // Mainline: + // https://github.com/openjdk/jdk/blob/53c864a881d2183d3664a6a5a56480bd99fffe45/src/hotspot/share/runtime/frame.cpp#L647-L661 + // Note: the marker set changes across JDK lines. In particular, "A" appears in some HotSpot + // versions but not all, so this mapping is best-effort rather than a stable cross-version enum. + private static String hotspotFrameType(char marker) { + switch (marker) { + case 'J': + return "compiled"; + case 'A': // exists in JDK 11 + return "aot_compiled"; + case 'j': + return "interpreted"; + case 'V': + return "vm"; + case 'v': + return "stub"; + case 'C': + return "native"; + default: + return null; + } + } private StackFrame parseLine(String line) { + if (line == null || line.isEmpty()) { + return null; + } + String functionName = null; Integer functionLine = null; String filename = null; + String ip = null; + String relAddress = null; + String symbolAddress = null; char firstChar = line.charAt(0); + String frameType = hotspotFrameType(firstChar); + if (line.length() > 1 && !Character.isSpaceChar(line.charAt(1))) { + // We can find entries like this in between the frames + // Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) + return null; + } switch (firstChar) { case 'J': + case 'A': { - // J 36572 c2 datadog.trace.util.AgentTaskScheduler$PeriodicTask.run()V (25 bytes) @ - // 0x00007f2fd0198488 [0x00007f2fd0198420+0x0000000000000068] + // spotless:off + // J 36572 c2 datadog.trace.util.AgentTaskScheduler$PeriodicTask.run()V (25 bytes) @ 0x00007f2fd0198488 [0x00007f2fd0198420+0x0000000000000068] + // J 3896 c2 java.nio.ByteBuffer.allocate(I)Ljava/nio/ByteBuffer; java.base@21.0.1 (20 bytes) @ 0x0000000112ad51e8 [0x0000000112ad4fc0+0x0000000000000228] + // J 302 java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00007fa287303dce [0x00007fa287303d00+0xce] + // spotless:on String[] parts = SPACE_SPLITTER.split(line); - functionName = parts[3]; + int bytesToken = -1; + for (int i = 0; i < parts.length - 1; i++) { + if (parts[i].startsWith("(") && "bytes)".equals(parts[i + 1])) { + bytesToken = i; + break; + } + } + if (bytesToken > 1) { + String candidate = parts[bytesToken - 1]; + // Newer JVMs insert a module token before "(NN bytes)". + if (candidate.contains("@")) { + candidate = parts[bytesToken - 2]; + } + if (!candidate.startsWith("(")) { + functionName = candidate; + } + } else if (parts.length > 3 && !parts[3].startsWith("(")) { + functionName = parts[3]; + } + + Matcher matcher = COMPILED_JAVA_ADDRESS_PARSER.matcher(line); + if (matcher.find()) { + ip = matcher.group(1); + symbolAddress = matcher.group(2); + relAddress = matcher.group(3); + } break; } case 'j': { // j one.profiler.AsyncProfiler.stop()V+1 String[] parts = PLUS_SPLITTER.split(line, 2); - functionName = parts[0].substring(3); - if (parts.length > 1) { - try { - functionLine = Integer.parseInt(parts[1]); - } catch (Throwable ignored) { + if (parts.length > 0 && parts[0].length() > 3) { + functionName = parts[0].substring(3); + if (parts.length > 1) { + try { + functionLine = Integer.parseInt(parts[1]); + } catch (NumberFormatException ignored) { + } } } break; @@ -74,23 +210,23 @@ private StackFrame parseLine(String line) { case 'V': { // V [libjvm.so+0x8fc20a] thread_entry(JavaThread*, JavaThread*)+0x8a - if (line.endsWith("]")) { - // C [libpthread.so.0+0x13d60] - functionName = line.substring(4, line.length() - 1); - } else { - int plusIdx = line.lastIndexOf('+'); - functionName = - plusIdx > -1 - ? line.substring(line.indexOf(']') + 3, plusIdx) - : line.substring(line.indexOf(']') + 3); - } + // C [libpthread.so.0+0x13d60] int libstart = line.indexOf('['); if (libstart > 0) { int libend = line.indexOf(']', libstart + 1); if (libend > 0) { - String[] parts = PLUS_SPLITTER.split(line.substring(libstart + 1, libend), 2); + String libAndRelAddress = line.substring(libstart + 1, libend); + String[] parts = PLUS_SPLITTER.split(libAndRelAddress, 2); filename = parts[0]; - // TODO: extract relative address for second part and send to the intake + if (parts.length > 1) { + relAddress = parts[1]; + } + + // Extract function name if present (after the bracket) + // Keep the relative address offset as part of the function name + if (libend + 3 < line.length() && !line.endsWith("]")) { + functionName = line.substring(libend + 3).trim(); + } } } break; @@ -98,29 +234,126 @@ private StackFrame parseLine(String line) { case 'v': { // v ~StubRoutines::call_stub - int plusIdx = line.lastIndexOf('+'); - functionName = - plusIdx > -1 - ? line.substring(line.indexOf(']') + 3, plusIdx) - : line.substring(line.indexOf(']') + 3); + // v ~RuntimeStub::_new_array_Java 0x00000001124cb638 + if (line.length() > 3) { + String remaining = line.substring(3).trim(); + // Check for address at the end (0x...) + int lastSpace = remaining.lastIndexOf(' '); + if (lastSpace > 0 && lastSpace + 1 < remaining.length()) { + String possibleAddress = remaining.substring(lastSpace + 1); + if (possibleAddress.startsWith("0x")) { + relAddress = possibleAddress; + remaining = remaining.substring(0, lastSpace).trim(); + } + } + // Keep the relative address offset as part of the function name + functionName = remaining; + } break; } default: // do nothing break; } - if (functionName != null) { - return new StackFrame(filename, functionLine, functionName); + if (filename != null && !filename.isEmpty()) { + buildIdCollector.addUnprocessedLibrary(filename); + } + + if (functionName != null || filename != null) { + return new StackFrame( + filename, + functionLine, + stripCompilerAnnotations(functionName), + frameType, + null, + null, + null, + ip, + symbolAddress, + relAddress); + } + return null; + } + + private static String stripCompilerAnnotations(String functionName) { + if (functionName == null) { + return null; + } + // Strip compiler annotations like [clone .isra.531], [clone .constprop.0], etc. + int bracketIdx = functionName.lastIndexOf(" ["); + if (bracketIdx > 0 && functionName.endsWith("]")) { + return functionName.substring(0, bracketIdx); + } + return functionName; + } + + private static String knownLibraryPrefix(String filename) { + final String lowerCased = filename.toLowerCase(Locale.ROOT); + for (String prefix : KNOWN_LIBRARY_NAMES) { + if (lowerCased.startsWith(prefix)) { + return prefix; + } } return null; } + private static String normalizeFilename(String filename) { + if (filename == null) { + return null; + } + final String prefix = knownLibraryPrefix(filename); + if (prefix == null) { + return filename; + } + + final int prefixLen = prefix.length(); + final int end = filename.indexOf('.', prefixLen); + if (end < prefixLen) { + return filename; + } + return filename.substring(0, prefixLen) + filename.substring(end); + } + + static String parseCurrentThreadName(String line) { + if (line == null || !line.startsWith("Current thread ")) { + return null; + } + final int separator = line.indexOf(':'); + if (separator < 0) { + return null; + } + + String threadDescriptor = line.substring(separator + 1).trim(); + final int metadataStart = threadDescriptor.indexOf('['); + if (metadataStart >= 0) { + threadDescriptor = threadDescriptor.substring(0, metadataStart).trim(); + } + if (threadDescriptor.isEmpty()) { + return null; + } + return threadDescriptor; + } + + private static List parseHotspotJvmArgs(String line) { + if (line == null || !line.startsWith(HOTSPOT_JVM_ARGS_PREFIX)) { + return null; + } + return RuntimeArgs.parseVmArgs(line.substring(HOTSPOT_JVM_ARGS_PREFIX.length())); + } + public CrashLog parse(String uuid, String crashLog) { - String signal = null; + SigInfo sigInfo = null; String pid = null; + String threadName = null; List frames = new ArrayList<>(); String datetime = null; - StringBuilder message = new StringBuilder(); + String datetimeRaw = null; + boolean incomplete = false; + String oomMessage = null; + Map registers = null; + List runtimeArgs = null; + List dynamicLibraryLines = null; + String dynamicLibraryKey = null; String[] lines = NEWLINE_SPLITTER.split(crashLog); outer: @@ -129,7 +362,6 @@ public CrashLog parse(String uuid, String crashLog) { case NEW: if (line.startsWith( "# A fatal error has been detected by the Java Runtime Environment:")) { - message.append("\n\n"); state = State.MESSAGE; // jump directly to MESSAGE state } break; @@ -137,21 +369,19 @@ public CrashLog parse(String uuid, String crashLog) { if (line.toLowerCase().contains("core dump")) { // break out of the message block state = State.HEADER; - } else if (!"#".equals(line)) { - if (signal == null) { - // first non-empty line after the message is the signal - signal = - line.substring( - 3, - line.indexOf( - ' ', 3)); // # SIGSEGV (0xb) at pc=0x00007f8b1c0b3d7d, pid=1, tid=1 + } else if (oomMessage == null + && (sigInfo == null || "INVALID".equals(sigInfo.name)) + && !"#".equals(line)) { + // note: some jvm might use INVALID to represent a OOM crash too. + final int oomIdx = line.indexOf(OOM_MARKER); + if (oomIdx > 0) { + oomMessage = line.substring(oomIdx + OOM_MARKER.length()); + } else { int pidIdx = line.indexOf("pid="); if (pidIdx > -1) { int endIdx = line.indexOf(',', pidIdx); pid = line.substring(pidIdx + 4, endIdx); } - } else { - message.append(line.substring(2)).append('\n'); } } break; @@ -171,23 +401,108 @@ public CrashLog parse(String uuid, String crashLog) { } break; case THREAD: + if (threadName == null) { + threadName = parseCurrentThreadName(line); + } // Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) if (line.startsWith("Native frames: ")) { - message.append('\n').append(line).append('\n'); state = State.STACKTRACE; } break; case STACKTRACE: - if (line.isEmpty()) { - state = State.DONE; + if (line.startsWith("siginfo:")) { + // spotless:off + // siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x70 + // siginfo: si_signo: 11 (SIGSEGV), si_code: 0 (SI_USER), si_pid: 554848, si_uid: 1000 + // spotless:on + final Matcher siginfoMatcher = SIGINFO_PARSER.matcher(line); + if (siginfoMatcher.matches()) { + Integer number = safelyParseInt(siginfoMatcher.group(1)); + String name = siginfoMatcher.group(2); + Integer siCode = safelyParseInt(siginfoMatcher.group(3)); + String sigAction = siginfoMatcher.group(4); + String address = siginfoMatcher.group(5); + Integer siPid = safelyParseInt(siginfoMatcher.group(6)); + Integer siUid = safelyParseInt(siginfoMatcher.group(7)); + sigInfo = new SigInfo(number, name, siCode, sigAction, address, siPid, siUid); + } + } else if (line.startsWith("Registers:")) { + registers = new LinkedHashMap<>(); + state = State.REGISTERS; + } else if (line.contains("P R O C E S S")) { + state = State.PROCESS; } else { // Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) - message.append(line).append('\n'); - frames.add(parseLine(line)); + final StackFrame frame = parseLine(line); + if (frame != null) { + frames.add(frame); + } + } + break; + case REGISTERS: + if (!line.isEmpty() && !REGISTER_LINE_START.matcher(line).find()) { + // non-empty line that does not start with a register entry signals end of section + state = State.STACKTRACE; + } else { + final Matcher m = REGISTER_ENTRY_PARSER.matcher(line); + while (m.find()) { + registers.put(m.group(1), m.group(2)); + } + } + break; + case PROCESS: + if (runtimeArgs == null && line.startsWith("VM Arguments:")) { + state = State.VM_ARGUMENTS; + } else if (line.startsWith("Dynamic libraries:")) { + state = State.DYNAMIC_LIBRARIES; + } else if (line.contains("S Y S T E M")) { + state = State.SYSTEM; + } else if (line.equals("END.")) { + state = State.DONE; + } + break; + case VM_ARGUMENTS: + if (line.isEmpty()) { + state = State.PROCESS; + } else if (runtimeArgs == null && line.startsWith(HOTSPOT_JVM_ARGS_PREFIX)) { + runtimeArgs = parseHotspotJvmArgs(line); + } + break; + case DYNAMIC_LIBRARIES: + if (line.isEmpty()) { + state = State.PROCESS; + } else { + if (dynamicLibraryKey == null) { + dynamicLibraryKey = detectDynamicLibrariesKey(line); + dynamicLibraryLines = new ArrayList<>(); + } + final Matcher matcher = DYNAMIC_LIBS_PATH_PARSER.matcher(line); + if (matcher.matches()) { + final String pathString = matcher.group(1); + if (pathString != null && !pathString.isEmpty()) { + dynamicLibraryLines.add(line); + try { + final Path path = Paths.get(pathString); + buildIdCollector.resolveBuildId(path); + } catch (InvalidPathException ignored) { + } + } + } + } + break; + case SYSTEM: + if (line.equals("END.")) { + state = State.DONE; + } else if (datetime == null && datetimeRaw == null && line.startsWith("time: ")) { + // JDK 8 fallback: no SUMMARY section, time is split across two lines here + datetimeRaw = line.substring(6).trim(); + } else if (datetime == null && datetimeRaw != null && line.startsWith("timezone: ")) { + datetime = dateTimeToISO(datetimeRaw + " " + line.substring(10).trim()); } break; case DONE: // skip + buildIdCollector.awaitCollectionDone(5); break outer; default: // unexpected parser state; bail out @@ -195,25 +510,88 @@ public CrashLog parse(String uuid, String crashLog) { } } - if (state != State.DONE) { + // PROCESS and SYSTEM sections are late enough that all critical data is captured + if (state != State.DONE && state != State.PROCESS && state != State.SYSTEM) { // incomplete crash log - return null; + incomplete = true; + } + final String kind; + final String message; + if (oomMessage != null) { + kind = "OutOfMemory"; + message = oomMessage; + } else { + kind = sigInfo != null && sigInfo.name != null ? sigInfo.name : "UNKNOWN"; + message = "Process terminated by signal " + kind; + } + + final List enrichedFrames = new ArrayList<>(frames.size()); + + for (StackFrame frame : frames) { + // enrich with the build id if collected (best effort) + if (frame.path == null) { + enrichedFrames.add(frame); + continue; + } + final BuildInfo buildInfo = buildIdCollector.getBuildInfo(frame.path); + if (buildInfo != null) { + enrichedFrames.add( + new StackFrame( + normalizeFilename(frame.path), + frame.line, + frame.function, + frame.frameType, + buildInfo.buildId, + buildInfo.buildIdType, + buildInfo.fileType, + frame.ip, + frame.symbolAddress, + frame.relativeAddress)); + } else { + enrichedFrames.add( + new StackFrame( + normalizeFilename(frame.path), + frame.line, + frame.function, + frame.frameType, + null, + null, + null, + frame.ip, + frame.symbolAddress, + frame.relativeAddress)); + } } ErrorData error = new ErrorData( - signal, message.toString(), new StackTrace(frames.toArray(new StackFrame[0]))); + kind, message, threadName, new StackTrace(enrichedFrames.toArray(new StackFrame[0]))); // We can not really extract the full metadata and os info from the crash log // This code assumes the parser is run on the same machine as the crash happened Metadata metadata = new Metadata("dd-trace-java", VersionInfo.VERSION, "java", null); - OSInfo osInfo = - new OSInfo( - SystemProperties.get("os.arch"), - SystemProperties.get("sun.arch.data.model"), - SystemProperties.get("os.name"), - SemanticVersion.of(SystemProperties.get("os.version"))); - ProcInfo procInfo = pid != null ? new ProcInfo(pid) : null; - return new CrashLog(uuid, false, datetime, error, metadata, osInfo, procInfo, "1.0"); + Integer parsedPid = safelyParseInt(pid); + ProcInfo procInfo = parsedPid != null ? new ProcInfo(parsedPid) : null; + Experimental experimental = + (registers != null && !registers.isEmpty()) + || (runtimeArgs != null && !runtimeArgs.isEmpty()) + ? new Experimental(registers, runtimeArgs) + : null; + DynamicLibs files = + (dynamicLibraryLines != null && !dynamicLibraryLines.isEmpty()) + ? new DynamicLibs(dynamicLibraryKey, dynamicLibraryLines) + : null; + return new CrashLog( + uuid, + incomplete, + datetime, + error, + metadata, + OSInfo.current(), + procInfo, + sigInfo, + "1.0", + experimental, + files); } static String dateTimeToISO(String datetime) { @@ -229,4 +607,49 @@ static String dateTimeToISO(String datetime) { } } } + + /** + * Detects whether the Dynamic libraries section comes from Linux {@code /proc/self/maps} (address + * range format {@code addr-addr perms ...}) or from the BSD/macOS dyld callback (format {@code + * 0xaddr\tpath}). Returns the appropriate map key. + */ + // The "Dynamic libraries:" section is written by os::print_dll_info(), whose implementation + // differs by platform: + // + // Linux + // ----- + // This reads `/proc/{tid}/maps` verbatim via _print_ascii_file(), producing the usual + // `/proc/self/maps` format: + // + // "addr-addr perms offset dev:inode [path]" + // + // Mainline: + // https://github.com/openjdk/jdk/blob/783f8f1adc4ea3ef7fd4c5ca5473aad76dfc7ed1/src/hotspot/os/linux/os_linux.cpp#L2086-L2099 + // + // BSD/macOS + // --------- + // This relies on `_dyld_image_count()`/`_dyld_get_image_name()` (on macOS) or + // `dlinfo(RTLD_DI_LINKMAP)` (on FreeBSD/OpenBSD) via a callback, producing a simpler format: + // + // "0xaddr\tpath" + // + // which lacks much of the information found in Linux's `/proc/self/maps`. + // Mainline: + // https://github.com/openjdk/jdk/blob/783f8f1adc4ea3ef7fd4c5ca5473aad76dfc7ed1/src/hotspot/os/bsd/os_bsd.cpp#L1382-L1387 + static String detectDynamicLibrariesKey(String firstLine) { + int dash = firstLine.indexOf('-'); + int space = firstLine.indexOf(' '); + return (dash > 0 && space > 0 && dash < space) ? "/proc/self/maps" : "dynamic_libraries"; + } + + static Integer safelyParseInt(String value) { + if (value == null) { + return null; + } + try { + return Integer.parseInt(value); + } catch (NumberFormatException e) { + return null; + } + } } diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/parsers/J9JavacoreParser.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/parsers/J9JavacoreParser.java new file mode 100644 index 00000000000..1d7214dde6d --- /dev/null +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/parsers/J9JavacoreParser.java @@ -0,0 +1,503 @@ +package datadog.crashtracking.parsers; + +import static java.time.format.DateTimeFormatter.ISO_OFFSET_DATE_TIME; + +import datadog.common.version.VersionInfo; +import datadog.crashtracking.buildid.BuildIdCollector; +import datadog.crashtracking.buildid.BuildInfo; +import datadog.crashtracking.dto.CrashLog; +import datadog.crashtracking.dto.ErrorData; +import datadog.crashtracking.dto.Experimental; +import datadog.crashtracking.dto.Metadata; +import datadog.crashtracking.dto.OSInfo; +import datadog.crashtracking.dto.ProcInfo; +import datadog.crashtracking.dto.SigInfo; +import datadog.crashtracking.dto.StackFrame; +import datadog.crashtracking.dto.StackTrace; +import java.nio.file.InvalidPathException; +import java.nio.file.Paths; +import java.time.LocalDateTime; +import java.time.ZoneId; +import java.time.ZonedDateTime; +import java.time.format.DateTimeFormatter; +import java.time.format.DateTimeParseException; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +/** + * Parser for J9/OpenJ9 javacore crash dump files. + * + *

J9 javacore files use a hierarchical tag-based format with numbered prefixes indicating + * section depth (0SECTION, 1TI*, 2XH*, 3XM*, 4XE*, etc). + * + *

Key sections: + * + *

    + *
  • TITLE - Contains dump event type and timestamp + *
  • GPINFO - General information including OS level and CPU architecture + *
  • ENVINFO - Environment info including process ID + *
  • THREADS - Thread information and stack traces + *
+ */ +public final class J9JavacoreParser { + private static final String J9_USER_ARG_PREFIX = "2CIUSERARG"; + + private final BuildIdCollector buildIdCollector; + + public J9JavacoreParser() { + this.buildIdCollector = new BuildIdCollector(); + } + + private static final String OOM_MARKER = "OutOfMemory"; + + // J9 event types mapped to signal names and numbers + private static final String EVENT_GPF = "gpf"; + private static final String EVENT_ABORT = "abort"; + private static final String EVENT_SYSTHROW = "systhrow"; + + // Section markers + private static final String SECTION_MARKER = "0SECTION"; + private static final String SECTION_TITLE = "TITLE"; + private static final String SECTION_GPINFO = "GPINFO"; + private static final String SECTION_ENVINFO = "ENVINFO"; + private static final String SECTION_THREADS = "THREADS"; + + // Tag patterns + private static final Pattern NEWLINE_SPLITTER = Pattern.compile("\\n"); + private static final Pattern SIG_INFO_PATTERN = + Pattern.compile("1TISIGINFO\\s+Dump Event \"(\\w+)\"(?:\\s+\\((\\w+)\\))?.*"); + private static final Pattern DATETIME_PATTERN = + Pattern.compile( + "1TIDATETIME\\s+Date:\\s+(\\d{4}/\\d{2}/\\d{2})\\s+at\\s+(\\d{2}:\\d{2}:\\d{2})(?::(\\d{3}))?.*"); + private static final Pattern PID_PATTERN = + Pattern.compile("1CIPROCESSID\\s+Process ID:\\s+(\\d+).*"); + private static final Pattern CURRENT_THREAD_PATTERN = + Pattern.compile("1XMCURTHDINFO\\s+Current thread.*"); + private static final Pattern THREAD_INFO_PATTERN = + Pattern.compile("3XMTHREADINFO\\s+\"(.+?)\".*"); + private static final Pattern JAVA_STACK_PATTERN = Pattern.compile("4XESTACKTRACE\\s+at\\s+(.+)"); + private static final Pattern NATIVE_STACK_PATTERN = Pattern.compile("4XENATIVESTACK\\s+(.+)"); + private static final Pattern EXCEPTION_DETAIL_PATTERN = + Pattern.compile("1TISIGINFO.*[Dd]etail\\s+\"(.+?)\".*"); + // Matches register entries in J9 GPINFO section, e.g.: + // 2XHREGISTER RDI: 0000000000000001 (x86-64) + // 2XHREGISTER R29: 0000FFFF990CDB50 (aarch64) + private static final Pattern REGISTER_ENTRY_PARSER = + Pattern.compile("([A-Za-z][A-Za-z0-9]*)\\s*:\\s*([0-9a-fA-F]+)"); + // Date time formatter for J9 format: YYYY/MM/DD at HH:MM:SS + private static final DateTimeFormatter J9_DATETIME_FORMATTER = + DateTimeFormatter.ofPattern("yyyy/MM/dd HH:mm:ss", Locale.ROOT); + + enum Section { + NONE, + TITLE, + GPINFO, + ENVINFO, + THREADS, + OTHER + } + + public CrashLog parse(String uuid, String javacoreContent) { + // Parser state - kept as local variables to ensure thread safety and reusability + Section currentSection = Section.NONE; + boolean inCurrentThread = false; + boolean collectingStack = false; + String eventType = null; + String eventCode = null; + String exceptionDetail = null; + String pid = null; + String datetime = null; + String currentThreadName = null; + List frames = new ArrayList<>(); + boolean incomplete = false; + boolean foundThreadSection = false; + + Map registers = null; + RuntimeArgs j9UserArgs = new RuntimeArgs(); + + String[] lines = NEWLINE_SPLITTER.split(javacoreContent); + + for (String line : lines) { + // Full command line is available under 1CICMDLINE, but it's harder to parse properly, + // than adding them from 2CIUSERARGS + if (line.startsWith(J9_USER_ARG_PREFIX)) { + j9UserArgs.addArg(line.substring(J9_USER_ARG_PREFIX.length()).trim()); + } + + // Track section changes + if (line.startsWith(SECTION_MARKER)) { + currentSection = detectSection(line); + if (currentSection == Section.THREADS) { + foundThreadSection = true; + inCurrentThread = false; + collectingStack = false; + } + continue; + } + + switch (currentSection) { + case TITLE: + // Extract event type (gpf, abort, systhrow) + Matcher sigMatcher = SIG_INFO_PATTERN.matcher(line); + if (sigMatcher.matches()) { + eventType = sigMatcher.group(1); + eventCode = sigMatcher.group(2); + } + + // Extract exception detail for systhrow events + Matcher detailMatcher = EXCEPTION_DETAIL_PATTERN.matcher(line); + if (detailMatcher.matches()) { + exceptionDetail = detailMatcher.group(1); + } + + // Extract timestamp + Matcher dtMatcher = DATETIME_PATTERN.matcher(line); + if (dtMatcher.matches()) { + datetime = parseDateTime(dtMatcher.group(1), dtMatcher.group(2)); + } + break; + + case GPINFO: + if (line.startsWith("1XHREGISTERS")) { + registers = new LinkedHashMap<>(); + } else if (registers != null && line.startsWith("2XHREGISTER")) { + final Matcher m = REGISTER_ENTRY_PARSER.matcher(line); + while (m.find()) { + registers.put(m.group(1), "0x" + m.group(2)); + } + } + break; + + case ENVINFO: + // Extract process ID + Matcher pidMatcher = PID_PATTERN.matcher(line); + if (pidMatcher.matches()) { + pid = pidMatcher.group(1); + } + break; + + case THREADS: + // Look for current thread marker + if (CURRENT_THREAD_PATTERN.matcher(line).matches()) { + inCurrentThread = true; + continue; + } + + // If in current thread section, look for thread info start + if (inCurrentThread && line.startsWith("3XMTHREADINFO")) { + Matcher threadMatcher = THREAD_INFO_PATTERN.matcher(line); + if (threadMatcher.matches()) { + currentThreadName = threadMatcher.group(1); + collectingStack = true; + } + continue; + } + + // Collect stack frames for current thread + if (collectingStack) { + // Java stack frame + Matcher javaStackMatcher = JAVA_STACK_PATTERN.matcher(line); + if (javaStackMatcher.matches()) { + StackFrame frame = parseJavaStackFrame(javaStackMatcher.group(1)); + if (frame != null) { + frames.add(frame); + } + continue; + } + + // Native stack frame + Matcher nativeStackMatcher = NATIVE_STACK_PATTERN.matcher(line); + if (nativeStackMatcher.matches()) { + StackFrame frame = parseNativeStackFrame(nativeStackMatcher.group(1)); + if (frame != null) { + frames.add(frame); + } + continue; + } + + // End of stack trace - blank line or new section + if (line.isEmpty() || line.startsWith("NULL") || line.startsWith("0SECTION")) { + collectingStack = false; + inCurrentThread = false; + } + } + break; + + default: + break; + } + } + + // Check for incomplete parse + if (!foundThreadSection || (eventType == null && exceptionDetail == null)) { + incomplete = true; + } + + // Wait for build ID collection to complete + buildIdCollector.awaitCollectionDone(5); + + // Build signal info from event type + SigInfo sigInfo = buildSigInfo(eventType, eventCode); + + // Determine error kind and message + String kind; + String message; + if (isOOMEvent(eventType, exceptionDetail)) { + kind = "OutOfMemory"; + message = exceptionDetail != null ? exceptionDetail : "OutOfMemoryError"; + } else if (eventType != null) { + kind = + sigInfo != null && sigInfo.name != null + ? sigInfo.name + : eventType.toUpperCase(Locale.ROOT); + message = "Process terminated by signal " + kind; + } else { + kind = "UNKNOWN"; + message = "Unknown crash event"; + } + + // Enrich frames with build IDs (best effort) + final List enrichedFrames = new ArrayList<>(frames.size()); + for (StackFrame frame : frames) { + if (frame.path == null) { + enrichedFrames.add(frame); + continue; + } + + // Try to resolve build ID for this library + final BuildInfo buildInfo = buildIdCollector.getBuildInfo(frame.path); + if (buildInfo != null) { + enrichedFrames.add( + new StackFrame( + frame.path, + frame.line, + frame.function, + frame.frameType, + buildInfo.buildId, + buildInfo.buildIdType, + buildInfo.fileType, + frame.ip, + frame.symbolAddress, + frame.relativeAddress)); + } else { + enrichedFrames.add(frame); + } + } + + ErrorData error = + new ErrorData( + kind, + message, + currentThreadName, + new StackTrace(enrichedFrames.toArray(new StackFrame[0]))); + Metadata metadata = new Metadata("dd-trace-java", VersionInfo.VERSION, "java", null); + Integer parsedPid = safelyParseInt(pid); + ProcInfo procInfo = parsedPid != null ? new ProcInfo(parsedPid) : null; + List runtimeArgs = j9UserArgs.build(); + Experimental experimental = + (registers != null && !registers.isEmpty()) + || (runtimeArgs != null && !runtimeArgs.isEmpty()) + ? new Experimental(registers, runtimeArgs) + : null; + + return new CrashLog( + uuid, + incomplete, + datetime, + error, + metadata, + OSInfo.current(), + procInfo, + sigInfo, + "1.0", + experimental, + null); + } + + private static Integer safelyParseInt(String value) { + if (value == null) { + return null; + } + try { + return Integer.parseInt(value); + } catch (NumberFormatException e) { + return null; + } + } + + private static Section detectSection(String line) { + if (line.contains(SECTION_TITLE)) { + return Section.TITLE; + } else if (line.contains(SECTION_GPINFO)) { + return Section.GPINFO; + } else if (line.contains(SECTION_ENVINFO)) { + return Section.ENVINFO; + } else if (line.contains(SECTION_THREADS)) { + return Section.THREADS; + } else { + return Section.OTHER; + } + } + + private boolean isOOMEvent(String eventType, String exceptionDetail) { + if (EVENT_SYSTHROW.equals(eventType) && exceptionDetail != null) { + return exceptionDetail.contains(OOM_MARKER); + } + return false; + } + + private SigInfo buildSigInfo(String eventType, String eventCode) { + if (eventType == null) { + return null; + } + + String signalName; + int signalNumber; + + switch (eventType.toLowerCase(Locale.ROOT)) { + case EVENT_GPF: + signalName = "SIGSEGV"; + signalNumber = 11; + break; + case EVENT_ABORT: + signalName = "SIGABRT"; + signalNumber = 6; + break; + case EVENT_SYSTHROW: + signalName = "EXCEPTION"; + signalNumber = 0; + break; + default: + signalName = eventType.toUpperCase(Locale.ROOT); + signalNumber = parseEventCode(eventCode); + } + + return new SigInfo(signalNumber, signalName, null, null, null, null, null); + } + + private int parseEventCode(String eventCode) { + if (eventCode == null) { + return 0; + } + try { + return Integer.decode(eventCode); + } catch (NumberFormatException e) { + return 0; + } + } + + /** + * Parse J9 Java stack frame format: package/Class.method(Source.java:line) or + * package/Class.method(Native Method) + */ + private StackFrame parseJavaStackFrame(String frameText) { + String function = frameText.trim(); + String file = null; + Integer line = null; + + // Extract source file and line: method(File.java:123) + int parenStart = function.lastIndexOf('('); + int parenEnd = function.lastIndexOf(')'); + String frameType = "java"; + if (parenStart > 0 && parenEnd > parenStart) { + String sourceInfo = function.substring(parenStart + 1, parenEnd); + function = function.substring(0, parenStart); + + if ("Native Method".equals(sourceInfo)) { + frameType = "native"; + } else { + int colonIdx = sourceInfo.lastIndexOf(':'); + if (colonIdx > 0) { + file = sourceInfo.substring(0, colonIdx); + try { + line = Integer.parseInt(sourceInfo.substring(colonIdx + 1)); + } catch (NumberFormatException ignored) { + // Keep line as null + } + } else { + file = sourceInfo; + } + } + } + + return new StackFrame(file, line, function, frameType, null, null, null, null, null, null); + } + + /** + * Parse J9 native stack frame format: (0xADDRESS [library+offset]) or functionName+offset + * (address [library+offset]) + */ + private StackFrame parseNativeStackFrame(String frameText) { + String text = frameText.trim(); + String function = null; + String file = null; + String relAddress = null; + + // Try to extract library from [lib+offset] pattern + int bracketStart = text.indexOf('['); + int bracketEnd = text.indexOf(']', bracketStart + 1); + if (bracketStart >= 0 && bracketEnd > bracketStart) { + String libInfo = text.substring(bracketStart + 1, bracketEnd); + int plusIdx = libInfo.indexOf('+'); + if (plusIdx > 0) { + file = libInfo.substring(0, plusIdx); + relAddress = libInfo.substring(plusIdx + 1); + } else { + file = libInfo; + } + } + + // Try to extract function name (before the first parenthesis or bracket) + int funcEnd = text.indexOf('('); + if (funcEnd < 0) { + funcEnd = bracketStart >= 0 ? bracketStart : text.length(); + } + if (funcEnd > 0) { + String funcPart = text.substring(0, funcEnd).trim(); + // Remove trailing +offset if present + int plusIdx = funcPart.lastIndexOf('+'); + if (plusIdx > 0) { + function = funcPart.substring(0, plusIdx).trim(); + } else if (!funcPart.isEmpty() && !funcPart.startsWith("0x")) { + function = funcPart; + } + } + + // If we couldn't extract a function name, use the whole text + if (function == null || function.isEmpty()) { + function = text; + } + + // Collect library for build ID resolution + if (file != null && !file.isEmpty()) { + buildIdCollector.addUnprocessedLibrary(file); + + // If the library name looks like a path, also try to resolve its build ID directly + if (file.contains("/")) { + try { + buildIdCollector.resolveBuildId(Paths.get(file)); + } catch (InvalidPathException ignored) { + // Not a valid path, skip + } + } + } + + return new StackFrame(file, null, function, "native", null, null, null, null, null, relAddress); + } + + private String parseDateTime(String datePart, String timePart) { + try { + String combined = datePart + " " + timePart; + LocalDateTime localDateTime = LocalDateTime.parse(combined, J9_DATETIME_FORMATTER); + ZonedDateTime zonedDateTime = localDateTime.atZone(ZoneId.systemDefault()); + return zonedDateTime.format(ISO_OFFSET_DATE_TIME); + } catch (DateTimeParseException e) { + return null; + } + } +} diff --git a/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/parsers/RuntimeArgs.java b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/parsers/RuntimeArgs.java new file mode 100644 index 00000000000..8a422f5166f --- /dev/null +++ b/dd-java-agent/agent-crashtracking/src/main/java/datadog/crashtracking/parsers/RuntimeArgs.java @@ -0,0 +1,235 @@ +package datadog.crashtracking.parsers; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +/** + * Utility for normalizing and filtering JVM runtime arguments captured from crash artifacts. + * + *

The helper supports two input shapes: + * + *

    + *
  • a raw JVM-args record, such as HotSpot {@code jvm_args: ...} + *
  • pre-split J9 user arguments, such as individual {@code 2CIUSERARG} records + *
+ * + *

Only a curated subset of arguments is retained for telemetry: {@code -Djdk.*}, {@code + * -Djava.*}, {@code -Dsun.*}, {@code -javaagent:}, {@code -agentlib:}, {@code -X*}, and + * module/native-access options. + */ +final class RuntimeArgs { + // Aligned with JDK JEP-8372760 (JFR In-Process Data Redaction) default filter list. + private static final String[] SECRET_PROPERTY_KEYWORDS = { + "auth", "password", "passwd", "pwd", "passphrase", "secret", "token", "key", "credential" + }; + private static final String[] MODULE_OPTIONS = { + "--add-modules", + "--add-exports", + "--add-opens", + "--add-reads", + "--patch-module", + "--limit-modules", + "--module-path", + "--upgrade-module-path", + "--enable-native-access", + "--illegal-native-access", + "--sun-misc-unsafe-memory-access" + }; + + private final List args = new ArrayList<>(); + + /** Returns a filtered args list from the raw JVM-arguments. */ + static List parseVmArgs(String raw) { + return filterArgs(joinArgumentTokens(splitArgs(raw))); + } + + void addArg(String arg) { + if (arg != null && !arg.isEmpty()) { + args.add(arg); + } + } + + List build() { + return filterArgs(args); + } + + private static List filterArgs(List args) { + if (args.isEmpty()) { + return Collections.emptyList(); + } + List filtered = new ArrayList<>(); + for (String arg : args) { + if (arg == null || arg.isEmpty()) { + continue; + } + if (isAllowedSystemProperty(arg)) { + filtered.add(arg); + } else if (arg.startsWith("-javaagent:") || arg.startsWith("-agentlib:")) { + // Redact options after '=' — only the jar path / library name is sent + int eq = arg.indexOf('=', arg.indexOf(':') + 1); + filtered.add(eq >= 0 ? arg.substring(0, eq) + "=REDACTED" : arg); + } else if (arg.startsWith("-X") || isModuleOrNativeAccessOption(arg)) { + filtered.add(arg); + } + } + return filtered; + } + + private static boolean isModuleOrNativeAccessOption(String arg) { + for (String option : MODULE_OPTIONS) { + if (arg.equals(option) || arg.startsWith(option + "=") || arg.startsWith(option + " ")) { + return true; + } + } + return false; + } + + private static boolean isAllowedSystemProperty(String arg) { + if (hasSecretLikePropertyName(arg)) { + return false; + } + if (arg.startsWith("-Djdk.") || arg.startsWith("-Dosgi.")) { + return true; + } + // J9 lists them as vm args. + if (arg.startsWith("-Djava.class.path=") || arg.startsWith("-Dsun.java.command=")) { + return false; + } + return arg.startsWith("-Djava.") || arg.startsWith("-Dsun."); + } + + private static boolean hasSecretLikePropertyName(String arg) { + if (!arg.startsWith("-D")) { + return false; + } + int separator = arg.indexOf('='); + String propertyName = separator >= 0 ? arg.substring(2, separator) : arg.substring(2); + String normalizedPropertyName = propertyName.toLowerCase(); + for (String keyword : SECRET_PROPERTY_KEYWORDS) { + if (normalizedPropertyName.contains(keyword)) { + return true; + } + } + return false; + } + + private static List splitArgs(String raw) { + if (raw == null || raw.isEmpty()) { + return Collections.emptyList(); + } + List tokens = new ArrayList<>(); + StringBuilder current = new StringBuilder(raw.length()); + boolean inSingleQuote = false; + boolean inDoubleQuote = false; + boolean escaped = false; + + for (int i = 0; i < raw.length(); i++) { + char c = raw.charAt(i); + // Keep the escaped character verbatim and clear the escape state. + if (escaped) { + current.append(c); + escaped = false; + continue; + } + // Backslashes escape the next character unless we are inside single quotes. + if (c == '\\' && !inSingleQuote) { + escaped = true; + continue; + } + // Single quotes only toggle quoting when not already inside double quotes. + if (c == '\'' && !inDoubleQuote) { + inSingleQuote = !inSingleQuote; + continue; + } + // Double quotes only toggle quoting when not already inside single quotes. + if (c == '"' && !inSingleQuote) { + inDoubleQuote = !inDoubleQuote; + continue; + } + // Outside of quotes, whitespace terminates the current token. + if (Character.isWhitespace(c) && !inSingleQuote && !inDoubleQuote) { + if (current.length() > 0) { + tokens.add(current.toString()); + current.setLength(0); + } + continue; + } + current.append(c); + } + if (current.length() > 0) { + tokens.add(current.toString()); + } + return tokens; + } + + /** + * Joins shell-like tokens into argument-shaped units. + * + *

This pass reconstructs options that span multiple tokens, such as module options whose value + * is separated by whitespace and {@code -XX:OnError=} style options that carry shell fragments + * over multiple tokens. + */ + private static List joinArgumentTokens(List tokens) { + if (tokens.isEmpty()) { + return Collections.emptyList(); + } + List joinedArgs = new ArrayList<>(); + boolean canContinuePreviousArg = false; + StringBuilder argBuilder = new StringBuilder(); + for (int i = 0; i < tokens.size(); i++) { + String token = tokens.get(i); + if (token.isEmpty()) { + continue; + } + if (!token.startsWith("-")) { + if (canContinuePreviousArg + && !joinedArgs.isEmpty() + && isContinuationToken(token) + && acceptsContinuation(joinedArgs.get(joinedArgs.size() - 1))) { + int last = joinedArgs.size() - 1; + argBuilder.setLength(0); + argBuilder.append(joinedArgs.get(last)).append(' ').append(token); + joinedArgs.set(last, argBuilder.toString()); + continue; + } + canContinuePreviousArg = false; + continue; + } + + argBuilder.setLength(0); + argBuilder.append(token); + if (requiresSeparateValue(token) && i + 1 < tokens.size()) { + argBuilder.append(' ').append(tokens.get(++i)); + } + while (i + 1 < tokens.size() + && isContinuationToken(tokens.get(i + 1)) + && acceptsContinuation(argBuilder.toString())) { + argBuilder.append(' ').append(tokens.get(++i)); + } + String arg = argBuilder.toString(); + joinedArgs.add(arg); + canContinuePreviousArg = acceptsContinuation(arg); + } + return joinedArgs; + } + + private static boolean requiresSeparateValue(String token) { + for (String option : MODULE_OPTIONS) { + if (token.equals(option)) { + return true; + } + } + return false; + } + + private static boolean acceptsContinuation(String arg) { + return arg.startsWith("-XX:OnError=") + || arg.startsWith("-XX:OnOutOfMemoryError=") + || arg.startsWith("-Xdump:"); + } + + private static boolean isContinuationToken(String token) { + return !token.isEmpty() && !token.startsWith("-"); + } +} diff --git a/dd-java-agent/agent-crashtracking/src/main/resources/datadog/crashtracking/notify_oome.bat b/dd-java-agent/agent-crashtracking/src/main/resources/datadog/crashtracking/notify_oome.bat index b0d8b5bdbb4..9a1262da067 100644 --- a/dd-java-agent/agent-crashtracking/src/main/resources/datadog/crashtracking/notify_oome.bat +++ b/dd-java-agent/agent-crashtracking/src/main/resources/datadog/crashtracking/notify_oome.bat @@ -34,6 +34,12 @@ echo Tags: %tags% echo JAVA_HOME: %java_home% echo PID: %PID% +:: Clear environment variables that the parent JVM may have set so the child JVM +:: starts with a minimal configuration (avoids port conflicts, memory contention, etc.) +set JDK_JAVA_OPTIONS= +set JAVA_TOOL_OPTIONS= +set _JAVA_OPTIONS= + :: Execute the Java command with the loaded values "%java_home%\bin\java" -Ddd.dogstatsd.start-delay=0 -jar "%agent%" sendOomeEvent "%tags%" set RC=%ERRORLEVEL% diff --git a/dd-java-agent/agent-crashtracking/src/main/resources/datadog/crashtracking/notify_oome.sh b/dd-java-agent/agent-crashtracking/src/main/resources/datadog/crashtracking/notify_oome.sh index 5e75f6a7563..05371504b03 100644 --- a/dd-java-agent/agent-crashtracking/src/main/resources/datadog/crashtracking/notify_oome.sh +++ b/dd-java-agent/agent-crashtracking/src/main/resources/datadog/crashtracking/notify_oome.sh @@ -49,6 +49,15 @@ echo "Tags: $config_tags" echo "JAVA_HOME: $config_java_home" echo "PID: $PID" +# Clear environment variables that the parent JVM may have set so the child JVM +# starts with a minimal configuration (avoids port conflicts, memory contention, etc.) +unset JDK_JAVA_OPTIONS +unset JAVA_TOOL_OPTIONS +unset _JAVA_OPTIONS +# Prevent the instrumentation injector from re-injecting the agent into the child JVM +unset LD_PRELOAD +unset DYLD_INSERT_LIBRARIES + # Execute the Java command with the loaded values "$config_java_home/bin/java" -Ddd.dogstatsd.start-delay=0 -jar "$config_agent" sendOomeEvent "$config_tags" RC=$? diff --git a/dd-java-agent/agent-crashtracking/src/main/resources/datadog/crashtracking/upload_crash.bat b/dd-java-agent/agent-crashtracking/src/main/resources/datadog/crashtracking/upload_crash.bat index a525630da6f..7ee2064f734 100644 --- a/dd-java-agent/agent-crashtracking/src/main/resources/datadog/crashtracking/upload_crash.bat +++ b/dd-java-agent/agent-crashtracking/src/main/resources/datadog/crashtracking/upload_crash.bat @@ -4,7 +4,11 @@ setlocal enabledelayedexpansion :: Check if PID is provided if "%1"=="" ( echo "Error: No PID provided. Running in legacy mode." - call :ensureJava "!JAVA_ERROR_FILE!" + :: Clear environment variables that the parent JVM may have set + set JDK_JAVA_OPTIONS= + set JAVA_TOOL_OPTIONS= + set _JAVA_OPTIONS= + "!JAVA_HOME!\bin\java" -jar "!AGENT_JAR!" uploadCrash "!JAVA_ERROR_FILE!" if %ERRORLEVEL% EQU 0 ( echo "Uploaded error file \"!JAVA_ERROR_FILE!\"" @@ -28,28 +32,123 @@ if not exist "%configFile%" ( exit /b 1 ) +:: Initialize config values +set config_agent= +set config_hs_err= +set config_java_home= +set config_javacore_path= + :: Read the configuration file :: The expected contents are :: - agent: Path to the dd-java-agent.jar -:: - hs_err: Path to the hs_err log file -for /f "tokens=1,2 delims=: " %%a in (%configFile%.cfg) do ( - set %%a=%%b +:: - hs_err: Path to the hs_err log file (optional for J9) +:: - java_home: Path to Java installation +:: - javacore_path: Path/pattern for J9 javacore files (optional) +for /f "tokens=1,* delims==" %%a in (%configFile%) do ( + if "%%a"=="agent" set config_agent=%%b + if "%%a"=="hs_err" set config_hs_err=%%b + if "%%a"=="java_home" set config_java_home=%%b + if "%%a"=="javacore_path" set config_javacore_path=%%b +) + +:: Exiting early if agent or java_home is missing +if not defined config_agent ( + echo Error: Missing configuration - agent + exit /b 1 +) +if not defined config_java_home ( + echo Error: Missing configuration - java_home + exit /b 1 +) + +:: Find crash file - support both HotSpot (hs_err) and J9 (javacore) formats +set crash_file= + +:: Check HotSpot error file first +if defined config_hs_err ( + if exist "%config_hs_err%" ( + set crash_file=%config_hs_err% + ) +) + +:: Check default HotSpot error file location +if not defined crash_file ( + if exist "hs_err_pid%PID%.log" ( + set crash_file=hs_err_pid%PID%.log + ) ) +:: Look for J9 javacore files if no HotSpot error file found +if not defined crash_file ( + :: Check custom javacore path if configured + if defined config_javacore_path ( + :: Check if it's an existing directory + if exist "%config_javacore_path%\*" ( + :: Search for javacore files in the directory - strict pattern with dots around PID + for /f "delims=" %%f in ('dir /b /o-d "%config_javacore_path%\javacore.*.%PID%.*.txt" 2^>nul') do ( + if not defined crash_file set crash_file=%config_javacore_path%\%%f + ) + ) else if exist "%config_javacore_path%" ( + :: It's an existing file - use it directly + set crash_file=%config_javacore_path% + ) else ( + :: Try pattern substitution with %%pid (J9 uses %pid token) + set "substituted_path=!config_javacore_path:%%pid=%PID%!" + if exist "!substituted_path!" ( + set crash_file=!substituted_path! + ) else ( + :: Try the directory containing the pattern + for %%d in ("!config_javacore_path!") do set pattern_dir=%%~dpd + if exist "!pattern_dir!" ( + for /f "delims=" %%f in ('dir /b /o-d "!pattern_dir!javacore.*.%PID%.*.txt" 2^>nul') do ( + if not defined crash_file set crash_file=!pattern_dir!%%f + ) + ) + ) + ) + ) + + :: Fallback: search in current directory if no custom path or file not found + if not defined crash_file ( + for /f "delims=" %%f in ('dir /b /o-d javacore.*.%PID%.*.txt 2^>nul') do ( + if not defined crash_file set crash_file=%%f + ) + ) +) + +if not defined crash_file ( + echo Error: No crash file found for PID %PID% + if defined config_javacore_path ( + echo Searched custom path: %config_javacore_path% + ) + exit /b 1 +) + +:: Use the found crash file +set config_hs_err=%crash_file% + :: Debug: Print the loaded values (Optional) -echo Config file: %configFile% -echo Agent Jar: %agent% -echo Error Log: %hs_err% -echo JAVA_HOME: %java_home% +echo Agent Jar: %config_agent% +echo Error Log: %config_hs_err% +echo JAVA_HOME: %config_java_home% echo PID: %PID% +:: Clear environment variables that the parent JVM may have set so the child JVM +:: starts with a minimal configuration (avoids port conflicts, memory contention, etc.) +set JDK_JAVA_OPTIONS= +set JAVA_TOOL_OPTIONS= +set _JAVA_OPTIONS= + :: Execute the Java command with the loaded values -"%java_home%\bin\java" -jar "%agent%" uploadCrash -c "%configFile%" %hs_err%" +"%config_java_home%\bin\java" -jar "%config_agent%" uploadCrash -c "%configFile%" "%config_hs_err%" set RC=%ERRORLEVEL% -del "%configFile%" :: Clean up the configuration file + +:: Clean up the configuration file +del "%configFile%" + if %RC% EQU 0 ( - echo "Error file %hs_err% was uploaded successfully" + echo Error file %config_hs_err% was uploaded successfully ) else ( - echo "Error: Failed to upload error file %hs_err%" + echo Error: Failed to upload error file %config_hs_err% exit /b %RC% ) diff --git a/dd-java-agent/agent-crashtracking/src/main/resources/datadog/crashtracking/upload_crash.sh b/dd-java-agent/agent-crashtracking/src/main/resources/datadog/crashtracking/upload_crash.sh index 7cac4bcfb94..37abd608f06 100644 --- a/dd-java-agent/agent-crashtracking/src/main/resources/datadog/crashtracking/upload_crash.sh +++ b/dd-java-agent/agent-crashtracking/src/main/resources/datadog/crashtracking/upload_crash.sh @@ -7,6 +7,14 @@ set +e if [ -z "$1" ]; then echo "Warn: No PID provided. Running in legacy mode." + # Clear environment variables that the parent JVM may have set + unset JDK_JAVA_OPTIONS + unset JAVA_TOOL_OPTIONS + unset _JAVA_OPTIONS + # Prevent the instrumentation injector from re-injecting the agent into the child JVM + unset LD_PRELOAD + unset DYLD_INSERT_LIBRARIES + "!JAVA_HOME!/bin/java" -jar "!AGENT_JAR!" uploadCrash "!JAVA_ERROR_FILE!" if [ $? -eq 0 ]; then echo "Error file !JAVA_ERROR_FILE! was uploaded successfully" @@ -35,6 +43,7 @@ fi config_agent="" config_hs_err="" config_java_home="" +config_javacore_path="" # Read the configuration file while IFS="=" read -r key value; do @@ -42,21 +51,91 @@ while IFS="=" read -r key value; do agent) config_agent=$value ;; hs_err) config_hs_err=$value ;; java_home) config_java_home=$value ;; + javacore_path) config_javacore_path=$value ;; esac done < "$configFile" -# Exiting early if configuration is missing -if [ -z "$config_agent" ] || [ -z "$config_hs_err" ] || [ -z "$config_java_home" ]; then - echo "Error: Missing configuration" +# Exiting early if agent or java_home is missing +if [ -z "$config_agent" ] || [ -z "$config_java_home" ]; then + echo "Error: Missing configuration (agent or java_home)" + exit 1 +fi + +# Function to find J9 javacore file for given PID in a directory +find_javacore() { + search_dir="$1" + search_pid="$2" + # J9 javacore files are named: javacore..