Skip to content

Use canonical FFE fixtures#11355

Open
leoromanovsky wants to merge 13 commits into
masterfrom
leo.romanovsky/ffe-canonical-fixtures-20260512
Open

Use canonical FFE fixtures#11355
leoromanovsky wants to merge 13 commits into
masterfrom
leo.romanovsky/ffe-canonical-fixtures-20260512

Conversation

@leoromanovsky
Copy link
Copy Markdown
Contributor

@leoromanovsky leoromanovsky commented May 12, 2026 •
edited
Loading

Motivation

Make DataDog/ffe-system-test-data the canonical source for FFE/OpenFeature JSON fixtures in dd-trace-java. This removes copied smoke-test fixtures and moves evaluator behavior coverage to the shared corpus. Depends on DataDog/ffe-system-test-data#9.

Changes

  • Add dd-smoke-tests/openfeature/src/test/resources/ffe-system-test-data as a git submodule pinned to the canonical fixture update.
  • Remove copied OpenFeature fixture JSON resources.
  • Update the smoke test to load canonical ufc-config.json and all sorted evaluation-cases/*.json files.
  • Assert canonical result values and reasons from the JSON corpus.
  • Remove the stale programmatic evaluator fixture set in favor of the canonical JSON-driven loop.
  • Fix the OpenFeature smoke CodeNarc violation by removing an unnecessary Groovy import.
  • Add weekly Dependabot gitsubmodule updates.

📖 https://datadoghq.atlassian.net/wiki/spaces/PANA/pages/6713639211/FFE+SDK+Fixture+Contribution+Guide

Decisions

  • Shared evaluator behavior must come from JSON fixtures in ffe-system-test-data, not programmatic or copied fixture sets.
  • dd-trace-java already supports submodules and recursive checkout in CI; this PR uses the same pattern for the OpenFeature fixture corpus.
  • PHP is intentionally excluded until its OpenFeature client lands.

Validation

  • JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home PATH=/opt/homebrew/opt/openjdk@17/bin:$PATH ./gradlew :dd-smoke-tests:openfeature:codenarcTest
  • JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home PATH=/opt/homebrew/opt/openjdk@17/bin:$PATH ./gradlew :products:feature-flagging:feature-flagging-api:test --tests datadog.trace.api.openfeature.DDEvaluatorTest

@leoromanovsky leoromanovsky marked this pull request as ready for review May 18, 2026 12:21
@leoromanovsky leoromanovsky requested review from a team as code owners May 18, 2026 12:21
@leoromanovsky leoromanovsky requested review from dougqh, sameerank and typotter and removed request for a team May 18, 2026 12:21
@dd-octo-sts
Copy link
Copy Markdown
Contributor

dd-octo-sts Bot commented May 18, 2026 •
edited
Loading

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed May 18, 2026
View reviewed changes
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 057cdefcb8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ...e-flagging/feature-flagging-api/src/main/java/datadog/trace/api/openfeature/DDEvaluator.java Outdated
@datadog-datadog-prod-us1

This comment has been minimized.

Sign in to view
dougqh
dougqh reviewed May 26, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@dougqh dougqh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two review comments authored by Claude (Opus 4.7), posted on @dougqh's behalf.

Comment thread ...e-flagging/feature-flagging-api/src/main/java/datadog/trace/api/openfeature/DDEvaluator.java Outdated
Comment thread ...openfeature/src/test/groovy/datadog/smoketest/springboot/OpenFeatureProviderSmokeTest.groovy Outdated
dougqh
dougqh approved these changes May 26, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@dougqh dougqh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude detected a couple behavioral changes that I'm uncertain are intentional.
I'll leave those to your discretion.
Otherwise, looks good to me.

@leoromanovsky leoromanovsky added the comp: openfeature OpenFeature label May 26, 2026
@leoromanovsky
Copy link
Copy Markdown
Contributor Author

leoromanovsky commented May 26, 2026

Claude detected a couple behavioral changes that I'm uncertain are intentional. I'll leave those to your discretion. Otherwise, looks good to me.

@dougqh Thanks for the review; agreed with the assessment to take a closer look and remediate. I'll merge after someone on FFE reviews.

sameerank
sameerank approved these changes May 26, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@sameerank sameerank left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe one documentation thing, not strictly required

dd-trace-java/BUILDING.md

Line 182 in fbf651e

> This will keep the submodule in `dd-java-agent/agent-jmxfetch/integrations-core` up-to-date.

could mention both submodules: This will keep the submodules in `dd-java-agent/agent-jmxfetch/integrations-core` and `dd-smoke-tests/openfeature/src/test/resources/ffe-system-test-data` up-to-date.

@leoromanovsky
Copy link
Copy Markdown
Contributor Author

leoromanovsky commented May 26, 2026

Maybe one documentation thing, not strictly required

dd-trace-java/BUILDING.md

Line 182 in fbf651e

> This will keep the submodule in `dd-java-agent/agent-jmxfetch/integrations-core` up-to-date.

could mention both submodules: This will keep the submodules in `dd-java-agent/agent-jmxfetch/integrations-core` and `dd-smoke-tests/openfeature/src/test/resources/ffe-system-test-data` up-to-date.

@sameerank Good idea!

@leoromanovsky leoromanovsky enabled auto-merge May 26, 2026 18:43
@leoromanovsky leoromanovsky added this pull request to the merge queue May 29, 2026
@dd-octo-sts
Copy link
Copy Markdown
Contributor

dd-octo-sts Bot commented May 29, 2026

/merge

@gh-worker-devflow-routing-ef8351
Copy link
Copy Markdown

gh-worker-devflow-routing-ef8351 Bot commented May 29, 2026 •
edited
Loading

View all feedbacks in Devflow UI.

2026-05-29 01:28:45 UTC ℹ️ Start processing command /merge

2026-05-29 01:28:50 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-05-29 02:06:08 UTC ❌ MergeQueue: This merge request was updated

This PR is rejected because it was updated

@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 29, 2026
@leoromanovsky leoromanovsky added this pull request to the merge queue May 29, 2026
@dd-octo-sts
Copy link
Copy Markdown
Contributor

dd-octo-sts Bot commented May 29, 2026

/merge

@gh-worker-devflow-routing-ef8351
Copy link
Copy Markdown

gh-worker-devflow-routing-ef8351 Bot commented May 29, 2026 •
edited
Loading

View all feedbacks in Devflow UI.

2026-05-29 01:41:02 UTC ℹ️ Start processing command /merge

2026-05-29 01:41:05 UTC ❌ MergeQueue

PR already in the queue with status in_progress

@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 29, 2026
@leoromanovsky
Copy link
Copy Markdown
Contributor Author

leoromanovsky commented May 29, 2026

/merge

@gh-worker-devflow-routing-ef8351
Copy link
Copy Markdown

gh-worker-devflow-routing-ef8351 Bot commented May 29, 2026 •
edited
Loading

View all feedbacks in Devflow UI.

2026-05-29 02:09:24 UTC ℹ️ Start processing command /merge

2026-05-29 02:09:29 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2026-05-29 06:55:14 UTC ⚠️ MergeQueue: This merge request was unqueued

devflow unqueued this merge request: It did not become mergeable within the expected time

@dd-octo-sts
Copy link
Copy Markdown
Contributor

dd-octo-sts Bot commented May 29, 2026

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Startup Time

Scenario This PR master Change
insecure-bank / iast 13,963 ms 14,027 ms -0.5%
insecure-bank / tracing 12,905 ms 13,031 ms -1.0%
petclinic / appsec 16,535 ms 16,368 ms +1.0%
petclinic / iast 16,634 ms 16,560 ms +0.4%
petclinic / profiling 16,464 ms 16,453 ms +0.1%
petclinic / tracing 15,803 ms 15,756 ms +0.3%

Commit: fbfd381c · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

@dougqh dougqh dougqh approved these changes

@sameerank sameerank sameerank approved these changes

@typotter typotter Awaiting requested review from typotter typotter is a code owner automatically assigned from DataDog/feature-flagging-and-experimentation-sdk

Assignees

No one assigned

Labels

comp: openfeature OpenFeature type: refactoring

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@leoromanovsky @dougqh @sameerank