Send agentless Feature Flags exposures directly to EVP - #12195
Open
leoromanovsky wants to merge 4 commits into
Open
Send agentless Feature Flags exposures directly to EVP#12195leoromanovsky wants to merge 4 commits into
leoromanovsky wants to merge 4 commits into
Conversation
Contributor
|
🎯 Code Coverage (details) 🔗 Commit SHA: 3716594 | Docs | Datadog PR Page | Give us feedback! |
There was a problem hiding this comment.
Pull request overview
This PR updates Feature Flagging exposure delivery to support agentless deployments by selecting an appropriate transport (local EVP proxy when available, with safe direct-intake fallback when necessary), and adds supporting test coverage across feature-flagging and communication modules.
Changes:
- Introduce an exposure-specific backend selection layer (
ExposureBackendApiFactory) and an agentless fallback transport (AgentlessExposureBackendApi) to switch from local EVP proxy to direct intake on definitive rejections. - Start exposure delivery earlier in agentless mode (before application provider activation) while keeping the configuration source lazy until activation.
- Improve HTTP failure observability for EVP proxy calls by surfacing status codes via
HttpResponseException, and add targeted tests.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| products/feature-flagging/feature-flagging-lib/src/test/java/com/datadog/featureflag/ExposureWriterTests.java | Adds coverage for agentless direct-intake exposure delivery (API key header + direct endpoint). |
| products/feature-flagging/feature-flagging-lib/src/test/java/com/datadog/featureflag/ExposureBackendApiFactoryTest.java | New unit tests for backend selection rules (remote-config vs agentless, local vs direct, invalid direct URL). |
| products/feature-flagging/feature-flagging-lib/src/test/java/com/datadog/featureflag/AgentlessExposureBackendApiTest.java | New unit tests validating fallback and replay behavior for definitive vs ambiguous failures. |
| products/feature-flagging/feature-flagging-lib/src/main/java/com/datadog/featureflag/ExposureWriterImpl.java | Switches exposure writer to use ExposureBackendApiFactory rather than the generic BackendApiFactory selection. |
| products/feature-flagging/feature-flagging-lib/src/main/java/com/datadog/featureflag/ExposureBackendApiFactory.java | New transport selection logic for exposures, including agentless direct-intake fallback (when API key exists). |
| products/feature-flagging/feature-flagging-lib/src/main/java/com/datadog/featureflag/AgentlessExposureBackendApi.java | New BackendApi wrapper that switches from local EVP to direct intake on definitive rejections. |
| products/feature-flagging/feature-flagging-lib/build.gradle.kts | Adds config module dependency for compile-time constants (and test dependency for new tests). |
| products/feature-flagging/feature-flagging-agent/src/test/java/com/datadog/featureflag/FeatureFlaggingSystemTest.java | Updates system test expectations for earlier exposure writer initialization in agentless mode. |
| products/feature-flagging/feature-flagging-agent/src/main/java/com/datadog/featureflag/FeatureFlaggingSystem.java | Initializes exposure writer pre-activation for agentless mode; adds state inspection helpers used by tests. |
| communication/src/test/java/datadog/communication/EvpProxyApiTest.java | New test ensuring non-2xx responses surface status codes and preserve expected request shape. |
| communication/src/main/java/datadog/communication/HttpResponseException.java | New exception type carrying HTTP status codes for failed requests. |
| communication/src/main/java/datadog/communication/EvpProxyApi.java | Throws HttpResponseException (instead of generic IOException) on non-success responses. |
| communication/src/main/java/datadog/communication/BackendApiFactory.java | Refactors backend creation into separate local-EVP vs direct-intake factory methods. |
Suppressed comments (1)
products/feature-flagging/feature-flagging-lib/src/main/java/com/datadog/featureflag/ExposureWriterImpl.java:143
ExposureBackendApiFactory#create()is explicitly nullable when no local EVP proxy or direct intake credentials are available, but the serializer thread treats this as an exceptional state and throws an uncaughtIllegalArgumentException. This will surface as a background-thread crash even though the factory already logs that delivery is disabled. Prefer a clean shutdown (log + return) aftererrorCallback.run()and use an error message that matches the broader set of failure reasons (not only EVP proxy).
evp = backendApiFactory.create();
if (evp == null) {
errorCallback.run();
throw new IllegalArgumentException("EVP Proxy not available");
}
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Contributor
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
leoromanovsky
marked this pull request as ready for review
August 12, 2026 01:01
leoromanovsky
requested review from
btthomas,
mhlidd,
sameerank,
sarahchen6 and
typotter
and removed request for
a team and
typotter
August 12, 2026 01:01
Contributor
There was a problem hiding this comment.
Early local-EVP discovery can permanently disable exposure delivery without an API key: the writer exits before provider activation, yet activation reuses that dead writer even after the local route becomes available.
📊 Validated against 10 scenarios · Open Bits AI session
🤖 Datadog Autotest · Commit 3716594 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
Comment on lines
+89
to
+94
| if (exposureWriter == null) { | ||
| final ExposureWriter newExposureWriter = new ExposureWriterImpl(sco, config); | ||
| initialize(configService, newExposureWriter); | ||
| } else { | ||
| initializeConfigurationSource(configService, exposureWriter); | ||
| } |
Contributor
There was a problem hiding this comment.
Restart a writer that failed before activation
Affected agentless processes silently lose every feature-flag exposure for their lifetime.
Assertion details
- Input: Agentless Feature Flags without an API key; local EVP discovery initially misses the proxy, the writer exits, and the proxy becomes available before provider activation.
- Expected:
Provider activation should replace a terminated early writer and rerun route selection, allowing a newly available local EVP proxy to receive exposures. - Actual:
The early writer terminated after the initial capability miss. Provider activation then reused that terminated writer, performing no second capability check.
Suggested change
| if (exposureWriter == null) { | |
| final ExposureWriter newExposureWriter = new ExposureWriterImpl(sco, config); | |
| initialize(configService, newExposureWriter); | |
| } else { | |
| initializeConfigurationSource(configService, exposureWriter); | |
| } | |
| if (exposureWriter == null | |
| || (exposureWriter instanceof ExposureWriterImpl | |
| && !((ExposureWriterImpl) exposureWriter).isSerializerThreadAlive())) { | |
| if (exposureWriter != null) { | |
| exposureWriter.close(); | |
| } | |
| final ExposureWriter newExposureWriter = new ExposureWriterImpl(sco, config); | |
| initialize(configService, newExposureWriter); | |
| } else { | |
| initializeConfigurationSource(configService, exposureWriter); | |
| } |
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Java can evaluate Feature Flags from the managed CDN without a local Datadog receiver. The current writer drops exposures when no local EVP proxy exists. This loses experimentation data in the default agentless deployment.
flowchart TD E[Agentless flag evaluation] --> I[GET /info on the standard tracer Agent URL] I --> R{EVP proxy available?} R -->|Yes| L[Send through the Agent or serverless-init] R -->|No| D[Send directly to EVP] L --> P[Event Platform] D --> PFeature Flags use normal Agent discovery. Direct delivery requires an API key. A definitive local rejection can also select direct delivery.
Changes and Decisions
This PR adds support for direct EVP exposure delivery in agentless Feature Flags deployments without a compatible local EVP route.
serverless-init.serverless-init. This change does not send them directly.Validation
The shared exposure contract is enabled in DataDog/system-tests#7494.
9992388df3with the Spring Boot weblog.serverless-init:1.9.13routes each passed.