Skip to content

Tag evals with the tracer version and ml app - #12184

Draft
ddog-thibault-nadin wants to merge 1 commit into
thibault/llmobs-submit-feedbackfrom
thibault/llmobs-eval-default-tags
Draft

Tag evals with the tracer version and ml app#12184
ddog-thibault-nadin wants to merge 1 commit into
thibault/llmobs-submit-feedbackfrom
thibault/llmobs-eval-default-tags

Conversation

@ddog-thibault-nadin

Copy link
Copy Markdown

What does this PR do?

Adds the ddtrace.version and ml_app tags to every submitted evaluation metric, so an eval can be filtered by tag and not only through the top level ml_app payload field.

Motivation

Spotted while reviewing #12129: feedback events carry those two tags but evals never did, making the Java tracer the odd one out.

  • dd-trace-js builds evaluation and feedback tags with the same buildMetricTags(tags, mlApp, kind) (packages/dd-trace/src/llmobs/eval-metric.js), which always prefixes ddtrace.version and ml_app.
  • dd-trace-py starts both evaluation_tags and feedback_tags from {"ddtrace.version": __version__, "ml_app": ml_app} (ddtrace/llmobs/_eval_metric.py).

Stacked on top of #12129, whose shared IntakeTags.flatten helper this reuses. Review that one first.

Wire change

Purely additive, but note the tags field is now always present in the eval payload — it used to be omitted entirely when the caller passed no tags.

 {
   "event_kind": "evaluation",
   "ml_app": "my-app",
-  "tags": ["source:web-ui"]
+  "tags": ["ddtrace.version:1.2.3", "ml_app:my-app", "source:web-ui"]
 }

Testing

LLMObsEvalTest, which pins the payload key set, is updated to cover both cases: user supplied tags, and none.

@ddog-thibault-nadin ddog-thibault-nadin added type: bug fix Bug fix comp: mlobs ML Observability (LLMObs) tag: ai generated Largely based on code generated by an AI or LLM labels Aug 11, 2026
@datadog-datadog-prod-us1-2

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.01 s 13.98 s [-0.4%; +0.9%] (no difference)
startup:insecure-bank:tracing:Agent 12.93 s 13.04 s [-1.8%; +0.2%] (no difference)
startup:petclinic:appsec:Agent 17.48 s 17.40 s [-0.6%; +1.5%] (no difference)
startup:petclinic:iast:Agent 17.37 s 17.47 s [-1.4%; +0.2%] (no difference)
startup:petclinic:profiling:Agent 16.62 s 17.37 s [-8.6%; -0.0%] (maybe better)
startup:petclinic:sca:Agent 17.40 s 16.90 s [-1.5%; +7.5%] (no difference)
startup:petclinic:tracing:Agent 16.43 s 16.23 s [-3.1%; +5.6%] (no difference)

Commit: e659809f · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: mlobs ML Observability (LLMObs) tag: ai generated Largely based on code generated by an AI or LLM type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant