Skip to content

Commit c253bdf

Browse files
zdenekmusil-gdzdenekmusil-gd
committed
feat(eval): multi-model comparison, gd-eval models command, Langfuse enhancements
Multi-model comparison (--model repeated): - gd-eval run --model gpt-5.2 --model gpt-4o evaluates the full dataset against each model sequentially and prints a side-by-side comparison table with winner. --model accepts provider/model syntax to disambiguate (e.g. --model Foundry4o/gpt-5.2 or --model HN_Anthropic/claude-opus-4-7). - --provider flag removed — superseded by provider/model syntax; ambiguous-model error now suggests the syntax directly. - Workspace original active model restored in finally block after all models run. - JSON report always uses nested {models, runs, comparison} shape. - Langfuse: one named dataset run per model (gd-eval-{ts}-{model_id}). New command: gd-eval models - Lists all LLM providers and their models with provider ID, family, and active marker. Marks the workspace's currently active model when --workspace given. Provider type labelling: - Always fetches model family live via list_llm_provider_models_by_id() — automatically correct for new families without code changes. - Combines gateway + family: OPENAI→family only, AWS_BEDROCK→BEDROCK/family, AZURE_FOUNDRY→AZURE/family (e.g. BEDROCK/ANTHROPIC, AZURE/OPENAI). - Model and provider_type in Langfuse trace metadata, tags, and dataset-run-item runDescription + metadata. EvalReport gains provider_name and provider_type; comparison table shows provider/model to distinguish runs with the same model id across providers. 111 tests, ruff + ty clean. JIRA: GDAI-1831 Risk: low — new isolated features; no changes to existing packages.
1 parent c028c31 commit c253bdf

11 files changed

Lines changed: 855 additions & 134 deletions

File tree

‎packages/gooddata-eval/README.md‎

Lines changed: 108 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# gooddata-eval
22

33
CLI to evaluate the GoodData AI agent against a dataset of natural-language
4-
questions on a chosen workspace and LLM model.
4+
questions on a chosen workspace and LLM model — including multi-model comparison.
55

66
## Install
77

@@ -11,68 +11,145 @@ Or install `gd-eval` as a standalone tool:
1111

1212
uv tool install gooddata-eval
1313

14-
## Quick start
14+
## Commands
15+
16+
| Command | Description |
17+
|---|---|
18+
| `gd-eval run` | Run an evaluation dataset against one or more models. |
19+
| `gd-eval models` | List LLM providers and models configured in the org. |
20+
21+
---
22+
23+
## `gd-eval run`
24+
25+
### Quick start — single model
1526

1627
```bash
1728
export GOODDATA_TOKEN='your-api-token'
1829

1930
gd-eval run \
2031
--host https://your.gooddata.cloud \
21-
--workspace demo \
32+
--workspace ecommerce_demo \
2233
--dataset ./my-dataset \
2334
--model gpt-5.2 \
24-
--runs 2 \
35+
--runs 1 \
2536
--json results.json
2637
```
2738

28-
## All flags
39+
### Multi-model comparison
40+
41+
Pass `--model` multiple times to evaluate the same dataset against several
42+
models and get a side-by-side comparison:
43+
44+
```bash
45+
gd-eval run \
46+
--host https://your.gooddata.cloud \
47+
--workspace ecommerce_demo \
48+
--dataset ./my-dataset \
49+
--model gpt-5.2 \
50+
--model claude-opus-4-7 \
51+
--runs 1 \
52+
--json comparison.json
53+
```
54+
55+
When the same model id is offered by multiple providers, use the
56+
`provider/model` syntax to disambiguate:
57+
58+
```bash
59+
--model "Foundry4o_4.1_5.2/gpt-5.2" \
60+
--model "HN_Anthropic/claude-opus-4-7"
61+
```
2962

30-
### Connection
63+
Both provider name and provider id are accepted as the prefix.
64+
65+
### All flags
66+
67+
#### Connection
3168

3269
| Flag | Env var | Description |
3370
|---|---|---|
34-
| `--host HOST` | — | GoodData host URL (e.g. `https://your.gooddata.cloud`). |
71+
| `--host HOST` | — | GoodData host URL. |
3572
| `--token TOKEN` | `GOODDATA_TOKEN` | API token. Pass via flag or env var. |
36-
| `--profile NAME` | — | Profile name in `~/.gooddata/profiles.yaml` (same file as the `gdc` CLI). Provides host + token when both flags are omitted. |
73+
| `--profile NAME` | — | Profile name in `~/.gooddata/profiles.yaml` (same file as the `gdc` CLI). |
3774
| `--workspace ID` | — | **Required.** Workspace id to evaluate against. |
3875

39-
### Dataset source (pick one)
76+
#### Dataset source (pick one)
4077

4178
| Flag | Description |
4279
|---|---|
43-
| `--dataset PATH` | Path to a flat folder of JSON files — one question per file. |
44-
| `--langfuse-dataset NAME` | Pull dataset items by name from Langfuse. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST` env vars. |
80+
| `--dataset PATH` | Flat folder of JSON files — one question per file. |
81+
| `--langfuse-dataset NAME` | Pull items by name from a Langfuse dataset. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
4582

46-
### Model selection
83+
#### Model selection
4784

4885
| Flag | Description |
4986
|---|---|
50-
| `--model ID` | LLM model id to evaluate (e.g. `gpt-5.2`). Defaults to the workspace's currently active model. If the model is offered by a different provider than the active one, the workspace's active provider is switched automatically. |
51-
| `--provider NAME_OR_ID` | LLM provider name or id. Use when `--model` is offered by multiple providers and you need to pick one. Accepts either the human-readable provider name or its UUID id. |
87+
| `--model MODEL` | Model id to evaluate. Repeat to compare multiple models. Accepts `provider/model` syntax to disambiguate when a model is offered by multiple providers (e.g. `--model "Foundry4o/gpt-5.2"`). Defaults to the workspace's current active model. |
5288

53-
### Evaluation
89+
#### Evaluation
5490

5591
| Flag | Default | Description |
5692
|---|---|---|
57-
| `--runs K` | `2` | Number of independent conversation runs per item (pass@K). An item passes if any run passes. |
93+
| `--runs K` | `2` | Independent runs per item (pass@K). An item passes if any run passes. |
5894

59-
### Output
95+
#### Output
6096

6197
| Flag | Description |
6298
|---|---|
63-
| `--json PATH` | Write a machine-readable JSON report (keyed by item id, with per-item scores) to this path. Console summary is always printed. |
64-
| `--quiet` | Suppress per-item progress output. Only the final table and summary are printed. |
99+
| `--json PATH` | Write a JSON report to this path. Always uses the nested `{models, runs, comparison}` shape even for a single model. |
100+
| `--quiet` | Suppress per-item progress. Per-model result tables and the comparison summary are still printed. |
65101

66-
### Langfuse sink
102+
#### Langfuse sink
67103

68104
| Flag | Description |
69105
|---|---|
70-
| `--langfuse` | Log evaluation results to Langfuse after each item. Requires `--langfuse-dataset` (so item ids can be linked to Langfuse dataset items). Creates a named experiment run (`gd-eval-{timestamp}-{model}`) in the Langfuse dataset. Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
106+
| `--langfuse` | Log scores and traces to Langfuse after each item. Requires `--langfuse-dataset`. Creates one named experiment run per model (`gd-eval-{timestamp}-{model}`). Requires `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`. |
107+
108+
### JSON report shape
109+
110+
The JSON report always uses the nested multi-model shape:
111+
112+
```json
113+
{
114+
"models": ["gpt-5.2", "claude-opus-4-7"],
115+
"runs": {
116+
"gpt-5.2": { "summary": { "passed": 22, ... }, "items": { ... } },
117+
"claude-opus-4-7": { "summary": { "passed": 18, ... }, "items": { ... } }
118+
},
119+
"comparison": {
120+
"gpt-5.2": { "passed": 22, "total": 31, "pass_rate": 0.71, "avg_quality_score": 0.81, ... },
121+
"claude-opus-4-7": { "passed": 18, "total": 31, "pass_rate": 0.58, "avg_quality_score": 0.72, ... }
122+
}
123+
}
124+
```
125+
126+
Winner is selected by **pass rate → quality score → latency** (lower latency wins all-equal ties).
127+
128+
---
129+
130+
## `gd-eval models`
131+
132+
List all LLM providers and their models in the org. Marks the active model
133+
for a workspace when `--workspace` is given:
134+
135+
```bash
136+
gd-eval models \
137+
--host https://your.gooddata.cloud \
138+
--workspace ecommerce_demo
139+
```
140+
141+
```
142+
┃ Provider ┃ Provider ID ┃ Model ID ┃ Family ┃ Active ┃
143+
│ Foundry4o │ foundry_… │ gpt-5.2 │ OPENAI │ ◀ active │
144+
│ │ │ gpt-4o │ OPENAI │ │
145+
│ HN_Anthropic │ hn_anthr_… │ claude-opus-4-7 │ ANTHROPIC │ │
146+
```
147+
148+
---
71149

72150
## Dataset format
73151

74-
A dataset is a folder of `.json` files, one per question. Each file must
75-
contain a common envelope:
152+
A dataset is a folder of `.json` files, one per question:
76153

77154
```json
78155
{
@@ -87,8 +164,6 @@ contain a common envelope:
87164
Supported `test_kind` values: `visualization`, `metric_skill`, `alert_skill`,
88165
`search_tool`, `general_question`, `guardrail`.
89166

90-
See the full dataset specification for `expected_output` shapes per test kind.
91-
92167
## Supported test kinds
93168

94169
| test_kind | What the agent must produce | Extra required |
@@ -104,31 +179,22 @@ See the full dataset specification for `expected_output` shapes per test kind.
104179

105180
### `[llm-judge]` — LLM-as-judge evaluators
106181

107-
`general_question` and `guardrail` items are scored by an LLM judge (GPT-4o)
108-
that compares the agent's text response against your expected-output description.
109-
This requires the OpenAI Python package and an API key:
182+
`general_question` and `guardrail` items are scored by a GPT-4o judge.
183+
Requires the OpenAI package and `OPENAI_API_KEY`:
110184

111185
```bash
112-
uv add 'gooddata-eval[llm-judge]' # project dependency
113-
# or, for the standalone gd-eval tool:
186+
uv add 'gooddata-eval[llm-judge]'
187+
# or for the standalone tool:
114188
uv tool install 'gooddata-eval[llm-judge]'
115189
```
116190

117-
Set your OpenAI key before running:
118-
119-
```bash
120-
export OPENAI_API_KEY='sk-...'
121-
```
122-
123-
Without `[llm-judge]`, items with `test_kind: general_question` or `guardrail`
124-
are reported as **skipped**.
125-
191+
Without `[llm-judge]`, those items are **skipped**.
126192

127193
## Exit codes
128194

129195
| Code | Meaning |
130196
|---|---|
131-
| `0` | Run completed. Evaluation failures do **not** cause a non-zero exit — check the report. |
197+
| `0` | Run completed. Evaluation failures do **not** cause a non-zero exit. |
132198
| `2` | Operational error: bad connection, missing model, unreadable dataset, missing credentials. |
133199

134200
## Scores (in JSON report and Langfuse)
@@ -137,5 +203,6 @@ are reported as **skipped**.
137203
|---|---|
138204
| `pass_at_k` | 1 if any of the K runs passed strict checks, else 0. |
139205
| `quality_score` | Fraction of strict check flags that are `True` (0.0–1.0). Shown in CLI as a percentage. |
140-
| `value_score` | Weighted blend: 0.6 × quality + 0.2 × speed (where speed = max(0, 1 − latency/60s)). |
206+
| `value_score` | Weighted blend: 0.6 × quality + 0.2 × speed (speed = max(0, 1 − latency/60s)). |
141207
| `latency_s` | Average per-run latency in seconds. |
208+
| `provider_type` | Model vendor + gateway label (e.g. `ANTHROPIC`, `BEDROCK/ANTHROPIC`, `AZURE/OPENAI`). Stored in Langfuse trace metadata and tags. |

0 commit comments

Comments
 (0)