test trigger#1752
Conversation
radu-mocanu
left a comment
There was a problem hiding this comment.
Golden-rule review
Reviewed the diff against the golden rules. The change adds tracing spans around eval execution and a new --trace-file CLI option exporting trace data to a local file. These are logic changes within the CLI eval layer. No layering, env-var, surface, typing, caching, fail-loud, or test-hygiene violations were found. The only golden-rule concern is GR-3: this is a logic-changing PR but no package version bump is visible in the diff.
1 golden-rule violation(s) found:
- [GR-3]
src/uipath/_cli/cli_eval.py:105: Logic change without a visible version bump- This PR changes behavior (adds tracing spans and a new --trace-file exporter) but no package version bump is present in the diff. GR-3 requires every logic-changing PR to bump the version so unrelated changes do not ride your version.
- Fix: Bump the package version (patch, since this is additive/non-breaking) and run uv sync to update uv.lock.
Automated review against the repo's golden rules. Reply if a rule is misapplied.
There was a problem hiding this comment.
Pull request overview
This PR enhances evaluation observability by (1) adding an option to export traces locally during uipath eval runs and (2) adding additional OpenTelemetry spans around key evaluation phases.
Changes:
- Add
--trace-fileoption touipath evalto export trace spans to a local JSON Lines file. - Add parent/child spans for “Evaluation Set Run”, per-“Evaluation”, and per-“Evaluator” execution within the eval runtime.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
src/uipath/_cli/cli_eval.py |
Adds a new CLI option and wires a JSONL span exporter for local trace export. |
src/uipath/_cli/_evals/_runtime.py |
Introduces additional OpenTelemetry spans around eval set execution, individual eval items, and evaluator runs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @click.option( | ||
| "--trace-file", | ||
| required=False, | ||
| type=click.Path(exists=False), | ||
| help="File path where the trace data will be exported for local testing", | ||
| ) |
| if ctx.job_id: | ||
| trace_manager.add_span_exporter(LlmOpsHttpExporter()) | ||
|
|
||
| if trace_file: | ||
| trace_manager.add_span_exporter(JsonLinesFileExporter(trace_file)) |
| ( | ||
| evaluation_set, | ||
| evaluators, | ||
| evaluation_iterable, | ||
| ) = await self.initiate_evaluation(runtime) |
| with tracer.start_as_current_span( | ||
| "Evaluation Set Run", | ||
| attributes=span_attributes | ||
| ): |
| with tracer.start_as_current_span( | ||
| "Evaluation", | ||
| attributes={ | ||
| "execution.id": execution_id, | ||
| "span_type": "evaluation", | ||
| "eval_item_id": eval_item.id, | ||
| "eval_item_name": eval_item.name, | ||
| } | ||
| ): |
| with tracer.start_as_current_span( | ||
| f"Evaluator: {evaluator.name}", | ||
| attributes={ | ||
| "span_type": "evaluator", | ||
| "evaluator_id": evaluator.id, | ||
| "evaluator_name": evaluator.name, | ||
| "eval_item_id": eval_item.id, | ||
| } | ||
| ): |
| agent_execution=agent_execution, | ||
| evaluation_criteria=evaluation_criteria, | ||
| ) | ||
|
|
||
| return result |
No description provided.