Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: UiPath/coder_eval
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 8dcd35f
Choose a base ref
...
head repository: UiPath/coder_eval
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: cff8737
Choose a head ref
  • 4 commits
  • 18 files changed
  • 2 contributors

Commits on Aug 4, 2026

  1. feat(agents): add OpenHands Software Agent SDK agent (openhands) 

    Add a new in-tree `openhands` agent (openhands-sdk + openhands-tools 1.40.0,
optional [openhands] extra) so coder_eval can evaluate the OpenHands harness
against any model reachable via its bundled LiteLLM — the provider is resolved
from the agent.model prefix (anthropic/ openai/ openrouter/ bedrock/
litellm_proxy/), making OpenHands the model-agnostic "universal harness" for
isolate-the-model comparisons. Registers via the plugin SPI (no enum dispatch),
emits the standard event protocol through EventCollector, and drives the
synchronous Conversation.run() off-thread with a watchdog->pause() timeout and a
cooperative early-stop seam.

- models: AgentKind.OPENHANDS + OpenHandsAgentConfig (discriminated-union member,
  exported from coder_eval.models); settings.openhands_model.
- pricing: _normalize_model strips litellm_proxy/ openrouter/ openai/ anthropic/
  prefixes so provider-prefixed OSS ids key the existing rate card.
- sandbox: OPENAI_API_KEY / OPENROUTER_API_KEY / OPENHANDS_BASE_URL /
  OPENHANDS_MODEL added to the default env_passthrough allowlist.
- agent: OpenHandsAgent + _OpenHandsTurnState (event mapping, token/cost mapping
  with cache-inclusive prompt handling, crash/timeout funnels, off-thread worker
  drain on cancel to avoid a main/worker state race, idempotent teardown).
- tests: tests/test_openhands_agent.py (SDK mocked / importorskip).
- docs/experiment: docs/agents/OPENHANDS.md + experiments/harness-comparison.yaml
  + mkdocs nav/blurb (README/index/llms.txt regenerated via make docs-indexes).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
    @CarlesUIPath @claude
    CarlesUIPath and claude committed Aug 4, 2026
    Configuration menu
    Copy the full SHA
    1c78cbb View commit details
    Browse the repository at this point in the history

  2. chore(harness): defer CE034 plan-phase-label lint candidate 

    Surfaced during the openhands-agent review — the no-plan-phase-refs-in-code
MEMORY rule is unenforced and leaked "Phase N" labels this run (caught by
review, reworded by hand). Deferred: needs a whole-tree comment/string scan with
an exemption for litellm_cost.py's legitimate algorithm-stage usage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
    @CarlesUIPath @claude
    CarlesUIPath and claude committed Aug 4, 2026
    Configuration menu
    Copy the full SHA
    df7d175 View commit details
    Browse the repository at this point in the history

  3. fix(agents): code-review fixes for the openhands agent 

    Final cross-cutting review findings:
- HIGH: terminal-status classification was a denylist (only ERROR/STUCK crashed),
  so a non-FINISHED status (WAITING_FOR_CONFIRMATION, an unexpected IDLE, or a
  PAUSED we did not cause) silently scored as COMPLETED. Switch to an ALLOWLIST:
  clean iff FINISHED, or PAUSED that WE caused (timeout/early-stop); everything
  else crashes. Covered by a parametrized STUCK/WAITING_FOR_CONFIRMATION/IDLE/PAUSED test.
- MEDIUM: doc CLI example pointed at a non-existent task; point at tasks/hello_date.yaml.
- MEDIUM: document max_turns-exhaustion-as-crash (SDK collapses cap-hit into ERROR)
  as a known limitation.
- LOW: correct the close()-ownership comment (kill_sync may also close from the
  watchdog thread; safe because idempotent + delete_on_close=False).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
    @CarlesUIPath @claude
    CarlesUIPath and claude committed Aug 4, 2026
    Configuration menu
    Copy the full SHA
    03d30fb View commit details
    Browse the repository at this point in the history

  4. feat(openhands): direct-only OpenRouter with real-cost recovery, drop… 

    … proxy, fix tool telemetry

- Direct path recovers REAL OpenRouter cost via litellm_extra_body usage.include
  (was falling back to a ~$0 estimate); pin compliant providers per model via
  _OPENROUTER_PROVIDER_ROUTING.
- Remove the LiteLLM-proxy path entirely — OpenHands reaches every provider
  natively and retrieves real cost in-agent, so the sidecar was dead weight:
  drop _resolve_base_url / OPENHANDS_BASE_URL, the cost_log_tags ctor param +
  supports_cost_log_tags override, and the base_url/extra_headers LLM args;
  fail fast on litellm_proxy/* model ids. __init__ now matches Codex's shape.
- Fix tool telemetry: join ObservationEvent to its ActionEvent on tool_call_id,
  not action_id (which is the ActionEvent's event id). Previously every tool
  orphaned as result_status="unknown", losing tool results and rendering as
  errors on the dashboard.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
    @CarlesUIPath @claude
    CarlesUIPath and claude committed Aug 4, 2026
    Configuration menu
    Copy the full SHA
    cff8737 View commit details
    Browse the repository at this point in the history
Loading