Author, run, and analyze Coder Eval suites from inside Claude Code — including whether your own Claude Code skills actually trigger.
Coder Eval runs a real coding agent in a sandbox against declarative YAML tasks and scores the files and commands it actually produced. This plugin puts the authoring, running, and analysis loop behind six slash commands.
/plugin marketplace add UiPath/coder_eval
/plugin install coder-eval@coder-eval
The first command registers this repository as a plugin marketplace; the second installs
the plugin from it. (Marketplace and plugin share the name coder-eval, hence the
coder-eval@coder-eval.)
The skills drive the coder-eval CLI, which is not bundled with the plugin. Install it
once:
uv tool install coder-eval # or: pip install coder-evalYou do not have to do this in advance. init, task and skill-check check
coder-eval --version before doing any work and, if it is missing, offer to install it and
ask first — they never install unprompted, and they verify it worked before continuing.
Running a suite also needs credentials for whichever agent the tasks use (e.g.
ANTHROPIC_API_KEY for the default claude-code agent). lint-tasks needs neither the CLI
nor credentials — it only reads files.
| Command | What it does |
|---|---|
/coder-eval:init |
Scans the repo for what is worth evaluating (skills, an MCP server, a CLI), then scaffolds a task directory with one real task. |
/coder-eval:skill-check |
Generates and runs an activation suite for one of your skills — does the agent engage it when it should, and leave it alone when it shouldn't? |
/coder-eval:task |
Turns a natural-language description into a task YAML with the right success criteria. |
/coder-eval:lint-tasks |
Reviews task YAML you already have and reports criteria that cannot fail, prompts that give away the answer, and fixtures with no cleanup. Read-only. |
/coder-eval:analyze |
Reads a finished run directory and reports systemic failure patterns, per-task findings, and concrete fixes. |
/coder-eval:ci |
Emits a GitHub Actions workflow that runs your suite as a CI gate (or on a schedule, to catch skill drift). |
init and ci are explicit-invocation only. skill-check, task, lint-tasks, and
analyze can also be reached by the agent on its own when a request clearly calls for them.
reference/ travels with the plugin so the skills work with no access to this repository:
criteria.md— every success-criterion type and its fields, generated from theSuccessCriterionmodel union (regenerated bymake plugin-reference; do not hand-edit).task-rubric.md— the adversarial task-quality checklist ("could this pass for the wrong reason?", fixture lifecycle, scope match) thattaskandlint-tasksboth apply.cli-setup.md— the missing-CLI policy the CLI-driving skills follow: offer, ask, verify.run-layout.md— the on-disk run-directory contract thatanalyzereads: what is inside a run directory.repo-layout.md— how every skill finds where your eval tree is: discovered by content (task_id:files,run.json), never assumed to betasks/andruns/.templates/— the canonical activation suiteskill-checkcopies into your repo.
- Documentation: https://coder-eval.com/docs
- Source and issues: https://github.com/UiPath/coder_eval
- License: Apache-2.0