Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: openai/openai-agents-python
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: main
Choose a base ref
...
head repository: codeactual/openai-agents-python
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref
Checking mergeability… Don’t worry, you can still create the pull request.
  • 3 commits
  • 28 files changed
  • 1 contributor

Commits on Apr 23, 2026

  1. Add skill-eval harness PoC for Claude Code Skills 

    Evaluate Claude Code Skills (SKILL.md with YAML frontmatter + markdown body)
by loading them into an Agent's system prompt and running a JSONL suite of
test cases through Runner.run. Scorers combine programmatic assertions
(contains/regex/jsonpath/equals/not_contains) with an optional LLM judge
that mirrors the llm_as_a_judge example pattern.

The harness lives under examples/skill_evals/ as a self-contained example
so it can iterate without touching src/agents/. Each case runs inside its
own trace(...) span so traces map one-to-one with cases. A compare()
helper diffs two reports to surface regressions when iterating on a skill
version.

Adds pyyaml to the dev dependency group for frontmatter parsing; ships a
json-summarizer sample skill plus unit + integration tests (using FakeModel)
so the PoC is verifiable offline.
    @claude
    claude committed Apr 23, 2026
    Configuration menu
    Copy the full SHA
    96f723c View commit details
    Browse the repository at this point in the history

  2. Add skill evals user guide to docs 

    Document the examples/skill_evals/ harness under the Documentation section
of the MkDocs site and link to it from the Examples index. Covers skill
layout, CLI usage, test-case format (with the assertion-kind table), the
default scorer stack, tracing, variant comparison, Python API, and
explicit non-goals for the PoC.
    @claude
    claude committed Apr 23, 2026
    Configuration menu
    Copy the full SHA
    e242fd4 View commit details
    Browse the repository at this point in the history

  3. Generate missing API reference stubs 

    Auto-generated by docs/scripts/generate_ref_files.py for source modules
that didn't yet have mkdocstrings stubs: mongodb_session, the new
sandbox/session ops modules (archive/manifest/mount/snapshot/tar), and
the sandbox box mount provider.
    @claude
    claude committed Apr 23, 2026
    Configuration menu
    Copy the full SHA
    3ce4543 View commit details
    Browse the repository at this point in the history
Loading