fix(conformance): improve OpenAI API conformance score from 89.1% to 91.2%#5490
Merged
leseb merged 2 commits intollamastack:mainfrom Apr 9, 2026
Merged
Conversation
The OpenAI spec includes type: object on 766 of 772 schemas that have properties defined. Previously we stripped it from all schemas, which hurt conformance across Chat, Embeddings, Vector stores, and other categories. This change narrows the strip to only the 6 schemas that actually omit it in the OpenAI spec. Overall conformance: 89.3% -> 91.0% Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
Contributor
✱ Stainless preview buildsThis PR will update the
|
…spec The _convert_standalone_const_to_enum transform strips all defaults from single-value enums, but a handful of OpenAI schemas (Conversations, Responses, Compact, Chat list) include explicit defaults on their object fields. Add _restore_const_enum_defaults to selectively re-add defaults on the 5 schemas that need them, fixing 6 conformance issues across Responses (+0.9%), Chat (+0.2%), Conversations (+0.1%), and Embeddings categories. Overall conformance improves from 91.0% to 91.2%. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
Collaborator
Author
|
@cdoern the reason the conformance went down in the compaction PR is because it's a api under I have another PR incoming that will do a broader change to handle some of the other type issues observed by the entire |
4 tasks
Collaborator
Author
leseb
approved these changes
Apr 9, 2026
leseb
added a commit
to leseb/llama-stack
that referenced
this pull request
Apr 9, 2026
Resolve conflicts in generated spec files by regenerating them with our changes (type: object retention and files_batch enum fix) applied on top of the merged conformance improvements. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Sébastien Han <seb@redhat.com>
github-merge-queue Bot
pushed a commit
that referenced
this pull request
Apr 9, 2026
…AI spec (#5494) ## Summary - Remove `null` from `anyOf` JSON schemas for 8 response body fields (`frequency_penalty`, `parallel_tool_calls`, `presence_penalty`, `temperature`, `top_p`, `top_logprobs`, `tools`, `service_tier`) and 3 request body fields (`store`, `stream`, `include`) to match OpenAI's non-nullable spec - Python types remain `T | None` for runtime compatibility — only the generated JSON schema output changes - Responses conformance score improves from 85.8% → 88.9% (7 schema issues resolved), overall 91.2% → 91.4% Builds on #5490, which improved overall conformance from 89.1% to 91.2%. ## BREAKING CHANGE Request properties `store`, `stream`, and `include` on `POST /v1/responses` no longer accept explicit `null` in the JSON schema. Clients should omit these fields instead of sending `null`. This aligns with the OpenAI spec behavior. ## Test plan - [x] `uv run pytest tests/unit/ -k "response" --ignore=tests/unit/providers/vector_io/` — 360 passed - [x] `uv run pytest tests/unit/ -k "openai or schema or spec"` — 431 passed - [x] `uv run pre-commit run --all-files` — all hooks pass - [x] `uv run python scripts/openai_coverage.py --update` — conformance score verified 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Francisco Javier Arceo <farceo@redhat.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two targeted fixes to the OpenAPI schema generation transforms that improve overall OpenAI API conformance from 89.1% to 91.2% (+2.1%) and reduce total issues from 374 to 303 (-71).
1. Keep
type: objecton schemas that include it in the OpenAI specPreviously,
_remove_type_object_from_openai_schemasstrippedtype: objectfrom all schemas withproperties. But 766 of 772 OpenAI schemas includetype: object— only 6 omit it. Changed to a targeted exemption list so only those 6 schemas havetype: objectremoved.2. Restore
defaulton single-value enum fields where OpenAI expects them_convert_standalone_const_to_enumstrips all defaults from single-value enums. A handful of OpenAI schemas (CompactResource,ConversationResource,Response,ChatCompletionList) include explicit defaults on theirobjectfields. Added_restore_const_enum_defaultsto selectively re-add them for the 5 affected schemas.Conformance delta (main vs this PR)
Compact endpoint (/responses/compact)
Test plan
uv run pre-commit run --all-files-- all hooks pass./scripts/unit-tests.sh-- all tests pass