Skip to content

feat(validate): new subcommand wrapping kin-openapi's Validate()#894

Draft
reuvenharrison wants to merge 23 commits into
mainfrom
feat/validate-command
Draft

feat(validate): new subcommand wrapping kin-openapi's Validate()#894
reuvenharrison wants to merge 23 commits into
mainfrom
feat/validate-command

Conversation

@reuvenharrison
Copy link
Copy Markdown
Collaborator

DO NOT MERGE — depends on getkin/kin-openapi#1166 landing and a kin release shipping. The go.mod replace directive currently pins kin to the fork's rfc branch via a pseudo-version. Cleanup before merge: drop the replace, bump kin to the released version, go mod tidy.

What this adds

A new oasdiff validate <spec> subcommand that flags per-RFC OpenAPI / JSON Schema spec violations.

Wraps kin-openapi's openapi3.T.Validate() walker and dispatches each typed error to a stable kebab-case rule ID via errors.As against kin's *RequiredFieldError / *FieldVersionMismatchError clusters (introduced in kin#1166). Both clusters carry the offending field path, which is normalised into the canonical rule ID (e.g. info.versioninfo-version-required, prefixItemsprefix-items-field-for-3-1-plus, $dynamicAnchordynamic-anchor-field-for-3-1-plus).

Sites kin hasn't migrated to a typed cluster yet fall through to a generic kin-validation-error rule ID. As more sites get migrated upstream, more findings get specific IDs automatically.

Output

Two formats:

  • -f text (default): header summary line plus per-finding block matching the changelog command's ApiChange.MultiLineError shape, so users can grep across both:
    1 findings: 1 error, 0 warning, 0 info
    error	[info-version-required] at spec.yaml
    	invalid info: value of version must be a non-empty string
    
  • -f yaml: structured records with the changelog's field names (id, text, level, source):
    - id: info-version-required
      text: 'invalid info: value of version must be a non-empty string'
      level: 3
      source: spec.yaml

Exit code: 0 on no findings, 1 if any finding is reported.

Tests

10 tests cover:

  • Happy path (valid spec → exit 0, no output)
  • Typed dispatch through five concrete cases:
    • info-version-required (RequiredFieldError, single-level wrap)
    • openapi-required (RequiredFieldError, top-level)
    • identifier-field-for-3-1-plus (FieldVersionMismatchError via openapi3 → info → license chain)
    • webhooks-field-for-3-1-plus (FieldVersionMismatchError, top-level)
    • const-field-for-3-1-plus (FieldVersionMismatchError through 3 levels of %w wrapping: paths → path → operation → schema)
  • Untyped fallback (kin-validation-error for sites kin hasn't migrated yet)
  • Text format header + block shape
  • YAML format round-trip through yaml.Unmarshal
  • Load-failure exit code (102) distinct from validation-finding exit code (1)

Why a separate subcommand

oasdiff diff / breaking / changelog stay forgiving — they silently skip invalid input today and continue producing useful output. The new validate is opt-in and surfaces exactly the violations the others quietly route around. CI users who want pre-flight strictness add a validate step; everyone else is unaffected.

Scoping rule

Only flags failures the OpenAPI or referenced JSON Schema spec explicitly declares invalid. No style preferences, no config file, no opinion-driven rules — that ground belongs to Spectral/Redocly. The strict scoping keeps the surface small and the UX uncontroversial.

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 7, 2026

Codecov Report

❌ Patch coverage is 65.81470% with 107 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.17%. Comparing base (effffd7) to head (9459be7).

Files with missing lines Patch % Lines
internal/validate.go 66.12% 83 Missing and 22 partials ⚠️
checker/colorize.go 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #894      +/-   ##
==========================================
- Coverage   90.65%   90.17%   -0.48%     
==========================================
  Files         267      268       +1     
  Lines       16067    16380     +313     
==========================================
+ Hits        14565    14771     +206     
- Misses        962     1047      +85     
- Partials      540      562      +22     
Flag Coverage Δ
unittests 90.17% <65.81%> (-0.48%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Adds 'oasdiff validate <spec>' that flags per-RFC OpenAPI / JSON
Schema spec violations. Wraps kin-openapi's openapi3.T.Validate()
walker and dispatches each typed error to a stable kebab-case rule
ID via errors.As against kin's RequiredFieldError /
FieldVersionMismatchError clusters (introduced in kin-openapi#1166).

Output format follows the changelog command's shape (id/text/level/
source) so a single CI script can parse both. Two formats: text
(default, with a header summary line + per-finding block matching
ApiChange.MultiLineError style) and yaml (-f yaml).

Exit code 0 on no findings, 1 if any.

10 tests cover: happy path, typed dispatch (info-version-required,
openapi-required, identifier-field-for-3-1-plus, webhooks-field-for-
3-1-plus, const-field-for-3-1-plus through 3 levels of %w wrapping),
untyped fallback (kin-validation-error for sites kin hasn't migrated
to a typed cluster yet), text + yaml format shapes, load-failure
exit code distinct from validation-finding exit code.

DO NOT MERGE - depends on kin-openapi#1166 landing and a kin release
shipping. The go.mod replace directive pins kin to the fork's rfc
branch.

Cleanup before merge:
  1. Remove the replace directive
  2. Bump kin-openapi to the released version
  3. go mod tidy
Loader now starts with IncludeOrigin=true. The Finding struct gains
Line/Column fields, populated from the kin cluster errors' Origin.Key
when available (info, license, server, schema). Document-root fields
(openapi, webhooks, jsonSchemaDialect) have nil Origin and emit
findings without line/column (yaml omitempty).

Text format renders <file>:<line>:<column> when origin is set, plain
<file> otherwise — matches the changelog command's location shape.

Three new tests pin: line/column populated for info-version-required,
both fields absent for openapi-required (doc-root), text format
includes :line:column when available.
kin errors like *SchemaError embed newlines in their Error() output
(Schema and Value dumps). Without indenting continuation lines, those
broke the finding's visual grouping in text format. Every \n in the
message now becomes \n\t so the whole finding stays under the same
tab indent.
Previous indent logic prefixed every \n with \t, including blank
lines — leaving stray tabs on otherwise-empty separator lines and a
trailing \t at the end of the message. Switch to a line-by-line
walk that skips blanks and trims trailing whitespace.
…olates-schema)

kin#1166 added a *SchemaValueError cluster wrapping the *SchemaError
returned by VisitJSON when a schema's example or default doesn't
satisfy its own constraints. Add a third dispatch arm so findings of
that shape get a specific rule ID derived from the cluster's
ValueKind: 'example' → 'example-violates-schema',
'default' → 'default-violates-schema'.

The cluster also carries the parameter/media-type/schema's Origin, so
Line+Column now populate for these findings as well.
All four typed-validation-error PRs now merged into upstream
getkin/kin-openapi:
- #1162 (openapi3conv 3.0→3.x canonicalization)
- #1166 (ValidationError framework)
- #1170 (long-tail RequiredFieldError leaves)
- #1180 (combined long-tail PR collapsing #1171-#1179)

Drop the replace directive; pin to upstream-master pseudo-version
e8145f8f4d2b (the #1180 merge commit). When getkin tags a release
containing this work, the require line will move from pseudo-version
to a stable v0.138.x.
@reuvenharrison reuvenharrison force-pushed the feat/validate-command branch from 7406391 to 0c3c169 Compare May 9, 2026 17:39
After bumping kin to upstream master post-#1180, eight cluster types
became available that the validate dispatch was still routing to the
'kin-validation-error' catchall. Wire each:

- *PathParametersError -> path-parameters-mismatch (static, since the
  cluster carries Path/Method/Missing not a single derivable field)
- *MutuallyExclusiveFieldsError -> <f1>-<f2>-mutually-exclusive
- *ForbiddenFieldError -> <field>-forbidden
- *ServerURLTemplateError -> server-url-template-invalid (static, the
  cluster carries only the offending URL)
- *EitherFieldRequiredError -> <f1>-or-<f2>-required
- *SchemaBothFormsExclusive -> <field>-both-forms-exclusive
- *ExactlyOneFieldError -> <f1>-or-<f2>-exactly-one
- *SingleEntryContentError -> <subject>-content-single-entry
- *WebhookNilError -> webhook-nil

keyOriginForKinError extended in the same way so line/column flow
through for clusters that carry Origin (all except WebhookNilError,
whose offending key sits on the document root that the loader
doesn't track per-key).

Updated the previously-flipping Test_ValidateCmd_UntypedKinErrorFallback
(server-url-mismatched-braces is now typed) and added a test for the
user-reported PathParametersError case.
reuvenharrison and others added 11 commits May 13, 2026 20:41
…tamps

Switches the kin-openapi dep to the oasdiff fork's
feat/yaml-disable-timestamps branch via a replace directive.

This is the same branch underlying getkin/kin-openapi#1181 (still
in review). It includes the typed validation errors from #1166 and
#1180 plus the DisableTimestamps integration with oasdiff/yaml
v0.1.0, which prevents YAML 1.1 implicit-timestamp resolution from
mangling date-shaped map keys in real-world specs.

Verified with the canonical case: unicourt.com/1.0.0/openapi.yaml
(originally cited in invopop/yaml#10 four years ago) now loads
cleanly and produces actual schema-validation findings rather than
the time.Time map-key explosion.

REMOVE BEFORE MERGE: replace this directive with a bump to the
released kin version once #1181 lands and a kin tag is cut.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Restructures the validate output to match the design's locked JSON
schema (mirrors oasdiff changelog --format json with a few additions).

Finding struct now exposes:
- source as an object {file, line, column} (was: string source + top-level
  line/column)
- section: which top-level doc section the finding belongs to
  ("info", "paths", "components", "webhooks", "servers", "security",
  "tags", or "" for unscoped doc-root findings). Determined per-cluster
  with a light Field-prefix check.
- fingerprint: stable 12-char sha256-prefix derived from
  "{id}:{operation}:{path}:{args}" — mirrors the existing
  formatters/changes.go:computeFingerprint scheme so the Pro
  PR-comment can partition findings into new/pre-existing/fixed via
  set membership on fingerprint across the base/revision spec pair.
- comment, operation, path: present but omitempty (Phase 1 leaves
  operation/path empty; extracting them from kin Origins requires
  walking the spec structure, deferred to Phase 2).
- All fields have both yaml: and json: tags.

Adds --format json support (encoder + enum value); existing yaml and
text paths unchanged in behaviour.

Tests:
- Existing yaml-format + origin-tracking tests updated to the new
  source-object shape.
- New JSON-format test pins the locked shape.
- New fingerprint-stability test pins determinism across runs (the
  property Pro PR-comment partitioning depends on).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the locked --color auto|always|never flag to oasdiff validate,
matching the convention used by changelog and breaking. Text output
now renders the severity in red/purple/cyan (error/warning/info,
via Level.StringCond) and the rule ID in yellow, matching the
established color scheme. Source path stays uncolored.

Auto mode disables color when stdout is piped or redirected; the
auto-detect helper lives in checker/piped_output.go and is shared
across subcommands so behaviour is consistent.

Exports checker.IsColorEnabled as a thin wrapper around the
package-internal isColorEnabled. Lets oasdiff packages outside
checker (validate, future subcommands) gate their own color logic
without duplicating the auto-detect convention.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…osing object

kin's Origin model carries two location handles:
- Origin.Key       — the start of the enclosing collection
                     (for a license-identifier error, the `license:` key)
- Origin.Fields[X] — the specific scalar field X inside that collection
                     (for a license-identifier error, the `identifier:` line)

The previous locator returned Origin.Key for every cluster, so a finding
in `data/validate/license-identifier-in-3-0.yaml` was pinned to line 5
(`license:`) instead of line 7 (`identifier: MIT`). Reviewers reading
the output had to scan from the parent key to find the actual offender.

Reworks locationForKinError (renamed from keyOriginForKinError) to prefer
Origin.Fields[Field] when the cluster carries a Field, falling back to
Origin.Key when the per-field entry is missing (e.g. empty values, which
kin doesn't track per-field). New fieldLoc helper centralises the lookup.

Per-cluster strategy:
- RequiredFieldError, FieldVersionMismatchError, ForbiddenFieldError,
  SchemaBothFormsExclusive — use Fields[Field]
- MutuallyExclusiveFieldsError — use Fields[Field1] (either field pins
  to the right object)
- SchemaValueError — use Fields[ValueKind] (e.g. "example", "default")
- SingleEntryContentError — use Fields[Subject]
- EitherFieldRequiredError, ExactlyOneFieldError — use Key (cluster fires
  when NONE of the fields are present, so per-field lookup wouldn't match)
- PathParametersError, ServerURLTemplateError — use Key (no Field metadata)
- WebhookNilError — no Origin (doc root)

Adds a regression test asserting the License-identifier fixture pins to
line 7:5 (the identifier line) rather than 5:3 (the license: line).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kin wraps validation errors in nested fmt.Errorf("...: %w") layers
that carry path/operation context as plain text rather than typed
fields:

  invalid paths: invalid path /thing: invalid operation GET: <inner>

These three layers (or any subset) are now stripped off the rendered
message and surfaced as Finding.Operation and Finding.Path, matching
the changelog command's convention of presenting them as discrete
fields. Finding.Text holds the cleaned inner message.

Text output gains a second header line "in API <op> <path>" rendered
in green when color is enabled, mirroring changelog / breaking. Doc-
root findings (info, openapi, license — no path/operation) skip the
line entirely.

Before:
  error [const-field-for-3-1-plus] at spec.yaml:18:21
        invalid paths: invalid path /thing: invalid operation GET: field const is for OpenAPI >=3.1

After:
  error [const-field-for-3-1-plus] at spec.yaml:18:21
        in API GET /thing
        field const is for OpenAPI >=3.1

Side benefit: Finding.Operation / Finding.Path are now populated for
operation-scoped findings, which feeds into the fingerprint (per
formatters/changes.go scheme: sha256(id:op:path:args)) and makes
Pro PR-comment partitioning stable when the same finding moves to a
new path between base and revision.

Component-scoped findings (kin's "invalid components: schema X:"
wrapper) are still inline in Text. Section is set correctly via
sectionForKinError, but extracting the schema name into a discrete
field is a separate enhancement.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This reverts commit f9bfe39.

The reverted commit used regex to strip kin's "invalid paths: invalid
path X: invalid operation Y:" prefixes off the rendered error message.
That was always meant to be interim — message-text parsing is exactly
the brittleness kin's typed-error work eliminates.

kin PR getkin/kin-openapi#1183 adds typed SectionContextError /
PathContextError / OperationContextError wrappers that carry the
context as structured fields. Once that merges and a kin release is
cut, the extraction becomes three errors.As calls with no string
parsing.

Carrying the regex in the interim isn't worth it: the whole
feat/validate-command branch is gated on a kin release anyway, so
there's no window where the regex would ship to users. Reverting now
keeps the branch honest and avoids a "delete the regex" follow-up
commit later.

The Finding struct keeps its Operation / Path fields (added in the
schema-alignment commit, not here) — they're simply unpopulated until
the typed-error extraction lands. omitempty elides them from output
in the meantime. Validate findings temporarily render the full
wrapped message inline again, as they did before f9bfe39.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resolve the go.mod/go.sum conflict by completing the planned kin
dependency swap: drop the temporary `replace => oasdiff/kin-openapi`
fork directive (#1181 + #1183 are now merged into getkin master) and
point the require at getkin/kin-openapi@master
(v0.138.1-0.20260514150620-69492dff6b62, the commit right after #1183).

Interim state: the @master pseudo-version must be swapped for a real
kin release tag before #894 can merge.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ror chain

Now that getkin/kin-openapi #1183 is merged, validate lifts the
structural scope of each finding out of the message text and into the
typed fields:

- Operation / Path come from PathValidationError + OperationValidationError
  via errors.As (replacing the reverted regex approach).
- sectionForKinError now prefers SectionValidationError.Section, kin's
  authoritative section name, falling back to the pre-#1183 cluster
  heuristics only for doc-root errors that aren't section-wrapped.

Adds data/validate/operation-missing-responses.yaml and a test asserting
an in-operation error surfaces operation=GET, path=/things, section=paths.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Section/path/operation scope now lives in the Finding's typed fields,
so the "invalid paths: invalid path X: invalid operation Y:" prefix that
kin's context wrappers add to Error() was pure duplication in Text.
unwrapContext peels those typed wrappers off the front of the chain so
Text carries only the underlying leaf message.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bump kin to oasdiff/kin-openapi @ e14b38a (the open getkin/kin-openapi#1185
branch with multi-error + simple-leaf conversions + plural-examples
typing fix). Lets oasdiff exercise the changes end-to-end before the kin
PR merges; the replace will drop and the dep will return to upstream
master once #1185 lands.

Wire-up:

 - internal/validate.go: pass openapi3.EnableMultiError() to Validate so
   simple-leaf and container errors aggregate rather than fail-fast.
 - internal/validate.go (fieldLoc): when the dotted Field name on a
   cluster error (e.g. "info.version") doesn't match a Fields key, fall
   back to the suffix after the last dot ("version"). Kin's Origin.Fields
   is keyed by the leaf name as it appears in the YAML mapping, while
   cluster errors use a dotted form for rule-ID disambiguation; without
   this fallback findings under aggregated leaves resolved to the parent
   object's Origin.Key instead of the precise field.

Concrete result on /tmp/multi-problem.yaml (empty info.title, empty
info.version, missing operation.responses): three findings at the exact
field lines and columns where the value is missing, instead of one
finding at the info section start.

Tests updated:

 - missing-required-info.yaml fixture: title now set, only version left
   empty so single-finding tests stay single.
 - Test_ValidateCmd_LineColumnFromOrigin: now asserts line 4 col 3 (the
   version line in the fixture).
 - Test_ValidateCmd_DocRootFieldHasLineColumn (renamed from
   ...HasNoLineColumn): asserts doc-root findings carry line:1 col:1 now
   that T.Origin is populated (kin #1184).
 - Test_ValidateCmd_TextFormatLocation: location string updated to 4:3.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants