fix: preserve Vale severity in CI annotations and block merge on errors#26587
Conversation
nickvigilante
added a commit
that referenced
this pull request
Jun 22, 2026
Collapse the Vale config to load only the Coder rule package. Drop the
Packages directive (no external packages sync), BasedOnStyles=Coder
only, drop every Google.X, write-good.X, and alex.X rule line. The
repo-root .vale.ini becomes a four-line config plus a doctrine comment.
Motivation: the previous config carried ~12,000 baseline findings
corpus-wide (412 errors / 5380 warnings / 6247 suggestions), almost
entirely from third-party rules whose false-positive patterns Vale
cannot distinguish from author intent. Google.Headings false-positives
on every acronym and product name (VM, AWS, Coder, Vale, JetBrains);
Google.Will fires on legitimate event-sequencing prose; alex.* rules
were enabled in DOCS-40 without a corpus cleanup commit. Engineers
stop reading annotations when CI cries wolf. The fix is a tight,
trustworthy ruleset rather than tuning around individual false
positives.
Doctrine added to docs/.style/README.md ('Adding a Vale rule'): every
rule ships clean (zero baseline findings) with a deliberate per-rule
severity choice. 'error' blocks merge in CI; 'warning' surfaces
annotations without failing CI; 'suggestion' surfaces 'notice'
annotations. The severity choice and the cleanup discipline are
independent, so a rule landing at any severity still ships with zero
baseline findings to keep the annotation channel trustworthy. Third-
party rules return via the same per-rule PR pattern after their corpus
is clean. See DOCS-425.
Effect on baseline: 412 errors -> 0, 5380 warnings -> 0, 6247
suggestions -> 0. Verified with 'mise exec aqua:errata-ai/vale -- vale
--no-exit docs/' against 465 files. Vale sync now skips the
third-party packages (the directories remain on disk for developers
who already synced; they're gitignored and no longer load). The CI
'Vale prose lint' step is currently advisory; PR #26587 (DOCS-426)
installs the merge gate that honors the doctrine's 'error blocks
merge' semantic.
95f6409 to
ba20bc2
Compare
The Vale prose lint step in CI ran 'vale --output=line' through a GitHub Actions problem matcher at .github/vale-problem-matcher.json. The matcher hard-codes severity=warning at the top level, and Vale's line format strips per-finding severity, so every Vale finding rendered as a GitHub 'warning' annotation, even error-level findings from Coder.BrandNames. Switch to 'vale --output=JSON' and pipe through jq to emit GitHub workflow commands directly. Mapping: Vale 'suggestion' -> ::notice::, Vale 'warning' -> ::warning::, Vale 'error' -> ::error::. URL-encode message bodies for '%', carriage return, and newline per the GitHub Actions workflow command spec. Delete the now-unused .github/vale-problem-matcher.json. The CI step stays continue-on-error: true so Vale annotations remain advisory and never block merge. See DOCS-426.
Adds three throwaway Coder.Demo* rules and a single demo doc that triggers each rule exactly once. Together with the matcher fix in the previous commit, this PR's CI surfaces three GitHub annotations in three distinct severities (notice, warning, error). Used to visually verify that severity rendering works correctly after the fix.
Files:
- docs/.style/styles/Coder/DemoError.yml (level: error)
- docs/.style/styles/Coder/DemoWarning.yml (level: warning)
- docs/.style/styles/Coder/DemoSuggestion.yml (level: suggestion)
- docs/.style/_vale-annotation-demo.md (one trigger phrase per rule)
Each rule uses Vale's 'existence' extension type to match a single literal token (vale-demo-{error,warning,suggestion}-marker) that appears only in the demo doc. The demo doc carries a NOTE callout explaining its purpose.
Demo content drops in a follow-up commit on this branch before merge; only the matcher fix lands. Or, on @Vigilante's call, the demo stays as a permanent canary that monitors severity rendering across future CI changes. See DOCS-426.
…ings Drop `continue-on-error: true` and `vale --no-exit` from the Vale prose lint step so error-level findings fail the step, fail the required `lint-docs` job, and block merge. Warning- and suggestion-level findings still render as PR annotations but do not affect the exit code, matching Vale's native exit semantics (only errors trigger non-zero exit). `set -o pipefail` is required so Vale's exit code propagates through the jq pipe; without it, jq's success masks Vale's non-zero exit. Pairs with DOCS-425's "rules ship clean" doctrine: any rule enabled in `.vale.ini` ships with zero baseline error-level violations, so the new merge gate fires only when a PR introduces new prose that violates an enabled rule. The three-severity demo doc deliberately includes one error-level finding, so this PR's `lint-docs` job is expected to fail until the demo content drops in the follow-up cleanup commit before merge.
5aefebf to
c774ad3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes DOCS-426. Stacked on #26586 (DOCS-425, strip).
Problem
Two coupled issues in the Vale CI step:
.github/vale-problem-matcher.jsonhard-codes"severity": "warning". Every Vale finding renders as a GitHubwarningannotation, regardless of Vale's actual severity. Nick observed this on PR #25501: error-level findings fromCoder.BrandNamesappear as warnings.continue-on-error: trueandvale --no-exit, so no Vale finding could fail the job, even at severityerror. There was no merge gate on prose style.Together these two issues collapsed the doctrine's three-severity ladder (
errorblocks merge,warningannotates without blocking,suggestionsurfaces as a notice) into a single advisory channel. This PR restores the ladder.Root cause (rendering)
GitHub Actions problem matchers expect either a regex capture group for severity or a hard-coded severity. Vale's
--output=lineformat producespath:line:col:rule:messagewith severity stripped, so the matcher had no severity to capture and fell back on the hard-coded value.Fix
Commit 1: severity rendering
Switch the Vale prose lint step to
vale --output=JSONand pipe throughjqto emit GitHub workflow commands directly. Drop the problem matcher file.suggestion::notice::warning::warning::error::error::Message bodies are URL-encoded for
%,\r, and\nper the GitHub Actions workflow command spec.Commit 2: three-severity demo
Three throwaway
Coder.Demo*rules atlevel: suggestion,level: warning, andlevel: error, plus adocs/.style/_vale-annotation-demo.mdfile that triggers each rule exactly once. Together with the rendering fix above, this PR's CI surfaces three GitHub annotations in three distinct severities (notice, warning, error). Use the Files Changed view to inspect rendering.Commit 3: block merge on error-level findings
Drop
continue-on-error: trueandvale --no-exitfrom the Vale prose lint step. Error-level findings now exit non-zero, fail the step, fail the requiredlint-docsjob, and block merge. Warning- and suggestion-level findings still render as annotations but do not affect the exit code. This matches Vale's native exit semantics (only errors trigger non-zero exit) and operationalizes the DOCS-425 doctrine: every rule lands clean (zero baseline findings) with a deliberate per-rule severity choice;erroris the severity that catches new violations of hard policy and blocks the PR, whilewarningandsuggestionannotate without forcing the gate.set -o pipefailis required so Vale's exit code propagates through the jq pipe; without it, jq's success masks Vale's non-zero exit.Expected CI state on this PR
The demo doc in commit 2 deliberately includes one error-level finding (
Coder.DemoError). With commit 3 in place, this PR'slint-docsjob is expected to fail with the demo error annotation rendered at line 21 ofdocs/.style/_vale-annotation-demo.md. That is the canary that proves the merge gate works end-to-end. Warning- and suggestion-level demo annotations render at lines 19 and 17 without affecting the exit code.Local verification of the full pipeline:
Sequencing before merge
A follow-up commit on this branch drops the four demo files (
docs/.style/_vale-annotation-demo.mdand the threeCoder.Demo*rules). After that drop,lint-docsgoes green and the PR is mergeable.Stacked on #26586. Targets the strip branch so the demo runs against a clean post-strip baseline (no third-party noise drowning the three demo annotations). After #26586 merges, this PR retargets to
main.Decision log
.github/.docs/.style/_vale-annotation-demo.mdfilename: underscore prefix follows Coder convention for files that exist outside the normal docs taxonomy. Not surfaced on coder.com/docs becausedocs/.style/is excluded from the manifest, deploy workflow, and docs preview.errorfor hard rules that must hold (e.g.,Coder.BrandNames, banned first-person pronouns, em-dash bans),warningfor strong guidance with legitimate human-judgment exceptions,suggestionfor soft guidance where the right fix is contextual. Blocking on anything belowerrorwould force every rule onto an all-or-nothing setting and defeat the gradation the doctrine deliberately built.set -o pipefailvs capturing exit then processing: pipefail is cleaner and keeps the streaming pipeline intact. The alternative (write to a temp file, capture exit code, jq from disk) trades one line of shell for an extra disk write and an extra variable to thread.Filed via Coder Agents on Nick's behalf.