Skip to content

fix: remove 403 from key failover and cooldown on 401 (#27419) - #27524

Merged
mtojek merged 1 commit into
release/2.35from
backport/27419-to-2.35
Jul 27, 2026
Merged

fix: remove 403 from key failover and cooldown on 401 (#27419)#27524
mtojek merged 1 commit into
release/2.35from
backport/27419-to-2.35

Conversation

@ssncferreira

@ssncferreira ssncferreira commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Backport of #27419

Original PR: #27419 — fix: remove 403 from key failover and cooldown on 401
Merge commit: dba45ce
Requested by: @ssncferreira

@ssncferreira
fix: remove 403 from key failover and cooldown on 401 (#27419)
322b915 
When a key returned 401 or 403, the pool marked it permanently
unavailable for the lifetime of that in-memory pool. This is bad UX: a
transient auth failure or a briefly-misconfigured key could take a key
out of rotation until the operator either restarted Coder or
reconfigured the key (even re-saving the same working value).

- **403 removed from key failover**: it's a per-request authorization
failure, not a key-level problem, so it's surfaced to the caller as-is
without marking the key or failing over.
- **401 now applies a temporary cooldown** (like 429) so the key
recovers on its own instead of staying blocked.
- When every key is in an auth-failure cooldown, the pool reports a
`502` with no `Retry-After`, but the keys still recover automatically
once the cooldown elapses.

Closes
https://linear.app/codercom/issue/AIGOV-421/ai-gateway-a-quarantined-centralized-key-never-recovers-without-a
Closes
https://linear.app/codercom/issue/AIGOV-533/403s-misclassifying-keys-as-permanently-down-in-ai-gateway

> [!NOTE]
> Initially generated by Claude Opus 4.7, modified and reviewed by
@ssncferreira

(cherry picked from commit dba45ce)
@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Docs preview

📖 View docs preview for docs/ai-coder/ai-gateway/providers.md

@ssncferreira
ssncferreira marked this pull request as ready for review July 27, 2026 13:41
@coderagents

coderagents Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Documentation Check

The docs/ai-coder/ai-gateway/providers.md change in this backport is accurate and matches the code: 401/429 now apply a temporary cooldown, 403 passes through untouched, exhaustion returns 429 with Retry-After when any key is rate-limited and 502 with no Retry-After when every cooldown came from an auth failure. It is byte-identical to the merged section on main.

One gap carried over from the original PR (#27419) remains unaddressed:

Updates Needed

  • scripts/metricsdocgen/metrics + docs/admin/integrations/prometheus.md - coder_ai_gateway_key_pool_state_transitions_total is still documented as (reason: rate_limited, unauthorized, forbidden), but aibridge/metrics/metrics.go now emits (reason: rate_limited, unauthorized) and no code path produces a forbidden reason. Update the static entry in scripts/metricsdocgen/metrics (AI Gateway metrics are manually maintained there), then run make gen to regenerate docs/admin/integrations/prometheus.md.

    Also stale on main, so the fix likely belongs there first and should be backported alongside this PR.

Optional, non-blocking: coder_ai_gateway_key_pool_state still lists a permanent state. MarkPermanent now has no non-test callers, so that state is unreachable in practice. Worth a follow-up decision rather than a docs edit here.

Automated review via Coder Agents

@ssncferreira
ssncferreira requested a review from mtojek July 27, 2026 13:59
@mtojek
mtojek merged commit c9388a1 into release/2.35 Jul 27, 2026
57 of 58 checks passed
@mtojek
mtojek deleted the backport/27419-to-2.35 branch July 27, 2026 14:01
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 27, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@mtojek mtojek mtojek approved these changes

Assignees

@ssncferreira ssncferreira

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ssncferreira @mtojek