Skip to content

fix(v2): tell a caller when to come back on every failure meant to be retried - #6625

Open
waleedlatif1 wants to merge 2 commits into
stagingfrom
feat/v2-http-standards
Open

fix(v2): tell a caller when to come back on every failure meant to be retried#6625
waleedlatif1 wants to merge 2 commits into
stagingfrom
feat/v2-http-standards

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Three related gaps in retry signalling on the v2 surface, found auditing it against RFC 9110 / RFC 6585 and against how Stripe, GitHub and Google's AIPs handle the same problems. Builds on #6620.

What was wrong

1. No 503 carried Retry-After. Not one of them: the three route builders' unhandledErrorResponse, the execute and resume routes, and serviceFailureResponse. A client's only defensible policy on a bare 503 is an immediate retry — which is exactly the traffic a degraded dependency cannot absorb, and Sim raises 503 precisely when the API-key store, the rollout gate, the rate-limit backend, or execution-identity allocation is briefly unavailable.

2. A 429 that already knew its wait threw it away. This is the more interesting one. ADMISSION_ERROR_DESCRIPTOR declares retryAfterSeconds: 5 for RESERVATION_CONCURRENCY, but mapping a descriptor onto a PreprocessExecutionError copied only statusCode, code and retryable. A concurrency denial therefore reached the client as a bare 429 with no Retry-After — despite the policy layer having named the wait five seconds earlier, and despite the OpenAPI already documenting RateLimited.headers: ['Retry-After']. The 503 infrastructure denial had the same leak; the new default happened to mask it with the same number.

3. One 503 must not advise a retry at all. ASYNC_ENQUEUE_AMBIGUOUS means the enqueue may have succeeded — it deliberately retains its execution-ID claim because a job may already exist. Telling that caller to come back in 5s invites a client with no X-Run-Id to start, and bill, a second run of the same workflow.

What changed

  • v2Error applies a Retry-After default keyed on the response status (not the v2 error code — Retry-After is defined against the status, and the status is the only half of the pair a client sees). A caller-supplied value still wins.
  • retryAfterSeconds now travels descriptorPreprocessExecutionError.retryAfterMsExecuteWorkflowServiceFailure.retryAfterMsserviceFailureResponse. The transport reads a number the policy owns instead of re-guessing it; the v2Error default is now only the floor for paths with no policy signal.
  • ASYNC_ENQUEUE_AMBIGUOUS opts out via omitRetryAfter and returns the run id so the caller reconciles rather than retries.
  • Reuses ADMISSION_RETRY_AFTER_SECONDS rather than restating 5, so the execute route's capacity 429 and every other surface's 503 cannot drift apart.
  • One central OpenAPI edit adds Retry-After to the shared ServiceUnavailable response — all 128 operations, 7 specs.

The audit

.agents/skills/v2-api-conventions/SKILL.md records the standards review so it is not re-litigated: the retry rule, the cursor-tamper-evidence invariants, and reasoned rejections of RFC 9457 problem+json, the IETF RateLimit-* draft fields, renaming X-RateLimit-* under RFC 6648, 422-for-semantic-validation, Location on 201, ETag / If-Match, and merge-patch+json — each with the spec text and the industry evidence.

Highlights of the "no" cases, since the evidence was not what I expected:

  • RFC 6648 does not ask anyone to rename deployed X- headers. §1 item 4 "makes no recommendation as to whether existing 'X-' parameters ought to remain in use or be migrated", and Appendix B argues the migration is the interoperability harm. The rule is a SHOULD NOT binding creators of new parameters.
  • The RateLimit-* draft is unpublished, was returned "Not ready" at HTTPDIR review, and is on its third mutually incompatible wire format. 0 of Stripe/GitHub/Google emit it.
  • 0 of 3 use RFC 9457, and 9457 §4 itself steers APIs with an existing format toward keeping it.
  • 0 of 3 do HTTP optimistic concurrency. Google does the semantics via a resource etag field (AIP-154), deliberately not If-Match.
  • Retry-After is only a MAY on 429 and 503 — it is SHOULD on exactly one status, 413. So this PR is a deliberate improvement, not a conformance fix, and says so.

Deprecation/Sunset on v1 is left open: emitting either commits Sim to a retirement date, which is a product decision, not an engineering one.

Idempotency is assessed and deliberately not extended here. X-Run-Id already gives the money path at-most-once semantics via a durable claim, and the operation description already says it is not an idempotency key. A true Stripe-style Idempotency-Key needs a request fingerprint, a retention window, an in-flight-vs-completed split, and somewhere to store a large synchronous body — a designed piece of work, not an increment. Aliasing the header name without the replay semantics would be worse than leaving it, since clients written against Stripe would read our 409 as a hard failure.

Verification

  • Every fix has a test proven to fail without it (reverted the source, watched it go red, restored).
  • bun run type-check, check:api-validation, check:openapi (7 specs, 128 operations, 225 examples), biome on all changed files, and 1299 tests across the v2 / routes / execution / billing suites — all green.

… retried

Three related gaps in retry signalling, found auditing the v2 surface against
RFC 9110/6585 and against how Stripe, GitHub and Google's AIPs handle the same
problems.

**No 503 carried `Retry-After`.** Every one of them — the three route builders'
`unhandledErrorResponse`, the execute and resume routes, and
`serviceFailureResponse` — funnels through `v2Error`, so the default lands
there, keyed on the response *status*: `Retry-After` is defined against the
status, and the status is the only half of the code/status pair a client sees.
A caller that supplies its own value still wins. RFC 9110 §15.6.4 makes this a
`MAY` rather than a `SHOULD`, so it is a deliberate improvement, not a
conformance fix: without it a client's only defensible policy on a 503 is an
immediate retry, and Sim raises 503 exactly when a dependency is too degraded to
absorb one.

**A 429 that already knew its wait threw it away.** The admission descriptors
declare `retryAfterSeconds` per denial, but mapping a descriptor onto a
preprocess error copied only `statusCode`, `code` and `retryable`. A
concurrency denial therefore reached the client as a bare 429 with no
`Retry-After` despite the policy layer having named the wait five seconds
earlier. The value now travels `descriptor.retryAfterSeconds` →
`PreprocessExecutionError.retryAfterMs` →
`ExecuteWorkflowServiceFailure.retryAfterMs` → `serviceFailureResponse`, so the
transport reads a number the policy owns instead of re-guessing one. The 503
default is now only the floor for paths with no policy signal.

**One failure must not advise a retry at all.** `ASYNC_ENQUEUE_AMBIGUOUS` is a
503 whose enqueue may have succeeded — it deliberately retains its execution-ID
claim because a job may already exist. Telling that caller to come back in five
seconds invites a client with no `X-Run-Id` to start, and bill, a second run of
the same workflow. It opts out via `omitRetryAfter` and returns the run id so
the caller reconciles instead.

`ADMISSION_RETRY_AFTER_SECONDS` is reused rather than restated, so the execute
route's capacity 429 and every other surface's 503 cannot drift apart.

Also records the audit in `.agents/skills/v2-api-conventions/SKILL.md`: the
retry rule, the cursor-tampering invariants, and reasoned rejections of RFC 9457
problem+json, the `RateLimit-*` draft fields, renaming `X-RateLimit-*` under RFC
6648, 422-for-semantic-validation, `Location` on 201, ETag/`If-Match`, and
`merge-patch+json` — each with the spec text and the industry evidence, so they
are not re-litigated. `Deprecation`/`Sunset` on v1 is left open pending a
retirement date, which is a product decision.
@vercel

vercel Bot commented Aug 12, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview Aug 12, 2026 4:47pm

Request Review

@cursor

cursor Bot commented Aug 12, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches shared v2 error responses and workflow execute failure paths (billing-sensitive ambiguous enqueue); behavior change is intentional for all 503s but could affect retrying clients.

Overview
Fixes v2 retry signalling so clients know when to back off instead of hammering degraded dependencies.

v2Error now adds a default Retry-After on 503 (shared ADMISSION_RETRY_AFTER_SECONDS floor); caller-supplied headers still win. 429 is not defaulted here—rate limits and admission denials must supply their own wait. omitRetryAfter suppresses the header for ASYNC_ENQUEUE_AMBIGUOUS on workflow execute so callers reconcile on the run id instead of risking a second billed run.

Admission retryAfterSeconds is no longer dropped in preprocessing: it flows through PreprocessExecutionError.retryAfterMs (including concurrency 429 and reservation infra 503), with UsageReservationUnavailableError exposing the descriptor value.

OpenAPI documents Retry-After on ServiceUnavailable and broadens the header description for 429 and 503. v2 API conventions gain Rule 6 (retry guidance), deliberate non-adoptions, idempotency (X-Run-Id), and cursor security notes, plus checklist items for Cache-Control on auth-sensitive 404s and retry headers.

Tests cover v2Error retry behavior, execute-route 503 / ambiguous enqueue, and preprocessing admission mapping.

Reviewed by Cursor Bugbot for commit 6de7ad9. Configure here.

@greptile-apps

greptile-apps Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds consistent retry guidance to transient v2 API failures while preventing blind retries when async enqueue success is ambiguous.

  • Adds a default Retry-After header to 503 responses while allowing explicit overrides and omission for ambiguous enqueue outcomes.
  • Propagates admission-policy retry timing through preprocessing and workflow execution responses.
  • Documents the normally-present header and its ASYNC_ENQUEUE_AMBIGUOUS exception in the shared OpenAPI response and regenerated specifications.
  • Adds focused tests for retry timing, policy propagation, and ambiguous enqueue reconciliation.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
apps/sim/app/api/v2/lib/response.ts Centralizes the default 503 retry delay, preserves explicit headers, and supports deliberate omission for ambiguous outcomes.
apps/sim/app/api/v2/workflows/[id]/execute/route.ts Propagates policy-owned retry timing and suppresses blind-retry guidance for ambiguous async enqueue failures.
apps/sim/lib/execution/preprocessing.ts Carries admission descriptor retry delays into preprocessing errors without assigning delays to non-retryable denials.
apps/sim/lib/api/contracts/v2/openapi/shared.ts The shared 503 response now accurately describes the normally-present Retry-After header and the ambiguous-enqueue exception.
apps/sim/lib/billing/calculations/usage-reservation.ts Exposes the admission policy's infrastructure retry delay for propagation to API callers.

Reviews (2): Last reviewed commit: "docs(v2): name the one 503 that omits Re..." | Re-trigger Greptile

Comment thread apps/sim/lib/api/contracts/v2/openapi/shared.ts
The shared ServiceUnavailable description claimed every 503 carries the header,
which the ASYNC_ENQUEUE_AMBIGUOUS response deliberately does not. It now says
the header is normally present and names that exception, so the published
contract matches the runtime behaviour for all 128 operations.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile review

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cursor review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 6de7ad9. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant