Skip to content

fix(billing): answer an in-flight retry at once, and tell Chat what provenance withheld - #9094

Merged
waleedlatif1 merged 7 commits into
stagingfrom
fix/billing-inflight-and-withheld-note
Oct 16, 2026
Merged

waleedlatif1 merged 7 commits into
stagingfrom
fix/billing-inflight-and-withheld-note

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Billing in-flight retry.
    • Before: the cost callback waited up to lock_timeout on the per-request ledger lock. While an earlier attempt of the same request was still committing (a commit can wait on a synchronous standby while holding that lock), a retry held a pooled connection for 3s and then failed with a 500 plus an ERROR log.
    • Now: the lock is try-acquired. A held lock answers 503 with code: BILLING_REQUEST_IN_FLIGHT and Retry-After: 5 immediately, logged at INFO.
    • The monotonic max-cumulative top-up is unchanged, so a resend can neither double-charge nor lose a charge.
    • lock_timeout still bounds the row locks taken under the ledger lock.
  • Provenance refusal logging.
    • Refusals are policy, not faults: they log at WARN instead of ERROR.
    • Each gate admission logs one line per (surface, cause), not one per item.
    • Each line keeps top-level surface, cause and refusedCount, plus the refused ids as each line names them today (up to 200, flagged resourceIdsTruncated beyond that), so every id stays paired with its cause. A single refusal keeps its resourceId field.
    • The table-row unknown-envelope line is now a WARN. Writer defects stay at ERROR.
  • Withheld items in Chat results.
    • Gate verdicts now carry the refusal reason.
    • A new countWithheldContent wraps each Chat CLI request. When items were withheld, the CLI's stderr says, per kind and reason, what the model can do next, and names no item:
      • unknown provenance: reading it again will not return it;
      • recorded secrets: read it as text for a redacted copy;
      • provenance not verified: it may be readable later.
    • v2 API bodies and headers are unchanged. Callers outside Chat never classify content, so they see no note.
  • In-flight refusal logging. An in-flight refusal no longer triggers the usage ledger's "did not complete promptly" WARN; the route logs it once at INFO. The span keeps the existing internal_error outcome, because the closed outcome set is generated from the legacy contracts.
  • Comment fixes. The worker's callback timeout is 15s, not 5s. No transaction setting bounds a commit that is waiting on a standby.
  • Contract. BillingInFlightCode is synced from the worker contracts.

Companion: simstudioai/mothership#705

Release order: either order is safe.

  • A worker without the companion change treats this 503 as retryable: it retries in-loop and then hands off to replay.
  • A worker with it, running against a Sim without this PR, simply never sees the code.

Type of Change

  • Bug fix

Testing

Each of these fails without this PR and passes with it:

  • usage-log.integration.ts:
    • New: a retry behind a holder of the same request is refused at once with the in-flight error. Before this PR it waited 3s and failed with a 55P03.
    • The recovery and concurrent-convergence tests now resend in-flight refusals the way replay does, and still converge to the exact totals.
  • route.test.ts: the in-flight error maps to the 503 with its code and Retry-After. Before this PR it returned a 500.
  • registry-gate.test.ts and session-gate.test.ts: verdicts carry the refusal reason, and 50 refusals produce one batched report.
  • durable-secret-provenance-telemetry.test.ts: 163 refusals produce one line per cause, each keeping every id. A 250-item group is capped at 200 ids and flagged as truncated.
  • content-delivery.test.ts: countWithheldContent counts refusals by kind and reason across nested requests, and counts nothing for non-Chat callers.
  • telemetry.test.ts: the note wording is checked for each refusal reason.
  • usage-log.test.ts: an in-flight refusal emits no slow-transaction WARN.
  • file-provenance.test.ts: the CLI result carries the reason-specific note.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…rovenance withheld

A cost callback retried while an earlier attempt of the same request was
still committing (a commit can wait on a synchronous standby while holding
the request's ledger lock) waited out lock_timeout on a pooled connection and
failed with a 500. The ledger lock is now taken without waiting: a held lock
answers 503 BILLING_REQUEST_IN_FLIGHT (Retry-After) at once, and the caller
resends later; the monotonic top-up applies the higher cumulative once.

Provenance refusals are policy, not faults: each gate admission now reports
its refusals as one warning with counts and a bounded sample instead of one
error per item, and the table envelope's unknown-row line is a warning.

Chat's CLI result now notes how many items of each kind a gate withheld, on
stderr, so the model can tell a withheld item from an absent one. Callers
outside Chat never classify content and see no change.

Also corrects two comments: the worker's callback timeout is 15 s, and no
transaction setting bounds a commit waiting on a standby.
@vercel

vercel Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Oct 16, 2026 12:36am UTC

Request Review

@github-actions github-actions Bot added the requires-mothership-merge Has a companion PR on the mothership/copilot side — merge in lockstep label Oct 15, 2026
@github-actions

Copy link
Copy Markdown

⚠️ Cross-repo companion check

One or more companion PRs aren't merged into staging yet. Merging this without them will leave copilot and sim out of sync — merge them in lockstep.

  • ❌ simstudioai/mothership#705 — OPEN, not merged (targets staging) — fix(worker): leave an unknown billing outcome to replay, count only replayable l

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 20 files

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/lib/mothership/agent-cli/index.ts
Comment thread apps/sim/lib/table/rows/secret-provenance.ts
Comment thread apps/sim/lib/table/rows/secret-provenance.ts Outdated
@greptile-apps

greptile-apps Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[High impact] The PR appears safe to merge; no blocking issues remain.

Summary

The PR returns a retryable 503 when another attempt holds the billing request’s lock. It also groups provenance refusal logs and tells Chat why content was withheld.

  • Billing retries get an immediate answer when a request is still in flight.
  • Chat tells the model why some results left items out.
  • Provenance refusals show up as grouped warnings, not error noise.

Reviews (4) · Last reviewed commit: "fix(provenance): give every refusal line..." · Reviewed by Greptile

Comment thread apps/sim/lib/mothership/content-delivery/telemetry.ts Outdated
…e route context

The usage-log unit mock answered every statement with undefined, which the
try-lock cannot read; it now answers whether the lock was taken, with a test
that a held lock refuses before any read or write. The new route test passes
the route context like the typed handler expects, and the refusal reason type
stays module-private.
…be the rows that read as unknown

A gate refuses either the item (unknown provenance) or only the requested
view (an excerpt or raw bytes redaction cannot cover), so the note no longer
says another read cannot return it. Untracked legacy rows read as exact-empty;
only tracked rows with a stale, missing or malformed sidecar read as unknown.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptileai review

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review

@cubic-dev-ai

cubic-dev-ai Bot commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 21 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

…ox files by hash, and tell Chat why each item was withheld

Refusal reports are grouped by surface, cause and owning record (document,
table or chat), each line with top-level surface, cause and refusedCount and
the refused ids (up to 200, flagged when truncated), so per-cause queries and
id-to-cause pairing survive aggregation. A sandbox file is attributed by a
chat-scoped hash of its path rather than the path, which carries the user's
file name. A streamed sandbox image declares its view opaque, so its refusal
reads as opaque content rather than a missing registry; the use case now
requires every caller to declare the view.

Gate verdicts carry the refusal reason, and Chat's CLI note says per reason
what a read can do next: unknown provenance will not return on a retry,
recorded secrets can be read as text for a redacted copy, and provenance that
could not be verified may be readable later. Chunk write refusals name their
document as the parent.
The usage ledger's slow-transaction warning fired for every in-flight
refusal (stage lock, not succeeded), although nothing waited and the route
already logs the refusal once at INFO. It now skips refusals. The span keeps
the existing internal_error outcome, since the closed outcome set is
generated from the legacy contracts; the response code and the INFO line tell
it apart.
…roduced

Drops the sandbox attribution id, the streamed-image view and the owning-record
ids from this change; they belong to the provenance owner's work and fix
behavior that predates it. Refusal lines stay one per surface and cause with
top-level surface, cause and refusedCount and every refused id as each line
names it today, and the note keeps its per-reason wording.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptileai review

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review

@cubic-dev-ai

cubic-dev-ai Bot commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 22 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

… tells an in-flight 503 apart

A single refusal now carries refusedCount 1 like a grouped one, so summing
refusedCount by cause covers both. The route comment now says the span is the
same as the retryable settlement 503's and that the body code and the INFO
line distinguish them.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptileai review

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review

@cubic-dev-ai

cubic-dev-ai Bot commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 22 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1
waleedlatif1 merged commit f4fd25e into staging Oct 16, 2026
53 checks passed
@waleedlatif1
waleedlatif1 deleted the fix/billing-inflight-and-withheld-note branch October 16, 2026 01:05

This branch was previously deployed

1 inactive deployment
Preview — 62aae616 Deployed Oct 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

requires-mothership-merge Has a companion PR on the mothership/copilot side — merge in lockstep

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant