Skip to content

feat: price AI Gateway 1h cache writes - #30872

Draft
ibetitsmike wants to merge 16 commits into
mike/aigw-prices/long-context-tiersfrom
mike/aigw-cache-1h-long-context-prices
Draft

ibetitsmike wants to merge 16 commits into
mike/aigw-prices/long-context-tiersfrom
mike/aigw-cache-1h-long-context-prices

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Note

Xum opened this PR on behalf of Mike.

Bills a Claude cache write with a 1-hour TTL at its model's 1-hour price, 2x the input price, instead of the 5-minute cache write price.

Stack context

AI Gateway prices every cache write at the 5-minute price and ignores the long-context prices that models.dev publishes, so Claude usage and long prompts are under-priced. On dogfood, Oct 1-14 recorded $7,535 against a list price of about $9,290. The 1-hour cache rule closes about 86% of that gap, and the tiers close almost all of the rest. The stack, bottom to top:

  1. chore: add 1h cache write and long-context tier prices to the price book #30915: adds the 1-hour cache write and long-context tier prices to the price book.
  2. feat: price AI Gateway long-context requests at their tier prices #30916: bills long prompts at their long-context tier prices.
  3. feat: price AI Gateway 1h cache writes #30872 (this PR): bills 1-hour cache writes at the 1-hour price.

Why

Claude bills a 1-hour cache write at 2x the input price and a 5-minute write at 1.25x, but AI Gateway billed both at the 5-minute price. On dogfood the 1-hour rule closes about 86% of the gap above. This PR first carried both pricing rules; it now holds only the 1-hour rule, on top of #30916.

Providers

  • Claude on Anthropic (including Claude Platform), Bedrock (InvokeModel and the Mantle /anthropic route), and Copilot goes through the Messages API, which reports the 1-hour part of each cache write. Those writes now use the 1-hour price.
  • OpenAI models, on OpenAI, Azure, Copilot, or Bedrock Mantle (openai.*), have no TTL-dependent cache price. GPT-5.6 and later bill a cache write at 1.25x the input price with a single 30m TTL, and earlier models do not charge for cache writes. One cache write price already covers them, so they do not change.
  • Claude through OpenRouter or Vercel goes through the OpenAI-compatible path, which does not report the 1-hour part, so those cache writes keep the cache write price.

What changed

  • aibridge sends the 1-hour part of each cache write on the token usage record (aibridged API v1.5), and structured token_usage logs include it as cache_write_one_hour_input_tokens.
  • Database (migration 000629): ai_model_prices.cache_write_1h_price, and on aibridge_token_usages the 1-hour token count and the 1-hour price that applied. A check keeps the 1-hour count within the cache write count.
  • Cost: 1-hour writes use the 1-hour price and the other cache writes use the cache write price. A model or tier without a 1-hour price uses the cache write price that applies.
  • API, SDK, and CLI carry cache_write_1h_price. It is optional, and omitting it clears it. The CLI adds --cache-write-1h-price.
  • Model form: a read-only "Cache write (1h)" price when the row has one, and the tier lines include it.
  • Docs: cost-controls.md states the rule and tells admins to set cache_write_1h_price on custom prices.

What to scrutinize

  • A tier without a 1-hour price falls back to the tier's cache write price, not to the base 1-hour price (pricesForPrompt).
  • usageFromRequest clamps a 1-hour count above the cache write count instead of rejecting it. The count comes from the provider, and a rejected record would drop the whole usage row.

Generated files

All from make gen: coderd/aibridged/proto/aibridged.pb.go (protoc); coderd/database/queries.sql.go, querier.go, models.go, dump.sql, and check_constraint.go (sqlc and the schema dump); coderd/apidoc/* and docs/reference/api/schemas.md (swag); site/src/api/typesGenerated.ts.

Validation

  • The go test and site checks listed on feat: price AI Gateway long-context requests at their tier prices #30916 pass on this head, with the new 1-hour cases.
  • Each new test was checked to fail when the behavior it covers is removed.
  • Before the split, dogfood UAT of the combined change passed four rounds. The cost of all 29 new usage rows matched a recomputation (13 had 1-hour writes and 12 used a tier). A custom price row and usage rows recorded before the upgrade did not change. Through the CLI and the API, a missing or null cache_write_1h_price clears the price, and a 0 is kept.
  • After the split, a smoke UAT on dogfood passed. It upgraded a test deployment from main to 6e7e4a67b68: migrations 000628 and 000629 applied, custom price rows and earlier usage rows did not change, and the cost of all 10 new usage rows matched a recomputation. On this head (ef70012cde3), which only adds log fields, live 1-hour writes were billed correctly, and the structured token_usage logs from both log sources carried prompt_tokens and cache_write_one_hour_input_tokens.

Model form for claude-haiku-5-5 with the 1h price and a tier line

Model form for gpt-4.1 without a 1h price or tiers

AI Gateway charged every cache write at the 5-minute price and ignored the
long-context price tiers models.dev publishes, so Claude and long prompts
were under-priced.

- Messages interceptions send the 1h cache write count and the prompt size
  of each upstream request (aibridged API v1.4). A stream's output-only
  message_delta record carries the prompt size of its message_start.
- ai_model_prices gains cache_write_1h_price and a tiers jsonb column;
  aibridge_token_usages records the 1h count, prompt size, applied 1h price
  and applied tier threshold (migration 000624). Past usage is not changed.
- Cost uses the 1h price for 1h writes and the prices of the highest tier
  the prompt is over; a missing tier price falls back to the base price.
- The price generator sets the 1h price to 2x input for Claude on
  anthropic, bedrock and copilot and keeps models.dev tiers; prices.json
  gains both fields.
- The API, SDK, CLI (--cache-write-1h-price, tiers in the JSON document)
  and the read-only model form show the new prices; docs explain the rules.
@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-17T04:43:32.303383Z ef70012 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ae44583f4b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/.style/style-guide/word-choice.md
Comment thread site/src/modules/aiModels/PriceTierNotes.tsx Outdated
An earlier commit checked these pages out from a newer main, which added
unrelated main changes to this branch.
…tier

A tier without a 1h price bills 1h writes at its own cache write price, so
the tier note lists that price whenever the model or the tier prices 1h
writes apart.
…ase prices

The API and the price book reject a price row without one of the four
base prices, so a row with only a 1h price or tiers cannot exist.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 46411a3f69

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/aibridgedserver/cost.go
Comment thread coderd/aibridgedserver/cost.go
Comment thread coderd/aibridgedserver/cost.go
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 46411a3f69

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

main added 000624_workspace_secrets, so the AI Gateway cache TTL and
tiers migration moves to 000625. Its SQL is unchanged.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: edc1168c9a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

A tier without a 1h price uses the cache write price that applies to the
tier, which can be the base cache write price. The upsert doc no longer
says a missing cache_write_1h_price clears it, because the generated
TypeScript type requires the key. The price tier note helpers get JSDoc,
and the comments that restated the code or the story are gone.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e299124e40

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread codersdk/aimodelprices.go Outdated
The endpoint and the CLI treat a missing cache_write_1h_price like null,
and tiers is already omitempty. Without omitempty the generated
AIModelPriceUpsert type required the key, so a TypeScript client that
built the four-price upsert stopped compiling.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 054ca2c595

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike ibetitsmike added the experimental Changes that might not necessarily be merged, until its approved to proceed with. label Oct 16, 2026
@ibetitsmike
ibetitsmike changed the base branch from main to mike/aigw-prices/long-context-tiers October 17, 2026 04:06
@ibetitsmike
ibetitsmike added this pull request to stack #30917 October 17, 2026 04:06
@ibetitsmike ibetitsmike changed the title feat: price AI Gateway 1h cache writes and long-context tiers feat: price AI Gateway 1h cache writes Oct 17, 2026
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e7e4a67b6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread enterprise/coderd/aimodelprices.go
Comment thread codersdk/aimodelprices.go
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: ef70012cde

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

experimental Changes that might not necessarily be merged, until its approved to proceed with.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant