Repository navigation
feat: price AI Gateway 1h cache writes - #30872
ibetitsmike wants to merge 16 commits into
Conversation
AI Gateway charged every cache write at the 5-minute price and ignored the long-context price tiers models.dev publishes, so Claude and long prompts were under-priced. - Messages interceptions send the 1h cache write count and the prompt size of each upstream request (aibridged API v1.4). A stream's output-only message_delta record carries the prompt size of its message_start. - ai_model_prices gains cache_write_1h_price and a tiers jsonb column; aibridge_token_usages records the 1h count, prompt size, applied 1h price and applied tier threshold (migration 000624). Past usage is not changed. - Cost uses the 1h price for 1h writes and the prices of the highest tier the prompt is over; a missing tier price falls back to the base price. - The price generator sets the 1h price to 2x input for Claude on anthropic, bedrock and copilot and keeps models.dev tiers; prices.json gains both fields. - The API, SDK, CLI (--cache-write-1h-price, tiers in the JSON document) and the read-only model form show the new prices; docs explain the rules.
…and tier prices in cost estimate story
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ae44583f4b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
An earlier commit checked these pages out from a newer main, which added unrelated main changes to this branch.
…tier A tier without a 1h price bills 1h writes at its own cache write price, so the tier note lists that price whenever the model or the tier prices 1h writes apart.
…ase prices The API and the price book reject a price row without one of the four base prices, so a row with only a 1h price or tiers cannot exist.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 46411a3f69
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. Hooray! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
main added 000624_workspace_secrets, so the AI Gateway cache TTL and tiers migration moves to 000625. Its SQL is unchanged.
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep it up! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
A tier without a 1h price uses the cache write price that applies to the tier, which can be the base cache write price. The upsert doc no longer says a missing cache_write_1h_price clears it, because the generated TypeScript type requires the key. The price tier note helpers get JSDoc, and the comments that restated the code or the story are gone.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e299124e40
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The endpoint and the CLI treat a missing cache_write_1h_price like null, and tiers is already omitempty. Without omitempty the generated AIModelPriceUpsert type required the key, so a TypeScript client that built the four-price upsert stopped compiling.
|
@codex review |
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6e7e4a67b6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Note
Xum opened this PR on behalf of Mike.
Bills a Claude cache write with a 1-hour TTL at its model's 1-hour price, 2x the input price, instead of the 5-minute cache write price.
Stack context
AI Gateway prices every cache write at the 5-minute price and ignores the long-context prices that models.dev publishes, so Claude usage and long prompts are under-priced. On dogfood, Oct 1-14 recorded $7,535 against a list price of about $9,290. The 1-hour cache rule closes about 86% of that gap, and the tiers close almost all of the rest. The stack, bottom to top:
Why
Claude bills a 1-hour cache write at 2x the input price and a 5-minute write at 1.25x, but AI Gateway billed both at the 5-minute price. On dogfood the 1-hour rule closes about 86% of the gap above. This PR first carried both pricing rules; it now holds only the 1-hour rule, on top of #30916.
Providers
/anthropicroute), and Copilot goes through the Messages API, which reports the 1-hour part of each cache write. Those writes now use the 1-hour price.openai.*), have no TTL-dependent cache price. GPT-5.6 and later bill a cache write at 1.25x the input price with a single30mTTL, and earlier models do not charge for cache writes. One cache write price already covers them, so they do not change.What changed
token_usagelogs include it ascache_write_one_hour_input_tokens.ai_model_prices.cache_write_1h_price, and onaibridge_token_usagesthe 1-hour token count and the 1-hour price that applied. A check keeps the 1-hour count within the cache write count.cache_write_1h_price. It is optional, and omitting it clears it. The CLI adds--cache-write-1h-price.cost-controls.mdstates the rule and tells admins to setcache_write_1h_priceon custom prices.What to scrutinize
pricesForPrompt).usageFromRequestclamps a 1-hour count above the cache write count instead of rejecting it. The count comes from the provider, and a rejected record would drop the whole usage row.Generated files
All from
make gen:coderd/aibridged/proto/aibridged.pb.go(protoc);coderd/database/queries.sql.go,querier.go,models.go,dump.sql, andcheck_constraint.go(sqlc and the schema dump);coderd/apidoc/*anddocs/reference/api/schemas.md(swag);site/src/api/typesGenerated.ts.Validation
go testand site checks listed on feat: price AI Gateway long-context requests at their tier prices #30916 pass on this head, with the new 1-hour cases.cache_write_1h_priceclears the price, and a 0 is kept.6e7e4a67b68: migrations 000628 and 000629 applied, custom price rows and earlier usage rows did not change, and the cost of all 10 new usage rows matched a recomputation. On this head (ef70012cde3), which only adds log fields, live 1-hour writes were billed correctly, and the structuredtoken_usagelogs from both log sources carriedprompt_tokensandcache_write_one_hour_input_tokens.