Repository navigation
feat: price AI Gateway long-context requests at their tier prices - #30916
ibetitsmike wants to merge 2 commits into
Conversation
When the prompt of one upstream request is over the threshold of a long-context price tier, every token of that request now uses the prices of the highest such tier. An unset tier price uses the base price. aibridge sends the prompt size on each token usage record (aibridged API v1.4); the output-only message_delta record of a stream carries the prompt size of its message_start. Migration 000628 adds the tiers column to ai_model_prices and records the prompt size and the applied tier on aibridge_token_usages. The seed, API, SDK, CLI, and model form carry the tiers, and the cost controls docs explain the rule.
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8cdccf73d8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Note
Xum opened this PR on behalf of Mike.
Bills an AI Gateway request at its model's long-context tier prices when the request's prompt is over the tier's threshold, instead of always at the base prices.
Stack context
AI Gateway prices every cache write at the 5-minute price and ignores the long-context prices that models.dev publishes, so Claude usage and long prompts are under-priced. On dogfood, Oct 1-14 recorded $7,535 against a list price of about $9,290. The 1-hour cache rule closes about 86% of that gap, and the tiers close almost all of the rest. The stack, bottom to top:
Why
Some models publish higher prices above a prompt size, for example Claude Haiku 5.5 above 100K tokens and GPT-5.6 and later above 272K tokens, on their own APIs and on Bedrock. AI Gateway billed those requests at the base prices. #30915 added the tier prices to the price book; this PR bills with them.
Pricing rule
threshold_tokens, every token of that request uses the tier's prices. A prompt equal to the threshold uses the base prices.What changed
token_usagelogs include it asprompt_tokens. In a stream, the output-onlymessage_deltarecord carries the prompt size of itsmessage_start, because the two records reach coderd in no fixed order.ai_model_prices.tiers(a jsonb array), and onaibridge_token_usagesthe prompt size and the threshold of the applied tier, so each row's price snapshot still adds up tocost_micros.pricesForPromptincoderd/aibridgedserver/cost.gopicks the tier, and the usage row stores the prices that applied.tiers. Validation rejects negative prices and thresholds that are not positive and strictly ascending. The CLI lists tiers as an optional column and shows tier changes in the update plan.CostEstimateWithTiers).cost-controls.mdstates the rule.What to scrutinize
usageFromRequest:max(prompt_tokens, input + cache read + cache write). Records from AI Gateway daemons on API v1.3 send no prompt size, so they still select a tier by their own input.tiersvalue that does not decode fails the usage record instead of billing at the base prices. The API and the seed validate tiers, so only a manual database edit can cause this.UpsertAIModelPricesnow also rewrites a default row when only its tiers change, so existing deployments pick up tiers on upgrade.Generated files
All from
make gen:coderd/aibridged/proto/aibridged.pb.go(protoc);coderd/database/queries.sql.go,querier.go,models.go,dump.sql, andcheck_constraint.go(sqlc and the schema dump);coderd/apidoc/*anddocs/reference/api/schemas.md(swag);site/src/api/typesGenerated.ts.Validation
go testpasses foraibridge/intercept/messages,aibridge/recorder,aibridge/internal/integrationtest,coderd/aibridgedserver,coderd/aibridge/prices/...,coderd/database/migrations,db2sdk,dbgen, andcoderd/aibridged/..., and with-run AIModelPriceinenterprise/coderdandenterprise/cli.tsc, Vitest (src/modules/aiModelsand theModelFormstories), Biome, oxlint, and knip pass.