Skip to content

feat: price AI Gateway long-context requests at their tier prices - #30916

Draft
ibetitsmike wants to merge 2 commits into
mike/aigw-prices/price-bookfrom
mike/aigw-prices/long-context-tiers
Draft

ibetitsmike wants to merge 2 commits into
mike/aigw-prices/price-bookfrom
mike/aigw-prices/long-context-tiers

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Note

Xum opened this PR on behalf of Mike.

Bills an AI Gateway request at its model's long-context tier prices when the request's prompt is over the tier's threshold, instead of always at the base prices.

Stack context

AI Gateway prices every cache write at the 5-minute price and ignores the long-context prices that models.dev publishes, so Claude usage and long prompts are under-priced. On dogfood, Oct 1-14 recorded $7,535 against a list price of about $9,290. The 1-hour cache rule closes about 86% of that gap, and the tiers close almost all of the rest. The stack, bottom to top:

  1. chore: add 1h cache write and long-context tier prices to the price book #30915: adds the 1-hour cache write and long-context tier prices to the price book.
  2. feat: price AI Gateway long-context requests at their tier prices #30916 (this PR): bills long prompts at their long-context tier prices.
  3. feat: price AI Gateway 1h cache writes #30872: bills 1-hour cache writes at the 1-hour price.

Why

Some models publish higher prices above a prompt size, for example Claude Haiku 5.5 above 100K tokens and GPT-5.6 and later above 272K tokens, on their own APIs and on Bedrock. AI Gateway billed those requests at the base prices. #30915 added the tier prices to the price book; this PR bills with them.

Pricing rule

  • The prompt size of one upstream request is its input, cache read, and cache write tokens.
  • When the prompt size is over a tier's threshold_tokens, every token of that request uses the tier's prices. A prompt equal to the threshold uses the base prices.
  • With several tiers, the highest threshold below the prompt size applies. A price that the tier does not set uses the base price.
  • Past usage is not repriced. On startup the seed adds tiers to default price rows; custom prices keep their own values.

What changed

  • aibridge sends the prompt size on each token usage record (aibridged API v1.4), and structured token_usage logs include it as prompt_tokens. In a stream, the output-only message_delta record carries the prompt size of its message_start, because the two records reach coderd in no fixed order.
  • Database (migration 000628): ai_model_prices.tiers (a jsonb array), and on aibridge_token_usages the prompt size and the threshold of the applied tier, so each row's price snapshot still adds up to cost_micros.
  • Cost: pricesForPrompt in coderd/aibridgedserver/cost.go picks the tier, and the usage row stores the prices that applied.
  • API, SDK, and CLI carry tiers. Validation rejects negative prices and thresholds that are not positive and strictly ascending. The CLI lists tiers as an optional column and shows tier changes in the update plan.
  • Model form: one read-only line per tier under the cost estimate (story CostEstimateWithTiers).
  • Docs: cost-controls.md states the rule.

What to scrutinize

  • The prompt-size floor in usageFromRequest: max(prompt_tokens, input + cache read + cache write). Records from AI Gateway daemons on API v1.3 send no prompt size, so they still select a tier by their own input.
  • A stored tiers value that does not decode fails the usage record instead of billing at the base prices. The API and the seed validate tiers, so only a manual database edit can cause this.
  • UpsertAIModelPrices now also rewrites a default row when only its tiers change, so existing deployments pick up tiers on upgrade.

Generated files

All from make gen: coderd/aibridged/proto/aibridged.pb.go (protoc); coderd/database/queries.sql.go, querier.go, models.go, dump.sql, and check_constraint.go (sqlc and the schema dump); coderd/apidoc/* and docs/reference/api/schemas.md (swag); site/src/api/typesGenerated.ts.

Validation

  • go test passes for aibridge/intercept/messages, aibridge/recorder, aibridge/internal/integrationtest, coderd/aibridgedserver, coderd/aibridge/prices/..., coderd/database/migrations, db2sdk, dbgen, and coderd/aibridged/..., and with -run AIModelPrice in enterprise/coderd and enterprise/cli.
  • Site: tsc, Vitest (src/modules/aiModels and the ModelForm stories), Biome, oxlint, and knip pass.
  • Each new test was checked to fail when the behavior it covers is removed.

When the prompt of one upstream request is over the threshold of a
long-context price tier, every token of that request now uses the prices
of the highest such tier. An unset tier price uses the base price.

aibridge sends the prompt size on each token usage record (aibridged API
v1.4); the output-only message_delta record of a stream carries the
prompt size of its message_start. Migration 000628 adds the tiers column
to ai_model_prices and records the prompt size and the applied tier on
aibridge_token_usages. The seed, API, SDK, CLI, and model form carry the
tiers, and the cost controls docs explain the rule.
@ibetitsmike
ibetitsmike added this pull request to stack #30917 October 17, 2026 04:06
@github-actions

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-17T04:30:55.241972Z bd9c5a5 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8cdccf73d8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/aibridgedserver/cost.go
Comment thread aibridge/recorder/types.go
Comment thread codersdk/aimodelprices.go
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: bd9c5a53f3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant