feat: automate weekly AI model price book refresh - #28146
Open
evgeniy-scherbina wants to merge 3 commits into
Open
feat: automate weekly AI model price book refresh#28146evgeniy-scherbina wants to merge 3 commits into
evgeniy-scherbina wants to merge 3 commits into
Conversation
The price book is regenerated by hand with `make gen/aibridge-prices`, so newly launched models stay unpriced and changed prices stay stale until someone remembers to run it. Add a weekly workflow that regenerates both artifacts from live models.dev data and opens a pull request when the output changes. The refresh is never merged automatically: prices are customer-visible cost numbers, so every change goes through human review. A generator failure, which happens by design when upstream drops a pinned or curated model, is announced in Slack. The pull request body carries a generated summary of models added, models removed, and prices changed, rendered by a new deterministic scripts/aibridgepricesdiff tool. Add CODEOWNERS entries for the price book, the frontend catalog, and both generators so the right reviewer is requested on every refresh.
The pull request diff already shows exact prices for every added, removed, and changed row, so repeating them in the body duplicated what a reviewer can read directly. Keep the counts and the model lists, which answer what moved without restating the diff.
… fields The per-field count did not match the model list below it and needed a qualifier to reconcile. Counting models keeps every figure on the same unit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The price book is regenerated by hand with
make gen/aibridge-prices, so keeping it current depends on someone remembering to run it. Newly launched models stay unpriced and changed prices stay stale until the next manual run.This adds a weekly workflow that regenerates both artifacts from live models.dev data and opens a pull request when the output changes.
For scale, a dry run of the generator against
maintoday produced 28 models added, 11 removed, and 92 price changes.What it does
workflow_dispatch. Exits without opening anything when regeneration produces no diff.prices.jsonandknownModelsGenerated.jsontogether, since both come from a single upstream snapshot.overrides.jqor curated incuration.json, and a silently red weekly cron would defeat the point.scripts/aibridgepricesdiffrenders the PR body summary: counts, then the models added, removed, and repriced. Exact figures stay in the diff rather than being restated in the body. It is deterministic and unit tested rather than AI-generated, so the model lists cannot drift from the file.CODEOWNERSnow covers the price book, the frontend catalog, and both generators.Follow-ups needed before the first run
AIBRIDGE_PRICES_SLACK_WEBHOOKrepository secret pointing at#feat-ai-gateway. The notification step skips with a warning while the secret is absent, so the workflow stays green until then.secrets.CDRCI_GITHUB_TOKEN. The defaultGITHUB_TOKENcannot trigger workflow runs, which would leave the refresh PR with no CI signal on a file that feeds cost calculation.Notes
scripts/biome_format.sh, which runspnpm exec biomefromsite/. Without it that script warns and exits 0, which would commit an unformatted file.Testing
go test ./scripts/aibridgepricesdiff/covers add, remove, change, price-unset, cross-provider collisions, sorting determinism, per-model collapsing of multi-field changes, and rendering.make lint/actions/actionlint,make lint/actions/zizmor, andmake pre-commitpass.make gen/aibridge-pricesend to end and fed the result through the diff tool to verify the generated summary against real upstream output.Implementation plan and decision log
Goal
Refresh
make gen/aibridge-pricesoutput on a schedule and open a human-reviewed PR when it changes, so releases never ship a stale price book.Schedule
Weekly, Mondays 09:00 UTC. A monthly run was considered first. Release ships Tuesday morning and the freeze is the last Tuesday before EOM, so a monthly run on the 19th or 20th collapses to 1-2 usable business days in roughly 10 of 132 months: February, when the release Tuesday is the 22nd and the run day falls on a weekend. Measured across 2026-2036:
The median is 6 days for every candidate, so moving to weekly costs nothing and removes the bad tail. The shipped book is never more than 7 days stale.
Branch strategy
One fixed branch, force-pushed, rather than a new branch and PR per run.
prices.jsonis a full regeneration from the current upstream snapshot, not an incremental patch, so two open refresh PRs are the same file at two points in time and the older one is strictly wrong. A single PR makes merging a stale snapshot impossible, keeps reviewer load at one PR, and avoids a full CI run per week per stale PR. Force-pushing also dismisses a stale approval, which is the desired behavior when the numbers have changed.Summary generation
Deterministic Go tool, not an AI summary. An LLM summarizing a JSON diff can drop or invent a row, and would be non-reproducible run to run. A jq implementation was considered and rejected as effectively untestable. The body lists which models moved; the diff in the Files tab remains the source of truth for figures.
Token
secrets.CDRCI_GITHUB_TOKEN, the cdrci machine user already used for bot-authored PRs inrelease.yamland cited for exactly this reason in the commented-outupdate-flakejob inci.yaml. A dedicated GitHub App scoped to this repo would be tighter, but needs org admin to create.Alerting
A dedicated
AIBRIDGE_PRICES_SLACK_WEBHOOKrather than reusingCI_FAILURE_SLACK_WEBHOOK. The repo already splits alerting per domain (docs, security, dependabot each have their own webhook), and the CI-failure channel carries a user mention plusvars.BLINK_CI_FAILURE_PROMPTthat feeds an automated triage flow this payload does not belong in.Out of scope
Triggering a refresh when a release branch is cut. The existing cherry-pick label already gets a late refresh into a release, so a separate mechanism is unnecessary.
Closes AIGOV-578.
Authored by Coder Agents on behalf of @evgeniy-scherbina.