feat(gateway): opt-in model routing on outbound LLM calls, plus Nadir app connection - #441

On process: CONTRIBUTING asks for an issue before a feature PR. I've opened this as code rather than prose because the design questions here are easier to judge against a diff than a description, and because it's cheap for you to close. Happy to move the discussion to an issue and let this sit, or to close it outright if routing doesn't belong in a credential gateway. Disclosure: I work on Nadir.

Adds Nadir to OneCLI in two independent pieces. The first is a plain catalog entry; the second is an opt-in body transform. They can be reviewed, and shipped, separately, and the first is useful without the second.

The problem

Agents pin one model and use it for everything. The same session that needs Opus for a refactor also uses it to rename a variable, reformat JSON, or answer "does this file exist". Those calls are billed at the top tier for work a small model does identically.

OneCLI is already the one component that sees every outbound LLM call from every agent, and already rewrites request bodies (BodyTransform::GitHubCommitTrailer, and hooks::prepare_request_body on the cloud build). That makes it the only place this can be fixed once instead of in every agent.

1. Nadir as a connectable app

An api_key app definition modeled on resend.ts, so the gateway injects a stored Nadir key as X-API-Key on api.getnadir.com. Ordinary credential injection, no routing config, does nothing until a user connects it.

The key field is optional because Nadir's classifier has a keyless tier; a key raises the rate limit and records savings.

2. BodyTransform::NadirModelRoute

On POST /v1/messages and POST /v1/chat/completions on known LLM hosts (policy::is_llm_host), classify the outbound prompt and rewrite model to the tier the operator mapped that bucket to. The agent's code, SDK, and credentials are untouched.

Dispatch follows the existing finalizer pattern in forward.rs:

rules.body_transform
    .or_else(|| crate::apps::body_transform_for_host(super::strip_port(host)))
    .or_else(|| super::transforms::nadir_model_route::transform_for_host(host))

LLM endpoints are reached with a plain secret rather than an OAuth app connection, so they have no provider-registry entry to hang a transform on and must be matched by host. body_transform_for_host is a pure registry mirror of the existing finalizer_for_host; the Nadir-specific host logic lives in the transform module, so apps.rs stays a provider registry.

The classifier is one vendor behind a self-contained module seam. If you'd rather the seam be explicitly generic so a second backend is a sibling file, say so and I'll restructure.

One deliberate divergence from the existing arm

The GitHubCommitTrailer arm falls back to reqwest::Body::from(vec![]) when the transform errors, forwarding an empty body. I did not reuse that. For an LLM request it would silently replace a real prompt with nothing and bill the user for the result.

Instead try_route_model returns the original bytes on every recoverable failure, and ? propagates only the unrecoverable one (an unbufferable body, where the stream is already consumed and there is nothing left to forward). Happy to fix the GitHub arm the same way in a separate PR if you want it.

Behaviour

Anthropic gets a default ladder. OpenAI does not, deliberately: its model names move fast enough that a hardcoded ladder would eventually route to a model the operator never chose, so OpenAI routing stays off until the ladder is set.

The tradeoff you should weigh

When routing is enabled, prompt text of routed requests is sent to api.getnadir.com to be classified. Classification needs the text; there is no way around that.

For a project whose pitch is "agents never see the keys", shipping anything that forwards prompt content deserves an explicit decision rather than a quiet default. That's why it's opt-in, off by default, and disclosed in docs/nadir-integration.md, the README feature bullet, .env.example, and the module doc comment. Nothing is sent while the feature is off. If this alone makes it a no, that's a reasonable call and I'd rather hear it early.

Verification

• pnpm build — pass
• pnpm check (lint + types + format, 9 tasks) — pass
• cargo clippy --all-targets — zero warnings
• cargo test — 523 passed, 0 failed (17 new)
• cargo test -- --ignored live_classifier — passes against the live API: an Opus request with a trivial prompt routes down, and max_tokens / messages survive the rewrite byte for byte

New tests cover flavor detection (wrong host, wrong method, wrong path, query strings), ladder ranking, Anthropic's sibling system field in both string and block-array form, max_tokens clamping in both directions, off-ladder and malformed-body passthrough, and byte-exact passthrough for unroutable requests.

Not included

Per-project configuration. AppConfig already has provider / enabled / settings and would need no migration, but threading it to the gateway means touching the connect path and ConnectResponse, which felt like a separate change. Env config matches how the gateway reads its other settings today.