The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Jun 18, 2026 2:52am

PR Summary

High Risk
Changes API key resolution, customer billing, and cost/metrics for all hosting-configured LLM providers when the flag is enabled; streaming cost settlement affects every provider using createStreamingExecution.

Overview
Adds a flag-gated (hosted-key-llm / HOSTED_KEY_LLM) path so hosted LLM calls use the same hosted-key framework as tools: BYOK → user key → platform pool, with round-robin key selection and no queue/rate limits (mode: 'none').

Key resolution & config: getApiKeyWithBYOK gains an early unified branch driven by per-provider hosting in PROVIDER_DEFINITIONS (envKeyPrefix, byokProviderId). When the flag is off or no pool keys exist, behavior stays on the legacy getRotatingApiKey / per-provider paths. Env key discovery now probes {PREFIX}_1.._N when _COUNT is missing (without picking up a bare ambient key).

Billing & telemetry: New shared helpers in hosted-cost.ts (calculateHostedCost, classifyHostedKeyFailure, emitHostedKeyUsage); tools delegate to them. The provider entrypoint records used / failed / cost for pool keys, bills non-BYOK pool usage even outside getHostedModels(), and passes hostedKey on the request for streaming.

Streaming: createStreamingExecution wraps streams to settle token cost on drain via settleStreamingLlmCost (cost multiplier + cached-input pricing on the hosted path), including post-tool streams that never call finalizeTiming. Providers thread hostedKey and isCachedInput(request.context).

^{Reviewed by Cursor Bugbot for commit ed64de9. Bugbot is set up for automated code reviews on this repo. Configure here.}

Greptile Summary

This PR routes LLM provider calls through the shared hosted-key framework (BYOK-first → platform key pool via acquireKey with mode: 'none'), gated by the hosted-key-llm feature flag. It centralises streaming cost settlement into settleStreamingLlmCost (applying the cost multiplier the per-provider drain callbacks previously omitted), emits recordUsed/recordCostCharged/recordFailed hosted-key metrics for both streaming and non-streaming LLM calls, and consolidates classifyHostedKeyFailure and calculateHostedCost into a shared hosted-cost module used by both tools and providers.

• Rate limiter: adds mode: 'none' (NoRateLimit) to HostedKeyRateLimitConfig and a selectKey helper so acquireKey bypasses the FIFO queue and token buckets entirely, plus upgrades resolveEnvKeys to probe _1.._N without requiring _COUNT.
• Key resolution: getApiKeyWithBYOK gains a flag-gated top-of-function block that checks BYOK first, then acquires from the platform pool, then falls through to the unchanged legacy path.
• Streaming cost seam: createStreamingExecution's finalizeTiming hook now calls settleStreamingLlmCost when hostedKey is set; Gemini's bespoke drain callbacks call it directly; both paths use output.tokens already written by the provider before finalizeTiming is invoked.

Confidence Score: 4/5

The non-streaming and simple-streaming paths are correctly wired end-to-end; tool-path streaming for most providers still lacks the finalizeTiming call that would settle the hosted-key cost metric and apply the cost multiplier on drain.

The flag-gated key resolution, rate-limiter mode:'none', and centralized cost settlement in settleStreamingLlmCost are all sound. The one structural gap — tool-path streaming providers omitting finalizeTiming in their createStream callbacks — was flagged in a prior review cycle and remains unaddressed.

apps/sim/providers/streaming-execution.ts and the tool-path createStream callbacks in openai/core.ts, baseten/index.ts, fireworks/index.ts, together/index.ts, mistral/index.ts, and ollama/core.ts

Important Files Changed

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant E as executeProviderRequest
    participant B as getApiKeyWithBYOK
    participant RL as HostedKeyRateLimiter
    participant P as Provider.executeRequest
    participant M as hostedKeyMetrics

    E->>B: provider, model, workspaceId, apiKey, userId
    B->>B: isHosted and workspaceId and flag on?
    alt BYOK workspace key exists
        B-->>E: apiKey isBYOK true
    else platform key available
        B->>RL: acquireKey mode none
        RL-->>B: success key envVarName
        B-->>E: apiKey isBYOK false hostedKeyEnvVar
    else fallback
        B-->>E: legacy key no hostedKeyEnvVar
    end
    E->>P: executeRequest sanitizedRequest
    alt throws
        E->>M: recordFailed
        E-->>E: re-throw
    else StreamingExecution
        E->>M: recordUsed
        Note over P,M: On stream drain
        P->>M: recordCostCharged via settleStreamingLlmCost
    else ProviderResponse
        E->>M: emitHostedKeyUsage
    end

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant E as executeProviderRequest
    participant B as getApiKeyWithBYOK
    participant RL as HostedKeyRateLimiter
    participant P as Provider.executeRequest
    participant M as hostedKeyMetrics

    E->>B: provider, model, workspaceId, apiKey, userId
    B->>B: isHosted and workspaceId and flag on?
    alt BYOK workspace key exists
        B-->>E: apiKey isBYOK true
    else platform key available
        B->>RL: acquireKey mode none
        RL-->>B: success key envVarName
        B-->>E: apiKey isBYOK false hostedKeyEnvVar
    else fallback
        B-->>E: legacy key no hostedKeyEnvVar
    end
    E->>P: executeRequest sanitizedRequest
    alt throws
        E->>M: recordFailed
        E-->>E: re-throw
    else StreamingExecution
        E->>M: recordUsed
        Note over P,M: On stream drain
        P->>M: recordCostCharged via settleStreamingLlmCost
    else ProviderResponse
        E->>M: emitHostedKeyUsage
    end

_{Reviews (3): Last reviewed commit: "Merge remote-tracking branch 'origin/sta..." | Re-trigger Greptile}

@greptile review

@greptile review

Addressed review:

• Bugbot (user key bypassed by pool): getApiKeyWithBYOK now returns a user-provided request.apiKey (not billed) before acquiring a platform pool key, matching tool hosted-key precedence (5f9046e57).
• Greptile (inaccurate comment): corrected the streaming-cost comment to note gemini settles via its bespoke finalizer, not createStreamingExecution.

Also fixed CI: completed provider test mocks (@/providers/utils isCachedInput; @/lib/core/config/env getEnv/isTruthy/isFalsy) that the new streaming wiring exposed. Full suite green locally (8868 tests).

Two more Bugbot findings reviewed:

• High — tool streaming skips hosted settlement: valid and fixed (8d6b768c9). The post-tool streaming paths (createStream: ({ output }) => …) never call finalizeTiming(), so hooking settlement there missed them. Moved settlement to fire on actual stream drain in createStreamingExecution (wraps the returned stream; single point, covers simple + post-tool paths for every provider). Added a regression test that a stream which never calls finalizeTiming still settles cost + emits recordCostCharged on drain.

• Medium — hosted metrics omit tool costs: by design, not a bug. hostedKeyMetrics.recordCostCharged for an LLM call is the LLM token cost (label tool: <model>). Tool costs are emitted separately by each hosted tool's own path (label tool: <tool.id>); folding sumToolCosts into the LLM metric would double-count hosted tools and mis-attribute cost to the model dimension. Streaming and non-streaming are consistent here. response.cost.total (LLM+tools) remains the billing/trace total — that's a separate channel from this CloudWatch observability metric.

Addressed the two streaming-metric symmetry findings (ed64de979):

• The stream-drain wrapper now records a hosted-key failure on mid-stream error (via classifyHostedKeyFailure), so a failed hosted stream emits recordFailed to match the recordUsed fired at dispatch. A client cancel runs neither (it's an abort, not a key failure), and cost is intentionally not settled on failure (no charge for a failed/partial response). Added a regression test for the error path.
• Net streaming semantics: recordUsed = attempt, recordCostCharged = successful completion, recordFailed = mid-stream error — so completed = Used − Failed (no silent over-count).

Full suite green (8870 tests), tsc 0, lint 16/16.