feat: add LiteLLM as AI gateway provider by RheagalFire · Pull Request #4644 · simstudioai/sim

RheagalFire · 2026-05-17T18:22:27Z

Summary

Adds LiteLLM as a provider, enabling access to 100+ LLM backends through a LiteLLM proxy server. Follows the vLLM provider pattern (OpenAI SDK with custom baseURL, dynamic model discovery via /v1/models).

Fixes #2859

Type of Change

Testing

Model discovery:

Discovered models: [ 'litellm/anthropic/claude-sonnet-4-6' ]

Basic completion through LiteLLM proxy:

Content: 4
Model: claude-sonnet-4-6
Tokens: {"completion_tokens":5,"prompt_tokens":20,"total_tokens":25}

Streaming:

Hello!
Full content: Hello!
PASS: streaming works

Tool calling:

Tool call: {"arguments":"{\"city\": \"London\"}","name":"get_weather"}
PASS: tool calling works

All tests use OpenAI SDK with baseURL: http://localhost:4000/v1 pointing at a LiteLLM proxy, which is exactly what the provider does.

Deep-dive bugs found and fixed:

attachments.ts: formatMessagesForProvider missing litellm (would hit exhaustive never check at runtime)
blocks/utils.ts: litellm not excluded from API key requirement check (dynamic providers don't need pre-configured keys)
providers/utils.ts: missing metadata, getBaseModelProviders exclusion, and getApiKey handler
stores/providers: missing ProviderName type and store defaults

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

vercel · 2026-05-17T18:22:35Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		May 17, 2026 6:40pm

cursor · 2026-05-17T18:22:36Z

PR Summary

Medium Risk
Adds a new provider path that makes outbound requests to a configured LiteLLM proxy and plugs it into model discovery, streaming, and tool-calling flows; correctness depends on upstream API compatibility and new env configuration.

Overview
Adds LiteLLM as a first-class AI provider, including a full provider executor (OpenAI SDK w/ custom baseURL) supporting completions, streaming, structured outputs, and tool-calling with timing/cost tracking.

Introduces a new GET /api/providers/litellm/models endpoint for dynamic model discovery (with provider/model blacklists) and wires the client loader/query/store to fetch and sync LiteLLM models into provider definitions.

Updates shared plumbing to recognize litellm/* models: new env vars (LITELLM_BASE_URL, LITELLM_API_KEY), provider registry/types/metadata, API-key visibility logic, attachment support, and a new LitellmIcon.

^{Reviewed by Cursor Bugbot for commit 475820a. Bugbot is set up for automated code reviews on this repo. Configure here.}

greptile-apps · 2026-05-17T18:26:34Z

Greptile Summary

This PR adds LiteLLM as a new AI gateway provider, enabling access to 100+ LLM backends through a LiteLLM proxy server. It follows the existing vLLM pattern throughout: OpenAI SDK with a custom baseURL, dynamic model discovery via /v1/models, env vars LITELLM_BASE_URL/LITELLM_API_KEY, and a full tool-calling loop with streaming support.

Server-side path is complete: litellmProvider is registered, initialize() discovers models from the proxy, and executeRequest correctly handles streaming, tool calls, forced-tool sequencing, and response-format stripping — all mirroring the vLLM implementation.
Frontend model discovery pipeline is missing: There is no /api/providers/litellm/models route, no getLitellmProviderModelsContract, no case 'litellm' in requestProviderModels, and ProviderModelsLoader never calls useSyncProvider('litellm') — so discovered models are never synced to the client and won't appear in the model selector.
updateLiteLLMProviderModels in providers/utils.ts and the 'litellm' branch added to all the right type unions/registries are correct groundwork that will be needed once the API route is added.

Confidence Score: 3/5

Safe to merge for server-side execution, but end-to-end model discovery in the UI is broken without additional files that were not included in this PR.

The server-side execution path (initialize, executeRequest, tool loop, streaming) is complete and correct. However, the client-side model discovery pipeline is entirely absent: no API route, no contract, no query hook case, and no ProviderModelsLoader entry for litellm. Without these, users can type in a model name manually but the model selector will show nothing for LiteLLM, making the feature appear non-functional in the UI.

The four files that need to be added or updated to complete the frontend pipeline: apps/sim/app/api/providers/litellm/models/route.ts (new), apps/sim/lib/api/contracts/providers.ts, apps/sim/hooks/queries/providers.ts, and apps/sim/app/workspace/[workspaceId]/providers/provider-models-loader.tsx.

Important Files Changed

Filename	Overview
apps/sim/providers/litellm/index.ts	New LiteLLM provider implementation mirroring the vLLM pattern; server-side model discovery via /v1/models and full tool-calling loop are correctly implemented, but the frontend model sync pipeline is missing.
apps/sim/app/workspace/[workspaceId]/providers/provider-models-loader.tsx	Not changed by this PR but is missing the litellm entry — no useSyncProvider('litellm') call, causing discovered models to never reach the client-side store or model selector.
apps/sim/providers/models.ts	Adds litellm to PROVIDER_DEFINITIONS, DYNAMIC_MODEL_PROVIDERS, getBaseModelProviders exclusion list, and updateLiteLLMModels helper; defaultModel placeholder 'litellm/generic' is a non-functional model ID.
apps/sim/providers/utils.ts	Adds litellm metadata, updateLiteLLMProviderModels, and getApiKey handler following the vLLM pattern; updateLiteLLMProviderModels is defined but never called from any production path yet.
apps/sim/providers/litellm/utils.ts	Thin wrapper around createOpenAICompatibleStream following the vLLM utils pattern exactly.
apps/sim/providers/registry.ts	Correctly registers litellmProvider in the provider registry alongside other providers.
apps/sim/providers/attachments.ts	Adds litellm to AttachmentProvider union, PROVIDER_SUPPORTED_LABELS, getAttachmentProvider, and isMimeTypeSupportedByProvider; correctly treats it as an image-capable provider.
apps/sim/providers/types.ts	Adds 'litellm' to the ProviderId union type correctly.
apps/sim/stores/providers/store.ts	Adds litellm to the default providers state in the Zustand store.
apps/sim/stores/providers/types.ts	Adds 'litellm' to the ProviderName union type.
apps/sim/blocks/utils.ts	Correctly excludes litellm from the API key requirement check alongside ollama and vllm.
apps/sim/lib/core/config/env.ts	Adds LITELLM_BASE_URL and LITELLM_API_KEY env vars following the same schema pattern as VLLM_BASE_URL/VLLM_API_KEY.
apps/sim/components/icons.tsx	Adds LitellmIcon SVG component with a simple LiteLLM logo representation.

Sequence Diagram

sequenceDiagram
    participant Client as Browser (ProviderModelsLoader)
    participant API as /api/providers/litellm/models ❌ missing
    participant Init as litellmProvider.initialize()
    participant Proxy as LiteLLM Proxy (/v1/models)
    participant Exec as litellmProvider.executeRequest()

    Note over Client,API: Frontend model discovery (NOT implemented)
    Client--xAPI: GET /api/providers/litellm/models (no route exists)

    Note over Init,Proxy: Server-side initialization (works correctly)
    Init->>Proxy: GET /v1/models
    Proxy-->>Init: "{ data: [{ id: 'anthropic/claude-...' }] }"
    Init->>Init: "this.models = ['litellm/anthropic/...']"
    Init->>Init: setProviderModels('litellm', models)

    Note over Client,Exec: Execution flow (works correctly)
    Client->>Exec: "POST /api/providers { provider: 'litellm', model: 'litellm/...' }"
    Exec->>Proxy: "chat.completions.create (OpenAI SDK, baseURL=LITELLM_BASE_URL/v1)"
    Proxy-->>Exec: ChatCompletion / Stream
    Exec-->>Client: ProviderResponse / StreamingExecution

_{Reviews (1): Last reviewed commit: "fix: add litellm to attachments, provide..." | Re-trigger Greptile}

greptile-apps · 2026-05-17T18:26:37Z

+    name: 'LiteLLM',
+    icon: LitellmIcon,
+    description: 'LiteLLM proxy with an OpenAI-compatible API',
+    defaultModel: 'litellm/generic',


The defaultModel value 'litellm/generic' is not a real model ID and will produce a confusing error when it is used as the initial default before any models are discovered (e.g. if the proxy is unreachable). Aligning with the vLLM pattern of leaving defaultModel as an empty string avoids sending a bogus model ID to the proxy.

Suggested change

defaultModel: 'litellm/generic',

defaultModel: '',

Add API route, contract, query hook case, and ProviderModelsLoader entry so litellm models are fetched and synced to the store on workspace load, matching the vllm/ollama/openrouter/fireworks pattern. Also fixes defaultModel to empty string and adds litellm/ prefix early-return in blocks/utils.ts (reviewer feedback).

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 475820a. Configure here.}

cursor · 2026-05-17T18:44:47Z

+      stream: !!request.stream,
+    })
+
+    const baseUrl = (request.azureEndpoint || env.LITELLM_BASE_URL || '').replace(/\/$/, '')


LiteLLM provider incorrectly uses azureEndpoint as base URL

Low Severity

The LiteLLM provider falls back to request.azureEndpoint before env.LITELLM_BASE_URL when determining its base URL. This is a copy-paste artifact from the vLLM provider. If a block previously configured with an Azure endpoint is switched to use a LiteLLM model and still carries the azureEndpoint field in its request payload, the LiteLLM request would be sent to the Azure endpoint instead of the LiteLLM proxy, causing unexpected failures or sending data to an unintended server.

^{Reviewed by Cursor Bugbot for commit 475820a. Configure here.}

RheagalFire added 2 commits May 17, 2026 23:23

feat: add LiteLLM as AI gateway provider

fe39f57

fix: add litellm to attachments, provider store, utils, and block guards

9667e88

vercel Bot temporarily deployed to Preview May 17, 2026 18:22 Inactive

greptile-apps Bot reviewed May 17, 2026

View reviewed changes

cursor Bot reviewed May 17, 2026

View reviewed changes

Comment thread apps/sim/blocks/utils.ts

Comment thread apps/sim/providers/utils.ts

vercel Bot temporarily deployed to Preview May 17, 2026 18:40 Inactive

cursor Bot reviewed May 17, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add LiteLLM as AI gateway provider#4644

feat: add LiteLLM as AI gateway provider#4644
RheagalFire wants to merge 3 commits into
simstudioai:mainfrom
RheagalFire:feat/add-litellm-provider

RheagalFire commented May 17, 2026

Uh oh!

vercel Bot commented May 17, 2026 •

edited

Loading

Uh oh!

cursor Bot commented May 17, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented May 17, 2026

Uh oh!

greptile-apps Bot May 17, 2026

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot May 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

RheagalFire commented May 17, 2026

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented May 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot commented May 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

greptile-apps Bot commented May 17, 2026

Greptile Summary

Confidence Score: 3/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps Bot May 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot May 17, 2026

Choose a reason for hiding this comment

LiteLLM provider incorrectly uses azureEndpoint as base URL

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented May 17, 2026 •

edited

Loading

cursor Bot commented May 17, 2026 •

edited

Loading

LiteLLM provider incorrectly uses `azureEndpoint` as base URL