feat(litellm): add LiteLLM as AI gateway provider by waleedlatif1 · Pull Request #4739 · simstudioai/sim

waleedlatif1 · 2026-05-26T17:17:00Z

Summary

Adds LiteLLM as an AI gateway provider (OpenAI-compatible proxy → 100+ backends), mirroring the vLLM pattern: dynamic model discovery via /v1/models, full OpenAI SDK call path, streaming, tool calling, JSON schema response format
Wires the provider end-to-end: contract + /api/providers/litellm/models route, ProviderModelsLoader, useProviderModels hook, store defaults, ProviderId/ProviderName/AttachmentProvider unions, registry, env vars (LITELLM_BASE_URL, LITELLM_API_KEY)
Closes audit gaps from the original PR feat: add LiteLLM as AI gateway provider #4644: BYOK path now resolves the proxy key (was throwing API key is required for litellm ... for any workflow run), getModelOptions() includes litellm models in the agent block picker, copilot server-side fallback state initialized, copilot edit-workflow message lists LiteLLM, .env.example documents the new env vars
Tests: mirrored vLLM tests for getApiKey('litellm', ...) and the litellm store-bucket guard
Brand: official LiteLLM icon and #040229 color on the provider definition

Type of Change

New feature

Testing

`bun run lint` — clean
`bun run check:api-validation` — passes (route baseline bumped 755 → 756)
`turbo run type-check --filter=sim` — passes
`vitest run blocks/utils.test.ts providers/utils.test.ts` — 179/179 passing
Verified against LiteLLM OpenAPI spec: `GET /v1/models` (APIKeyHeader auth, OpenAI-shape response), `POST /v1/chat/completions` (fully OpenAI-compatible, error envelope, streaming chunks, tool_calls)

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Add API route, contract, query hook case, and ProviderModelsLoader entry so litellm models are fetched and synced to the store on workspace load, matching the vllm/ollama/openrouter/fireworks pattern. Also fixes defaultModel to empty string and adds litellm/ prefix early-return in blocks/utils.ts (reviewer feedback).

Copy-paste artifact from vLLM provider. LiteLLM should only use LITELLM_BASE_URL, not fall back to azureEndpoint which could cause requests to be routed to the wrong server.

- byok.ts: add litellm branch to getApiKeyWithBYOK so workflow block execution can resolve the proxy key instead of throwing "API key is required for litellm ..." - check-api-validation-contracts.ts: bump route baseline 755 -> 756 to account for the new /api/providers/litellm/models route - .env.example: document LITELLM_BASE_URL / LITELLM_API_KEY - copilot edit-workflow validation: include LiteLLM in the list of user-configured prefixed providers shown to the model - providers/utils.ts: drop stray optional-chain on providers.litellm to match the vllm pattern - lint: apply biome formatting fixes (multi-line if, SVG path, multi-line DYNAMIC_MODEL_PROVIDERS)

- blocks/utils.ts getModelOptions(): include litellm models in the combined model dropdown — was previously dropping any proxy-discovered models from the agent block model picker. - get-blocks-metadata-tool.ts mockProvidersState: add litellm bucket so the server-side copilot block-metadata fallback can render model options when the providers store is not initialized. - blocks/utils.test.ts: add litellm to mock providers state (initial + beforeEach reset) and add a parallel store-bucket guard test mirroring the vLLM case. - providers/utils.test.ts: add parallel getApiKey test for litellm.

- icons.tsx: replace the placeholder letterform with the official LiteLLM brand mark embedded as a PNG data URI in an SVG image. - models.ts: set color: #040229 on the litellm provider definition to match the brand background.

vercel · 2026-05-26T17:17:06Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		May 26, 2026 5:31pm

cursor · 2026-05-26T17:17:10Z

PR Summary

Medium Risk
Introduces a large new provider execution path and env-based proxy credentials for workflow LLM calls; scope is broad but mirrors an existing vLLM pattern with tests and graceful empty-model fallbacks.

Overview
Adds LiteLLM as a self-hosted AI gateway provider, following the same pattern as vLLM: OpenAI-compatible proxy, dynamic model discovery from GET /v1/models, and a full execution path (chat completions, streaming, tools, JSON schema).

Configuration & discovery: LITELLM_BASE_URL and optional LITELLM_API_KEY in env and .env.example; new GET /api/providers/litellm/models route and contract; workspace ProviderModelsLoader and React Query hook load and sync litellm/* models into the providers store and agent model picker.

Runtime: New litellm provider module (registry, types, attachments for images, BYOK/key resolution using server env key like vLLM—no per-model user API key). Agent blocks treat litellm/ models like vLLM/Ollama for API-key visibility. Copilot validation and metadata mention LiteLLM-prefixed model IDs.

Tests & tooling: Unit tests for LiteLLM API-key and store-bucket behavior; API validation baseline 755 → 756 routes.

^{Reviewed by Cursor Bugbot for commit b965f70. Configure here.}

greptile-apps · 2026-05-26T17:28:27Z

Greptile Summary

This PR adds LiteLLM as a first-class AI gateway provider, mirroring the existing vLLM integration end-to-end: dynamic model discovery via /v1/models, full OpenAI SDK execution path (streaming, tool calling, JSON schema response format), BYOK key resolution, store/hook wiring, and env-var documentation.

New provider module (providers/litellm/index.ts, utils.ts): ~700 lines implementing initialize(), executeRequest(), streaming, and multi-turn tool loops, all following the established vLLM pattern with vllmUpstreamResponseSchema validation for model-list responses.
Full wiring: ProviderId/ProviderName/AttachmentProvider union types, contract + API route (/api/providers/litellm/models), ProviderModelsLoader, useProviderModels hook, Zustand store defaults, getModelOptions(), and BYOK getApiKey/getApiKeyWithBYOK paths are all updated.
Tests and tooling: mirrored vLLM unit tests for getApiKey and the store-bucket guard; route baseline bumped; .env.example documents the two new env vars.

Confidence Score: 5/5

Safe to merge — the integration is a faithful, end-to-end port of the vLLM provider with no new data-loss, auth-bypass, or broken-contract issues introduced.

The PR follows every established pattern from the vLLM provider: Zod validation of the upstream model list, BYOK key resolution, store/hook wiring, streaming, and multi-turn tool loops are all correctly implemented. The previous audit gaps are addressed and tests cover the new code paths.

No files require special attention — the changes are additive and self-contained within the provider layer.

Important Files Changed

Filename	Overview
apps/sim/providers/litellm/index.ts	New provider: ~700-line implementation of initialize(), executeRequest(), streaming, and tool-loop — faithfully mirrors the vLLM provider with correct schema validation and BYOK wiring.
apps/sim/app/api/providers/litellm/models/route.ts	API route for model discovery: uses vllmUpstreamResponseSchema for validation, handles missing base URL and upstream errors gracefully.
apps/sim/lib/api-key/byok.ts	BYOK path correctly resolves LiteLLM key (env var or user-provided), consistent with the vLLM branch it mirrors.
apps/sim/providers/utils.ts	updateLiteLLMProviderModels uses require('@/providers/models') synchronously, matching the existing vLLM pattern exactly; getApiKey and getBaseModelProviders updated correctly.
apps/sim/providers/models.ts	PROVIDER_DEFINITIONS entry, updateLiteLLMModels, DYNAMIC_MODEL_PROVIDERS, and getBaseModelProviders all updated; litellm correctly excluded from base model map.
apps/sim/providers/attachments.ts	Adds litellm to AttachmentProvider union and image-mime-type switch — consistent with the vLLM/fireworks/ollama cases.
apps/sim/blocks/utils.ts	litellm models included in getModelOptions() and excluded from API-key requirement in shouldRequireApiKeyForModel — correct.

_{Reviews (2): Last reviewed commit: "chore(litellm): validate /v1/models resp..." | Re-trigger Greptile}

…itialize() Match the API route handler — both code paths now run the same vllmUpstreamResponseSchema.parse() over the upstream /v1/models JSON instead of a raw type-cast, so malformed upstream payloads surface a descriptive ZodError instead of a downstream TypeError. Addresses Greptile review feedback on PR #4739.

waleedlatif1 · 2026-05-26T17:32:09Z

@greptile

waleedlatif1 · 2026-05-26T17:32:12Z

@cursor review

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit b965f70. Configure here.}

RheagalFire and others added 7 commits May 26, 2026 10:16

feat: add LiteLLM as AI gateway provider

98926c5

fix: add litellm to attachments, provider store, utils, and block guards

a991d5c

fix: remove azureEndpoint fallback from LiteLLM provider

72d9a9a

Copy-paste artifact from vLLM provider. LiteLLM should only use LITELLM_BASE_URL, not fall back to azureEndpoint which could cause requests to be routed to the wrong server.

feat(litellm): use official LiteLLM brand icon and color

4185090

- icons.tsx: replace the placeholder letterform with the official LiteLLM brand mark embedded as a PNG data URI in an SVG image. - models.ts: set color: #040229 on the litellm provider definition to match the brand background.

vercel Bot deployed to Preview May 26, 2026 17:20 View deployment

greptile-apps Bot reviewed May 26, 2026

View reviewed changes

Comment thread apps/sim/providers/litellm/index.ts Outdated

Comment thread apps/sim/components/icons.tsx

vercel Bot temporarily deployed to Preview May 26, 2026 17:31 Inactive

cursor Bot reviewed May 26, 2026

View reviewed changes

waleedlatif1 merged commit 81bf93b into staging May 26, 2026
10 checks passed

waleedlatif1 deleted the waleedlatif1/audit-pr-4644 branch May 26, 2026 17:38

waleedlatif1 mentioned this pull request May 26, 2026

v0.6.91: file zoom, Zoom KB connector, error classifications, LiteLLM support, executor code cleanup #4743

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(litellm): add LiteLLM as AI gateway provider#4739

feat(litellm): add LiteLLM as AI gateway provider#4739
waleedlatif1 merged 8 commits into
stagingfrom
waleedlatif1/audit-pr-4644

waleedlatif1 commented May 26, 2026

Uh oh!

vercel Bot commented May 26, 2026 •

edited

Loading

Uh oh!

cursor Bot commented May 26, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented May 26, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented May 26, 2026

Uh oh!

waleedlatif1 commented May 26, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

waleedlatif1 commented May 26, 2026

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

greptile-apps Bot commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented May 26, 2026

Uh oh!

waleedlatif1 commented May 26, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vercel Bot commented May 26, 2026 •

edited

Loading

cursor Bot commented May 26, 2026 •

edited

Loading

greptile-apps Bot commented May 26, 2026 •

edited

Loading