Skip to content

improvement(media-gen): retire vision block, add hosted key for fal ai for image/video gen, search visibility in cmd-k #4684

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
icecrasher321 wants to merge 2 commits into
base: staging
Choose a base branch
from
Open

improvement(media-gen): retire vision block, add hosted key for fal ai for image/video gen, search visibility in cmd-k #4684

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dacdbe3
improvement(media-gen): retire vision block, add hosted key for fal a…
icecrasher321 May 20, 2026
3c7fd57
address comments
icecrasher321 May 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions apps/docs/content/docs/en/tools/azure_devops.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -280,7 +280,7 @@ Get the execution timeline for an Azure DevOps build — every stage, job, and t
| ↳ `warningCount` | number | Number of warnings |
| ↳ `startTime` | string | ISO 8601 start timestamp |
| ↳ `finishTime` | string | ISO 8601 finish timestamp |
| ↳ `failedRecords` | array | Subset of records where result === "failed" — use logId to fetch logs |
| ↳ `failedRecords` | array | Subset of records where result is failed, partiallySucceeded, or succeededWithIssues — use logId to fetch logs |
| ↳ `id` | string | Record GUID |
| ↳ `name` | string | Step name |
| ↳ `type` | string | Stage \| Phase \| Job \| Task |
Expand Down Expand Up @@ -333,7 +333,8 @@ Execute a WIQL query to search for work items in Azure DevOps and return full fi
| --------- | ---- | ----------- |
| `content` | string | Human-readable summary of matching work items |
| `metadata` | object | Work items metadata |
| ↳ `count` | number | Number of work items returned |
| ↳ `count` | number | Number of work items returned \(after hydration\) |
| ↳ `totalMatched` | number | Total number of work items matched by the WIQL query before hydration |
| ↳ `workItems` | array | Array of work item details |
| ↳ `id` | number | Work item ID |
| ↳ `title` | string | Work item title |
Expand Down Expand Up @@ -372,15 +373,15 @@ Fetch full details of a single work item by ID from Azure DevOps, including titl

### `azure_devops_get_work_items_batch`

Fetch full details for multiple work items by ID from Azure DevOps in a single call. Pass comma-separated IDs (e.g.
Fetch full details for multiple work items by ID from Azure DevOps. Pass comma-separated IDs (e.g.

#### Input

| Parameter | Type | Required | Description |
| --------- | ---- | -------- | ----------- |
| `organization` | string | Yes | Azure DevOps organization name |
| `project` | string | Yes | Azure DevOps project name |
| `ids` | string | Yes | Comma-separated work item IDs to fetch \(e.g. "123,456,789"\). Maximum 200 IDs. |
| `ids` | string | Yes | Comma-separated work item IDs to fetch \(e.g. "123,456,789"\). Lists longer than 200 IDs are chunked automatically. |

#### Output

Expand All @@ -389,6 +390,7 @@ Fetch full details for multiple work items by ID from Azure DevOps in a single c
| `content` | string | Human-readable summary of the fetched work items |
| `metadata` | object | Work items metadata |
| ↳ `count` | number | Number of work items returned |
| ↳ `totalRequested` | number | Total number of IDs requested \(across all chunks\) |
| ↳ `workItems` | array | Array of work item details |
| ↳ `id` | number | Work item ID |
| ↳ `title` | string | Work item title |
Expand Down
1 change: 0 additions & 1 deletion apps/docs/content/docs/en/tools/meta.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,6 @@
"upstash",
"vercel",
"video_generator",
"vision",
"wealthbox",
"webflow",
"whatsapp",
Expand Down
60 changes: 0 additions & 60 deletions apps/docs/content/docs/en/tools/vision.mdx
View file
Open in desktop

This file was deleted.

8 changes: 4 additions & 4 deletions apps/docs/content/docs/en/triggers/azure_devops.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ Trigger workflow when an Azure DevOps build fails, is canceled, or partially suc
| `branch` | string | Source branch name \(refs/heads/ prefix stripped\) |
| `commitSha` | string | Source commit SHA |
| `triggeredBy` | string | Display name of the person who triggered the build |
| `triggeredByEmail` | string | Email/unique name of the person who triggered the build |
| `triggeredByEmail` | string | Email/unique name of the person who triggered the build, or null if not set |
| `startTime` | string | Build start time \(ISO 8601\) |
| `finishTime` | string | Build finish time \(ISO 8601\) |
| `buildUrl` | string | API URL for the build resource |
Expand Down Expand Up @@ -72,12 +72,12 @@ Trigger workflow when a work item is created in Azure DevOps
| `workItemType` | string | Work item type for Basic process \(e.g. Issue, Task, Epic\) |
| `title` | string | Work item title |
| `state` | string | Work item state for Basic process \(e.g. To Do, Doing, Done\) |
| `createdBy` | string | Display name of the creator |
| `assignedTo` | string | Assignee display name, or empty string if unassigned |
| `createdBy` | string | Display name of the creator, or null if not set |
| `assignedTo` | string | Assignee display name, or null if unassigned |
| `priority` | number | Priority \(1–4\), or 0 if not set |
| `areaPath` | string | Area path |
| `iterationPath` | string | Iteration path |
| `description` | string | Work item description \(HTML\), or empty string if not set |
| `description` | string | Work item description \(HTML\), or null if not set |
| `projectName` | string | Azure DevOps project name |
| `workItemUrl` | string | API URL for the work item resource |

2 changes: 0 additions & 2 deletions apps/sim/app/(landing)/integrations/data/icon-mapping.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,6 @@ import {
EvernoteIcon,
ExaAIIcon,
ExtendIcon,
EyeIcon,
FathomIcon,
FindymailIcon,
FirecrawlIcon,
Expand Down Expand Up @@ -405,7 +404,6 @@ export const blockTypeToIconMap: Record<string, IconComponent> = {
upstash: UpstashIcon,
vercel: VercelIcon,
video_generator_v3: VideoIcon,
vision_v2: EyeIcon,
wealthbox: WealthboxIcon,
webflow: WebflowIcon,
whatsapp: WhatsAppIcon,
Expand Down
26 changes: 2 additions & 24 deletions apps/sim/app/(landing)/integrations/data/integrations.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1938,7 +1938,7 @@
},
{
"name": "Get Work Items Batch",
"description": "Fetch full details for multiple work items by ID from Azure DevOps in a single call. Pass comma-separated IDs (e.g. "
"description": "Fetch full details for multiple work items by ID from Azure DevOps. Pass comma-separated IDs (e.g. "
},
{
"name": "Create Work Item",
Expand Down Expand Up @@ -4156,10 +4156,6 @@
"name": "Fetch",
"description": "Fetch and parse a file from a URL with optional custom headers."
},
{
"name": "Get",
"description": "Get a workspace file object from a selected file or canonical workspace file ID."
},
{
"name": "Write",
"description": "Create a new workspace file. If a file with the same name already exists, a numeric suffix is added (e.g., "
Expand All @@ -4169,7 +4165,7 @@
"description": "Append content to an existing workspace file. The file must already exist. Content is added to the end of the file."
}
],
"operationCount": 5,
"operationCount": 4,
"triggers": [],
"triggerCount": 0,
"authType": "none",
Expand Down Expand Up @@ -14253,24 +14249,6 @@
"integrationTypes": ["ai", "design"],
"tags": ["video-generation", "llm"]
},
{
"type": "vision_v2",
"slug": "vision",
"name": "Vision",
"description": "Analyze images with vision models",
"longDescription": "Integrate Vision into the workflow. Can analyze images with vision models.",
"bgColor": "#4D5FFF",
"iconName": "EyeIcon",
"docsUrl": "https://docs.sim.ai/tools/vision",
"operations": [],
"operationCount": 0,
"triggers": [],
"triggerCount": 0,
"authType": "api-key",
"category": "tools",
"integrationTypes": ["ai", "documents"],
"tags": ["llm", "document-processing", "ocr"]
},
{
"type": "wealthbox",
"slug": "wealthbox",
Expand Down
15 changes: 15 additions & 0 deletions apps/sim/app/api/tools/image/route.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import {
import { generateRequestId } from '@/lib/core/utils/request'
import { getBaseUrl } from '@/lib/core/utils/urls'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import { type FalAICostMetadata, getFalAICostMetadata } from '@/lib/tools/falai-pricing'

const logger = createLogger('ImageProxyAPI')

Expand All @@ -42,6 +43,7 @@ interface GeneratedImageResult {
revisedPrompt?: string
seed?: number
jobId?: string
falaiCost?: FalAICostMetadata
}

interface StoredImageResponse {
Expand All @@ -61,6 +63,8 @@ interface StoredImageResponse {
jobId?: string
contentType: string
}
__falaiCostDollars?: number
__falaiBilling?: FalAICostMetadata
}

export const POST = withRouteHandler(async (request: NextRequest) => {
Expand Down Expand Up @@ -869,6 +873,13 @@ async function generateWithFalAI(
revisedPrompt: getStringProperty(resultData, 'revised_prompt'),
seed: getNumberProperty(resultData, 'seed'),
jobId: falRequestId,
falaiCost: body.useHostedCostTracking
? await getFalAICostMetadata({
apiKey,
endpointId: modelConfig.endpoint,
requestId: falRequestId,
})
: undefined,
Comment thread
icecrasher321 marked this conversation as resolved.
Comment thread
icecrasher321 marked this conversation as resolved.
}
}

Expand Down Expand Up @@ -926,6 +937,8 @@ async function storeGeneratedImage(
jobId: imageResult.jobId,
contentType: imageResult.contentType,
},
__falaiCostDollars: imageResult.falaiCost?.costDollars,
__falaiBilling: imageResult.falaiCost,
}
}

Expand Down Expand Up @@ -958,5 +971,7 @@ async function storeGeneratedImage(
jobId: imageResult.jobId,
contentType: imageResult.contentType,
},
__falaiCostDollars: imageResult.falaiCost?.costDollars,
__falaiBilling: imageResult.falaiCost,
}
}
25 changes: 24 additions & 1 deletion apps/sim/app/api/tools/video/route.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import { getValidationErrorMessage, parseRequest, validationErrorResponse } from
import { checkInternalAuth } from '@/lib/auth/hybrid'
import { getMaxExecutionTimeout } from '@/lib/core/execution-limits'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import { type FalAICostMetadata, getFalAICostMetadata } from '@/lib/tools/falai-pricing'
import { downloadFileFromStorage } from '@/lib/uploads/utils/file-utils.server'
import { assertToolFileAccess } from '@/app/api/files/authorization'
import type { UserFile } from '@/executor/types'
Expand Down Expand Up @@ -102,6 +103,7 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
let height: number | undefined
let jobId: string | undefined
let actualDuration: number | undefined
let falaiCost: FalAICostMetadata | undefined

if (body.visualReference) {
const denied = await assertToolFileAccess(
Expand Down Expand Up @@ -200,6 +202,7 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
resolution,
body.promptOptimizer,
body.generateAudio,
body.useHostedCostTracking === true,
requestId,
logger
)
Expand All @@ -208,6 +211,7 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
height = result.height
jobId = result.jobId
actualDuration = result.duration
falaiCost = result.falaiCost
} else {
return NextResponse.json({ error: `Unknown provider: ${provider}` }, { status: 400 })
}
Expand Down Expand Up @@ -262,6 +266,8 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
provider,
model: model || 'default',
jobId,
__falaiCostDollars: falaiCost?.costDollars,
__falaiBilling: falaiCost,
})
}

Expand Down Expand Up @@ -294,6 +300,8 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
provider,
model: model || 'default',
jobId,
__falaiCostDollars: falaiCost?.costDollars,
__falaiBilling: falaiCost,
})
} catch (error) {
logger.error(`[${requestId}] Video proxy error:`, error)
Expand Down Expand Up @@ -1082,9 +1090,17 @@ async function generateWithFalAI(
resolution: string | undefined,
promptOptimizer: boolean | undefined,
generateAudio: boolean | undefined,
useHostedCostTracking: boolean,
requestId: string,
logger: ReturnType<typeof createLogger>
): Promise<{ buffer: Buffer; width: number; height: number; jobId: string; duration: number }> {
): Promise<{
buffer: Buffer
width: number
height: number
jobId: string
duration: number
falaiCost?: FalAICostMetadata
}> {
logger.info(`[${requestId}] Starting Fal.ai generation with model: ${model}`)

const modelConfig = FALAI_MODEL_CONFIGS[model]
Expand Down Expand Up @@ -1229,6 +1245,13 @@ async function generateWithFalAI(
height,
jobId: requestIdFal,
duration: getNumberProperty(videoOutput, 'duration') || duration || 5,
falaiCost: useHostedCostTracking
? await getFalAICostMetadata({
apiKey,
endpointId: modelConfig.endpoint,
requestId: requestIdFal,
})
: undefined,
Comment thread
icecrasher321 marked this conversation as resolved.
}
}

Expand Down
8 changes: 8 additions & 0 deletions apps/sim/app/workspace/[workspaceId]/settings/components/byok/byok.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import {
FireworksIcon,
GeminiIcon,
GoogleIcon,
ImageIcon,
JinaAIIcon,
LinkupIcon,
MistralIcon,
Expand Down Expand Up @@ -85,6 +86,13 @@ const PROVIDERS: {
description: 'LLM calls',
placeholder: 'Enter your Fireworks API key',
},
{
id: 'falai',
name: 'Fal.ai',
icon: ImageIcon,
description: 'Image and video generation',
placeholder: 'Enter your Fal.ai API key',
},
{
id: 'firecrawl',
name: 'Firecrawl',
Expand Down
Loading
Loading