Skip to content

improvement(mothership): add workflow lint for custom tool/skills/mcp tool additions to agent block#5199

Open
Sg312 wants to merge 3 commits into
stagingfrom
dev
Open

improvement(mothership): add workflow lint for custom tool/skills/mcp tool additions to agent block#5199
Sg312 wants to merge 3 commits into
stagingfrom
dev

Conversation

@Sg312

@Sg312 Sg312 commented Jun 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds workflow lint for custom tool/skills/mcp tool additions to agent block
Companion: https://github.com/simstudioai/mothership/pull/335

Type of Change

  • Improvement

Testing

Manual

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel

vercel Bot commented Jun 24, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Skipped Skipped Jun 24, 2026 6:19pm

Request Review

@github-actions github-actions Bot added the requires-mothership-merge Has a companion PR on the mothership/copilot side — merge in lockstep label Jun 24, 2026
@github-actions

Copy link
Copy Markdown

⚠️ Cross-repo companion check

One or more companion PRs aren't merged into staging yet. Merging this without them will leave copilot and sim out of sync — merge them in lockstep.

  • simstudioai/mothership#335OPEN, not merged (targets staging) — improvement(workflow-linter): add custom tool validation to workflow linter

@greptile-apps

greptile-apps Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a second tier of agent-block lint checks to the workflow edit pipeline. It extends the existing WorkflowLintUnresolvedReference kind union with custom-tool, mcp-tool, and skill, adds synchronous shape validation for tool-input and skill-input sub-block types inside validateValueForSubBlockType, and introduces collectUnresolvedAgentToolReferences — an async, DB-backed existence checker that runs after the existing credential/resource pass.

  • Shape validation (validateAgentToolEntry / validateAgentSkillEntry) catches malformed entries (missing type: \"custom-tool\", raw OpenAI function schemas, MCP tools without params.serverId, skills using id instead of skillId) and surfaces them as ValidationErrors so the LLM gets actionable feedback before the workflow is saved.
  • DB-backed resolution in collectUnresolvedAgentToolReferences is gated on workspaceId for all three kinds, preventing false positives when workspace context is unavailable; inline custom tools (carrying their own schema) are skipped from the DB check.
  • formatWorkflowLintMessage now renders credential/resource and tool/skill unresolved references as two separate sections, improving LLM readability.

Confidence Score: 5/5

Safe to merge. The new lint path is entirely additive, gated on workspaceId, and wrapped in try/catch so any DB failure degrades gracefully without blocking the edit.

All three DB-backed checks (custom tool, MCP server, skill) are correctly gated on workspaceId to avoid false positives, inline tools that carry their own schema are skipped, and the logger is now at module scope. The only note is that tool/skill DB lookups are issued sequentially rather than in parallel, which is a latency concern but not a correctness one.

validation.ts — the sequential-await pattern in collectUnresolvedAgentToolReferences is worth optimising as agent blocks grow.

Important Files Changed

Filename Overview
apps/sim/lib/copilot/tools/server/workflow/edit-workflow/validation.ts Adds shape validation for tool-input/skill-input sub-block types and a new async DB-backed existence check (collectUnresolvedAgentToolReferences). Logic is sound; custom-tool DB lookups and skill DB lookups are issued sequentially per entry, which could accumulate latency on large agent blocks.
apps/sim/lib/copilot/tools/server/workflow/edit-workflow/index.ts Plumbs the new collectUnresolvedAgentToolReferences into the existing lint pipeline, mirroring the pattern used for collectUnresolvedReferences; both unresolvedReferences and validationErrors are populated correctly.
apps/sim/lib/copilot/tools/server/workflow/edit-workflow/lint.ts Extends WorkflowLintUnresolvedReference.kind union and splits the single unresolved-references section of formatWorkflowLintMessage into separate credential/resource and tool/skill segments with clear copy for the LLM.
apps/sim/lib/copilot/tools/server/workflow/edit-workflow/validation.test.ts Adds comprehensive unit tests for tool-input and skill-input shape validation and for the new collectUnresolvedAgentToolReferences function, including workspaceId-absent path, inline-schema bypass, and resolution happy/sad paths.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[edit-workflow request] --> B[applyOperationsToWorkflowState]
    B --> C[collectUnresolvedReferences\ncredentials / resources]
    C --> D[collectUnresolvedAgentToolReferences\ncustom-tool / mcp-tool / skill]
    D --> E{workspaceId present?}
    E -- No --> F[skip all DB checks\nno false positives]
    E -- Yes --> G[Walk agent blocks]
    G --> H{tool type?}
    H -- custom-tool + no inline schema --> I[getCustomToolById\nDB lookup]
    H -- mcp --> J[validateSelectorIds\nmcp-server-selector]
    H -- other/workflow/integration --> K[pass through]
    I --> L{resolved?}
    J --> L
    L -- No --> M[push UnresolvedSelectorReference\nkind: custom-tool / mcp-tool]
    L -- Yes --> N[no lint error]
    G --> O[Walk skills array]
    O --> P[getSkillById\nbuiltin check + DB]
    P --> Q{resolved?}
    Q -- No --> R[push UnresolvedSelectorReference\nkind: skill]
    Q -- Yes --> S[no lint error]
    M --> T[unresolvedReferences + validationErrors]
    R --> T
    T --> U[formatWorkflowLintMessage\nseparate credential/resource vs tool/skill sections]
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[edit-workflow request] --> B[applyOperationsToWorkflowState]
    B --> C[collectUnresolvedReferences\ncredentials / resources]
    C --> D[collectUnresolvedAgentToolReferences\ncustom-tool / mcp-tool / skill]
    D --> E{workspaceId present?}
    E -- No --> F[skip all DB checks\nno false positives]
    E -- Yes --> G[Walk agent blocks]
    G --> H{tool type?}
    H -- custom-tool + no inline schema --> I[getCustomToolById\nDB lookup]
    H -- mcp --> J[validateSelectorIds\nmcp-server-selector]
    H -- other/workflow/integration --> K[pass through]
    I --> L{resolved?}
    J --> L
    L -- No --> M[push UnresolvedSelectorReference\nkind: custom-tool / mcp-tool]
    L -- Yes --> N[no lint error]
    G --> O[Walk skills array]
    O --> P[getSkillById\nbuiltin check + DB]
    P --> Q{resolved?}
    Q -- No --> R[push UnresolvedSelectorReference\nkind: skill]
    Q -- Yes --> S[no lint error]
    M --> T[unresolvedReferences + validationErrors]
    R --> T
    T --> U[formatWorkflowLintMessage\nseparate credential/resource vs tool/skill sections]
Loading

Reviews (2): Last reviewed commit: "improvement(validation): ensure type is ..." | Re-trigger Greptile

Comment thread apps/sim/lib/copilot/tools/server/workflow/edit-workflow/validation.ts Outdated
Comment thread apps/sim/lib/copilot/tools/server/workflow/edit-workflow/validation.ts Outdated
@Sg312

Sg312 commented Jun 24, 2026

Copy link
Copy Markdown
Collaborator Author

@greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

requires-mothership-merge Has a companion PR on the mothership/copilot side — merge in lockstep

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant