diff --git a/.agents/skills/add-block/SKILL.md b/.agents/skills/add-block/SKILL.md index 378c9b68799..091a8307c31 100644 --- a/.agents/skills/add-block/SKILL.md +++ b/.agents/skills/add-block/SKILL.md @@ -993,11 +993,15 @@ After adding or changing one, run: ```bash bun run scripts/generate-docs.ts bun run integration-catalog:check +bun run docs:check ``` The catalog check independently derives deployment metadata from the executable block registry and -compares it with the committed `apps/sim/lib/integrations/integrations.json`. Review the generated -diff and keep only intentional changes. +compares it with the committed `apps/sim/lib/integrations/integrations.json`. `docs:check` re-renders +every generated docs artifact in memory and fails on any committed file that differs — it runs in CI +via `check:audits`, so commit the full generator output. If the generator also trues up pages an +earlier PR left stale, commit that catch-up too; reverting it as "unrelated drift" makes `docs:check` +fail. ## Checklist Before Finishing - [ ] `integrationType` is set to the correct `IntegrationType` enum value @@ -1018,6 +1022,7 @@ diff and keep only intentional changes. - [ ] If any tool was added, changed or removed alongside the block: ran `bun run tool-metadata:generate` and committed the artifacts - [ ] Ran `bun run scripts/generate-docs.ts`, reviewed the generated diff, and committed the integration catalog changes - [ ] `bun run integration-catalog:check` passes +- [ ] `bun run docs:check` passes (CI gate — fails on any stale generated docs page) - [ ] If icon missing: asked user to provide SVG - [ ] If triggers exist: `triggers` config set, trigger subBlocks spread - [ ] Optional/rarely-used fields set to `mode: 'advanced'` diff --git a/.agents/skills/add-integration/SKILL.md b/.agents/skills/add-integration/SKILL.md index 3369828d6f2..22ed2ef4195 100644 --- a/.agents/skills/add-integration/SKILL.md +++ b/.agents/skills/add-integration/SKILL.md @@ -561,6 +561,7 @@ Run the documentation generator: ```bash bun run scripts/generate-docs.ts bun run integration-catalog:check +bun run docs:check ``` This creates `apps/docs/content/docs/en/integrations/{service}.mdx` — one page per service carrying the block's Actions and, if it has one, its Triggers section. Never hand-edit generated pages; the only editable region is the `{/* MANUAL-CONTENT */}` block (see `scripts/README.md`). @@ -651,6 +652,9 @@ If creating V2 versions (API-aligned outputs): - [ ] Verified docs file created - [ ] Reviewed and committed the generated `apps/sim/lib/integrations/integrations.json` change - [ ] `bun run integration-catalog:check` passes +- [ ] `bun run docs:check` passes — CI fails on stale generated docs, so commit the full generator + output, including catch-up regeneration for pages another PR left stale (never revert it as + "unrelated drift") ### Final Validation (Required) - [ ] Read every tool file and cross-referenced inputs/outputs against the API docs diff --git a/.agents/skills/add-tools/SKILL.md b/.agents/skills/add-tools/SKILL.md index 734c03bcd9c..6ae100ce86b 100644 --- a/.agents/skills/add-tools/SKILL.md +++ b/.agents/skills/add-tools/SKILL.md @@ -475,6 +475,9 @@ All tool IDs MUST use `snake_case`: `{service}_{action}` (e.g., `x_create_tweet` - [ ] Index.ts exports all tools and re-exports types (`export * from './types'`) - [ ] Tools registered in `tools/registry.ts` - [ ] `bun run tool-metadata:generate` run and the regenerated artifacts committed +- [ ] `bun run scripts/generate-docs.ts` run and the refreshed docs committed — the integration's + docs page is rendered from each tool's description, params, and outputs, and CI's + `bun run docs:check` fails on stale pages - [ ] Block wired: `tools.access`, dropdown options, subBlocks, `tools.config`, outputs, inputs - [ ] Model, durable-storage, and internal-execution boundaries use the shared provenance mechanisms only where a concrete Sim `{{...}}` resolution path requires them diff --git a/.agents/skills/add-trigger/SKILL.md b/.agents/skills/add-trigger/SKILL.md index 2720d94f99d..3175d46e1c8 100644 --- a/.agents/skills/add-trigger/SKILL.md +++ b/.agents/skills/add-trigger/SKILL.md @@ -511,3 +511,6 @@ Add to `helm/sim/values.yaml` under the existing polling cron jobs: - [ ] `bun run type-check` passes - [ ] Manually verify output keys match trigger `outputs` keys - [ ] Trigger UI shows correctly in the block +- [ ] Ran `bun run scripts/generate-docs.ts` and committed the refreshed pages — trigger sections + render into the owning integration's docs page, and CI's `bun run docs:check` fails on stale + pages diff --git a/.agents/skills/validate-integration/SKILL.md b/.agents/skills/validate-integration/SKILL.md index da5ac1dd984..68ed9ee7620 100644 --- a/.agents/skills/validate-integration/SKILL.md +++ b/.agents/skills/validate-integration/SKILL.md @@ -388,6 +388,7 @@ Several files are generated from tool and block definitions. Editing a tool or b bun run tool-metadata:generate # repo root — apps/sim/tools/generated/* bun run scripts/generate-docs.ts # docs .mdx + lib/integrations/integrations.json + docs icons bun run integration-catalog:check # registry ↔ committed deployment metadata drift +bun run docs:check # committed docs ↔ what the generator renders today ``` - **`tool-metadata:generate`** — required whenever a tool's `outputs`, `params`, or descriptions change. CI enforces this with `bun run tool-metadata:check`, which fails with *"Generated tool metadata is stale"*. This is the easiest gate to miss, because nothing in the tool file hints that a generated artifact mirrors it. @@ -395,8 +396,17 @@ bun run integration-catalog:check # registry ↔ committed deployment metadat - **`integration-catalog:check`** — loads the executable block registry, derives visible integration deployment fields, and compares them with the committed catalog. It catches missing/unexpected entries and stale auth/service IDs without loading the executable registry in client code. - -**Always diff the regen output before committing.** These generators rewrite every file they own, so they will also sweep in unrelated drift that accumulated on the base branch — pages losing sections, unrelated icons appearing. Keep only the hunks belonging to the integration under validation and `git checkout --` the rest, otherwise an unrelated doc regression rides along in the PR. Verify no page was silently dropped by comparing the directory listing before and after. +- **`docs:check`** — check mode of `generate-docs.ts`: renders every generated docs artifact in + memory and fails listing any committed file that differs. Runs in CI via `check:audits`. + +**Always diff the regen output before committing — but commit all of it.** These generators rewrite +every file they own, so they also true up drift that accumulated on the base branch (pages whose +source changed without a regen). That catch-up is correct output, not a regression: `docs:check` +fails CI on any page left stale, so reverting swept-in hunks with `git checkout --` reintroduces the +failure. Review the diff to confirm each hunk is explained by a real source change (yours or an +upstream PR that skipped regeneration), and investigate anything that looks like content loss — a +page losing a section usually means its source block moved or a generator input broke, not that the +hunk should be reverted. If an icon changed, `apps/sim/components/icons.tsx` is the source of truth and `apps/docs/components/icons.tsx` is its generated mirror — they must end up byte-identical for that component. @@ -408,9 +418,10 @@ After fixing, confirm: 3. The integration's tests pass, and any test you added actually fails without its fix (revert it once and watch it go red) 4. Derived artifacts regenerated and their diffs reviewed (see above) 5. `bun run integration-catalog:check` passes -6. For OAuth or service-account changes, `bun test apps/sim/lib/integrations/availability.server.test.ts` passes -7. Re-read all modified files to verify fixes are correct -8. Any remaining unknown response schemas were explicitly reported to the user instead of guessed +6. `bun run docs:check` passes +7. For OAuth or service-account changes, `bun test apps/sim/lib/integrations/availability.server.test.ts` passes +8. Re-read all modified files to verify fixes are correct +9. Any remaining unknown response schemas were explicitly reported to the user instead of guessed ## Checklist Summary @@ -439,7 +450,7 @@ After fixing, confirm: - [ ] Reported all issues grouped by severity - [ ] Fixed all critical and warning issues - [ ] Ran `bun run tool-metadata:generate` if any tool outputs/params changed, and confirmed `bun run tool-metadata:check` passes -- [ ] Ran `bun run generate-docs` if any block metadata changed, and reverted unrelated drift the generator swept in +- [ ] Ran `bun run generate-docs` if any block metadata changed, and committed the full generated diff — including stale-page catch-up for other integrations (`bun run docs:check` fails CI on reverted generator output) - [ ] Ran `bun run lint` after fixes - [ ] Verified TypeScript compiles clean - [ ] Verified added tests fail without their fix diff --git a/.claude/commands/add-block-preview.md b/.claude/commands/add-block-preview.md deleted file mode 100644 index 8345170767f..00000000000 --- a/.claude/commands/add-block-preview.md +++ /dev/null @@ -1,60 +0,0 @@ ---- -description: Gate a block's visibility — ship an unreleased block as a preview (hidden until revealed via AppConfig/env), reveal it to admins/orgs, GA it, or kill-switch a shipped block -argument-hint: ---- - -# Add Block Preview Skill - -You manage **block visibility gating** in Sim — hiding blocks from every discovery surface (toolbar, cmd+K search, copilot @-mentions, agent tool picker, mothership VFS/metadata/tools, Access Control list, public docs/catalog) while **never** gating execution of already-placed instances. - -## The model - -Three levers, evaluated in `apps/sim/lib/core/config/block-visibility.ts` and folded into the registry accessors (`apps/sim/blocks/registry.ts`): - -1. **`preview: true`** on the `BlockConfig` (static, in code) — the block is default-hidden EVERYWHERE (hosted, self-hosted, dev, SSR) until revealed. Fail-closed. -2. **The hosted `block-visibility` AppConfig document** — per-block rule keyed by the existing block type: - - ```jsonc - { - "": { - "enabled": false, // required. true = GA (visible to everyone) - "orgIds": ["org_..."], // optional allowlist clauses (any match reveals) - "userIds": ["user_..."], - "adminEnabled": true // platform admins (user.role === 'admin') - } - } - ``` - -3. **`PREVIEW_BLOCKS` env** (comma-separated block types) — the off-AppConfig reveal path for self-hosters and local dev. - -A revealed block that is not globally GA (`enabled !== true`, or env-revealed) renders with a **" (Preview)"** name suffix on discovery surfaces. `getBlock()` stays pure, so placed instances keep their canonical name and always execute. - -## Lifecycle of a preview block - -1. **Author** the block normally (`/add-block` etc.) and set `preview: true` on its `BlockConfig`. **Ship no `BlockMeta` and no docs until GA** — `check-block-registry` deliberately skips preview blocks in meta coverage, and `generate-docs` skips them at every gate. -2. **Local dev:** set `PREVIEW_BLOCKS=` in your env to see it (with the suffix). -3. **Merge/deploy.** The block's code is live everywhere but visible nowhere — no AppConfig rule exists and self-hosters have no env entry. -4. **Hosted preview:** add a rule to the `block-visibility` AppConfig document and start a deployment (no code deploy): - - Admins only: `{ "enabled": false, "adminEnabled": true }` - - Design-partner org: `{ "enabled": false, "orgIds": ["org_123"] }` - - GA via config (code cleanup pending): `{ "enabled": true }` — suffix disappears everywhere within ~30s (AppConfig TTL) + client refetch. - - Same runbook as `feature-flags`: edit the hosted document, `aws appconfig start-deployment` with the `sim--fast` strategy (see the infra README). -5. **GA cleanup:** delete `preview: true` from the block (now visible to self-hosters on their next upgrade), add its `BlockMeta` + regen docs, and drop the AppConfig entry. For a v2 upgrade, this is also when v1 gets `hideFromToolbar: true` (the superseded-version paradigm). - -## Kill switch (shipped blocks) - -To pull an already-GA block from discovery surfaces on hosted (incident, deprecation): add `{ "": { "enabled": false } }` to the document. Allowlist clauses can carve out exceptions. **Execution is NOT stopped** — workflows already using the block keep running; the kill switch only prevents new placement/discovery. - -## Invariants (do not violate) - -- **Execution is never gated.** The executor, serializer, drop-naming, and `isBlockTypeAccessControlExempt` resolve via pure `getBlock`. Do not add visibility checks to execution paths. -- **Clone-not-remove:** gated blocks stay in `getAllBlocks()` output as clones with `hideFromToolbar: true` — `.find`-by-type consumers rely on this. Never filter them out. -- **Keys are registry block types.** Never `custom_block_*` (parse drops them — custom blocks have their own enabled/disabled lifecycle). -- **The shared hidden-predicate is `isHiddenUnder`** (`apps/sim/blocks/visibility/context.ts`). Never restate the preview/disabled rule inline at a new consumer. -- **Process-global caches stay ungated.** `getStaticComponentFiles` (VFS) and `getExposedIntegrationTools` build the ungated universe; per-viewer filtering happens at stamp/consumer time. Never move gating into a shared builder. -- Gating is **surface hiding, not secrecy** — the full config ships in the client JS bundle. Anything truly secret cannot be a registered block. - -## Tests - -Evaluation semantics: `apps/sim/lib/core/config/block-visibility.test.ts`. Registry projection: `apps/sim/blocks/visibility/visibility.test.ts`. When gating behavior changes, extend those — mock `isPlatformAdmin` for the admin clause; use the local `withAppConfig` harness. diff --git a/.claude/commands/add-block.md b/.claude/commands/add-block.md deleted file mode 100644 index 1f9a8554af8..00000000000 --- a/.claude/commands/add-block.md +++ /dev/null @@ -1,1042 +0,0 @@ ---- -description: Create or update a Sim integration block with correct subBlocks, conditions, dependsOn, modes, canonicalParamId usage, outputs, and tool wiring. Use when working on `apps/sim/blocks/blocks/{service}.ts` or aligning a block with its tools. -argument-hint: ---- - -# Add Block Skill - -You are an expert at creating block configurations for Sim. You understand the serializer, subBlock types, conditions, dependsOn, modes, and all UI patterns. - -## Your Task - -When the user asks you to create a block: -1. Create the block file in `apps/sim/blocks/blocks/{service}.ts` -2. Configure all subBlocks with proper types, conditions, and dependencies -3. Wire up tools correctly - -## Hard Rule: No Guessed Tool Outputs - -Blocks depend on tool outputs. If the underlying tool response schema is not documented or live-verified, you MUST tell the user instead of guessing block outputs. - -- Do NOT invent block outputs for undocumented tool responses -- Do NOT describe unknown JSON shapes as if they were confirmed -- Do NOT wire fields into the block just because they seem likely to exist - -If the tool outputs are not known, do one of these instead: -1. Ask the user for sample tool responses -2. Ask the user for test credentials so the tool responses can be verified -3. Limit the block to operations whose outputs are documented -4. Leave uncertain outputs out and explicitly tell the user what remains unknown - -## Block Configuration Structure - -```typescript -import { {ServiceName}Icon } from '@/components/icons' -import type { BlockConfig } from '@/blocks/types' -import { AuthMode, IntegrationType } from '@/blocks/types' -import { getScopesForService } from '@/lib/oauth/utils' - -export const {ServiceName}Block: BlockConfig = { - type: '{service}', // snake_case identifier - name: '{Service Name}', // Human readable - description: 'Brief description', // One sentence - longDescription: 'Detailed description for docs', - docsLink: 'https://docs.sim.ai/integrations/{service}', - category: 'tools', // 'tools' | 'blocks' | 'triggers' - integrationType: IntegrationType.X, // Primary category (see IntegrationType enum) - tags: ['oauth', 'api'], // Cross-cutting tags (see IntegrationTag type) - bgColor: '#HEXCOLOR', // Brand color - icon: {ServiceName}Icon, - - // Auth mode - authMode: AuthMode.OAuth, // or AuthMode.ApiKey - - // Card summary sentences — see "Canvas Sentences" below - canvasPresentation: { - defaultTitle: '{Default Operation}', - sentences: { byOperation: { /* one per operation dropdown option id */ } }, - }, - - subBlocks: [ - // Define all UI fields here - ], - - tools: { - access: ['tool_id_1', 'tool_id_2'], // Array of tool IDs this block can use - config: { - tool: (params) => `{service}_${params.operation}`, // Tool selector function - params: (params) => ({ - // Transform subBlock values to tool params - }), - }, - }, - - inputs: { - // Optional: define expected inputs from other blocks - }, - - outputs: { - // Define outputs available to downstream blocks - }, -} -``` - -## SubBlock Types Reference - -**Critical:** Every subblock `id` must be unique within the block. Duplicate IDs cause conflicts even with different conditions. - -### Text Inputs -```typescript -// Single-line input -{ id: 'field', title: 'Label', type: 'short-input', placeholder: '...' } - -// Multi-line input -{ id: 'field', title: 'Label', type: 'long-input', placeholder: '...', rows: 6 } - -// Password input -{ id: 'apiKey', title: 'API Key', type: 'short-input', password: true } -``` - -### Selection Inputs -```typescript -// Dropdown (static options) -{ - id: 'operation', - title: 'Operation', - type: 'dropdown', - options: [ - { label: 'Create', id: 'create' }, - { label: 'Update', id: 'update' }, - ], - value: () => 'create', // Default value function -} - -// Combobox (searchable dropdown) -{ - id: 'field', - title: 'Label', - type: 'combobox', - options: [...], - searchable: true, -} -``` - -### Code/JSON Inputs -```typescript -{ - id: 'code', - title: 'Code', - type: 'code', - language: 'javascript', // 'javascript' | 'json' | 'python' - placeholder: '// Enter code...', -} -``` - -### OAuth/Credentials -```typescript -{ - id: 'credential', - title: 'Account', - type: 'oauth-input', - serviceId: '{service}', // Must match OAuth provider service key - requiredScopes: getScopesForService('{service}'), // Import from @/lib/oauth/utils - placeholder: 'Select account', - required: true, -} -``` - -**Scopes:** Always use `getScopesForService(serviceId)` from `@/lib/oauth/utils` for `requiredScopes`. Never hardcode scope arrays — the single source of truth is `OAUTH_PROVIDERS` in `lib/oauth/oauth.ts`. - -**Scope descriptions:** When adding a new OAuth provider, also add human-readable descriptions for all scopes in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts`. - -**Service accounts (shared, app-level credentials):** A plain `oauth-input` already lets users *select* an existing service account — those credentials fold into the picker automatically (a Google service account created for any Google service appears in every Google block's picker). You only set `credentialKind` when you want to change the *connect* action: - -```typescript -{ - id: 'credential', - title: 'Account', - type: 'oauth-input', - serviceId: '{service}', - requiredScopes: getScopesForService('{service}'), - credentialKind: 'any', // omit | 'service-account' | 'any' -} -``` - -- **omit (default):** lists OAuth accounts + any existing service accounts; the only connect action is "Connect account" (OAuth). Use this for the common "let users pick a service account someone set up elsewhere, but don't offer inline setup" case — no config needed. -- **`'service-account'`:** service-account credentials *only*, plus an inline setup action that opens the provider's connect modal. Use when a block accepts *only* an app credential. -- **`'any'`:** merged picker — OAuth accounts *and* service accounts in one grouped dropdown, with a connect action for each. Use when a block supports both (e.g. Slack: a personal account *or* a custom bot). - -Optional companions: `credentialLabels` (override the picker's section/connect-row copy) and `allowServiceAccounts: true` (trigger-mode only — list service accounts, which triggers otherwise exclude; set only when the trigger's polling path can resolve a service-account token). The connect modal, provider families (Google JSON key, Atlassian token, token-paste, client-credential, Slack bot), and the preview gate are all resolved from `serviceAccountProviderId` — you don't wire them per block. - -### OAuth deployment availability (required for integration blocks) - -A visible tools-category block with OAuth is deployment-gated. Its `oauth-input.serviceId` is -projected into `apps/sim/lib/integrations/integrations.json`, then resolved through -`resolveOAuthClientCapabilityId()` in `apps/sim/lib/core/config/env-capabilities.ts`. - -When adding or changing an OAuth integration block: - -1. Keep exactly one distinct OAuth `serviceId` across the block's `oauth-input` subBlocks. -2. Confirm that service ID resolves to an entry in `OAUTH_CLIENT_CAPABILITIES`. Google and - Microsoft service IDs intentionally share their provider-level capability; do not add duplicate - entries for those aliases. -3. For a new capability, add its required client fields to `OAUTH_CLIENT_CAPABILITIES` and ensure - every referenced field exists in the env schema in `apps/sim/lib/core/config/env.ts`. Then add - the matching `text` or `secret` input modes to `OAUTH_CLIENT_SETUP_FIELDS` in - `scripts/setup/capability-config.ts`. The CLI catalog is exhaustively typed and checked against - the runtime field list; do not infer secrecy from the field name. -4. If the canonical OAuth service declares `serviceAccountProviderId`, keep - `SERVICE_ACCOUNT_METADATA_BY_OAUTH_SERVICE_ID` in - `apps/sim/lib/integrations/service-account-metadata.ts` aligned. Set - `deploymentRequirement` only when the service-account path is preview-gated or depends on the - OAuth client fields; otherwise omit it. - -Missing capability metadata is a runtime configuration error, not a reason to make the integration -silently available. - -### Selectors (with dynamic options) -```typescript -// Channel selector (Slack, Discord, etc.) -{ - id: 'channel', - title: 'Channel', - type: 'channel-selector', - serviceId: '{service}', - placeholder: 'Select channel', - dependsOn: ['credential'], -} - -// Project selector (Jira, etc.) -{ - id: 'project', - title: 'Project', - type: 'project-selector', - serviceId: '{service}', - dependsOn: ['credential'], -} - -// File selector (Google Drive, etc.) -{ - id: 'file', - title: 'File', - type: 'file-selector', - serviceId: '{service}', - mimeType: 'application/pdf', - dependsOn: ['credential'], -} - -// User selector -{ - id: 'user', - title: 'User', - type: 'user-selector', - serviceId: '{service}', - dependsOn: ['credential'], -} -``` - -### Other Types -```typescript -// Switch/toggle -{ id: 'enabled', type: 'switch' } - -// Slider -{ id: 'temperature', title: 'Temperature', type: 'slider', min: 0, max: 2, step: 0.1 } - -// Table (key-value pairs) -{ id: 'headers', title: 'Headers', type: 'table', columns: ['Key', 'Value'] } - -// File upload -{ - id: 'files', - title: 'Attachments', - type: 'file-upload', - multiple: true, - acceptedTypes: 'image/*,application/pdf', -} -``` - -## File Input Handling - -When your block accepts file uploads, use the basic/advanced mode pattern with `normalizeFileInput`. - -### Basic/Advanced File Pattern - -```typescript -// Basic mode: Visual file upload -{ - id: 'uploadFile', - title: 'File', - type: 'file-upload', - canonicalParamId: 'file', // Both map to 'file' param - placeholder: 'Upload file', - mode: 'basic', - multiple: false, - required: true, - condition: { field: 'operation', value: 'upload' }, -}, -// Advanced mode: Reference from other blocks -{ - id: 'fileRef', - title: 'File', - type: 'short-input', - canonicalParamId: 'file', // Both map to 'file' param - placeholder: 'Reference file (e.g., {{file_block.output}})', - mode: 'advanced', - required: true, - condition: { field: 'operation', value: 'upload' }, -}, -``` - -**Keep the pair to one logical thing.** Basic is the file upload, advanced is *only* a reference to -a file from a previous block. Gmail attachments are the reference implementation -(`apps/sim/blocks/blocks/gmail.ts` — `attachmentFiles` / `attachments`). - -Do not overload the advanced side with alternate identifiers (a remote URL, a provider asset ID, a -path). A subblock whose meaning changes based on what the string looks like is impossible to reason -about, forces the params function to sniff the value, and makes the field's type meaningless. Give -each alternative its own subblock outside the pair: - -```typescript -// ✓ Good — the pair is "a file"; other sources are their own fields -{ id: 'mediaFile', type: 'file-upload', canonicalParamId: 'media', mode: 'basic' }, -{ id: 'mediaFileRef', type: 'short-input', canonicalParamId: 'media', mode: 'advanced' }, -{ id: 'mediaId', type: 'short-input', mode: 'advanced' }, // separate concept -{ id: 'mediaLink', type: 'short-input', mode: 'advanced' }, // separate concept - -// ✗ Bad — one field meaning three things, resolved by guessing -{ id: 'mediaRef', type: 'short-input', canonicalParamId: 'media', mode: 'advanced', - placeholder: 'File reference, media ID, or public URL' }, -``` - -When several fields are mutually exclusive alternatives, mark them all `required: false` and enforce -"exactly one" at execution — a conditionally-required canonical pair rejects the workflow before the -other paths ever get a chance to supply the value. - -**Critical constraints:** -- `canonicalParamId` must NOT match any subblock's `id` in the same block -- A canonical group is **block-wide**, not per-operation: `buildCanonicalIndex` keys groups by - `canonicalParamId` across every subblock, and a group has exactly one `basicId`. Two operations - that each need a file pair need two distinct `canonicalParamId` values. -- All members of a group must share the same `required` status - -### Normalizing File Input in tools.config - -Put the normalization in `tools.config.params`, never in `tools.config.tool` — `tool` runs at -serialization, before variable resolution, so a `` file reference is not yet a value -there. - -```typescript -import { normalizeFileInput } from '@/blocks/utils' - -tools: { - access: ['service_upload'], - config: { - tool: (params) => `service_${params.operation}`, - params: (params) => { - // Read the CANONICAL id, not the subblock ids - const { file: fileParam, ...rest } = params - const file = normalizeFileInput(fileParam, { single: true }) - return { - ...rest, - ...(file ? { file } : {}), - } - }, - }, -} -``` - -**Where the value actually lives at runtime.** The subblock `id` is where the UI *stores* the value, -but it is not what the params function receives. `extractBlockParams` -(`apps/sim/serializer/index.ts`) collapses each canonical group at serialization time: - -```typescript -const sourceIds = [group.basicId, ...group.advancedIds].filter(Boolean) -sourceIds.forEach((id) => delete params[id]) // subblock ids are deleted -if (chosen !== undefined) params[group.canonicalId] = chosen -``` - -So by the time `tools.config.params(inputs)` runs (`executor/handlers/generic/generic-handler.ts`), -`params.uploadFile` and `params.fileRef` are **gone** and the value is under `params.file`. Reading a -subblock id there yields `undefined` and silently sends no file. - -Only the active mode's value survives — `getCanonicalValues` returns the basic value in basic mode -and the first non-empty advanced value in advanced mode, so a stale value in the dormant mode can -never leak. `normalizeFileInput` then handles the JSON string that advanced-mode template resolution -produces. - -Note that `generic-handler` merges rather than replaces (`{ ...inputs, ...transformedParams }`), so -omitting a key from the returned object does not strip it from what the tool receives. Tools simply -ignore params they do not declare. - -### File Input Types in `inputs` - -Declare the **canonical** id with `type: 'json'` — the subblock ids never reach `inputs`: - -```typescript -inputs: { - file: { type: 'json', description: 'File to upload (UserFile or reference)' }, - // Legacy field for backwards compatibility - fileContent: { type: 'string', description: 'Legacy: base64 encoded content' }, -} -``` - -### Multiple Files - -For multiple file uploads: - -```typescript -{ - id: 'attachments', - title: 'Attachments', - type: 'file-upload', - multiple: true, // Allow multiple files - maxSize: 25, // Max size in MB per file - acceptedTypes: 'image/*,application/pdf,.doc,.docx', -} - -// In tools.config: -const normalizedFiles = normalizeFileInput( - params.attachments || params.attachmentRefs, - // No { single: true } - returns array -) -if (normalizedFiles) { - params.files = normalizedFiles -} -``` - -## Condition Syntax - -Controls when a field is shown based on other field values. - -### Simple Condition -```typescript -condition: { field: 'operation', value: 'create' } -// Shows when operation === 'create' -``` - -### Multiple Values (OR) -```typescript -condition: { field: 'operation', value: ['create', 'update'] } -// Shows when operation is 'create' OR 'update' -``` - -### Negation -```typescript -condition: { field: 'operation', value: 'delete', not: true } -// Shows when operation !== 'delete' -``` - -### Compound (AND) -```typescript -condition: { - field: 'operation', - value: 'send', - and: { - field: 'type', - value: 'dm', - not: true, - } -} -// Shows when operation === 'send' AND type !== 'dm' -``` - -### Complex Example -```typescript -condition: { - field: 'operation', - value: ['list', 'search'], - not: true, - and: { - field: 'authMethod', - value: 'oauth', - } -} -// Shows when operation NOT in ['list', 'search'] AND authMethod === 'oauth' -``` - -## DependsOn Pattern - -Controls when a field is enabled and when its options are refetched. - -### Simple Array (all must be set) -```typescript -dependsOn: ['credential'] -// Enabled only when credential has a value -// Options refetch when credential changes - -dependsOn: ['credential', 'projectId'] -// Enabled only when BOTH have values -``` - -### Complex (all + any) -```typescript -dependsOn: { - all: ['authMethod'], // All must be set - any: ['credential', 'apiKey'] // At least one must be set -} -// Enabled when authMethod is set AND (credential OR apiKey is set) -``` - -## Required Pattern - -Can be boolean or condition-based. - -### Simple Boolean -```typescript -required: true -required: false -``` - -### Conditional Required -```typescript -required: { field: 'operation', value: 'create' } -// Required only when operation === 'create' - -required: { field: 'operation', value: ['create', 'update'] } -// Required when operation is 'create' OR 'update' -``` - -## Mode Pattern (Basic vs Advanced) - -Controls which UI view shows the field. - -### Mode Options -- `'basic'` - Only in basic view (default UI) -- `'advanced'` - Only in advanced view -- `'both'` - Both views (default if not specified) -- `'trigger'` - Only in trigger configuration - -### canonicalParamId Pattern - -Maps multiple UI fields to a single serialized parameter: - -```typescript -// Basic mode: Visual selector -{ - id: 'channel', - title: 'Channel', - type: 'channel-selector', - mode: 'basic', - canonicalParamId: 'channel', // Both map to 'channel' param - dependsOn: ['credential'], -} - -// Advanced mode: Manual input -{ - id: 'channelId', - title: 'Channel ID', - type: 'short-input', - mode: 'advanced', - canonicalParamId: 'channel', // Both map to 'channel' param - placeholder: 'Enter channel ID manually', -} -``` - -**How it works:** -- In basic mode: `channel` selector value → `params.channel` -- In advanced mode: `channelId` input value → `params.channel` -- The serializer consolidates based on current mode - -**Critical constraints:** -- `canonicalParamId` must NOT match any other subblock's `id` in the same block (causes conflicts) -- A `canonicalParamId` links exactly one basic/advanced pair for a single logical parameter. Do NOT reuse the same `canonicalParamId` for different parameters, even under mutually-exclusive conditions/operations -- ONLY use `canonicalParamId` to link basic/advanced mode alternatives for the same logical parameter -- Do NOT use it for any other purpose - -## WandConfig Pattern - -Enables AI-assisted field generation. - -```typescript -{ - id: 'query', - title: 'Query', - type: 'code', - language: 'json', - wandConfig: { - enabled: true, - prompt: 'Generate a query based on the user request. Return ONLY the JSON.', - placeholder: 'Describe what you want to query...', - generationType: 'json-object', // Optional: affects AI behavior - maintainHistory: true, // Optional: keeps conversation context - }, -} -``` - -### Generation Types -- `'javascript-function-body'` - JS code generation -- `'json-object'` - Raw JSON (adds "no markdown" instruction) -- `'json-schema'` - JSON Schema definitions -- `'sql-query'` - SQL statements -- `'timestamp'` - Adds current date/time context - -## Tools Configuration - -**Important:** `tools.config.tool` runs during serialization before variable resolution. Put `Number()` and other type coercions in `tools.config.params` instead, which runs at execution time after variables are resolved. - -**Preferred:** Use tool names directly as dropdown option IDs to avoid switch cases: -```typescript -// Dropdown options use tool IDs directly -options: [ - { label: 'Create', id: 'service_create' }, - { label: 'Read', id: 'service_read' }, -] - -// Tool selector just returns the operation value -tool: (params) => params.operation, -``` - -### With Parameter Transformation -```typescript -tools: { - access: ['service_action'], - config: { - tool: (params) => 'service_action', - params: (params) => ({ - id: params.resourceId, - data: typeof params.data === 'string' ? JSON.parse(params.data) : params.data, - }), - }, -} -``` - -### V2 Versioned Tool Selector -```typescript -import { createVersionedToolSelector } from '@/blocks/utils' - -tools: { - access: [ - 'service_create_v2', - 'service_read_v2', - 'service_update_v2', - ], - config: { - tool: createVersionedToolSelector({ - baseToolSelector: (params) => `service_${params.operation}`, - suffix: '_v2', - fallbackToolId: 'service_create_v2', - }), - }, -} -``` - -## Outputs Definition - -**IMPORTANT:** Block outputs have a simpler schema than tool outputs. Block outputs do NOT support: -- `optional: true` - This is only for tool outputs -- `items` property - This is only for tool outputs with array types - -Block outputs only support: -- `type` - The data type ('string', 'number', 'boolean', 'json', 'array') -- `description` - Human readable description -- `condition` - Optional visibility condition -- `hiddenFromDisplay` - Optional flag to hide from the output display - -**Nested object/`properties` outputs are tool-output-only and will fail TypeScript at build time on block outputs.** For complex shapes use `type: 'json'` and describe the inner fields in the `description` string. - -```typescript -outputs: { - // Simple outputs - id: { type: 'string', description: 'Resource ID' }, - success: { type: 'boolean', description: 'Whether operation succeeded' }, - - // Use type: 'json' for complex objects or arrays (NOT type: 'array' with items) - items: { type: 'json', description: 'List of items' }, - metadata: { type: 'json', description: 'Response metadata' }, - - // Nested outputs (for structured data) - user: { - id: { type: 'string', description: 'User ID' }, - name: { type: 'string', description: 'User name' }, - email: { type: 'string', description: 'User email' }, - }, -} -``` - -### Typed JSON Outputs - -When using `type: 'json'` and you know the object shape in advance, **describe the inner fields in the description** so downstream blocks know what properties are available. Block outputs have no nested `properties` form — always keep the output flat and put the shape in the `description`: - -```typescript -outputs: { - // BAD: Opaque json with no info about what's inside - plan: { type: 'json', description: 'Zone plan information' }, - - // GOOD: Describe the known fields in the description - plan: { - type: 'json', - description: 'Zone plan information (id, name, price, currency, frequency, is_subscribed)', - }, -} -``` - -Nested object outputs (`plan: { id: { type: 'string' }, ... }`) are a **tool-output** feature only — `OutputFieldDefinition` for blocks does not allow them and they fail TypeScript at build time. - -If the output shape is unknown because the underlying tool response is undocumented, you MUST tell the user and stop. Unknown is not the same as variable. Never guess block outputs. - -## V2 Block Pattern - -When creating V2 blocks (alongside legacy V1): - -```typescript -// V1 Block - mark as legacy -export const ServiceBlock: BlockConfig = { - type: 'service', - name: 'Service (Legacy)', - hideFromToolbar: true, // Hide from toolbar - // ... rest of config -} - -// V2 Block - visible, uses V2 tools -export const ServiceV2Block: BlockConfig = { - type: 'service_v2', - name: 'Service', // Clean name - hideFromToolbar: false, // Visible - subBlocks: ServiceBlock.subBlocks, // Reuse UI - tools: { - access: ServiceBlock.tools?.access?.map(id => `${id}_v2`) || [], - config: { - tool: createVersionedToolSelector({ - baseToolSelector: (params) => (ServiceBlock.tools?.config as any)?.tool(params), - suffix: '_v2', - fallbackToolId: 'service_default_v2', - }), - params: ServiceBlock.tools?.config?.params, - }, - }, - outputs: { - // Flat, API-aligned outputs (not wrapped in content/metadata) - }, -} -``` - -## Registering Blocks - -After creating the block, remind the user to register it in `apps/sim/blocks/registry-maps.ts` (the data maps live here; `registry.ts` holds only the accessor functions). Add the import and an entry to each map alphabetically: - -```typescript -import { ServiceBlock, ServiceBlockMeta } from '@/blocks/blocks/service' - -export const BLOCK_REGISTRY: Record = { - // ... existing blocks ... - service: ServiceBlock, -} - -export const BLOCK_META_REGISTRY: Record = { - // ... existing metas ... - service: ServiceBlockMeta, -} -``` - -## Complete Example - -```typescript -import { ServiceIcon } from '@/components/icons' -import type { BlockConfig } from '@/blocks/types' -import { AuthMode, IntegrationType } from '@/blocks/types' -import { getScopesForService } from '@/lib/oauth/utils' - -export const ServiceBlock: BlockConfig = { - type: 'service', - name: 'Service', - description: 'Integrate with Service API', - longDescription: 'Full description for documentation...', - docsLink: 'https://docs.sim.ai/integrations/service', - category: 'tools', - integrationType: IntegrationType.DeveloperTools, - tags: ['oauth', 'api'], - bgColor: '#FF6B6B', - icon: ServiceIcon, - authMode: AuthMode.OAuth, - - subBlocks: [ - { - id: 'operation', - title: 'Operation', - type: 'dropdown', - options: [ - { label: 'Create', id: 'create' }, - { label: 'Read', id: 'read' }, - { label: 'Update', id: 'update' }, - { label: 'Delete', id: 'delete' }, - ], - value: () => 'create', - }, - { - id: 'credential', - title: 'Service Account', - type: 'oauth-input', - serviceId: 'service', - requiredScopes: getScopesForService('service'), - placeholder: 'Select account', - required: true, - }, - { - id: 'resourceId', - title: 'Resource ID', - type: 'short-input', - placeholder: 'Enter resource ID', - condition: { field: 'operation', value: ['read', 'update', 'delete'] }, - required: { field: 'operation', value: ['read', 'update', 'delete'] }, - }, - { - id: 'name', - title: 'Name', - type: 'short-input', - placeholder: 'Resource name', - condition: { field: 'operation', value: ['create', 'update'] }, - required: { field: 'operation', value: 'create' }, - }, - ], - - tools: { - access: ['service_create', 'service_read', 'service_update', 'service_delete'], - config: { - tool: (params) => `service_${params.operation}`, - }, - }, - - outputs: { - id: { type: 'string', description: 'Resource ID' }, - name: { type: 'string', description: 'Resource name' }, - createdAt: { type: 'string', description: 'Creation timestamp' }, - }, -} -``` - -## Connecting Blocks with Triggers - -If the service supports webhooks, connect the block to its triggers. - -```typescript -import { getTrigger } from '@/triggers' - -export const ServiceBlock: BlockConfig = { - // ... basic config ... - - triggers: { - enabled: true, - available: ['service_event_a', 'service_event_b', 'service_webhook'], - }, - - subBlocks: [ - // Tool subBlocks first... - { id: 'operation', /* ... */ }, - - // Then spread trigger subBlocks - ...getTrigger('service_event_a').subBlocks, - ...getTrigger('service_event_b').subBlocks, - ...getTrigger('service_webhook').subBlocks, - ], -} -``` - -See the `/add-trigger` skill for creating triggers. - -## Icon Requirement - -If the icon doesn't already exist in `@/components/icons.tsx`, **do NOT search for it yourself**. After completing the block, ask the user to provide the SVG: - -``` -The block is complete, but I need an icon for {Service}. -Please provide the SVG and I'll convert it to a React component. - -You can usually find this in the service's brand/press kit page, or copy it from their website. -``` - -When converting the SVG: a **monochrome** logo (single white or black mark) must -use `fill='currentColor'`, never a hardcoded `#fff`/`#000000`. Block icons render -both inside their `bgColor` tile and "bare" on a neutral page (the home Suggested -actions list) in light and dark mode; a hardcoded white/black mark goes invisible -bare on the matching background. Multi-color brand logos keep their own fills. -Verify with `bun run check:bare-icons`. - -## Advanced Mode for Optional Fields - -Optional fields that are rarely used should be set to `mode: 'advanced'` so they don't clutter the basic UI. This includes: -- Pagination tokens -- Time range filters (start/end time) -- Sort order options -- Reply settings -- Rarely used IDs (e.g., reply-to tweet ID, quote tweet ID) -- Max results / limits - -```typescript -{ - id: 'startTime', - title: 'Start Time', - type: 'short-input', - placeholder: 'ISO 8601 timestamp', - condition: { field: 'operation', value: ['search', 'list'] }, - mode: 'advanced', // Rarely used, hide from basic view -} -``` - -## WandConfig for Complex Inputs - -Use `wandConfig` for fields that are hard to fill out manually, such as timestamps, comma-separated lists, and complex query strings. This gives users an AI-assisted input experience. - -```typescript -// Timestamps - use generationType: 'timestamp' to inject current date context -{ - id: 'startTime', - title: 'Start Time', - type: 'short-input', - mode: 'advanced', - wandConfig: { - enabled: true, - prompt: 'Generate an ISO 8601 timestamp based on the user description. Return ONLY the timestamp string.', - generationType: 'timestamp', - }, -} - -// Comma-separated lists - simple prompt without generationType -{ - id: 'mediaIds', - title: 'Media IDs', - type: 'short-input', - mode: 'advanced', - wandConfig: { - enabled: true, - prompt: 'Generate a comma-separated list of media IDs. Return ONLY the comma-separated values.', - }, -} -``` - -## Naming Convention - -All tool IDs referenced in `tools.access` and returned by `tools.config.tool` MUST use `snake_case` (e.g., `x_create_tweet`, `slack_send_message`). Never use camelCase or PascalCase. - -## BlockMeta (Required) - -Every block file must export a `{Service}BlockMeta` alongside the block — **minimum 7 templates**. Look at existing examples in `apps/sim/blocks/blocks/` (e.g. `browser_use.ts`, `google_sheets.ts`) for the pattern. - -```typescript -import type { BlockMeta } from '@/blocks/types' - -export const {Service}BlockMeta = { - tags: ['tag1', 'tag2'], // IntegrationTag[] - url: 'https://{service}.com', // external service homepage (verify it resolves) — NOT docs.sim.ai - templates: [ - { - icon: {Service}Icon, - title: '{Service} ', // 2–5 words - prompt: 'Build a workflow that...', // specific use case, 1–3 sentences - modules: ['agent', 'workflows'], // 'agent' | 'workflows' | 'tables' | 'files' | 'scheduled' | 'knowledge-base' - category: 'operations', // 'operations' | 'marketing' | 'sales' | 'engineering' | 'productivity' | 'support' | 'popular' - tags: ['automation'], - alsoIntegrations: ['slack'], // optional — other block IDs referenced in the prompt - featured: true, // optional - }, - // ... at least 6 more - ], - skills: [ // SuggestedSkill[] — 3–5 mainstream, 2–3 niche - { - name: 'summarize-thread', // kebab-case, ≤64 chars, unique, verb-led - description: 'One line: what it does and when to use it.', // ≤1024 chars - content: - '# Summarize Thread\n\n...\n\n## Steps\n1. ...\n\n## Output\n...', // markdown - }, - // ... more - ], -} as const satisfies BlockMeta -``` - -Derive templates from the service's real use cases. Each prompt should name a concrete trigger, transformation, and output — not a generic description of what the service does. - -`skills` are curated, ready-to-add agent skills shown on the integration's detail page (users click **Add** to create them in their workspace). Two hard rules: - -- **Ground every skill in operations the block actually exposes** — cross-check each skill's steps against `tools.access`. Never describe an action the integration cannot perform. -- **Derive skills from real, popular use cases found online — never invent them.** Web-search the service's documented use cases (vendor use-case/solutions pages, official docs describing the workflow, reputable "top automations for X" articles) and only add a skill you can source as something people genuinely do with the service. Do not hallucinate skills. - -## Canvas Sentences - -Every block declares a one-line prose summary that replaces its card's field rows: - -``` -Slack ← header (already names the block) -Posts ⟨Ship it 🚀⟩ to ⟨#eng⟩ ← the sentence; ⟨…⟩ are live value chips -``` - -Write one `byOperation` entry per operation dropdown option (or a single `default` -when the block has no operation dropdown). - -**The full authoring contract — voice, structure, and the two mistakes that break -cards silently — is `apps/sim/blocks/AGENTS.md` → "Canvas sentences". Read it -before writing any.** The two failures worth repeating here, because both are -invisible at runtime: - -1. A clause naming only one member of a `canonicalParamId` pair drops the sentence - for every advanced-mode user. List all members: - `field: ['channelSelector', 'manualChannel']`. -2. A clause referencing a subblock whose `condition` excludes that operation can - never render. - -Validate before finishing: - -```bash -bun run apps/sim/scripts/check-canvas-sentences.ts --block={service} -``` - -## Generated artifacts - -Adding a block on its own needs no **tool metadata** regeneration — a block references existing -tool IDs through `tools.access` and does not change any tool's shape. - -But if the same change also adds, edits **or removes** a tool, run `bun run tool-metadata:generate` and commit the result, or CI fails on stale artifacts. That matters here because a block's `outputs` are authored to match its tools' outputs, and the UI now reads those from the generated metadata rather than the executable registry — an unregenerated tool change makes the block's outputs disagree with what the panel renders. See `.agents/skills/tool-registry-boundary/SKILL.md`. - -A visible integration block does require the generated integration catalog and docs to be refreshed. -After adding or changing one, run: - -```bash -bun run scripts/generate-docs.ts -bun run integration-catalog:check -``` - -The catalog check independently derives deployment metadata from the executable block registry and -compares it with the committed `apps/sim/lib/integrations/integrations.json`. Review the generated -diff and keep only intentional changes. -## Checklist Before Finishing - -- [ ] `integrationType` is set to the correct `IntegrationType` enum value -- [ ] `tags` array includes all applicable `IntegrationTag` values -- [ ] All subBlocks have `id`, `title` (except switch), and `type` -- [ ] Conditions use correct syntax (field, value, not, and) -- [ ] DependsOn set for fields that need other values -- [ ] Required fields marked correctly (boolean or condition) -- [ ] OAuth inputs have correct `serviceId` and `requiredScopes: getScopesForService(serviceId)` -- [ ] Every OAuth `serviceId` resolves through `resolveOAuthClientCapabilityId()` to the correct `OAUTH_CLIENT_CAPABILITIES` entry -- [ ] Any new OAuth capability fields exist in `apps/sim/lib/core/config/env.ts` -- [ ] If the OAuth service supports service accounts, `SERVICE_ACCOUNT_METADATA_BY_OAUTH_SERVICE_ID` matches its canonical `serviceAccountProviderId` and deployment requirement -- [ ] Scope descriptions added to `SCOPE_DESCRIPTIONS` in `lib/oauth/utils.ts` for any new scopes -- [ ] Tools.access lists all tool IDs (snake_case) -- [ ] Tools.config.tool returns correct tool ID (snake_case) -- [ ] Outputs match tool outputs -- [ ] Block + meta registered in registry-maps.ts (`BLOCK_REGISTRY` / `BLOCK_META_REGISTRY`) -- [ ] If any tool was added, changed or removed alongside the block: ran `bun run tool-metadata:generate` and committed the artifacts -- [ ] Ran `bun run scripts/generate-docs.ts`, reviewed the generated diff, and committed the integration catalog changes -- [ ] `bun run integration-catalog:check` passes -- [ ] If icon missing: asked user to provide SVG -- [ ] If triggers exist: `triggers` config set, trigger subBlocks spread -- [ ] Optional/rarely-used fields set to `mode: 'advanced'` -- [ ] Timestamps and complex inputs have `wandConfig` enabled -- [ ] Exported `{Service}BlockMeta` with at least 7 templates -- [ ] `url` set on `{Service}BlockMeta` to the external service's verified homepage (omit only for first-party blocks with no external service) -- [ ] `skills` added to `{Service}BlockMeta`, each grounded in `tools.access` and sourced from a real online use case (not invented) -- [ ] `canvasPresentation.sentences` covers every operation, and `bun run apps/sim/scripts/check-canvas-sentences.ts --block={service}` passes with 100% coverage - -## Final Validation (Required) - -After creating the block, you MUST validate it against every tool it references: - -1. **Read every tool definition** that appears in `tools.access` — do not skip any -2. **For each tool, verify the block has correct:** - - SubBlock inputs that cover all required tool params (with correct `condition` to show for that operation) - - SubBlock input types that match the tool param types (e.g., dropdown for enums, short-input for strings) - - `tools.config.params` correctly maps subBlock IDs to tool param names (if they differ) - - Type coercions in `tools.config.params` for any params that need conversion (Number(), Boolean(), JSON.parse()) -3. **Verify block outputs** cover the key fields returned by all tools -4. **Verify conditions** — each subBlock should only show for the operations that actually use it -5. **Verify `{Service}BlockMeta` is exported** with at least 7 templates, each having `icon`, `title`, `prompt`, `modules`, `category`, and `tags` -6. **If any tool outputs are still unknown**, explicitly tell the user instead of guessing block outputs diff --git a/.claude/commands/add-column-type.md b/.claude/commands/add-column-type.md deleted file mode 100644 index 7b362016218..00000000000 --- a/.claude/commands/add-column-type.md +++ /dev/null @@ -1,159 +0,0 @@ ---- -description: Add a new table column type to Sim — registry entry, icon, storage shape, coercion, and the behavioral hooks the grid and API read. Use when adding a value kind under `apps/sim/lib/table/column-types/`. -argument-hint: ---- - -# Adding a Table Column Type - -A column type is **one file** in `apps/sim/lib/table/column-types/` plus a registry entry. Everything that varies per type — label, icon, storage cast, coercion, validation, conversion compatibility, formatting, editor, filter operators — lives on that one object, so no consumer needs editing. - -This was not always true: adding `currency` originally took ~40 edits across 32 `switch` arms and 26 UI branches, each of which failed **silently** when missed. The registry exists to make that impossible, so the rule is absolute: **if you find yourself adding a `case 'yourtype':` anywhere outside `column-types/`, the registry is missing a field. Add the field instead.** - -## Hard Rule: the compiler tells you what to do - -Do **not** hunt for places to edit. Add your type to the `ColumnType` union first and let `tsc` produce the list: - -```bash -cd apps/sim && bun run type-check -``` - -You will get two errors, naming `column-types/registry.ts` and `column-types/registry.server.ts`. Register in both. - -If your type owns metadata, adding its key to `TYPE_SPECIFIC_COLUMN_KEYS` produces two more legitimate errors — `FOREIGN_METADATA_VERB` in `validation.ts` (a `Record` over those keys) and the key's absence from `ColumnDefinition`. Those are the gate working, not sites to "fix". - -Any error beyond those four is a site reading a hardcoded type list that should read the registry — fix that site, don't work around it. - -## Directory Structure - -``` -apps/sim/lib/table/column-types/ -├── types.ts # ColumnTypeDefinition — the contract you implement -├── types.server.ts # ColumnTypeServerDefinition — cell migrations only -├── registry.ts # Record ← client-safe, the gate -├── registry.server.ts # Record ← adds migrations (drizzle) -├── index.ts # barrel + accessors (columnTypeOf, columnTypeById, …) -└── {type}.ts # one file per type — what you write -``` - -## Step 1: Pick the storage shape - -Decide what a cell literally holds in `user_table_rows.data` (JSONB). This drives almost everything else: - -| Storage | `jsonbCast` | Notes | -|---------|-------------|-------| -| number | `'numeric'` | Filters/sorts compare numerically. `currency` does this. | -| ISO string | `'timestamptz'` | `date` does this. | -| string / bool / object | `null` | Text comparison is correct. | - -**Prefer an existing primitive over a new shape.** `currency` stores a plain number and keeps its ISO code as *display metadata* — which is why filtering, sorting, uniqueness, and CSV export all reuse the numeric paths untouched, and why re-denominating a column rewrites zero rows. - -## Step 2: Add the icon - -Create `packages/emcn/src/icons/type-{name}.tsx`, copying the geometry conventions of its siblings exactly: - -```tsx -import type { SVGProps } from 'react' - -/** - * Type {name} icon component - {what the glyph is} for {name} columns - * @param props - SVG properties including className, fill, etc. - */ -export function Type{Pascal}(props: SVGProps) { - return ( - - ) -} -``` - -- `viewBox='-1.75 -1.5 24 24'` is the **`type-*` family** value, not the set-wide default. Match the family. -- Center the glyph on the viewBox's optical center (**y = 10.5**, **x = 10.25**) — every sibling does, and a few tenths off is visible at `size-[14px]`. -- Export alphabetically **by component name** in `packages/emcn/src/icons/index.ts`. - -## Step 3: Write the type file - -`apps/sim/lib/table/column-types/{name}.ts`. Copy the closest existing type and change what differs. Every field is required by the interface, so the compiler enumerates them for you — read the TSDoc in `types.ts` rather than guessing. - -The three that are easy to get wrong: - -- **`coerce`** is the *single* write-path implementation. The server runs it before persisting **and** the grid runs it to fill the optimistic cache. Accept every shape the value legitimately arrives in (paste, CSV, tool write), because rejecting means the cell is nulled. -- **`isCompatibleWith`** gates type conversion and must read the value **exactly as `coerce` will**, or a conversion will pass its check and then null the cell. -- **`ownedMetadata`** lists the `ColumnDefinition` keys your type owns. Anything you add must also be added to `TYPE_SPECIFIC_COLUMN_KEYS` in `types.ts` and given a phrase in `FOREIGN_METADATA_VERB` in `validation.ts` — both are `Record`-typed, so the compiler will tell you. - -## Step 4: Register - -Add the entry to `COLUMN_TYPE_REGISTRY` in `registry.ts` **and** `COLUMN_TYPE_SERVER_REGISTRY` in `registry.server.ts`. - -`COLUMN_TYPES` is declared in `types.ts` (not derived from the registry — the registry is annotated `Record` against it, which is the gate). `constants.ts` re-exports it, so `columnTypeSchema = z.enum(COLUMN_TYPES)` picks your type up with no edit. **Type-specific metadata does not** — see the next step. - -## Step 5: Migrations (only if the stored bytes change) - -If converting an existing column **to** your type must rewrite cells, add `migrateCellsTo` in `registry.server.ts`; if converting **away** must rewrite them, add `migrateCellsFrom`. - -This is load-bearing, not cosmetic: filters and sorts apply `jsonbCast` to whatever is stored, so leaving a non-castable string behind makes **every query on that column fail** — not merely render oddly. - -Prefer set-based SQL. When the transform genuinely needs JS (`currency`'s separator disambiguation), compute the values during the compatibility scan and pass them through `resolved`, then apply them in one batched statement. - -## Naming Convention - -- Type id: lowercase, singular — `currency`, not `Currency` or `currencies` -- File: `column-types/{id}.ts`, export `const {id}ColumnType` -- Icon: `type-{kebab}.tsx`, export `Type{Pascal}` - -## Watch out - -- **Import cycles.** `column-types/select.ts` imports `select-values.ts`, so `select-values.ts` must **not** import the registry — that closes a cycle and fails at module init. Inside a type's own helper module the string literal is the implementation, not a config leak. -- **The client-safe boundary.** `registry.ts` and everything it imports must stay free of `@sim/db`, `drizzle-orm`, and `next/server` — the tables grid imports it directly. A React icon is fine (it's a component *reference*, never called server-side). Only `registry.server.ts` may touch drizzle. -- **Don't re-export the registry from `@/lib/table`.** 44 server modules import that barrel; routing this through it pulls `@sim/emcn/icons` into all of them. Deep-import `@/lib/table/column-types`. -- **`import.ts`'s `coerceValue` is a SECOND write path and is not opt-in.** Importing into a column of your type always hits it, and its `default` arm silently `String(value)`s — so a missing `case` stores text in a column whose `jsonbCast` is numeric, and then every filter and sort on that column errors in Postgres. Add a `case`, even though the switch compiles without one. (It is deliberately separate from the registry's `coerce`: an import wants an unparseable value to survive as its raw string so the row error can name it.) -- **CSV inference** is an ordered heuristic in `import.ts`, deliberately not registry-driven. A new type is not inferred from a CSV unless you extend `inferColumnType` — usually you should not, since inference cannot supply configuration (an option set, a currency code). - -## If your type owns metadata, read this - -Registering the *type* is compiler-enforced. Registering its *metadata* is not, and that is where the remaining manual work lives. A key like `precision` has to be added in each of these, none of which will fail to compile if you forget: - -| Where | What happens if you forget | -|---|---| -| `lib/table/types.ts` `ColumnDefinition` | (this one DOES fail — the ownership loop indexes it) | -| `column-types/types.ts` `TYPE_SPECIFIC_COLUMN_KEYS` | it is never stripped on conversion, and poisons the target type | -| `lib/api/contracts/tables.ts` — the schema slot in all three column schemas, plus `refineColumnOptions` | zod strips it at the boundary; silently never saved | -| `columns/service.ts` `addTableColumn` param type | callers cannot pass it | -| A metadata-only update path (`updateColumnCurrency` is the model) + a branch in both column routes + the copilot tool | changing it on an existing column is a silent 200 no-op | -| `column-config-sidebar.tsx` | no UI to set it | -| `table-grid.tsx` delete-column undo + `use-table-undo.ts` restore | undo silently resets it to the default | - -`normalizeColumn`, `buildConvertedColumn`, and the undo snapshot read `TYPE_SPECIFIC_COLUMN_KEYS` generically, so those three are already zero-edit. - -**Known gap:** the metadata-only update path is ~6 near-identical copies (service + 2 routes + copilot). A `metadataUpdate` descriptor on `ColumnTypeServerDefinition` would collapse them; until that exists, copy `currency`'s. - -## Checklist Before Finishing - -- [ ] Added to the `ColumnType` union in `column-types/types.ts` -- [ ] `column-types/{id}.ts` created, every interface field filled in -- [ ] Registered in **both** `registry.ts` and `registry.server.ts` -- [ ] Icon added, centered on the family's optical center, exported alphabetically -- [ ] `migrateCellsTo` / `migrateCellsFrom` added if the stored bytes change -- [ ] New metadata keys added to `TYPE_SPECIFIC_COLUMN_KEYS` + `FOREIGN_METADATA_VERB` -- [ ] Unit tests for `coerce` / `isCompatibleWith` round-trips, verified to fail without the code -- [ ] Docs row added to `apps/docs/content/docs/en/tables/index.mdx` - -## Final Validation (Required) - -1. **`cd apps/sim && bun run type-check`** — must be clean. If any file *outside* `column-types/` errors, that file has a hardcoded type list; fix it to read the registry. -2. **Grep for leaks** — `grep -rnE "(===|!==) '{id}'|case '{id}':" apps/sim --include='*.ts' --include='*.tsx' | grep -v column-types/`. (All three forms: a plain `!==` and a `case` are how half of `currency`'s real branches are written.) Hits are expected; judge each. A hit is fine when it mounts a specific React component or encodes a genuinely one-off behavior (`json`'s mono textarea, `date`'s timezone-aware parsing). A hit is a **leak** when it restates something the registry could answer — an icon, a label, a colour, an operator set, a cast, a coercion. Leaks get a registry field, not a new branch. -3. **Run the suite** — `bunx vitest run lib/table 'app/workspace/[workspaceId]/tables' lib/api app/api/table app/api/v1 lib/copilot/tools/server/table`. Existing tests must pass **unchanged**; needing to edit one means you changed behavior for the other types. -4. **`bun run lint:check`, `bun run check:api-validation`, `bun run check:client-boundary`** from the repo root. -5. **Exercise it in the running app** on a table with one column of every type: create, edit inline / in the expanded popover / in the row modal, paste from a spreadsheet, filter, sort, convert to and from other types, export CSV, undo a column delete. diff --git a/.claude/commands/add-connector.md b/.claude/commands/add-connector.md deleted file mode 100644 index c73921b22fc..00000000000 --- a/.claude/commands/add-connector.md +++ /dev/null @@ -1,625 +0,0 @@ ---- -description: Add or update a Sim knowledge base connector for syncing documents from an external source, including auth mode, config fields, pagination, document mapping, tags, and registry wiring. Use when working in `apps/sim/connectors/{service}/` or adding a new external document source. -argument-hint: [api-docs-url] ---- - -# Add Connector Skill - -You are an expert at adding knowledge base connectors to Sim. A connector syncs documents from an external source (Confluence, Google Drive, Notion, etc.) into a knowledge base. - -## Your Task - -When the user asks you to create a connector: -1. Use Context7 or WebFetch to read the service's API documentation -2. Determine the auth mode: **OAuth** (if Sim already has an OAuth provider for the service) or **API key** (if the service uses API key / Bearer token auth) -3. Create the connector directory: a client-safe `meta.ts` (declarative metadata) plus the runtime module that spreads it -4. Register it in BOTH the server registry and the client-safe meta registry - -## Hard Rule: No Guessed Response Or Document Schemas - -If the service docs do not clearly show the document list response, document fetch response, pagination shape, or metadata fields, you MUST tell the user instead of guessing. - -- Do NOT invent document fields -- Do NOT guess pagination cursors or next-page fields -- Do NOT infer metadata/tag mappings from unrelated endpoints -- Do NOT fabricate `ExternalDocument` content structure from partial docs - -If the source schema is unknown, do one of these instead: -1. Ask the user for sample API responses -2. Ask the user for test credentials so you can verify live payloads -3. Implement only the documented parts of the connector -4. Leave the connector incomplete and explicitly say which fields remain unknown - -## Directory Structure - -Each connector is split into a client-safe metadata file and a server-only runtime file. This mirrors the `XBlockMeta` / `BLOCK_META_REGISTRY` split in `apps/sim/blocks` — client components (the knowledge UI) only need the metadata (icon, name, auth, config fields), so the runtime functions (which pull server-only helpers like `input-validation.server` → `undici` → `node:net`) must stay out of the client bundle. - -Create files in `apps/sim/connectors/{service}/`: -``` -connectors/{service}/ -├── index.ts # Barrel export (re-exports the runtime connector) -├── meta.ts # ConnectorMeta — client-safe declarative metadata -└── {service}.ts # ConnectorConfig — spreads the meta + adds runtime functions -``` - -- `meta.ts` exports `{service}ConnectorMeta: ConnectorMeta`. It imports ONLY the icon from `@/components/icons`, `import type { ConnectorMeta } from '@/connectors/types'`, and any pure-data constants. It must NEVER import server/runtime code. -- `{service}.ts` exports `{service}Connector: ConnectorConfig`. It imports the meta via `import { {service}ConnectorMeta } from '@/connectors/{service}/meta'`, spreads it as the first property, and holds the runtime functions (which may import server-only helpers like `@/lib/knowledge/documents/utils`). - -## Authentication - -Connectors use a discriminated union for auth config (`ConnectorAuthConfig` in `connectors/types.ts`): - -```typescript -type ConnectorAuthConfig = - | { mode: 'oauth'; provider: OAuthService; requiredScopes?: string[] } - | { mode: 'apiKey'; label?: string; placeholder?: string } -``` - -### OAuth mode -For services with existing OAuth providers in `apps/sim/lib/oauth/types.ts`. The `provider` must match an `OAuthService`. The modal shows a credential picker and handles token refresh automatically. - -### API key mode -For services that use API key / Bearer token auth. The modal shows a password input with the configured `label` and `placeholder`. The API key is encrypted at rest using AES-256-GCM and stored in a dedicated `encryptedApiKey` column on the connector record. The sync engine decrypts it automatically — connectors receive the raw access token in `listDocuments`, `getDocument`, and `validateConfig`. - -## Connector Structure (meta.ts + runtime) - -The declarative metadata lives in `meta.ts` (`ConnectorMeta`). The runtime functions live in `{service}.ts` (`ConnectorConfig`), which spreads the meta as its first property. - -### `meta.ts` — client-safe metadata - -```typescript -import { {Service}Icon } from '@/components/icons' -import type { ConnectorMeta } from '@/connectors/types' - -export const {service}ConnectorMeta: ConnectorMeta = { - id: '{service}', - name: '{Service}', - description: 'Sync documents from {Service} into your knowledge base', - version: '1.0.0', - icon: {Service}Icon, - - auth: { - mode: 'oauth', - provider: '{service}', // Must match OAuthService in lib/oauth/types.ts - requiredScopes: ['read:...'], - }, - - configFields: [ - // Rendered dynamically by the add-connector modal UI - // Supports 'short-input', 'dropdown', and 'selector' types — see ConfigField Types below - ], - - // Optional: tag definitions are metadata too — declare them here - // tagDefinitions: [ ... ], -} -``` - -Keep `meta.ts` free of any server/runtime import. Only the icon, the `ConnectorMeta` type, and pure-data constants belong here. - -### `{service}.ts` — runtime (OAuth example) - -```typescript -import { createLogger } from '@sim/logger' -import { fetchWithRetry } from '@/lib/knowledge/documents/utils' -import { {service}ConnectorMeta } from '@/connectors/{service}/meta' -import type { ConnectorConfig, ExternalDocument, ExternalDocumentList } from '@/connectors/types' - -const logger = createLogger('{Service}Connector') - -export const {service}Connector: ConnectorConfig = { - ...{service}ConnectorMeta, - - listDocuments: async (accessToken, sourceConfig, cursor) => { - // Return metadata stubs with contentDeferred: true (if per-doc content fetch needed) - // Or full documents with content (if list API returns content inline) - // Return { documents: ExternalDocument[], nextCursor?, hasMore } - }, - - getDocument: async (accessToken, sourceConfig, externalId) => { - // Fetch full content for a single document - // Return ExternalDocument with contentDeferred: false, or null - }, - - validateConfig: async (accessToken, sourceConfig) => { - // Return { valid: true } or { valid: false, error: 'message' } - }, - - // Optional: map source metadata to semantic tag keys (translated to slots by sync engine) - mapTags: (metadata) => { - // Return Record with keys matching tagDefinitions[].id - }, -} -``` - -Only map fields in `listDocuments`, `getDocument`, `validateConfig`, and `mapTags` when the source payload shape is documented or live-verified. If not, tell the user and stop rather than guessing. - -### API key connector example - -The split is identical — `auth` lives in `meta.ts`, runtime functions in `{service}.ts`. - -```typescript -// meta.ts -export const {service}ConnectorMeta: ConnectorMeta = { - id: '{service}', - name: '{Service}', - description: 'Sync documents from {Service} into your knowledge base', - version: '1.0.0', - icon: {Service}Icon, - - auth: { - mode: 'apiKey', - label: 'API Key', // Shown above the input field - placeholder: 'Enter your {Service} API key', // Input placeholder - }, - - configFields: [ /* ... */ ], -} - -// {service}.ts -export const {service}Connector: ConnectorConfig = { - ...{service}ConnectorMeta, - listDocuments: async (accessToken, sourceConfig, cursor) => { /* ... */ }, - getDocument: async (accessToken, sourceConfig, externalId) => { /* ... */ }, - validateConfig: async (accessToken, sourceConfig) => { /* ... */ }, -} -``` - -## ConfigField Types - -The add-connector modal renders these automatically — no custom UI needed. - -Three field types are supported: `short-input`, `dropdown`, and `selector`. - -```typescript -// Text input -{ - id: 'domain', - title: 'Domain', - type: 'short-input', - placeholder: 'yoursite.example.com', - required: true, -} - -// Dropdown (static options) -{ - id: 'contentType', - title: 'Content Type', - type: 'dropdown', - required: false, - options: [ - { label: 'Pages only', id: 'page' }, - { label: 'Blog posts only', id: 'blogpost' }, - { label: 'All content', id: 'all' }, - ], -} -``` - -## Dynamic Selectors (Canonical Pairs) - -Use `type: 'selector'` to fetch options dynamically from the existing selector registry (`hooks/selectors/registry.ts`). Selectors are always paired with a manual fallback input using the **canonical pair** pattern — a `selector` field (basic mode) and a `short-input` field (advanced mode) linked by `canonicalParamId`. - -The user sees a toggle button (ArrowLeftRight) to switch between the selector dropdown and manual text input. On submit, the modal resolves each canonical pair to the active mode's value, keyed by `canonicalParamId`. - -### Rules - -1. **Every selector field MUST have a canonical pair** — a corresponding `short-input` (or `dropdown`) field with the same `canonicalParamId` and `mode: 'advanced'`. -2. **`required` must be set identically on both fields** in a pair. If the selector is required, the manual input must also be required. -3. **`canonicalParamId` must match the key the connector expects in `sourceConfig`** (e.g. `baseId`, `channel`, `teamId`). The advanced field's `id` should typically match `canonicalParamId`. -4. **`dependsOn` references the selector field's `id`**, not the `canonicalParamId`. The modal propagates dependency clearing across canonical siblings automatically — changing either field in a parent pair clears dependent children. - -### Selector canonical pair example (Airtable base → table cascade) - -```typescript -configFields: [ - // Base: selector (basic) + manual (advanced) - { - id: 'baseSelector', - title: 'Base', - type: 'selector', - selectorKey: 'airtable.bases', // Must exist in hooks/selectors/registry.ts - canonicalParamId: 'baseId', - mode: 'basic', - placeholder: 'Select a base', - required: true, - }, - { - id: 'baseId', - title: 'Base ID', - type: 'short-input', - canonicalParamId: 'baseId', - mode: 'advanced', - placeholder: 'e.g. appXXXXXXXXXXXXXX', - required: true, - }, - // Table: selector depends on base (basic) + manual (advanced) - { - id: 'tableSelector', - title: 'Table', - type: 'selector', - selectorKey: 'airtable.tables', - canonicalParamId: 'tableIdOrName', - mode: 'basic', - dependsOn: ['baseSelector'], // References the selector field ID - placeholder: 'Select a table', - required: true, - }, - { - id: 'tableIdOrName', - title: 'Table Name or ID', - type: 'short-input', - canonicalParamId: 'tableIdOrName', - mode: 'advanced', - placeholder: 'e.g. Tasks', - required: true, - }, - // Non-selector fields stay as-is - { id: 'maxRecords', title: 'Max Records', type: 'short-input', ... }, -] -``` - -### Selector with domain dependency (Jira/Confluence pattern) - -When a selector depends on a plain `short-input` field (no canonical pair), `dependsOn` references that field's `id` directly. The `domain` field's value maps to `SelectorContext.domain` automatically via `SELECTOR_CONTEXT_FIELDS`. - -```typescript -configFields: [ - { - id: 'domain', - title: 'Jira Domain', - type: 'short-input', - placeholder: 'yoursite.atlassian.net', - required: true, - }, - { - id: 'projectSelector', - title: 'Project', - type: 'selector', - selectorKey: 'jira.projects', - canonicalParamId: 'projectKey', - mode: 'basic', - dependsOn: ['domain'], - placeholder: 'Select a project', - required: true, - }, - { - id: 'projectKey', - title: 'Project Key', - type: 'short-input', - canonicalParamId: 'projectKey', - mode: 'advanced', - placeholder: 'e.g. ENG, PROJ', - required: true, - }, -] -``` - -### How `dependsOn` maps to `SelectorContext` - -The connector selector field builds a `SelectorContext` from dependency values. For the mapping to work, each dependency's `canonicalParamId` (or field `id` for non-canonical fields) must exist in `SELECTOR_CONTEXT_FIELDS` (`lib/workflows/subblocks/context.ts`): - -``` -oauthCredential, domain, teamId, projectId, knowledgeBaseId, planId, -siteId, collectionId, spreadsheetId, fileId, baseId, datasetId, serviceDeskId -``` - -### Available selector keys - -Check `hooks/selectors/types.ts` for the full `SelectorKey` union. Common ones for connectors: - -| SelectorKey | Context Deps | Returns | -|-------------|-------------|---------| -| `airtable.bases` | credential | Base ID + name | -| `airtable.tables` | credential, `baseId` | Table ID + name | -| `slack.channels` | credential | Channel ID + name | -| `gmail.labels` | credential | Label ID + name | -| `google.calendar` | credential | Calendar ID + name | -| `linear.teams` | credential | Team ID + name | -| `linear.projects` | credential, `teamId` | Project ID + name | -| `jira.projects` | credential, `domain` | Project key + name | -| `confluence.spaces` | credential, `domain` | Space key + name | -| `notion.databases` | credential | Database ID + name | -| `asana.workspaces` | credential | Workspace GID + name | -| `microsoft.teams` | credential | Team ID + name | -| `microsoft.channels` | credential, `teamId` | Channel ID + name | -| `webflow.sites` | credential | Site ID + name | -| `outlook.folders` | credential | Folder ID + name | - -## ExternalDocument Shape - -Every document returned from `listDocuments`/`getDocument` must include: - -```typescript -{ - externalId: string // Source-specific unique ID - title: string // Document title - content: string // Extracted plain text (or '' if contentDeferred) - contentDeferred?: boolean // true = content will be fetched via getDocument - mimeType: 'text/plain' // Always text/plain (content is extracted) - contentHash: string // Metadata-based hash for change detection - sourceUrl?: string // Link back to original (stored on document record) - metadata?: Record // Source-specific data (fed to mapTags) -} -``` - -## Content Deferral (Required for file/content-download connectors) - -**All connectors that require per-document API calls to fetch content MUST use `contentDeferred: true`.** This is the standard pattern — `listDocuments` returns lightweight metadata stubs, and content is fetched lazily by the sync engine via `getDocument` only for new/changed documents. - -This pattern is critical for reliability: the sync engine processes documents in batches and enqueues each batch for processing immediately. If a sync times out, all previously-batched documents are already queued. Without deferral, content downloads during listing can exhaust the sync task's time budget before any documents are saved. - -### When to use `contentDeferred: true` - -- The service's list API does NOT return document content (only metadata) -- Content requires a separate download/export API call per document -- Examples: Google Drive, OneDrive, SharePoint, Dropbox, Notion, Confluence, Gmail, Obsidian, Evernote, GitHub - -### When NOT to use `contentDeferred` - -- The list API already returns the full content inline (e.g., Slack messages, Reddit posts, HubSpot notes) -- No per-document API call is needed to get content - -### Content Hash Strategy - -Use a **metadata-based** `contentHash` — never a content-based hash. The hash must be derivable from the list response metadata alone, so the sync engine can detect changes without downloading content. - -Good metadata hash sources: -- `modifiedTime` / `lastModifiedDateTime` — changes when file is edited -- Git blob SHA — unique per content version -- API-provided content hash (e.g., Dropbox `content_hash`) -- Version number (e.g., Confluence page version) - -Format: `{service}:{id}:{changeIndicator}` - -```typescript -// Google Drive: modifiedTime changes on edit -contentHash: `gdrive:${file.id}:${file.modifiedTime ?? ''}` - -// GitHub: blob SHA is a content-addressable hash -contentHash: `gitsha:${item.sha}` - -// Dropbox: API provides content_hash -contentHash: `dropbox:${entry.id}:${entry.content_hash ?? entry.server_modified}` - -// Confluence: version number increments on edit -contentHash: `confluence:${page.id}:${page.version.number}` -``` - -**Critical invariant:** The `contentHash` MUST be identical whether produced by `listDocuments` (stub) or `getDocument` (full doc). Both should use the same stub function to guarantee this. - -### Implementation Pattern - -```typescript -// 1. Create a stub function (sync, no API calls) -function fileToStub(file: ServiceFile): ExternalDocument { - return { - externalId: file.id, - title: file.name || 'Untitled', - content: '', - contentDeferred: true, - mimeType: 'text/plain', - sourceUrl: `https://service.com/file/${file.id}`, - contentHash: `service:${file.id}:${file.modifiedTime ?? ''}`, - metadata: { /* fields needed by mapTags */ }, - } -} - -// 2. listDocuments returns stubs (fast, metadata only) -listDocuments: async (accessToken, sourceConfig, cursor) => { - const response = await fetchWithRetry(listUrl, { ... }) - const files = (await response.json()).files - const documents = files.map(fileToStub) - return { documents, nextCursor, hasMore } -} - -// 3. getDocument fetches content and returns full doc with SAME contentHash -getDocument: async (accessToken, sourceConfig, externalId) => { - const metadata = await fetchWithRetry(metadataUrl, { ... }) - const file = await metadata.json() - if (file.trashed) return null - - try { - const content = await fetchContent(accessToken, file) - if (!content.trim()) return null - const stub = fileToStub(file) - return { ...stub, content, contentDeferred: false } - } catch (error) { - logger.warn(`Failed to fetch content for: ${file.name}`, { error }) - return null - } -} -``` - -### Reference Implementations - -- **Google Drive**: `connectors/google-drive/google-drive.ts` — file download/export with `modifiedTime` hash -- **GitHub**: `connectors/github/github.ts` — git blob SHA hash -- **Notion**: `connectors/notion/notion.ts` — blocks API with `last_edited_time` hash -- **Confluence**: `connectors/confluence/confluence.ts` — version number hash - -## tagDefinitions — Declared Tag Definitions - -Declare which tags the connector populates using semantic IDs. Shown in the add-connector modal as opt-out checkboxes. -On connector creation, slots are **dynamically assigned** via `getNextAvailableSlot` — connectors never hardcode slot names. - -```typescript -tagDefinitions: [ - { id: 'labels', displayName: 'Labels', fieldType: 'text' }, - { id: 'version', displayName: 'Version', fieldType: 'number' }, - { id: 'lastModified', displayName: 'Last Modified', fieldType: 'date' }, -], -``` - -Each entry has: -- `id`: Semantic key matching a key returned by `mapTags` (e.g. `'labels'`, `'version'`) -- `displayName`: Human-readable name shown in the UI (e.g. "Labels", "Last Modified") -- `fieldType`: `'text'` | `'number'` | `'date'` | `'boolean'` — determines which slot pool to draw from - -Users can opt out of specific tags in the modal. Disabled IDs are stored in `sourceConfig.disabledTagIds`. -The assigned mapping (`semantic id → slot`) is stored in `sourceConfig.tagSlotMapping`. - -## `@/connectors/utils` Helpers - -Reuse these instead of inlining the same logic (the validator enforces them): - -- `htmlToPlainText(html)` — strip HTML to plain text before indexing `ExternalDocument.content`. Never index raw HTML. -- `computeContentHash(content)` — stable content hash for change detection. -- `parseTagDate(value)` — parse to a valid `Date` or `undefined` (guards Invalid Date). Use in `mapTags` for date fields. -- `joinTagArray(value)` — validate an array and join to a comma-separated string, or `undefined`. Use in `mapTags` for array/label fields. -- `parseMultiValue(value)` — normalize a value into a `string[]`. - -## mapTags — Metadata to Semantic Keys - -Maps source metadata to semantic tag keys. Required if `tagDefinitions` is set. -The sync engine calls this automatically and translates semantic keys to actual DB slots -using the `tagSlotMapping` stored on the connector. - -Return keys must match the `id` values declared in `tagDefinitions`. - -Use the `@/connectors/utils` helpers for the common transforms — don't hand-roll date/array validation: - -```typescript -import { joinTagArray, parseTagDate } from '@/connectors/utils' - -mapTags: (metadata: Record): Record => { - const result: Record = {} - - // joinTagArray validates the array and joins to a comma-separated string (undefined if empty) - const labels = joinTagArray(metadata.labels) - if (labels) result.labels = labels - - // Validate numbers — guard against NaN - if (metadata.version != null) { - const num = Number(metadata.version) - if (!Number.isNaN(num)) result.version = num - } - - // parseTagDate returns a valid Date or undefined (guards against Invalid Date) - const lastModified = parseTagDate(metadata.lastModified) - if (lastModified) result.lastModified = lastModified - - return result -} -``` - -## External API Calls — Use `fetchWithRetry` - -All external API calls must use `fetchWithRetry` from `@/lib/knowledge/documents/utils` instead of raw `fetch()`. This provides exponential backoff with retries on 429/502/503/504 errors. It returns a standard `Response` — all `.ok`, `.json()`, `.text()` checks work unchanged. - -For `validateConfig` (user-facing, called on save), pass `VALIDATE_RETRY_OPTIONS` to cap wait time at ~7s. Background operations (`listDocuments`, `getDocument`) use the built-in defaults (5 retries, ~31s max). - -```typescript -import { VALIDATE_RETRY_OPTIONS, fetchWithRetry } from '@/lib/knowledge/documents/utils' - -// Background sync — use defaults -const response = await fetchWithRetry(url, { - method: 'GET', - headers: { Authorization: `Bearer ${accessToken}` }, -}) - -// validateConfig — tighter retry budget -const response = await fetchWithRetry(url, { ... }, VALIDATE_RETRY_OPTIONS) -``` - -## sourceUrl - -If `ExternalDocument.sourceUrl` is set, the sync engine stores it on the document record. Always construct the full URL (not a relative path). - -## Capped or Incomplete Listings — `syncContext.listingCapped` (REQUIRED) - -If `listDocuments` can ever return **less than the full source set** on a non-incremental sync — a `maxItems`/`maxDocuments`-style cap, or a transient per-item error that drops a still-existing document from the listing — it MUST set `syncContext.listingCapped = true` when that happens. - -The sync engine reconciles deletions by comparing the full listing against stored documents: anything not seen is **hard-deleted** (sync-engine.ts, gated on `!syncContext?.listingCapped`). A truncated listing without this flag deletes every real document beyond the cap. This was the single most common bug found when auditing connectors — do not omit it. - -```typescript -if (hitLimit && syncContext) { - syncContext.listingCapped = true -} -``` - -Rules: -- Set it when a user-configured cap truncates the listing while more documents exist -- Set it when a thrown error caused a still-present document to be skipped during listing -- Do NOT set it when the source is genuinely exhausted (deleted documents must still reconcile) -- Do NOT set it for intentional scope filters (e.g. a date cutoff) — out-of-scope documents should be reconciled normally - -## Sync Engine Behavior (Do Not Modify) - -The sync engine (`lib/knowledge/connectors/sync-engine.ts`) is connector-agnostic. It: -1. Calls `listDocuments` with pagination until `hasMore` is false -2. Compares `contentHash` to detect new/changed/unchanged documents -3. Stores `sourceUrl` and calls `mapTags` on insert/update automatically -4. Handles soft-delete of removed documents -5. Resolves access tokens automatically — OAuth tokens are refreshed, API keys are decrypted from the `encryptedApiKey` column - -You never need to modify the sync engine when adding a connector. - -## Icon - -The `icon` field on `ConnectorConfig` is used throughout the UI — in the connector list, the add-connector modal, and as the document icon in the knowledge base table (replacing the generic file type icon for connector-sourced documents). The icon is read from `CONNECTOR_REGISTRY[connectorType].icon` at runtime — no separate icon map to maintain. - -If the service already has an icon in `apps/sim/components/icons.tsx` (from a tool integration), reuse it. Otherwise, ask the user to provide the SVG. - -## Registering - -Register in BOTH registries, keeping the same alphabetical-by-id ordering in each. - -1. **Server registry** — `apps/sim/connectors/registry.server.ts` (server-only full registry; holds full connectors with runtime functions, imported by the sync engine and knowledge API routes): - -```typescript -import { {service}Connector } from '@/connectors/{service}' - -export const CONNECTOR_REGISTRY: ConnectorRegistry = { - // ... existing connectors ... - {service}: {service}Connector, -} -``` - -2. **Client-safe meta registry** — `apps/sim/connectors/registry.ts` (imports each connector's `meta.ts` only, so client components can use it without pulling server-only code; the metadata counterpart to `BLOCK_META_REGISTRY`): - -```typescript -import { {service}ConnectorMeta } from '@/connectors/{service}/meta' - -export const CONNECTOR_META_REGISTRY: ConnectorMetaRegistry = { - // ... existing connector metas ... - {service}: {service}ConnectorMeta, -} -``` - -`registry.ts` exports `CONNECTOR_META_REGISTRY: ConnectorMetaRegistry` plus the helpers `getConnectorMeta(id)` and `getAllConnectorMeta()`, importing each `@/connectors/{service}/meta` directly — never the runtime module. `registry.server.ts` exports `CONNECTOR_REGISTRY: ConnectorRegistry`. - -## Reference Implementations - -- **OAuth + contentDeferred**: `apps/sim/connectors/google-drive/google-drive.ts` — file download with metadata-based hash, `orderBy` for deterministic pagination -- **OAuth + contentDeferred (blocks API)**: `apps/sim/connectors/notion/notion.ts` — complex block content extraction deferred to `getDocument` -- **OAuth + contentDeferred (git)**: `apps/sim/connectors/github/github.ts` — blob SHA hash, tree listing -- **OAuth + inline content**: `apps/sim/connectors/confluence/confluence.ts` — multiple config field types, `mapTags`, label fetching -- **API key**: `apps/sim/connectors/fireflies/fireflies.ts` — GraphQL API with Bearer token auth - -## Checklist - -- [ ] Created `connectors/{service}/meta.ts` with `{service}ConnectorMeta: ConnectorMeta` (icon, name, auth, configFields, tagDefinitions) — no server/runtime imports -- [ ] Created `connectors/{service}/{service}.ts` with `{service}Connector: ConnectorConfig` spreading the meta + runtime functions -- [ ] Created `connectors/{service}/index.ts` barrel export -- [ ] **Auth configured correctly:** - - OAuth: `auth.provider` matches an existing `OAuthService` in `lib/oauth/types.ts` - - API key: `auth.label` and `auth.placeholder` set appropriately -- [ ] **Selector fields configured correctly (if applicable):** - - Every `type: 'selector'` field has a canonical pair (`short-input` or `dropdown` with same `canonicalParamId` and `mode: 'advanced'`) - - `required` is identical on both fields in each canonical pair - - `selectorKey` exists in `hooks/selectors/registry.ts` - - `dependsOn` references selector field IDs (not `canonicalParamId`) - - Dependency `canonicalParamId` values exist in `SELECTOR_CONTEXT_FIELDS` -- [ ] `listDocuments` handles pagination with metadata-based content hashes -- [ ] `syncContext.listingCapped = true` set whenever the listing is truncated (max-items cap or transient per-item error) — required to prevent the engine's deletion reconciliation from removing unseen documents -- [ ] `contentDeferred: true` used if content requires per-doc API calls (file download, export, blocks fetch) -- [ ] `contentHash` is metadata-based (not content-based) and identical between stub and `getDocument` -- [ ] `sourceUrl` set on each ExternalDocument (full URL, not relative) -- [ ] `metadata` includes source-specific data for tag mapping -- [ ] `tagDefinitions` declared for each semantic key returned by `mapTags` -- [ ] `mapTags` implemented if source has useful metadata (labels, dates, versions) -- [ ] `validateConfig` verifies the source is accessible -- [ ] All external API calls use `fetchWithRetry` (not raw `fetch`) -- [ ] All optional config fields validated in `validateConfig` -- [ ] Icon exists in `components/icons.tsx` (or asked user to provide SVG) -- [ ] Registered the full connector in `connectors/registry.server.ts` -- [ ] Registered the meta in `connectors/registry.ts` (same alphabetical-by-id ordering as registry.server.ts) diff --git a/.claude/commands/add-enrichment.md b/.claude/commands/add-enrichment.md deleted file mode 100644 index b8da9bb3bb4..00000000000 --- a/.claude/commands/add-enrichment.md +++ /dev/null @@ -1,142 +0,0 @@ ---- -description: Add a code-defined table enrichment (registry entry) under `apps/sim/enrichments/` backed by an ordered provider cascade, ensuring every provider tool it calls has hosted-key support. Use when adding a per-row table enrichment that fills cells via existing Sim tools. -argument-hint: ---- - -# Adding a Table Enrichment - -Enrichments are code-defined entries in `apps/sim/enrichments/` that run **directly per table row** (no workflow). Each enrichment declares inputs, outputs, and an ordered list of **providers**; the cascade runner tries providers in order and the first non-empty result fills the cell. Each provider calls one existing Sim tool via `executeTool`, which injects the workspace's BYOK key or a **hosted key** and bills usage automatically. - -Because enrichments run on Sim's hosted keys by default, **every provider tool you reference must have hosted-key support** — otherwise it can only run when the workspace brings its own key. This command makes that check a required step. - -## Overview - -| Step | What | Where | -|------|------|-------| -| 1 | Pick the data-source tool(s) for each output | `tools/{service}/` + `tools/registry.ts` | -| 2 | **Verify each tool has `hosting`; if not, run `/add-hosted-key`** | `tools/{service}/{action}.ts` | -| 3 | Write the enrichment definition | `enrichments/{name}/{name}.ts` + `index.ts` | -| 4 | Register it | `enrichments/registry.ts` | -| 5 | Verify | tsc / biome / manual run | - -## Architecture (what you're plugging into) - -- **`enrichments/types.ts`** — `EnrichmentConfig { id, name, description, icon, inputs, outputs, providers }` and `EnrichmentProvider { id, label, toolId, buildParams, mapOutput }`. Providers are **plain data** (no `@/tools` import) so the catalog stays client-safe. -- **`enrichments/providers.ts`** — `toolProvider(...)` (typed passthrough) plus shared input helpers: `str(v)`, `normalizeDomain(v)`, `firstNonEmpty(arr)`, `splitName(fullName)`. -- **`enrichments/run.ts`** — the server-only cascade runner. Calls `executeTool(provider.toolId, { ...params, _context: { workspaceId } })`, accumulates hosted-key cost, returns the first non-empty mapped result. **You do not edit this** — it works for any registry entry. -- **`enrichments/registry.ts`** — `ENRICHMENT_REGISTRY` / `ALL_ENRICHMENTS` / `getEnrichment`. Register new entries here. - -Outputs automatically become table columns; billing, the catalog/sidebar UI, the column meta-header icon, and per-row execution all work with no extra wiring. - -## Step 1: Pick the data-source tool(s) - -For each output the enrichment produces, decide which existing tool provides it. Look up the service's API and the tool in `apps/sim/tools/{service}/` (e.g. `hunter_email_finder`, `pdl_person_enrich`, `pdl_company_enrich`). Confirm: - -- The tool id is registered in `apps/sim/tools/registry.ts`. -- Its `params` accept what you can derive from table columns (read the tool's `params`). -- Its `outputs` / `transformResponse` actually expose the field you need (read the real output shape — don't assume). - -Order providers **cheapest / most-likely-to-hit first**; the cascade stops at the first non-empty result. Apollo / LinkedIn are not hosted-safe (ToS) — don't use them. - -## Step 2: Verify hosted-key support — chain to `/add-hosted-key` if missing - -**This is the required gate.** For every tool a provider calls, open `apps/sim/tools/{service}/{action}.ts` and check for a `hosting` block: - -```typescript -hosting: { - envKeyPrefix: 'SERVICE_API_KEY', - apiKeyParam: 'apiKey', - byokProviderId: 'service', - pricing: { /* ... */ }, - rateLimit: { /* ... */ }, -} -``` - -- **If `hosting` is present** — good. Note the `envKeyPrefix`; the deployment needs `{PREFIX}_COUNT` + `{PREFIX}_1..N` env vars set for the hosted key to actually resolve at runtime (ops concern, not code). If those env vars aren't set in the target environment, the provider will only run with a workspace BYOK key. -- **If `hosting` is absent** — the tool can't use a Sim-provided key, so the enrichment would silently produce blank cells on hosted Sim. **Stop and run `/add-hosted-key `** to add hosted-key support to that tool first, then come back. Do this for every provider tool that lacks it. - -Why it matters: the cascade runner only bills (and only reads `output.cost.total`) when `executeTool` injected a hosted key, which requires the tool's `hosting` config. No `hosting` → no hosted key → the enrichment depends entirely on per-workspace BYOK. - -## Step 3: Write the enrichment definition - -Create `apps/sim/enrichments/{name}/{name}.ts` and a barrel `index.ts`. Mirror the existing entries (`work-email`, `phone-number`, `company-domain`, `company-info`). - -```typescript -import { SomeIcon } from '@sim/emcn/icons' -import { filterUndefined } from '@sim/utils/object' -import { normalizeDomain, splitName, str, toolProvider } from '@/enrichments/providers' -import type { EnrichmentConfig } from '@/enrichments/types' - -export const myEnrichment: EnrichmentConfig = { - id: 'my-enrichment', - name: 'My Enrichment', - description: 'One concise sentence describing what it finds.', - icon: SomeIcon, - inputs: [ - // Person enrichments take a single canonical `fullName` (Clay-style); - // split it with splitName() for tools that need first/last. - { id: 'fullName', name: 'Full name', type: 'string', required: true }, - { id: 'companyDomain', name: 'Company domain', type: 'string' }, - ], - outputs: [{ id: 'value', name: 'value', type: 'string' }], - providers: [ - toolProvider({ - id: 'provider-a', - label: 'Provider A', - toolId: 'service_action', // must have `hosting` (Step 2) - buildParams: (inputs) => { - // Return null when there aren't enough inputs → cascade skips this provider. - const name = splitName(inputs.fullName) - const domain = normalizeDomain(inputs.companyDomain) - if (!name || !domain) return null - return { domain, first_name: name.firstName, last_name: name.lastName } - }, - mapOutput: (output) => { - // Return { [outputId]: value } on a hit, or null to fall through. - const value = str(output.value) - return value ? { value } : null - }, - }), - // ...additional fallback providers, in priority order. - ], -} -``` - -```typescript -// apps/sim/enrichments/{name}/index.ts -export { myEnrichment } from './my-enrichment' -``` - -Rules: -- Keep the file **client-safe**: import only `@sim/emcn/icons`, `@sim/utils/*`, `@/enrichments/providers`, and the types. **Never import `@/tools`** here — the runner does the tool call. -- `buildParams` returns `null` when inputs are insufficient (provider skipped). `mapOutput` returns `null`/empty for a miss (falls through). Use `filterUndefined` when assembling optional tool params; coerce numbers explicitly (don't pass `''` to number outputs). -- Output `id`s are the keys `mapOutput` returns; output `name`s are the default column names (the user can rename them in the config). - -## Step 4: Register it - -In `apps/sim/enrichments/registry.ts`, import and add the entry (catalog order is registration order): - -```typescript -import { myEnrichment } from '@/enrichments/my-enrichment' - -export const ENRICHMENT_REGISTRY: EnrichmentRegistry = { - // ...existing - [myEnrichment.id]: myEnrichment, -} -``` - -## Step 5: Verify - -1. `bun run type-check` (from `apps/sim`) and `bunx biome check` on the changed files. -2. In a table → **+ New column → Enrichments** → pick the new enrichment, map its inputs to columns, name the output column(s), Save. Confirm it appears in the catalog with its icon/description. -3. With hosted keys (or a workspace BYOK key) configured for each provider's service, run a row and confirm the cell fills; the dev-server log shows `Enrichment hit { provider }`. A row whose providers all miss completes blank; a row where every provider errored shows an error cell. - -## Checklist - -- [ ] Each output mapped to a real tool field (verified against the tool's `params`/`outputs`) -- [ ] **Every provider tool has a `hosting` block — ran `/add-hosted-key` for any that didn't** -- [ ] Providers ordered cheapest / most-likely-first; Apollo/LinkedIn not used -- [ ] Enrichment file is client-safe (no `@/tools` import); uses `toolProvider` + shared helpers -- [ ] `buildParams` returns `null` on insufficient inputs; `mapOutput` returns `null` on a miss -- [ ] Registered in `enrichments/registry.ts` -- [ ] tsc + biome clean; created and ran the column end-to-end diff --git a/.claude/commands/add-feature-flag.md b/.claude/commands/add-feature-flag.md deleted file mode 100644 index 07e38a50443..00000000000 --- a/.claude/commands/add-feature-flag.md +++ /dev/null @@ -1,94 +0,0 @@ ---- -description: Add a runtime feature flag (AppConfig-backed on prod, secret fallback off-prod), global by default or optionally gated by org id, user id, or platform admin -argument-hint: ---- - -# Add Feature Flag Skill - -You add a **runtime feature flag** to Sim that can change on prod with no redeploy (AWS AppConfig). Prefer a global on/off flag unless the rollout actually needs per-organization, per-user, or platform-admin targeting. When AppConfig isn't the source of truth, the flag falls back to a single **secret** (on/off only). - -## When to use this vs `env-flags.ts` - -- **Feature flag** (`@/lib/core/config/feature-flags.ts`): runtime global on/off by default, optionally scoped by `userId`/`orgId`/admin. This skill. -- **Env flag** (`@/lib/core/config/env-flags.ts`): deploy-time capability/environment detection (`isProd`, `isHosted`, `isBillingEnabled`). A module-load boolean. **Do not add gated flags here.** - -If the user wants a fixed per-deployment toggle, send them to `env-flags.ts` instead. - -## The flag model - -A flag's **gating rule lives only in the hosted AppConfig document**. It is ON for a context when any configured clause matches: - -```ts -interface FeatureFlagRule { - enabled?: boolean // global default for everyone - orgIds?: string[] // allowlisted organization ids - userIds?: string[] // allowlisted user ids - adminEnabled?: boolean // platform admins (user.role === 'admin') -} -``` - -Critically, **none of this is expressible in code** — gating (especially `adminEnabled`) can only be set through AppConfig, so no environment can grant access from a code literal. Off-AppConfig (self-hosted/OSS/local), a flag is simply on or off, derived from its fallback secret. - -## Steps - -1. **Confirm the granularity before editing code.** If the user has not already specified it, stop and ask: - - > Should `` be a global on/off flag (recommended), or does it need rollout targeting by organization, user, and/or platform admin? - - - Recommend **global**. Do not infer scoped gating merely because the call site already has a user or organization id. - - If the user chooses scoped gating but does not name the dimensions, ask which of organization, user, and platform admin it needs. Wire only the selected dimensions. - - If the user wants a fixed per-deployment toggle rather than a runtime AppConfig flag, use `env-flags.ts` instead. - -2. **Define the flag.** Add one entry to the `FEATURE_FLAGS` registry in `apps/sim/lib/core/config/feature-flags.ts`. Each entry is the flag's whole definition — name (kebab-case key), `description`, and the `fallback` secret consulted when AppConfig isn't the source of truth (truthy ⇒ on globally): - - ```ts - const FEATURE_FLAGS = { - '': { - description: '', - fallback: '', - }, - } - ``` - - `fallback` is the env/secret key (typed as `keyof typeof env`), so add `` to `apps/sim/lib/core/config/env.ts` first (and the deployment's secret store) — it won't typecheck otherwise. Do **not** add org/user/admin defaults here — that gating exists only in AppConfig. Adding the entry makes `` a valid `FeatureFlagName`. - -3. **Gate the call site at the chosen granularity.** For the recommended global mode, pass no context: - - ```ts - import { isFeatureEnabled } from '@/lib/core/config/feature-flags' - - if (await isFeatureEnabled('')) { - // gated behavior - } - ``` - - Do not fetch, resolve, or thread through user or organization context solely for a global flag. - - For scoped rollout, pass only the dimensions the user selected. Admin status is resolved internally, so ordinary callers pass `userId`, not a role: - - ```ts - import { isFeatureEnabled } from '@/lib/core/config/feature-flags' - - if (await isFeatureEnabled('', { userId, orgId })) { - // gated behavior - } - ``` - - - Organization targeting uses `orgId`; user and platform-admin targeting require `userId`. - - Missing ids are fine — a clause with no matching id is skipped; with no `userId`, the admin clause resolves to `false` without a DB read. - - Admin routes that already know the caller is an admin may pass `{ userId, isAdmin: true }` to skip the role lookup. - - **Client/UI flags:** resolve server-side (in a server component, route, or loader) and pass the boolean down as a prop. There is no client AppConfig. - -4. **(Prod) configure in AppConfig.** The infra `feature-flags` profile schema is permissive, so a new flag needs **no infra change**. Operators add the flag to the hosted `feature-flags` document using `enabled` for global rollout or only the selected `orgIds`/`userIds`/`adminEnabled` clauses for scoped rollout, then start a `sim--fast` deployment (see the AppConfig runbook in the infra README — same flow as `access-control`). The fallback secret only applies when AppConfig is disabled. - -5. **Test.** Add a case to `apps/sim/lib/core/config/feature-flags.test.ts` that matches the chosen granularity. For a global flag, exercise `isFeatureEnabled('')` with an AppConfig `enabled` rule and toggle the fallback secret for the off-AppConfig path. For scoped rollout, cover only the selected clauses and mock `isPlatformAdmin` when testing `adminEnabled`. - -6. **Clean up after rollout.** When the feature ships to everyone, delete the flag's entry from `FEATURE_FLAGS`, the `` env entry, the AppConfig document, the call sites, and the test. Leaving dead flags around is the main failure mode of flag systems. - -## Notes - -- Flag keys are `kebab-case`. -- Never read flags via raw `fetch` or a new AppConfig client — always go through `isFeatureEnabled` / `getFeatureFlags`. -- Never bake gating into code. The fallback is a single boolean secret; org/user/admin scoping is AppConfig-only. -- Never add or propagate request context unless the user chose scoped rollout. -- The admin check reads the DB **replica** (`dbReplica`) and is resolved lazily, so an admin-gated flag adds at most one cheap replica read, and only when `adminEnabled` is the deciding clause. diff --git a/.claude/commands/add-hosted-key.md b/.claude/commands/add-hosted-key.md deleted file mode 100644 index cf1b4a4b4b1..00000000000 --- a/.claude/commands/add-hosted-key.md +++ /dev/null @@ -1,301 +0,0 @@ ---- -description: Add hosted API key support to a tool so Sim provides the key (metered and billed to the workspace) when a user has not brought their own. Use when adding a `hosting` config to a tool under `apps/sim/tools/{service}/`. -argument-hint: ---- - -# Adding Hosted Key Support to a Tool - -When a tool has hosted key support, Sim provides its own API key if the user hasn't configured one (via BYOK or env var). Usage is metered and billed to the workspace. - -## Overview - -| Step | What | Where | -|------|------|-------| -| 1 | Register BYOK provider ID | `tools/types.ts`, `lib/api/contracts/byok-keys.ts` | -| 2 | Research the API's pricing and rate limits | API docs / pricing page (before writing any code) | -| 3 | Add `hosting` config to the tool | `tools/{service}/{action}.ts` | -| 4 | Hide API key field when hosted | `blocks/blocks/{service}.ts` | -| 5 | Add to BYOK settings UI | BYOK settings component (`byok.tsx`) | -| 6 | Summarize pricing and throttling comparison | Output to user (after all code changes) | - -## Step 1: Register the BYOK Provider ID - -Add the new provider to the `BYOKProviderId` union in `tools/types.ts`: - -```typescript -export type BYOKProviderId = - | 'openai' - | 'anthropic' - // ...existing providers - | 'your_service' -``` - -Then add the same provider id to the `byokProviderIdSchema` enum in `lib/api/contracts/byok-keys.ts` (this is what the byok-keys route validates against): - -```typescript -export const byokProviderIdSchema = z.enum([ - 'openai', - 'anthropic', - // ...existing providers - 'your_service', -]) -``` - -## Step 2: Research the API's Pricing Model and Rate Limits - -**Before writing any `getCost` or `rateLimit` code**, look up the service's official documentation for both pricing and rate limits. You need to understand: - -### Pricing - -1. **How the API charges** — per request, per credit, per token, per step, per minute, etc. -2. **Whether the API reports cost in its response** — look for fields like `creditsUsed`, `costDollars`, `tokensUsed`, or similar in the response body or headers -3. **Whether cost varies by endpoint/options** — some APIs charge more for certain features (e.g., Firecrawl charges 1 credit/page base but +4 for JSON format, +4 for enhanced mode) -4. **The dollar-per-unit rate** — what each credit/token/unit costs in dollars on our plan - -### Rate Limits - -1. **What rate limits the API enforces** — requests per minute/second, tokens per minute, concurrent requests, etc. -2. **Whether limits vary by plan tier** — free vs paid vs enterprise often have different ceilings -3. **Whether limits are per-key or per-account** — determines whether adding more hosted keys actually increases total throughput -4. **What the API returns when rate limited** — HTTP 429, `Retry-After` header, error body format, etc. -5. **Whether there are multiple dimensions** — some APIs limit both requests/min AND tokens/min independently - -Search the API's docs/pricing page (use WebSearch/WebFetch). Capture the pricing model as a comment in `getCost` so future maintainers know the source of truth. - -### Setting Our Rate Limits - -Our rate limiter (`lib/core/rate-limiter/hosted-key/`) uses a token-bucket algorithm applied **per billing actor** (workspace). It supports two modes: - -- **`per_request`** — simple; just `requestsPerMinute`. Good when the API charges flat per-request or cost doesn't vary much. -- **`custom`** — `requestsPerMinute` plus additional `dimensions` (e.g., `tokens`, `search_units`). Each dimension has its own `limitPerMinute` and an `extractUsage` function that reads actual usage from the response. Use when the API charges on a variable metric (tokens, credits) and you want to cap that metric too. - -When choosing values for `requestsPerMinute` and any dimension limits: - -- **Stay well below the API's per-key limit** — our keys are shared across all workspaces. If the API allows 60 RPM per key and we have 3 keys, the global ceiling is ~180 RPM. Set the per-workspace limit low enough (e.g., 20-60 RPM) that many workspaces can coexist without collectively hitting the API's ceiling. -- **Account for key pooling** — our round-robin distributes requests across `N` hosted keys, so the effective API-side rate per key is `(total requests) / N`. But per-workspace limits are enforced *before* key selection, so they apply regardless of key count. -- **Prefer conservative defaults** — it's easy to raise limits later but hard to claw back after users depend on high throughput. - -## Step 3: Add `hosting` Config to the Tool - -Add a `hosting` object to the tool's `ToolConfig`. This tells the execution layer how to acquire hosted keys, calculate cost, and rate-limit. - -```typescript -hosting: { - envKeyPrefix: 'YOUR_SERVICE_API_KEY', - apiKeyParam: 'apiKey', - byokProviderId: 'your_service', - pricing: { - type: 'custom', - getCost: (_params, output) => { - if (output.creditsUsed == null) { - throw new Error('Response missing creditsUsed field') - } - const creditsUsed = output.creditsUsed as number - const cost = creditsUsed * 0.001 // dollars per credit - return { cost, metadata: { creditsUsed } } - }, - }, - rateLimit: { - mode: 'per_request', - requestsPerMinute: 100, - }, -}, -``` - -### Hosted Key Env Var Convention - -Keys use a numbered naming pattern driven by a count env var: - -``` -YOUR_SERVICE_API_KEY_COUNT=3 -YOUR_SERVICE_API_KEY_1=sk-... -YOUR_SERVICE_API_KEY_2=sk-... -YOUR_SERVICE_API_KEY_3=sk-... -``` - -The `envKeyPrefix` value (`YOUR_SERVICE_API_KEY`) determines which env vars are read at runtime. Adding more keys only requires bumping the count and adding the new env var. - -### Pricing: Prefer API-Reported Cost - -Always prefer using cost data returned by the API (e.g., `creditsUsed`, `costDollars`). This is the most accurate because it accounts for variable pricing tiers, feature modifiers, and plan-level discounts. - -**When the API reports cost** — use it directly and throw if missing: - -```typescript -pricing: { - type: 'custom', - getCost: (params, output) => { - if (output.creditsUsed == null) { - throw new Error('Response missing creditsUsed field') - } - // $0.001 per credit — from https://example.com/pricing - const cost = (output.creditsUsed as number) * 0.001 - return { cost, metadata: { creditsUsed: output.creditsUsed } } - }, -}, -``` - -**When the API does NOT report cost** — compute it from params/output based on the pricing docs, but still validate the data you depend on: - -```typescript -pricing: { - type: 'custom', - getCost: (params, output) => { - if (!Array.isArray(output.searchResults)) { - throw new Error('Response missing searchResults, cannot determine cost') - } - // Serper: 1 credit for <=10 results, 2 credits for >10 — from https://serper.dev/pricing - const credits = Number(params.num) > 10 ? 2 : 1 - return { cost: credits * 0.001, metadata: { credits } } - }, -}, -``` - -**`getCost` must always throw** if it cannot determine cost. Never silently fall back to a default — this would hide billing inaccuracies. - -### Capturing Cost Data from the API - -If the API returns cost info, capture it in `transformResponse` so `getCost` can read it from the output: - -```typescript -transformResponse: async (response: Response) => { - const data = await response.json() - return { - success: true, - output: { - results: data.results, - creditsUsed: data.creditsUsed, // pass through for getCost - }, - } -}, -``` - -For async/polling tools, capture it in `postProcess` when the job completes: - -```typescript -if (jobData.status === 'completed') { - result.output = { - data: jobData.data, - creditsUsed: jobData.creditsUsed, - } -} -``` - -## Step 4: Hide the API Key Field When Hosted - -In the block config (`blocks/blocks/{service}.ts`), add `hideWhenHosted: true` to the API key subblock. This hides the field on hosted Sim since the platform provides the key: - -```typescript -{ - id: 'apiKey', - title: 'API Key', - type: 'short-input', - placeholder: 'Enter your API key', - password: true, - required: true, - hideWhenHosted: true, -}, -``` - -The visibility is controlled by `isSubBlockHidden()` in `lib/workflows/subblocks/visibility.ts`, which checks both the `isHosted` feature flag (`hideWhenHosted`) and optional env var conditions (`hideWhenEnvSet`). - -### Excluding Specific Operations from Hosted Key Support - -When a block has multiple operations but some operations should **not** use a hosted key (e.g., the underlying API is deprecated, unsupported, or too expensive), use the **duplicate apiKey subblock** pattern. This is the same pattern Exa uses for its `research` operation: - -1. **Remove the `hosting` config** from the tool definition for that operation — it must not have a `hosting` object at all. -2. **Duplicate the `apiKey` subblock** in the block config with opposing conditions: - -```typescript -// API Key — hidden when hosted for operations with hosted key support -{ - id: 'apiKey', - title: 'API Key', - type: 'short-input', - placeholder: 'Enter your API key', - password: true, - required: true, - hideWhenHosted: true, - condition: { field: 'operation', value: 'unsupported_op', not: true }, -}, -// API Key — always visible for unsupported_op (no hosted key support) -{ - id: 'apiKey', - title: 'API Key', - type: 'short-input', - placeholder: 'Enter your API key', - password: true, - required: true, - condition: { field: 'operation', value: 'unsupported_op' }, -}, -``` - -Both subblocks share the same `id: 'apiKey'`, so the same value flows to the tool. The conditions ensure only one is visible at a time. The first has `hideWhenHosted: true` and shows for all hosted operations; the second has no `hideWhenHosted` and shows only for the excluded operation — meaning users must always provide their own key for that operation. - -To exclude multiple operations, use an array: `{ field: 'operation', value: ['op_a', 'op_b'] }`. - -**Reference implementations:** -- **Exa** (`blocks/blocks/exa.ts`): `exa_research` operation excluded from hosting — duplicate `apiKey` pair around lines ~348-365 -- **Google Maps** (`blocks/blocks/google_maps.ts`): `speed_limits` operation excluded from hosting (deprecated Roads API) - -## Step 5: Add to the BYOK Settings UI - -Add an entry to the `PROVIDERS` array in the BYOK settings component so users can bring their own key. You need the service icon from `components/icons.tsx`: - -```typescript -{ - id: 'your_service', - name: 'Your Service', - icon: YourServiceIcon, - description: 'What this service does', - placeholder: 'Enter your API key', -}, -``` - -## Step 6: Summarize Pricing and Throttling Comparison - -After all code changes are complete, output a detailed summary to the user covering: - -### What to include - -1. **API's pricing model** — how the service charges (per token, per credit, per request, etc.), the specific rates found in docs, and whether the API reports cost in responses. -2. **Our `getCost` approach** — how we calculate cost, what fields we depend on, and any assumptions or estimates (especially when the API doesn't report exact dollar cost). -3. **API's rate limits** — the documented limits (RPM, TPM, concurrent, etc.), which plan tier they apply to, and whether they're per-key or per-account. -4. **Our `rateLimit` config** — what we set for `requestsPerMinute` (and dimensions if custom mode), why we chose those values, and how they compare to the API's limits. -5. **Key pooling impact** — how many hosted keys we expect, and how round-robin distribution affects the effective per-key rate at the API. -6. **Gaps or risks** — anything the API charges for that we don't meter, rate limit dimensions we chose not to enforce, or pricing that may be inaccurate due to variable model/tier costs. - -### Format - -Present this as a structured summary with clear headings. Example: - -``` -### Pricing -- **API charges**: $X per 1M tokens (input), $Y per 1M tokens (output) — varies by model -- **Response reports cost?**: No — only token counts in `usage` field -- **Our getCost**: Estimates cost at $Z per 1M total tokens based on median model pricing -- **Risk**: Actual cost varies by model; our estimate may over/undercharge for cheap/expensive models - -### Throttling -- **API limits**: 300 RPM per key (paid tier), 60 RPM (free tier) -- **Per-key or per-account**: Per key — more keys = more throughput -- **Our config**: 60 RPM per workspace (per_request mode) -- **With N keys**: Effective per-key rate is (total RPM across workspaces) / N -- **Headroom**: Comfortable — even 10 active workspaces at full rate = 600 RPM / 3 keys = 200 RPM per key, under the 300 RPM API limit -``` - -This summary helps reviewers verify that the pricing and rate limiting are well-calibrated and surfaces any risks that need monitoring. - -## Checklist - -- [ ] Provider added to `BYOKProviderId` in `tools/types.ts` -- [ ] Provider added to `byokProviderIdSchema` enum in `lib/api/contracts/byok-keys.ts` -- [ ] API pricing docs researched — understand per-unit cost and whether the API reports cost in responses -- [ ] API rate limits researched — understand RPM/TPM limits, per-key vs per-account, and plan tiers -- [ ] `hosting` config added to the tool with `envKeyPrefix`, `apiKeyParam`, `byokProviderId`, `pricing`, and `rateLimit` -- [ ] `getCost` throws if required cost data is missing from the response -- [ ] Cost data captured in `transformResponse` or `postProcess` if API provides it -- [ ] `hideWhenHosted: true` added to the API key subblock in the block config -- [ ] Provider entry added to the BYOK settings UI with icon and description -- [ ] Env vars documented: `{PREFIX}_COUNT` and `{PREFIX}_1..N` -- [ ] Pricing and throttling summary provided to reviewer diff --git a/.claude/commands/add-integration.md b/.claude/commands/add-integration.md deleted file mode 100644 index 08d8fc92f08..00000000000 --- a/.claude/commands/add-integration.md +++ /dev/null @@ -1,1000 +0,0 @@ ---- -description: Add a complete Sim integration from API docs, covering tools, block, icon, optional triggers, registrations, resolved-secret/model-input safety, and integration conventions. Use when introducing a new service under `apps/sim/tools`, `apps/sim/blocks`, and `apps/sim/triggers`. -argument-hint: [api-docs-url] ---- - -# Add Integration Skill - -You are an expert at adding complete integrations to Sim. This skill orchestrates the full process of adding a new service integration. - -## Overview - -Adding an integration involves these steps in order: -1. **Research** - Read the service's API documentation -2. **Create Tools** - Build tool configurations for each API operation -3. **Create Block** - Build the block UI configuration -4. **Add Icon** - Add the service's brand icon -5. **Create Triggers** (optional) - If the service supports webhooks -6. **Register** - Register tools, block, and triggers in their registries -7. **Configure Deployment Availability** - Wire OAuth client and service-account metadata -8. **Generate and Validate the Catalog** - Regenerate docs/catalog artifacts and run drift checks - -## Step 1: Research the API - -Before writing any code: -1. Use Context7 to find official documentation: `mcp__context7__resolve-library-id`, then fetch with `mcp__context7__query-docs` -2. Or use WebFetch to read API docs directly -3. Identify: - - Authentication method (OAuth, API Key, both) - - Available operations (CRUD, search, etc.) - - Required vs optional parameters - - Response structures - -### Hard Rule: No Guessed Response Schemas - -If the official docs do not clearly show the response JSON shape for an endpoint, you MUST stop and tell the user exactly which outputs are unknown. - -- Do NOT guess response field names -- Do NOT infer nested JSON paths from related endpoints -- Do NOT invent output properties just because they seem likely -- Do NOT implement `transformResponse` against unverified payload shapes - -If response schemas are missing or incomplete, do one of the following before proceeding: -1. Ask the user for sample responses -2. Ask the user for test credentials so you can verify the live payload -3. Reduce the scope to only endpoints whose response shapes are documented -4. Leave the tool unimplemented and explicitly report why - -## Step 2: Create Tools - -### Directory Structure -``` -apps/sim/tools/{service}/ -├── index.ts # Barrel exports -├── types.ts # TypeScript interfaces -├── {action1}.ts # Tool for action 1 -├── {action2}.ts # Tool for action 2 -└── ... -``` - -### Key Patterns - -**types.ts:** -```typescript -import type { ToolResponse } from '@/tools/types' - -export interface {Service}{Action}Params { - accessToken: string // For OAuth services - // OR - apiKey: string // For API key services - - requiredParam: string - optionalParam?: string -} - -export interface {Service}Response extends ToolResponse { - output: { - // Define output structure - } -} -``` - -**Tool file pattern:** -```typescript -export const {service}{Action}Tool: ToolConfig = { - id: '{service}_{action}', - name: '{Service} {Action}', - description: '...', - version: '1.0.0', - - oauth: { required: true, provider: '{service}' }, // If OAuth - - params: { - accessToken: { type: 'string', required: true, visibility: 'hidden', description: '...' }, - // ... other params - }, - - request: { url, method, headers, body }, - - transformResponse: async (response) => { - const data = await response.json() - return { - success: true, - output: { - field: data.field ?? null, // Always handle nullables - }, - } - }, - - outputs: { /* ... */ }, -} -``` - -### Critical Rules -- `visibility: 'hidden'` for OAuth tokens -- `visibility: 'user-only'` for API keys and user credentials -- `visibility: 'user-or-llm'` for operation parameters -- Always use `?? null` for nullable API response fields -- Always use `?? []` for optional array fields -- Set `optional: true` for outputs that may not exist -- Never output raw JSON dumps - extract meaningful fields -- When using `type: 'json'` and you know the object shape, define `properties` with the inner fields so downstream consumers know the structure. Only use bare `type: 'json'` when the shape is truly dynamic -- If you do not know the response JSON shape from docs or verified examples, you MUST tell the user and stop. Never guess outputs or response mappings. - -### Resolved Secrets at Model and Persistence Boundaries - -Classify every request field before implementing the tool: - -This is opt-in, not a blanket integration migration. Add a model-input declaration only when the -service's official documentation or an unambiguous local execution path proves that the exact -field is consumed by an AI model. If that cannot be established, preserve existing tool behavior -and leave the field unannotated. - -- **Ordinary provider/API input:** leave it unchanged. Explicit `{{...}}` references resolve and are - sent with their normal request semantics. A URL, domain, resource ID, control field, or opaque - payload is not model-visible merely because the provider is AI-backed or may process the - referenced resource later. -- **Text or structured content consumed by an AI model:** declare `request.modelInput` with - `mode: 'project'` and select only the exact model-visible fields. The shared executor replaces - activated Sim secrets with canonical `{{NAME}}` labels before request formatting. For nested or - JSON-string fields, use a small shared selector plus `applyProjected`; verify that selecting the - rebuilt params reproduces the projected selection. -- **Serialized model content sent directly to an external provider:** include the serialized - top-level param in `request.modelInput`. Project the private copy before the existing request - formatter parses it; keep formatter behavior deterministic when a whole-value placeholder is not - valid in the serialized grammar. Do not introduce a second hard-rejection path. -- **Opaque model input owned by an authenticated internal route** such as inline audio, image, - video, or document bytes: add `privateProvenance` to a projected request, or use - `mode: 'private-provenance'` when there is no textual projection. Do not select storage keys, - paths, signed URLs, or ordinary remote URLs as byte provenance; the owning route must authorize - stored bytes independently at model egress. The route must call - `validateOpaqueModelInputProvenance` before downloading or sending content to the model and must - apply the workspace-file provenance guard before reading a persisted workspace file. -- **Sim-owned durable storage or internal execution handoff** that can later enter a workflow/model - (table cells, Agent memory, knowledge documents/chunks, workspace-file contents, or child-workflow - input): transport encrypted field-scoped provenance with `request.secretProvenance`. The - authenticated receiver validates the exact selection and scope, strips the private envelope, and - persists, imports, or propagates it at the owning boundary. Preserve shared legacy behavior for - headerless internal calls and rows/files whose provenance marker is `NULL`; never invent a - tool-local migration rule. - -Hard rules: - -- Never substitute secret plaintext into source or serialize plaintext provenance. -- Never hand-roll private provenance headers/envelopes; the shared `executeTool` boundary owns - transport and strips private metadata from functional results. -- Never attach private provenance to an external URL or to `directExecution`. Project proven - model-visible external fields with `request.modelInput`; otherwise preserve ordinary request - semantics. Use an authenticated internal route when encrypted provenance must cross the boundary. -- Never sanitize arbitrary third-party tool results. Projection applies only to secrets activated - by Sim's resolved-secret provenance for that execution/tool call. -- Do not add provenance merely because a value is persisted, returned by a tool, or appears in a - filename. Require a concrete Sim `{{...}}` resolution path and a later model/log boundary. If an - unsupported field can resolve a secret but does not justify durable tracking (for example a - `file_write` path), reject it at that exact ingress. -- At diagnostic boundaries, project only values carrying execution-scoped provenance. Ordinary - provider responses, filenames, URLs, and errors remain unchanged when Sim did not resolve a - secret into them. - -Add focused tests covering named projection, ordinary identical text without provenance, nested and -serialized shape handling, unchanged ordinary external inputs, malformed/incomplete private metadata -failing closed, headerless legacy requests, and absence of private metadata in the public tool result. -For durable sinks, also cover legacy `NULL` markers, exact-empty new writes, tracked secret writes, -stale/missing sidecars, and scope isolation. - -## Step 3: Create Block - -### File Location -`apps/sim/blocks/blocks/{service}.ts` - -### Block Structure -```typescript -import { {Service}Icon } from '@/components/icons' -import type { BlockConfig } from '@/blocks/types' -import { AuthMode, IntegrationType } from '@/blocks/types' -import { getScopesForService } from '@/lib/oauth/utils' - -export const {Service}Block: BlockConfig = { - type: '{service}', - name: '{Service}', - description: '...', - longDescription: '...', - docsLink: 'https://docs.sim.ai/integrations/{service}', - category: 'tools', - integrationType: IntegrationType.X, // Primary category (see IntegrationType enum) - tags: ['oauth', 'api'], // Cross-cutting tags (see IntegrationTag type) - bgColor: '#HEXCOLOR', - icon: {Service}Icon, - authMode: AuthMode.OAuth, // or AuthMode.ApiKey - - subBlocks: [ - // Operation dropdown - { - id: 'operation', - title: 'Operation', - type: 'dropdown', - options: [ - { label: 'Operation 1', id: 'action1' }, - { label: 'Operation 2', id: 'action2' }, - ], - value: () => 'action1', - }, - // Credential field - { - id: 'credential', - title: '{Service} Account', - type: 'oauth-input', - serviceId: '{service}', - requiredScopes: getScopesForService('{service}'), - required: true, - }, - // Conditional fields per operation - // ... - ], - - tools: { - access: ['{service}_action1', '{service}_action2'], - config: { - tool: (params) => `{service}_${params.operation}`, - }, - }, - - outputs: { /* ... */ }, -} -``` - -### Key SubBlock Patterns - -**Condition-based visibility:** -```typescript -{ - id: 'resourceId', - title: 'Resource ID', - type: 'short-input', - condition: { field: 'operation', value: ['read', 'update', 'delete'] }, - required: { field: 'operation', value: ['read', 'update', 'delete'] }, -} -``` - -**DependsOn for cascading selectors:** -```typescript -{ - id: 'project', - type: 'project-selector', - dependsOn: ['credential'], -}, -{ - id: 'issue', - type: 'file-selector', - dependsOn: ['credential', 'project'], -} -``` - -**Basic/Advanced mode for dual UX:** -```typescript -// Basic: Visual selector -{ - id: 'channelSelector', - type: 'channel-selector', - mode: 'basic', - canonicalParamId: 'channel', - dependsOn: ['credential'], -}, -// Advanced: Manual input -{ - id: 'channelId', - type: 'short-input', - mode: 'advanced', - canonicalParamId: 'channel', -} -``` - -Note neither subblock `id` is `channel` — the canonical id is a third name that both members map -onto, and it is the only one that survives serialization. - -**Critical Canonical Param Rules:** -- `canonicalParamId` must NOT match any subblock's `id` in the block -- `canonicalParamId` must be unique **block-wide**, not per operation. `buildCanonicalIndex` keys - groups by `canonicalParamId` across all subblocks and a group holds exactly one `basicId`, so two - operations that each need their own pair must use two different canonical ids -- Only use `canonicalParamId` to link basic/advanced alternatives for the same logical parameter. - A pair carries ONE concept — for files that means upload (basic) + file reference (advanced), as - in Gmail attachments (`blocks/blocks/gmail.ts`). Never overload the advanced side with alternate - identifiers like a URL or a provider asset ID; give those their own subblocks, mark all the - mutually exclusive sources `required: false`, and enforce "exactly one" at execution -- `mode` only controls UI visibility, NOT serialization. Without `canonicalParamId`, both basic and advanced field values would be sent -- Every subblock `id` must be unique within the block. Duplicate IDs cause conflicts even with different conditions -- **Required consistency:** If one subblock in a canonical group has `required: true`, ALL subblocks in that group must have `required: true` (prevents bypassing validation by switching modes) -- **Inputs section:** Must list canonical param IDs (e.g., `fileId`), NOT raw subblock IDs (e.g., `fileSelector`, `manualFileId`) -- **Params function:** Must use canonical param IDs, NOT raw subblock IDs (raw IDs are deleted after canonical transformation) - -### BlockMeta (Required) - -Export a `{Service}BlockMeta` in the same file as the block — **minimum 7 templates**. See `.agents/skills/add-block/SKILL.md` → "BlockMeta (Required)" for valid `modules` and `category` values and the full pattern. - -```typescript -export const {Service}BlockMeta = { - tags: ['tag1', 'tag2'], - templates: [ - { - icon: {Service}Icon, - title: '{Service} ', - prompt: 'Build a workflow that...', // concrete trigger → transformation → output - modules: ['agent', 'workflows'], - category: 'operations', - tags: ['automation'], - alsoIntegrations: ['slack'], // when the prompt references another service - }, - // ... at least 6 more - ], -} as const satisfies BlockMeta -``` - -## Step 4: Add Icon - -### File Location -`apps/sim/components/icons.tsx` - -### Pattern -```typescript -export function {Service}Icon(props: SVGProps) { - return ( - - {/* SVG paths from user-provided SVG */} - - ) -} -``` - -### Getting Icons -**Do NOT search for icons yourself.** At the end of implementation, ask the user to provide the SVG: - -``` -I've completed the integration. Before I can add the icon, please provide the SVG for {Service}. -You can usually find this in the service's brand/press kit page, or copy it from their website. - -Paste the SVG code here and I'll convert it to a React component. -``` - -Once the user provides the SVG: -1. Extract the SVG paths/content -2. Create a React component that spreads props -3. Ensure viewBox is preserved from the original SVG - -### Theme-safety (bare rendering) — REQUIRED - -The icon renders both inside its colored `bgColor` tile AND "bare" (no tile) on a -neutral page — e.g. the home **Suggested actions** list — in both light and dark -mode. A monochrome logo whose paths hardcode a single near-white or near-black -fill is invisible bare on the matching background (white-on-white in light mode, -black-on-black in dark mode). - -Rules when adding the SVG: - -- **Monochrome logos** (a single white or black mark): draw the shape with - `fill='currentColor'`, not `fill='#fff'` / `fill='#000000'`. It then inherits - white inside dark tiles, near-black inside light tiles (via - `getTileIconColorClass`), and the theme-aware `var(--text-icon)` bare — legible - everywhere. Do NOT set `iconColor` for these. -- **Multi-color brand logos** (their own vivid fills): keep the hardcoded fills. - They read on any background. Only set `iconColor` (a vivid brand hex, never a - near-black/near-white tile color) if the bare icon should adopt a brand tint. -- A large white shape with a tiny vivid accent (e.g. a logo where the body is the - white negative space) still vanishes bare — convert the body to `currentColor`. - -Verify with `bun run check:bare-icons` (also runs in CI). It flags purely -monochrome hazards; for partial-accent logos, eyeball the suggested-actions list -in both light and dark mode. - -## Step 5: Create Triggers (Optional) - -If the service supports webhooks, create triggers using the generic `buildTriggerSubBlocks` helper. - -### Directory Structure -``` -apps/sim/triggers/{service}/ -├── index.ts # Barrel exports -├── utils.ts # Trigger options, setup instructions, extra fields -├── {event_a}.ts # Primary trigger (includes dropdown) -├── {event_b}.ts # Secondary triggers (no dropdown) -└── webhook.ts # Generic webhook (optional) -``` - -### Key Pattern - -```typescript -import { buildTriggerSubBlocks } from '@/triggers' -import { {service}TriggerOptions, {service}SetupInstructions, build{Service}ExtraFields } from './utils' - -// Primary trigger - includeDropdown: true -export const {service}EventATrigger: TriggerConfig = { - id: '{service}_event_a', - subBlocks: buildTriggerSubBlocks({ - triggerId: '{service}_event_a', - triggerOptions: {service}TriggerOptions, - includeDropdown: true, // Only for primary trigger! - setupInstructions: {service}SetupInstructions('Event A'), - extraFields: build{Service}ExtraFields('{service}_event_a'), - }), - // ... -} - -// Secondary triggers - no dropdown -export const {service}EventBTrigger: TriggerConfig = { - id: '{service}_event_b', - subBlocks: buildTriggerSubBlocks({ - triggerId: '{service}_event_b', - triggerOptions: {service}TriggerOptions, - // No includeDropdown! - setupInstructions: {service}SetupInstructions('Event B'), - extraFields: build{Service}ExtraFields('{service}_event_b'), - }), - // ... -} -``` - -### Connect to Block -```typescript -import { getTrigger } from '@/triggers' - -export const {Service}Block: BlockConfig = { - triggers: { - enabled: true, - available: ['{service}_event_a', '{service}_event_b'], - }, - subBlocks: [ - // Tool fields... - ...getTrigger('{service}_event_a').subBlocks, - ...getTrigger('{service}_event_b').subBlocks, - ], -} -``` - -See `/add-trigger` skill for complete documentation. - -## Step 6: Register Everything - -### Tools Registry (`apps/sim/tools/registry.ts`) - -```typescript -// Add import (alphabetically) -import { - {service}Action1Tool, - {service}Action2Tool, -} from '@/tools/{service}' - -// Add to tools object (alphabetically) -export const tools: Record = { - // ... existing tools ... - {service}_action1: {service}Action1Tool, - {service}_action2: {service}Action2Tool, -} -``` - -Then regenerate the generated tool metadata and commit it: - -```bash -bun run tool-metadata:generate -``` - -Client code reads `params`/`outputs` from these artifacts rather than importing -the registry, so a tool you add, change or remove is invisible to the UI until they are regenerated, -and CI fails on stale ones. See `.agents/skills/tool-registry-boundary/SKILL.md`. - -### Block Registry (`apps/sim/blocks/registry-maps.ts`) - -The data maps (`BLOCK_REGISTRY` + `BLOCK_META_REGISTRY`) live in `registry-maps.ts`; `registry.ts` holds only the accessor functions. Add the import and an entry to each map alphabetically: - -```typescript -// Add import (alphabetically) -import { {Service}Block, {Service}BlockMeta } from '@/blocks/blocks/{service}' - -// Add to the config map (alphabetically) -export const BLOCK_REGISTRY: Record = { - // ... existing blocks ... - {service}: {Service}Block, -} - -// Add to the catalog-meta map (alphabetically) -export const BLOCK_META_REGISTRY: Record = { - // ... existing metas ... - {service}: {Service}BlockMeta, -} -``` - -### Trigger Registry (`apps/sim/triggers/registry.ts`) - If triggers exist - -```typescript -// Add import (alphabetically) -import { - {service}EventATrigger, - {service}EventBTrigger, - {service}WebhookTrigger, -} from '@/triggers/{service}' - -// Add to TRIGGER_REGISTRY (alphabetically) -export const TRIGGER_REGISTRY: TriggerRegistry = { - // ... existing triggers ... - {service}_event_a: {service}EventATrigger, - {service}_event_b: {service}EventBTrigger, - {service}_webhook: {service}WebhookTrigger, -} -``` - -## Step 7: Configure Deployment Availability - -Do this for every visible OAuth integration. API-key and unauthenticated integrations do not need -an OAuth client capability. - -The block's `oauth-input.serviceId` is the canonical link between the generated integration catalog, -the OAuth service configuration, deployment availability, and the setup CLI. - -1. Ensure the block has exactly one distinct OAuth `serviceId` and that it matches the canonical - service entry in `apps/sim/lib/oauth/oauth.ts`. -2. Confirm `resolveOAuthClientCapabilityId(serviceId)` resolves to the intended provider entry in - `OAUTH_CLIENT_CAPABILITIES` in `apps/sim/lib/core/config/env-capabilities.ts`. Google and - Microsoft service IDs deliberately share provider-level capabilities. -3. For a new OAuth provider, add the required client fields to `OAUTH_CLIENT_CAPABILITIES`, add - every referenced field to the env schema in `apps/sim/lib/core/config/env.ts`, and add the - matching `text` or `secret` entries to `OAUTH_CLIENT_SETUP_FIELDS` in - `scripts/setup/capability-config.ts`. Do not create integration-specific setup logic or infer - secret fields from naming; the CLI mapping is exhaustively checked against the runtime fields. -4. If the canonical OAuth service has `serviceAccountProviderId`, add the matching projection to - `SERVICE_ACCOUNT_METADATA_BY_OAUTH_SERVICE_ID` in - `apps/sim/lib/integrations/service-account-metadata.ts`. Use: - - no `deploymentRequirement` when the service-account path works independently of OAuth client fields; - - `'oauth-client'` when it requires the same deployment OAuth client fields; - - `'preview-gated'` when availability is controlled by the service-account preview block. - -Never add a permissive fallback for missing capability metadata. A visible OAuth integration without -a resolvable capability must fail validation. - -## Step 8: Generate and Validate the Catalog - -Run the documentation generator: -```bash -bun run scripts/generate-docs.ts -bun run integration-catalog:check -``` - -This creates `apps/docs/content/docs/en/integrations/{service}.mdx` — one page per service carrying the block's Actions and, if it has one, its Triggers section. Never hand-edit generated pages; the only editable region is the `{/* MANUAL-CONTENT */}` block (see `scripts/README.md`). - -The same generator refreshes `apps/sim/lib/integrations/integrations.json`. The catalog check then -derives the deployment-relevant fields from the executable block registry and compares them with the -committed projection. Review the generated diff and keep only intentional changes. - -## V2 Integration Pattern - -If creating V2 versions (API-aligned outputs): - -1. **V2 Tools** - Add `_v2` suffix, version `2.0.0`, flat outputs -2. **V2 Block** - Add `_v2` type, use `createVersionedToolSelector` -3. **V1 Block** - Add `(Legacy)` to name, set `hideFromToolbar: true` -4. **Registry** - Register both versions - -```typescript -// In registry -{service}: {Service}Block, // V1 (legacy, hidden) -{service}_v2: {Service}V2Block, // V2 (visible) -``` - -## Complete Checklist - -### Tools -- [ ] Created `tools/{service}/` directory -- [ ] Created `types.ts` with all interfaces -- [ ] Created tool file for each operation -- [ ] All params have correct visibility -- [ ] All nullable fields use `?? null` -- [ ] All optional outputs have `optional: true` -- [ ] Created `index.ts` barrel export -- [ ] Registered all tools in `tools/registry.ts` -- [ ] Ran `bun run tool-metadata:generate` and committed the regenerated artifacts -- [ ] Classified every model-visible, opaque, Sim-durable, and internal-execution request field -- [ ] Added shared model-input projection or private provenance only where required; ordinary - external resource locators and control inputs retain their request semantics -- [ ] Confirmed ordinary third-party tool results are not generically sanitized -- [ ] Added provenance compatibility and fail-closed boundary tests where applicable - -### Block -- [ ] Created `blocks/blocks/{service}.ts` -- [ ] Set `integrationType` to the correct `IntegrationType` enum value -- [ ] Set `tags` array with all applicable `IntegrationTag` values -- [ ] Defined operation dropdown with all operations -- [ ] Added credential field with `requiredScopes: getScopesForService('{service}')` -- [ ] Added conditional fields per operation -- [ ] Set up dependsOn for cascading selectors -- [ ] Configured tools.access with all tool IDs -- [ ] Configured tools.config.tool selector -- [ ] Defined outputs matching tool outputs -- [ ] Registered block + meta in `blocks/registry-maps.ts` (`BLOCK_REGISTRY` / `BLOCK_META_REGISTRY`) -- [ ] If triggers: set `triggers.enabled` and `triggers.available` -- [ ] If triggers: spread trigger subBlocks with `getTrigger()` -- [ ] Exported `{Service}BlockMeta` with at least 7 templates - -### OAuth Scopes (if OAuth service) -- [ ] Defined scopes in `lib/oauth/oauth.ts` under `OAUTH_PROVIDERS` -- [ ] Added scope descriptions in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts` -- [ ] Used `getCanonicalScopesForProvider()` in `auth.ts` (never hardcode) -- [ ] Used `getScopesForService()` in block `requiredScopes` (never hardcode) - -### Deployment Availability (if OAuth service) -- [ ] Block declares exactly one distinct `oauth-input.serviceId` -- [ ] `resolveOAuthClientCapabilityId(serviceId)` resolves to the intended `OAUTH_CLIENT_CAPABILITIES` entry -- [ ] Every new OAuth capability field exists in `apps/sim/lib/core/config/env.ts` -- [ ] Runtime OAuth fields live in `OAUTH_CLIENT_CAPABILITIES`; matching CLI input modes live in the exhaustively checked `OAUTH_CLIENT_SETUP_FIELDS` -- [ ] If `serviceAccountProviderId` is configured, `SERVICE_ACCOUNT_METADATA_BY_OAUTH_SERVICE_ID` has the matching projection and deployment requirement - -### Icon -- [ ] Asked user to provide SVG -- [ ] Added icon to `components/icons.tsx` -- [ ] Icon spreads props correctly -- [ ] Monochrome marks use `fill='currentColor'` (not hardcoded white/black) so the icon renders bare in light AND dark mode — verified with `bun run check:bare-icons` - -### Triggers (if service supports webhooks) -- [ ] Created `triggers/{service}/` directory -- [ ] Created `utils.ts` with options, instructions, and extra fields helpers -- [ ] Primary trigger uses `includeDropdown: true` -- [ ] Secondary triggers do NOT have `includeDropdown` -- [ ] All triggers use `buildTriggerSubBlocks` helper -- [ ] Created `index.ts` barrel export -- [ ] Registered all triggers in `triggers/registry.ts` - -### Docs -- [ ] Ran `bun run scripts/generate-docs.ts` -- [ ] Verified docs file created -- [ ] Reviewed and committed the generated `apps/sim/lib/integrations/integrations.json` change -- [ ] `bun run integration-catalog:check` passes - -### Final Validation (Required) -- [ ] Read every tool file and cross-referenced inputs/outputs against the API docs -- [ ] Verified block subBlocks cover all required tool params with correct conditions -- [ ] Verified block outputs match what the tools actually return -- [ ] Verified `tools.config.params` correctly maps and coerces all param types -- [ ] Verified every tool output and `transformResponse` path against documented or live-verified JSON responses -- [ ] If any response schema remained unknown, explicitly told the user instead of guessing -- [ ] `{Service}BlockMeta` exported with at least 7 templates, each having `icon`, `title`, `prompt`, `modules`, `category`, and `tags` - -## Example Command - -When the user asks to add an integration: - -``` -User: Add a Stripe integration - -You: I'll add the Stripe integration. Let me: - -1. First, research the Stripe API using Context7 -2. Create the tools for key operations (payments, subscriptions, etc.) -3. Create the block with operation dropdown -4. Register everything -5. Generate docs -6. Ask you for the Stripe icon SVG - -[Proceed with implementation...] - -[After completing steps 1-5...] - -I've completed the Stripe integration. Before I can add the icon, please provide the SVG for Stripe. -You can usually find this in the service's brand/press kit page, or copy it from their website. - -Paste the SVG code here and I'll convert it to a React component. -``` - -## File Handling - -When your integration handles file uploads or downloads, follow these patterns to work with `UserFile` objects consistently. - -### What is a UserFile? - -A `UserFile` is the standard file representation in Sim: - -```typescript -interface UserFile { - id: string // Unique identifier - name: string // Original filename - url: string // Presigned URL for download - size: number // File size in bytes - type: string // MIME type (e.g., 'application/pdf') - base64?: string // Optional base64 content (if small file) - key?: string // Internal storage key - context?: object // Storage context metadata -} -``` - -### File Input Pattern (Uploads) - -For tools that accept file uploads, **always route through an internal API endpoint** rather than calling external APIs directly. This ensures proper file content retrieval. - -#### 1. Block SubBlocks for File Input - -Use the basic/advanced mode pattern: - -```typescript -// Basic mode: File upload UI -{ - id: 'uploadFile', - title: 'File', - type: 'file-upload', - canonicalParamId: 'file', // Maps to 'file' param - placeholder: 'Upload file', - mode: 'basic', - multiple: false, - required: true, - condition: { field: 'operation', value: 'upload' }, -}, -// Advanced mode: Reference from previous block -{ - id: 'fileRef', - title: 'File', - type: 'short-input', - canonicalParamId: 'file', // Same canonical param - placeholder: 'Reference file (e.g., {{file_block.output}})', - mode: 'advanced', - required: true, - condition: { field: 'operation', value: 'upload' }, -}, -``` - -**Critical:** `canonicalParamId` must NOT match any subblock `id`. - -#### 2. Normalize File Input in Block Config - -In `tools.config.tool`, use `normalizeFileInput` to handle all input variants: - -```typescript -import { normalizeFileInput } from '@/blocks/utils' - -tools: { - config: { - tool: (params) => { - // Normalize file from basic (uploadFile), advanced (fileRef), or legacy (fileContent) - const normalizedFile = normalizeFileInput( - params.uploadFile || params.fileRef || params.fileContent, - { single: true } - ) - if (normalizedFile) { - params.file = normalizedFile - } - return `{service}_${params.operation}` - }, - }, -} -``` - -#### 3. Create Special Internal Tool Execution Route - -Create `apps/sim/app/api/tools/{service}/{action}/route.ts`. This raw route pattern is only for an integration's provider-execution boundary when it needs special file normalization, large-body handling, or protocol behavior. It is not the pattern for CRUD or other operations on protected Sim resources. For those, use the `migrate-application-operation` skill and an authorized application use case with the ordinary internal/v2 route builders. - -Internal tool routes are HTTP boundaries and follow the same contract policy as public routes — define the request/response shape in `apps/sim/lib/api/contracts/tools/{service}.ts` (or an existing aggregate) and validate with canonical helpers from `@/lib/api/server`. Never write a route-local Zod schema. Authenticate and perform cheap admission before parsing or downloading files. - -```typescript -// apps/sim/lib/api/contracts/tools/{service}.ts -import { z } from 'zod' -import { defineRouteContract } from '@/lib/api/contracts' -import { FileInputSchema } from '@/lib/uploads/utils/file-schemas' - -export const {service}UploadBodySchema = z.object({ - accessToken: z.string(), - file: FileInputSchema.optional().nullable(), - fileContent: z.string().optional().nullable(), - // ... other params -}) - -export const {service}UploadResponseSchema = z.object({ - success: z.boolean(), - output: z.object({ id: z.string(), url: z.string() }).optional(), - error: z.string().optional(), -}) - -export const {service}UploadContract = defineRouteContract({ - method: 'POST', - path: '/api/tools/{service}/upload', - body: {service}UploadBodySchema, - response: { mode: 'json', schema: {service}UploadResponseSchema }, -}) - -export type {Service}UploadBody = z.input -export type {Service}UploadResponse = z.output -``` - -```typescript -// apps/sim/app/api/tools/{service}/upload/route.ts -import { createLogger } from '@sim/logger' -import { NextResponse, type NextRequest } from 'next/server' -import { {service}UploadContract } from '@/lib/api/contracts/tools/{service}' -import { parseRequest } from '@/lib/api/server' -import { checkInternalAuth } from '@/lib/auth/hybrid' -import { generateRequestId } from '@/lib/core/utils/request' -import { withRouteHandler } from '@/lib/core/utils/with-route-handler' -import { type RawFileInput } from '@/lib/uploads/utils/file-schemas' -import { processFilesToUserFiles } from '@/lib/uploads/utils/file-utils' -import { downloadFileFromStorage } from '@/lib/uploads/utils/file-utils.server' - -const logger = createLogger('{Service}UploadAPI') - -export const POST = withRouteHandler(async (request: NextRequest) => { - const requestId = generateRequestId() - - // Auth always runs BEFORE parseRequest — never validate untrusted input before authenticating. - const authResult = await checkInternalAuth(request, { requireWorkflowId: false }) - if (!authResult.success) { - return NextResponse.json({ success: false, error: 'Unauthorized' }, { status: 401 }) - } - - const parsed = await parseRequest({service}UploadContract, request, {}) - if (!parsed.success) return parsed.response - const data = parsed.data.body - - let fileBuffer: Buffer - let fileName: string - - // Prefer UserFile input, fall back to legacy base64 - if (data.file) { - const userFiles = processFilesToUserFiles([data.file as RawFileInput], requestId, logger) - if (userFiles.length === 0) { - return NextResponse.json({ success: false, error: 'Invalid file' }, { status: 400 }) - } - const userFile = userFiles[0] - fileBuffer = await downloadFileFromStorage(userFile, requestId, logger) - fileName = userFile.name - } else if (data.fileContent) { - // Legacy: base64 string (backwards compatibility) - fileBuffer = Buffer.from(data.fileContent, 'base64') - fileName = 'file' - } else { - return NextResponse.json({ success: false, error: 'File required' }, { status: 400 }) - } - - // Now call external API with fileBuffer - const response = await fetch('https://api.{service}.com/upload', { - method: 'POST', - headers: { Authorization: `Bearer ${data.accessToken}` }, - body: new Uint8Array(fileBuffer), // Convert Buffer for fetch - }) - - // ... handle response -}) -``` - -#### 4. Update Tool to Use Internal Route - -```typescript -export const {service}UploadTool: ToolConfig = { - id: '{service}_upload', - // ... - params: { - file: { type: 'file', required: false, visibility: 'user-or-llm' }, - fileContent: { type: 'string', required: false, visibility: 'hidden' }, // Legacy - }, - request: { - url: '/api/tools/{service}/upload', // Internal route - method: 'POST', - body: (params) => ({ - accessToken: params.accessToken, - file: params.file, - fileContent: params.fileContent, - }), - }, -} -``` - -### File Output Pattern (Downloads) - -For tools that return files, use `FileToolProcessor` to store files and return `UserFile` objects. - -#### In Tool transformResponse - -```typescript -import { FileToolProcessor } from '@/executor/utils/file-tool-processor' - -transformResponse: async (response, context) => { - const data = await response.json() - - // Process file outputs to UserFile objects - const fileProcessor = new FileToolProcessor(context) - const file = await fileProcessor.processFileData({ - data: data.content, // base64 or buffer - mimeType: data.mimeType, - filename: data.filename, - }) - - return { - success: true, - output: { file }, - } -} -``` - -#### In API Route (for complex file handling) - -```typescript -// Return file data that FileToolProcessor can handle -return NextResponse.json({ - success: true, - output: { - file: { - data: base64Content, - mimeType: 'application/pdf', - filename: 'document.pdf', - }, - }, -}) -``` - -### Key Helpers Reference - -| Helper | Location | Purpose | -|--------|----------|---------| -| `normalizeFileInput` | `@/blocks/utils` | Normalize file params in block config | -| `processFilesToUserFiles` | `@/lib/uploads/utils/file-utils` | Convert raw inputs to UserFile[] | -| `downloadFileFromStorage` | `@/lib/uploads/utils/file-utils.server` | Get file Buffer from UserFile | -| `FileToolProcessor` | `@/executor/utils/file-tool-processor` | Process tool output files | -| `isUserFile` | `@/lib/core/utils/user-file` | Type guard for UserFile objects | -| `FileInputSchema` | `@/lib/uploads/utils/file-schemas` | Zod schema for file validation | - -### Advanced Mode for Optional Fields - -Optional fields that are rarely used should be set to `mode: 'advanced'` so they don't clutter the basic UI. Examples: pagination tokens, time range filters, sort order, max results, reply settings. - -### WandConfig for Complex Inputs - -Use `wandConfig` for fields that are hard to fill out manually: -- **Timestamps**: Use `generationType: 'timestamp'` to inject current date context into the AI prompt -- **JSON arrays**: Use `generationType: 'json-object'` for structured data -- **Complex queries**: Use a descriptive prompt explaining the expected format - -```typescript -{ - id: 'startTime', - title: 'Start Time', - type: 'short-input', - mode: 'advanced', - wandConfig: { - enabled: true, - prompt: 'Generate an ISO 8601 timestamp. Return ONLY the timestamp string.', - generationType: 'timestamp', - }, -} -``` - -### OAuth Scopes (Centralized System) - -Scopes are maintained in a single source of truth and reused everywhere: - -1. **Define scopes** in `lib/oauth/oauth.ts` under `OAUTH_PROVIDERS[provider].services[service].scopes` -2. **Add descriptions** in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts` for the OAuth modal UI -3. **Reference in auth.ts** using `getCanonicalScopesForProvider(providerId)` from `@/lib/oauth/utils` -4. **Reference in blocks** using `getScopesForService(serviceId)` from `@/lib/oauth/utils` - -**Never hardcode scope arrays** in `auth.ts` or block `requiredScopes`. Always import from the centralized source. - -```typescript -// In auth.ts (Better Auth config) -scopes: getCanonicalScopesForProvider('{service}'), - -// In block credential sub-block -requiredScopes: getScopesForService('{service}'), -``` - -### Common Gotchas - -1. **OAuth serviceId must match** - The `serviceId` in oauth-input must match the OAuth provider configuration -2. **All tool IDs MUST be snake_case** - `stripe_create_payment`, not `stripeCreatePayment`. This applies to tool `id` fields, registry keys, `tools.access` arrays, and `tools.config.tool` return values -3. **Block type is snake_case** - `type: 'stripe'`, not `type: 'Stripe'` -4. **Alphabetical ordering** - Keep imports and registry entries alphabetically sorted -5. **Required can be conditional** - Use `required: { field: 'op', value: 'create' }` instead of always true -6. **DependsOn clears options** - When a dependency changes, selector options are refetched -7. **Never pass Buffer directly to fetch** - Convert to `new Uint8Array(buffer)` for TypeScript compatibility -8. **Always handle legacy file params** - Keep hidden `fileContent` params for backwards compatibility -9. **Optional fields use advanced mode** - Set `mode: 'advanced'` on rarely-used optional fields -10. **Complex inputs need wandConfig** - Timestamps, JSON arrays, and other hard-to-type values should have `wandConfig` enabled -11. **Never hardcode scopes** - Use `getScopesForService()` in blocks and `getCanonicalScopesForProvider()` in auth.ts -12. **Always add scope descriptions** - New scopes must have entries in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts` -13. **OAuth service IDs need deployment capabilities** - Every visible OAuth integration must resolve through `OAUTH_CLIENT_CAPABILITIES`; shared Google/Microsoft aliases map to their provider capability -14. **Keep runtime and presentation separate** - Runtime OAuth fields live in `env-capabilities.ts`; CLI input modes live in the exhaustively checked `scripts/setup/capability-config.ts` mapping diff --git a/.claude/commands/add-managed-cli.md b/.claude/commands/add-managed-cli.md deleted file mode 100644 index 4b5bfcd0840..00000000000 --- a/.claude/commands/add-managed-cli.md +++ /dev/null @@ -1,148 +0,0 @@ ---- -description: Add or upgrade a curated, immutable managed CLI for Sim Function sandboxes, including client-safe catalog metadata, a pinned server-only installation recipe, checksum and executable verification, provider compatibility, PATH propagation, content-addressed image identity, and tests. Use when adding a CLI to the Sandbox managed-CLI selector or changing an existing managed CLI version or recipe. ---- - -# Add a Managed CLI - -Add CLIs through the curated registry. Never turn this surface into arbitrary commands or package names: system packages already cover validated Debian/APT coordinates, while managed CLIs require immutable artifacts and reproducible recipes. - -## Read First - -Read these live sources before editing; do not copy their current entries into this skill: - -1. `apps/sim/lib/execution/remote-sandbox/cli-tools.ts` — persisted IDs and client-safe metadata. -2. `apps/sim/lib/execution/remote-sandbox/cli-tools.server.ts` — server-only recipes and recipe helpers. -3. `apps/sim/lib/execution/remote-sandbox/cli-tools.test.ts` — catalog and supply-chain invariants. -4. `apps/sim/lib/execution/remote-sandbox/cli-tools-boundary.test.ts` — client/server import boundary. -5. `apps/sim/lib/execution/remote-sandbox/sandbox-spec.ts` — content-addressed hash inputs. - -Read `resolve.ts` and `e2b.ts` only when changing provisioning mechanics. A normal catalog addition should not require UI, API, database, resolver, or provider edits; those paths derive from the registries. - -Do not modify the dedicated Function base image or the separate Mothership Shell template for a normal managed CLI addition. Managed recipes layer on the Function base image. Do not change `MAX_SANDBOX_CLI_TOOLS` from ten unless the user separately requests a product-limit change. - -## 1. Verify the Upstream Release - -Use primary upstream release documentation and official artifacts. Establish all of the following before writing code: - -- Exact version and stable Linux x86-64 artifact URL. Never use `latest`, mutable redirects, or an unversioned installer. -- SHA-256 for the exact artifact. Prefer a publisher-signed checksum; otherwise download the official artifact and compute it independently. -- Archive layout and the exact executable paths to install. -- Noninteractive, credential-free verification commands for every advertised executable, normally version commands. -- Required PATH entries under `/opt/sim-cli`. -- E2B and Daytona compatibility. Default to both only when the same Linux recipe works on both. - -When the user does not name a version, select the current stable upstream release from primary sources and state the exact version chosen. Do not silently choose a prerelease or infer a version from an unverified secondary source. - -Reject curl-to-shell installers, `npm install`, `pip install`, distro package repositories, arbitrary user commands, and artifacts from unofficial mirrors. Never place credentials, tokens, login commands, or account configuration in an image recipe or build log; authentication is runtime-only. - -## 2. Choose an Immutable ID - -Use `@-r`. - -- New upstream version: append a new ID ending in `-r1`. -- Recipe-only change for the same upstream version: append `-r2`, `-r3`, and so on. -- Never mutate or delete an existing ID or recipe. Persisted sandboxes must continue resolving to the bytes and behavior they selected. -- On upgrade, retain the old ID and recipe and set its metadata to `selectable: false`. Only the newest version keeps the public label selectable. - -Before shipping the first upgrade for a tool family, verify that editing a sandbox cannot leave both the retired and replacement IDs selected. If the generic selector and API validation do not already replace or reject colliding versions, address that once at the generic registry boundary with focused UI and contract tests; never special-case the individual CLI or silently install two versions that expose the same executable. - -Recipe identity includes the ID, revision, and SHA-256 in the sandbox image hash. Keeping old entries is what makes that identity reproducible rather than merely cache-busting. - -## 3. Add Client-Safe Metadata - -In `cli-tools.ts`: - -1. Append the ID to `SANDBOX_CLI_TOOL_IDS` in the same order used by the metadata and recipe registries. -2. Add a `SANDBOX_CLI_TOOLS` entry whose key and `id` exactly match. -3. Provide a unique selectable `label`, concise `description`, existing `category`, and useful executable/vendor aliases in `searchTerms`. -4. Add a category only when no existing category is accurate, then ensure it has at least one selectable entry. - -Keep this file safe for client bundles. It must not contain artifact URLs, checksums, install commands, verification commands, PATH recipes, provider SDKs, or imports from `cli-tools.server.ts`. - -The API enum and searchable grouped selector derive from this registry. Do not add parallel option arrays or route-local wire types. - -## 4. Add the Server-Only Recipe - -In `cli-tools.server.ts`, use the narrowest existing helper: - -- `defineBinaryRecipe` for one downloaded binary. -- `defineTarGzipRecipe` or `defineZipRecipe` for archives containing binaries. -- `defineVerifiedRecipe` for a vendor archive or installer layout that needs explicit commands. -- A direct typed entry only when the helpers cannot faithfully model the release. - -Provide every field the recipe contract requires: - -- Exact `version`, `artifactUrl`, `artifactName`, and lowercase 64-character `sha256`. -- Every installed `executable` and a corresponding `verificationCommands` entry. -- Deterministic extraction/install commands into `/opt/sim-cli`; quote fixed paths and clean temporary artifacts. -- `pathEntries` when the executable is not installed into the helper's default `bin` directory. -- `supportedProviders` only when it differs from the E2B-and-Daytona default. -- `revision` when it differs from `1`; it must agree with the ID suffix. - -Verification must prove the command is discoverable through `sandboxCliEnvironment`, not authenticate or contact a user account. Recipe commands run as root during both prebuilt image creation and runtime provisioning. - -If the artifact host is new, add only the exact official hostname to the `officialHosts` allowlist in `cli-tools.test.ts`. Treat that as a supply-chain review, not a way to silence the test. - -## 5. Preserve Generic Behavior - -Confirm the existing generic paths remain sufficient: - -- `sandboxCliToolRecipes` canonicalizes and resolves the recipe. -- `sandboxCliEnvironment` propagates PATH to Python subprocesses, JavaScript subprocesses, and Shell. -- E2B bakes the recipe into the custom image; runtime-strategy providers install it within the Function timeout. -- CLI-only sandboxes remain buildable even with no language packages. -- `hashSandboxSpec` includes recipe ID, revision, and checksum while preserving the legacy hash for an empty CLI list. -- The settings selector derives groups and search aliases from client-safe metadata. - -Do not special-case a CLI in those layers unless the registry contract cannot express a genuine provider requirement. Extend the registry contract generically when multiple CLIs need the same new behavior. - -## 6. Test the Addition - -Extend tests when the new entry introduces behavior not already covered: - -- For every upgrade, add a regression proving the old ID and recipe remain resolvable but non-selectable, while the replacement ID is selectable. -- Add important executable aliases to the table-driven search assertion. -- Add a focused assertion for a multi-executable recipe, custom PATH, or restricted provider. -- Add an opt-in credentialed smoke test only when installation plus a real minimal command cannot be validated without authentication. Read credentials from test-only environment variables, skip by default, create them only at runtime, and always tear down the sandbox. - -Never commit downloaded artifacts or credentials. - -## Required Validation - -From `apps/sim`: - -```bash -bunx vitest run \ - lib/execution/remote-sandbox/cli-tools.test.ts \ - lib/execution/remote-sandbox/cli-tools-boundary.test.ts \ - lib/execution/remote-sandbox/sandbox-spec.test.ts \ - lib/execution/remote-sandbox/resolve.test.ts \ - lib/api/contracts/sandboxes.test.ts \ - 'app/workspace/[workspaceId]/settings/components/sandboxes/utils.test.ts' \ - 'app/workspace/[workspaceId]/settings/components/sandboxes/components/sandbox-editor.test.tsx' -``` - -From the repository root: - -```bash -bun run type-check -bun run check:api-validation -bunx biome check \ - apps/sim/lib/execution/remote-sandbox/cli-tools.ts \ - apps/sim/lib/execution/remote-sandbox/cli-tools.server.ts \ - apps/sim/lib/execution/remote-sandbox/cli-tools.test.ts -git diff --check -``` - -For a new recipe, also exercise its install and every verification command in an actual E2B or Daytona sandbox when credentials and network access are available. Report clearly when only registry/unit validation ran. - -## Completion Checklist - -- [ ] Official immutable Linux x86-64 artifact and SHA-256 verified. -- [ ] Versioned ID appended; old IDs and recipes retained. -- [ ] Client metadata is searchable, categorized, unique, and recipe-free. -- [ ] Server recipe is pinned, integrity-checked, noninteractive, and credential-free. -- [ ] Every advertised executable has an offline verification command and PATH entry. -- [ ] Provider compatibility is explicit and accurate. -- [ ] Catalog, boundary, hash, resolver, type, API-validation, format, and diff checks pass. -- [ ] Real provider installation was tested, or the missing live verification is disclosed. diff --git a/.claude/commands/add-model.md b/.claude/commands/add-model.md deleted file mode 100644 index 9cef8634689..00000000000 --- a/.claude/commands/add-model.md +++ /dev/null @@ -1,221 +0,0 @@ ---- -description: Add a new LLM model to apps/sim/providers/models.ts with specs verified against the provider's live API docs (no hallucination) -argument-hint: [docs-url] ---- - -# Add Model Skill - -You add a new model entry to `apps/sim/providers/models.ts`. **Every numeric and capability claim MUST be derived from a live web fetch of the provider's official docs in this session.** Marketing emails, training data, and your prior knowledge are not sources of truth — they routinely hallucinate pricing, context windows, and capability lists. - -## Hard rules (do not skip) - -1. **Live-fetch or refuse.** Before writing the entry, you must successfully WebFetch the provider's official models/pricing page in this session. If you cannot reach an authoritative source for any field, **mark the field as UNVERIFIED in your report and ask the user before guessing**. Never fill in pricing or capabilities from memory. -2. **Two-source rule for pricing.** Cross-check input/output/cached pricing against at least one secondary source (OpenRouter, Artificial Analysis, CloudPrice, mem0, intuitionlabs). If sources disagree, the provider's own docs win — but flag the disagreement. -3. **Read the code before setting capability flags.** Capability flags are dead unless the provider's implementation under `apps/sim/providers/{provider}/` actually consumes them (see Consumption Matrix below). Setting a flag the provider ignores is a silent bug. -4. **Cite every fact.** Your final report must list the URL each value came from. No URL → not verified. - -## Your Task - -1. Identify provider and model id from user args -2. Live-fetch official docs + pricing page + capability/parameter pages + at least one secondary source -3. Apply the Consumption Matrix to know which capability flags are real -4. Read 2-3 sibling entries in `models.ts` and match their pattern exactly -5. Check the repo-side touchpoints that are NOT data-driven (hosted-key billing, tests, provider code) -6. Insert the entry, run `bun run lint`, print the verification report - -## Step 1: Live source-of-truth lookup - -In priority order — fetch all that exist for the provider: - -| Provider | Models index | Pricing | Reasoning/parameter caveats | -|---|---|---|---| -| OpenAI | platform.openai.com/docs/models | openai.com/api/pricing | platform.openai.com/docs/guides/reasoning | -| Anthropic | docs.anthropic.com/en/docs/about-claude/models | anthropic.com/pricing | docs.anthropic.com/en/docs/build-with-claude/extended-thinking | -| Google (Gemini) | ai.google.dev/gemini-api/docs/models | ai.google.dev/pricing | ai.google.dev/gemini-api/docs/thinking | -| xAI | docs.x.ai/developers/models | docs.x.ai/developers/models (per-model detail page) | docs.x.ai/developers/model-capabilities/text/reasoning | -| Mistral | docs.mistral.ai/getting-started/models/models_overview | mistral.ai/pricing | n/a | -| DeepSeek | api-docs.deepseek.com/quick_start/pricing | same | api-docs.deepseek.com/guides/reasoning_model | -| Groq | console.groq.com/docs/models | groq.com/pricing | n/a | -| Cerebras | inference-docs.cerebras.ai/models | cerebras.ai/pricing | n/a | - -Secondary verification (use at least one): `openrouter.ai//`, `artificialanalysis.ai/models/`, `cloudprice.net/models/-`. - -Use a precise WebFetch prompt: *"Extract for {model_id}: exact model id string, context window in tokens, input price per 1M, cached input price per 1M, output price per 1M, max output tokens, supported reasoning effort levels, accepted parameters (temperature, top_p), release date. Do not fill in fields you cannot find."* - -## Step 2: Consumption Matrix (which provider honors which capability) - -| Capability | Honored by | Effect if set elsewhere | -|---|---|---| -| `temperature` | All providers (passed through if set) | Safe but inert on always-reasoning models that reject it | -| `toolUsageControl` | All providers (provider-level, not per-model) | n/a — set on `ProviderDefinition`, not models | -| `reasoningEffort` | `openai/core.ts`, `azure-openai`, `anthropic/core.ts` (mapped to thinking), `gemini/core.ts` | **Dead on xai, deepseek, mistral, groq, cerebras, openrouter, fireworks, bedrock, vertex** unless their core consumes it — re-grep before assuming | -| `verbosity` | `openai/core.ts`, `azure-openai/index.ts` only | Dead elsewhere | -| `thinking` | `anthropic/core.ts`, `gemini/core.ts` | Dead elsewhere | -| `thinking.streamed` | Docs generator + `getThinkingStreamVisibility` (`models.ts`); `anthropic/core.ts` uses `'summary'` to request `display: 'summarized'` on agent-events runs | **Mandatory on Anthropic-family thinking models** (`agent-stream-docs:check` fails without it); other families fall back to provider defaults | -| `nativeStructuredOutputs` | `anthropic/core.ts`, `fireworks/index.ts`, `openrouter/index.ts` | Dead on openai, xai, google, vertex, bedrock, azure-openai, deepseek, mistral, groq, cerebras | -| `maxOutputTokens` | Read by UI + executor for token estimation | Always meaningful — set if provider documents a cap | -| `computerUse` | `anthropic/core.ts` | Dead elsewhere | -| `deepResearch` | UI flag for routing to deep-research SKUs | Set only on actual deep-research model IDs | -| `memory: false` | Conversation persistence opt-out | Set only when model genuinely cannot maintain history (e.g., deep-research) | - -**Always re-grep before relying on this table** — the codebase moves: - -```bash -rg "reasoningEffort|reasoning_effort" apps/sim/providers// -rg "verbosity" apps/sim/providers// -rg "request\.thinking|thinking:" apps/sim/providers// -rg "supportsNativeStructuredOutputs|nativeStructuredOutputs" apps/sim/providers// -``` - -## Step 3: Match the provider's existing entry pattern - -Open `apps/sim/providers/models.ts`, find `PROVIDER_DEFINITIONS[].models`, read 2-3 sibling entries. Match field order exactly: - -```ts -{ - id: '', - pricing: { - input: , - cachedInput: , // omit if provider doesn't offer caching - output: , - updatedAt: '', - }, - capabilities: { - // only flags the provider actually consumes — see matrix - }, - contextWindow: , - releaseDate: '', - recommended: true, // only if new flagship; ask user before swapping - speedOptimized: true, // only on smallest/fastest tier - deprecated: true, // only on retired models -} -``` - -### Reseller providers (azure-openai, azure-anthropic, vertex, bedrock, openrouter) - -Model id MUST be prefixed: `azure/`, `azure-anthropic/`, `vertex/`, `bedrock/`, `openrouter/`. Pricing usually mirrors the upstream provider but verify on the reseller's own pricing page. - -### Insertion order - -Within a family, newest first (matches existing convention: GPT-5.5 above GPT-5.4 above GPT-5.2). Across families, biggest/flagship at top of list. - -### `recommended` / `speedOptimized` - -- At most one or two `recommended: true` per provider — the current flagship(s). -- If you're adding a new flagship, ask the user before removing `recommended` from the previous flagship. Never silently flip it. -- `speedOptimized: true` only on the smallest/fastest tier (nano, flash-lite, haiku class). - -## Step 4: Repo-side touchpoints beyond the entry - -Adding the `models.ts` entry is most of the job because nearly every consumer is **data-driven** and picks the model up automatically: the ~40 query helpers in `models.ts` / `providers/utils.ts`, the public `/models` catalog (`app/(landing)/models/utils.ts` iterates `PROVIDER_DEFINITIONS`), the agent-block model dropdown, and copilot's `isKnownModelId` / `suggestModelIdsForUnknownModel` validation. The touchpoints below are the exceptions — they are **not** data-driven, so check each one. - -### Hosted = auto-billed, by provider - -`getHostedModels()` in `apps/sim/providers/models.ts` returns **every** model under `openai`, `anthropic`, and `google`: - -```ts -export function getHostedModels(): string[] { - return [ - ...getProviderModels('openai'), - ...getProviderModels('anthropic'), - ...getProviderModels('google'), - ] -} -``` - -So a model added to any of those three providers is **automatically served with Sim's rotating hosted key and billed** to the workspace via `shouldBillModelUsage()` (`providers/utils.ts`). Before you insert: - -- **If the model should be BYOK-only / never-billed**, do NOT drop it under `openai`/`anthropic`/`google` as-is — that silently enrolls it in hosted billing. Confirm hosting/billing intent with the user. (Precedent: Ollama Cloud is a deliberately separate `isReseller` provider specifically to stay BYOK-only/never-billed.) -- **If the model should be hosted**, the deployment must actually have a key for it — the provider's `{PREFIX}_COUNT` / `{PREFIX}_1..N` env vars must be set, or hosted runs fail at execution time. -- State the hosted/billing status explicitly in the verification report. - -### Tests with hardcoded model IDs - -`bun run lint` does **not** run tests. A few tests assert specific model IDs and can break or need updating when you touch a hosted or flagship model: - -- `apps/sim/providers/utils.test.ts` — asserts membership of `getHostedModels()` / `shouldBillModelUsage()` -- `apps/sim/providers/index.test.ts` and serializer tests — reference concrete model IDs - -```bash -rg "|getHostedModels|shouldBillModelUsage" apps/sim/providers/*.test.ts -``` - -If anything matches, run the affected provider tests and update assertions as needed. - -### New API behavior is NOT data-driven - -The Consumption Matrix (Step 2) tells you which capability *flags* are honored by existing provider code. But if the new model needs **net-new** request handling that the provider doesn't implement yet — a new beta header (e.g. Anthropic's `anthropic-beta` structured-outputs header in `anthropic/index.ts`), a new thinking/reasoning encoding, a Responses-API quirk — you must edit `apps/sim/providers//core.ts` / `index.ts`. Setting a flag whose behavior isn't implemented is a silent no-op. When you do edit provider code, reuse the shared helpers rather than hand-rolling: streaming responses are assembled via `createStreamingExecution` (`@/providers/streaming-execution`) and tool schemas via `adaptOpenAIChatToolSchema` / `adaptAnthropicToolSchema` (`@/providers/tool-schema-adapter`). - -### Thinking/reasoning models: `streamed` visibility + generated docs - -If the entry has `capabilities.thinking` or `capabilities.reasoningEffort`, it appears in the autogenerated "Streamed thinking and tool calls" table on the Agent block docs page: - -- **Anthropic-family (`anthropic`, `azure-anthropic`) thinking models MUST declare `capabilities.thinking.streamed`** (`'full' | 'summary' | 'none'`). Verify against Anthropic's current thinking-display and streaming docs: visible thinking returned by the API is summarized, including when Sim opts models whose default display is `omitted` into `display: 'summarized'` on agent-events runs, so current Claude thinking models use `'summary'`. Use `'full'` only if future official API docs explicitly guarantee raw thinking deltas. `bun run agent-stream-docs:check` (CI) fails if the field is missing. -- Other families usually omit the field and inherit the provider default in `getThinkingStreamVisibility` (Gemini/OpenAI → summaries; Bedrock/Meta → none; OpenAI-compatible vendors with documented reasoning fields → full deltas). Set it explicitly only when the model deviates from its family. -- After inserting the entry, run `bun run agent-stream-docs:generate` and commit the regenerated `apps/docs/content/docs/en/workflows/blocks/agent.mdx` — CI diffs it. -- Include the `streamed` value (with its source URL) in the verification report when set. - -### Wrong family entirely? - -- **Embedding or rerank model** → it does NOT go in the `models[]` array. Use `EMBEDDING_MODEL_PRICING` / `RERANK_MODEL_PRICING` in `models.ts` instead. -- **Brand-new provider** (not just a new model under an existing one) → much larger surface: add the id to `ProviderId` in `providers/types.ts`, a registry entry in `providers/registry.ts`, a provider implementation under `providers//` (assemble streaming responses with `createStreamingExecution` and wrap tool schemas with the `@/providers/tool-schema-adapter` helpers), an icon in `components/icons.tsx`, and the `PROVIDER_DEFINITIONS` block. That is beyond this skill — tell the user. - -## Step 5: Write, lint - -```bash -bun run lint -bun run agent-stream-docs:generate # only when the entry has thinking/reasoningEffort -``` - -Lint must pass before reporting done. **If lint fails:** read the error, fix the syntax/typing issue in the entry you just wrote (do not delete the entry — it's the work product), re-run lint, and note the fix in a "Lint adjustments" line in the verification report. Never report done with lint failing. - -## Step 6: Verification report (mandatory format) - -End with this exact structure: - -```markdown -### Verification — - -| Field | Value | Source URL | Status | -|---|---|---|---| -| `id` | `grok-4.3` | https://docs.x.ai/... | ✓ verified | -| `contextWindow` | 1,000,000 | https://docs.x.ai/... + https://openrouter.ai/... | ✓ verified (2 sources agree) | -| `input` | $1.25/M | https://docs.x.ai/... | ✓ verified | -| `cachedInput` | $0.20/M | https://cloudprice.net/... | ⚠️ single source | -| `output` | $2.50/M | https://docs.x.ai/... + https://openrouter.ai/... | ✓ verified | -| `capabilities.temperature` | `{ min: 0, max: 1 }` | matches sibling entries | — pattern-match only | -| `capabilities.reasoningEffort` | NOT SET | provider docs say API rejects it for this model | ✓ correctly omitted | -| `releaseDate` | 2026-04-30 | https://docs.x.ai/... announcement | ✓ verified | -| hosted/billing | BYOK-only (xai not in `getHostedModels`) | `providers/models.ts` | — confirmed intent | - -**Disagreements** -- _none_ OR _OpenRouter says X, provider docs say Y — used Y per provider rule_ - -**Unverified fields** -- _none_ OR _: could not find authoritative source — left as based on sibling pattern; please confirm_ -``` - -If any row is ⚠️ single-source or "unverified," **state it plainly to the user and ask whether to proceed**. Do not silently merge. - -## What to do if you cannot find a source - -Omitting a field is **not the same as verifying it**. Any field you cannot confirm from a live fetch must be **both** omitted from the entry **and** listed as ❓ UNVERIFIED in the report's "Unverified fields" section, with the URLs you attempted. Then ask the user to confirm before merging. - -- Pricing missing → do NOT guess. Omit `cachedInput`. Mark ❓ UNVERIFIED. Ask the user for the price or the docs URL. -- Context window missing → do NOT guess. Ask the user; mark ❓ UNVERIFIED. -- Release date missing → omit the field; mark ❓ UNVERIFIED in the report. -- Capability uncertain → omit the flag (safer than setting a dead/wrong one); mark ❓ UNVERIFIED so the user knows you didn't confirm it either way. - -## Anti-patterns this skill exists to prevent - -- ❌ Trusting a marketing email (xAI's grok-4.3 email claimed "3 reasoning efforts" but the API rejects `reasoning_effort` — verified by official docs only) -- ❌ Setting `nativeStructuredOutputs: true` on xai/openai/google (dead — only anthropic/fireworks/openrouter consume it) -- ❌ Setting `thinking` on non-Anthropic/non-Gemini providers -- ❌ Adding an Anthropic-family thinking model without `thinking.streamed` (CI `agent-stream-docs:check` fails), or skipping `bun run agent-stream-docs:generate` after adding any thinking/reasoning model -- ❌ Setting `verbosity` on anything other than OpenAI gpt-5.x -- ❌ Copying `pricing.updatedAt` from a sibling instead of using today's date -- ❌ Inventing a `cachedInput` price by dividing input by 4 (varies by provider — find an explicit number) -- ❌ Stamping `recommended: true` on the new model without removing it from the previous flagship -- ❌ Adding a BYOK-only model under `openai`/`anthropic`/`google` (silently enrolls it in hosted billing via `getHostedModels()`) -- ❌ Reporting "done" after only `bun run lint` when you touched a hosted (openai/anthropic/google) or flagship model with assertions in `providers/utils.test.ts` -- ❌ Reporting "done" with any UNVERIFIED row in the table diff --git a/.claude/commands/add-tools.md b/.claude/commands/add-tools.md deleted file mode 100644 index 6b390520b64..00000000000 --- a/.claude/commands/add-tools.md +++ /dev/null @@ -1,501 +0,0 @@ ---- -description: Create tool configurations for a Sim integration by reading API docs -argument-hint: [api-docs-url] ---- - -# Add Tools Skill - -You are an expert at creating tool configurations for Sim integrations. Your job is to read API documentation and create properly structured tool files. - -## Your Task - -When the user asks you to create tools for a service: -1. Use Context7 or WebFetch to read the service's API documentation -2. Create the tools directory structure -3. Generate properly typed tool configurations - -## Hard Rule: No Guessed Response Schemas - -If the docs do not clearly show the response JSON for a tool, you MUST tell the user exactly which outputs are unknown and stop short of guessing. - -- Do NOT invent response field names -- Do NOT infer nested paths from nearby endpoints -- Do NOT guess array item shapes -- Do NOT write `transformResponse` against unverified payloads - -If the response shape is unknown, do one of these instead: -1. Ask the user for sample responses -2. Ask the user for test credentials so you can verify live responses -3. Implement only the endpoints whose outputs are documented -4. Leave the tool unimplemented and explicitly say why - -## Directory Structure - -Create files in `apps/sim/tools/{service}/`: -``` -tools/{service}/ -├── index.ts # Barrel export -├── types.ts # Parameter & response types -└── {action}.ts # Individual tool files (one per operation) -``` - -## Tool Configuration Structure - -Every tool MUST follow this exact structure: - -```typescript -import type { {ServiceName}{Action}Params } from '@/tools/{service}/types' -import type { ToolConfig } from '@/tools/types' - -interface {ServiceName}{Action}Response { - success: boolean - output: { - // Define output structure here - } -} - -export const {serviceName}{Action}Tool: ToolConfig< - {ServiceName}{Action}Params, - {ServiceName}{Action}Response -> = { - id: '{service}_{action}', // snake_case, matches tool name - name: '{Service} {Action}', // Human readable - description: 'Brief description', // One sentence - version: '1.0.0', - - // OAuth config (if service uses OAuth) - oauth: { - required: true, - provider: '{service}', // Must match OAuth provider ID - }, - - params: { - // Hidden params (system-injected, only use hidden for oauth accessToken) - accessToken: { - type: 'string', - required: true, - visibility: 'hidden', - description: 'OAuth access token', - }, - // User-only params (credentials, api key, IDs user must provide) - someId: { - type: 'string', - required: true, - visibility: 'user-only', - description: 'The ID of the resource', - }, - // User-or-LLM params (everything else, can be provided by user OR computed by LLM) - query: { - type: 'string', - required: false, // Use false for optional - visibility: 'user-or-llm', - description: 'Search query', - }, - }, - - request: { - url: (params) => `https://api.service.com/v1/resource/${params.id}`, - method: 'POST', - headers: (params) => ({ - Authorization: `Bearer ${params.accessToken}`, - 'Content-Type': 'application/json', - }), - body: (params) => ({ - // Request body - only for POST/PUT/PATCH - // Trim ID fields to prevent copy-paste whitespace errors: - // userId: params.userId?.trim(), - }), - }, - - transformResponse: async (response: Response) => { - const data = await response.json() - return { - success: true, - output: { - // Map API response to output - // Use ?? null for nullable fields - // Use ?? [] for optional arrays - }, - } - }, - - outputs: { - // Define each output field - }, -} -``` - -## Critical Rules for Parameters - -### Visibility Options -- `'hidden'` - System-injected (OAuth tokens, internal params). User never sees. -- `'user-only'` - User must provide (credentials, api keys, account-specific IDs) -- `'user-or-llm'` - User provides OR LLM can compute (search queries, content, filters, most fall into this category) - -### Parameter Types -- `'string'` - Text values -- `'number'` - Numeric values -- `'boolean'` - True/false -- `'json'` - Complex objects (NOT 'object', use 'json') -- `'file'` - Single file -- `'file[]'` - Multiple files - -### Required vs Optional -- Always explicitly set `required: true` or `required: false` -- Optional params should have `required: false` - -## Resolved Secrets and Provenance Boundaries - -- Leave ordinary external API inputs and third-party results unchanged. Add provenance handling only - when an exact field is proven to cross a Sim model, durable-storage, or internal-execution boundary. -- Project AI-consumed text/structured fields with the smallest exact `request.modelInput` selector. -- Treat URLs, domains, resource IDs, and control fields as ordinary request values unless the exact - field is proven model-visible. For serialized external model content, project the serialized - top-level param through `request.modelInput` before the existing formatter parses it; do not add a - separate hard-rejection mechanism. -- For authenticated internal routes, use `privateProvenance` for actual inline/raw model bytes or - `request.secretProvenance` for durable writes and execution handoffs. Do not treat a storage key, - path, signed URL, or remote URL as provenance for fetched bytes; authorize tracked stored bytes at - the owning model-egress boundary. Authenticate first, validate the exact selection and scope, - strip the private envelope, then import or propagate provenance at the receiving boundary. - Preserve documented headerless legacy behavior. -- Never substitute secret plaintext into source, serialize plaintext provenance, hand-roll private - headers, or blanket-sanitize tool results. -- Add focused tests for named projection, identical unproven public text, malformed/incomplete - metadata, metadata stripping, scope isolation, and legacy compatibility where applicable. - -## Critical Rules for Outputs - -### Output Types -- `'string'`, `'number'`, `'boolean'` - Primitives -- `'json'` - Complex objects (use this, NOT 'object') -- `'array'` - Arrays with `items` property -- `'object'` - Objects with `properties` property - -### Optional Outputs -Add `optional: true` for fields that may not exist in the response: -```typescript -closedAt: { - type: 'string', - description: 'When the issue was closed', - optional: true, -}, -``` - -### Typed JSON Outputs - -When using `type: 'json'` and you know the object shape in advance, **always define the inner structure** using `properties` so downstream consumers know what fields are available: - -```typescript -// BAD: Opaque json with no info about what's inside -metadata: { - type: 'json', - description: 'Response metadata', -}, - -// GOOD: Define the known properties -metadata: { - type: 'json', - description: 'Response metadata', - properties: { - id: { type: 'string', description: 'Unique ID' }, - status: { type: 'string', description: 'Current status' }, - count: { type: 'number', description: 'Total count' }, - }, -}, -``` - -For arrays of objects, define the item structure: -```typescript -items: { - type: 'array', - description: 'List of items', - items: { - type: 'object', - properties: { - id: { type: 'string', description: 'Item ID' }, - name: { type: 'string', description: 'Item name' }, - }, - }, -}, -``` - -Only use bare `type: 'json'` without `properties` when the shape is truly dynamic or unknown. - -If the response shape is unknown because the docs do not provide it, you MUST tell the user and stop. Unknown is not the same as dynamic. Never guess outputs. - -## Critical Rules for transformResponse - -### Handle Nullable Fields -ALWAYS use `?? null` for fields that may be undefined: -```typescript -transformResponse: async (response: Response) => { - const data = await response.json() - return { - success: true, - output: { - id: data.id, - title: data.title, - body: data.body ?? null, // May be undefined - assignee: data.assignee ?? null, // May be undefined - labels: data.labels ?? [], // Default to empty array - closedAt: data.closed_at ?? null, // May be undefined - }, - } -} -``` - -### Never Output Raw JSON Dumps -DON'T do this: -```typescript -output: { - data: data, // BAD - raw JSON dump -} -``` - -DO this instead - extract meaningful fields: -```typescript -output: { - id: data.id, - name: data.name, - status: data.status, - metadata: { - createdAt: data.created_at, - updatedAt: data.updated_at, - }, -} -``` - -## Types File Pattern - -Create `types.ts` with interfaces for all params and responses: - -```typescript -import type { ToolResponse } from '@/tools/types' - -// Parameter interfaces -export interface {Service}{Action}Params { - accessToken: string - requiredField: string - optionalField?: string -} - -// Response interfaces (extend ToolResponse) -export interface {Service}{Action}Response extends ToolResponse { - output: { - field1: string - field2: number - optionalField?: string | null - } -} -``` - -## Index.ts Barrel Export Pattern - -```typescript -// Export all tools -export { serviceTool1 } from './{action1}' -export { serviceTool2 } from './{action2}' - -// Export types -export * from './types' -``` - -## Registering Tools - -After creating tools: -1. Import tools in `apps/sim/tools/registry.ts` -2. Add to the `tools` object with snake_case keys (alphabetically): -```typescript -import { serviceActionTool } from '@/tools/{service}' - -export const tools = { - // ... existing tools ... - {service}_{action}: serviceActionTool, -} -``` - -3. Regenerate the tool metadata artifacts: - -```bash -bun run tool-metadata:generate -``` - -Client code reads a tool's `params`/`outputs` from generated metadata rather than -importing the registry, so a tool you add, change or remove is invisible to the UI until -these are regenerated — and CI fails on stale artifacts. Commit the result. See -`.agents/skills/tool-registry-boundary/SKILL.md`. - -## Wiring Tools into the Block (Required) - -After registering in `tools/registry.ts`, you MUST also update the block definition at `apps/sim/blocks/blocks/{service}.ts`. This is not optional — tools are only usable from the UI if they are wired into the block. - -### 1. Add to `tools.access` - -```typescript -tools: { - access: [ - // existing tools... - 'service_new_action', // Add every new tool ID here - ], - config: { ... } -} -``` - -### 2. Add operation dropdown options - -If the block uses an operation dropdown, add an option for each new tool: - -```typescript -{ - id: 'operation', - type: 'dropdown', - options: [ - // existing options... - { label: 'New Action', id: 'new_action' }, // id maps to what tools.config.tool returns - ], -} -``` - -### 3. Add subBlocks for new tool params - -For each new tool, add subBlocks covering all its required params (and optional ones where useful). Apply `condition` to show them only for the right operation, and mark required params with `required`: - -```typescript -// Required param for new_action -{ - id: 'someParam', - title: 'Some Param', - type: 'short-input', - placeholder: 'e.g., value', - condition: { field: 'operation', value: 'new_action' }, - required: { field: 'operation', value: 'new_action' }, -}, -// Optional param — put in advanced mode -{ - id: 'optionalParam', - title: 'Optional Param', - type: 'short-input', - condition: { field: 'operation', value: 'new_action' }, - mode: 'advanced', -}, -``` - -### 4. Update `tools.config.tool` - -Ensure the tool selector returns the correct tool ID for every new operation. The simplest pattern: - -```typescript -tool: (params) => `service_${params.operation}`, -// If operation dropdown IDs already match tool IDs, this requires no change. -``` - -If the dropdown IDs differ from tool IDs, add explicit mappings: - -```typescript -tool: (params) => { - const map: Record = { - new_action: 'service_new_action', - // ... - } - return map[params.operation] ?? `service_${params.operation}` -}, -``` - -### 5. Update `tools.config.params` - -Add any type coercions needed for new params (runs at execution time, after variable resolution): - -```typescript -params: (params) => { - const result: Record = {} - if (params.limit != null && params.limit !== '') result.limit = Number(params.limit) - if (params.newParamName) result.toolParamName = params.newParamName // rename if IDs differ - return result -}, -``` - -### 6. Add new outputs - -Add any new fields returned by the new tools to the block `outputs`: - -```typescript -outputs: { - // existing outputs... - newField: { type: 'string', description: 'Description of new field' }, -} -``` - -### 7. Add new inputs - -Add new subBlock param IDs to the block `inputs` section: - -```typescript -inputs: { - // existing inputs... - someParam: { type: 'string', description: 'Param description' }, - optionalParam: { type: 'string', description: 'Optional param description' }, -} -``` - -### Block wiring checklist - -- [ ] New tool IDs added to `tools.access` -- [ ] Operation dropdown has an option for each new tool -- [ ] SubBlocks cover all required params for each new tool -- [ ] SubBlocks have correct `condition` (only show for the right operation) -- [ ] Optional/rarely-used params set to `mode: 'advanced'` -- [ ] `tools.config.tool` returns correct ID for every new operation -- [ ] `tools.config.params` handles any ID remapping or type coercions -- [ ] New outputs added to block `outputs` -- [ ] New params added to block `inputs` - -## V2 Tool Pattern - -If creating V2 tools (API-aligned outputs), use `_v2` suffix: -- Tool ID: `{service}_{action}_v2` -- Variable name: `{action}V2Tool` -- Version: `'2.0.0'` -- Outputs: Flat, API-aligned (no content/metadata wrapper) - -## Naming Convention - -All tool IDs MUST use `snake_case`: `{service}_{action}` (e.g., `x_create_tweet`, `slack_send_message`). Never use camelCase or PascalCase for tool IDs. - -## Checklist Before Finishing - -- [ ] All tool IDs use snake_case -- [ ] All params have explicit `required: true` or `required: false` -- [ ] All params have appropriate `visibility` -- [ ] All nullable response fields use `?? null` -- [ ] All optional outputs have `optional: true` -- [ ] No raw JSON dumps in outputs -- [ ] Types file has all interfaces -- [ ] Index.ts exports all tools and re-exports types (`export * from './types'`) -- [ ] Tools registered in `tools/registry.ts` -- [ ] `bun run tool-metadata:generate` run and the regenerated artifacts committed -- [ ] Block wired: `tools.access`, dropdown options, subBlocks, `tools.config`, outputs, inputs -- [ ] Model, durable-storage, and internal-execution boundaries use the shared provenance mechanisms - only where a concrete Sim `{{...}}` resolution path requires them -- [ ] Ordinary third-party inputs/results remain unchanged and private metadata never leaves Sim - -## Final Validation (Required) - -After creating all tools, you MUST validate every tool before finishing: - -1. **Read every tool file** you created — do not skip any -2. **Cross-reference with the API docs** to verify: - - All required params are marked `required: true` - - All optional params are marked `required: false` - - Param types match the API (string, number, boolean, json) - - Request URL, method, headers, and body match the API spec - - `transformResponse` extracts the correct fields from the API response - - All output fields match what the API actually returns - - No fields are missing from outputs that the API provides - - No extra fields are defined in outputs that the API doesn't return - - Every output field and JSON path is backed by docs or live-verified sample responses -3. **Verify consistency** across tools: - - Shared types in `types.ts` match all tools that use them - - Tool IDs in the barrel export match the tool file definitions - - Error handling is consistent (error checks, meaningful messages) -4. **If any response schema is still unknown**, explicitly tell the user instead of guessing diff --git a/.claude/commands/add-trigger.md b/.claude/commands/add-trigger.md deleted file mode 100644 index 45065c69f1d..00000000000 --- a/.claude/commands/add-trigger.md +++ /dev/null @@ -1,512 +0,0 @@ ---- -description: Create webhook or polling triggers for a Sim integration -argument-hint: ---- - -# Add Trigger - -You are an expert at creating webhook and polling triggers for Sim. You understand the trigger system, the generic `buildTriggerSubBlocks` helper, polling infrastructure, and how triggers connect to blocks. - -## Your Task - -1. Research what webhook events the service supports — if the service lacks reliable webhooks, use polling -2. Create the trigger files using the generic builder (webhook) or manual config (polling) -3. Create a provider handler (webhook) or polling handler (polling) -4. Register triggers and connect them to the block - -## Hard Rule: No Guessed Webhook Payload Schemas - -If the service docs do not clearly show the webhook payload JSON for an event, you MUST tell the user instead of guessing trigger outputs or `formatInput` mappings. - -- Do NOT invent payload field names -- Do NOT guess nested event object paths -- Do NOT infer output fields from the UI or marketing docs -- Do NOT write `formatInput` against unverified webhook bodies - -If the payload shape is unknown, do one of these instead: -1. Ask the user for sample webhook payloads -2. Ask the user for a test webhook source so you can inspect a real event -3. Implement only the event registration/setup portions whose payloads are documented -4. Leave the trigger unimplemented and explicitly say which payload fields are unknown - -## Directory Structure - -``` -apps/sim/triggers/{service}/ -├── index.ts # Barrel exports -├── utils.ts # Service-specific helpers (options, instructions, extra fields, outputs) -├── {event_a}.ts # Primary trigger (includes dropdown) -├── {event_b}.ts # Secondary trigger (no dropdown) -└── webhook.ts # Generic webhook trigger (optional, for "all events") - -apps/sim/lib/webhooks/ -├── provider-subscription-utils.ts # Shared subscription helpers (getProviderConfig, getNotificationUrl) -├── providers/ -│ ├── {service}.ts # Provider handler (auth, formatInput, matchEvent, subscriptions) -│ ├── types.ts # WebhookProviderHandler interface -│ ├── utils.ts # Shared helpers (createHmacVerifier, verifyTokenAuth, skipByEventTypes) -│ └── registry.ts # Handler map + default handler -``` - -## Step 1: Create `utils.ts` - -This file contains all service-specific helpers used by triggers. - -```typescript -import type { SubBlockConfig } from '@/blocks/types' -import type { TriggerOutput } from '@/triggers/types' - -export const {service}TriggerOptions = [ - { label: 'Event A', id: '{service}_event_a' }, - { label: 'Event B', id: '{service}_event_b' }, -] - -export function {service}SetupInstructions(eventType: string): string { - const instructions = [ - 'Copy the Webhook URL above', - 'Go to {Service} Settings > Webhooks', - `Select the ${eventType} event type`, - 'Paste the webhook URL and save', - 'Click "Save" above to activate your trigger', - ] - return instructions - .map((instruction, index) => - `
${index + 1}. ${instruction}
` - ) - .join('') -} - -export function build{Service}ExtraFields(triggerId: string): SubBlockConfig[] { - return [ - { - id: 'projectId', - title: 'Project ID (Optional)', - type: 'short-input', - placeholder: 'Leave empty for all projects', - mode: 'trigger', - condition: { field: 'selectedTriggerId', value: triggerId }, - }, - ] -} - -export function build{Service}Outputs(): Record { - return { - eventType: { type: 'string', description: 'The type of event' }, - resourceId: { type: 'string', description: 'ID of the affected resource' }, - resource: { - id: { type: 'string', description: 'Resource ID' }, - name: { type: 'string', description: 'Resource name' }, - }, - } -} -``` - -## Step 2: Create Trigger Files - -**Primary trigger** — MUST include `includeDropdown: true`: - -```typescript -import { {Service}Icon } from '@/components/icons' -import { buildTriggerSubBlocks } from '@/triggers' -import { build{Service}ExtraFields, build{Service}Outputs, {service}SetupInstructions, {service}TriggerOptions } from '@/triggers/{service}/utils' -import type { TriggerConfig } from '@/triggers/types' - -export const {service}EventATrigger: TriggerConfig = { - id: '{service}_event_a', - name: '{Service} Event A', - provider: '{service}', - description: 'Trigger workflow when Event A occurs', - version: '1.0.0', - icon: {Service}Icon, - subBlocks: buildTriggerSubBlocks({ - triggerId: '{service}_event_a', - triggerOptions: {service}TriggerOptions, - includeDropdown: true, - setupInstructions: {service}SetupInstructions('Event A'), - extraFields: build{Service}ExtraFields('{service}_event_a'), - }), - outputs: build{Service}Outputs(), - webhook: { method: 'POST', headers: { 'Content-Type': 'application/json' } }, -} -``` - -**Secondary triggers** — NO `includeDropdown` (it's already in the primary): - -```typescript -export const {service}EventBTrigger: TriggerConfig = { - // Same as above but: id: '{service}_event_b', no includeDropdown -} -``` - -## Step 3: Register and Wire - -### `apps/sim/triggers/{service}/index.ts` - -```typescript -export { {service}EventATrigger } from './event_a' -export { {service}EventBTrigger } from './event_b' -``` - -### `apps/sim/triggers/registry.ts` - -```typescript -import { {service}EventATrigger, {service}EventBTrigger } from '@/triggers/{service}' - -export const TRIGGER_REGISTRY: TriggerRegistry = { - // ... existing ... - {service}_event_a: {service}EventATrigger, - {service}_event_b: {service}EventBTrigger, -} -``` - -### Block file (`apps/sim/blocks/blocks/{service}.ts`) - -Wire triggers into the block so the trigger UI appears and `generate-docs.ts` discovers them. Two changes are needed: - -1. **Spread trigger subBlocks** at the end of the block's `subBlocks` array -2. **Add `triggers` property** after `outputs` with `enabled: true` and `available: [...]` - -```typescript -import { getTrigger } from '@/triggers' - -export const {Service}Block: BlockConfig = { - // ... - subBlocks: [ - // Regular tool subBlocks first... - ...getTrigger('{service}_event_a').subBlocks, - ...getTrigger('{service}_event_b').subBlocks, - ], - // ... tools, inputs, outputs ... - triggers: { - enabled: true, - available: ['{service}_event_a', '{service}_event_b'], - }, -} -``` - -**Versioned blocks (V1 + V2):** Many integrations have a hidden V1 block and a visible V2 block. Where you add the trigger wiring depends on how V2 inherits from V1: - -- **V2 uses `...V1Block` spread** (e.g., Google Calendar): Add trigger to V1 — V2 inherits both `subBlocks` and `triggers` automatically. -- **V2 defines its own `subBlocks`** (e.g., Google Sheets): Add trigger to V2 (the visible block). V1 is hidden and doesn't need it. -- **Single block, no V2** (e.g., Google Drive): Add trigger directly. - -`generate-docs.ts` deduplicates by base type (first match wins). If V1 is processed first without triggers, the V2 triggers won't appear in `integrations.json`. Always verify by checking the output after running the script. - -## Provider Handler - -All provider-specific webhook logic lives in a single handler file: `apps/sim/lib/webhooks/providers/{service}.ts`. - -### When to Create a Handler - -| Behavior | Method | Examples | -|---|---|---| -| HMAC signature auth | `verifyAuth` via `createHmacVerifier` | Ashby, Jira, Linear, Typeform | -| Custom token auth | `verifyAuth` via `verifyTokenAuth` | Generic, Google Forms | -| Event filtering | `matchEvent` | GitHub, Jira, Attio, HubSpot | -| Idempotency dedup | `extractIdempotencyId` | Slack, Stripe, Linear, Jira | -| Custom input formatting | `formatInput` | Slack, Teams, Attio, Ashby | -| Auto webhook creation | `createSubscription` | Ashby, Grain, Calendly, Airtable | -| Auto webhook deletion | `deleteSubscription` | Ashby, Grain, Calendly, Airtable | -| Challenge/verification | `handleChallenge` | Slack, WhatsApp, Teams | -| Custom success response | `formatSuccessResponse` | Slack, Twilio Voice, Teams | - -If none apply, you don't need a handler. The default handler provides bearer token auth. - -### Example Handler - -```typescript -import crypto from 'crypto' -import { createLogger } from '@sim/logger' -import { safeCompare } from '@/lib/core/security/encryption' -import type { EventMatchContext, FormatInputContext, FormatInputResult, WebhookProviderHandler } from '@/lib/webhooks/providers/types' -import { createHmacVerifier } from '@/lib/webhooks/providers/utils' - -const logger = createLogger('WebhookProvider:{Service}') - -function validate{Service}Signature(secret: string, signature: string, body: string): boolean { - if (!secret || !signature || !body) return false - const computed = crypto.createHmac('sha256', secret).update(body, 'utf8').digest('hex') - return safeCompare(computed, signature) -} - -export const {service}Handler: WebhookProviderHandler = { - verifyAuth: createHmacVerifier({ - configKey: 'webhookSecret', - headerName: 'X-{Service}-Signature', - validateFn: validate{Service}Signature, - providerLabel: '{Service}', - }), - - async matchEvent({ body, requestId, providerConfig }: EventMatchContext) { - const triggerId = providerConfig.triggerId as string | undefined - if (triggerId && triggerId !== '{service}_webhook') { - const { is{Service}EventMatch } = await import('@/triggers/{service}/utils') - if (!is{Service}EventMatch(triggerId, body as Record)) return false - } - return true - }, - - async formatInput({ body }: FormatInputContext): Promise { - const b = body as Record - return { - input: { - eventType: b.type, - resourceId: (b.data as Record)?.id || '', - resource: b.data, - }, - } - }, - - extractIdempotencyId(body: unknown) { - const obj = body as Record - return obj.id && obj.type ? `${obj.type}:${obj.id}` : null - }, -} -``` - -### Register the Handler - -In `apps/sim/lib/webhooks/providers/registry.ts`: - -```typescript -import { {service}Handler } from '@/lib/webhooks/providers/{service}' - -const PROVIDER_HANDLERS: Record = { - // ... existing (alphabetical) ... - {service}: {service}Handler, -} -``` - -## Output Alignment (Critical) - -There are two sources of truth that **MUST be aligned**: - -1. **Trigger `outputs`** — schema defining what fields SHOULD be available (UI tag dropdown) -2. **`formatInput` on the handler** — implementation that transforms raw payload into actual data - -If they differ: the tag dropdown shows fields that don't exist, or actual data has fields users can't discover. - -**Rules for `formatInput`:** -- Return `{ input: { ... } }` where inner keys match trigger `outputs` exactly -- Return `{ input: ..., skip: { message: '...' } }` to skip execution -- No wrapper objects or duplication -- Use `null` for missing optional data - -## Automatic Webhook Registration - -If the service API supports programmatic webhook creation, implement `createSubscription` and `deleteSubscription` on the handler. The orchestration layer calls these automatically — **no code touches `route.ts`, `provider-subscriptions.ts`, or `deploy.ts`**. - -```typescript -import { getNotificationUrl, getProviderConfig } from '@/lib/webhooks/provider-subscription-utils' -import type { DeleteSubscriptionContext, SubscriptionContext, SubscriptionResult } from '@/lib/webhooks/providers/types' - -export const {service}Handler: WebhookProviderHandler = { - async createSubscription(ctx: SubscriptionContext): Promise { - const config = getProviderConfig(ctx.webhook) - const apiKey = config.apiKey as string - if (!apiKey) throw new Error('{Service} API Key is required.') - - const res = await fetch('https://api.{service}.com/webhooks', { - method: 'POST', - headers: { Authorization: `Bearer ${apiKey}`, 'Content-Type': 'application/json' }, - body: JSON.stringify({ url: getNotificationUrl(ctx.webhook) }), - }) - - if (!res.ok) throw new Error(`{Service} error: ${res.status}`) - const { id } = (await res.json()) as { id: string } - return { providerConfigUpdates: { externalId: id } } - }, - - async deleteSubscription(ctx: DeleteSubscriptionContext): Promise { - const config = getProviderConfig(ctx.webhook) - const { apiKey, externalId } = config as { apiKey?: string; externalId?: string } - if (!apiKey || !externalId) return - await fetch(`https://api.{service}.com/webhooks/${externalId}`, { - method: 'DELETE', - headers: { Authorization: `Bearer ${apiKey}` }, - }).catch(() => {}) - }, -} -``` - -**Key points:** -- Throw from `createSubscription` — orchestration rolls back the DB webhook -- Never throw from `deleteSubscription` — log non-fatally -- Return `{ providerConfigUpdates: { externalId } }` — orchestration merges into `providerConfig` -- Add `apiKey` field to `build{Service}ExtraFields` with `password: true` - -## Trigger Outputs Schema - -Trigger outputs use the same schema as block outputs (NOT tool outputs). - -**Supported:** `type` + `description` for leaf fields, nested objects for complex data. -**NOT supported:** `optional: true`, `items` (those are tool-output-only features). - -```typescript -export function buildOutputs(): Record { - return { - eventType: { type: 'string', description: 'Event type' }, - timestamp: { type: 'string', description: 'When it occurred' }, - payload: { type: 'json', description: 'Full event payload' }, - resource: { - id: { type: 'string', description: 'Resource ID' }, - name: { type: 'string', description: 'Resource name' }, - }, - } -} -``` - -## Polling Triggers - -Use polling when the service lacks reliable webhooks (e.g., Google Sheets, Google Drive, Google Calendar, Gmail, RSS, IMAP). Polling triggers do NOT use `buildTriggerSubBlocks` — they define subBlocks manually. - -### Directory Structure - -``` -apps/sim/triggers/{service}/ -├── index.ts # Barrel export -└── poller.ts # TriggerConfig with polling: true - -apps/sim/lib/webhooks/polling/ -└── {service}.ts # PollingProviderHandler implementation -``` - -### Polling Handler (`apps/sim/lib/webhooks/polling/{service}.ts`) - -```typescript -import { pollingIdempotency } from '@/lib/core/idempotency/service' -import type { PollingProviderHandler, PollWebhookContext } from '@/lib/webhooks/polling/types' -import { markWebhookFailed, markWebhookSuccess, resolveOAuthCredential, updateWebhookProviderConfig } from '@/lib/webhooks/polling/utils' -import { processPolledWebhookEvent } from '@/lib/webhooks/processor' - -export const {service}PollingHandler: PollingProviderHandler = { - provider: '{service}', - label: '{Service}', - - async pollWebhook(ctx: PollWebhookContext): Promise<'success' | 'failure'> { - const { webhookData, workflowData, requestId, logger } = ctx - const webhookId = webhookData.id - - try { - // For OAuth services: - const accessToken = await resolveOAuthCredential(webhookData, '{service}', requestId) - const config = webhookData.providerConfig as unknown as {Service}WebhookConfig - - // First poll: seed state, emit nothing - if (!config.lastCheckedTimestamp) { - await updateWebhookProviderConfig(webhookId, { lastCheckedTimestamp: new Date().toISOString() }, logger) - await markWebhookSuccess(webhookId, logger) - return 'success' - } - - // Fetch changes since last poll, process with idempotency - // ... - - await markWebhookSuccess(webhookId, logger) - return 'success' - } catch (error) { - logger.error(`[${requestId}] Error processing {service} webhook ${webhookId}:`, error) - await markWebhookFailed(webhookId, logger) - return 'failure' - } - }, -} -``` - -**Key patterns:** -- First poll seeds state and emits nothing (avoids flooding with existing data) -- Use `pollingIdempotency.executeWithIdempotency(provider, key, callback)` for dedup -- Use `processPolledWebhookEvent(webhookData, workflowData, payload, requestId)` to fire the workflow -- Use `updateWebhookProviderConfig(webhookId, partialConfig, logger)` for read-merge-write on state -- Use the latest server-side timestamp from API responses (not wall clock) to avoid clock skew - -### Trigger Config (`apps/sim/triggers/{service}/poller.ts`) - -```typescript -import { {Service}Icon } from '@/components/icons' -import type { TriggerConfig } from '@/triggers/types' - -export const {service}PollingTrigger: TriggerConfig = { - id: '{service}_poller', - name: '{Service} Trigger', - provider: '{service}', - description: 'Triggers when ...', - version: '1.0.0', - icon: {Service}Icon, - polling: true, // REQUIRED — routes to polling infrastructure - - subBlocks: [ - { id: 'triggerCredentials', type: 'oauth-input', title: 'Credentials', serviceId: '{service}', requiredScopes: [], required: true, mode: 'trigger' }, - // ... service-specific config fields (dropdowns, inputs, switches) ... - { id: 'triggerInstructions', type: 'text', title: 'Setup Instructions', hideFromPreview: true, mode: 'trigger', defaultValue: '...' }, - ], - - outputs: { - // Must match the payload shape from processPolledWebhookEvent - }, -} -``` - -### Registration (3 places) - -1. **`apps/sim/triggers/constants.ts`** — add provider to `POLLING_PROVIDERS` Set -2. **`apps/sim/lib/webhooks/polling/registry.ts`** — import handler, add to `POLLING_HANDLERS` -3. **`apps/sim/triggers/registry.ts`** — import trigger config, add to `TRIGGER_REGISTRY` - -### Helm Cron Job - -Add to `helm/sim/values.yaml` under the existing polling cron jobs: - -```yaml -{service}WebhookPoll: - schedule: "*/1 * * * *" - concurrencyPolicy: Forbid - url: "http://sim:3000/api/webhooks/poll/{service}" -``` - -### Reference Implementations - -- Simple: `apps/sim/lib/webhooks/polling/rss.ts` + `apps/sim/triggers/rss/poller.ts` -- Complex (OAuth, attachments): `apps/sim/lib/webhooks/polling/gmail.ts` + `apps/sim/triggers/gmail/poller.ts` -- Cursor-based (changes API): `apps/sim/lib/webhooks/polling/google-drive.ts` -- Timestamp-based: `apps/sim/lib/webhooks/polling/google-calendar.ts` - -## Checklist - -### Trigger Definition -- [ ] Created `utils.ts` with options, instructions, extra fields, and output builders -- [ ] Primary trigger has `includeDropdown: true`; secondary triggers do NOT -- [ ] All triggers use `buildTriggerSubBlocks` helper -- [ ] Created `index.ts` barrel export - -### Registration -- [ ] All triggers in `triggers/registry.ts` → `TRIGGER_REGISTRY` -- [ ] Block has `triggers.enabled: true` and lists all trigger IDs in `triggers.available` -- [ ] Block spreads all trigger subBlocks: `...getTrigger('id').subBlocks` - -### Provider Handler (if needed) -- [ ] Handler file at `apps/sim/lib/webhooks/providers/{service}.ts` -- [ ] Registered in `providers/registry.ts` (alphabetical) -- [ ] Signature validator is a private function inside the handler file -- [ ] `formatInput` output keys match trigger `outputs` exactly -- [ ] Event matching uses dynamic `await import()` for trigger utils - -### Auto Registration (if supported) -- [ ] `createSubscription` and `deleteSubscription` on the handler -- [ ] NO changes to `route.ts`, `provider-subscriptions.ts`, or `deploy.ts` -- [ ] API key field uses `password: true` - -### Polling Trigger (if applicable) -- [ ] Handler implements `PollingProviderHandler` at `lib/webhooks/polling/{service}.ts` -- [ ] Trigger config has `polling: true` and defines subBlocks manually (no `buildTriggerSubBlocks`) -- [ ] Provider string matches across: trigger config, handler, `POLLING_PROVIDERS`, polling registry -- [ ] First poll seeds state and emits nothing -- [ ] Added provider to `POLLING_PROVIDERS` in `triggers/constants.ts` -- [ ] Added handler to `POLLING_HANDLERS` in `lib/webhooks/polling/registry.ts` -- [ ] Added cron job to `helm/sim/values.yaml` -- [ ] Payload shape matches trigger `outputs` schema - -### Testing -- [ ] `bun run type-check` passes -- [ ] Manually verify output keys match trigger `outputs` keys -- [ ] Trigger UI shows correctly in the block diff --git a/.claude/commands/babysit.md b/.claude/commands/babysit.md deleted file mode 100644 index 0c75a25a962..00000000000 --- a/.claude/commands/babysit.md +++ /dev/null @@ -1,143 +0,0 @@ ---- -description: Drive a PR to a clean review (Greptile 5/5, zero open threads) — ships if needed, keeps it mergeable against staging, triggers Greptile/Cursor Bugbot, fixes real findings, replies to and resolves every thread, and loops until clean ---- - -# Babysit PRs - -Owns a PR end-to-end through review: ship it, wait for the automatic review round, and if it -isn't already clean, drive fix → reply → resolve → re-review cycles until Greptile reports 5/5 -and there are zero open comment threads, keeping the branch mergeable against staging along the -way. Designed to be run under `/loop` (no fixed interval — let it self-pace on review latency) -so it survives across multiple wakeups in the same session. - -## When to use - -- The user says "babysit this PR", "keep working the reviews until it's clean", or similar -- As the natural follow-up to `/ship` when the user wants the review loop automated rather than - manually re-triggering reviews and answering comments themselves - -## Inputs - -Needs a PR number. If none is given and there's no open PR for the current branch, run `/ship` -first (which includes the `origin/staging` sync check — see `.agents/skills/ship/SKILL.md`) to -create one. - -## Definition of "clean" - -Both must hold: -1. The latest Greptile summary comment reports **Confidence Score: 5/5** -2. `reviewThreads` (GraphQL, see below) has **zero threads with `isResolved: false`** - -Do not stop early on "no new comments this round" alone — a thread can be open from an earlier -round. Always check both conditions freshly after every push. - -## Loop - -1. **Check current state** before doing anything, including whether the PR is still mergeable: - ```bash - gh pr view --json mergeable - gh pr view --json comments -q '[.comments[] | select(.author.login=="greptile-apps")] | last | .body' - gh api graphql -f query=' - query { repository(owner: "", name: "") { pullRequest(number: ) { - reviewThreads(first: 50) { pageInfo { hasNextPage endCursor } nodes { id isResolved path line - comments(first: 5) { nodes { id databaseId author { login } body } } } } } } }' - ``` - `[.comments[]] | last | .body`, not `... | .body | tail -1` — the latter pipes every matching - comment's full multi-line body through the pipeline and keeps only the final *line* of that - combined output (usually the "Reviews (n): Last reviewed commit..." footer), not the last - *comment*, so it silently misses the actual "Confidence Score: X/5" line. - `reviewThreads(first: 50)` is a single page — check `pageInfo.hasNextPage`. If `true`, don't - stop yet: re-run the same query with `after: ""` and keep paging until - `hasNextPage` is `false` before evaluating "clean." A PR with more than 50 threads is rare but - stopping on a partial page would silently miss unresolved ones past the cutoff. - If `mergeable` is `CONFLICTING`, fix that first (step 2). Otherwise, if Greptile is 5/5 and - every thread across all pages has `isResolved: true`, stop — report the outcome (see - "Reporting" below) and skip the rest of this list. - -2. **If the PR has a merge conflict**, merge `origin/staging`, resolve the conflicts, run the - usual pre-push checks, push, and go to step 8 to re-trigger review. - -3. **If no review has run yet** (fresh PR, no Greptile/Cursor comments): they usually run - automatically on PR open — confirm via `gh pr checks ` (look for `Cursor Bugbot` / - `Greptile Review`) and wait for that first round before doing anything else. - -4. **If a review round has landed and it isn't clean**: for every thread where - `isResolved: false`, triage the finding on its own merits — this is the part that requires - judgment, not a mechanical loop: - - **Real bug**: fix it in the cleanest way available. Match the codebase's existing - conventions for that kind of problem before inventing a new one (e.g. an SSRF-prone - user-supplied-host fetch should use whatever `validateUrlWithDNS`/`secureFetchWithPinnedIP` - pattern the rest of the codebase already uses for that exact situation — grep for a sibling - integration solving the same problem first). Never patch around a finding with a - workaround, a broad try/catch, or a suppression comment — fix the actual cause. - - **False positive**: don't change code. Reply with the specific reason it doesn't apply - (cite the type definition, the established pattern it matches, or the doc it follows) so - the reviewer bot and a human skimming later both understand why it was left as-is. - - **Already fixed by an earlier finding in the same round**: note that and resolve without a - duplicate code change. - -5. **Reply to every thread individually** before resolving it — never resolve silently: - ```bash - gh api repos///pulls//comments//replies -f body="" - ``` - Then resolve via GraphQL (needs the thread `id` from step 1, not the comment id): - ```bash - gh api graphql -f query='mutation { resolveReviewThread(input: {threadId: ""}) { thread { isResolved } } }' - ``` - -6. **Before pushing, re-run the full sync check from `/ship` step 2** — not just the log command, - the whole check-and-recover flow (stash WIP if needed, rebase, verify the rebase didn't just - cleanly replay stray commits, cherry-pick rebuild if it did or if it conflicted). A babysit - loop spanning a long session is exactly the scenario where a branch can drift, and pushing - review fixes on top of undetected drift is how an oversized PR happens even after the branch - was fixed once. Then run the repo's pre-ship checks the same way `/ship` does before - committing — not just lint/typecheck/boundary-validation, but also the conditional `/cleanup` - (if this round's fix touched UI code) and `/db-migrate` (if it touched schema/migrations) - gates from `/ship` steps 4 and 5. A review-fix round is still a code change and can trip - either gate just as easily as the original commit did. - -7. **Commit and push** the round's fixes as one commit — `--force-with-lease` whenever step 6's - sync check rewrote history, which includes a plain `git rebase origin/staging` that completed - with no conflicts, not only the cherry-pick rebuild path; both rewrite commits already - published to the remote, so a plain `git push` can be rejected either way — then run `/ship` - step 9's post-push verify — not just before the first push, every push in the loop: - ```bash - git fetch origin staging && git log --oneline --reverse origin/staging..HEAD - gh pr view --json commits -q '.commits[].messageHeadline' - ``` - `--reverse` makes `git log` oldest-first, matching the PR commit list's order — plain - `git log` is newest-first, so without it a positional comparison can spuriously fail on any - multi-commit branch. - These two lists must describe the same commits. A review loop runs many pushes across many - rounds; checking sync only before the push (step 6) and never after is how a bad push or a - PR whose commit history quietly went stale between rounds goes unnoticed. - -8. **Re-trigger review** by posting `@greptile` and `@cursor review` as **two separate PR - comments** — never combine them into one comment, each bot only responds to its own mention: - ```bash - gh pr comment --body "@greptile" - gh pr comment --body "@cursor review" - ``` - -9. **Wait for the new round**, then go back to step 1. Pace the wait with `ScheduleWakeup` using - a fallback delay of ~250–300s (Greptile/Cursor typically take 1–3 minutes) — never busy-poll - in a sleep loop. Pass the same `/loop babysit PR ` prompt on each wakeup so the loop - resumes correctly. - -10. **Stop conditions**: clean state reached (see above), or the same unresolved finding or - merge conflict survives two consecutive rounds with no new information (surface it to the - user instead of looping forever), or the user interrupts. - -## Reporting - -When the loop ends, summarize: how many rounds it took, what was actually fixed (one line each), -what was pushed back on as a false positive and why, and the final Greptile score / thread count. - -## Hard rules - -- Never post the two re-review mentions as a single combined comment. -- Never resolve a thread without replying to it first. -- Never fix a finding with a hacky workaround — if the clean fix isn't obvious, find the sibling - pattern elsewhere in the codebase solving the same class of problem and match it. -- Never silently drop a finding — every thread gets either a code fix or a reasoned reply. -- Always re-run the `/ship`-style sync check before every push in the loop, not just the first. diff --git a/.claude/commands/cleanup.md b/.claude/commands/cleanup.md deleted file mode 100644 index e4f3a2f6f2d..00000000000 --- a/.claude/commands/cleanup.md +++ /dev/null @@ -1,62 +0,0 @@ ---- -description: Run all code quality skills — effects, memo, callbacks, state, React Query, emcn design review, url-state, and comments — analyzing in parallel, then applying fixes sequentially -argument-hint: "[scope] [fix=true|false]" ---- - -# Cleanup - -Arguments: -- scope: what to review (default: your current changes). Examples: "diff to main", "PR #123", "src/components/", "whole codebase" -- fix: whether to apply fixes (default: true). Set to false to only propose changes. - -User arguments: $ARGUMENTS - -## Step 1 — Parallel analysis (read-only) - -First parse the user's `$ARGUMENTS` into `scope` and `fix`: extract the `fix=true|false` token wherever it appears in the string (start, middle, or end), and treat everything else — with that token removed — as `scope`. Defaults: `scope` = your current changes, `fix` = true. The `fix` value is consumed by Step 3 — it does NOT propagate to these passes, which always run `fix=false`. - -Spawn all eight passes concurrently as subagents in a **single message** (multiple Agent tool calls). Each runs its skill on the parsed `scope` with `fix=false` — analysis and proposals ONLY, no edits. Instruct each agent to return its findings as a structured list: for every proposed change, the file path, line range, a one-line description of the change, and the exact before/after so the orchestrator can apply it without re-deriving. - -Run these eight in parallel, substituting the parsed `scope` for `` in each invocation (pass the real scope text, never the literal ``): - -1. `/you-might-not-need-an-effect fix=false` -2. `/you-might-not-need-a-memo fix=false` -3. `/you-might-not-need-a-callback fix=false` -4. `/you-might-not-need-state fix=false` -5. `/react-query-best-practices fix=false` -6. `/emcn-design-review fix=false` -7. `/you-might-not-need-url-state fix=false` -8. `/you-might-not-need-a-comment fix=false` - -## Step 2 — Converge - -Collect all findings into one list, **keeping each proposal tagged with the pass that produced it** — do NOT collapse a file's proposals into a single unlabeled patch, because Step 3 applies in pass order and needs those labels. Detect overlaps where two passes touch the same region (common: a state pass and an effect pass on the same block, or a memo and callback pass on the same component). Reconcile only genuine same-region conflicts, and drop proposals a sibling pass has made moot; a reconciled change inherits the pass label of whichever of its passes comes first in the Step 3 dependency order (effects → state → memo → callback → React Query → url-state → emcn → comments), so it is applied at the earliest safe point. Non-overlapping proposals stay as-is with their own labels. The output is a per-pass list of surviving changes, not a per-file patch. - -## Step 3 — Sequential apply - -If `fix=false`, skip this step — just report the proposals from Step 2. - -Otherwise apply the surviving changes yourself (in the main context, not delegated), iterating **pass by pass** in this dependency order so earlier structural changes settle before later passes build on them: - -1. effects → 2. state → 3. memo → 4. callback → 5. React Query → 6. url-state → 7. emcn design → 8. comments - -For each pass in turn, apply all of that pass's changes, then move to the next pass. A file touched by several passes is therefore edited once per pass, in this order — not once as a merged patch. This is what makes the ordering real: a single merged-per-file patch would collapse all passes into one edit and lose it. - -Comments apply last, on purpose: that pass operates on whatever the earlier structural passes settled the code into, so it never edits lines a sibling pass is about to delete or rewrite. - -**Treat every Step 1 proposal as snapshot-relative, not authoritative.** All passes analyzed the *original* files in parallel, so a proposal's line ranges and before/after text describe the code as it was *before* any edits — once an earlier pass has run, a later pass's snippet may no longer match. So for each change, before applying: - -1. Re-read the file and locate the target by its **content** (the proposal's `old_string` snippet), not by its line number — line numbers from Step 1 are only a hint for where to look, since earlier edits shift them. -2. If the `old_string` still matches verbatim, apply it — a content-anchored edit is safe even if its line moved. -3. If it no longer matches (an earlier pass altered that region), do **not** force the stale patch. Re-derive the change from the current code by re-applying that pass's rule to the construct, or drop it if a prior pass already made it moot. Never apply a proposal against text it wasn't computed from. - -After all edits, run `bun run lint:check` (it runs `turbo run lint:check` across the repo — there is no per-file target, so run the full check). - -## Step 4 — Summary - -Output a summary across all eight passes: what each found, what was applied vs. skipped-as-redundant, and any proposals that need a human decision. - -## Boundary Audit Guidance - -- When removing route-local Zod schemas, replacing raw `fetch(` calls in hooks, or removing `as unknown as X` casts, do not introduce `// boundary-raw-fetch: ` or `// double-cast-allowed: ` annotations to silence the audit. Fix the underlying call instead — adopt a contract from `@/lib/api/contracts/**` and use `requestJson(contract, ...)` from `@/lib/api/client/request`, or refine the type so the double cast is unnecessary. -- Annotations are reserved for legitimate exceptions only: streaming responses, binary downloads, multipart uploads, signed-URL flows, OAuth redirects, external-origin requests, and double casts where no narrower type is available. Each annotation requires a non-empty reason; empty reasons fail `bun run check:api-validation:strict`. diff --git a/.claude/commands/council.md b/.claude/commands/council.md deleted file mode 100644 index 8ea47fe727f..00000000000 --- a/.claude/commands/council.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -description: Spawn parallel task agents to explore a given area of the codebase from multiple angles, then use their findings to answer the question or build a plan. Use when a task needs broad fan-out exploration across many files before acting. -argument-hint: ---- - -Based on the given area of interest, please: - -1. Dig around the codebase in terms of that given area of interest, gather general information such as keywords and architecture overview. -2. Spawn off n=10 (unless specified otherwise) task agents to dig deeper into the codebase in terms of that given area of interest, some of them should be out of the box for variance. -3. Once the task agents are done, use the information to do what the user wants. - -If user is in plan mode, use the information to create the plan. diff --git a/.claude/commands/db-migrate.md b/.claude/commands/db-migrate.md deleted file mode 100644 index 6a53b472c68..00000000000 --- a/.claude/commands/db-migrate.md +++ /dev/null @@ -1,88 +0,0 @@ ---- -description: Author or review a Drizzle DB migration for zero-downtime safety — expand/contract phasing, backward-compatibility with the deployed app version, and writing the `-- migration-safe` acknowledgment the check:migrations lint requires. Use when adding/editing files under `packages/db/migrations/` or changing `packages/db/schema.ts`. ---- - -# DB Migrate Skill - -You make schema changes that survive a deploy without downtime. The `check:migrations` lint (`scripts/check-migrations-safety.ts`) is the deterministic gate; you are the judgment that decides whether a flagged change is actually safe and writes the annotation that satisfies it. - -## The window (why this matters) - -A deploy runs the migration, then rolls out the new app image via blue/green. The two are **not atomic and cannot be** — during cutover the old task set keeps serving against the **already-migrated** schema. So: - -> Every migration must be backward-compatible with the app version that is *already deployed*. - -If a migration drops a column the old code still reads, renames one, or adds a `NOT NULL` the old inserts don't populate, the old code throws until traffic fully shifts — the downtime we're guarding against. You can't fix this by reordering the pipeline; the only fix is discipline. - -## Expand / contract - -Split every breaking change across **two deploys**: - -1. **Expand** (this PR): additive, backward-compatible schema + code that tolerates *both* the old and new shape. -2. **Contract** (a later PR, after expand is fully deployed): remove the old thing, now that nothing reads it. - -Never put expand and contract in the same PR. If this PR both removes the code that used a column *and* drops the column, the old code is still live during cutover — split it. - -### Per-operation playbook - -| You want to | Do (deploy 1 = expand) | Do (deploy 2 = contract) | -|---|---|---| -| Add a required column | `ADD COLUMN` nullable or `DEFAULT`; code writes it | backfill, then `SET NOT NULL` | -| Rename a column/table | add the new name; code dual-writes / reads new-then-old | drop the old name | -| Drop a column/table | stop all reads/writes in code; ship it | `DROP` (annotate) | -| Change a column type | add a new column of the new type; dual-write | backfill, swap reads, drop old | -| Add FK / CHECK | `ADD CONSTRAINT ... NOT VALID` | `VALIDATE CONSTRAINT` separately | -| Index an existing table | `COMMIT;` breakpoint → `SET lock_timeout = 0` → `CREATE INDEX CONCURRENTLY IF NOT EXISTS` (see `packages/db/scripts/migrate.ts`) | — | -| Drop an index | `COMMIT;` breakpoint → `DROP INDEX CONCURRENTLY` — plain `DROP INDEX` takes ACCESS EXCLUSIVE on the table | — | -| Backfill data | batched + idempotent `UPDATE` (keyset/`WHERE`, bounded) | — | - -A `CREATE INDEX`, `ADD COLUMN`, or `ADD CONSTRAINT` against a table **created in the same migration** is always safe (no rows, no live traffic) — the lint already suppresses those. - -## Tracking the contract (don't let it rot) - -The contract half is deferred to a later deploy — and that is exactly when it gets forgotten, leaving dead columns, orphaned tables, and `NOT NULL`s that never land. Every deferred contract must become a durable, greppable TODO. - -When an expand defers a drop, leave a **`contract-pending`** marker on the legacy column/table in `packages/db/schema.ts` — that is the file you will be editing when you finally do the drop, so the reminder lives where the work happens: - -```ts -// contract-pending(after #5035 is fully deployed): drop once permission-check.ts stops reading it -workspaceId: text('workspace_id'), -``` - -Format: `contract-pending(): — `. The precondition names the PR/release that removes the last reader and **must be fully deployed** before the contract ships. - -- **The TODO list is a grep** — always accurate, never drifts: `grep -rn "contract-pending" packages/db apps/sim`. Run it when starting migration work to see what is owed. -- For anything with a real owner or schedule, also open a tracking issue and put its number in the marker. -- **Close the loop in the contract PR:** the contract migration's `-- migration-safe:` annotation references the expand, and you **delete the `contract-pending` marker** in the same PR: - ```sql - -- migration-safe: contract of #5035 — workspace_id readers removed there, deployed 2026-06-10 - ALTER TABLE "permission_group" DROP COLUMN "workspace_id"; - ``` -- An expand merged **without** a marker for the drop it defers, or a contract merged **without** removing its marker, is a bug — flag it in review. - -## The judgment the lint can't do - -The lint flags risky *shapes*; it cannot know whether a given drop is *safe right now*. For each flagged statement, do the work it can't: - -1. **Is the dependency gone?** Grep the app for the table/column: search `apps/sim` and `packages` for the column name, the Drizzle field (camelCase), and the table object. If any live read/write remains, it is **not** safe — fix the code first. -2. **Did the expand already ship?** The removal of that read/write must be in a deploy that is *already out*, not this same PR. If it's in this PR, split: land the code change now, do the destructive migration in a follow-up after it deploys. -3. **Backfills:** confirm the `UPDATE`/`DELETE` is batched (bounded `WHERE`/keyset, not a single whole-table statement), idempotent (safe to replay — a failed migration re-runs unjournaled files from the top), and safe under concurrent writes from the still-live old app. - -## Workflow - -1. Edit `packages/db/schema.ts`, then `cd packages/db && bunx drizzle-kit generate` to produce the SQL. If this is an expand that defers a drop, leave a `contract-pending` marker on the legacy column (see "Tracking the contract"). If this is the contract, delete the marker it resolves. -2. Hand-edit the generated SQL where the playbook requires it: `CONCURRENTLY` + `COMMIT;` breakpoint for indexes on existing tables, `NOT VALID` for constraints, batching for backfills. -3. Run `bun run check:migrations` (base defaults to `origin/staging`). - - **Hard errors** (`add-not-null-no-default`, `rename`, `index-not-concurrent`, `constraint-not-valid`, …): rewrite into expand/contract. Do **not** try to annotate them away — the lint won't accept it. - - **Annotate tier** (`drop-table`, `drop-column`, `drop-default`, `set-not-null`, `alter-type`, `drop-index`): only after you've confirmed steps 1–3 above, add a comment on the line directly above the statement: - ```sql - -- migration-safe: `secret` read removed in v0.6.1 (#1234), shipped two deploys ago - ALTER TABLE "webhook" DROP COLUMN "secret"; - ``` - The reason must be specific and name the PR/version that removed the dependency. An empty reason fails the lint. - - **Warnings** (`data-backfill`): non-blocking, but confirm the batching/idempotency before merging. -4. Verify locally: `cd packages/db && bun run db:migrate` against a dev DB. - -## Hard rule - -Never annotate a destructive statement just to make the lint pass. The annotation is a claim that you verified the old code no longer depends on it. If you can't make that claim truthfully, the change belongs in a later deploy — tell the user to split it. diff --git a/.claude/commands/design-taste-frontend.md b/.claude/commands/design-taste-frontend.md deleted file mode 100644 index 2f58011ea16..00000000000 --- a/.claude/commands/design-taste-frontend.md +++ /dev/null @@ -1,1205 +0,0 @@ ---- -description: Anti-slop frontend skill for landing pages, portfolios, and redesigns. The agent reads the brief, infers the right design direction, and ships interfaces that do not look templated. Real design systems when applicable, audit-first on redesigns, strict pre-flight check. ---- - -# tasteskill: Anti-Slop Frontend Skill - -> Landing pages, portfolios, and redesigns. Not dashboards, not data tables, not multi-step product UI. -> Every rule below is **contextual**. None of it fires automatically. First read the brief, then pull only what fits. - ---- - -## 0. BRIEF INFERENCE (Read the Room Before Anything Else) - -Before touching code or tweaking dials, **infer what the user actually wants**. Most LLM design output is bad because the model jumps to a default aesthetic instead of reading the room. - -### 0.A Read these signals first -1. **Page kind** - landing (SaaS / consumer / agency / event), portfolio (dev / designer / creative studio), redesign (preserve vs overhaul), editorial / blog. -2. **Vibe words** the user used - "minimalist", "calm", "Linear-style", "Awwwards", "brutalist", "premium consumer", "Apple-y", "playful", "serious B2B", "editorial", "agency-y", "glassy", "dark tech". -3. **Reference signals** - URLs they linked, screenshots they pasted, products they named, brands they're competing with. -4. **Audience** - B2B procurement panel vs. design-conscious consumer vs. recruiter scanning a portfolio. The audience picks the aesthetic, not your taste. -5. **Brand assets that already exist** - logo, color, type, photography. For redesigns, these are starting material, not optional input (see Section 11). -6. **Quiet constraints** - accessibility-first audiences, public-sector, regulated industries, trust-first commerce, kids' products. These constraints OVERRIDE aesthetic preference. - -### 0.B Output a one-line "Design Read" before generating -Before any code, state in one line: **"Reading this as: \ for \, with a \ language, leaning toward \."** - -Example reads: -- *"Reading this as: B2B SaaS landing for technical buyers, with a Linear-style minimalist language, leaning toward Tailwind utilities + Geist + restrained motion."* -- *"Reading this as: solo designer portfolio for hiring managers, with an editorial / kinetic-type language, leaning toward native CSS + scroll-driven animation + custom typography."* -- *"Reading this as: redesign of a public-sector service site, with a trust-first language, leaning toward GOV.UK Frontend or USWDS."* - -### 0.C If the brief is ambiguous, ask one question, do not guess -Ask exactly **one** clarifying question - never a multi-question dump - and only when the design read genuinely diverges. Example: *"Should this feel closer to Linear-clean or Awwwards-experimental?"* - -If you can confidently infer from context, **do not ask**. Just declare the design read and proceed. - -### 0.D Anti-Default Discipline -Do not default to: AI-purple gradients, centered hero over dark mesh, three equal feature cards, generic glassmorphism on everything, infinite-loop micro-animations everywhere, Inter + slate-900. These are the LLM defaults. Reach past them deliberately based on the design read. - ---- - -## 1. THE THREE DIALS (Core Configuration) - -After the design read, set three dials. Every layout, motion, and density decision below is gated by these. - -* **`DESIGN_VARIANCE: 8`** - 1 = Perfect Symmetry, 10 = Artsy Chaos -* **`MOTION_INTENSITY: 6`** - 1 = Static, 10 = Cinematic / Physics -* **`VISUAL_DENSITY: 4`** - 1 = Art Gallery / Airy, 10 = Cockpit / Packed Data - -**Baseline:** `8 / 6 / 4`. Use these unless the design read overrides them. Do not ask the user to edit this file - overrides happen conversationally. - -### 1.A Dial Inference (design read → dial values) -| Signal | VARIANCE | MOTION | DENSITY | -|---|---|---|---| -| "minimalist / clean / calm / editorial / Linear-style" | 5-6 | 3-4 | 2-3 | -| "premium consumer / Apple-y / luxury / brand" | 7-8 | 5-7 | 3-4 | -| "playful / wild / Dribbble / Awwwards / experimental / agency" | 9-10 | 8-10 | 3-4 | -| "landing page / portfolio / marketing site (default)" | 7-9 | 6-8 | 3-5 | -| "trust-first / public-sector / regulated / accessibility-critical" | 3-4 | 2-3 | 4-5 | -| "redesign - preserve" | match existing | +1 | match existing | -| "redesign - overhaul" | +2 | +2 | match existing | - -### 1.B Use-Case Presets -| Use case | VARIANCE | MOTION | DENSITY | -|---|---|---|---| -| Landing (SaaS, mainstream) | 7 | 6 | 4 | -| Landing (Agency / creative) | 9 | 8 | 3 | -| Landing (Premium consumer) | 7 | 6 | 3 | -| Portfolio (Designer / studio) | 8 | 7 | 3 | -| Portfolio (Developer) | 6 | 5 | 4 | -| Editorial / Blog | 6 | 4 | 3 | -| Public-sector service | 3 | 2 | 5 | -| Redesign - preserve | match | match+1 | match | -| Redesign - overhaul | +2 | +2 | match | - -### 1.C How the Dials Drive Output -Use these (or user-overridden values) as global variables. Cross-references throughout this document refer to these exact variable names - never invent aliases like `LAYOUT_VARIANCE` or `ANIM_LEVEL`. - ---- - -## 2. BRIEF → DESIGN SYSTEM MAP - -Once you have the design read (Section 0) and dials (Section 1), pick the right foundation. Do not invent CSS for things that have an official package. Do not pretend an aesthetic trend is an official system. - -### 2.A When to reach for a real design system (use official packages) -| Brief reads as… | Reach for | Why | -|---|---|---| -| Microsoft / enterprise SaaS / dashboards | `@fluentui/react-components` or `@fluentui/web-components` | Official Fluent UI, Microsoft tokens, accessibility done | -| Google-ish UI, Material-flavored product | `@material/web` + Material 3 tokens | Official, theme-able via Material Theming | -| IBM-style B2B / enterprise analytics | `@carbon/react` + `@carbon/styles` | Official Carbon, mature data-density patterns | -| Shopify app surfaces | `polaris.js` web components / Polaris React | Required for Shopify admin UI | -| Atlassian / Jira-style product | `@atlaskit/*` + `@atlaskit/tokens` | Official Atlassian DS | -| GitHub-style devtool / community page | `@primer/css` or `@primer/react-brand` | Official Primer; Brand variant for marketing | -| Public-sector UK service | `govuk-frontend` | Legally / regulatorily expected | -| US public-sector / trust-first | `uswds` | Same | -| Fast local-business / agency MVP | Bootstrap 5.3 | Boring, fast, works | -| Modern accessible React foundation | `@radix-ui/themes` | Primitives + polished theme | -| Modern SaaS where you own the components | shadcn/ui (`npx shadcn@latest add ...`) | You own the code, easy to customise; never ship default state | -| Tailwind-based modern SaaS / AI marketing | Tailwind v4 utilities + `dark:` variant | Default for indie + small team builds | - -**Honesty rule:** if the brief reads as one of the systems above, install and use the **official** package. Do not recreate its CSS by hand. Do not import a system's tokens but then override 90% of them. - -**One system per project.** Do not mix Fluent React with Carbon in the same tree. Do not import shadcn/ui components into a Material 3 app. - -### 2.B When the brief is an aesthetic, not a system -For these directions, there is **no single official package**. Build with native CSS + Tailwind + a maintained component library. Be honest in code comments about what is borrowed inspiration vs. official material. - -| Aesthetic | Honest implementation | -|---|---| -| Glassmorphism / "frosted glass" | `backdrop-filter`, layered borders, highlight overlays. Provide solid-fill fallback for `prefers-reduced-transparency`. | -| Bento (Apple-style tile grids) | CSS Grid with mixed cell sizes. No single library owns this. | -| Brutalism | Native CSS, monospace, raw borders. No library. | -| Editorial / magazine | Serif type, asymmetric grid, generous whitespace. No library. | -| Dark tech / hacker | Mono + accent neon, terminal motifs. No library. | -| Aurora / mesh gradients | SVG or layered radial gradients. No library. | -| Kinetic typography | Native CSS animations, scroll-driven animations, GSAP for hijacks. No library. | -| **Apple Liquid Glass** | Apple documents this for Apple platforms only. **There is no official `liquid-glass.css`.** Web implementations are approximations using `backdrop-filter` + layered borders + highlights. Label clearly as approximation. | - ---- - -## 3. DEFAULT ARCHITECTURE & CONVENTIONS - -Unless the design read picks a real design system (Section 2.A), these are the defaults: - -### 3.A Stack -* **Framework:** React or Next.js. Default to Server Components (RSC). - * **RSC SAFETY:** Global state works ONLY in Client Components. In Next.js, wrap providers in a `"use client"` component. - * **INTERACTIVITY ISOLATION:** Any component using Motion, scroll listeners, or pointer physics MUST be an isolated leaf with `'use client'` at the top. Server Components render static layouts only. -* **Styling:** **Tailwind v4** (default). Tailwind v3 only if the existing project demands it. - * For v4: do NOT use `tailwindcss` plugin in `postcss.config.js`. Use `@tailwindcss/postcss` or the Vite plugin. -* **Animation:** **Motion** (the library formerly known as Framer Motion). Import from `motion/react` (`import { motion } from "motion/react"`). The `framer-motion` package still works as a legacy alias - prefer `motion/react` in new code. -* **Fonts:** Always use `next/font` (Next.js) or self-host with `@font-face` + `font-display: swap`. Never link Google Fonts via `` in production. - -### 3.B State -* Local `useState` / `useReducer` for isolated UI. -* Global state ONLY for deep prop-drilling avoidance - Zustand, Jotai, or React context. -* **NEVER** use `useState` to track continuous values driven by user input (mouse position, scroll progress, pointer physics, magnetic hover). Use Motion's `useMotionValue` / `useTransform` / `useScroll`. `useState` re-renders the React tree on every change and collapses on mobile. - -### 3.C Icons -* **Allowed libraries (priority order):** `@phosphor-icons/react`, `hugeicons-react`, `@radix-ui/react-icons`, `@tabler/icons-react`. -* **Discouraged:** `lucide-react`. Acceptable only when the user explicitly asks for it or the project already depends on it. -* **NEVER hand-roll SVG icons.** If a glyph is missing, install a second library or compose from primitives - do not draw icon paths from scratch. -* **One family per project.** Do not mix Phosphor with Lucide in the same component tree. -* **Standardize `strokeWidth` globally** (e.g. `1.5` or `2.0`). - -### 3.D Emoji Policy -Discouraged by default in code, markup, and visible text. Replace symbols with icon-library glyphs. **Override:** allow emojis only when the user explicitly asks for a playful / chat-style / social-native vibe - and even then use them sparingly with intent. - -### 3.E Responsiveness & Layout Mechanics -* Standardize breakpoints (`sm 640`, `md 768`, `lg 1024`, `xl 1280`, `2xl 1536`). -* Contain page layouts using `max-w-[1400px] mx-auto` or `max-w-7xl`. -* **Viewport Stability:** NEVER use `h-screen` for full-height Hero sections. ALWAYS use `min-h-[100dvh]` to prevent layout jumping on mobile (iOS Safari address bar). -* **Grid over Flex-Math:** NEVER use complex flexbox percentage math (`w-[calc(33%-1rem)]`). ALWAYS use CSS Grid (`grid grid-cols-1 md:grid-cols-3 gap-6`). - -### 3.F Dependency Verification (mandatory) -Before importing ANY 3rd-party library, check `package.json`. If the package is missing, output the install command first. **Never** assume a library exists. - ---- - -## 4. DESIGN ENGINEERING DIRECTIVES (Bias Correction) - -LLMs default to clichés. Override these defaults proactively. Each rule has a context-aware override path. - -### 4.1 Typography -* **Display / Headlines:** Default `text-4xl md:text-6xl tracking-tighter leading-none`. -* **Body / Paragraphs:** Default `text-base text-gray-600 leading-relaxed max-w-[65ch]`. -* **Sans font choice:** - * **Discouraged as default:** `Inter`. Pick `Geist`, `Outfit`, `Cabinet Grotesk`, `Satoshi`, or a brand-appropriate serif first. - * **Override:** Inter is acceptable when the user explicitly asks for a neutral / standard / Linear-style feel, or when the brief is a public-sector / accessibility-first site. -* **Pairings to know:** `Geist` + `Geist Mono`, `Satoshi` + `JetBrains Mono`, `Cabinet Grotesk` + `Inter Tight`, `GT America` + `IBM Plex Mono`. - -* **SERIF DISCIPLINE (VERY DISCOURAGED AS DEFAULT):** - * Serif is **very discouraged as the default font for any project.** "It feels creative / premium / editorial" is NOT a reason to reach for serif. The agent's default mental model that "creative brief = serif" is the single most-tested AI tell in production rounds. - * **Serif is only acceptable when ONE of these is explicitly true:** - - The brand brief literally names a serif font, OR - - The aesthetic family is genuinely editorial / luxury / publication / manuscript / heritage / vintage AND you can articulate why this specific serif fits this specific brand - * For everything else (creative agency, design studio, modern brand, premium consumer, portfolio, lifestyle), **default sans-serif display** (Geist Display, ABC Diatype, Söhne Breit, Cabinet Grotesk Display, Migra Sans, GT Walsheim, Inter Display, PP Neue Montreal). Sans display fonts are not "boring" — they are the default for the same reason black is the default in fashion. - * **EMPHASIS RULE (related):** When you want to emphasize a word within a headline (the kinetic "and `spatial` design" type move), use **italic or bold of the SAME font**. Do NOT inject a random serif word into a sans headline (or vice versa) just to add visual interest. Mixed-family emphasis is amateur. Italic/bold emphasis in the same family is the right move. - * **Specifically BANNED as defaults:** `Fraunces` and `Instrument_Serif` (the two LLM-favorite display serifs). - * **If a serif is justified** (rare, per the above), rotate from this pool, do NOT reuse the same serif across consecutive projects: PP Editorial New, GT Sectra Display, Cardinal Grotesque, Reckless Neue, Tiempos Headline, Recoleta, Cormorant Garamond, Playfair Display, EB Garamond, IvyPresto, Migra, Editorial Old, Saol Display, Söhne Breit Kursiv, Domaine Display, Canela, Schnyder, Tobias, NB Architekt, ITC Galliard. - -* **ITALIC DESCENDER CLEARANCE (mandatory):** When italic is used in display type and the word contains a descender letter (`y g j p q`), `leading-[1]` or `leading-none` will clip the descender. Use `leading-[1.1]` minimum and add `pb-1` or `mb-1` reserve on the wrapping element. Audit every italic word in display headlines before shipping. - -### 4.2 Color Calibration -* Max 1 accent color. Saturation < 80% by default. -* **THE LILA RULE:** The "AI Purple / Blue glow" aesthetic is discouraged as a default. No automatic purple button glows, no random neon gradients. Use neutral bases (Zinc / Slate / Stone) with high-contrast singular accents (Emerald, Electric Blue, Deep Rose, Burnt Orange, etc.). -* **Override:** if the brand or brief explicitly asks for purple / violet / lila, embrace it. But execute with intent: consistent palette, harmonised neutrals, restrained gradients. Not generic AI gradient slop. -* **One palette per project.** Do not fluctuate between warm and cool grays within the same project. -* **COLOR CONSISTENCY LOCK (mandatory):** Once an accent color is chosen for a page, it is used on the WHOLE page. A warm-grey site does not suddenly get a blue CTA in section 7. A rose-accented site does not get a teal status badge in the footer. Pick one accent, lock it, audit every component before shipping. - -* **PREMIUM-CONSUMER PALETTE BAN (mandatory, second-most-recurring AI-tell):** - * For premium-consumer briefs (cookware, wellness, artisan, luxury, heritage craft, DTC home goods, etc.) the LLM default is **warm beige/cream + brass/clay/oxblood/ochre + espresso/ink dark text**. Concretely banned hex families as default backgrounds and accents: - - Backgrounds: `#f5f1ea`, `#f7f5f1`, `#fbf8f1`, `#efeae0`, `#ece6db`, `#faf7f1`, `#e8dfcb` (all "warm paper / cream / chalk / bone") - - Accents: `#b08947`, `#b6553a`, `#9a2436`, `#9c6e2a`, `#bc7c3a`, `#7d5621` (all "brass / clay / oxblood / ochre") - - Text: `#1a1714`, `#1a1814`, `#1b1814` (all "espresso / warm near-black") - * This palette is BANNED as the default reach for premium-consumer briefs. Every premium-consumer site you have ever shipped uses this exact palette. The brand becomes invisible. - * **Default alternatives (rotate, do not reuse):** - - **Cold Luxury:** silver-grey + chrome + smoke (think Tesla, Apple Watch Hermes-without-the-leather) - - **Forest:** deep green + bone + amber accent (think Filson, Patagonia premium) - - **Black and Tan:** true off-black + warm tan, sharp contrast, no beige - - **Cobalt + Cream:** saturated blue against a single neutral, no brass - - **Terracotta + Slate:** warm rust against cool grey, no brass - - **Olive + Brick + Paper:** muted olive plus brick-red accent - - **Pure monochrome + single saturated pop:** off-white + off-black + one bright accent (electric blue, emerald, hot pink, etc.) - * **Palette-rotation rule:** if the previous premium-consumer project you generated used the beige+brass family, this one MUST use a different family. Do not ship the same warm-craft palette twice in a row. - * **Override:** the beige+brass+espresso palette is acceptable ONLY when the brand brief explicitly names those colors, or when the brand identity is genuinely vintage / artisan / warm-craft AND you can articulate why this specific palette fits this specific brand. Default-reaching for it because "this is a cookware brief" is banned. - -### 4.3 Layout Diversification -* **ANTI-CENTER BIAS:** Centered Hero / H1 sections are avoided when `DESIGN_VARIANCE > 4`. Force "Split Screen" (50/50), "Left-aligned content / right-aligned asset", "Asymmetric white-space", or scroll-pinned structures. -* **Override:** centered hero is OK for editorial / manifesto / launch-announcement briefs where the message itself is the design. - -### 4.4 Materiality, Shadows, Cards -* Use cards ONLY when elevation communicates real hierarchy. Otherwise group with `border-t`, `divide-y`, or negative space. -* When a shadow is used, tint it to the background hue. No pure-black drop shadows on light backgrounds. -* For `VISUAL_DENSITY > 7`: generic card containers are banned. Data metrics breathe in plain layout. -* **SHAPE CONSISTENCY LOCK (mandatory):** Pick ONE corner-radius scale for the page and stick to it. Options: all-sharp (radius 0), all-soft (radius 12-16px), all-pill (full radius for interactive). Mixed systems are allowed only when there is a documented rule (e.g. "buttons are full-pill, cards are 16px, inputs are 8px") and that rule is followed everywhere. Round buttons in a square layout, or square cards on a pill-button page, is broken design. - -### 4.5 Interactive UI States -LLMs default to "static successful state only." Always implement full cycles: -* **Loading:** Skeletal loaders matching the final layout's shape. Avoid generic circular spinners. -* **Empty States:** Beautifully composed; indicate how to populate. -* **Error States:** Clear, inline (forms), or contextual (toasts only for transient). -* **Tactile Feedback:** On `:active`, use `-translate-y-[1px]` or `scale-[0.98]` to simulate a physical push. -* **BUTTON CONTRAST CHECK (mandatory, a11y):** Before shipping any button, verify the button text is readable against the button background. White button + white text, `bg-white` CTA with `text-white` label, transparent button against the page background with no border → all banned. Audit every CTA: contrast ratio WCAG AA min (4.5:1 for body, 3:1 for large text 18px+). Same rule applies to ghost buttons over photographic backgrounds (use a backdrop, scrim, or stroke). -* **CTA BUTTON WRAP BAN (mandatory):** Button text MUST fit on one line at desktop. If a label like "VIEW SELECTED WORK" wraps to 2 or 3 lines, the button is broken. Fix by EITHER shortening the label (3 words max for primary CTAs, ideally 1-2) OR widening the button (do not artificially constrain `max-width` on CTAs). Wrapped CTAs at desktop are a Pre-Flight Fail. -* **NO DUPLICATE CTA INTENT (mandatory):** Two CTAs with the same intent on one page is a Pre-Flight Fail. Examples of same intent: "Get in touch" + "Contact us" + "Let's talk" + "Start a project" + "Start something" + "Reach out" = all "contact" intent → pick ONE label and use it everywhere on the page (nav, hero, footer). Same for "Try free" + "Get started" + "Sign up free" (all "signup" intent) and "View work" + "See selected work" + "Browse projects" (all "portfolio" intent). One label per intent. -* **FORM CONTRAST CHECK (mandatory, a11y):** Form inputs, placeholder text, focus rings, helper text, and error text all pass WCAG AA contrast against the section background. Light placeholders on a near-white form, white form on white page section, form labels grayer than 4.5:1 contrast → all banned. Audit every form before shipping. - -### 4.6 Data & Form Patterns -* Label ABOVE input. Helper text optional but present in markup. Error text BELOW input. Standard `gap-2` for input blocks. -* No placeholder-as-label. Ever. - -### 4.7 Layout Discipline (Hard Rules. Failing any of these is shipping broken work) - -* **Hero MUST fit in the initial viewport.** Headline max 2 lines on desktop, subtext max **20 words** AND max 3-4 lines, CTAs visible without scroll. If the copy is too long: reduce font scale OR cut copy. If you cannot describe the value-prop in 20 words of subtext, the value-prop is unclear, not the rule too tight. Never let the hero overflow and force scroll to find the CTA. -* **Hero font-scale discipline.** Plan font size and image size *together*. If the hero asset is large and the headline is more than 6 words, do not start at `text-7xl/text-8xl`. Default sensible range: `text-4xl md:text-5xl lg:text-6xl` for most heroes; `text-6xl md:text-7xl` only when the headline is 3-5 words. A 4-line hero headline is always a font-size error, never a copy-length error. -* **HERO TOP PADDING CAP (mandatory):** Hero top padding max `pt-24` (≈6rem) at desktop. More than that means the hero content floats halfway down the viewport and reads as a layout bug, not as intentional space. If your hero needs more breathing room, increase font scale or asset size, not top padding. -* **HERO STACK DISCIPLINE (max 4 text elements).** The hero is a single moment, not a feature list. Allowed text elements, max 4 in total: - 1. Eyebrow (small uppercase label) OR brand strip OR neither - pick zero or one - 2. Headline (max 2 lines, see above) - 3. Subtext (max 20 words, max 4 lines) - 4. CTAs (1 primary + max 1 secondary) - - **BANNED in the hero:** tiny tagline below CTAs ("Works with GitHub, GitLab, and self-hosted Git"), trust micro-strip ("Used by engineering teams at..."), pricing teaser ("Free for solo, $10/user for teams"), feature bullet list, social-proof avatar row. All of those move to dedicated sections directly below the hero. - - If you have an eyebrow AND a tagline below CTAs in the same hero, drop the tagline. If you have a brand strip AND a tagline, drop the tagline. One small text element per hero, max. -* **"Used by" / "Trusted by" logo wall belongs UNDER the hero, never inside it.** The hero is for the value prop and primary CTA. The logo wall is a separate section directly below. Do not stuff trust logos into the same flex row as the hero copy. -* **Navigation MUST render on a single line on desktop.** If items don't fit at `lg` (1024px), condense labels, drop secondary items, or move to a hamburger. A two-line nav at desktop is broken design. -* **Navigation height cap: 80px max desktop, default 64-72px.** No huge "agency" nav bars that eat 15% of the viewport. -* **Bento grids MUST have rhythm, not one-sided repetition.** Do not stack 6 left-image / right-text rows. Vary the composition: alternate full-width feature rows, asymmetric tile sizes, vertical breaks. -* **BENTO CELL COUNT RULE (mandatory):** A bento grid has EXACTLY as many cells as you have content for. 3 items → 3 cells (1+2 split, or 2+1, or asymmetric trio). 5 items → 5 cells (2+3, 3+2, hero+4, etc.). If your grid has an empty cell in the middle or at the end, you planned wrong. Re-shape the grid; do not paste a blank tile. -* **Section-Layout-Repetition Ban.** Once you use a layout family for a section (e.g., 3-column-image-cards, full-width-quote, split-text-image), that family can appear at most ONCE on the page. "Selected commissions" must not look like "What we do." A landing page with 8 sections must use at least 4 different layout families. -* **ZIGZAG ALTERNATION CAP (mandatory).** Alternating "left-image + right-text" then "left-text + right-image" zigzag layout = banal. Max 2 sections in a row with this image+text-split pattern. The 3rd consecutive image+text split is a Pre-Flight Fail. Break the pattern with a full-width section, a vertical-stack section, a bento grid, a marquee, or a different layout family. -* **EYEBROW RESTRAINT (mandatory, the #1 violated rule in production tests).** An "eyebrow" is the small uppercase wide-tracking label sitting above a section headline (e.g. `FOUR COLORWAYS`, `SELECTED WORK`, `THE HARDWARE`, `Git-native task management`). Typical CSS signature: `text-[11px] uppercase tracking-[0.18em]`, `font-mono text-[10.5px] uppercase tracking-[0.22em]`. Every AI-built site puts an eyebrow above EVERY section header, producing the same templated rhythm. Hard rule: - - **Maximum 1 eyebrow per 3 sections.** Hero counts as 1. So a page with 9 sections may use at most 3 eyebrows total. - - If section A has an eyebrow, the next 2 sections cannot have one. - - **Pre-Flight Check is mechanical:** count instances of `uppercase tracking` (or similar small-caps mono labels above headlines) across all section components. If count > ceil(sectionCount / 3), the output fails. - - **What to do instead of an eyebrow:** drop it entirely. The headline alone is enough. If you need to categorize a section, the section's location on the page already categorizes it; no label needed. -* **SPLIT-HEADER BAN (mandatory).** The pattern "left big headline + right small explainer paragraph" as a section header (left col-span-7/8, right col-span-4/5 with a small body paragraph floating in the right column) is **banned as default**. Sections should have ONE focused message. If you genuinely need both a headline and an explainer paragraph, stack them vertically (headline on top, body below, max-width 65ch). Reach for the split-header pattern only when there is a real compositional reason (e.g., the right column carries a visual or interactive element, not just filler text). -* **Bento Background Diversity (mandatory).** Bento and feature-grid sections cannot be 6 white-on-white cards with text inside. At least 2-3 cells in any multi-cell grid need real visual variation: a real image, a brand-appropriate gradient (not AI-purple), a pattern, a tinted background. A cream-on-cream bento with only typography inside reads as boring AI default, even when the rest of the page is good. -* **Mobile collapse must be explicit per section.** For every multi-column layout, declare the `< 768px` fallback in the same component. No "it'll work, Tailwind handles it" assumptions. - -### 4.8 Image & Visual Asset Strategy - -Landing pages and portfolios are **visual products**. Text-only pages with fake-screenshot divs are slop. - -**Priority order for visual assets:** -1. **Image-generation tool first.** If ANY image-gen tool is available in the environment (`generate_image`, MCP image tool, IDE-integrated gen, OpenAI image tools, etc.) you MUST use it to create section-specific assets: hero photography, product shots, texture backgrounds, mood images. Generate at the right aspect ratio for the section. Do not skip this step because hand-rolled CSS feels faster. -2. **Real web images second.** When no gen tool is available, use real photography sources. Acceptable defaults: - * `https://picsum.photos/seed/{descriptive-seed}/{w}/{h}` for placeholder photography (seed should describe the section, e.g. `marrow-cookware-kitchen`) - * Actual stock or brand URLs when the brief provides them - * Open-license sources (Unsplash via direct URL, Pexels) if explicitly allowed -3. **Last resort: tell the user.** If neither is possible, do NOT fill the page with hand-rolled SVG illustrations or div-based "fake screenshots." Instead, leave clearly-labeled placeholder slots (``) and at the end of the response say: *"This page needs real images at: \[list of placements\]. Please generate or provide them."* - -**Even minimalist sites need real images.** A pure-text page is not minimalism. It is incomplete work. Even an editorial Linear-style site needs at least 2-3 real images (hero, one product/lifestyle shot, one supporting image). Generate B&W minimalist photography if the brief is restrained; do not skip images entirely because the dial is low. - -**Real company logos for social proof.** When the brief calls for a "Trusted by / Used by / Customers" logo wall, do NOT default to plain text wordmarks (`Acme Co` styled in a row). Use real SVG logos: -* **Source: Simple Icons** (`https://cdn.simpleicons.org/{slug}/ffffff` for any color, or `simple-icons` npm package). Covers most known brands. -* **Alternative: devicon** for tech-stack logos (`@svgr/cli` or CDN). -* **Make-up the brand name? Then make-up an SVG mark too.** Generate a simple monogram (one letter in a circle, two-letter ligature, abstract glyph) rendered as an inline `` matching the page style. Plain text wordmarks for invented brand names look generic. -* **Always** ensure logos render in both light and dark mode (white-on-dark, black-on-light, or single-color theme variable). -* **LOGO-ONLY rule (mandatory):** logo wall = logos and nothing else. Do NOT print industry / category labels below each logo (no `Vercel` + `hosting` underneath, no `Stripe` + `payments`, no `Cloudflare` + `infra`). The logo is the credibility, the label adds nothing the user does not already know. Optional: brand name as alt-text for screen readers, optional link to the brand's site. That is it. - -**Hand-rolled illustrations:** -* SVG icons from libraries: fine (see Section 3.C). -* Hand-rolled decorative SVGs (custom illustrations, logos, marks): **strongly discouraged**, never as default. Acceptable only when: - - The brief explicitly calls for it ("draw me an SVG logo") - - It's a single, simple geometric mark (a square, a circle, a wordmark in display type) - - You're confident in the output quality - -**Div-based fake screenshots are banned.** A "hand-built product preview" rendered with `
` rectangles, fake task lists, fake dashboards, fake terminal windows is a Tell. If you need to show a product: -* Use a real screenshot URL if one exists -* Generate one via image tool -* Use a real component preview (an actual mini-version of the UI inside the page) -* Or skip the preview entirely and use editorial photography - -**Hero needs a real visual.** Text + gradient blob is not a hero - it's a placeholder. - -### 4.9 Content Density - -Landing pages live on the **first impression**, not the full read. Cut ruthlessly. - -* **Default content shape per section:** short headline (≤ 8 words) + short sub-paragraph (≤ 25 words) + one visual asset OR one CTA. Anything more must be justified by the section's job. -* **No data-dump sections.** A 20-row publication table, a 30-row award list, a giant pricing matrix on a marketing page = wrong layout. Use: - - Top 3-5 highlights + "View full list" link - - Marquee / carousel for breadth - - Different page entirely if the data is the product -* **Long lists need a different UI component, not a longer list.** Default `
    ` with bullets / `divide-y` rows is the lazy choice. If you have > 5 items, reach for one of these instead: - - 2-column split with grouped items - - Card grid with image + label per item - - Tabs / accordion if items are categorisable - - Horizontal scroll-snap pills - - Carousel for breadth-heavy lists (testimonials, logos, capabilities) - - Marquee for "lots-of-things-that-don't-need-individual-attention" - A spec sheet with 10 rows + a hairline under every row is the WORST default. Either group rows into 2-3 chunks with sparse dividers, or move to a card-per-spec layout. -* **Spec sheets specifically (the Marrow-cookware pattern).** A long product specification table with `border-b` on every row is the AI default for cookware / hardware / apparel / artisan-goods briefs. Banned. Concrete alternatives: - - **2-col card grid:** each spec gets its own card with the spec name, the value (large display number), and a one-line "why it matters" body. Cards arranged 2-col on desktop, 1-col mobile. - - **Scroll-snap horizontal pills:** each spec is a pill, user can flick through. - - **Grouped chunks:** group 10 specs into 3 logical clusters (e.g. "Materials", "Cooking", "Warranty"), each cluster gets ONE soft divider and a cluster heading. - - **Featured-vs-rest:** 3-4 hero specs visualised as large display tiles, the rest collapsed under a "View full specifications" disclosure. - -* **COPY SELF-AUDIT (mandatory before ship):** Before declaring any task done, re-read every visible string on the page (headlines, subheads, eyebrows, button labels, body copy, captions, alt text, footer text, error messages). Flag any string that is: - - **Grammatically broken** ("free on its past", "two plans but one is honest", "to put it on the table" out of context) - - **Has unclear referents** ("we plan to stay that way" without prior context) - - **Sounds like AI hallucination** (cute-but-wrong wordplay, forced metaphors that don't track, "elegant nothing" phrases) - - **Reads like an LLM trying to sound thoughtful** (passive-aggressive humility, fake-craftsman labels, mock-poetic micro-meta) - Rewrite every flagged string. If unsure whether a string makes sense, replace it with a plain functional sentence. AI-generated cute copy is worse than boring copy. -* **Fake-precise numbers are flagged.** Numbers like `92%`, `4.1×`, `48k`, `5.8 mm`, `13.4 lb` either: - - Come from real data (brief, brand guidelines, public metrics) - fine - - Are explicitly labeled as mock (``, "example", "sample data") - fine - - Are AI-invented spec aesthetics - banned. Don't fake engineering precision the brand doesn't claim. -* **One copy register per page.** Don't mix technical mono ("47 tasks · 0.6 ctx-switches/day"), editorial prose, and marketing punch in the same composition unless the brand voice explicitly calls for it. - -### 4.10 Quotes & Testimonials - -* **Max 3 lines** of quote body. Never 6. If the original quote is longer → cut it. A landing-page quote is a snippet, not the full review. -* For very small font sizes (e.g. footer-style testimonials), the line cap can stretch slightly. Spirit: "fits in a glance." -* **No em-dashes inside the quote text** as design flourish (long pauses, kinetic em-dashes, em-dash-bullets). See Section 9.G - em-dash is completely banned. -* Attribution: name + role + (optionally) company. Never name only ("- Sarah"). -* Quote marks: use real typographic quotes ( " " ) or none at all. Not straight ASCII ( " ). - -### 4.11 Page Theme Lock (Light / Dark Mode Consistency) - -The page has ONE theme. Sections do not invert. - -* If the page is dark mode, ALL sections are dark mode. No light-mode-warm-paper section sandwiched between dark sections (or vice versa). The user must not feel they walked into a different website mid-scroll. -* The exception: if the brief explicitly calls for a "Color Block Story" or "Theme Switch on Scroll" device AND that is a deliberate composition (one full theme switch with a strong transition, not random alternation), it is allowed once per page. -* Default behaviour: pick light, dark, or auto (`prefers-color-scheme`) at the page level and lock it. Section-level background tints within the same theme family are fine (`bg-zinc-950` next to `bg-zinc-900`); flipping to `bg-amber-50` in the middle of a `bg-zinc-950` page is broken. -* When using a design system with built-in theming (Radix Themes, shadcn/ui with ``), set the theme ONCE in `layout.tsx` or the page root. Do not let individual sections override. - ---- - -## 5. CONTEXT-AWARE PROACTIVITY - -These are tools, not defaults. Use them when the design read calls for them. **None of these fire automatically.** - -* **Liquid Glass / Glassmorphism:** Appropriate for premium consumer, Apple-adjacent, luxury brand, or media-overlay vibes. Inappropriate for dashboards, public-sector, or "boring B2B." When used, go beyond `backdrop-blur`: add a 1px inner border (`border-white/10`) and a subtle inner shadow (`shadow-[inset_0_1px_0_rgba(255,255,255,0.1)]`) for physical edge refraction. Provide a solid-fill fallback under `prefers-reduced-transparency`. -* **Magnetic Micro-physics:** Use when `MOTION_INTENSITY > 5` AND the brief reads premium / playful / agency. Implement EXCLUSIVELY with Motion's `useMotionValue` / `useTransform` outside the React render cycle. Never `useState`. See Section 3.B. -* **Perpetual Micro-Interactions** (Pulse, Typewriter, Float, Shimmer, Carousel): Use when `MOTION_INTENSITY > 5` AND the section actively benefits from motion (status indicators, live feeds, AI-feel). **Not every card needs an infinite loop.** If a section is informational, leave it still. Apply Spring Physics (`type: "spring", stiffness: 100, damping: 20`) - no linear easing. -* **"Motion claimed, motion shown."** If `MOTION_INTENSITY > 4`, the page must actually move: entry transitions on hero, scroll-reveal on key sections, hover physics on CTAs, at minimum. A static page that claims `MOTION_INTENSITY: 7` is broken. Conversely, if you cannot ship working motion in the available scope, drop the dial to 3 and ship a clean static page. Never half-build motion that breaks (cut-off ScrollTriggers, jumpy enters, missing cleanups). -* **MOTION MUST BE MOTIVATED (mandatory).** Before adding any animation, ask: "what does this animation communicate?" Valid answers: hierarchy (drawing attention to the right thing), storytelling (revealing content in sequence that matches a narrative), feedback (acknowledging a user action), state transition (showing something changed). Invalid answer: "it looked cool". GSAP everywhere because GSAP is available is amateur. Each ScrollTrigger, each marquee, each pinned section needs a reason. If you cannot articulate the reason in one sentence, drop the animation. -* **MARQUEE MAX-ONE-PER-PAGE (mandatory).** Horizontal scrolling text marquees ("logos endlessly scrolling", "manifesto scrolling sideways", "kinetic word strip") are appropriate at most ONCE per page. Two or more marquees on the same page reads as lazy filler. Pick the one section where the marquee actually serves the content; the others get a different layout. -* **GSAP Sticky-Stack Pattern (when scroll-stack is used).** A "card stack on scroll" must be a REAL sticky-stack, not a sequential reveal list. See Section 5.A below for the canonical code skeleton. Common failure: trigger fires halfway through scroll instead of pinning at viewport top. Fix: `start: "top top"` not `start: "top center"` or `"top 80%"`. -* **GSAP Horizontal-Pan Pattern (when horizontal scroll-hijack is used).** See Section 5.B below for the canonical skeleton. Common failure: animation starts before the section is pinned, so the user sees half a slide. Same fix: `start: "top top"`, pin the wrapper, scrub the inner track. - -### 5.A Sticky-Stack - Canonical Skeleton - -```tsx -"use client"; -import { useRef, useEffect } from "react"; -import { gsap } from "gsap"; -import { ScrollTrigger } from "gsap/ScrollTrigger"; -import { useReducedMotion } from "motion/react"; - -gsap.registerPlugin(ScrollTrigger); - -export function StickyStack({ cards }: { cards: React.ReactNode[] }) { - const ref = useRef(null); - const reduce = useReducedMotion(); - - useEffect(() => { - if (reduce || !ref.current) return; - const ctx = gsap.context(() => { - const cardEls = gsap.utils.toArray(".stack-card"); - cardEls.forEach((card, i) => { - if (i === cardEls.length - 1) return; - ScrollTrigger.create({ - trigger: card, - start: "top top", // pin at viewport top - endTrigger: cardEls[cardEls.length - 1], - end: "top top", - pin: true, - pinSpacing: false, - }); - gsap.to(card, { - scale: 0.92, - opacity: 0.55, - ease: "none", - scrollTrigger: { - trigger: cardEls[i + 1], - start: "top bottom", - end: "top top", - scrub: true, - }, - }); - }); - }, ref); - return () => ctx.revert(); - }, [reduce]); - - return ( -
    - {cards.map((card, i) => ( -
    - {card} -
    - ))} -
    - ); -} -``` - -Critical points: `start: "top top"`, `pin: true`, every card except the last is pinned, the scale/opacity transform is driven by the NEXT card's scroll trigger (so previous card shrinks as next one arrives). - -### 5.B Horizontal-Pan - Canonical Skeleton - -```tsx -"use client"; -import { useRef, useEffect } from "react"; -import { gsap } from "gsap"; -import { ScrollTrigger } from "gsap/ScrollTrigger"; -import { useReducedMotion } from "motion/react"; - -gsap.registerPlugin(ScrollTrigger); - -export function HorizontalPan({ children }: { children: React.ReactNode }) { - const wrap = useRef(null); - const track = useRef(null); - const reduce = useReducedMotion(); - - useEffect(() => { - if (reduce || !wrap.current || !track.current) return; - const ctx = gsap.context(() => { - const distance = track.current!.scrollWidth - window.innerWidth; - gsap.to(track.current, { - x: -distance, - ease: "none", - scrollTrigger: { - trigger: wrap.current, - start: "top top", // pin starts when section top hits viewport top - end: () => `+=${distance}`, // scroll distance = track width minus viewport - pin: true, - scrub: 1, - invalidateOnRefresh: true, - }, - }); - }, wrap); - return () => ctx.revert(); - }, [reduce]); - - return ( -
    -
    - {children} -
    -
    - ); -} -``` - -Critical points: `start: "top top"`, `pin: true`, `end: "+=${distance}"` (scroll length = horizontal travel needed), `scrub: 1`. The wrapper is pinned, the inner track slides horizontally as the user scrolls vertically. - -### 5.C Scroll-Reveal Stagger - Canonical Skeleton (lighter alternative) - -For simple "items appear as they enter viewport" (no pinning), prefer Motion's `whileInView` over GSAP - lighter, no ScrollTrigger needed: - -```tsx -"use client"; -import { motion, useReducedMotion } from "motion/react"; - -export function RevealStagger({ items }: { items: string[] }) { - const reduce = useReducedMotion(); - return ( -
      - {items.map((item, i) => ( - - {item} - - ))} -
    - ); -} -``` - -Use this for: feature lists, testimonial grids, logo walls, anything that just needs "enter on scroll." Save GSAP for actual pin/scrub work. - -### 5.D Forbidden Animation Patterns - -* **`window.addEventListener("scroll", ...)`** is banned. It runs on every scroll frame, jank-prone, no batching. Use Motion's `useScroll()`, GSAP's `ScrollTrigger`, IntersectionObserver, or CSS `scroll-driven animations` (`animation-timeline: view()`). -* **Custom scroll progress calculations using `window.scrollY`** in React state. Same reason. Re-renders on every frame. -* **`requestAnimationFrame` loops that touch React state.** Use motion values (`useMotionValue` + `useTransform`) instead. -* **Layout Transitions:** Use Motion's `layout` and `layoutId` props for visible state changes (re-ordering lists, expanding modals, shared elements between routes). Do not wrap static content in `layout` props "for safety" - it costs measurement work. -* **Staggered Orchestration:** Use `staggerChildren` (Motion) or CSS cascade (`animation-delay: calc(var(--index) * 100ms)`) for reveal moments where sequence matters. For `staggerChildren`, parent (`variants`) and children MUST share the same Client Component tree. - ---- - -## 6. PERFORMANCE & ACCESSIBILITY GUARDRAILS - -### 6.A Hardware Acceleration -* Animate ONLY `transform` and `opacity`. Never animate `top`, `left`, `width`, `height`. -* Use `will-change: transform` sparingly - only on elements that will actually animate. - -### 6.B Reduced Motion (mandatory) -* **Any motion above `MOTION_INTENSITY > 3` MUST honor `prefers-reduced-motion`.** This is non-negotiable. -* In Motion: wrap with `useReducedMotion()` and degrade to static. -* In CSS: gate animations behind `@media (prefers-reduced-motion: no-preference)` or provide an override block under `@media (prefers-reduced-motion: reduce)` that disables. -* Infinite loops, parallax, scroll-hijack, and magnetic physics MUST collapse to static / instant under reduced motion. - -### 6.C Dark Mode (mandatory for any consumer-facing page) -* Design for **both modes from the start**. Never ship light-only or dark-only without explicit user instruction. -* Use Tailwind `dark:` variant OR CSS variables for tokens. Pick one strategy per project. -* **Do not prescribe specific dark-mode colors here.** The brief decides. Maintain visual hierarchy, brand identity, and WCAG AA contrast (AAA for body) across both modes. -* Respect `prefers-color-scheme: dark`. Default to system preference unless the brand insists on one mode. - -### 6.D Core Web Vitals Targets -* **LCP** < 2.5s. Hero image must be `next/image priority` or preloaded. -* **INP** < 200ms. Heavy work off main thread. -* **CLS** < 0.1. Reserve space for images, fonts, embeds. -* Run Lighthouse before declaring a page done. - -### 6.E DOM Cost -* Apply grain / noise filters EXCLUSIVELY to fixed, `pointer-events-none` pseudo-elements (e.g., `fixed inset-0 z-[60] pointer-events-none`). NEVER on scrolling containers - continuous GPU repaints destroy mobile FPS. -* Be aware of bundle size. Motion is not tiny. Three.js is large. Lazy-load anything that's not above-the-fold. - -### 6.F Z-Index Restraint -NEVER spam arbitrary `z-50` or `z-10`. Use z-index strictly for systemic layer contexts (sticky navbars, modals, overlays, grain). Document the z-index scale in a project constants file. - ---- - -## 7. DIAL DEFINITIONS (Technical Reference) - -### DESIGN_VARIANCE (Level 1-10) -* **1-3 (Predictable):** Symmetrical CSS Grid (12-col, equal fr-units), equal paddings, centered alignment. -* **4-7 (Offset):** `margin-top: -2rem` overlaps, varied image aspect ratios (4:3 next to 16:9), left-aligned headers over center-aligned data. -* **8-10 (Asymmetric):** Masonry layouts, CSS Grid with fractional units (`grid-template-columns: 2fr 1fr 1fr`), massive empty zones (`padding-left: 20vw`). -* **MOBILE OVERRIDE:** For levels 4-10, asymmetric layouts above `md:` MUST collapse to strict single-column (`w-full`, `px-4`, `py-8`) on viewports `< 768px`. - -### MOTION_INTENSITY (Level 1-10) -* **1-3 (Static):** No automatic animations. CSS `:hover` and `:active` states only. `prefers-reduced-motion` is the default mode anyway. -* **4-7 (Fluid CSS):** `transition: all 0.3s cubic-bezier(0.16, 1, 0.3, 1)`. `animation-delay` cascades for load-ins. Focus on `transform` and `opacity`. -* **8-10 (Advanced Choreography):** Complex scroll-triggered reveals, parallax, scroll-driven animation (CSS `animation-timeline` or GSAP ScrollTrigger). Use Motion hooks. **NEVER use `window.addEventListener('scroll')`** - it is a hard ban, not a "prefer-not." See Section 5.D for the allowed alternatives. - -### VISUAL_DENSITY (Level 1-10) -* **1-3 (Art Gallery):** Lots of white space. Huge section gaps (`py-32` to `py-48`). Expensive, clean. -* **4-7 (Daily App):** Standard web app spacing (`py-16` to `py-24`). -* **8-10 (Cockpit):** Tight paddings. No card boxes; 1px lines separate data. Mandatory: `font-mono` for all numbers. - ---- - -## 8. DARK MODE PROTOCOL - -Dual-mode by default. Never assume light-only unless the brief is print-emulating editorial. - -### 8.A Token Strategy (pick one, stick to it) -* **Tailwind `dark:` variant** (default for utility-first projects): every color utility paired with its dark variant (`bg-white dark:bg-zinc-950`, `text-gray-900 dark:text-gray-100`). -* **CSS variables** (for shadcn/ui, Radix Themes, or component libraries with theming): define semantic tokens (`--surface`, `--surface-elevated`, `--text-primary`, `--accent`) and swap values under `[data-theme="dark"]` or `@media (prefers-color-scheme: dark)`. - -### 8.B Do Not Prescribe Specific Colors Here -The brief and brand decide. This skill enforces only: -* **Contrast** - WCAG AA minimum for body text, AAA target for hero copy. -* **Hierarchy parity** - visual hierarchy that works in light must work in dark. If a CTA pops in light, it pops in dark. -* **Brand fidelity** - primary brand color stays recognisable. Don't desaturate the brand into a dark mode. -* **No pure `#000000` and no pure `#ffffff`** - use off-black (zinc-950, near-black warm gray) and off-white. Pure values kill depth. - -### 8.C Default Mode -Respect `prefers-color-scheme` unless the brand insists. Add a manual toggle if either mode would lose key brand expression. - -### 8.D Test in Both Modes Before Finishing -Open the page in both modes during development. Do not ship a page you've only seen in one mode. - ---- - -## 9. AI TELLS (Forbidden Patterns) - -Avoid these signatures unless the brief explicitly asks for them. - -### 9.A Visual & CSS -* **NO neon / outer glows** by default. Use inner borders or subtle tinted shadows. -* **NO pure black (`#000000`).** Off-black, zinc-950, or charcoal. -* **NO oversaturated accents.** Desaturate to blend with neutrals. -* **NO excessive gradient text** for large headers. -* **NO custom mouse cursors.** Outdated, accessibility-hostile, perf-hostile. - -### 9.B Typography -* **AVOID Inter as default.** See Section 4.1. Override path exists. -* **NO oversized H1s** that just scream. Control hierarchy with weight + color, not raw scale. -* **Serif constraints:** Serif for editorial / luxury / publication. Not for dashboards. - -### 9.C Layout & Spacing -* **Mathematically perfect** padding and margins. No floating elements with awkward gaps. -* **NO 3-column equal feature cards.** The generic "three identical cards horizontally" feature row is banned. Use 2-column zig-zag, asymmetric grid, scroll-pinned, or horizontal-scroll alternative. - -### 9.D Content & Data ("Jane Doe" Effect) -* **NO generic names.** "John Doe", "Sarah Chan", "Jack Su" → use creative, realistic, locale-appropriate names. -* **NO generic avatars.** No SVG "egg" or Lucide user icons → use believable photo placeholders or specific styling. -* **NO fake-perfect numbers.** Avoid `99.99%`, `50%`, `1234567`. Use organic, messy data (`47.2%`, `+1 (312) 847-1928`). -* **NO startup-slop brand names.** "Acme", "Nexus", "SmartFlow", "Cloudly" → invent contextual, premium names that sound real. -* **NO filler verbs.** "Elevate", "Seamless", "Unleash", "Next-Gen", "Revolutionize" → concrete verbs only. - -### 9.E External Resources & Components -* **NO hand-rolled SVG icons.** Use Phosphor / HugeIcons / Radix / Tabler. Lucide on explicit request only. -* **Hand-rolled decorative SVGs strongly discouraged** as default (see Section 4.8). -* **NO div-based fake screenshots.** Never build a fake product UI out of `
    ` rectangles to simulate a screenshot. Use real images, generated images, or skip the preview. -* **NO broken Unsplash links.** Use `https://picsum.photos/seed/{descriptive-string}/{w}/{h}`, or generated photo placeholders, or actual assets. -* **shadcn/ui customization:** Allowed, but NEVER in default state. Customize radii, colors, shadows, typography to the project aesthetic. -* **Production-Ready Cleanliness:** Code visually clean, memorable, meticulously refined. - -### 9.F Production-Test Tells (banned outright) - -These patterns came out of real LLM-generated landing-page tests. They are the signatures the model defaults to when it tries to "look designed." Treat them as hard bans unless the brief explicitly calls for one. - -**Hero & top-of-page** -* **NO version labels in the hero.** `V0.6`, `v2.0`, `BETA`, `INVITE-ONLY PREVIEW`, `EARLY ACCESS`, `ALPHA` - banned as default eyebrows. Only acceptable when the brief is explicitly about a product launch / preview status. -* **NO "Brand · No. 01"-style sub-eyebrows.** "Marrow · No. 01 · The 6-quart" type micro-meta lines. Skip them. - -**Section numbering & micro-labels** -* **NO section-number eyebrows.** `00 / INDEX`, `001 · Capabilities`, `002 · Featured commission`, `06 · how it works`, `05 · The honest table` - banned. Eyebrows should name the topic in plain language, not enumerate. -* **NO `01 / 4`-style pagination on images or bento tiles.** If the user can count, they don't need the label. -* **NO `Scroll · 001 Capabilities`-style scroll cues.** A simple arrow or "Scroll" is enough; no section-number prefix. -* **NO "Index of Work, 2018 - 2026"-style range labels** as eyebrows. Just say what the section is. - -**Separators & dots** -* **The middle-dot (`·`) is rationed.** Maximum 1 per line in metadata strips. Do NOT use it as the default separator for everything ("foo · bar · baz · qux · quux"). If you need a separator family, prefer line breaks, hairlines, or columns. -* **NO decorative colored status dots on every list/nav/badge.** A colored dot before "ONE Q4 SLOT OPEN" or before every nav link, or every task row - banned by default. Acceptable only when the dot conveys actual semantic state (a server status, an availability flag) and is used sparingly. - -**Em-dashes & typography flourishes** -* **NO em-dash (`—`) as a design element OR anywhere else.** See Section 9.G below for the complete, non-negotiable ban. The em-dash character is forbidden in headlines, eyebrows, pills, body copy, quotes, attribution, captions, button text, and alt text. Use the regular hyphen (`-`). -* **NO `
    `-broken-and-italicized headlines** as a default "design move." "for thirty\*years.*" type splits. Headlines should read naturally first, get clever only when the brief demands it. -* **NO vertical rotated text** ("INDEX OF WORK, 2018 - 2026" rotated 90°). Agency-portfolio cliché. Use it only when the brief is explicitly agency / Awwwards / experimental AND it serves a real composition purpose. -* **NO crosshair / hairline grid lines as decoration.** Vertical and horizontal lines drawn just to make the page "feel designed" - banned. Use them only when they organize real content. - -**Fake product previews** -* **NO div-based fake product UI in the hero** (fake task list, fake terminal, fake dashboard built from styled divs). It is the #1 LLM-design Tell. Use a real screenshot, a generated image, a real component preview, or none at all. -* **NO fake version footers** ("v0.6.2-rc.1", "last sync 4s ago · main") inside fake screenshots. Adds nothing, screams AI. - -**Marketing-copy Tells** -* **NO "Quietly in use at" / "Quietly trusted by"** social-proof headers. Use natural language: "Trusted by", "Used at", "Customers include", or skip the heading entirely if the logos speak. -* **NO "From the field" / "Field notes" / "Currently on the bench" / "On our desks" / "Loose plates" style poetic labels** on quote, blog, or sidebar sections. Reads as performative-craftsman. Use plain functional labels ("Testimonials", "Latest writing", "Now working on") or skip the label. -* **NO "We respect the French ones"-style** mock-humble industry-references in body copy. Cute and AI-y. -* **NO weather / locale strips** ("LIS 14:23 · 18°C") in headers/footers unless the brief is explicitly about a place / time-zone-distributed studio. -* **NO micro-meta-sentences under eyebrows.** Sentences like *"Each of these is a feature we ship today, not a roadmap promise. The list will stay short on purpose."* sitting under a section heading are clutter. Eyebrow + Headline + Body is enough. -* **NO generic step labels.** "Stage 1 / Stage 2 / Stage 3", "Step 1 / Step 2 / Step 3", "Phase 01 / Phase 02 / Phase 03", "Pass One / Pass Two / Pass Three". Banned. The actual step content is the label. If you must show progression, use the verb-noun directly ("Install", "Configure", "Ship") not "Stage 1: Install". - -**Pills, labels and version stamps** -* **NO pills/labels/tags overlaid on images.** No `` overlays on photos with tags like `Brand · 02`, `PLATE · BRAND`, `Field notes - journal`. Either let the image speak alone, or add a caption directly below (outside the image). -* **NO photo-credit captions as decoration.** Strings like `Field study no. 12 · Ines Caetano`, `Plate 03 · House archive`, `Frame XII · 35mm` under stock/picsum images are pretentious. Photo credit is allowed ONLY when there is a real photographer being credited for a real photo (with permission). Otherwise: skip the caption or use a one-line functional caption ("The 6-quart, in Sage."). -* **NO version footers on marketing pages.** Footer strings like `v1.4.2`, `Build 0048`, `last sync 4s ago · main` are CLI / devtool fixtures, not landing-page content. Banned on marketing/landing/portfolio pages. -* **NO "Reservation 412 of 800"-style live-stock counters** as decoration. Only if the brief is explicitly a limited-run waitlist with real data. - -**Decoration text strips** -* **NO decoration text strip at hero bottom.** Patterns like `BRAND. MOTION. SPATIAL.`, `TYPE / FORM / MOTION`, `DESIGN · BUILD · SHIP`, `ESTD. 2018 · LISBON · BRAND. MOTION. SPATIAL.` as a small mono-caps strip across the bottom of the hero are an agency-portfolio cliché. Banned by default. Only acceptable when the strip carries real, navigable links (sticky bottom nav) or real status info (cookie banner, build info on a docs site). -* **NO floating top-right sub-text in section headings.** Pattern: section has a giant left-aligned headline; in the top-right corner of the same section header there is a small explainer paragraph floating with no clear alignment to anything else. That floater is the Tell. Either put the sub-text directly under the headline, or build a clean 2-column header (left: headline, right: aligned body), but not a tiny corner paragraph. - -**Lists, dividers and scoring** -* **NO `border-t` + `border-b` on every row of a long list / spec table.** Pick one (bottom-border between rows OR top-border above the group) and use it sparsely. A 10-row spec table with hairlines under each row is the laziest layout - see Section 4.9 for alternative UI components. -* **NO scoring/progress bars with filled background tracks** as comparison visuals. If you need to show "X out of Y" comparisons, prefer a number + small icon, or a tiny inline bar WITHOUT a background track. Big filled `bg-zinc-200` tracks with a partial fill on top are dashboard-UI clutter on a landing page. - -**Locale, time, scroll cues** -* **Locale / city-name / time / weather strips are banned for 99% of briefs.** "Lisbon, working with founders" in the hero, "1200-690 Lisbon, Portugal" in the footer, "Lisbon 14:23 · 18°C" in the nav. These are agency-portfolio decoration tells. Allowed ONLY when: the brief explicitly describes a globally-distributed studio with timezone-relevant work, OR a travel-focused brand, OR a real-world physical venue. A single contact-address mention in the footer is fine; an atmospheric locale strip is not. -* **Scroll cues are banned.** `Scroll`, `↓ scroll`, `Scroll to explore`, `Scroll to walk through it`, animated mouse-wheel icons. If the user has not scrolled yet, they are looking at the hero. They know what scroll is. The bottom of the viewport does not need a label. -* **ZERO decorative status dots by default.** A coloured dot before nav items, before list rows, before badges, before status labels is a Tell. Only acceptable when conveying real semantic state (a live indicator on actual server status, a live availability flag) and limited to one per page section. - -### 9.G EM-DASH BAN (the single most-violated Tell) - -**Em-dash (`—`) is COMPLETELY banned.** It is the LLM's signature stylistic crutch and it is the #1 visual Tell in production tests. There is no "limited use" allowance, no "natural language frequency" allowance, no "in body copy is fine" allowance. None. - -* **Banned in headlines.** Use a period or a comma. -* **Banned in eyebrows / labels / pills / button text / image captions / nav items.** Replace with line breaks, columns, or hairlines. -* **Banned in body copy.** Restructure the sentence: two sentences with a period, OR a comma, OR parentheses, OR a colon. -* **Banned in quote attribution.** Use a normal hyphen with spaces (` - `) or a line break + smaller-weight name. -* **Banned in en-dash form too (`–`) when used as a separator.** Date ranges (`2018-2026`) use a hyphen. Number ranges (`€40-80k`) use a hyphen. - -The ONLY permitted dash characters on the page are: -* Regular hyphen `-` (for compound words, ranges, line dividers in markup) -* Minus sign in math (`-5°C`) - -If your output contains a single `—` or `–` anywhere visible to the user, the output fails the Pre-Flight Check and must be rewritten. - -This rule is non-negotiable. The agent has historically ignored em-dash limits when phrased as "use sparingly." The phrasing here is binary: zero em-dashes. - ---- - -## 10. REFERENCE VOCABULARY (Pattern Names the Agent Should Know) - -This is a vocabulary, not a library. The agent should KNOW these pattern names to communicate about them, design with them in mind, and reach for them when the design read calls for them. **Implementations and code sketches live in the Block Library (Section 12), which is populated iteratively.** - -### Hero Paradigms -* **Asymmetric Split Hero** - Text on one side, asset on the other, generous white space. -* **Editorial Manifesto Hero** - Large type, no asset, almost-poster. -* **Video / Media Mask Hero** - Type cut out as mask over video background. -* **Kinetic-Type Hero** - Animated typography as the primary visual. -* **Curtain-Reveal Hero** - Hero parts on scroll like a curtain. -* **Scroll-Pinned Hero** - Hero stays pinned while content scrolls behind. - -### Navigation & Menus -* **Mac OS Dock Magnification** - Edge nav, icons scale fluidly on hover. -* **Magnetic Button** - Pulls toward cursor. -* **Gooey Menu** - Sub-items detach like viscous liquid. -* **Dynamic Island** - Morphing pill for status / alerts. -* **Contextual Radial Menu** - Circular menu expanding at click point. -* **Floating Speed Dial** - FAB springing into curved secondary actions. -* **Mega Menu Reveal** - Full-screen dropdown, stagger-fade content. - -### Layout & Grids -* **Bento Grid** - Asymmetric tile grouping (Apple Control Center). -* **Masonry Layout** - Staggered grid, no fixed row height. -* **Chroma Grid** - Borders / tiles with subtle animating gradients. -* **Split-Screen Scroll** - Two halves sliding in opposite directions. -* **Sticky-Stack Sections** - Sections that pin and stack on scroll. - -### Cards & Containers -* **Parallax Tilt Card** - 3D tilt tracking mouse coordinates. -* **Spotlight Border Card** - Borders illuminate under cursor. -* **Glassmorphism Panel** - Frosted glass with inner refraction. -* **Holographic Foil Card** - Iridescent rainbow shift on hover. -* **Tinder Swipe Stack** - Physical card stack, swipe-away. -* **Morphing Modal** - Button expands into its own dialog. - -### Scroll Animations -* **Sticky Scroll Stack** - Cards stick and physically stack. -* **Horizontal Scroll Hijack** - Vertical scroll → horizontal pan. -* **Locomotive / Sequence Scroll** - Video / 3D sequence tied to scrollbar. -* **Zoom Parallax** - Central background image zooming on scroll. -* **Scroll Progress Path** - SVG line drawing along scroll. -* **Liquid Swipe Transition** - Page transition like viscous liquid. - -### Galleries & Media -* **Dome Gallery** - 3D panoramic gallery. -* **Coverflow Carousel** - 3D carousel with angled edges. -* **Drag-to-Pan Grid** - Boundless draggable canvas. -* **Accordion Image Slider** - Narrow strips expanding on hover. -* **Hover Image Trail** - Mouse leaves popping image trail. -* **Glitch Effect Image** - RGB-channel shift on hover. - -### Typography & Text -* **Kinetic Marquee** - Endless text bands reversing on scroll. -* **Text Mask Reveal** - Massive type as transparent window to video. -* **Text Scramble Effect** - Matrix-style decoding on load / hover. -* **Circular Text Path** - Text curving along spinning circle. -* **Gradient Stroke Animation** - Outlined text with running gradient. -* **Kinetic Typography Grid** - Letters dodging the cursor. - -### Micro-Interactions & Effects -* **Particle Explosion Button** - CTA shatters into particles on success. -* **Liquid Pull-to-Refresh** - Reload indicator like detaching droplets. -* **Skeleton Shimmer** - Shifting light reflection across placeholders. -* **Directional Hover-Aware Button** - Fill enters from cursor's exact side. -* **Ripple Click Effect** - Wave from click coordinates. -* **Animated SVG Line Drawing** - Vectors drawing themselves in real time. -* **Mesh Gradient Background** - Organic lava-lamp blobs. -* **Lens Blur Depth** - Background UI blurred to focus foreground action. - -### Animation Library Choice -* **Motion (`motion/react`)** - default for UI / Bento / state-change motion. -* **GSAP + ScrollTrigger** - for full-page scrolltelling and scroll hijacks. Isolate in dedicated leaf components with `useEffect` cleanup. -* **Three.js / WebGL** - for canvas backgrounds and 3D scenes. Same isolation rule. -* **NEVER mix GSAP / Three.js with Motion in the same component tree.** They fight over the same frames. - ---- - -## 11. REDESIGN PROTOCOL - -This skill handles **greenfield builds AND redesigns**. Misclassifying the mode is the single biggest source of bad redesign output. - -### 11.A Detect the Mode (first action) -* **Greenfield** - no existing site, or full overhaul approved. Dial baseline from Section 1. -* **Redesign - Preserve** - modernise without breaking the brand. Audit first, extract brand tokens, evolve gradually. -* **Redesign - Overhaul** - new visual language on top of existing content. Treat as greenfield for visuals; preserve content and IA. - -If ambiguous, ask **once**: *"Should this redesign preserve the existing brand, or are we starting visually from scratch?"* - -### 11.B Audit Before Touching -Document the current state before proposing changes: -* **Brand tokens** - primary / accent colors, type stack, logo treatment, radii. -* **Information architecture** - page tree, primary nav, key conversion paths. -* **Content blocks** - what exists, what's doing work, what's filler. -* **Patterns to preserve** - signature interactions, recognisable hero, copy voice. -* **Patterns to retire** - AI-slop tells, broken layouts, dead links, generic stock imagery, perf traps. -* **Dial reading of the existing site** - infer current `DESIGN_VARIANCE` / `MOTION_INTENSITY` / `VISUAL_DENSITY`. That's your starting point, not the baseline. -* **SEO baseline** - current ranking pages, meta titles, structured data, OG cards. **SEO migration is the #1 redesign risk.** - -### 11.C Preservation Rules -* **Do not change information architecture** unless asked. Keep page slugs, anchor IDs, primary nav labels stable for SEO and muscle memory. -* **Extract brand colors before applying Section 4.2.** A brand that is already purple stays purple - apply the LILA RULE's override. -* **Preserve copy voice** unless asked for a rewrite. Visual modernisation ≠ content rewrite. -* **Honor existing accessibility wins.** Do not regress focus states, alt text, keyboard nav, contrast. -* **Respect existing analytics events.** Do not rename buttons, form fields, section IDs that downstream tracking depends on. - -### 11.D Modernisation Levers (priority order) -Apply in order - stop when the brief is satisfied: -1. **Typography refresh** - biggest visual lift per unit of risk. -2. **Spacing & rhythm** - increase section padding, fix vertical rhythm. -3. **Color recalibration** - desaturate, unify neutrals, keep brand accent. -4. **Motion layer** - add `MOTION_INTENSITY`-appropriate micro-interactions to existing components. -5. **Hero & key-section recomposition** - restructure top-of-funnel using Section 10 vocabulary. -6. **Full block replacement** - only when the existing block is unsalvageable. - -### 11.E Decision Tree: Targeted Evolution vs Full Redesign -* IA, content, and SEO sound → **targeted evolution** (Levers 1-4). ~70% of value at ~40% of risk. -* Visual debt is structural (broken IA, no design system, broken mobile) → **full redesign** with strict content preservation. -* Brand itself is changing → **greenfield**. - -### 11.F What Never Changes Silently -Never modify without explicit user approval: -* URL structure / route slugs. -* Primary nav labels. -* Form field names or order (breaks analytics + autofill). -* Brand logo or wordmark. -* Existing legal / consent / cookie copy. - ---- - -## 12. THE BLOCK LIBRARY (Contract - Implementations Land Here Iteratively) - -The Reference Vocabulary (Section 10) names patterns. The Block Library implements them with real props, real motion specs, and real code sketches. - -**Status:** schema defined here. Blocks will be added iteratively. Do not freelance new blocks without following this schema. - -### 12.A File Location -``` -skills/taste-skill/blocks/ - hero/ - asymmetric-split.md - editorial-manifesto.md - kinetic-type.md - ... - feature/ - bento-grid.md - sticky-scroll-stack.md - zig-zag.md - ... - social-proof/ - pricing/ - cta/ - footer/ - navigation/ - portfolio/ - transition/ -``` - -### 12.B Required Frontmatter -```yaml ---- -name: asymmetric-split-hero -category: hero -dial_compatibility: - variance: [6, 10] - motion: [3, 10] - density: [2, 5] -when_to_use: "Landing pages with one strong asset and one strong message. Default hero for SaaS, agency, premium consumer." -not_for: "Editorial / manifesto launches where the message IS the design." -stack: ["react", "next", "tailwind", "motion"] ---- -``` - -### 12.C Required Body Sections -1. **Visual sketch** - short ASCII or description of the layout. -2. **Props API** - the component's interface. -3. **Code sketch** - minimal working implementation (Server Component default, Client island for motion). -4. **Mobile fallback** - explicit collapse rules for `< 768px`. -5. **Motion variants** - one variant per `MOTION_INTENSITY` band (1-3, 4-7, 8-10). Reduced-motion fallback explicit. -6. **Dark-mode notes** - token strategy specific to this block. -7. **Anti-patterns** - common ways this block goes wrong. -8. **References** - links to real examples in production. - -### 12.D Block-Library Discipline -* One block per file. No multi-block files. -* Every block must work standalone (drop it into a page, it renders). -* Every block must pass the Pre-Flight Check (Section 14). -* Blocks that depend on a design system from Section 2.A live under `blocks//--.md` (e.g. `feature/bento-grid--material.md`). - ---- - -## 13. OUT OF SCOPE - -This skill is NOT for: -* Dashboards / dense product UI / admin panels (use Fluent, Carbon, Atlassian, or Polaris from Section 2.A). -* Data tables (use TanStack Table or AG Grid). -* Multi-step forms / wizards (use Form-specific patterns; this skill won't make them better). -* Code editors (use Monaco / CodeMirror with their official skinning). -* Native mobile (use Apple HIG / Material directly). -* Realtime collab UIs (presence, cursors, OT-aware - different problem class). - -If the brief is one of the above, **say so explicitly**, point to the right tool, and only apply this skill's marketing-page / about-page / landing-page parts to the surfaces where they apply. - ---- - -## 14. FINAL PRE-FLIGHT CHECK - -Run this matrix before outputting code. This is the last filter. - -**THIS IS NOT OPTIONAL. Run every box. If any box fails, the output is not done.** - -- [ ] **Brief inference** declared (Section 0.B one-liner)? -- [ ] **Dial values** explicit and reasoned from the brief, not silently using baseline? -- [ ] **Design system** chosen from Section 2 if applicable, or aesthetic labeled honestly? -- [ ] **Redesign mode** detected and audit performed (if applicable, Section 11)? -- [ ] **ZERO em-dashes (`—`) anywhere on the page.** Headlines, eyebrows, pills, body, quotes, attribution, captions, buttons, alt text. Zero. (Section 9.G - non-negotiable.) -- [ ] **Page Theme Lock**: ONE theme (light, dark, or auto) for the whole page. No section flips to inverted mode mid-page (Section 4.11)? -- [ ] **Color Consistency Lock**: one accent color used identically across all sections (Section 4.2)? -- [ ] **Shape Consistency Lock**: one corner-radius system applied consistently (Section 4.4)? -- [ ] **Button Contrast Check**: every CTA text is readable against its background (no white-on-white, WCAG AA 4.5:1)? -- [ ] **CTA Button Wrap**: no CTA label wraps to 2+ lines at desktop? -- [ ] **Form Contrast Check**: form inputs, placeholders, focus rings, labels all pass WCAG AA against the section background? -- [ ] **Serif discipline**: if a serif is used, it is NOT Fraunces or Instrument_Serif (or it is, with explicit brand justification)? Different serif from your previous project? -- [ ] **Premium-consumer palette check**: if the brief is premium-consumer (cookware / wellness / artisan / luxury), the palette is NOT the AI-default beige+brass+oxblood+espresso family? Different family from your previous premium-consumer project? -- [ ] **Italic descender clearance**: every italic word with `y g j p q` has `leading-[1.1]` min + `pb-1` reserve? -- [ ] **Hero fits the viewport**: headline ≤ 2 lines, subtext ≤ 20 words AND ≤ 4 lines, CTA visible without scroll, font scale planned around image? -- [ ] **Hero top padding**: max `pt-24` at desktop, hero content does not float halfway down the viewport? -- [ ] **Hero stack discipline**: max 4 text elements in hero (eyebrow OR brand strip, headline, subtext, CTAs)? No tiny tagline below CTAs, no trust micro-strip in hero? -- [ ] **EYEBROW COUNT (mechanical)**: count instances of `uppercase tracking` micro-labels above section headlines across all components. Count ≤ ceil(sectionCount / 3)? Hero counts as 1. -- [ ] **Split-Header Ban**: no "left big headline + right small explainer paragraph" pattern as a section header (vertical stack instead)? -- [ ] **Zigzag Alternation Cap**: no 3+ consecutive sections with the same image+text-split layout? -- [ ] **No Duplicate CTA Intent**: no two CTAs with the same intent ("Get in touch" + "Let's talk" both on page = Fail)? -- [ ] **Logo wall = logo only**: no industry / category labels printed below logos? -- [ ] **Bento Background Diversity**: at least 2-3 bento cells have real visual variation (image, gradient, pattern), not all white-on-white text cards? -- [ ] **"Used by / Trusted by" logo wall** lives UNDER the hero, not inside it, uses REAL SVG logos (Simple Icons / devicon) or generated SVG marks, NOT plain text wordmarks? -- [ ] **Copy Self-Audit**: every visible string re-read, no grammatically-broken or AI-hallucinated phrases ("free on its past" type) shipped? -- [ ] **Motion motivated**: every animation can be justified in one sentence (hierarchy / storytelling / feedback / state transition), no GSAP-for-show? -- [ ] **Marquee max-one-per-page**: no two horizontal marquees on the same page? -- [ ] **Navigation on ONE line** at desktop, height ≤ 80px? -- [ ] **Section-Layout-Repetition** check: no two sections share the same layout family (at least 4 different families across 8 sections)? -- [ ] **Bento has rhythm AND exact cell count** (N items → N cells, no empty cells in middle or at end)? -- [ ] **Long lists use the right UI component** (not default `
      ` with `divide-y` for > 5 items - see Section 4.9 alternatives)? -- [ ] **Real images used** (gen-tool first, then Picsum-seed, then explicit placeholder slots) - NO div-based fake screenshots, NO hand-rolled decorative SVGs, NO pure-text minimalism? -- [ ] **No pills/labels overlaid on images** (no `Plate · Brand`, no `Field notes - journal`)? -- [ ] **No photo-credit captions as decoration** (`Field study no. 12 · Ines Caetano`)? -- [ ] **No version footers** (`v1.4.2`, `Build 0048`) on marketing pages? -- [ ] **No micro-meta-sentences** under eyebrows ("Each of these is a feature we ship today...")? -- [ ] **No decoration text strip at hero bottom** (`BRAND. MOTION. SPATIAL.`)? -- [ ] **No floating top-right sub-text** in section headings? -- [ ] **No scoring/progress bars with filled background tracks** as comparison visuals? -- [ ] **No locale / city-name / time / weather strips** unless brief is genuinely globally-distributed or place-focused? -- [ ] **No scroll cues** (`Scroll`, `↓ scroll`, `Scroll to explore`)? -- [ ] **No version labels in hero** (V0.6, BETA, INVITE-ONLY) unless the brief is a launch? -- [ ] **No section-numbering eyebrows** (`00 / INDEX`, `001 · Capabilities`, `06 · how it works`)? -- [ ] **No decorative dots** (zero by default, only for real semantic state)? -- [ ] **No `border-t` + `border-b` on every row** of long lists / spec tables? -- [ ] **Content density** sane: no 20-row data tables, no fake-precise specs without justification, ≤ 25-word sub-paragraphs by default? -- [ ] **Quotes ≤ 3 lines** of body, attribution clean (no em-dash)? -- [ ] **Motion claimed = motion shown**: if `MOTION_INTENSITY > 4`, page actually animates, not just claimed? -- [ ] **GSAP sticky-stack / horizontal-pan** implemented per Section 5.A / 5.B canonical skeleton (`start: "top top"`, `pin: true`, correct scrub)? -- [ ] **No `window.addEventListener('scroll')`** - using Motion `useScroll()` / ScrollTrigger / IntersectionObserver / CSS scroll-driven animations only? -- [ ] **Reduced motion** wrapped for everything `MOTION_INTENSITY > 3`? -- [ ] **Dark mode** tokens defined and tested in both modes? -- [ ] **Mobile collapse** explicit (`w-full`, `px-4`, `max-w-7xl mx-auto`) for high-variance layouts? -- [ ] **Viewport stability**: `min-h-[100dvh]`, never `h-screen`? -- [ ] **`useEffect` animations** have strict cleanup functions? -- [ ] **Empty / loading / error** states provided? -- [ ] **Cards omitted** in favor of spacing where possible? -- [ ] **Icons** from an allowed library only (Phosphor / HugeIcons / Radix / Tabler), no hand-rolled SVG paths? -- [ ] **Motion** isolated in client-leaf components with `'use client'` at the top, memoized? -- [ ] **No AI Tells** from Section 9 (Inter as default, AI-purple, three-equal cards, Jane Doe, Acme, "Quietly in use at")? -- [ ] **Core Web Vitals** plausibly hit (LCP < 2.5s, INP < 200ms, CLS < 0.1)? -- [ ] **One design system** per project (no Material + shadcn mixed)? - -If a single checkbox cannot be honestly ticked, the page is not done. Fix it before delivering. - ---- - -# APPENDICES - Real Source-Backed Reference Material - -The sections below are vendored reference content. They give the agent real install commands, real canonical doc links, and real working starter snippets for each design system named in Section 2. Use them to ground decisions in production reality, not training-data fiction. - -## Appendix A - Install Commands per Design System - -```bash -# Material Web (Material 3) -npm install @material/web - -# Fluent UI React (v9) -npm install @fluentui/react-components - -# Fluent UI Web Components (framework-free) -npm install @fluentui/web-components @fluentui/tokens - -# IBM Carbon -npm install @carbon/react @carbon/styles - -# Radix Themes -npm install @radix-ui/themes - -# shadcn/ui (open code, owned components) -npx shadcn@latest init -npx shadcn@latest add button card badge separator input - -# Primer CSS (GitHub product/devtool UI) -npm install --save @primer/css - -# Primer Brand (GitHub marketing UI) -npm install @primer/react-brand - -# GOV.UK Frontend -npm install govuk-frontend - -# USWDS (US Web Design System) -npm install uswds - -# Atlassian Design System (Atlaskit) -yarn add @atlaskit/css-reset @atlaskit/tokens @atlaskit/button @atlaskit/badge @atlaskit/section-message @atlaskit/card - -# Bootstrap 5.3 -npm install bootstrap - -# Shopify Polaris Web Components (Shopify apps only) -# Add this to your app HTML head: -# -# -``` - -## Appendix B - Canonical Sources (read these before reinventing) - -### Material Web -- https://github.com/material-components/material-web -- https://material-web.dev/theming/material-theming/ -- https://m3.material.io/develop/web - -### Fluent UI -- https://fluent2.microsoft.design/get-started/develop -- https://fluent2.microsoft.design/components/web/react/ -- https://github.com/microsoft/fluentui -- https://learn.microsoft.com/en-us/fluent-ui/web-components/ - -### Carbon -- https://carbondesignsystem.com/ -- https://github.com/carbon-design-system/carbon -- https://carbondesignsystem.com/developing/react-tutorial/overview/ -- https://carbondesignsystem.com/developing/web-components-tutorial/overview/ - -### Shopify Polaris -- https://shopify.dev/docs/api/app-home/web-components -- https://github.com/Shopify/polaris-react -- https://polaris-react.shopify.com/components - -### Atlassian -- https://atlassian.design/get-started/develop -- https://atlassian.design/components/button/examples -- https://atlaskit.atlassian.com/packages/design-system/button/example/disabled -- https://atlassian.design/tokens/design-tokens - -### Primer -- https://primer.style/ -- https://github.com/primer/css -- https://github.com/primer/brand - -### GOV.UK -- https://design-system.service.gov.uk/components/button/ -- https://design-system.service.gov.uk/styles/layout/ -- https://github.com/alphagov/govuk-frontend - -### USWDS -- https://designsystem.digital.gov/documentation/developers/ -- https://designsystem.digital.gov/components/button/ -- https://designsystem.digital.gov/components/card/ -- https://github.com/uswds/uswds - -### Bootstrap -- https://getbootstrap.com/docs/5.3/layout/grid/ -- https://getbootstrap.com/docs/5.3/components/card/ - -### Tailwind -- https://tailwindcss.com/docs/dark-mode -- https://tailwindcss.com/blog/tailwindcss-v4 - -### Radix -- https://www.radix-ui.com/themes/docs/components/theme -- https://www.radix-ui.com/themes/docs/components/card -- https://github.com/radix-ui/themes - -### shadcn/ui -- https://ui.shadcn.com/docs -- https://ui.shadcn.com/docs/components/card -- https://github.com/shadcn-ui/ui - -### Native CSS / W3C standards -- https://developer.mozilla.org/en-US/docs/Web/CSS/Reference/Properties/backdrop-filter -- https://developer.mozilla.org/en-US/docs/Web/CSS/Reference/At-rules/@media/prefers-color-scheme -- https://developer.mozilla.org/en-US/docs/Web/CSS/Reference/At-rules/@media/prefers-reduced-motion -- https://developer.mozilla.org/en-US/docs/Web/CSS/Guides/Grid_layout -- https://developer.mozilla.org/en-US/docs/Web/CSS/Guides/Scroll-driven_animations -- https://drafts.csswg.org/scroll-animations-1/ - -### Apple Liquid Glass (Apple platforms only) -- https://developer.apple.com/design/human-interface-guidelines/materials -- https://developer.apple.com/documentation/TechnologyOverviews/liquid-glass -- https://developer.apple.com/documentation/TechnologyOverviews/adopting-liquid-glass -- https://developer.apple.com/documentation/SwiftUI/Material - ---- - -## Appendix C - Apple Liquid Glass: Honest Web Approximation - -Do **not** treat random CSS snippets as official Apple Liquid Glass. - -### What is official -Apple documents Liquid Glass inside Apple's Human Interface Guidelines and Developer Documentation for **Apple platforms**. It is a dynamic material used across Apple platform UI. Apple's native implementation belongs to Apple platform APIs and system components, **not a public web CSS package**. - -Relevant official docs: -- Apple Human Interface Guidelines → Materials -- Apple Developer Documentation → Liquid Glass -- Apple Developer Documentation → Adopting Liquid Glass -- SwiftUI → Material - -### What is NOT official -There is no `liquid-glass.css` from Apple for normal websites. - -A web approximation can use: -- `backdrop-filter` -- transparent backgrounds -- layered borders -- highlight overlays -- gradients -- motion -- strong contrast fallbacks - -But that is **web glassmorphism / frosted-glass approximation**, not official Apple Liquid Glass. Label it as such in comments. - -### Safer web approximation skeleton - -```css -.liquid-glass-web-approx { - position: relative; - isolation: isolate; - overflow: hidden; - border-radius: 999px; - border: 1px solid rgb(255 255 255 / .32); - background: - linear-gradient(135deg, rgb(255 255 255 / .30), rgb(255 255 255 / .08)), - rgb(255 255 255 / .12); - backdrop-filter: blur(24px) saturate(180%) contrast(1.05); - -webkit-backdrop-filter: blur(24px) saturate(180%) contrast(1.05); - box-shadow: - inset 0 1px 0 rgb(255 255 255 / .48), - inset 0 -1px 0 rgb(255 255 255 / .12), - 0 18px 60px rgb(0 0 0 / .18); -} - -.liquid-glass-web-approx::before { - content: ""; - position: absolute; - inset: 0; - z-index: -1; - border-radius: inherit; - background: - radial-gradient(circle at 20% 0%, rgb(255 255 255 / .55), transparent 34%), - linear-gradient(90deg, rgb(255 255 255 / .18), transparent 42%, rgb(255 255 255 / .14)); - pointer-events: none; -} - -.liquid-glass-web-approx::after { - content: ""; - position: absolute; - inset: 1px; - border-radius: inherit; - border: 1px solid rgb(255 255 255 / .14); - pointer-events: none; -} - -@media (prefers-color-scheme: dark) { - .liquid-glass-web-approx { - border-color: rgb(255 255 255 / .18); - background: - linear-gradient(135deg, rgb(255 255 255 / .16), rgb(255 255 255 / .04)), - rgb(15 23 42 / .42); - box-shadow: - inset 0 1px 0 rgb(255 255 255 / .22), - 0 18px 60px rgb(0 0 0 / .42); - } -} - -@media (prefers-reduced-transparency: reduce) { - .liquid-glass-web-approx { - background: rgb(255 255 255 / .96); - backdrop-filter: none; - -webkit-backdrop-filter: none; - } -} -``` - -**Important:** `prefers-reduced-transparency` has uneven browser support; test it. Always provide enough contrast even without blur. - ---- - -**End of appendices.** Install commands above are reality anchors. The Apple Liquid Glass skeleton is a labeled approximation, not an Apple-issued package. For canonical docs per design system, consult the system's official docs (links in Section 2 plus Appendix B). diff --git a/.claude/commands/emcn-design-review.md b/.claude/commands/emcn-design-review.md deleted file mode 100644 index 1a5c562facd..00000000000 --- a/.claude/commands/emcn-design-review.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -description: Review UI code for alignment with the emcn design system — components, tokens, patterns, and conventions -argument-hint: "[scope] [fix=true|false]" ---- - -# EMCN Design Review - -Arguments: -- scope: what to review (default: your current changes). Examples: "diff to main", "PR #123", "src/components/", "whole codebase" -- fix: whether to apply fixes (default: true). Set to false to only propose changes. - -User arguments: $ARGUMENTS - -## Context - -This codebase uses **emcn**, a custom component library built on Radix UI primitives with CVA variants and CSS variable design tokens. All UI must use emcn components and tokens. - -## Steps - -1. Read the emcn public barrel at `packages/emcn/src/index.ts` (re-exports components, Calendar, Table*, and icons) to know what's available; for the full icon set read `packages/emcn/src/icons/index.ts` -2. Read `apps/sim/app/_styles/globals.css` for CSS variable tokens -3. Analyze the specified scope against every rule below -4. If fix=true, apply the fixes. If fix=false, propose the fixes without applying. - ---- - -## Imports - -- Import from `@/components/emcn` barrel, never subpaths -- Icons from `@sim/emcn/icons` -- Use `cn` from `@/lib/core/utils/cn` for conditional classes - -## Design Tokens - -Use CSS variable pattern (`text-[var(--text-primary)]`), never Tailwind semantics (`text-muted-foreground`) or hardcoded colors (`text-gray-500`, `#333`). - -**Text**: `--text-primary`, `--text-secondary`, `--text-tertiary`, `--text-muted`, `--text-body` (canonical value text), `--text-icon`, `--text-placeholder`, `--text-subtle`, `--text-inverse`, `--text-error` -**Surfaces**: `--bg`, `--surface-1` through `--surface-7`, `--surface-hover`, `--surface-active` -**Borders**: `--border`, `--border-1`, `--border-muted` -**Brand/accent**: `--brand-secondary`, `--brand-accent` -**Z-Index**: `--z-dropdown` (100), `--z-modal` (200), `--z-popover` (300), `--z-tooltip` (400), `--z-toast` (500) -**Shadows**: `shadow-subtle`, `shadow-medium`, `shadow-overlay`, `shadow-card` -**Badges**: `--badge-*` semantic families (success/error/gray/blue/purple/orange/amber/teal/cyan/pink, each with `-bg`/`-text`) - -## Buttons - -Intent-to-variant mapping (read the actual `buttonVariants` in `packages/emcn/src/components/button/button.tsx` for the full variant set — it exposes more than listed here): - -| Action | Variant | -|--------|---------| -| Toolbar, icon-only | `ghost` | -| Create, save, submit | `primary` | -| Cancel, close | `default` | -| Delete, remove | `destructive` | -| Selected state | `active` | -| Toggle | `outline` | - -## Delete/Remove Confirmations - -`ChipModal` `size='sm'`, title "Delete/Remove {ItemType}", destructive confirm button, plain Cancel (follow the chip footer layout in `.claude/rules/emcn-components.md`). Use `text-[var(--text-error)]` for irreversible warnings. - -## Toast - -`toast.success()`, `toast.error()`, `toast()` from `@/components/emcn`. Never custom notification UI. - -## Badges - -`red`=error/failed, `gray-secondary`=metadata/roles, `type`=type annotations, `green`=success/active, `gray`=neutral, `amber`=processing, `orange`=paused, `blue`=info. Use `dot` prop for status indicators. - -## Icons - -Default: `size-[14px]`. Color: `text-[var(--text-icon)]`. Scale: 14px > 16px > 12px > 20px. Use the `size-*` shorthand — flag `h-[Npx] w-[Npx]` and `h-N w-N` pairs as refactor targets. - -## Anti-patterns to flag - -- Raw `