Skip to content

Commit d4642bd

Browse files
waleedlatif1waleedlatif1
committed
refactor(connectors): split client metadata from server runtime + cover node:net in client bundle
The browser build broke with `Cannot find module 'node:net'`. Server-only SSRF code in `input-validation.server.ts` (`dns/promises`, and since PR #5060 `undici` → `node:net`/`node:tls`) is statically reachable from the client bundle via the tool/connector registries, which the workflow editor imports for metadata. Node networking builtins have no browser shim, so Turbopack cannot compile them for the client. Two changes: 1. Split each connector's client-safe declarative metadata into a sibling `meta.ts` (`<name>ConnectorMeta`), mirroring the `XBlockMeta` / `BLOCK_META_REGISTRY` pattern. `connectors/registry.ts` is now the client-safe `CONNECTOR_META_REGISTRY` (+ `getConnectorMeta` / `getAllConnectorMeta`); the full registry with runtime fns moves to `connectors/registry.server.ts`. Client components consume the meta registry; the sync engine and knowledge API routes consume the server registry. This removes connectors from the client's server-only graph. Connector metadata is byte-for-byte identical before/after; runtime fns are untouched. 2. Extend the existing #4899 `turbopack.resolveAlias` browser stub — which already mapped `dns`/`dns/promises` to an empty module for the browser — to also cover `net`/`tls` (+ `node:` variants), since `undici` now pulls those in. The remaining tool/provider definitions still reach `input-validation.server` server-side; the browser-only stub keeps those Node builtins out of the client bundle while the real modules stay on the server, so SSRF validation and IP pinning are unaffected. Connector authoring/validation skills updated to teach the meta.ts split.
1 parent 3fe061e commit d4642bd

127 files changed

Lines changed: 4305 additions & 3647 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/add-connector/SKILL.md‎

Lines changed: 65 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@ You are an expert at adding knowledge base connectors to Sim. A connector syncs
1212
When the user asks you to create a connector:
1313
1. Use Context7 or WebFetch to read the service's API documentation
1414
2. Determine the auth mode: **OAuth** (if Sim already has an OAuth provider for the service) or **API key** (if the service uses API key / Bearer token auth)
15-
3. Create the connector directory and config
16-
4. Register it in the connector registry
15+
3. Create the connector directory: a client-safe `meta.ts` (declarative metadata) plus the runtime module that spreads it
16+
4. Register it in BOTH the server registry and the client-safe meta registry
1717

1818
## Hard Rule: No Guessed Response Or Document Schemas
1919

@@ -32,13 +32,19 @@ If the source schema is unknown, do one of these instead:
3232

3333
## Directory Structure
3434

35+
Each connector is split into a client-safe metadata file and a server-only runtime file. This mirrors the `XBlockMeta` / `BLOCK_META_REGISTRY` split in `apps/sim/blocks` — client components (the knowledge UI) only need the metadata (icon, name, auth, config fields), so the runtime functions (which pull server-only helpers like `input-validation.server` → `undici` → `node:net`) must stay out of the client bundle.
36+
3537
Create files in `apps/sim/connectors/{service}/`:
3638
```
3739
connectors/{service}/
38-
├── index.ts # Barrel export
39-
└── {service}.ts # ConnectorConfig definition
40+
├── index.ts # Barrel export (re-exports the runtime connector)
41+
├── meta.ts # ConnectorMeta — client-safe declarative metadata
42+
└── {service}.ts # ConnectorConfig — spreads the meta + adds runtime functions
4043
```
4144

45+
- `meta.ts` exports `{service}ConnectorMeta: ConnectorMeta`. It imports ONLY the icon from `@/components/icons`, `import type { ConnectorMeta } from '@/connectors/types'`, and any pure-data constants. It must NEVER import server/runtime code.
46+
- `{service}.ts` exports `{service}Connector: ConnectorConfig`. It imports the meta via `import { {service}ConnectorMeta } from '@/connectors/{service}/meta'`, spreads it as the first property, and holds the runtime functions (which may import server-only helpers like `@/lib/knowledge/documents/utils`).
47+
4248
## Authentication
4349

4450
Connectors use a discriminated union for auth config (`ConnectorAuthConfig` in `connectors/types.ts`):
@@ -55,19 +61,17 @@ For services with existing OAuth providers in `apps/sim/lib/oauth/types.ts`. The
5561
### API key mode
5662
For services that use API key / Bearer token auth. The modal shows a password input with the configured `label` and `placeholder`. The API key is encrypted at rest using AES-256-GCM and stored in a dedicated `encryptedApiKey` column on the connector record. The sync engine decrypts it automatically — connectors receive the raw access token in `listDocuments`, `getDocument`, and `validateConfig`.
5763
58-
## ConnectorConfig Structure
64+
## Connector Structure (meta.ts + runtime)
65+
66+
The declarative metadata lives in `meta.ts` (`ConnectorMeta`). The runtime functions live in `{service}.ts` (`ConnectorConfig`), which spreads the meta as its first property.
5967
60-
### OAuth connector example
68+
### `meta.ts` — client-safe metadata
6169
6270
```typescript
63-
import { createLogger } from '@sim/logger'
6471
import { {Service}Icon } from '@/components/icons'
65-
import { fetchWithRetry } from '@/lib/knowledge/documents/utils'
66-
import type { ConnectorConfig, ExternalDocument, ExternalDocumentList } from '@/connectors/types'
67-
68-
const logger = createLogger('{Service}Connector')
72+
import type { ConnectorMeta } from '@/connectors/types'
6973

70-
export const {service}Connector: ConnectorConfig = {
74+
export const {service}ConnectorMeta: ConnectorMeta = {
7175
id: '{service}',
7276
name: '{Service}',
7377
description: 'Sync documents from {Service} into your knowledge base',
@@ -85,6 +89,26 @@ export const {service}Connector: ConnectorConfig = {
8589
// Supports 'short-input' and 'dropdown' types
8690
],
8791

92+
// Optional: tag definitions are metadata too — declare them here
93+
// tagDefinitions: [ ... ],
94+
}
95+
```
96+
97+
Keep `meta.ts` free of any server/runtime import. Only the icon, the `ConnectorMeta` type, and pure-data constants belong here.
98+
99+
### `{service}.ts` — runtime (OAuth example)
100+
101+
```typescript
102+
import { createLogger } from '@sim/logger'
103+
import { fetchWithRetry } from '@/lib/knowledge/documents/utils'
104+
import { {service}ConnectorMeta } from '@/connectors/{service}/meta'
105+
import type { ConnectorConfig, ExternalDocument, ExternalDocumentList } from '@/connectors/types'
106+
107+
const logger = createLogger('{Service}Connector')
108+
109+
export const {service}Connector: ConnectorConfig = {
110+
...{service}ConnectorMeta,
111+
88112
listDocuments: async (accessToken, sourceConfig, cursor) => {
89113
// Return metadata stubs with contentDeferred: true (if per-doc content fetch needed)
90114
// Or full documents with content (if list API returns content inline)
@@ -111,8 +135,11 @@ Only map fields in `listDocuments`, `getDocument`, `validateConfig`, and `mapTag
111135

112136
### API key connector example
113137

138+
The split is identical — `auth` lives in `meta.ts`, runtime functions in `{service}.ts`.
139+
114140
```typescript
115-
export const {service}Connector: ConnectorConfig = {
141+
// meta.ts
142+
export const {service}ConnectorMeta: ConnectorMeta = {
116143
id: '{service}',
117144
name: '{Service}',
118145
description: 'Sync documents from {Service} into your knowledge base',
@@ -126,6 +153,11 @@ export const {service}Connector: ConnectorConfig = {
126153
},
127154

128155
configFields: [ /* ... */ ],
156+
}
157+
158+
// {service}.ts
159+
export const {service}Connector: ConnectorConfig = {
160+
...{service}ConnectorMeta,
129161
listDocuments: async (accessToken, sourceConfig, cursor) => { /* ... */ },
130162
getDocument: async (accessToken, sourceConfig, externalId) => { /* ... */ },
131163
validateConfig: async (accessToken, sourceConfig) => { /* ... */ },
@@ -499,7 +531,9 @@ If the service already has an icon in `apps/sim/components/icons.tsx` (from a to
499531

500532
## Registering
501533

502-
Add one line to `apps/sim/connectors/registry.ts`:
534+
Register in BOTH registries, keeping the same alphabetical-by-id ordering in each.
535+
536+
1. **Server registry** — `apps/sim/connectors/registry.server.ts` (server-only full registry; holds full connectors with runtime functions, imported by the sync engine and knowledge API routes):
503537

504538
```typescript
505539
import { {service}Connector } from '@/connectors/{service}'
@@ -510,6 +544,19 @@ export const CONNECTOR_REGISTRY: ConnectorRegistry = {
510544
}
511545
```
512546

547+
2. **Client-safe meta registry** — `apps/sim/connectors/registry.ts` (imports each connector's `meta.ts` only, so client components can use it without pulling server-only code; the metadata counterpart to `BLOCK_META_REGISTRY`):
548+
549+
```typescript
550+
import { {service}ConnectorMeta } from '@/connectors/{service}/meta'
551+
552+
export const CONNECTOR_META_REGISTRY: ConnectorMetaRegistry = {
553+
// ... existing connector metas ...
554+
{service}: {service}ConnectorMeta,
555+
}
556+
```
557+
558+
`registry.ts` exports `CONNECTOR_META_REGISTRY: ConnectorMetaRegistry` plus the helpers `getConnectorMeta(id)` and `getAllConnectorMeta()`, importing each `@/connectors/{service}/meta` directly — never the runtime module. `registry.server.ts` exports `CONNECTOR_REGISTRY: ConnectorRegistry`.
559+
513560
## Reference Implementations
514561

515562
- **OAuth + contentDeferred**: `apps/sim/connectors/google-drive/google-drive.ts` — file download with metadata-based hash, `orderBy` for deterministic pagination
@@ -520,7 +567,8 @@ export const CONNECTOR_REGISTRY: ConnectorRegistry = {
520567

521568
## Checklist
522569

523-
- [ ] Created `connectors/{service}/{service}.ts` with full ConnectorConfig
570+
- [ ] Created `connectors/{service}/meta.ts` with `{service}ConnectorMeta: ConnectorMeta` (icon, name, auth, configFields, tagDefinitions) — no server/runtime imports
571+
- [ ] Created `connectors/{service}/{service}.ts` with `{service}Connector: ConnectorConfig` spreading the meta + runtime functions
524572
- [ ] Created `connectors/{service}/index.ts` barrel export
525573
- [ ] **Auth configured correctly:**
526574
- OAuth: `auth.provider` matches an existing `OAuthService` in `lib/oauth/types.ts`
@@ -542,4 +590,5 @@ export const CONNECTOR_REGISTRY: ConnectorRegistry = {
542590
- [ ] All external API calls use `fetchWithRetry` (not raw `fetch`)
543591
- [ ] All optional config fields validated in `validateConfig`
544592
- [ ] Icon exists in `components/icons.tsx` (or asked user to provide SVG)
545-
- [ ] Registered in `connectors/registry.ts`
593+
- [ ] Registered the full connector in `connectors/registry.server.ts`
594+
- [ ] Registered the meta in `connectors/registry.ts` (same alphabetical-by-id ordering as registry.server.ts)

‎.agents/skills/validate-connector/SKILL.md‎

Lines changed: 20 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,12 @@ When the user asks you to validate a connector:
2121
Read **every** file for the connector — do not skip any:
2222

2323
```
24-
apps/sim/connectors/{service}/{service}.ts # Connector implementation
24+
apps/sim/connectors/{service}/meta.ts # ConnectorMeta — client-safe metadata (icon, name, auth, configFields, tagDefinitions)
25+
apps/sim/connectors/{service}/{service}.ts # Connector implementation — spreads the meta + runtime functions
2526
apps/sim/connectors/{service}/index.ts # Barrel export
26-
apps/sim/connectors/registry.ts # Connector registry entry
27-
apps/sim/connectors/types.ts # ConnectorConfig interface, ExternalDocument, etc.
27+
apps/sim/connectors/registry.server.ts # Server-only full registry entry (CONNECTOR_REGISTRY; full connector)
28+
apps/sim/connectors/registry.ts # Client-safe meta registry entry (CONNECTOR_META_REGISTRY)
29+
apps/sim/connectors/types.ts # ConnectorMeta / ConnectorConfig interfaces, ExternalDocument, etc.
2830
apps/sim/connectors/utils.ts # Shared utilities (computeContentHash, htmlToPlainText, etc.)
2931
apps/sim/lib/oauth/oauth.ts # OAUTH_PROVIDERS — single source of truth for scopes
3032
apps/sim/lib/oauth/utils.ts # getCanonicalScopesForProvider, getScopesForService, SCOPE_DESCRIPTIONS
@@ -262,10 +264,18 @@ Connectors where the list API already returns content inline (e.g., Slack messag
262264
- [ ] Logs sync progress at `info` level
263265
- [ ] Logs errors at `warn` or `error` level with context
264266

267+
### Meta / Runtime Split
268+
- [ ] `connectors/{service}/meta.ts` exports `{service}ConnectorMeta: ConnectorMeta` (id, name, description, version, icon, auth, configFields, and any `tagDefinitions` / `supportsIncrementalSync`)
269+
- [ ] `meta.ts` imports ONLY the icon from `@/components/icons`, `ConnectorMeta` (type-only), and pure-data constants — NO server/runtime imports (`@/lib/knowledge/...`, `input-validation.server`, `fetchWithRetry`, etc.); any such import in `meta.ts` is **critical** (breaks the client bundle)
270+
- [ ] `connectors/{service}/{service}.ts` spreads `...{service}ConnectorMeta` as the first property and adds the runtime functions (`listDocuments`, `getDocument`, `validateConfig`, `mapTags?`)
271+
- [ ] Metadata fields (id, name, auth, configFields, etc.) live ONLY in `meta.ts`, not duplicated in `{service}.ts`
272+
265273
### Registry
266274
- [ ] Connector is exported from `connectors/{service}/index.ts`
267-
- [ ] Connector is registered in `connectors/registry.ts`
268-
- [ ] Registry key matches the connector's `id` field
275+
- [ ] Full connector is registered in `connectors/registry.server.ts` (server-only registry, `CONNECTOR_REGISTRY`)
276+
- [ ] Meta is registered in `connectors/registry.ts` (client-safe registry, `CONNECTOR_META_REGISTRY`), importing `@/connectors/{service}/meta`
277+
- [ ] Both registries use the same key and it matches the connector's `id` field
278+
- [ ] Both registries keep the same alphabetical-by-id ordering
269279

270280
## Step 11: Report and Fix
271281

@@ -284,6 +294,8 @@ Group findings by severity:
284294
- Query/filter injection: user-controlled values interpolated into OData `$filter`, SOQL, or query strings without escaping
285295
- Per-document content download in `listDocuments` without `contentDeferred: true` — causes sync timeouts for large document sets
286296
- `contentHash` mismatch between `listDocuments` stub and `getDocument` return — causes unnecessary re-processing every sync
297+
- Server/runtime import in `meta.ts` (e.g. `@/lib/knowledge/...`, `input-validation.server`, `fetchWithRetry`) — pulls server-only code into the client bundle and breaks the build
298+
- Connector missing from `connectors/registry.ts` (the client-safe meta registry) — or its entry there imports the runtime module instead of `meta.ts` — the knowledge UI can't render it
287299

288300
**Warning** (incorrect behavior, data quality issues, or convention violations):
289301
- HTML content not stripped via `htmlToPlainText`
@@ -323,7 +335,7 @@ After fixing, confirm:
323335

324336
## Checklist Summary
325337

326-
- [ ] Read connector implementation, types, utils, registry, and OAuth config
338+
- [ ] Read connector meta.ts, implementation, types, utils, both registries, and OAuth config
327339
- [ ] Pulled and read official API documentation for the service
328340
- [ ] Validated every API endpoint URL, method, headers, and body against API docs
329341
- [ ] Validated input sanitization: no query/filter injection, URL fields normalized
@@ -342,7 +354,8 @@ After fixing, confirm:
342354
- [ ] Validated API efficiency: field selection used, no redundant calls, sequential fetches batched
343355
- [ ] Validated error handling: graceful failures, no unhandled rejections
344356
- [ ] Validated logging: createLogger, no console.log
345-
- [ ] Validated registry: correct export, correct key
357+
- [ ] Validated meta/runtime split: `meta.ts` holds metadata with no server/runtime imports, `{service}.ts` spreads the meta + adds runtime functions
358+
- [ ] Validated registry: exported from index.ts, full connector in `registry.server.ts`, meta in `registry.ts`, matching keys and alphabetical-by-id ordering in both
346359
- [ ] Reported all issues grouped by severity
347360
- [ ] Fixed all critical and warning issues
348361
- [ ] Ran `bun run lint` after fixes

0 commit comments

Comments
 (0)