fix(connectors): validate and repair the knowledge-base connector fleet - #6757

Four review passes over all 61 knowledge-base connectors, validated against each provider's live documentation or machine-readable spec. Passes 3 and 4 reviewed the previous pass's edits, which is where most of what follows was caught — including regressions the audit itself introduced.

The dominant bug class

shouldReconcileDeletions hard-deletes any stored document absent from a listing that did not set syncContext.listingCapped. So any path that turns a failure into an empty or short listing is a silent mass-delete. Most fixes here are instances of that one shape:

• fireflies — firefliesGraphQL ended in return data?.data ?? {} after a .catch(() => null) parse, so a 2xx with an unparseable body (gateway HTML, truncation) became a confident empty listing with no error and no flag. fireflies declares no incremental sync, so every run is a full sync: a fault persisting across two consecutive syncs would have tombstoned all 1,469 live documents. Now throws, with regression tests confirmed to fail on revert.
• linear — same shape via (data.issues || {}) on a non-nullable IssueConnection!.
• gmail — a failed labels.list was swallowed, producing a query matching nothing across every label-scoped source.
• github — pass 2's lastDot <= 0 change dropped dotfiles from listings. The revert is differential-tested at 15 paths × 8 configs (120 combos, 0 differences vs staging); the same harness reproduces the regression at 4 combos.
• sharepoint — the cap gate missed in-page truncation. Now documents.length < stubs.length, mutation-verified.

Cap-at-exhaustion

The inverse error: setting listingCapped when the cap lands exactly on source exhaustion permanently blocks reconciliation, so deleted documents are never cleaned up. Fixed across gmail, ashby, azure-devops, and others.

Pagination

"Last-page precision" (shrinking page size to the remaining budget) is only valid for cursor/offset APIs. On page-number APIs it duplicates and permanently skips records. Reverted on greenhouse (Harvest) and intercom (/articles), both confirmed page-numbered against their specs. Also reverted on airtable, asana, and ashby, where the page size varied across requests reusing an opaque cursor token — undocumented whether the token survives a changed page size, and the cap was already enforced server-side.

Security

SecureFetchHeaders declares private setCookies: string[], but TypeScript private is compile-time only — at runtime it is an own enumerable property. @sim/logger copies own enumerable properties into the formatted line and the retry loop logs { error } on every failed attempt, so newly-attached response headers wrote upstream Set-Cookie values into connector-sync logs. Now attached non-enumerably. Scope is the dev/colorize branch only — production reduces errors to .message — so this is a local exposure, not a prod incident.

SFTP now requires a host-key fingerprint on all sources (fails closed, verified by mutation: 7 tests go red). Zero live SFTP sources, so no migration.

Fabricated doc citations

Six TSDoc comments quoted provider sentences that do not exist (github, jira, jsm, linear, google-meet, dropbox). One inverted the usual failure mode: google-meet's code was correct and the comment invented a sentence to justify it, which nearly got a correct fallback reverted. Worth noting two of three "unverifiable" flags carried into the final pass turned out to be fully documented — re-checking beat reverting on suspicion.

Removed

Evernote, entirely — its auth flow was non-functional, so this is removal rather than repair. Verified complete: 6 surviving references, all prose in agent-tooling markdown, zero in shipped code; both icon registries de-registered.

No user impact. Zero Evernote knowledge-base sources exist, no deployed workflow references the Evernote block, and no workflow containing it has ever been executed.

Customer impact

One-time re-index, deliberate (externalId unchanged in all three, so these are in-place updates — no deletes, no re-creates, no re-auth):

Other behavior changes worth watching on first sync:

• airtable — records with attachment/collaborator/barcode/button cells re-index once. This ends permanent churn: the old hash included 2-hour-expiring attachment URLs, so those records were re-indexing on every sync.
• asana — archived projects are now excluded; tasks living only under archived projects reconcile away.
• github — file ceiling rises 15,000 → 100,000. Repos previously hitting it had deletions permanently suppressed and will now reconcile normally. An add wave, not a delete wave. Worth watching peak stub memory on the largest source (22 sources / 102,995 docs).
• google-drive — getDocument now throws instead of returning null, so persistent content-read failures become visible failed rows. Expect a one-time bump in reported failures across 39 sources; this is the intended invariant, not a defect.
• confluence — previously-truncated label-filtered sources will index their full corpus once.
• monday — MONDAY_API_VERSION extraction silently bumps two selector routes from an inline 2024-10 pin to 2026-04. Almost certainly a fix (the old pin was deprecated and resolving to Maintenance anyway; the affected queries are trivial), but it is a behavior change riding in what reads as a pure refactor.

Verified against production rather than assumed: the linear page-size clamp has no live exposure — only one of six sources sets maxIssues, at 200, a multiple of the 50-item page size, so the old and new code fetch identically. microsoft-teams and mintlify have zero sources, which is what makes the shared htmlToPlainText entity-decoding change free.

Operational notes for the first post-deploy sync

Two things will look alarming in logs and are not incidents. Both were measured against the code, not estimated.

notion's re-index will not finish in one task window. 9,596 documents must re-hydrate against a documented ~3 req/s average. The engine hydrates 5 documents concurrently (SYNC_BATCH_SIZE = 5) and the task budget is 30 minutes (maxDuration: 1800), so expect the run to be killed mid-sync, with heavy 429 backoff before that. It is not destructive and needs no intervention: batches commit as they go, a page that exhausts its retries rejects and is therefore excluded from deletion reconciliation, and a re-hydrated page classifies unchanged next run. It should converge over roughly four to seven scheduled syncs.

jira's deletion reconciliation will run for the first time. PAGE_SIZE 50 → 100 raises the effective listing ceiling from 25,000 to 50,000 (MAX_PAGES = 500). At ~49,931 live documents the source previously always hit the ceiling, which set listingTruncated and unconditionally blocked reconciliation — so stale rows have been accumulating since setup. It now fits under the ceiling, and the first full sync may tombstone a large batch of issues genuinely deleted in Jira. That is the intended correction and it is two-phase (tombstone, then delete only on a second consecutive absence, behind the suspect-listing guard), but it is worth an eyes-on rather than a surprise. Note the margin is thin: if that source grows past 50,000 it truncates again and fails safe.

Smaller one-time re-indexes, all deliberate: reddit (hash now keyed on edit revision + comment count, so edits are finally detected — the old hash was immutable), microsoft-teams (content-derived hash, and content now includes threaded replies), and outlook (conversations whose listing-derived and recomputed dates had diverged).

Two deliberate scope narrowings that delete already-indexed rows: s3 drops rtf from the default extension list (the indexed "text" was RTF control words) and excludes keys containing ./.. path segments. Both are intentional filters, so they correctly do not set listingCapped and the affected documents reconcile away.

Verification

27/27 audits (check:audits), 24/24 type-check, biome clean, 1315 tests passing.

Final validation sweep

A last read-only /validate-connector pass ran over all 59 changed connectors plus the shared files, verifying every quoted claim against provider specs. It found six real defects, all now fixed:

• ashby, azure-devops — getDocument returned null on an API-shape fault. Both connectors set contentDeferred, so these paths were live: a fulfilled null records no failure and no log, and the document silently vanishes.
• dropbox — 409 covers Dropbox's entire LookupError union, and restricted_content / locked both mean the file still exists. Only not_found is absence now.
• docusign — fetchFormValues swallowed every non-OK status, baking a permanently incomplete document (the hash is metadata-only, so the next sync sees it as unchanged forever).
• typeform — all sent response_type=started,partial,completed, but Typeform documents only partial and completed. An unknown enum member risks a 400 that fails the whole sync, and staging omitted the parameter entirely — so this shipped as a regression. Now sends the widest documented set, which is strictly wider than staging's effective behavior.
• github — a utf-8 blob branch was added on a misattributed quote. That sentence describes the encoding request parameter of Create a blob; the GET response is documented as always base64. Branch removed, and two comments corrected that were hiding a real drop (>1 MB files under vnd.github+json 403 rather than returning encoding: none).

That makes eight fabricated citations found across the audit, two of them introduced by the audit itself. Every one is now either corrected or removed.

The sweep also confirmed the htmlToPlainText blast radius independently: exactly microsoft-teams and mintlify both decode HTML and hash the result, verified by intersecting the 26 htmlToPlainText importers against the 7 computeContentHash callers.

hubspot additionally moved to a shared anchored HTML-detection helper. The loose pattern matched angle-bracketed prose (Reply from John <john@acme.com>), and htmlToPlainText deletes the span and collapses line structure — so a false positive loses data rather than passing it through. This was worth fixing now rather than later precisely because the hubspot:v2: bump rewrites every live document once.

Known gaps

All 61 connectors have now been through the final pass. The last six (s3, typeform, x, youtube, zendesk, zoho-desk) were reviewed against the four invariants — cap gating, getDocument absence semantics, contentHash parity, and pagination shape. All six have zero live production sources.

Findings from that pass:

• typeform / zendesk getDocument returned null on missing required config (formId, subdomain). null reads as documented absence, so on an add the document is dropped with neither a failure counter nor a log. Unreachable today — both listDocuments paths throw on the same condition, aborting the sync first — but inconsistent with the invariant and live the moment getDocument gains a second caller. Fixed: both now throw.
• Cap gating is correct in all six. typeform (hitLimit && (slicedSome || sourceHasMore)), x (slicedByCap || nextToken || moreUsernames), youtube (hitMax && (trimmedByCap > 0 || nextPageToken)) and s3 (hitLimit && moreAvailable) all cover in-page truncation without firing at cap-at-exhaustion. zoho-desk uses an explicit probe-for-more, and zendesk additionally cross-checks the API's own totalMatches — both are stronger than the fleet norm.
• No page-number taper: zendesk follows next_page link URLs through an origin check, and the rest are cursor- or token-paginated.

One pre-existing issue found, deliberately not fixed here. youtube builds its contentHash at two call sites from different source fields — the stub uses playlistItems.contentDetails.videoPublishedAt, the hydrated document uses videos.list snippet.publishedAt. Because the stub sets contentDeferred: true, getDocument runs for every video, so any divergence between those two fields would make the hydrated hash never match the stub hash and re-index every video on every sync — the same non-converging churn fixed in greenhouse here. It is unchanged from staging (not a regression from this PR), has zero live sources, and confirming whether the two fields actually diverge needs a live API call, so it belongs in its own change rather than an unreviewed edit at a ship gate.

isScopeSatisfiedBy (lib/oauth/utils.ts) was the one shared-OAuth item flagged as needing scrutiny — it treats a granted non-readonly scope as satisfying its .readonly variant by string suffix, which would be unsound if any provider had a foo.readonly scope where foo is not a superset. Checked and clear. Exactly three .readonly requiredScopes exist fleet-wide, all Google, and all three appear verbatim in their provider scope lists, so exact match already satisfies them. The suffix rule fires for only one: ediscovery (bare) is grantable and is a genuine documented superset of ediscovery.readonly. The other two cannot trigger it — bare meetings.space and forms.responses are not real scopes and appear in no provider list (Meet grants meetings.space.created, which is not a superset). No false-positive path exists.

Two structural issues found but deliberately not fixed here:

1. outlook passes a conversation date from listDocuments to getDocument through an untyped syncContext key. Correct today — sync-engine.ts:978 is the only caller — but the fallback fails silently and permanently if that coupling breaks, re-indexing on every sync with no error. connectors/types.ts documents syncContext as a cache, not a correctness-critical handoff.
2. "Full resync" is destructive on capped sources. fullSync overrides listingCapped and skips the tombstone grace period, hard-deleting hidden documents in one pass. Correct semantics, but it deserves a UI warning.