Skip to content

v0.8.4: nextjs upgrade, cli updates, jotform triggers, secrets descriptions, helm updates - #6805

Open
waleedlatif1 wants to merge 21 commits into
mainfrom
staging
Open

v0.8.4: nextjs upgrade, cli updates, jotform triggers, secrets descriptions, helm updates#6805
waleedlatif1 wants to merge 21 commits into
mainfrom
staging

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

waleedlatif1 and others added 20 commits August 17, 2026 14:04
…live code (#6777)

* chore(deps): upgrade next to 16.3.1, its optimizer no longer deletes live code

16.3.0 was reverted in #6242 because its Turbopack optimizer modelled a bare
`return <asyncCall>()` tail call inside an async function as returning the
promise object, propagated that always-truthy fact through the caller's `await`,
and deleted everything after the resulting `if`. That shipped two dead code
paths to production: the whole `POST /api/credentials` create path, and the
insert inside `upsertAsyncToolCall`.

We reported it as vercel/next.js#96595. The fix — "[turbopack] Collapse nested
promises in the analyzer" (vercel/next.js#96601) — folds `Promise<Promise<T>>`
to `Promise<T>` in the analyzer, and was backported as #96675 and released in
16.3.1.

Verified before taking the bump:

- The minimal reproduction from the issue no longer reproduces on 16.3.1. All
  four routes keep their code; on 16.3.0 `/api/broken` lost everything after
  the `if`.
- A production build of `apps/sim` on 16.3.1 still emits the markers whose
  disappearance was the original signal: `credential_connected` (43 files),
  `acquireOrganizationUserMutationLocks` (28), and the `upsertAsyncToolCall`
  insert-path warning (10).

The `return await` hardening added to both sites in the revert stays as is, and
so does the TypeScript toolchain configuration.

16.3.1 published 2026-08-13, so it is inside the 7-day `minimumReleaseAge`
supply-chain window until 2026-08-20 and needs an exclusion to install. The
alternative is sitting on 16.2.12, whose successor we already reverted once, so
the entries go in dated and come out on the next touch of the file. The mermaid
and js-yaml exclusions aged out on 2026-08-11 and 2026-08-07 and are dropped
here per that same rule.

* fix(deps): keep the musl and win32 SWC binaries in the lockfile

The release-age exclusion only listed the four @next/swc platforms that
package.json pins, but next declares all eight as its own optionalDependencies,
so all eight are normally resolved into bun.lock. A gated optional dependency
does not fail the install — bun drops it silently — so the first install
stripped both musl variants and both win32 variants from the lockfile.

That left the Alpine devcontainer and any Windows machine with no SWC binary to
resolve. Adding the remaining four to the exclusion list restores all eight
entries at 16.3.1.

Worth knowing for the next time this happens: bun.lock is sticky here. Once an
optional dependency has been dropped, re-running the install — even with
--force, even with the age gate switched off entirely — does not bring it back,
because the resolution is not reattempted. The lockfile has to be regenerated
from a base that still contains the entries, which is why this restores
bun.lock from staging before re-applying the bump.
…elease-age waivers (#6784)

js-yaml < 4.3.1 has quadratic CPU consumption in !!omap resolution
(GHSA-5p4m-2wfm-xmqj / CVE-2026-59870). sim-cli builds with
--packages=bundle, so its dev-scoped js-yaml is bundled into the
published CLI. 4.3.1 is already what apps/sim pins, so this collapses
sim-cli onto the hoisted copy.

The minimumReleaseAge waivers for js-yaml and mermaid were temporary
and have both aged past the 7-day window; leaving them behind would
disable the supply-chain gate for those packages indefinitely.
…OneDrive (#6785)

* fix(connectors): index Office documents and PDFs from SharePoint and OneDrive

The SharePoint and OneDrive connectors filtered their listings against a
12-item plain-text extension whitelist, so a document library of .docx, .pdf
or .xlsx files synced as "success, 0 documents" — no document, no failed row,
and no log line, which is indistinguishable from a wrong folder path. Both
whitelists had been unchanged since the connectors shipped, and Sim already
parses all of these formats for a manually uploaded knowledge base document.

Adds a shared `extractConnectorText` in connectors/utils that routes binary
document formats through the same `parseBuffer` the upload path uses, so the
OOXML zip-bomb guard and each parser's extraction limits apply. The
previously-accepted text formats stay on their exact existing path: sending
.csv through CsvParser would silently reformat every already-indexed connector
document on its next re-index.

Also logs a per-page count of files skipped for an unsupported extension.
Unsupported files are counted rather than turned into failed document rows, so
a library full of images does not fill the knowledge base with noise.

* fix(connectors): never index a degraded document extraction

`DocParser` and `PptxParser` never throw by design — on a legacy OLE `.doc`/`.ppt`
or a deck with no extractable text they return a placeholder sentence or scraped
ZIP internals so an interactive upload still shows the user something. Verified
against real OOXML fixtures: an image-only `.pptx` yields 1.9KB of
`[Content_Types].xml…` as "content", and a legacy `.ppt` yields "Unable to
extract text from PowerPoint file."

A connector sync would embed that into the vector index at scale, so it needs to
tell a real extraction from a fabricated one. Adds a declared `degraded` flag to
`FileParseMetadata`, set by exactly those two fallback paths, rather than having
callers sniff `extractionMethod`. `DocParser`'s plaintext branch stays unflagged:
a text file misnamed `.doc` is a genuine extraction.

`extractConnectorText` now raises `ConnectorTextExtractionError` when a parsed
format comes back degraded or blank, and SharePoint/OneDrive surface it as a
skipped document via the existing `markSkipped` path — so the file appears in the
knowledge base as a failed row telling the user to re-save it as DOCX/PPTX/XLSX,
instead of being silently dropped or indexed as junk.

The upload path is unaffected; it ignores the new flag.

* fix(file-parsers): register the document variants the parsers already handle

A document library holds whole format families, not just the headline extension
of each. These all extract correctly with the libraries already installed — they
were simply never registered, so every one of them was reported as an
unsupported file type:

  docm dotx  (WordprocessingML — mammoth reads word/document.xml regardless of
              the package content type)
  xlsm xlsb xltx ods  (SheetJS reads every workbook container natively)
  pptm potx  (PresentationML)
  odt odp    (OpenDocument, via a new OpenDocumentParser)

Verified against real fixtures built with jszip and SheetJS rather than assumed:
officeparser identifies a Buffer by sniffing content with `file-type`, not by the
name we pass, so the routing had to be measured. `ods` goes to the spreadsheet
parser rather than OpenDocumentParser so its output keeps per-sheet structure.

`rtf` is deliberately excluded: nothing bundled extracts it, and DocParser's
plaintext branch would pass its control words through as if they were prose.

Converts the registry from `require()` inside per-parser `try/catch` blocks that
only logged to static imports. Every parser dependency is a regular, non-optional
one, so a resolution failure should fail loudly — the old form produced a silently
**empty** registry in which every format became `Unsupported file type`, with an
empty "Supported types are:" list as the only clue. The heavy extraction libraries
are still deferred inside the individual parsers, and connectors now import the
registry lazily so the ~60 connectors that never touch a file do not pull SheetJS.

Adds registry.test.ts, which exercises the real module: index.test.ts mocks
`@/lib/file-parsers` itself, so it validated its own fake routing table and the
real registry had no coverage at all. The new test gates every member of
SupportedFileType on having a registered parser that supports buffer parsing.

* fix(file-parsers): resolve the parser registry through a Map, not object keys

The registry rewrite switched extension lookup from
`Object.keys(parsers).includes(ext)` to a bracket read on an object literal,
which also resolves inherited keys. `PARSERS['constructor']` therefore returned
`Object` — truthy, with no parse methods — so a caller-supplied extension of
`constructor` fell through to "does not support buffer parsing" instead of being
rejected as an unsupported type, and `parseFile` would have raised a TypeError.
It also disagreed with `isSupportedFileType`, which used `Object.hasOwn` and
correctly returned false for the same input.

A Map has no prototype chain to walk, so lookup and support check now agree by
construction. `isSupportedFileType` also guards a non-string argument, which the
try/catch it replaced used to absorb.
* fix(cli): resolve findings from a full command-surface audit

Exercised all 147 commands against a live deployment. Fixes the defects
that surfaced, plus the docs and generator drift they exposed.

Transport
- Stop following redirects. A bare domain that 301s to www silently
  converted POST to GET and dropped the body, so reads worked while every
  write failed with a misleading validation error and login returned 405.
  Both the client and the device flow now explain the redirect and name
  the endpoint to configure, rather than carrying credentials off-origin.
- Report a non-JSON response as one instead of printing the HTML page.
- Name the personal-API-key remedy on a workspace-key refusal, reading the
  machine-readable code the API actually sends.
- Drop union-branch noise from validation errors that contradicted itself.
- Show paging progress on stderr for multi-page fetches.

Output
- Clamp record values for table only. text is the format built for pipes,
  and it was truncating signed URLs and tool source mid-value.
- Infer timestamp, duration, bytes and boolean formatting for API-owned
  keys so undeclared commands stop printing raw ISO and float ms. Skips
  user-defined table cells and leaves json/yaml on the raw payload.
- Render a declared-but-absent field as an em dash; billing credits were
  vanishing silently.

Paths, naming and validation
- Percent-encode folder paths per segment and decode them for display, so
  a folder reads and types as the name shown in the app.
- Reject a malformed endpoint where it is set and where it resolves,
  instead of crashing with a URL parse trace.
- Request the detail level logs list's own columns need; its workflow
  column could never populate.
- Rename three commands that described themselves wrongly and align two
  flags with their siblings. Old spellings still work: hidden, warned on
  stderr, and kept out of help and docs.
- Verify whoami against the API, separating a bad key from an unreachable
  endpoint, and report the workspace by name.
- Correct the --yes help text, which advertised skipping a prompt that
  does not exist.

Docs
- Teach the docs generator that a flag required by the runtime is required,
  and that hidden commands are not documented.

* fix(cli): clear the paging progress line when a page fails

Progress is written without a trailing newline so it can be overwritten in
place, and both paging loops cleaned it up only on success. A page that
threw part-way through left `fetched 1200…` on the line the error was then
printed onto, so the two ran together.

* fix(cli): name a working API root when an endpoint redirects

The suggested endpoint was the redirect target's origin, which drops a path
prefix. A self-hosted deployment reached at https://host/sim was told to set
https://www.host — not an API root, so following the advice replaced one
broken endpoint with another.

Derive it by stripping the request's own path from the target instead, so a
prefix survives, and say nothing about --set-endpoint when the target
resolves to the endpoint already configured: a trailing-slash or path
normalization redirect keeps the origin, and naming the value the caller
already has explains nothing. The login poll shared both faults and now
shares the helper.
* fix(workflows,connectors): close pre-merge audit findings

Recover subblock values orphaned by the id renames in this release, and
stop truncated knowledge-base listings from reporting themselves complete.

- Add operation-scoped subblock id migrations so a saved workflow's stored
  value survives a rename. Cloudflare create/update DNS record, ServiceNow
  read record, and Okta deactivate/delete previously lost their stored value:
  the create path substituted a seeded default (an A record where the user
  chose CNAME, and unproxied where they chose proxied), and the update path
  silently no-opped while reporting success. A migration is used rather than
  a legacy-id fallback so no subblock id carries two value spaces at runtime.
- Webflow, Zendesk: a listing that stops for a reason the connector cannot
  rule out now reports as capped instead of exhausted. A malformed envelope,
  an unfollowable continuation link, or an absent collection list previously
  read as a complete listing and let deletion reconciliation hard-delete
  every document past the truncation point.
- Sentry: pin the listing window in the request rather than inheriting the
  server default, so the range cannot silently narrow into hard deletes.
- Fork sync: a parent re-pick no longer writes a blank over a hidden optional
  dependent's stored target value, and a required field stays on screen once
  it is filled. Add hook-level coverage for the submitted payload.
- Fork file copy: a file whose name is already taken in a reused target folder
  is de-duplicated instead of dropped.
- Delete an orphaned Shopify OAuth route that built a credential from unsigned
  cookies. It had no writer, no caller, and no inbound link.
- Tailwind: drop two content globs that scanned 5.4k files to emit one unused
  rule, keeping the ones that fix brand tile icon color.
- Correct the API route-count baseline, add an Evernote docs redirect, align
  library copy with the language rules, and fix a stale turbo filter.

* fix(connectors,forking): trim the audit fixes to their minimum

A legitimacy review found several changes closed no live defect, and two
introduced problems of their own.

- Zendesk: narrow the cursor fix to a signal change. Treating a missing meta
  envelope as truncation had also made the walk follow links.next and keep
  paginating, and the ticket cursor has no page-depth valve, so a source
  advertising a next page with no meta could loop without terminating. The
  page-fetch set now matches the previous behavior; only the flag is new.
- Zendesk: drop the search next_page branch. The existing count check already
  caps every case where a missing key could lose documents.
- Webflow: drop the empty-collections flag. The sync engine already blocks the
  first sync on an empty listing and reconciles only when a second sync agrees,
  which handles a transient fault better and still removes documents when a
  source is genuinely emptied. The flag short-circuited that and suppressed
  reconciliation permanently. Restore the previous loud failure on a non-array
  envelope, and drop the unreachable collection-id filter.
- Webflow: soften a docstring that claimed pagination.total is always present.
  It is documented optional, so its absence proves nothing either way and
  treating it as unprovable truncation is the fail-safe reading.
- Sentry: drop the pinned statsPeriod. Sentry's issue search floors every query
  at 90 days in the executor regardless of the request, and the endpoint this
  release moved away from hit the same floor, so there was no window to close.
  Keep the tests and the docstring recording that.
- Fork copy: drop the renamed counter, which no caller reads.
- Repair check-block-registry, which stopped exempting migrated subblock ids
  when the migration map became an array — `in` was testing array indices.
- Drop mdx from a Tailwind content glob that emits nothing, and loosen an
  exact compiled-SQL assertion to the invariant it was pinning.

* fix(migrations): keep a ServiceNow write body off the read projection

Review findings from the first round.

- A legacy ServiceNow block can hold a Create/Update Record JSON body under
  `fields` while its stored operation is Read Records: the id served both value
  spaces before the rename, and a subblock value is not cleared when the
  operation changes. The scoped migration moved that body onto `readFields`,
  where it would reach the wire as sysparm_fields. Migration entries can now
  carry a `whenValue` predicate for the case where the stored operation alone
  cannot separate two value spaces, and the ServiceNow entry uses it to move
  only a plausible comma-separated projection.
- Type the fork copy test harness instead of using `any`, without weakening it:
  every predicate shape it does not model still throws rather than matching.
- Correct the dependent-omission comments. Omitting a parent-invalidated field
  preserves the target's stored value on Save and across an undo, where the
  parent nets out unchanged; on a Sync the written state is source-derived, so
  what it prevents there is an explicit blank reaching the fields the remap's
  clearing pass does not cover, nested tool params in particular.

Okta's migration scope is left as-is: `okta_remove_user_from_app` and the
sendEmail split shipped in the same release, so no saved block can hold legacy
state for it, and widening the scope would promote an activation-era value onto
the deactivation switch. Tests document the boundary.

* chore(forking): move the fork-sync changes to their own PR

The dependent-omission fix and the fork file-copy de-duplication are reviewed
separately in #6787. They are the only changes here that overlap #6776, and
they carry their own design tradeoff, so they should not ride along with the
unrelated audit fixes in this PR.

* fix(migrations): separate a ServiceNow write body from a projection by parsing

The guard tested for a `{` or `[` prefix, so a stored scalar body — `true`,
`"short_description"`, `42` — read as a field list and was promoted onto
`readFields`, where it would go out as sysparm_fields.

A Create/Update Record body is JSON and a projection is a bare comma-separated
field list, which is never valid JSON, so parsing is the whole test rather than
a guess at its opening character. Ambiguity still resolves to "not a
projection", leaving the value where the Create/Update control owns it.

* test(connectors,credentials): tie two assertions to what they actually prove

- Webflow: a non-array collections envelope reaching `for...of` throws, which
  is the intended loud failure. Assert the spec-mandated TypeError plus a
  single request and no write-back, rather than matching V8's wording.
- Credentials: the second guard test cannot observe "not deleted" — the proxy
  driver replays canned rows — so name it for what it does verify, that the
  reference check carries no workspace predicate and an empty RETURNING logs
  nothing. Making the driver decide the outcome would fake the database.
- Drop `vi.importActual`; a plain `drizzle-orm/pg-proxy` import works now that
  `drizzle-orm` is un-mocked.

* fix(migrations): identify a ServiceNow projection by its own shape

Recognising a write body was the wrong way round. A saved body is not always
well-formed: it can be a half-typed draft or carry an unquoted block reference,
so neither "opens with a brace" nor "fails to parse as JSON" identifies one —
and a body misread as a projection is moved to readFields with its original key
dropped, losing the draft.

Match the projection instead: a comma-separated list of ServiceNow field names,
which are word characters plus the dot of a dotted walk. A brace, quote, colon,
angle bracket or interior space fails that shape. Parsing then removes the bare
scalars that satisfy it by accident.
…e key (#6789)

A mothership chat attachment is minted with the same storage key shape as a
workspace file — `resolveUploadStorage` calls `generateWorkspaceFileKey` for
`mothership_attachment` — but its row is written with `context = 'mothership'`.
The serve route branched on the key prefix alone, so every attachment entered
the workspace-file use case, which matches on `context = 'workspace'` and
resolved it to nothing: `{"error":"FileNotFoundError","message":"File not
found"}` for every chat thumbnail and click-through. The `context=mothership`
query param on the serve URL is decorative; the route never read it.

Serve now resolves the storage context from the key's stored binding, which is
server-authored at upload and the only thing that separates the two, and passes
it into the cloud and local handlers instead of re-inferring. Genuine workspace
files still go through the authorized use case.

`verifyWorkspaceFileAccess` takes the context too, so an attachment authorizes
from its database row rather than falling through to storage-object metadata.
A soft-deleted attachment is now denied, matching workspace files.

The route tests are what let this ship: they mocked `inferContextFromKey` to
return 'mothership' for a `workspace/…` key, which it never does. With the mock
made honest, nine of them fail against the old route.
`sim.ai` answers /api/** with a 301 to `www.sim.ai`, and the client refuses
to follow redirects — a 301 rewrites a POST into a bodyless GET, so
following one turns a write into a silent no-op and hands the API key to
whatever host Location names. Defaulting to the apex therefore failed every
command for anyone who never set an endpoint. Before the refusal shipped it
was quieter and worse: reads succeeded while writes did nothing.

Also trims the provider catalogue from eleven inferred columns to seven.
`docsUrl`, `helpText`, `requiresClientGeneratedCredentialId` and the nested
`fields` are what you read once you have chosen a provider, not what you
scan to choose one, and they pushed the table well past a terminal. Both
ids stay: `credentials connect` names an OAuth provider by `serviceId`,
`credentials create` matches a service account on `providerId`.
…PI (#6790)

* fix(sap_concur): align the integration with SAP Concur's documented API

Validated all 70 tools, the block, and both proxy routes against SAP's
published API docs.

Auth:
- add the password and companyUuid to the token cache key so a request
  with the wrong password can no longer be served a cached token minted
  from someone else's
- wire the documented company-level flow (username = company UUID,
  credtype = authtoken) so companyUuid actually scopes a token
- expand the datacenter allowlist to the documented set (adds glz, apj1,
  usg, the impl hosts, and the www- twins) and drop the undocumented cn
  host; validate the returned geolocation by shape instead of membership
- coalesce concurrent token fetches so a fan-out mints one token
- forward Retry-After so 429 retries pace off Concur's own hint
- handle the errorMessageList, SCIM detail, and legacy Error.Message
  shapes instead of falling through to a generic HTTP message
- pin redirects and cap the response body

Block:
- collapse six contextType subBlocks that disagreed on their default, so
  a new block no longer seeds MANAGER for every operation
- clamp contextType to each operation's documented set
- stop requiring a userId and contextType that the default operation's
  tool does not accept, and scope the receipt fields to the upload ops
- reach six params that had no subBlock, and pass userId on travel
  request updates so a stale value cannot impersonate

Tools:
- correct response shapes that resolved to undefined: budget headers,
  budget categories, allocations, receipts, SCIM nextCursor, and the
  delete endpoints that return a bare boolean
- use the Travel Request Amount schema (currency, not currencyCode)
- narrow the four XML-only travel tools to a documented string payload
  and request application/xml
- surface real errors instead of a JSON parse failure when the proxy
  returns a non-JSON body
- cap receipt uploads at the documented sizes before downloading

Adds 106 tests covering the token cache, geolocation validation, path
traversal, and error extraction.

* fix(sap_concur): drop the removed forwardId subblock via a migration

Removing the `forwardId` subblock without a migration entry breaks
deployed workflows that still carry a value under that key.

It fed a `concur-forwardid` request header that is documented nowhere in
Concur's Receipts v4 or Image v1 references, so it was never honored.
There is no replacement subblock and the value is an opaque
caller-chosen string rather than a secret, so it is dropped outright.

* fix(sap_concur): stop swallowing upload response-read failures

The upload route caught every error from the bounded response read and
continued down the success path, so a size-limit breach or a stream
failure surfaced as an upstream success with a null or header-only body.

Concur returns Content-Length: 0 on a successful image-only upload, and
readResponseTextWithLimit already returns an empty string for that
without throwing, so dropping the catch keeps the legitimate empty-body
case working while letting real read failures reach the route's handler.

* fix(sap_concur): unblock company auth and correct the body wand prompt

The password grant marked username required, so the company-level flow —
which sends the company UUID as the token username and has no user login
— could not be configured at all, even though the request schema and
token fetch already accept companyUuid without a username. Username is
now optional for that grant and the server-side check reports which of
the two is missing. Relabels the password and companyUuid fields to say
what they carry in the company flow.

The shared body wand prompt also still described several payloads the
way they looked before this branch: quick expenses in PascalCase rather
than v4 camelCase, travel requests and expected expenses using
currencyCode where the Request v4 Amount schema uses currency, the
standard SCIM SearchRequest URN instead of Concur's, startIndex as a
search parameter when it is unsupported, and a cash advance shape that
does not match the documented request. A wand-generated body was
therefore rejected for most of the create operations it covers.

* fix(sap_concur): keep Concur's status when an error body fails to read

Removing the blanket catch from the upload read fixed one failure mode
and introduced its inverse: a cap breach or stream error while reading a
non-success body threw before the route reached the branch that
preserves Concur's status, so an upstream 4xx surfaced as a Sim 500 and
could trigger a retry the caller should not make.

Both routes now split the two cases. On a success status the body is the
result, so a read failure still propagates. On an error status the body
only supplies the message, so a read failure resolves empty and the
upstream status survives, with the message falling back to the generic
HTTP-status form.

Adds 21 tests covering both helpers over success, error, empty-body and
boundary statuses; inverting the status check turns 14 of them red.
…y refusal (#6792)

Sends a `User-Agent` on both request paths. Without one a CLI request is
indistinguishable from any other API traffic, so a bug that reproduces on a
single CLI version cannot be found in the server's own logs; the runtime and
platform ride along because they are the first things asked about a
transport failure only some users hit.

The version moves into its own module so the HTTP client can read it. It
could not import `program.ts` — program builds the commands, which reach the
client — and duplicating the manifest read would let the two disagree.

Also recognises `PRINCIPAL_KIND_NOT_PERMITTED`. The same refusal is raised at
two layers under two codes, and only the workspace-key one was matched, so
the audit-log commands reported "Principal kind workspace_api_key cannot
perform operation audit_logs.list" — accurate, written for a server log, and
missing the one sentence that tells the reader a personal key resolves it.
* fix(cli): keep a path prefix in the login URLs

`sim login` built both of its URLs with `new URL('/path', endpoint)`. A
leading-slash path is absolute, so it resolves against the endpoint's origin
and drops any path the endpoint carries: a deployment served at
`https://host/sim` sent the browser to `https://host/cli/auth` and polled
`https://host/api/cli/auth/poll`, neither of which exists there.

Every other command builds its URL by concatenation and was unaffected, so
the endpoint looked correct and login alone failed.

Both now go through the client's `buildUrl`, which is exported for it rather
than duplicated — one URL builder for the whole CLI is the point, since two
of them is how the halves drifted apart. Its TSDoc now names the trap.

* test(cli): restore fetch with spyOn so the stub cannot outlive its test

vi.stubGlobal is not undone by restoreAllMocks, so the completed-auth
response would have leaked into whatever ran next. The rest of this file
already spies on globalThis.fetch, which the existing teardown restores.
* feat(files): support zip extraction

* fix(files): harden zip extraction safety

* fix(files): batch extraction notifications

* fix(files): defer rollback storage cleanup

* fix(files): restore reliable drag uploads

* fix(files): use explicit archive extraction route

* fix(ci): account for archive extraction route

* refactor(files): bound every archive extraction and trim the extractor's option surface

`maxMaterializedItems` was opt-in, so only the new unzip route bounded its output
tree — the copilot `materialize_file` and `POST /api/tools/file/manage` extract
paths had no cap on folder creation at all. An archive within MAX_ARCHIVE_ENTRIES
can still imply far more folders than files, so the cap now defaults to
MAX_WORKSPACE_FILE_BULK_AFFECTED_ITEMS and applies to all three callers.

`materializedRootFolderCount` was a hand-maintained number that had to agree with
what an opaque callback would create, and the callee could not check it; drift
surfaced only as an over-limit archive slipping past the cap. It is now derived
from whether `prepareRootFolder` ran, so the contract is just "the callback
creates exactly one folder".

Also single-sources the ArchiveError -> HTTP status map (it was copied into both
the internal error policy and the tools route), drops IdempotencyService config
that only `executeWithIdempotency` reads (the extraction lease uses
atomicallyClaim/release, so no result is ever stored), hoists the duplicated
predicates in purgeCreatedWorkspaceFile and archiveWorkspaceFileFolderIfEmpty so
a lock and its write cannot diverge, and names UPLOAD_SESSION_LOCAL_PUT_MAX_BYTES
rather than overloading the multipart part size as the local single-PUT ceiling.

Adds coverage for the two guards nothing exercised: the re-validation of the
segments `prepareRootFolder` actually returned, and the default cap applying with
no caller opt-in.

UI: the drop overlay used --surface-4 unconditionally, which renders grey over the
light-mode canvas; matches the canonical overlay's --white/dark:--surface-4 and
swaps arbitrary px type sizes for named tokens.

* fix(files): bound the extraction write loop so it cannot outlive its lease

Cursor Bugbot flagged two related holes, both rooted in the write loop being
unbounded:

1. `maxDuration` is a Next.js route-segment config that serverless platforms
   enforce and self-hosted deployments do not. A slow extraction (up to 1000
   sequential uploads) could therefore outrun the six-minute lease, and
   `IdempotencyService` reclaims an expired in-progress claim — so a second unzip
   of the same archive could start beside the first.
2. Nothing rolls back a process killed mid-pass-2, so a timeout stranded the
   destination folder and every file written so far.

`decompressArchiveBufferToWorkspaceFiles` now takes an `AbortSignal` and checks it
between entries in both passes, and the extraction use case supplies a 180s
deadline. The abort unwinds through the existing all-or-nothing rollback, so the
work stops on our terms with the tree cleaned up, well inside both the route's
300s budget and the 360s lease. That closes (1) outright — the holder can no
longer outlive its lease on any platform — and converts (2) from a stranded
partial tree into a clean rollback for the slow case that actually triggers it. A
SIGKILL still cannot be caught; that needs a durable job and is out of scope here.

The overrun surfaces as a caller-fixable 413 naming the archive rather than an
opaque 500 from the raw DOMException.

* fix(files): only remap the deadline abort itself, and stop overclaiming rollback

Two follow-ups on the budget deadline, both reported by Cursor Bugbot:

`deadline.aborted` stays true for the rest of the request once the timer fires, so
it cannot decide whether *this* error was the abort. An `ArchiveError` or storage
failure thrown mid-entry after the timer fired was being relabelled as a timeout
and returned as a 413, hiding the real cause. The catch now matches the thrown
value against `deadline.reason` — `throwIfAborted()` throws exactly that object,
so the check is identity-exact and cannot capture an unrelated failure.

The message also claimed a rollback that has not necessarily happened: the budget
covers the archive download too, so it can fire before the first write, when there
is nothing to roll back. It now says the unzip was cancelled and claims nothing
about what was written. Including the download in the budget is deliberate — the
lease it has to fit inside starts earlier still — so the TSDoc says that rather
than "the extraction itself".

---------

Co-authored-by: Waleed Latif <walif6@gmail.com>
…nt (#6794)

* fix(sap_concur): key the token cache with an HMAC and document rate casing

The cache key hashed a user-chosen password with a bare SHA-256. The key
never leaves the process, but a password is low-entropy enough to
brute-force out of a plain digest if one ever reached a heap dump or a
debug log, which is what CodeQL flags. Keying the digest with a
server-side secret makes it useless without that secret. A
password-hashing KDF would be the wrong tool here: this runs on every
token fetch, and the goal is collision-free partitioning rather than
verification of a stored credential.

The body wand prompt also claimed every payload family is camelCase.
Exchange rate uploads are the exception — they take a snake_case
currency_sets array of from_crn_code, to_crn_code, start_date and rate —
and that operation is in BODY_OPS, so the blanket claim produced bodies
Concur rejects.

* fix(sap_concur): wire the travel request sendback comment through the block

move_travel_request accepts a documented query comment that Concur
applies to the sendback action, but the params branch never passed it
and the block's only comment field is gated to create_report_comment, so
the value was unreachable from the UI.

Uses a dedicated sendbackComment subblock rather than widening the
existing comment field: that one is required for create_report_comment
while this is optional, so sharing an id would both clash on
required-ness and let a value bleed between the two operations.

* fix(sap_concur): gate the sendback comment and merge duplicate TSDoc

The sendbackComment field was conditioned only on the operation, so it
rendered for submit, approve, cancel and every other workflow action
even though Concur applies the comment to sendback alone. It is now
gated on the action as well, and the params branch only forwards it for
sendback so a value retained from an earlier sendback cannot ride along
once the field is hidden.

Also folds the two consecutive TSDoc blocks left above tokenCacheKey
into one. Only the nearest block binds to the declaration, so the
separator and collision reasoning in the earlier block was detached.
…idth (#6795)

Two loose ends from the command-surface audit.

`--help` stated where the profile files live and named only `~/.sim`, so it
was wrong for anyone who had set `SIM_CONFIG_DIR` — which every CI job
pointing the CLI at a scratch directory has. The variable is documented in
the README and the guides; help was the one place that omitted it.

`knowledge search` printed the raw similarity double, `0.2818957269585687`:
a nineteen-character column whose last dozen digits cannot separate one
result from another. A `score` format fixes it to four decimals, the width
`cost` already uses, so the column stays put down the page. `json` and
`yaml` still carry the full double, which is what a script compares.
* feat(secrets): add optional descriptions to workspace secrets

Workspace secrets already have a backing credential row with a description
column, but nothing surfaced it. Teammates had no way to record what a
secret is for.

- Add a Description field to the secret detail page, matching the
  integrations credential page, gated on workspace-secret admin
- Fold the value and description editors into one Save/Discard pair and
  one unsaved-changes guard; two guards cannot coexist, since each seeds
  its own same-URL history entry
- Match descriptions in the secrets settings search
- Expose description on GET/PUT /api/v2/secrets and in the CLI

Descriptions are workspace-only: env_personal credential rows are
per-workspace mirrors of one user-global secret, so one saved there would
exist in a single workspace, and a personal secret has no teammates to
inform. The API rejects a description on personal scope rather than
silently dropping it, and omitting it on PUT leaves any existing
description untouched so a value rotation cannot erase it.

* fix(secrets): address review findings on secret descriptions

- Patch the credential detail cache optimistically on update. `onMutate`
  cancelled the detail query but only patched the lists, so a detail-backed
  editor stayed dirty after a successful save until the refetch landed —
  long enough for Discard to restore the pre-save value over the committed
  one, and for Back to open the unsaved-changes guard.
- Memoize `useSecretValue`'s returned callbacks and object, per the hook
  convention, so the composed form's save/discard stop churning per render.
- Reject a description on a personal secret in the domain layer rather than
  only at the v2 boundary. The internal credential update path accepted one
  for any type, writing data every reader hides.
- Normalize an empty description to null so the API and UI agree.
- Correct the secrets documentation, which described a Display Name field
  the detail view does not have and omitted the scope rule.
- Drop the CLI's copy of the 500-character bound; it can't import the
  contract, so a copy only drifts from the message the API already returns.
- Collapse a redundant save guard and align the description write gate with
  the render gate.

Leaves the integrations credential page byte-identical to staging.

* fix(secrets): keep the API docs example and CLI column order stable

Backward-compatibility fixes for anyone who never sets a description.

- Move the blank-to-null normalization out of the contract and into the
  route. A Zod `.transform()` on any property drops the whole request
  schema's OpenAPI examples, which had silently removed the Set Secret
  request example from the published docs.
- Append the CLI `description` column instead of inserting it before
  `updated`. `--output text` is positional, so inserting would shift every
  field an existing script cuts.
- Reject a description on a personal secret with a message that says so,
  rather than dropping the field and falling through to the generic
  "no updatable fields" error.
* fix(cli): bound, trace, and explain the requests the CLI makes

Four transport gaps, all of which failed silently.

A request had no timeout, so a connection that was accepted and then never
answered hung the terminal indefinitely. `SIM_TIMEOUT_SECONDS` now bounds
one, defaulting to 3600s — deliberately above every timeout the server
itself applies, since a synchronous workflow run is allowed 3000s on a paid
plan and a tighter default would abort real work and report it as a
transport failure. `0` removes the bound, for a self-hosted deployment that
runs executions without one of its own. The caller's abort signal is
composed with the timeout rather than replaced, so neither masks the other.

Node ignores HTTP(S)_PROXY unless NODE_USE_ENV_PROXY opts in, and only from
v22.21 and v24.5, so on a network that reaches the API only through a proxy
every command failed to connect while the variable that would have fixed it
was already set. The CLI cannot enable that from inside the process — Node
reads it at startup — so it says what to do rather than bundling an HTTP
stack for a setting the platform now owns.

An API key was sent to any http:// endpoint with no signal. Now a warning,
not a refusal: http is the documented way to reach a local dev server, and
a deployment terminating TLS at a gateway is real. Loopback stays silent.

`SIM_DEBUG=1` traces method, URL, status and duration. Bodies and headers
are deliberately absent — the request carries the API key, and `secrets set`
carries the secret itself.

All four write to stderr, so a piped stdout stays parseable.

* fix(cli): make the request bound safe on every runtime it supports

Two ways the new timeout could fail before the request was made.

`AbortSignal.any` arrived in Node 20.3 and this package supports Node 20, so
composing a caller's abort signal with the timeout threw a bare TypeError on
the earliest 20.x releases. It is now used when present and composed through
an AbortController when not.

`AbortSignal.timeout` rejects a fractional millisecond outright, and past
2^31-1 ms it does not fail at all — it clamps to 1ms, so the longest timeout
anyone asked for became the shortest. The value is now rounded and refused
above what Node can actually wait, pointing at 0 for an unbounded wait.

Also unstubs env vars between tests: `stubEnv` is not undone by
`unstubAllGlobals`, so a SIM_TIMEOUT_SECONDS set for one test configured
every test after it.

* fix(cli): correct the proxy version table, and classify a timeout mid-body

`runtimeCanProxy` treated any release between 22 and 24 as capable, so on
Node 23 — which reached end of life before the backport — a configured proxy
was ignored and the CLI stayed silent about it, which is the exact failure
the warning exists to report. The table is now the two lines that shipped the
support, and anything after them.

`AbortSignal.timeout` keeps firing after `fetch` resolves, so a bound that
elapsed while the body was still being read — a large `files get` — escaped
the client's own handling and printed a raw TimeoutError stack. The top-level
handler now names it, which covers the streaming path as well as the JSON
one. A user's own Ctrl-C raises AbortError and is deliberately left alone.

* fix(cli): report a timed-out download as a timeout

`files get --output-file` streams the body to disk, and `streamToFile`
converted anything the stream threw into a write failure. So a request bound
elapsing mid-download read as `Could not write <path>: ...`, sending the
reader to check permissions and free space for a timeout they can raise, and
hiding the one instruction that resolves it.

The predicate and that instruction now live beside the timeout that raises
them, so the client, the top-level handler and the download path all say the
same thing. The wrapping stays where it is: the staged-download cleanup runs
off that failure, and rethrowing past it would leak the temporary directory.

* fix(cli): keep a sub-millisecond timeout bounded

Zero is how this function says "no bound", so rounding a positive
SIM_TIMEOUT_SECONDS down to zero inverted the request: anything under
0.0005s asked for the shortest possible timeout and got none at all, leaving
a stalled request to hang. Introduced by the rounding that fixed the
fractional-millisecond rejection.

Floored at 1ms for every positive value; only a literal 0 still disables.
* feat(jotform): trigger a workflow on every new form submission

Jotform's only webhook event is a new submission, so the block gets one
trigger. Deploying it registers the callback on the form through the API
and undeploying removes it again.

Two things about this provider needed handling:

Jotform posts submissions as multipart/form-data, which the shared webhook
body parser did not read — the delivery died as a 400 before any handler
saw it. The parser now flattens a multipart body the same way it already
flattens a urlencoded one, reducing an uploaded part to its filename so a
stray file cannot inflate the execution input.

The form's webhooks are identified by their position in the form's webhook
map, so an id captured at registration goes stale the moment any other
webhook on that form is removed. Nothing persists it; cleanup re-resolves
the id by matching the callback URL. Registration checks the same way,
because Jotform answers a rejected request with the unchanged list rather
than an error.

Answers are exposed as the parsed `rawRequest` rather than re-keyed by
question label — the labels are not unique, and the payload shape is only
documented as the raw q{qid}_{slug} map.

The trigger's region field is named `apiRegion` so it does not collide
with the block's own advanced-mode `region`.

* fix(jotform): make webhook registration idempotent and URL matching tolerant

Validated the trigger against Jotform's API reference and a captured
delivery (zulip's multipart fixture), which confirmed every mapped field —
formID, submissionID, formTitle, username, ip, type, pretty, rawRequest —
and turned up three things worth correcting.

Jotform keeps a form's webhooks as a plain list and does not treat the URL
as a key, so posting one it already holds leaves the form delivering every
submission twice. Registration now consults the list first and only posts
when the URL is absent. The documented POST sample returns the new entry as
"0", renumbering the rest, which is further reason nothing persists an id.

URL matching no longer lets a trailing slash decide the outcome. Jotform
stores the URL verbatim in every sample seen, but an exact match failing
would hard-fail deploy, and Pipedream's client normalizes the same way.

The rawRequest description claimed the field holds the submitted answers.
A real payload also carries slug, buildDate, submitSource and
jsExecutionTracker, and a file answer appears under the bare slugified
label as upload URLs rather than under a q{qid}_ key — which is also why
filtering to q-prefixed keys would silently drop file answers.

* fix(jotform): keep the callback when an active deployment still needs it

Redeploying prepares the replacement webhook row alongside the live one and
a workflow keeps its path across deployments, so both rows resolve to a
single callback on a single form. Registration adopts the callback already
present instead of posting a duplicate, which left the retired row's
cleanup deleting the one the new row had just adopted — the trigger went
silent after a redeploy that changed the trigger config.

Teardown now skips when another webhook row belonging to an active
deployment resolves to the same form and callback URL, matching how the
Telegram handler skips deleteWebhook while an active deployment still uses
the same bot. A genuine undeploy has no such row and still cleans up.
…27.0.0.1] (#6799)

* fix(email): greet SMTP relays with a qualified hostname instead of [127.0.0.1]

Nodemailer derives the EHLO greeting from os.hostname() and substitutes the
address literal [127.0.0.1] whenever that name contains no dot. Kubernetes pod
hostnames never contain one, so every k8s deployment introduced itself to the
relay as loopback and strict relays refused the session before any mail moved.

Send the domain the app is served from instead, as RFC 5321 4.1.4 asks, with
SMTP_EHLO_NAME to override it for relays that expect a different identity.

* fix(email): parse EHLO address literals and drop a port from the app domain

Review round 1. The bracketed branch matched a character class rather than an
address, so [::::] and [13] reached the relay as a greeting it would refuse.
Parse the address with node:net instead, which also admits the RFC 5321
IPv6: form.

getEmailDomain reports a URL host, so a deployment served on a non-default
port failed the qualified-name check and fell back to nodemailer's default —
[127.0.0.1] again on Kubernetes, the exact failure this change exists to fix.
Strip the port before validating.

* fix(email): accept any casing of the IPv6 literal tag, and stop owning SMTP_EHLO_NAME in setup

Review round 2. RFC 5321 tags the IPv6 address-literal form, and RFC 5234
makes ABNF string literals case-insensitive, so [ipv6:2001:db8::1] is as valid
as [IPv6:...]. The exact-prefix check routed it to isIPv4 and discarded it.

Drop SMTP_EHLO_NAME from the email capability's optional fields. SMTP_SECURE,
the same kind of optional transport knob on the same provider, is not modelled
there either, and claiming the field obliged the setup wizard to prompt for it
— a field whose entire purpose is to stay unset now that the default is right.
…xt (#6803)

Follows the serve fix by stating the contract it relies on, so the next
reader does not re-derive module ownership from the key prefix.

The prefix is authoritative for where the bytes live — bucket and tenant —
and nothing more. Which module owns an object is `workspace_files.context`,
which is server-authored like the key but, unlike the key, mutable: a chat
attachment becomes a workspace file when `materialize_file` flips that
column, and encoding a mutable fact in an immutable key would mean copying
the bytes on every such transition just to restate them.

`resolveTrustedFileContext` claimed the prefix was flatly authoritative.
That claim is what made routing on it look safe. It is now scoped to what
it actually defends — a caller-supplied context can still never relabel a
private key — and `resolveStoredFileContext` is documented as the sanctioned
way to ask who owns an object rather than as a workaround.

`verifyWorkspaceFileAccess` resolved its binding with the lookup filtered to
`context = 'workspace'`, so an attachment missed the row and fell through to
object metadata, which cannot see a soft delete. It now matches either
workspace-scoped context, which is also what every caller already wanted:
the LLM-attachment and presigned-URL paths pass 'workspace' for attachment
keys today. A soft-deleted attachment is now denied on all of them.

The parse route carried the same gate and labelled parsed attachments with
the raw storage segment instead of the uploaded filename.

Module-scoped filters are deliberately untouched: the Files module, its
folder manager, forking and the workspace-file use cases all match
`context = 'workspace'` because they mean the Files module, not the bucket.
…get value (#6787)

* fix(forking): stop a parent re-pick blanking a dependent's stored target value

A dependent selector (a sheet under a spreadsheet, a label under a mailbox)
is invalidated when its parent is re-picked, because the stored child no
longer exists under the new parent. That invalidation was recorded by writing
an empty string into the in-session override map — the same value the user's
own "clear this field" produces. The two are not the same thing, and the map
is submitted verbatim and written into the target workflow's configuration,
so an invalidated field cleared the target's real stored value.

The sharpest case is an undo. Re-pick a parent away from its original target,
then back. The parent nets out unchanged, so nothing is remapped and the
remap's own clearing pass never runs — but the child is still blank, and that
blank lands on a value the user never touched, with nothing in the UI showing
it happened.

Record the invalidation with a distinct marker instead. It reads as blank in
the selector, the in-block chain context, and the Sync gate, so a required
invalidated field still blocks Sync and still renders; but it is omitted from
the submitted payload rather than sent as empty, so no override is written and
the target keeps what it had. A blank the user picked themselves is still
submitted and still clears the target.

Also: skip the cascade entirely when a re-pick selects the value the field
already had, since the selector fires its change handler either way.

Fork file copy: a file whose name is already taken in a reused target folder
is de-duplicated with the same allocator the ordinary upload path uses, rather
than colliding with the folder-name unique index and being dropped from the
fork with its blob deleted.

Adds hook-level coverage for the submitted payload, which had none.

* fix(forking): preserve dependent-chain semantics

* fix(forking): preserve edits during fork sync

* fix(workflows): clear stale dependent inputs in Mothership edits

---------

Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
…6801)

* docs(helm): document null as the way to remove an inherited env key

Setting `app.env.KEY: ""` cannot clear a key that `app.envDefaults` sets:
the Secret template drops empty values, and the deployment template treats
an empty override as "not overridden" and still inlines the default. Helm's
own `KEY: null` deletion is the supported mechanism and already works.

The empty-string behavior is load-bearing, not a bug — every key under
`app.env` ships as a "" placeholder, and ten collide with a real
`envDefaults` value (NEXT_PUBLIC_APP_URL, BETTER_AUTH_URL, ...), so "" has
to read as "unspecified" or a default install would blank them out.

- README: document `null`, with the --reuse-values and Argo CD valuesObject
  caveats; correct the claim that `app.env` always wins over `app.envDefaults`
- values.yaml + self-hosting docs: same guidance where operators look
- sim-helm skill: record why an unset list is the wrong shape here
- tests: lock in that null removes a key and "" does not

* docs(helm): correct the verify command's chart path and scope the required-secret claim

- The verify snippet used a `sim/sim` repo alias that this chart never
  publishes; every other instruction installs from the local `./helm/sim`
  path, so the command could not run as written
- Nulling a boot-critical key only fails at template time with the
  chart-managed Secret. `existingSecret` mode skips that validation
  entirely (the chart cannot read a pre-created Secret), and under ESO the
  key must instead be mapped in externalSecrets.remoteRefs.app

* docs(helm): say null must be applied in every layer that sets a key

`null` deletes a key from the map it is applied to, not from the pod. A key
set in both `app.env` and `app.envDefaults` survives a null on the app.env
entry alone — the deployment then inlines the envDefaults value again. Under
ESO a retained `externalSecrets.remoteRefs.app` mapping keeps syncing the key
regardless of app.env.

- README and self-hosting docs: drop the "works in all three secret modes"
  shorthand and spell out that every layer setting the key must be nulled,
  including the ESO remote mapping
- tests: cover both halves — nulling only app.env restores the envDefault,
  nulling both actually removes the key
- chart 1.5.4; staging took 1.5.3 in the meantime
@waleedlatif1
waleedlatif1 requested a review from a team as a code owner August 18, 2026 01:19
@greptile-apps

greptile-apps Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Too many files changed for review (280 files, 100 file limit).

Bypass the limit by tagging @greptile-apps to review.

@vercel

vercel Bot commented Aug 18, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Skipped Skipped Aug 18, 2026 6:43am

Request Review

@cursor

cursor Bot commented Aug 18, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Changes are almost entirely documentation and one internal skill checklist line; no application runtime or auth logic appears in this diff.

Overview
Updates English docs so they match the v0.8.4 CLI and integration surface rather than describing older command shapes.

CLI reference and guides now document the default endpoint as https://www.sim.ai, sim whoami live verification (with --no-verify), and exit codes 1 vs 2 for scripting. Destructive commands document --yes as required instead of optional skip-confirmation. Folder options are described as paths as shown in the app. Several commands are renamed or relocated in the docs: sim files restore, sim workflows deployment status, sim tables rows count (replacing older count create / nested restore paths), --query on row find, knowledge knowledgeBaseId arguments, logs --details defaulting to full, and sim secrets set --description.

Configuration docs add SIM_TIMEOUT_SECONDS, SIM_DEBUG, and the Node HTTPS_PROXY / NODE_USE_ENV_PROXY warning.

Jotform integration docs add a webhook trigger section (form submission payload fields).

SAP Concur integration docs are expanded to match SAP’s API: OAuth client-credentials wording, optional vs required bodies, XML responses as strings, pagination cursors, SCIM/search payloads, workflow and recall token requirements, and many output field corrections.

Platform credentials docs note viewing/editing key/value and workspace secret descriptions.

A migrate-application-operation skill checklist line changes the type-check turbo filter from sim to @sim/app.

Reviewed by Cursor Bugbot for commit edc25aa. Configure here.

…ite paths (#6808)

* test(table): characterize the single-row route before migrating it

The single-row surface (GET/PATCH/DELETE) had no route-level tests despite
being the hottest table write path. Pin the behavior it emits today so the
migration onto the shared internal route builder is verifiable rather than
hopeful.

Covers status codes, body shapes, ISO-8601 timestamp serialization, the
access level each method demands, collaborator invocation, and the
dual-caller wire keying — session callers pass column ids through untouched
while internal-JWT callers translate names to ids in both directions.

Verified to fail: mutating the wire translator, the deleted count, and the
workspace-ownership guard each turn the corresponding tests red.

* feat(table): model wire keying and actor attribution on row write use cases

The row write use cases assumed every caller speaks column names. That holds
for /api/v2, /api/v1 and the Copilot tools, but not for the first-party grid
or the internal /api/table routes, which address cells by stable storage id.
Feeding id-keyed data through the name remap drops every key it does not
recognise — a storage id names no column name — so the write would store
nothing and still report success.

Make the wire an explicit, required property of the input rather than an
assumption. `dataKeying: 'names' | 'ids'` sits alongside `strictWrite` and is
required for the same reason: a new write surface must state which contract it
publishes. Strictness now means the same thing on either wire — an unknown
column id is refused exactly as an unknown column name already was.

Single-row writes also gain optional actor attribution, so the acting tab can
skip refetching its own write. It is optional and absent by default, so every
existing caller keeps broadcasting to all subscribers as before. Only the
single-row create, update and delete paths accept it; a batch write is not
reconciled locally by the actor and must still refetch.

The attribution pin moves with the behaviour: what selects the actor-scoped
signal is no longer which file calls it but which surface supplies an actor,
so that is now what the test pins.

Verified to fail: ignoring the keying discriminator, and dropping actor
attribution, each turn the corresponding tests red.

* perf(table): remove two round trips from every single-row update

A single-cell PATCH spends far more time in sequential round trips to a remote
Postgres than in the UPDATE it issues. Prepared statements are disabled for
PgBouncer transaction mode, so every await is a full Parse/Bind/Execute.

Two of them were avoidable.

getRowById issued the row lookup and its executions sidecar in series, but the
sidecar is keyed on the row id the caller already supplied, so it never
depended on the lookup. Issuing both together makes it one round trip. A miss
now pays one redundant sidecar read, which is the rare path and costs no extra
wall time.

The uniqueness probe ran whenever the table had any unique column, passing the
fully merged row, so editing an unrelated cell re-probed every unique column —
its own transaction plus one query per column. It is now scoped to the columns
the patch actually writes. A merge cannot newly violate uniqueness on a column
it leaves alone: that value is the one already stored, and it satisfied the
constraint when it was written.

Sized before changing: few tables declare a unique column, but write traffic
concentrates in the ones that do, so this is the larger of the two savings in
practice.

Verified to fail: reverting the probe scoping turns the covering test red.

* perf(table): start the workspace load with the table load when a workspace is asserted

resolveActiveTableContext ran two sequential round trips: load the table,
then load the workspace it turned out to live in. When the caller asserts a
workspace the second input is already in hand, so both can start together.

What makes that safe is unchanged: requireTable still compares the table's
canonical workspaceId against the assertion and reports a mismatch as
not_found. The table outcome is inspected first and unconditionally, so any
path that returns has proven the assertion equal to the canonical id, and a
failing workspace load can never replace the concealing not_found. A final
identity check on the loaded context restates the invariant at the point of
return, so even with the first check removed the function cannot hand back a
foreign workspace. Promise.allSettled keeps the discarded branch from
surfacing as an unhandled rejection.

With no asserted workspace the path stays sequential — the table load is what
reveals which workspace to load, so there is nothing to start early.

One existing assertion changed: it required the workspace load not to have
been issued yet on a mismatched assertion, which is internal sequencing rather
than caller-observable behaviour and is definitionally untrue once the loads
overlap. The observable half is kept, and two timing tests now cover the
sequencing directly.

Verified to fail: removing either mismatch check, reading the workspace
outcome first, and swapping allSettled for bare awaits each turn the
corresponding tests red.

* perf(table): read a table and its latest job in one round trip

getTableById issued the table SELECT and then awaited latestJobForTable, so
every table request paid two sequential round trips. With prepared statements
disabled for PgBouncer transaction mode every await is a full round trip, and
this loader is on essentially every table route.

The job read cannot be skipped: a table's reported rowCount is the stored
count minus the job's pendingDeleteRemaining, so dropping it would overstate
the count during a pending delete and could wrongly reject inserts as over
capacity. An opt-out flag would have made that a caller's trap. Instead the
job is read in the same statement, as a correlated jsonb subquery in the
select list — the select-list form of a LEFT JOIN LATERAL, which is what
drizzle can type here. Output is unchanged for every input.

latestJobForTable is deleted rather than left dangling: getTableById was its
only caller, and keeping it would have carried a third copy of the
exports-excluded / newest-started_at / limit-one rule. mapJobRow is now
exported so the batch path and the lateral share one implementation of the
doomedCount and pendingDeleteRemaining logic. The batch DISTINCT ON path used
by the list endpoint is untouched.

Verified to fail: dropping the export filter, reversing or re-keying the sort,
dropping the limit, dropping the correlation, loosening either doomedCount
condition, and removing the rowCount subtraction each turn tests red. Dropping
the lateral from the projection initially survived, because the shared db mock
returns queued rows regardless of predicate; a projection assertion now
covers it.

* chore(table): drop the unused single-row GET contract

Authored as groundwork for migrating the internal row routes onto the shared
builder, which is not in this change. An exported contract nothing consumes is
dead code, so it lands with the migration that needs it instead.

* fix(table): resolve strict id-keyed columns through getColumnId

assertKnownColumnIds read column.id directly, but a column id is optional —
pre-backfill columns have none and are stored under their name, which is why
getColumnId exists and is what every other consumer of the schema uses. Such columns still
exist, so a strict id-keyed write naming one would have been refused as
unknown.

Latent today: no surface yet combines dataKeying 'ids' with strictWrite. Fixed
before one does.

Verified to fail: reading column.id turns the covering test red.

* refactor(table): apply review findings from the quality pass

Four parallel reviews (reuse, simplification, efficiency, altitude) converged
on the same set. Applied:

Removed actorClientId entirely. It had no supplier anywhere in the repo, so
every call reached signalTableRowsChangedByActor(id, undefined), which is
byte-identical to the broadcast it replaced — three optional fields, three
verbatim doc blocks and a pin test asserting the empty set, all inert. It
belongs with the route migration that supplies an actor. The attribution pin
is restored to its original form.

The uniqueness probe was only half-narrowed: the patched column list was
computed and then discarded, and the probe re-derived every unique column from
the full schema. It now receives only the columns the patch touched, so a
table with several unique columns runs one query instead of all of them.

assertKnownColumnIds hand-rolled an id index and duplicated the sibling
assert's message verbatim. It now reuses buildColumnNameById — which already
keys by getColumnId, so the legacy pre-backfill column case is handled by the
shared helper rather than by a special case here — and both asserts share one
message builder.

The job field list had become two copies, one drizzle-checked and one an
unchecked sql<T> cast that could silently return undefined for a renamed
field. The lateral now derives its jsonb pairs from JOB_PROJECTION, which
satisfies Record<keyof LatestJobRow, Column>. That compile-time guarantee
replaces the runtime drift test it makes redundant.

Also: hoisted the id index out of the batch loop to match the names path,
dropped a never-supplied parameter, narrowed an over-broad parameter type,
removed a dead timer cleanup and its now-unused import, replaced dynamic
re-imports with the static one already present, hoisted a repeated stub, and
documented why filters need no keying counterpart and how laxness differs
between the two wires.

Verified to fail: removing a field from JOB_PROJECTION breaks the build in two
places.

* test(table): share one table-definition fixture factory

buildTable was copy-pasted into 13 route test files under app/api/table, each
a near-identical TableDefinition literal. A required field added to that type
would have failed 13 files individually.

packages/testing already owned createTableColumn and createTableRow but no
definition factory, and — as it turns out — did not export any of them from
the barrel, so they were unreachable from @sim/testing. Adds
createTableDefinition beside them and exports all three.

The fixture type is a structural stand-in rather than TableDefinition itself,
because packages/* must not import from apps/* — the same approach the
existing factories in that file already take.

No assertion changed. Call sites that varied a field pass it as an override;
the four files with several call sites hoist a shared options const and spread
it so each call still gets a fresh object.

* perf(table): project only the job field the row count needs

Both job reads selected the whole payload jsonb, but mapJobRow reads exactly
one number out of it, and only for a running delete. The payload also carries
the delete job's filter and an unbounded excludeRowIds array, and the latest
non-export job is read on essentially every table request — a table that once
ran a large delete would ship that id list on every read, forever.

LatestJobRow.payload becomes doomedCount, extracted in SQL. Both readers share
JOB_PROJECTION so one edit reaches the batch DISTINCT ON and the correlated
subquery alike; the compile-time constraint widens to Column | SQL rather than
being dropped.

Behaviour is identical. `->` keeps the value jsonb, which postgres-js decodes
through its built-in JSON.parse handler, so it arrives as a number with no
boundary coercion. A null payload, a payload without the key, a non-object
payload and an explicit JSON null all collapse to the same `?? 0` the previous
optional chain produced.

Sized honestly before claiming a win: payloads are small in practice today, so
this is defensive rather than impactful — it removes an unbounded growth path,
not a measured cost.

Verified against a real Postgres, not just the mocked driver: the
generated correlated subquery returns doomedCount 12 for a delete job, null
for an import job, and a null row for a table with no job.

Also from the review pass: re-homes the strictWrite explanation onto
rowWriteOptions, where six {@link} references now point; records why
replaceProjectedWireRows carries no keying discriminator; notes the one case
the uniqueness-narrowing invariant does not cover; pins the lax id-wire
passthrough with a test; and renames a parameter that misled once only its
keys were read.

* chore(table): drop two comments the code already says

One restated the identifier below it (checkUniqueConstraintsDb), and one was a
section-divider banner in the factories barrel, which the repo's comment
convention rules out. Everything else that survives is a why the code cannot
express: round-trip rationale, sql.raw input safety, the narrowing invariant
and the one case it does not cover.

* refactor(table): move the internal row routes onto the application boundary (#6809)

* refactor(table): move the single-row route onto the application boundary

The hottest table write path authorized in its own handler and queried the
database from the adapter — the two things a surface adapter must never do. It
now declares itself with defineInternalJsonRoute against the readRow, updateRow
and deleteRow use cases: 127 lines instead of 274, with no db import, no
drizzle import and no checkAccess.

Doing that surfaced why the violation existed. Write-provenance resolution
needs the canonical schema to map a caller's column key to the storage column
it certifies, and the adapter could only do that because it was already loading
the table illegally. The envelope is now split along the real seam: the adapter
reads the header and payload field, which is transport, and the use case
resolves the selections against the canonical table, which is domain.

That split has to preserve a distinction the defaulting logic would erase. An
internal caller that sends no envelope stays deliberately untracked; defaulting
it to an exact-empty stamp would certify "this write introduced no secrets" on
a runtime write that may well have introduced some. Only an interactive caller
certifies exact-empty, over the storage columns its write actually persists.

Two further changes fell out of it:

present() now receives the same { principal, input } pair its sibling hooks
responseHeaders and finalizeResponse already got. This route serves a session
and a workflow execution on one path and owes them different column keyings, so
rendering per caller kind is presentation rather than domain. That was a gap in
the builder, not a special case for this route.

tableRowWireSchema describes what the single-row routes actually return. The
contract claimed a full TableRow, carrying the executions sidecar and Date
objects — true of the list and query routes, and never true here. The hand-
rolled handler was never checked against its own contract, so the drift was
invisible until the builder started validating it.

Wire changes, both deliberate and both narrower than before: a cross-tenant
table now conceals as 404 where the old blanket handler answered 403, while an
in-workspace role denial still answers 403. Nothing in hooks/queries/tables.ts
branches on either.

Verified to fail: forcing one keying, dropping the actor, pre-resolving the
envelope, certifying an untracked internal write, and skipping the bundle
completeness check each turn the covering tests red.

* refactor(table): move the upsert route onto the application boundary

Same shape as the single-row route: declares itself against upsertTableRow,
hands the provenance envelope over unresolved, and derives its column keying
from the principal rather than assuming one.

The keying and presentation helpers the two routes shared are now in row-wire
beside the translators they wrap, so a third route does not restate them.

Two response details changed on purpose. The row now carries `position`, which
the use case always had and this route alone omitted — every other single-row
response already returned it, and the contract now describes one shape instead
of three. The upsert result also carries read-back provenance, which the route
previously assembled for itself.

The surface had no route-level tests; it has six now, covering both caller
keyings, both operations, and the envelope handover.

* refactor(table): move the enrichment-detail route onto the application boundary

The last internal adapter that queried the database for itself. It now runs
through readTableRowEnrichmentDetail, a new use case that shares
tableOperations.readRow — reading a cell's cascade breakdown is a projection of
the same row under the same role, not a second semantic operation.

Its tests move to the same seam and gain one the old suite could not express:
a cross-tenant table now conceals rather than confirming it exists.

* fix(table): mirror the storage rule when keying write provenance

storageKeyByWireKey mapped an unrecognised id-keyed key to null, but
rowDataToStorage persists that key when the caller is not writing strictly. A
cell would have been written with no provenance recorded, under a stamp still
marked complete — the same failure the bundle completeness check exists to
prevent, arriving through the keying map instead of the selection set.

The two wires genuinely differ and the code now says so: the name path drops an
unrecognised key, so it maps to null; the id path stores what it is given, so
every key it sends is a storage key.

Unreachable today, since no delegated surface uses id keying and a session
bundle is refused earlier. Fixed because the function's stated invariant — that
it mirrors how the row data itself is normalized — was not true.

Also corrects the delegated principal fixture in these tests, which used a kind
that is not in the Principal union, so the subject-id branch was never actually
exercised. It is now, and the scope check is asserted to receive the acting
principal's own subject id.

Verified to fail: restoring the schema-based lookup turns the covering test red.

* fix(table): restore executor access to the migrated row routes

The migration swapped checkSessionOrInternalAuth for the delegation policy,
and that broke every Table block call to these endpoints in two ways at once.

The old policy accepted a legacy internal token. The new one requires a
delegation token, which the executor only mints when the tool asks for it —
and none of the four table row tools did, so get/update/delete/upsert row
would each have failed with a 401. The knowledge tools already declare it,
because their routes migrated first.

Even with a valid token the operations denied the caller: readRow, updateRow,
deleteRow and upsertRow ran under a policy whose delegatedServices is
['copilot'], so the executor got a 403. They now use the tool-facing policy
that already existed for the group operations.

Neither was visible to the route tests, which mock the auth policy wholesale —
so the gap is closed at the layer that actually decides: one test pinning that
each tool requests delegation and that its operation admits the executor,
mutation-verified against both failure modes.

Also fixes findings from the review pass: the read surfaces no longer load an
executions sidecar none of them put on the wire (two readers rather than a
flag, so a caller cannot silently read an empty one); the provenance name
index is built once per batch instead of once per row; the uniqueness comment
now names the concurrent-insert race as well as the retro-added constraint;
and the presenter context gets NoInfer plus a note that the v2 builder passes
something different.

* fix(table): keep the lock on a 423 and pin the remaining wire changes

The rows error policy was built on the concealment base rather than the
lock-aware one, so a TableLockedError fell through to the generic handler and
the response lost its `lock` field — the only thing that tells a client which
lock to clear. A row write is exactly as lockable as a group mutation, so it
now shares that base.

Also pins the two wire changes the review found undocumented: a mismatched
workspace assertion answers 404 rather than 400, which is a superset of the
cross-tenant concealment already intended, and an unclassified failure answers
the builder's shared "Internal server error" rather than the old per-route
text. Both are consistent with the ~80 routes already on this builder; they are
asserted so they read as decisions rather than drift.

Verified to fail: reverting the policy base turns the lock test red.

* refactor(table): apply the quality pass

Four parallel reviews (reuse, simplification, efficiency, altitude). The
highest-value finding cut against the branch's own purpose: the rows error
policy sat in the barrel-exported route-policies module, so its import of the
1,200-line row use-case graph was paid by every one of the ~28 table routes
that can never throw a row error — the barrel's import cost went from ~1.1s to
~1.7s. row-route-policies.ts already existed for exactly this and is
deliberately not re-exported; the policy now lives there with its v2 sibling.

The upsert path still loaded an executions sidecar no surface puts on the wire,
and did it inside the write transaction, holding it open for a discarded
result. The read path got that fix earlier; the write path next to it did not.

rowKeyingForPrincipal fell through to name keying for anything that was not a
session. The operation policy admits API-key principals, so the first one to
reach these routes would have had every id-keyed cell dropped and the write
reported as successful. It is now an exhaustive switch over the two kinds the
auth policy yields — which immediately failed three tests using a principal
kind that exists nowhere in the repo, so those fixtures are real now too.

Also: reuses toWireTimestamp and createUnknownTableRowSecretProvenance instead
of re-inlining them; shares one helper for the provenance choice the update and
upsert use cases both make; keeps one canonical actorClientId doc with two
cross-references; merges two maps keyed by the same string into one; stops
round-tripping the principal through the legacy AuthType enum; hoists the
presenter function type out of both conditional branches; drops a subsumed
test; and freezes the shared locks fixture so a mutating test cannot poison its
siblings.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants