fix(v2): close seven correctness and honesty gaps found sweeping the API - #6702

A ten-slice sweep of the live v2 surface found no regression from the recent cancellation work, but did surface a set of pre-existing defects where an endpoint lost data, hid a failure, or reported something untrue. Each is fixed at the layer that owns the behavior.

What's fixed

Force-failed runs vanished from duration queries. The two force-fail boundaries wrote status: 'failed' with no ended_at and no total_duration_ms — the same defect class already closed for cancellation, still open on its sibling. A null duration reads as "no duration recorded" and drops the run out of every minDurationMs/maxDurationMs query. The payload factory is generalized to take the status; the five cancellation call sites are untouched and still emit a byte-identical row (asserted through PgDialect.sqlToQuery).

One bad row made a whole page unreachable. GET /api/v2/custom-tools failed the entire page on a single malformed row, and because the list is keyset-paginated, that row made every page containing it permanently unreachable — two workspaces have had no access for months. The projection now validates against the same contract schema the route builder applies. Repairs are hypotheses that must still pass validation, so a wrong guess can only downgrade a row to skipped, never emit invalid data. Both rows observed in production are recovered rather than discarded; the unrepairable case (function.parameters.type, an open string where a substitution would fabricate JSON-Schema semantics) is skipped with a warning. Single-resource reads still fail loudly — there is no other row to serve.

Table filters silently matched nothing. eq/ne/in/nin compiled a wrongly-typed operand into a containment test that matched nothing, so eq: "8" returned an empty page where eq: 8 returned 1172 rows — indistinguishable from genuinely empty. The write path already coerces through the column-type registry, so this was the read and write paths disagreeing about the same column. The operand is now read through that same registry and rejected only where it refuses. null (a legitimate JSON-null query) and '' (the cleared-cell sentinel) pass through untouched; select is excluded because its operands are already resolved upstream. Error messages now name the caller's column rather than the internal col_… id.

Messages described the wrong problem. A custom error on a string schema also replaced the wrong-type wording, so {"name": 123} reported that the name was missing. Messages now distinguish an omitted field from a mistyped one, topK names its own bounds, the knowledge-search refine reports against a field instead of the whole body, and a workspace id is bounded before it reaches a lookup.

Archived files were listed but unreadable. A soft-deleted file appeared under ?scope=archived yet 404'd on metadata, leaving no way to check share state before restoring. The read takes the same scope selector the list already exposes. The default is unchanged, and the parameter relaxes only the deleted_at predicate — the workspace join, asserted-scope check, per-principal authorization, and concealment policy are all untouched.

Cancellation reported a write that never happened. Cancelling an already-terminal run returned durablyRecorded: true, reason: "recorded". The service now distinguishes the no-op and names the state it observed (already_cancelled / already_completed / already_failed), and both surfaces share one vocabulary instead of the internal route deriving its own. The status read is observational, gates no effect, and falls back to the old report on failure. No claim predicate, WHERE clause, or write changed.

Protocol. A 401 now carries a challenge naming the header the API actually reads (Bearer would be false advertising — v2 doesn't read Authorization; Basic makes browsers prompt for credentials that can't be an API key). A body that failed to parse is reported as 415 only when the caller positively declared a non-JSON type, and only after the read already failed — so curl -d '{...}' without -H, which succeeds today, still succeeds.

Verification

• 25,116 tests pass, type-check clean, 26/26 audits including check:api-validation:strict and check:openapi.
• Every fix has tests proven able to fail — reverted, watched red, restored.
• The 415 guards encode "must not break" and pass either way by construction, so they were mutation-tested against a naive pre-parse implementation instead: all five went red, plus five HEAD tests (a HEAD carries no content-type), independently confirming a pre-parse gate is the wrong shape.

Behavior changes worth review attention

• Table containment filters. A filter returning an empty page today will now either return the rows it always meant, or 400 for an uncoercible operand. A saved view holding a stale predicate (e.g. eq: "abc" after a string→number retype) will now 400 rather than render empty. The range path already carries this exact exposure.
• executionDeadlineAt is now cleared on force-failed rows. This matches the documented column contract; the only DB reader is the stale-execution sweeper, gated on status='running'.
• Cancel reason enum widened. Client-safe: the TS SDK exposes no cancel endpoint, and all three durablyRecorded consumers go through the internal contract, which was already a superset. None branch on a specific reason.

Deliberately not fixed

Reranking returning unavailable (configuration, not code — and rerankerStatus is already reporting the fallback correctly); cancel latency reaching 29s (a real perf issue, but a refactor of ~15 sequential awaits on the executor's terminal path that deserves its own change with measurement); error.details being an object on 403 and an array on 400 (a breaking wire change needing a product decision); validation running before the workspace-key policy check (reordering authz against parsing to fix an information-shape nit is a bad trade).

Second commit: three regressions this branch introduced

Five adversarial reviews of the first commit found that three of its "behavior-preserving" claims were wrong. All are corrected in eb50a58.

Table filters were unsafe in two ways. date is the one column type whose registry coerce is not idempotent — it drops sub-second precision — and the leaf that compiles a filter also builds the unique-constraint and upsert-conflict probes. Re-coercing an already-coerced operand could stop it matching the row it was written from, admitting a duplicate inside the write transaction with no error. Separately, throwing moved failure from submission into the delete/update/dispatch/cancel runners, because the v2 predicate grammar type-checks structure but not operand values — worst case, a filter that cannot compile means the cells it started can no longer be cancelled. Coercion is now total and non-throwing, and date is excluded alongside select. eq: "8" on a number column still resolves to 8; everything else passes through exactly as before.

Reviving a force-failed run inherited its terminal duration. Writing ended_at/total_duration_ms on the force-fail boundary was right in isolation, but a partial resume flips that row back to pending and those columns survived. The preserved value is meant to be the pause checkpoint — the run's active time — and had become wall clock measured at the failed resume, which the checkpoint rule then carried into the next terminal write. The revival clears them only for a row that was terminal, so an ordinary paused row keeps the checkpoint it should.

Cancellation reclassification had two holes, both raised in review and answered in-thread: the guard discarded genuine paused-reconciliation failures for completed/failed priors, and an entry snapshot could not see a claim that lost a race. Reclassification now requires that nothing else went wrong, and the claim's own row count decides whether this cancel terminalized the run. The snapshot also stopped costing an extra query — status rides along on the ownership read, whose own TSDoc warned against the second SELECT we had added.

Also: an unprojectable custom tool now answers consistently (list omits it, GET/PATCH by id report it absent rather than as a server fault), and the workflow_execution_cancelled analytics event no longer fires for a cancel that cancelled nothing.

Correction to the first commit's description

read-workspace-file-metadata.ts's diff is comment-only — includeDeleted was already plumbed through the use case. The three tests there are characterization tests for pre-existing behavior, not evidence that this change authorizes the archived path. The real change for that fix is the contract + route mapping; the authorization argument is unchanged from origin/staging and was verified independently rather than introduced here.

Test hardening

Every fix was mutation-tested — the fix reverted, the suite watched for red. Where the suite stayed green, the assertion was strengthened:

• The absent Content-Type branch had zero coverage: the helper looked like it omitted the header, but undici derives text/plain from a string body. It now sends an encoded body and asserts the header is genuinely absent — this is the branch the source itself calls the likeliest way to turn a working client into a 415.
• The duration encoder was pinned with expect.anything(), which would have accepted endedAt in place of startedAt — silently computing a zero interval. Now pinned to the exact column.
• Execution ownership is pinned to both ids it must match; the mock previously discarded its where argument entirely, and the paused-only branch had no coverage at all.
• The archived-file concealment test passed with the fix reverted; it now proves it conceals the archived read specifically.
• Two tests asserting a paused branch they could not observe were deleted — the rendered-SQL test that can decide it already covers them.

Closed coverage gaps: multipart still succeeds on the body-lifecycle route (the 415 safety argument was prose-only), and a status-read failure logs and continues rather than blocking the cancel.

25,187 tests pass; 26/26 audits; type-check clean.