[High] [Lens 3, 7] — F1 The callback reports an aborted run as COMPLETED: pipeline SUCCESS, COMPLETED webhook, final COMPLETED log

• Location: workers/callback/tasks.py:L338-L344, L397, L1494, L1536, L1783, L1805; reached through backend/workflow_manager/execution/abort.py:L127-L129 and workers/file_processing/tasks.py:L768-L770; contradicted doc at backend/workflow_manager/execution/README.md:L119
• Failure mode: abort a live batch of 10 after 1 file finishes. The batch returns 1 ok and 9 failed, the last decrement fires the callback, and because failed is not equal to total it computes COMPLETED. The backend refuses the write with a 200 skipped, the callback ignores that, and it carries on with COMPLETED: last_run_status=SUCCESS, run count + 1, a COMPLETED webhook and a final COMPLETED log, while the row stays STOPPED with 9 files unprocessed. If no file had finished, the same path sends ERROR and pipeline FAILURE. With the default MAX_PARALLEL_FILE_BATCHES=1 there is no queued batch, so this is what every live abort does.
• Evidence: tasks.py:L342-L344 "Some or all files succeeded ... mark as COMPLETED"; the return of update_workflow_execution_status at L397 is discarded. test_later_completion_cannot_undo_the_abort asserts only the row status. README L119 says the pipeline's last-run status is not changed; README L156-L158 says it can be.
• Suggested fix: in both callback paths, let a STOPPED row override the computed status (context.execution_status, or workflow_execution.get("status") in the API path). The mapping at L127 then gives FAILURE and cleanup still runs. Delete README L119. Billing needs a decision: _track_subscription_usage_if_available commits only on COMPLETED (L1312), so files that really finished in a STOPPED run need an explicit rule.
• Confidence: High. High rather than Critical because the execution row stays correct and nothing is lost, but customers' webhooks receive a false status.
• Verification: VERIFIED — read the status computation, the discarded response and all downstream uses; a mock probe of _determine_execution_status_unified returned COMPLETED for 1 ok, 9 failed. Not run end to end.

[High] [Lens 3, 17] — F2 When a batch is still queued or the worker is dead, the callback never fires: no cleanup, webhook or usage commit, and the pipeline stays INPROGRESS

• Location: workers/queue_backend/pg_barrier.py:L1382-L1393, workers/file_processing/tasks.py:L299-L301, L336
• Made stale by: backend/workflow_manager/execution/abort.py:L127-L129
• Failure mode: a batch picked up after the abort takes the terminal skip and bypasses the barrier decrement, so remaining never reaches 0. A dead worker gives the same result, and that is this feature's main use. Lost: _cleanup_execution_resources (the execution directory with the customer's documents), every webhook, the usage commit for finished files, and the pipeline status, which stays INPROGRESS until the next run. The barrier row waits for the 2.5h stuck timeout, then the reaper deletes it with a warning. The API orchestrator has no terminal guard (workers/api-deployment/tasks.py:L489, unlike workers/general/tasks.py:L304), so an API run aborted while PENDING still arms a barrier and dispatches batches that all skip.
• Evidence: pg_barrier.py:L1383 "the reaper has by definition already torn the barrier down" is now false, and it is the stated reason for the bypass. abort.py has no barrier call. _callback_already_ran (workers/callback/tasks.py:L1391-L1394) says a STOPPED run's first callback must run "incl. resource cleanup". README L156 covers only the opposite case.
• Suggested fix: for STOPPED, omit the skip marker so the decrement runs, after F1 so the callback reports STOPPED. A dead worker needs abort itself to finalize the pipeline status and cleanup, or a written waiver. Correct the three stale comments and the skip log line, and add this to README Limits.
• Confidence: High on the flow, Medium on document retention. A storage lifecycle rule would reduce that part; see open questions.
• Verification: VERIFIED — read the guard, skip result, barrier bypass and reaper; a mock probe of run_batch_with_barrier with the marker set called the decrement 0 times. Not run end to end.

[High] [Lens 6, 3] — F3 Re-running right after an abort can process the in-flight file twice and write to the destination twice

• Location: backend/workflow_manager/execution/abort.py:L127-L141, backend/workflow_manager/internal_views.py:L1364-L1366, L1393, workers/file_processing/tasks.py:L1527
• Failure mode: abort leaves the current file running but marks the run and that file's row STOPPED at once. The active-file check needs both to be PENDING/EXECUTING, so the file now reads as free. Its Redis marker was already removed at pre-create, and file history has no entry until the file finishes. A manual re-run or the next scheduled run picks up the file while the old worker is still on it: double LLM spend and duplicate rows in a DB destination. The destination lock does not help because it is keyed per execution.
• Evidence: internal_views.py:L1364 filters executions on status__in=["PENDING", "EXECUTING"], and L1393 filters file rows the same way. README Limits does not mention this.
• Suggested fix: unsure of the best shape. One option: abort leaves recently touched EXECUTING file rows alone and the check counts them as active regardless of run status, with a staleness bound so a dead worker cannot block forever. At minimum, document it and warn on re-run, with a written waiver.
• Confidence: Medium. A run with an ETL DB destination and a slow file would raise it. High rather than Medium because the damage is silent, reaches customer destinations and follows the natural next action.
• Verification: VERIFIED — read every guard on the path. Not reproduced at runtime.

[High] [Lens 13] — F4 No test stops an org admin or service account aborting another org's run

• Location: backend/workflow_manager/execution/tests/test_abort_executions.py:L175-L186, backend/workflow_manager/execution/abort.py:L89-L95
• Failure mode: _abort_one re-reads by pk with no org filter, so _abortable is the only tenant boundary on this write. If its for_user base were lost, an admin or service account could stop another tenant's runs and the suite would stay green. The code is correct today; the guard is untested on a new mutating endpoint.
• Evidence: grep finds no second org in test_abort_executions.py or stuck_abort_base.py. Replacing visible with WorkflowExecution.objects.all() at abort.py:L90 survives all 13 abort tests (traced).
• Suggested fix: create an org-b workflow and run as test_shared_execution_access.py:L181 does, abort it as an admin and as a service account, and assert not_permitted, status unchanged and no audit row.
• Confidence: High
• Verification: VERIFIED — grep run; mutant traced by reading, because Postgres is unreachable.

[Medium] [Lens 13] — F5 30 of the 33 new backend tests could not run here: Postgres and Redis unreachable

• Location: backend/workflow_manager/execution/tests/stuck_abort_base.py:L32
• Failure mode: nothing in this review shows these tests pass at this head. Their absence masks the stuck filter, stuck_since values, count query budget, abort outcomes and permission matrix, audit row, file-row updates and internal status 404s. CI gives no cover: the test job is skipped while the PR is a draft. The PR body's "216 passed" could not be confirmed. The implementation session reports all 33 passed with Postgres up on the code that went into this head; that is a report from another session, not re-run here.
• Evidence: pytest on the three backend files gave 3 passed, 30 setup errors (connection refused on localhost:5432). workers/tests/test_abort_between_files.py gave 18 passed; those use mocks and need no services.
• Suggested fix: run these files against live Postgres and Redis, or mark the PR ready so integration-backend runs, before relying on them.
• Confidence: High
• Verification: VERIFIED — the pytest run and gh pr checks 2415.

[Low] [Lens 3] — F23 A 200 with no status fails open without a log — workers/file_processing/tasks.py:L703. Warn when the status is None.

[Low] [Lens 13] — F24 The worker tests' mock.Mock() has no spec, so a misnamed argument survives — workers/tests/test_abort_between_files.py:L83. Use create_autospec(InternalAPIClient, instance=True).

[Low] [Lens 13] — F25 "Later of" is half pinned: no case has a run newer than its files — backend/workflow_manager/execution/tests/test_stuck_executions.py:L36-L47. Add that case.

[Low] [Lens 13] — F26 Only stuck=true is checked for stuck_since, not 1 or other case — backend/workflow_manager/execution/tests/test_execution_contract.py:L82-L87. Add both.

[Low] [Lens 13] — F27 Tests rely on the ambient 9000s threshold — backend/workflow_manager/execution/tests/stuck_abort_base.py:L27-L29. Use @override_settings.

[High] [Lens 3, 1] — An aborted run with any finished file is reported as a success

• Location: workers/file_processing/tasks.py:L768-L770
• Failure mode: A user aborts mid-batch after at least one file finished. The batch counts its remaining files as failed and decrements the barrier, so the callback runs. The callback never reads the run's real status and computes COMPLETED, because failed_files differs from total_files. The backend refuses that write, so the run stays STOPPED, but the callback carries on with COMPLETED. The pipeline's last_run_status becomes SUCCESS, webhooks send a success status, and subscribers who asked for failures only get that success payload too. At the default MAX_PARALLEL_FILE_BATCHES=1 the running batch is always the last, so this is the normal case for a mid-run abort. The spec says an aborted run is "marked aborted".
• Evidence: workers/callback/tasks.py:L338-L344 (the else sets COMPLETED), L1494 and L1536 (used for pipeline and notifications), backend/workflow_manager/internal_views.py:L612-L630 (the refused write). README.md:L119 says "The pipeline's last-run status is not changed", and L156-L158 does not say the reported outcome is success. No test covers the callback on this path.
• Suggested fix: In _determine_execution_status_unified, return STOPPED when the fetched run is STOPPED. The pipeline mapping already turns STOPPED into FAILURE, and update_status accepts an equal status, so the file counts get written too. This changes what the callback's usage step does, see the first open question. Correct the README lines.
• Confidence: High
• Verification: VERIFIED — traced by reading from _stop_remaining_files through the callback, the pipeline endpoint and the notification payload. Not run end to end.

[High] [Lens 3, 6, 16, 1] — When a worker is dead or a batch is still queued, the callback never runs for an aborted run

• Location: backend/workflow_manager/execution/abort.py:L127-L130
• Failure mode: Abort sets STOPPED and leaves the barrier row in place. A batch picked up later hits the terminal check and returns without decrementing, and a dead worker never decrements at all. So remaining never reaches 0, the callback never fires, and the reaper later deletes only the barrier row. This covers the feature's main target, a stuck run. Only the callback does these things, so they are lost:
  • Directory cleanup: the execution directory, which holds an API run's uploaded documents, stays in storage.
  • Usage tracking: the callback's usage step does not run for finished files.
  • Webhooks: subscribers hear nothing.
  • Pipeline status: stays INPROGRESS until the next run.
• Evidence: workers/queue_backend/pg_barrier.py:L1382-L1393 returns before _barrier_pg_decrement. _cleanup_execution_resources is called only from workers/callback/tasks.py:L1512 and L1779. The spec says abort "releases queued work for that execution", and README Limits covers only the case where the callback does run. Comments that this makes false: pg_barrier.py:L1310-L1312 and L1383-L1388 ("the reaper has by definition already torn the barrier down"), workers/file_processing/tasks.py:L299-L301. The skip's log at tasks.py:L336-L339 ("stale/redelivered after reaper-recovery") will mislead on-call. The reaper's ERROR path has the same gap today; this PR gives users a direct route into it.
• Suggested fix: Decide what finalizes a stopped run. A skip caused by STOPPED could still decrement, so the last batch fires the callback. A dead-worker run never decrements, so the reaper's already-terminal branch also has to trigger cleanup for STOPPED runs. I am unsure which split is simplest; check it against the barrier's zero-row path. Then correct the comments and the README.
• Confidence: High on the flow, Medium on how long storage lasts (see open questions).
• Verification: VERIFIED — read _run_batch_stages, run_batch_with_barrier, _recover_one_barrier, and the callback. Not run on a multi-batch execution.

[High] [Lens 13] — No test covers abort's tenant boundary

• Location: backend/workflow_manager/execution/tests/test_abort_executions.py:L182
• Failure mode: for_user at abort.py:L90 is the only org scoping on this path, because _abort_one re-reads the row by id alone. If it regresses, an admin or service account in one org can stop another org's runs with a green suite. Today's code is correct. Separately, test_other_org_unknown_and_malformed_ids_are_404 uses "some-other-org", which does not exist, so it never tests a real second org.
• Evidence: With visible = WorkflowExecution.objects.all(), all 33 tests pass. A probe with a real second org then gives aborted; the real code answers not_permitted.
• Suggested fix: Create a run in a second real org. Assert not_permitted and an unchanged status for the admin and the service account, and use that org in the status 404 case.
• Confidence: High
• Verification: VERIFIED — mutant in a scratch copy, plus a two-org probe on real and mutated code.

[High] [Lens 13] — The abort row lock and per-id transaction cannot fail any test

• Location: backend/workflow_manager/execution/tests/stuck_abort_base.py:L32
• Failure mode: Everything runs inside TestCase's transaction. Remove the lock and a run that finishes during the abort is overwritten and reported as aborted. Remove atomic() and every abort returns 500 in production. Both pass all 33 tests. No test aborts twice either, though README.md:L140 promises retries are safe.
• Evidence: Probes on real connections: with a writer committing COMPLETED mid-abort, real code gives already_finished and the unlocked mutant gives aborted. In autocommit, real code gives 200 and the mutant gives TransactionManagementError. The pattern to copy is in test_update_status_locking.py (_lock_statements, ConcurrentStatusWriteTests).
• Suggested fix: Assert one FOR NO KEY UPDATE with CaptureQueriesContext. Add a TransactionTestCase for the race and an autocommit abort, and an abort-twice case.
• Confidence: High
• Verification: VERIFIED — mutants plus concurrent and autocommit probes.

[High] [Lens 13] — Nothing pins the status route, and the worker fails open if it breaks

• Location: backend/workflow_manager/execution/tests/test_execution_contract.py:L104
• Failure mode: The tests bind the view directly, so the router is never involved, and the worker test pins only its own URL string. If the route breaks, every status check gets a 404 and every file is processed. The abort API still answers aborted, so the stop between files silently does nothing, with only a worker warning. The route works today.
• Evidence: Renaming url_path="status" or removing the @action at internal_views.py:L520 passes all 33 backend tests. A real request through the middleware returned 200 for the right org and 404 for another.
• Suggested fix: Assert resolve(f"/internal/v1/workflow-execution/{id}/status/").func.actions == {"get": "execution_status"}, or send one request through the test client.
• Confidence: High
• Verification: VERIFIED — mutants in a scratch copy, plus a real-stack probe.

[Low] [Lens 7] — stuck_since is in no schema component

• Location: backend/workflow_manager/execution/views/execution.py:L40-L54
• The spec asks for a typed contract, but at schema time get_serializer_class returns the plain serializer, so the field exists only in prose. Nothing generates types from these routes today. Simplest fix: say the README is the contract.
• Verification: VERIFIED — generated the schema; git grep on main found no generator.

[Low] [Lens 7] — stuck-count.json and abort.json return 500

• Location: backend/workflow_manager/execution/urls.py:L32-L37
• format_suffix_patterns passes format= to handlers that do not accept it, and the TypeError fires before any abort runs. Move these two paths outside the wrapper, or add the kwarg.
• Verification: VERIFIED — ran it for stuck-count; abort has the same signature and was read only.

[Low] [Lens 3, 13] — The except in _mark_file_stopped never fires on a real failure

• Location: workers/file_processing/tasks.py:L707-L718
• The real client catches errors and returns APIResponse(success=False), and the return value is ignored. The test at workers/tests/test_abort_between_files.py:L110 raises a RuntimeError the client cannot produce, and its unspecced mock would not catch a renamed kwarg. Check response.success, log the file id, and use create_autospec.
• Verification: VERIFIED — read the client; mutant for the rename.

[Low] [Lens 3] — The status check is silent on a missing status and catches programming errors

• Location: workers/file_processing/tasks.py:L695-L704
• A 2xx with an unusable body reads as "not stopped" with no log, unlike _raise_if_execution_terminal in the same file. except Exception also hides bugs behind a per-file warning. Log missing or unknown values and narrow the catch to InternalAPIClientError.
• Verification: VERIFIED — read _make_request and _parse_response.

[Low] [Lens 13] — Small test pins

• Location: backend/workflow_manager/execution/tests/stuck_abort_base.py:L27-L29, backend/workflow_manager/execution/tests/test_abort_executions.py:L195
• Running with STUCK_EXECUTION_THRESHOLD_SECONDS=30 fails 3 tests; pin it with override_settings. The 100-id limit test derives its limit from the constant, so MAX_ABORT_IDS = 5 passes while the README says 100.
• Verification: VERIFIED — ran both in a scratch copy.