The comment block sat where #2176 reworded the e2e-login comment and inserted
a new group, creating a textual merge conflict against main. Move it inside
each group body; the optional: true removals merge cleanly on their own.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P4tU5vyB4G6BgfAnGqjxLs

…2170)

* UN-3636 [DEV] add hermetic LLM mock hook for execute-path tests

Execute-path critical paths (workflow-create-execute, usage-token-tracking,
etc.) need a deterministic LLM without real provider secrets. Add a single
sdk1 escape hatch: when UNSTRACT_LLM_MOCK_RESPONSE is set, inject litellm's
mock_response so completions return that string with fixed usage (10/20/30),
tunable via DEFAULT_MOCK_RESPONSE_* and error sentinels like
"litellm.RateLimitError". No-op in production (env unset).

Wire the env through worker-executor-v2 / worker-file-processing-v2 in the e2e
compose overlay so the mock reaches the process that runs the LLM.

Covered by unit tests pinning both our injector and the litellm mock contract
(string + deterministic usage + error sentinel) so a litellm bump can't
silently break hermetic execute-path coverage.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU

* UN-3636 [DEV] cover execute-path critical tests via mocked LLM

Add e2e tests that exercise the full execute path hermetically using the
UNSTRACT_LLM_MOCK_RESPONSE hook (no real LLM/secret):

- api-deployment-run + usage-token-tracking: deploy a workflow as an API,
  POST a doc synchronously, assert the mocked answer and fixed 10/20/30
  usage come back as structured JSON.
- workflow-create-execute: two-step /workflow/execute/ then poll to
  COMPLETED; COMPLETED + a successful file is the hermetic proof (without
  the mock the completion would fail).

A shared provisioned_workflow fixture stands up a Prompt Studio-backed API
workflow (NoOp x2text/vectordb, chunk_size=0 so embedding/vectordb are
skipped). The rig seeds a canonical mock string in ComposeRuntime so both
the workers and pytest see the same value; tests skip if it is unset.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU

* UN-3636 [MISC] Fix lint, promote e2e groups, document LLM mock

- test_mock_response.py: split the module docstring summary line (D205 was
  failing pre-commit.ci, the only red hook on the PR).
- groups.yaml: drop `optional: true` from e2e-workflow / e2e-api-deployment.
  They now hold real execute-path tests rather than placeholders, so a red one
  should fail the build directly instead of only surfacing as a critical-path
  gap via --fail-on-critical-gap.
- README: the workflows/ and api_deployment/ dirs are no longer "(future)", and
  document UNSTRACT_LLM_MOCK_RESPONSE under E2E runtime — including that only
  completions are mocked (embedding.py is unhooked; chunk_size=0 is what keeps
  indexing off the real provider).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P4tU5vyB4G6BgfAnGqjxLs

* UN-3636 [MISC] Reposition groups.yaml comment to avoid main conflict

The comment block sat where #2176 reworded the e2e-login comment and inserted
a new group, creating a textual merge conflict against main. Move it inside
each group body; the optional: true removals merge cleanly on their own.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P4tU5vyB4G6BgfAnGqjxLs

* UN-3636 [MISC] Tighten comments to state why, not how

Drop the restated-critical-path and mechanism-narrating comments across the
LLM mock and the execute-path e2e tests; keep only the constraints the code
can't show (litellm's fixed usage, the mandatory filter arg, the two-call
execute contract). Also drop the groups.yaml 'not optional' comment: the field
is self-evident and would go stale.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P4tU5vyB4G6BgfAnGqjxLs

* UN-3636 [DEV] Warn on active LLM mock, skip groups whose deps failed

Three follow-ups from review:

- llm.py: warn once per process when UNSTRACT_LLM_MOCK_RESPONSE is active. The
  hatch was silent, so a stray env var in production would fake every
  completion and its billing with nothing in the logs.

- rig: implement the runtime-gate-skip TODO in cmd_run. Groups run in topo
  order, so a dep's result is known before its dependents start; a red dep
  means the precondition it gates does not hold and dependents would only
  produce noise against a half-up stack. Adds GroupManifest.transitive_deps and
  a 'blocked' GroupResult status which is never green, so a blocked group
  attests no coverage and --fail-on-critical-gap still catches a path that
  quietly stopped being covered. This mattered less while every platform
  dependent was optional; it does now that the execute-path groups gate.

- api-deployment e2e: the 10/20/30 usage assert now explains itself, since
  counts are summed per reason and a multiple means extra completions rather
  than broken tracking.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P4tU5vyB4G6BgfAnGqjxLs

* UN-3636 [DEV] Cover the last four OSS critical paths

Closes the remaining gaps in the registry; every path is now proof: marker.

- callback-result-delivery: async API execution (timeout=0). With a file to
  process, COMPLETED is written only by the chord callback, so a polled result
  proves the callback worker ran. Pins the one-shot read too — fetching a
  delivered result acknowledges it and the next read 406s, which a retry loop
  added later would otherwise swallow silently.

- workflow-execution-fan-out: multi-file execution, one batch per file. Needed
  MAX_PARALLEL_FILE_BATCHES on the *backend* (workers ask it for the value and
  only fall back to their own env); at the default of 1 the files share a batch
  and run serially, so the test would have passed proving nothing. File bodies
  differ because identical ones are deduplicated by hash on ingest.

- prompt-studio-fetch-response: the authoring surface. Async, so the answer is
  polled from prompt-output/ rather than the task status, which reports the
  executor finishing and not the callback having stored anything. Upload is
  content-sniffed and OSS takes PDF only, hence the real (if empty) PDF.

- pipeline-etl-execute: source connector to destination, over MinIO — the only
  storage connector the compose stack both boots and registers. Reuses the
  exported tool but needs its own workflow, since endpoints are per-workflow
  and the existing one is committed to the API deployment tests.

None of these have run: the e2e job is skipped while the PR is a draft. Locally
verified only as far as it goes here — rig validate, rig self-tests, ruff, and
pytest collection.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P4tU5vyB4G6BgfAnGqjxLs

* UN-3636 [FIX] make api-deployment e2e poll instead of blocking, install minio

The e2e tier failed on two counts:

- api-deployment run/fan-out asserted COMPLETED on the synchronous POST, but a
  file-bearing execution only reaches COMPLETED once the chord callback rejoins
  — the sync path reports EXECUTING under CI load. All three api-deployment
  tests now dispatch async (timeout=0) and poll the status endpoint via shared
  dispatch_async/poll_delivered helpers, tolerating transient poll timeouts
  (the async test's earlier ReadTimeout). execution_id is parsed from status_api
  since the async PENDING body omits it as a top-level field.

- e2e-etl skipped ("minio client not installed"), leaving pipeline-etl-execute a
  gap. Add minio to the rig's injected pytest deps so the test runs.

Verified against a full local stack: api_deployment (3), workflow, etl,
prompt_studio all green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU

* UN-3636 [MISC] Drop verbose minio comment, make MinIO scheme env-driven

Clears the SonarCloud S5332 clear-text-HTTP finding on the test fixture
(compose-internal MinIO has no TLS) by sourcing the scheme from env, and
trims the low-value minio dependency comment.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU

* UN-3636 [MISC] Fail fast when no workflow endpoint matches

Assert at least one endpoint was patched to API in the provisioning
fixture, so a field-name mismatch surfaces here instead of as a confusing
downstream execute failure (CodeRabbit).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU

* UN-3636 [MISC] Trim compose overlay comments to WHY only

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU

* UN-3636 [MISC] Cache e2e image build via type=gha (bake)

The e2e job runs on ephemeral GitHub-hosted runners, so a plain
`docker compose build` rebuilt every image cold each run (~3min). Switch
to docker/bake-action with per-service type=gha cache scopes matching the
release workflow, so each runner restores layers from the shared GHA cache
(including entries the release build already wrote). load=true exports the
images into the daemon for the compose stack to boot.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU

* UN-3636 [MISC] Pin buildx/bake actions to commit SHA (Sonar S7637)

SonarCloud flagged the two floating action tags added for the e2e gha
cache as new MAJOR vulnerabilities (security rating C). Pin to the commit
SHA the tag points to so a moved tag can't inject code.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU

* UN-3636 [DEV] Skip e2e on frontend-only and docs-only changes

e2e ran on every non-draft PR regardless of paths, so image-only or
docs-only changes triggered a full ~7min platform boot for nothing.

Add an `e2e_relevant` path scope that skips frontend/** and docs/**
while keeping docker/** in scope (image changes must still be
e2e-tested), gate the e2e job on it, and accept a skipped e2e as a
pass in the report gate (matching how a skipped test tier is handled).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L549habQ6K2m1ACJ7DwtsE

* UN-3636 [FIX] Fail CI gate when the changes job fails

`report` needs `changes` but never consulted its result. A failed filter job
(paths-filter error, checkout failure, runner death) skips `test`, which the
gate reads as a legitimate path-filter skip and reports success with no unit
or integration tests having run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU

* UN-3636 [MISC] Address PR review feedback on rig and e2e tests

Rig:
- Blocked groups now persist a synthetic junit carrying `rig-blocked-by`, so
  the cascade survives `rig report`, which rebuilds results from junit alone.
  Without it a blocked group parsed as a zero-count pass and falsely attested
  critical-path coverage.
- GroupResult rejects a blocked row with non-zero counters.
- Report statuses are module constants instead of scattered literals.
- An exit-5 (no tests collected) group that others depend on now counts as
  failed, so dependents are blocked rather than run against nothing.
- The LLM mock default moves to cmd_run so it applies to every runtime.
- groups.py docstring no longer claims runtime gating is unimplemented.

E2E:
- Shared `wait_for_execution` helper replaces three near-identical polls; a
  CsrfSession subclass stamps X-CSRFToken so per-call helpers can't drift.
- api_deployment poll: sleeps on every iteration, captures exceptions as
  evidence, treats 406 as a lost-200 acknowledge rather than a poll assert,
  and surfaces non-422 statuses as themselves. Timeout drops to 150s so the
  group's three tests fit one 600s budget.
- MinIO scheme drives both the workers' URL and the client's TLS flag.
- Missing rig-provisioned prerequisites fail instead of silently skipping.

Core/SDK:
- `ExecutionStatus.terminal_statuses()` names the terminal set.
- Mock-response tests cover the empty-string no-op and assert every
  completion path still calls the injection hook.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU

* UN-3636 [FIX] Keep frontend in e2e scope

The e2e image build is the only CI check that runs `bun install` and
`bun run build`; ci-frontend-lint only runs biome. Excluding frontend/**
from e2e_relevant let a broken build or a desynced bun.lock merge green
and surface on main via production-build.yaml.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU

* UN-3636 [FEAT] Prove fan-out by timing, not by assuming it

The fan-out test asserted only the rejoin: every assertion held identically
for three files processed serially in one batch, while critical_paths.yaml
retired the workflow-execution-fan-out gap on its strength.

Nothing distinguishes a batch — the batch index is a discarded loop local and
the celery task id reaches only worker stdout — so assert the timing instead.
WorkflowFileExecution rows are created by the worker owning the batch rather
than up front by the dispatcher, so fanned out each row's window is one delay
wide, while serialised into one batch the windows open together and grow by a
delay per file. Comparing durations rather than overlap is the point:
serialised rows are created in one tight loop and overlap too.

That needs each completion to cost a known, measurable amount, so add
UNSTRACT_LLM_MOCK_DELAY alongside the existing mock hatch. It is inert without
UNSTRACT_LLM_MOCK_RESPONSE, since litellm ignores mock_delay on a real
completion, and a non-numeric value degrades to no delay rather than breaking
every completion in the worker.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU

* UN-3636 [FIX] Close the two remaining review holes

Manual runs reported green with nothing tested. paths-filter has no base to
diff on workflow_dispatch, so it resolves against the default branch and a
manual run on main diffs main against itself: every filter false, every tier
skipped, gate green. Filtering is a PR-time saving, so force both scopes true
off-PR and skip the filter step entirely there.

Gate the LLM mock on ENVIRONMENT as well as the mock var, so a stray variable
in a real deployment can't fake completions and their billing. Production sets
neither, so this fails closed. `development` is allowed alongside `test`
because that is what base compose sets on the two workers that run the
injection -- proven by the mock working today with the var set on exactly
those two services -- and the overlay now pins ENVIRONMENT=test on them so the
tier can't lose its mock to a base-compose edit. A refusal warns rather than
passing silently: setting the var at all means someone expected mocking.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU

* UN-3636 [FIX] Capture compose logs before the rig tears the stack down

The rig's `down -v` destroys the containers, so the workflow step that ran
`docker compose logs` afterwards always wrote an empty file — and overwrote
nothing useful only because there was nothing else to write. Capture inside
`ComposeRuntime.down()` while the containers still exist, and drop the step.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* UN-3636 [FIX] Give the fan-out timing check room above CI jitter

The bound was half a delay — a 1s window on per-file durations of ~12s, which
measures runner contention as much as batching. One delay is the midpoint
between the two outcomes the check separates (~0 spread fanned out, ~2 delays
serialised), and the delay itself has to be large enough for that midpoint to
sit above the noise.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* UN-3636 [MISC] Clear SonarCloud float equality and duplicate literal

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* UN-3636 [FIX] Record fan-out as a gap instead of proving it by timing

The worker log settles what the test could not: `Arranging 3 files into 3
batches` — the fan-out happens. Yet the per-file durations it was judged by came
out 6.6 / 18.1 / 22.9. Three tool containers running at once on a CI runner cost
more in contention than they gain in overlap, so genuine fan-out lands further
apart than serialised files do, and no threshold separates the two. Raising the
delay does not help: the contention scales with the parallelism being measured.

So the proof is removed rather than retuned, and with it the mock delay that
existed only to feed it -- every mocked completion across the whole e2e tier was
stalling for it. `workflow-execution-fan-out` becomes a declared gap (visible in
the report, non-gating), and the test keeps asserting the rejoin, which is
directly observable. Closing the gap wants a batch id on WorkflowFileExecution,
which is execution observability worth having on its own.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>