UN-3636 [DEV] Hermetic LLM mock hook for execute-path critical tests#2170
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe change adds deterministic LLM mocking, shared hermetic E2E provisioning, API deployment and ETL execution tests, expanded critical-path coverage, CI image caching, and dependency-aware test-group blocking and reporting. ChangesHermetic E2E coverage
Dependency-aware group execution
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant E2ETests
participant ProvisionedWorkflow
participant API
participant Workers
participant LLM
E2ETests->>ProvisionedWorkflow: provision mocked workflow
ProvisionedWorkflow->>API: create and configure workflow
E2ETests->>API: dispatch execution
API->>Workers: process execution
Workers->>LLM: request completion with mock_response
LLM-->>Workers: return canned answer
Workers-->>API: update execution result
API-->>E2ETests: deliver completed result
Suggested reviewers: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Execute-path critical paths (workflow-create-execute, usage-token-tracking, etc.) need a deterministic LLM without real provider secrets. Add a single sdk1 escape hatch: when UNSTRACT_LLM_MOCK_RESPONSE is set, inject litellm's mock_response so completions return that string with fixed usage (10/20/30), tunable via DEFAULT_MOCK_RESPONSE_* and error sentinels like "litellm.RateLimitError". No-op in production (env unset). Wire the env through worker-executor-v2 / worker-file-processing-v2 in the e2e compose overlay so the mock reaches the process that runs the LLM. Covered by unit tests pinning both our injector and the litellm mock contract (string + deterministic usage + error sentinel) so a litellm bump can't silently break hermetic execute-path coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
f26d44c to
fc0d056
Compare
Add e2e tests that exercise the full execute path hermetically using the UNSTRACT_LLM_MOCK_RESPONSE hook (no real LLM/secret): - api-deployment-run + usage-token-tracking: deploy a workflow as an API, POST a doc synchronously, assert the mocked answer and fixed 10/20/30 usage come back as structured JSON. - workflow-create-execute: two-step /workflow/execute/ then poll to COMPLETED; COMPLETED + a successful file is the hermetic proof (without the mock the completion would fail). A shared provisioned_workflow fixture stands up a Prompt Studio-backed API workflow (NoOp x2text/vectordb, chunk_size=0 so embedding/vectordb are skipped). The rig seeds a canonical mock string in ComposeRuntime so both the workers and pytest see the same value; tests skip if it is unset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
- test_mock_response.py: split the module docstring summary line (D205 was failing pre-commit.ci, the only red hook on the PR). - groups.yaml: drop `optional: true` from e2e-workflow / e2e-api-deployment. They now hold real execute-path tests rather than placeholders, so a red one should fail the build directly instead of only surfacing as a critical-path gap via --fail-on-critical-gap. - README: the workflows/ and api_deployment/ dirs are no longer "(future)", and document UNSTRACT_LLM_MOCK_RESPONSE under E2E runtime — including that only completions are mocked (embedding.py is unhooked; chunk_size=0 is what keeps indexing off the real provider). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P4tU5vyB4G6BgfAnGqjxLs
The comment block sat where #2176 reworded the e2e-login comment and inserted a new group, creating a textual merge conflict against main. Move it inside each group body; the optional: true removals merge cleanly on their own. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P4tU5vyB4G6BgfAnGqjxLs
Drop the restated-critical-path and mechanism-narrating comments across the LLM mock and the execute-path e2e tests; keep only the constraints the code can't show (litellm's fixed usage, the mandatory filter arg, the two-call execute contract). Also drop the groups.yaml 'not optional' comment: the field is self-evident and would go stale. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P4tU5vyB4G6BgfAnGqjxLs
Three follow-ups from review: - llm.py: warn once per process when UNSTRACT_LLM_MOCK_RESPONSE is active. The hatch was silent, so a stray env var in production would fake every completion and its billing with nothing in the logs. - rig: implement the runtime-gate-skip TODO in cmd_run. Groups run in topo order, so a dep's result is known before its dependents start; a red dep means the precondition it gates does not hold and dependents would only produce noise against a half-up stack. Adds GroupManifest.transitive_deps and a 'blocked' GroupResult status which is never green, so a blocked group attests no coverage and --fail-on-critical-gap still catches a path that quietly stopped being covered. This mattered less while every platform dependent was optional; it does now that the execute-path groups gate. - api-deployment e2e: the 10/20/30 usage assert now explains itself, since counts are summed per reason and a multiple means extra completions rather than broken tracking. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P4tU5vyB4G6BgfAnGqjxLs
Closes the remaining gaps in the registry; every path is now proof: marker. - callback-result-delivery: async API execution (timeout=0). With a file to process, COMPLETED is written only by the chord callback, so a polled result proves the callback worker ran. Pins the one-shot read too — fetching a delivered result acknowledges it and the next read 406s, which a retry loop added later would otherwise swallow silently. - workflow-execution-fan-out: multi-file execution, one batch per file. Needed MAX_PARALLEL_FILE_BATCHES on the *backend* (workers ask it for the value and only fall back to their own env); at the default of 1 the files share a batch and run serially, so the test would have passed proving nothing. File bodies differ because identical ones are deduplicated by hash on ingest. - prompt-studio-fetch-response: the authoring surface. Async, so the answer is polled from prompt-output/ rather than the task status, which reports the executor finishing and not the callback having stored anything. Upload is content-sniffed and OSS takes PDF only, hence the real (if empty) PDF. - pipeline-etl-execute: source connector to destination, over MinIO — the only storage connector the compose stack both boots and registers. Reuses the exported tool but needs its own workflow, since endpoints are per-workflow and the existing one is committed to the API deployment tests. None of these have run: the e2e job is skipped while the PR is a draft. Locally verified only as far as it goes here — rig validate, rig self-tests, ruff, and pytest collection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P4tU5vyB4G6BgfAnGqjxLs
|
| Filename | Overview |
|---|---|
| unstract/sdk1/src/unstract/sdk1/llm.py | Adds the environment-gated mock response hook around LiteLLM completion calls. |
| tests/compose/docker-compose.test.yaml | Forwards the mock response setting into the worker services used by execute-path tests. |
| tests/e2e/conftest.py | Adds shared workflow provisioning and mock-response fixtures for execute-path tests. |
| tests/e2e/etl/conftest.py | Adds MinIO-backed ETL workflow provisioning for filesystem source and destination coverage. |
| tests/rig/cli.py | Adds dependency-blocked group handling and mock response setup for platform runs. |
| tests/rig/reporting.py | Adds explicit blocked-group status handling in rig reports and summaries. |
Reviews (14): Last reviewed commit: "Merge branch 'main' into feat/un-3636-ex..." | Re-trigger Greptile
…ll minio
The e2e tier failed on two counts:
- api-deployment run/fan-out asserted COMPLETED on the synchronous POST, but a
file-bearing execution only reaches COMPLETED once the chord callback rejoins
— the sync path reports EXECUTING under CI load. All three api-deployment
tests now dispatch async (timeout=0) and poll the status endpoint via shared
dispatch_async/poll_delivered helpers, tolerating transient poll timeouts
(the async test's earlier ReadTimeout). execution_id is parsed from status_api
since the async PENDING body omits it as a top-level field.
- e2e-etl skipped ("minio client not installed"), leaving pipeline-etl-execute a
gap. Add minio to the rig's injected pytest deps so the test runs.
Verified against a full local stack: api_deployment (3), workflow, etl,
prompt_studio all green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
Clears the SonarCloud S5332 clear-text-HTTP finding on the test fixture (compose-internal MinIO has no TLS) by sourcing the scheme from env, and trims the low-value minio dependency comment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (3)
tests/e2e/api_deployment/conftest.py (1)
84-96: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winNo backoff on
ConnectionError/Timeoutretries.
except _TRANSIENT: continue(Lines 89-90) retries immediately with no sleep, unlike the other branches. ATimeoutis naturally throttled by the 30s request timeout, but a fast-failingConnectionError(e.g., connection reset) would busy-loop against the status endpoint for up to_POLL_TIMEOUT_SECONDS.Proposed fix
except _TRANSIENT: + time.sleep(2) continue🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/e2e/api_deployment/conftest.py` around lines 84 - 96, Update the polling retry logic around the transient exception handler in the status-wait function to sleep for the existing polling interval before continuing. Preserve the current handling for successful responses and non-transient failures, while ensuring fast-failing ConnectionError retries do not busy-loop.tests/e2e/conftest.py (1)
116-131: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winCSRF-wrapping
_post/_patchhelpers are reimplemented per file. Both sites attachX-CSRFTokenfrom the session cookie and POST/PATCH with a fixed timeout — one root cause (no shared e2e HTTP helper), duplicated per file.
tests/e2e/conftest.py#L116-L131: keep as the canonical_post/_patch(accepting arequests.Session); consider exporting them for reuse instead of re-deriving per-module wrappers.tests/e2e/prompt_studio/test_prompt_studio_fetch_response.py#L101-L103: replace the local_postwith a thin call intotests/e2e/conftest.py's_post(pw.session, ...)instead of reimplementing the CSRF-attach logic.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/e2e/conftest.py` around lines 116 - 131, The CSRF-aware HTTP helpers are duplicated across e2e files. Keep tests/e2e/conftest.py lines 116-131 as the canonical _post and _patch implementations, making them reusable if needed; in tests/e2e/prompt_studio/test_prompt_studio_fetch_response.py lines 101-103, remove the local _post wrapper and delegate to conftest.py’s _post using pw.session, preserving the existing request arguments and timeout behavior.tests/rig/cli.py (1)
449-457: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider extracting the failure predicate to avoid 3-way drift.
"Did this group fail" is now computed independently in three places with slightly different
optionalgating (cascade tracking at Line 478-481 ignoresoptional; the twooverall_exitfolds at Line 493-498 and 502-508 respect it). A future tweak to failure criteria (e.g. new non-failing exit code) risks updating only some sites.♻️ Suggested consolidation
+def _group_failed(exit_code: int, result: GroupResult | None) -> bool: + return exit_code not in _NON_FAILING_PYTEST_EXIT_CODES or ( + result is not None and (result.errors or result.failed) + ) + if result is not None: group_results.append(result) # Tracked regardless of `optional` — see the cascade note above. - if exit_code not in _NON_FAILING_PYTEST_EXIT_CODES or ( - result is not None and (result.errors or result.failed) - ): + if _group_failed(exit_code, result): failed_groups.add(name) ... - if ( - exit_code not in _NON_FAILING_PYTEST_EXIT_CODES - and not group.optional - and overall_exit == 0 - ): + if _group_failed(exit_code, None) and not group.optional and overall_exit == 0: overall_exit = exit_code - if ( - result is not None - and (result.errors or result.failed) - and not group.optional - and overall_exit == 0 - ): + if _group_failed(0, result) and not group.optional and overall_exit == 0: overall_exit = 1Also applies to: 477-481, 493-508
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/rig/cli.py` around lines 449 - 457, Extract a shared failure predicate for group results and use it consistently in the runnable-group cascade tracking and both overall_exit folds. Ensure the predicate applies the same optional-group gating everywhere, replacing the duplicated exit-code checks while preserving blocked-result handling.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/e2e/conftest.py`:
- Around line 250-263: Track whether the workflow endpoint setup loop in the
fixture successfully patches at least one matching endpoint to API, and assert
that this occurred after iterating through eps. Keep the existing matching and
patch status assertion, but fail immediately with a clear fixture-level message
when eps is empty or no endpoint matches workflow_id.
---
Nitpick comments:
In `@tests/e2e/api_deployment/conftest.py`:
- Around line 84-96: Update the polling retry logic around the transient
exception handler in the status-wait function to sleep for the existing polling
interval before continuing. Preserve the current handling for successful
responses and non-transient failures, while ensuring fast-failing
ConnectionError retries do not busy-loop.
In `@tests/e2e/conftest.py`:
- Around line 116-131: The CSRF-aware HTTP helpers are duplicated across e2e
files. Keep tests/e2e/conftest.py lines 116-131 as the canonical _post and
_patch implementations, making them reusable if needed; in
tests/e2e/prompt_studio/test_prompt_studio_fetch_response.py lines 101-103,
remove the local _post wrapper and delegate to conftest.py’s _post using
pw.session, preserving the existing request arguments and timeout behavior.
In `@tests/rig/cli.py`:
- Around line 449-457: Extract a shared failure predicate for group results and
use it consistently in the runnable-group cascade tracking and both overall_exit
folds. Ensure the predicate applies the same optional-group gating everywhere,
replacing the duplicated exit-code checks while preserving blocked-result
handling.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 13ad4ba2-a18b-4cb3-b034-91f3430b5f52
📒 Files selected for processing (25)
tests/README.mdtests/compose/docker-compose.test.yamltests/critical_paths.yamltests/e2e/api_deployment/__init__.pytests/e2e/api_deployment/conftest.pytests/e2e/api_deployment/test_api_deployment_async.pytests/e2e/api_deployment/test_api_deployment_fan_out.pytests/e2e/api_deployment/test_api_deployment_run.pytests/e2e/conftest.pytests/e2e/etl/__init__.pytests/e2e/etl/conftest.pytests/e2e/etl/test_pipeline_etl_execute.pytests/e2e/prompt_studio/__init__.pytests/e2e/prompt_studio/test_prompt_studio_fetch_response.pytests/e2e/workflows/__init__.pytests/e2e/workflows/test_workflow_execute.pytests/groups.yamltests/rig/cli.pytests/rig/groups.pytests/rig/reporting.pytests/rig/runtime.pytests/rig/tests/test_groups.pytests/rig/tests/test_reporting.pyunstract/sdk1/src/unstract/sdk1/llm.pyunstract/sdk1/tests/test_mock_response.py
Assert at least one endpoint was patched to API in the provisioning fixture, so a field-name mismatch surfaces here instead of as a confusing downstream execute failure (CodeRabbit). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
The e2e job runs on ephemeral GitHub-hosted runners, so a plain `docker compose build` rebuilt every image cold each run (~3min). Switch to docker/bake-action with per-service type=gha cache scopes matching the release workflow, so each runner restores layers from the shared GHA cache (including entries the release build already wrote). load=true exports the images into the daemon for the compose stack to boot. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
There was a problem hiding this comment.
🧹 Nitpick comments (1)
.github/workflows/ci-test.yaml (1)
171-178: 🔒 Security & Privacy | 🔵 Trivial | 💤 Low valuePin GitHub Actions to specific commit SHAs.
The newly added
setup-buildx-actionandbake-actionuse mutable version tags (@v4and@v7). Consider pinning them to specific commit SHAs to maintain consistency with thecheckout,login-action, andsetup-uvsteps earlier in this workflow and to strengthen supply chain security against upstream tag hijacking.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/ci-test.yaml around lines 171 - 178, Pin the newly added docker/setup-buildx-action and docker/bake-action steps to immutable commit SHAs instead of the mutable v4 and v7 tags, preserving their current action versions and configuration.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In @.github/workflows/ci-test.yaml:
- Around line 171-178: Pin the newly added docker/setup-buildx-action and
docker/bake-action steps to immutable commit SHAs instead of the mutable v4 and
v7 tags, preserving their current action versions and configuration.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: d4e93703-fb7d-4774-aa7c-c19dd594ce6a
📒 Files selected for processing (1)
.github/workflows/ci-test.yaml
SonarCloud flagged the two floating action tags added for the e2e gha cache as new MAJOR vulnerabilities (security rating C). Pin to the commit SHA the tag points to so a moved tag can't inject code. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
e2e ran on every non-draft PR regardless of paths, so image-only or docs-only changes triggered a full ~7min platform boot for nothing. Add an `e2e_relevant` path scope that skips frontend/** and docs/** while keeping docker/** in scope (image changes must still be e2e-tested), gate the e2e job on it, and accept a skipped e2e as a pass in the report gate (matching how a skipped test tier is handled). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L549habQ6K2m1ACJ7DwtsE
ritwik-g
left a comment
There was a problem hiding this comment.
Automated review — UN-3636 execute-path e2e
Verified against the actual head commit, the live backend source, and the PR's own green CI run (29740023152): all six new e2e tests genuinely executed and passed there (e2e-workflow 1, e2e-prompt-studio 1, e2e-etl 1, e2e-api-deployment 3), and all seven test_mock_response.py tests pass with no provider key. That evidence retires a lot of speculation — bake builds and loads correctly, e2e_relevant does evaluate true, the mock reaches the workers, the adapter/connector UUIDs resolve, and the critical-path markers all map to real tests.
Findings below are what survived that verification. Nothing here disputes the approach — the injection-site hook is the right shape, and removing optional: true plus wiring dep-failure cascade genuinely strengthens the gate.
One P1, in the CI gate rather than the test code.
Note on the PR description (not a code finding)
The description is stale and now contradicts the diff. It still says under "Not in this PR" that the execute-path e2e tests "will land next" and that "critical-path mappings are intentionally left unflipped until a real passing test exists" — but this PR contains exactly those tests and flips six mappings in critical_paths.yaml. Worth rewriting before merge so the merge commit describes what actually shipped.
PR title conforms to TICKET [TYPE] TITLE.
Posted as COMMENT, not an approval.
jaseemjaskp
left a comment
There was a problem hiding this comment.
PR Review — hermetic LLM mock hook + execute-path e2e coverage
Ran the full PR Review Toolkit (code review, silent-failure hunt, type design, test coverage, comment analysis, simplification) plus independent verification of each claim against the code.
Overall this is unusually disciplined test infrastructure. The proof: marker ratchet is the strongest thing here: because passed_critical_path_ids counts only passing marked tests, a skipped or blocked test degrades to a reported gap rather than a silent green — that safety net catches most of what would otherwise be serious findings. The SDK hook itself is well-built: env-gated, never clobbers an explicit mock_response, applied uniformly at all four completion sites so there's no half-mocked path, value never logged. The e2e tests assert real system outputs (the answer arriving through the API, landing in the object store, written back to the Prompt Studio row) rather than asserting on the mock. No tautological tests found.
Three blocking findings:
test_api_deployment_fan_out.pynames its own false-pass mode in the docstring and then doesn't guard against it — andcritical_paths.yamlretires a declared gap on that test's word._blocked_resultnever reaches the CI report.cmd_reportrebuilds from junit only, and the blocked branch writes none — so the new ⏭️ status andblocked_bynote are invisible in the sticky PR comment, which is the only place they'd be read.e2e-api-deploymentcan be SIGKILLed before any test reports why: 3 × 300s poll budget against a 630s subprocess cap.
The rest are low/nit — mostly skip-paths that read green outside CI, a few comments whose stated reason is wrong even where the conclusion is right, and some duplication worth collapsing now that there are four e2e packages.
Findings verified against the code; already-posted review threads were cross-checked and excluded.
muhammad-ali-e
left a comment
There was a problem hiding this comment.
LGTM
Did a manual review since already done couple of AI reviews
`report` needs `changes` but never consulted its result. A failed filter job (paths-filter error, checkout failure, runner death) skips `test`, which the gate reads as a legitimate path-filter skip and reports success with no unit or integration tests having run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
Rig: - Blocked groups now persist a synthetic junit carrying `rig-blocked-by`, so the cascade survives `rig report`, which rebuilds results from junit alone. Without it a blocked group parsed as a zero-count pass and falsely attested critical-path coverage. - GroupResult rejects a blocked row with non-zero counters. - Report statuses are module constants instead of scattered literals. - An exit-5 (no tests collected) group that others depend on now counts as failed, so dependents are blocked rather than run against nothing. - The LLM mock default moves to cmd_run so it applies to every runtime. - groups.py docstring no longer claims runtime gating is unimplemented. E2E: - Shared `wait_for_execution` helper replaces three near-identical polls; a CsrfSession subclass stamps X-CSRFToken so per-call helpers can't drift. - api_deployment poll: sleeps on every iteration, captures exceptions as evidence, treats 406 as a lost-200 acknowledge rather than a poll assert, and surfaces non-422 statuses as themselves. Timeout drops to 150s so the group's three tests fit one 600s budget. - MinIO scheme drives both the workers' URL and the client's TLS flag. - Missing rig-provisioned prerequisites fail instead of silently skipping. Core/SDK: - `ExecutionStatus.terminal_statuses()` names the terminal set. - Mock-response tests cover the empty-string no-op and assert every completion path still calls the injection hook. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
The e2e image build is the only CI check that runs `bun install` and `bun run build`; ci-frontend-lint only runs biome. Excluding frontend/** from e2e_relevant let a broken build or a desynced bun.lock merge green and surface on main via production-build.yaml. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
The fan-out test asserted only the rejoin: every assertion held identically for three files processed serially in one batch, while critical_paths.yaml retired the workflow-execution-fan-out gap on its strength. Nothing distinguishes a batch — the batch index is a discarded loop local and the celery task id reaches only worker stdout — so assert the timing instead. WorkflowFileExecution rows are created by the worker owning the batch rather than up front by the dispatcher, so fanned out each row's window is one delay wide, while serialised into one batch the windows open together and grow by a delay per file. Comparing durations rather than overlap is the point: serialised rows are created in one tight loop and overlap too. That needs each completion to cost a known, measurable amount, so add UNSTRACT_LLM_MOCK_DELAY alongside the existing mock hatch. It is inert without UNSTRACT_LLM_MOCK_RESPONSE, since litellm ignores mock_delay on a real completion, and a non-numeric value degrades to no delay rather than breaking every completion in the worker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
Manual runs reported green with nothing tested. paths-filter has no base to diff on workflow_dispatch, so it resolves against the default branch and a manual run on main diffs main against itself: every filter false, every tier skipped, gate green. Filtering is a PR-time saving, so force both scopes true off-PR and skip the filter step entirely there. Gate the LLM mock on ENVIRONMENT as well as the mock var, so a stray variable in a real deployment can't fake completions and their billing. Production sets neither, so this fails closed. `development` is allowed alongside `test` because that is what base compose sets on the two workers that run the injection -- proven by the mock working today with the var set on exactly those two services -- and the overlay now pins ENVIRONMENT=test on them so the tier can't lose its mock to a base-compose edit. A refusal warns rather than passing silently: setting the var at all means someone expected mocking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
The rig's `down -v` destroys the containers, so the workflow step that ran `docker compose logs` afterwards always wrote an empty file — and overwrote nothing useful only because there was nothing else to write. Capture inside `ComposeRuntime.down()` while the containers still exist, and drop the step. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The bound was half a delay — a 1s window on per-file durations of ~12s, which measures runner contention as much as batching. One delay is the midpoint between the two outcomes the check separates (~0 spread fanned out, ~2 delays serialised), and the delay itself has to be large enough for that midpoint to sit above the noise. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The worker log settles what the test could not: `Arranging 3 files into 3 batches` — the fan-out happens. Yet the per-file durations it was judged by came out 6.6 / 18.1 / 22.9. Three tool containers running at once on a CI runner cost more in contention than they gain in overlap, so genuine fan-out lands further apart than serialised files do, and no threshold separates the two. Raising the delay does not help: the contention scales with the parallelism being measured. So the proof is removed rather than retuned, and with it the mock delay that existed only to feed it -- every mocked completion across the whole e2e tier was stalling for it. `workflow-execution-fan-out` becomes a declared gap (visible in the report, non-gating), and the test keeps asserting the rejoin, which is directly observable. Closing the gap wants a batch id on WorkflowFileExecution, which is execution observability worth having on its own. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ritwik-g
left a comment
There was a problem hiding this comment.
Approved — all review findings addressed, verified against head 6dd8d756
Re-reviewed the full delta since my first pass (293695cff..6dd8d756, 579 insertions / 19 files), not just the P1 line. Every finding is resolved, and I verified the fixes against the actual head + its green CI run (29812358973), not the diff alone.
P1 — CI gate hole (blocker): fixed. report's gate now hard-fails on needs.changes.result (ci-test.yaml:340), so a broken filter job can no longer skip both tiers to green. The related workflow_dispatch main-diff-against-itself hole I'd flagged unverified is also closed — relevant/e2e_relevant are forced true off-PR and the paths-filter step is gated to pull_request.
P2s: all fixed.
poll_deliveredrewritten — sleeps on the transient path and records the exception; 406 → clear "200 lost in transit" failure instead of a misleading== 422assert; non-422/406 → explicit "unexpected poll status" rather than a poll-protocol assert (the 500 tool-not-found case now reads correctly).- Frontend e2e coverage restored —
!frontend/**removed frome2e_relevant, so a frontend change again exercises the image build (bun run build). groups.pydocstring corrected to describe the runtime skip-on-gate-failure this PR implements.
P3s: all fixed — MinIO secure= now derives from UNSTRACT_MINIO_SCHEME (one scheme drives both halves); README no longer claims 10/20/30 for the streaming path and documents the test_connection/adapter-register-llm gap; the api_deployment fixture docstring now says per-xdist-worker.
Beyond the findings, and welcome: the mock now fails closed unless ENVIRONMENT ∈ {test, development} — a real hardening of the billing-corruption risk, with parametrized refusal tests (production/staging/unset all pass in CI). workflow-execution-fan-out was honestly demoted to a covered_by: [] gap rather than proven by timing that can't discriminate contention from serialisation — the right call under "no false greens." Blocked-group state now round-trips through junit into CI's rig report, with a __post_init__ invariant and new cascade/round-trip tests.
Verified on head 6dd8d756: full CI green (unit 490, integration, e2e 10m); the six execute-path e2e tests all genuinely executed and passed (workflow 1, prompt-studio 1, etl 1, api-deployment 3); the new ENVIRONMENT-guard tests pass including the production-refusal cases; no unresolved review threads.
Nit, non-blocking: the PR description still reads as if the e2e tests and critical-path flips are not in this PR ("will land next") — worth a rewrite before merge so the squash commit matches what shipped.
Approving the review. Not merging — this is stacked on #2164, and merge is a separate call.
|
Unstract test resultsPer-group results
Critical paths
|



Stacked on #2164. Adds the deterministic-LLM capability the execute-path critical-path gaps need, without dragging real provider secrets into CI.
What
llm.py): one module-level_inject_mock_response+ a one-line call at all fourlitellm.completion/acompletionsites. WhenUNSTRACT_LLM_MOCK_RESPONSEis set, litellm returns that string as the completion with fixed usage (10/20/30), tunable viaDEFAULT_MOCK_RESPONSE_*; error sentinels likelitellm.RateLimitErrorforce error paths. Unset in production → no-op.worker-executor-v2/worker-file-processing-v2(the processes that run the LLM). Empty unless the rig/CI sets it.Why this shape
test_connection(), so a fixed mock runs every execute path to completion.Not in this PR
The actual execute-path e2e tests (workflow-create-execute, usage-token-tracking, …) that consume this hook — those need the full compose stack booted to author the API flow correctly and will land next. Critical-path mappings are intentionally left unflipped until a real passing test exists (no false greens).
🤖 Generated with Claude Code