Skip to content

UN-3636 [DEV] Hermetic LLM mock hook for execute-path critical tests#2170

Merged
chandrasekharan-zipstack merged 24 commits into
mainfrom
feat/un-3636-execute-path-llm-mock
Jul 21, 2026
Merged

UN-3636 [DEV] Hermetic LLM mock hook for execute-path critical tests#2170
chandrasekharan-zipstack merged 24 commits into
mainfrom
feat/un-3636-execute-path-llm-mock

Conversation

@chandrasekharan-zipstack

Copy link
Copy Markdown
Contributor

Stacked on #2164. Adds the deterministic-LLM capability the execute-path critical-path gaps need, without dragging real provider secrets into CI.

What

  • sdk1 hook (llm.py): one module-level _inject_mock_response + a one-line call at all four litellm.completion/acompletion sites. When UNSTRACT_LLM_MOCK_RESPONSE is set, litellm returns that string as the completion with fixed usage (10/20/30), tunable via DEFAULT_MOCK_RESPONSE_*; error sentinels like litellm.RateLimitError force error paths. Unset in production → no-op.
  • compose overlay: pass the env through worker-executor-v2 / worker-file-processing-v2 (the processes that run the LLM). Empty unless the rig/CI sets it.
  • unit tests: pin both our injector and the litellm mock contract (string + deterministic usage + error sentinel), so a litellm bump can't silently break hermetic execute-path coverage.

Why this shape

  • Single-place change, no cross-codebase adapter churn (unlike the retired NoOp adapter).
  • Hermetic + secret-free by construction — a cache-miss can't fall through to a real paid call (contrast: a record/replay proxy needs real keys to warm + as miss fallback).
  • Execute path tolerates a plain string (TEXT prompts stored verbatim; JSON prompts use tolerant repair) and never calls test_connection(), so a fixed mock runs every execute path to completion.

Not in this PR

The actual execute-path e2e tests (workflow-create-execute, usage-token-tracking, …) that consume this hook — those need the full compose stack booted to author the API flow correctly and will land next. Critical-path mappings are intentionally left unflipped until a real passing test exists (no false greens).

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

The change adds deterministic LLM mocking, shared hermetic E2E provisioning, API deployment and ETL execution tests, expanded critical-path coverage, CI image caching, and dependency-aware test-group blocking and reporting.

Changes

Hermetic E2E coverage

Layer / File(s) Summary
Deterministic mock runtime
unstract/sdk1/src/unstract/sdk1/llm.py, unstract/sdk1/tests/test_mock_response.py, tests/rig/runtime.py, tests/compose/docker-compose.test.yaml
LLM completion paths inject UNSTRACT_LLM_MOCK_RESPONSE, with runtime defaults, Compose wiring, warning behavior, and unit coverage.
Shared workflow and storage provisioning
tests/e2e/conftest.py, tests/e2e/api_deployment/conftest.py, tests/e2e/etl/conftest.py
Fixtures provision mocked workflows, API deployments, and MinIO-backed ETL workflows with authenticated HTTP helpers and polling support.
E2E execution scenarios
tests/e2e/api_deployment/*, tests/e2e/workflows/*, tests/e2e/prompt_studio/*, tests/e2e/etl/*
Tests cover async delivery, fan-out rejoin behavior, workflow execution, token usage, Prompt Studio responses, and ETL output.
Coverage registration and CI execution
tests/critical_paths.yaml, tests/groups.yaml, tests/README.md, .github/workflows/ci-test.yaml
Critical paths and required E2E groups are updated, documentation describes the new setup, and CI uses Buildx-based image builds with caching.

Dependency-aware group execution

Layer / File(s) Summary
Dependency traversal and group blocking
tests/rig/groups.py, tests/rig/cli.py, tests/rig/tests/test_groups.py
The rig computes transitive dependencies, tracks failed groups, skips downstream groups with synthetic blocked results, and includes the MinIO pytest plugin.
Blocked result reporting
tests/rig/reporting.py, tests/rig/tests/test_reporting.py
Blocked status, icons, dependency notes, and non-green classification are implemented and tested.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant E2ETests
  participant ProvisionedWorkflow
  participant API
  participant Workers
  participant LLM
  E2ETests->>ProvisionedWorkflow: provision mocked workflow
  ProvisionedWorkflow->>API: create and configure workflow
  E2ETests->>API: dispatch execution
  API->>Workers: process execution
  Workers->>LLM: request completion with mock_response
  LLM-->>Workers: return canned answer
  Workers-->>API: update execution result
  API-->>E2ETests: deliver completed result
Loading

Suggested reviewers: jaseemjaskp, muhammad-ali-e, ritwik-g

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description covers What and Why, but most template sections are missing, including How, breakage impact, testing, and checklist. Fill in the missing template sections: How, breakage impact, migrations, env config, related issues, testing, screenshots, and checklist.
Docstring Coverage ⚠️ Warning Docstring coverage is 32.76% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and clearly describes the main change: a hermetic LLM mock hook for execute-path tests.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/un-3636-execute-path-llm-mock

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Base automatically changed from feat/un-3636-critical-path-integration-tests to main July 14, 2026 10:31
Execute-path critical paths (workflow-create-execute, usage-token-tracking,
etc.) need a deterministic LLM without real provider secrets. Add a single
sdk1 escape hatch: when UNSTRACT_LLM_MOCK_RESPONSE is set, inject litellm's
mock_response so completions return that string with fixed usage (10/20/30),
tunable via DEFAULT_MOCK_RESPONSE_* and error sentinels like
"litellm.RateLimitError". No-op in production (env unset).

Wire the env through worker-executor-v2 / worker-file-processing-v2 in the e2e
compose overlay so the mock reaches the process that runs the LLM.

Covered by unit tests pinning both our injector and the litellm mock contract
(string + deterministic usage + error sentinel) so a litellm bump can't
silently break hermetic execute-path coverage.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
@chandrasekharan-zipstack
chandrasekharan-zipstack force-pushed the feat/un-3636-execute-path-llm-mock branch from f26d44c to fc0d056 Compare July 14, 2026 12:06
chandrasekharan-zipstack and others added 6 commits July 14, 2026 18:39
Add e2e tests that exercise the full execute path hermetically using the
UNSTRACT_LLM_MOCK_RESPONSE hook (no real LLM/secret):

- api-deployment-run + usage-token-tracking: deploy a workflow as an API,
  POST a doc synchronously, assert the mocked answer and fixed 10/20/30
  usage come back as structured JSON.
- workflow-create-execute: two-step /workflow/execute/ then poll to
  COMPLETED; COMPLETED + a successful file is the hermetic proof (without
  the mock the completion would fail).

A shared provisioned_workflow fixture stands up a Prompt Studio-backed API
workflow (NoOp x2text/vectordb, chunk_size=0 so embedding/vectordb are
skipped). The rig seeds a canonical mock string in ComposeRuntime so both
the workers and pytest see the same value; tests skip if it is unset.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
- test_mock_response.py: split the module docstring summary line (D205 was
  failing pre-commit.ci, the only red hook on the PR).
- groups.yaml: drop `optional: true` from e2e-workflow / e2e-api-deployment.
  They now hold real execute-path tests rather than placeholders, so a red one
  should fail the build directly instead of only surfacing as a critical-path
  gap via --fail-on-critical-gap.
- README: the workflows/ and api_deployment/ dirs are no longer "(future)", and
  document UNSTRACT_LLM_MOCK_RESPONSE under E2E runtime — including that only
  completions are mocked (embedding.py is unhooked; chunk_size=0 is what keeps
  indexing off the real provider).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P4tU5vyB4G6BgfAnGqjxLs
The comment block sat where #2176 reworded the e2e-login comment and inserted
a new group, creating a textual merge conflict against main. Move it inside
each group body; the optional: true removals merge cleanly on their own.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P4tU5vyB4G6BgfAnGqjxLs
Drop the restated-critical-path and mechanism-narrating comments across the
LLM mock and the execute-path e2e tests; keep only the constraints the code
can't show (litellm's fixed usage, the mandatory filter arg, the two-call
execute contract). Also drop the groups.yaml 'not optional' comment: the field
is self-evident and would go stale.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P4tU5vyB4G6BgfAnGqjxLs
Three follow-ups from review:

- llm.py: warn once per process when UNSTRACT_LLM_MOCK_RESPONSE is active. The
  hatch was silent, so a stray env var in production would fake every
  completion and its billing with nothing in the logs.

- rig: implement the runtime-gate-skip TODO in cmd_run. Groups run in topo
  order, so a dep's result is known before its dependents start; a red dep
  means the precondition it gates does not hold and dependents would only
  produce noise against a half-up stack. Adds GroupManifest.transitive_deps and
  a 'blocked' GroupResult status which is never green, so a blocked group
  attests no coverage and --fail-on-critical-gap still catches a path that
  quietly stopped being covered. This mattered less while every platform
  dependent was optional; it does now that the execute-path groups gate.

- api-deployment e2e: the 10/20/30 usage assert now explains itself, since
  counts are summed per reason and a multiple means extra completions rather
  than broken tracking.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P4tU5vyB4G6BgfAnGqjxLs
Closes the remaining gaps in the registry; every path is now proof: marker.

- callback-result-delivery: async API execution (timeout=0). With a file to
  process, COMPLETED is written only by the chord callback, so a polled result
  proves the callback worker ran. Pins the one-shot read too — fetching a
  delivered result acknowledges it and the next read 406s, which a retry loop
  added later would otherwise swallow silently.

- workflow-execution-fan-out: multi-file execution, one batch per file. Needed
  MAX_PARALLEL_FILE_BATCHES on the *backend* (workers ask it for the value and
  only fall back to their own env); at the default of 1 the files share a batch
  and run serially, so the test would have passed proving nothing. File bodies
  differ because identical ones are deduplicated by hash on ingest.

- prompt-studio-fetch-response: the authoring surface. Async, so the answer is
  polled from prompt-output/ rather than the task status, which reports the
  executor finishing and not the callback having stored anything. Upload is
  content-sniffed and OSS takes PDF only, hence the real (if empty) PDF.

- pipeline-etl-execute: source connector to destination, over MinIO — the only
  storage connector the compose stack both boots and registers. Reuses the
  exported tool but needs its own workflow, since endpoints are per-workflow
  and the existing one is committed to the API deployment tests.

None of these have run: the e2e job is skipped while the PR is a draft. Locally
verified only as far as it goes here — rig validate, rig self-tests, ruff, and
pytest collection.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P4tU5vyB4G6BgfAnGqjxLs
@chandrasekharan-zipstack
chandrasekharan-zipstack marked this pull request as ready for review July 17, 2026 11:51
@greptile-apps

greptile-apps Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds hermetic LLM mocking and expands execute-path test coverage. The main changes are:

  • Adds an environment-gated LiteLLM mock response hook in the SDK.
  • Passes the mock response into the worker test compose overlay.
  • Adds workflow, API deployment, Prompt Studio, and ETL e2e tests.
  • Updates the rig to handle blocked groups and critical-path reporting.
  • Updates CI path filtering and e2e image build caching.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.

Important Files Changed

Filename Overview
unstract/sdk1/src/unstract/sdk1/llm.py Adds the environment-gated mock response hook around LiteLLM completion calls.
tests/compose/docker-compose.test.yaml Forwards the mock response setting into the worker services used by execute-path tests.
tests/e2e/conftest.py Adds shared workflow provisioning and mock-response fixtures for execute-path tests.
tests/e2e/etl/conftest.py Adds MinIO-backed ETL workflow provisioning for filesystem source and destination coverage.
tests/rig/cli.py Adds dependency-blocked group handling and mock response setup for platform runs.
tests/rig/reporting.py Adds explicit blocked-group status handling in rig reports and summaries.

Reviews (14): Last reviewed commit: "Merge branch 'main' into feat/un-3636-ex..." | Re-trigger Greptile

Comment thread tests/e2e/etl/conftest.py
Comment thread tests/rig/cli.py
…ll minio

The e2e tier failed on two counts:

- api-deployment run/fan-out asserted COMPLETED on the synchronous POST, but a
  file-bearing execution only reaches COMPLETED once the chord callback rejoins
  — the sync path reports EXECUTING under CI load. All three api-deployment
  tests now dispatch async (timeout=0) and poll the status endpoint via shared
  dispatch_async/poll_delivered helpers, tolerating transient poll timeouts
  (the async test's earlier ReadTimeout). execution_id is parsed from status_api
  since the async PENDING body omits it as a top-level field.

- e2e-etl skipped ("minio client not installed"), leaving pipeline-etl-execute a
  gap. Add minio to the rig's injected pytest deps so the test runs.

Verified against a full local stack: api_deployment (3), workflow, etl,
prompt_studio all green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
Comment thread tests/e2e/etl/conftest.py Fixed
Clears the SonarCloud S5332 clear-text-HTTP finding on the test fixture
(compose-internal MinIO has no TLS) by sourcing the scheme from env, and
trims the low-value minio dependency comment.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (3)
tests/e2e/api_deployment/conftest.py (1)

84-96: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

No backoff on ConnectionError/Timeout retries.

except _TRANSIENT: continue (Lines 89-90) retries immediately with no sleep, unlike the other branches. A Timeout is naturally throttled by the 30s request timeout, but a fast-failing ConnectionError (e.g., connection reset) would busy-loop against the status endpoint for up to _POLL_TIMEOUT_SECONDS.

Proposed fix
         except _TRANSIENT:
+            time.sleep(2)
             continue
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/e2e/api_deployment/conftest.py` around lines 84 - 96, Update the
polling retry logic around the transient exception handler in the status-wait
function to sleep for the existing polling interval before continuing. Preserve
the current handling for successful responses and non-transient failures, while
ensuring fast-failing ConnectionError retries do not busy-loop.
tests/e2e/conftest.py (1)

116-131: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

CSRF-wrapping _post/_patch helpers are reimplemented per file. Both sites attach X-CSRFToken from the session cookie and POST/PATCH with a fixed timeout — one root cause (no shared e2e HTTP helper), duplicated per file.

  • tests/e2e/conftest.py#L116-L131: keep as the canonical _post/_patch (accepting a requests.Session); consider exporting them for reuse instead of re-deriving per-module wrappers.
  • tests/e2e/prompt_studio/test_prompt_studio_fetch_response.py#L101-L103: replace the local _post with a thin call into tests/e2e/conftest.py's _post(pw.session, ...) instead of reimplementing the CSRF-attach logic.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/e2e/conftest.py` around lines 116 - 131, The CSRF-aware HTTP helpers
are duplicated across e2e files. Keep tests/e2e/conftest.py lines 116-131 as the
canonical _post and _patch implementations, making them reusable if needed; in
tests/e2e/prompt_studio/test_prompt_studio_fetch_response.py lines 101-103,
remove the local _post wrapper and delegate to conftest.py’s _post using
pw.session, preserving the existing request arguments and timeout behavior.
tests/rig/cli.py (1)

449-457: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider extracting the failure predicate to avoid 3-way drift.

"Did this group fail" is now computed independently in three places with slightly different optional gating (cascade tracking at Line 478-481 ignores optional; the two overall_exit folds at Line 493-498 and 502-508 respect it). A future tweak to failure criteria (e.g. new non-failing exit code) risks updating only some sites.

♻️ Suggested consolidation
+def _group_failed(exit_code: int, result: GroupResult | None) -> bool:
+    return exit_code not in _NON_FAILING_PYTEST_EXIT_CODES or (
+        result is not None and (result.errors or result.failed)
+    )
+
             if result is not None:
                 group_results.append(result)
             # Tracked regardless of `optional` — see the cascade note above.
-            if exit_code not in _NON_FAILING_PYTEST_EXIT_CODES or (
-                result is not None and (result.errors or result.failed)
-            ):
+            if _group_failed(exit_code, result):
                 failed_groups.add(name)
             ...
-            if (
-                exit_code not in _NON_FAILING_PYTEST_EXIT_CODES
-                and not group.optional
-                and overall_exit == 0
-            ):
+            if _group_failed(exit_code, None) and not group.optional and overall_exit == 0:
                 overall_exit = exit_code
-            if (
-                result is not None
-                and (result.errors or result.failed)
-                and not group.optional
-                and overall_exit == 0
-            ):
+            if _group_failed(0, result) and not group.optional and overall_exit == 0:
                 overall_exit = 1

Also applies to: 477-481, 493-508

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/rig/cli.py` around lines 449 - 457, Extract a shared failure predicate
for group results and use it consistently in the runnable-group cascade tracking
and both overall_exit folds. Ensure the predicate applies the same
optional-group gating everywhere, replacing the duplicated exit-code checks
while preserving blocked-result handling.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/e2e/conftest.py`:
- Around line 250-263: Track whether the workflow endpoint setup loop in the
fixture successfully patches at least one matching endpoint to API, and assert
that this occurred after iterating through eps. Keep the existing matching and
patch status assertion, but fail immediately with a clear fixture-level message
when eps is empty or no endpoint matches workflow_id.

---

Nitpick comments:
In `@tests/e2e/api_deployment/conftest.py`:
- Around line 84-96: Update the polling retry logic around the transient
exception handler in the status-wait function to sleep for the existing polling
interval before continuing. Preserve the current handling for successful
responses and non-transient failures, while ensuring fast-failing
ConnectionError retries do not busy-loop.

In `@tests/e2e/conftest.py`:
- Around line 116-131: The CSRF-aware HTTP helpers are duplicated across e2e
files. Keep tests/e2e/conftest.py lines 116-131 as the canonical _post and
_patch implementations, making them reusable if needed; in
tests/e2e/prompt_studio/test_prompt_studio_fetch_response.py lines 101-103,
remove the local _post wrapper and delegate to conftest.py’s _post using
pw.session, preserving the existing request arguments and timeout behavior.

In `@tests/rig/cli.py`:
- Around line 449-457: Extract a shared failure predicate for group results and
use it consistently in the runnable-group cascade tracking and both overall_exit
folds. Ensure the predicate applies the same optional-group gating everywhere,
replacing the duplicated exit-code checks while preserving blocked-result
handling.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 13ad4ba2-a18b-4cb3-b034-91f3430b5f52

📥 Commits

Reviewing files that changed from the base of the PR and between 0786a0a and faa0d35.

📒 Files selected for processing (25)
  • tests/README.md
  • tests/compose/docker-compose.test.yaml
  • tests/critical_paths.yaml
  • tests/e2e/api_deployment/__init__.py
  • tests/e2e/api_deployment/conftest.py
  • tests/e2e/api_deployment/test_api_deployment_async.py
  • tests/e2e/api_deployment/test_api_deployment_fan_out.py
  • tests/e2e/api_deployment/test_api_deployment_run.py
  • tests/e2e/conftest.py
  • tests/e2e/etl/__init__.py
  • tests/e2e/etl/conftest.py
  • tests/e2e/etl/test_pipeline_etl_execute.py
  • tests/e2e/prompt_studio/__init__.py
  • tests/e2e/prompt_studio/test_prompt_studio_fetch_response.py
  • tests/e2e/workflows/__init__.py
  • tests/e2e/workflows/test_workflow_execute.py
  • tests/groups.yaml
  • tests/rig/cli.py
  • tests/rig/groups.py
  • tests/rig/reporting.py
  • tests/rig/runtime.py
  • tests/rig/tests/test_groups.py
  • tests/rig/tests/test_reporting.py
  • unstract/sdk1/src/unstract/sdk1/llm.py
  • unstract/sdk1/tests/test_mock_response.py

Comment thread tests/e2e/conftest.py
chandrasekharan-zipstack and others added 2 commits July 20, 2026 12:53
Assert at least one endpoint was patched to API in the provisioning
fixture, so a field-name mismatch surfaces here instead of as a confusing
downstream execute failure (CodeRabbit).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
The e2e job runs on ephemeral GitHub-hosted runners, so a plain
`docker compose build` rebuilt every image cold each run (~3min). Switch
to docker/bake-action with per-service type=gha cache scopes matching the
release workflow, so each runner restores layers from the shared GHA cache
(including entries the release build already wrote). load=true exports the
images into the daemon for the compose stack to boot.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
.github/workflows/ci-test.yaml (1)

171-178: 🔒 Security & Privacy | 🔵 Trivial | 💤 Low value

Pin GitHub Actions to specific commit SHAs.

The newly added setup-buildx-action and bake-action use mutable version tags (@v4 and @v7). Consider pinning them to specific commit SHAs to maintain consistency with the checkout, login-action, and setup-uv steps earlier in this workflow and to strengthen supply chain security against upstream tag hijacking.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/ci-test.yaml around lines 171 - 178, Pin the newly added
docker/setup-buildx-action and docker/bake-action steps to immutable commit SHAs
instead of the mutable v4 and v7 tags, preserving their current action versions
and configuration.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In @.github/workflows/ci-test.yaml:
- Around line 171-178: Pin the newly added docker/setup-buildx-action and
docker/bake-action steps to immutable commit SHAs instead of the mutable v4 and
v7 tags, preserving their current action versions and configuration.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: d4e93703-fb7d-4774-aa7c-c19dd594ce6a

📥 Commits

Reviewing files that changed from the base of the PR and between 540a035 and d409dac.

📒 Files selected for processing (1)
  • .github/workflows/ci-test.yaml

SonarCloud flagged the two floating action tags added for the e2e gha
cache as new MAJOR vulnerabilities (security rating C). Pin to the commit
SHA the tag points to so a moved tag can't inject code.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
Comment thread .github/workflows/ci-test.yaml Fixed
Comment thread .github/workflows/ci-test.yaml Fixed
e2e ran on every non-draft PR regardless of paths, so image-only or
docs-only changes triggered a full ~7min platform boot for nothing.

Add an `e2e_relevant` path scope that skips frontend/** and docs/**
while keeping docker/** in scope (image changes must still be
e2e-tested), gate the e2e job on it, and accept a skipped e2e as a
pass in the report gate (matching how a skipped test tier is handled).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L549habQ6K2m1ACJ7DwtsE

@ritwik-g ritwik-g left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review — UN-3636 execute-path e2e

Verified against the actual head commit, the live backend source, and the PR's own green CI run (29740023152): all six new e2e tests genuinely executed and passed there (e2e-workflow 1, e2e-prompt-studio 1, e2e-etl 1, e2e-api-deployment 3), and all seven test_mock_response.py tests pass with no provider key. That evidence retires a lot of speculation — bake builds and loads correctly, e2e_relevant does evaluate true, the mock reaches the workers, the adapter/connector UUIDs resolve, and the critical-path markers all map to real tests.

Findings below are what survived that verification. Nothing here disputes the approach — the injection-site hook is the right shape, and removing optional: true plus wiring dep-failure cascade genuinely strengthens the gate.

One P1, in the CI gate rather than the test code.

Note on the PR description (not a code finding)

The description is stale and now contradicts the diff. It still says under "Not in this PR" that the execute-path e2e tests "will land next" and that "critical-path mappings are intentionally left unflipped until a real passing test exists" — but this PR contains exactly those tests and flips six mappings in critical_paths.yaml. Worth rewriting before merge so the merge commit describes what actually shipped.

PR title conforms to TICKET [TYPE] TITLE.

Posted as COMMENT, not an approval.

Comment thread .github/workflows/ci-test.yaml
Comment thread tests/e2e/api_deployment/conftest.py Outdated
Comment thread .github/workflows/ci-test.yaml
Comment thread tests/rig/cli.py
Comment thread tests/e2e/etl/conftest.py Outdated
Comment thread tests/README.md Outdated
Comment thread tests/e2e/api_deployment/conftest.py Outdated

@jaseemjaskp jaseemjaskp left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review — hermetic LLM mock hook + execute-path e2e coverage

Ran the full PR Review Toolkit (code review, silent-failure hunt, type design, test coverage, comment analysis, simplification) plus independent verification of each claim against the code.

Overall this is unusually disciplined test infrastructure. The proof: marker ratchet is the strongest thing here: because passed_critical_path_ids counts only passing marked tests, a skipped or blocked test degrades to a reported gap rather than a silent green — that safety net catches most of what would otherwise be serious findings. The SDK hook itself is well-built: env-gated, never clobbers an explicit mock_response, applied uniformly at all four completion sites so there's no half-mocked path, value never logged. The e2e tests assert real system outputs (the answer arriving through the API, landing in the object store, written back to the Prompt Studio row) rather than asserting on the mock. No tautological tests found.

Three blocking findings:

  1. test_api_deployment_fan_out.py names its own false-pass mode in the docstring and then doesn't guard against it — and critical_paths.yaml retires a declared gap on that test's word.
  2. _blocked_result never reaches the CI report. cmd_report rebuilds from junit only, and the blocked branch writes none — so the new ⏭️ status and blocked_by note are invisible in the sticky PR comment, which is the only place they'd be read.
  3. e2e-api-deployment can be SIGKILLed before any test reports why: 3 × 300s poll budget against a 630s subprocess cap.

The rest are low/nit — mostly skip-paths that read green outside CI, a few comments whose stated reason is wrong even where the conclusion is right, and some duplication worth collapsing now that there are four e2e packages.

Findings verified against the code; already-posted review threads were cross-checked and excluded.

Comment thread tests/e2e/api_deployment/test_api_deployment_fan_out.py Outdated
Comment thread tests/rig/cli.py
Comment thread tests/groups.yaml
Comment thread tests/rig/runtime.py Outdated
Comment thread tests/rig/cli.py Outdated
Comment thread tests/critical_paths.yaml Outdated
Comment thread tests/README.md
Comment thread unstract/sdk1/tests/test_mock_response.py Outdated
Comment thread tests/e2e/etl/conftest.py
Comment thread tests/rig/cli.py Outdated

@muhammad-ali-e muhammad-ali-e left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Did a manual review since already done couple of AI reviews

Comment thread tests/e2e/workflows/test_workflow_execute.py Outdated
Comment thread tests/rig/reporting.py Outdated
Comment thread tests/rig/reporting.py Outdated
chandrasekharan-zipstack and others added 9 commits July 21, 2026 11:22
`report` needs `changes` but never consulted its result. A failed filter job
(paths-filter error, checkout failure, runner death) skips `test`, which the
gate reads as a legitimate path-filter skip and reports success with no unit
or integration tests having run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
Rig:
- Blocked groups now persist a synthetic junit carrying `rig-blocked-by`, so
  the cascade survives `rig report`, which rebuilds results from junit alone.
  Without it a blocked group parsed as a zero-count pass and falsely attested
  critical-path coverage.
- GroupResult rejects a blocked row with non-zero counters.
- Report statuses are module constants instead of scattered literals.
- An exit-5 (no tests collected) group that others depend on now counts as
  failed, so dependents are blocked rather than run against nothing.
- The LLM mock default moves to cmd_run so it applies to every runtime.
- groups.py docstring no longer claims runtime gating is unimplemented.

E2E:
- Shared `wait_for_execution` helper replaces three near-identical polls; a
  CsrfSession subclass stamps X-CSRFToken so per-call helpers can't drift.
- api_deployment poll: sleeps on every iteration, captures exceptions as
  evidence, treats 406 as a lost-200 acknowledge rather than a poll assert,
  and surfaces non-422 statuses as themselves. Timeout drops to 150s so the
  group's three tests fit one 600s budget.
- MinIO scheme drives both the workers' URL and the client's TLS flag.
- Missing rig-provisioned prerequisites fail instead of silently skipping.

Core/SDK:
- `ExecutionStatus.terminal_statuses()` names the terminal set.
- Mock-response tests cover the empty-string no-op and assert every
  completion path still calls the injection hook.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
The e2e image build is the only CI check that runs `bun install` and
`bun run build`; ci-frontend-lint only runs biome. Excluding frontend/**
from e2e_relevant let a broken build or a desynced bun.lock merge green
and surface on main via production-build.yaml.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
The fan-out test asserted only the rejoin: every assertion held identically
for three files processed serially in one batch, while critical_paths.yaml
retired the workflow-execution-fan-out gap on its strength.

Nothing distinguishes a batch — the batch index is a discarded loop local and
the celery task id reaches only worker stdout — so assert the timing instead.
WorkflowFileExecution rows are created by the worker owning the batch rather
than up front by the dispatcher, so fanned out each row's window is one delay
wide, while serialised into one batch the windows open together and grow by a
delay per file. Comparing durations rather than overlap is the point:
serialised rows are created in one tight loop and overlap too.

That needs each completion to cost a known, measurable amount, so add
UNSTRACT_LLM_MOCK_DELAY alongside the existing mock hatch. It is inert without
UNSTRACT_LLM_MOCK_RESPONSE, since litellm ignores mock_delay on a real
completion, and a non-numeric value degrades to no delay rather than breaking
every completion in the worker.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
Manual runs reported green with nothing tested. paths-filter has no base to
diff on workflow_dispatch, so it resolves against the default branch and a
manual run on main diffs main against itself: every filter false, every tier
skipped, gate green. Filtering is a PR-time saving, so force both scopes true
off-PR and skip the filter step entirely there.

Gate the LLM mock on ENVIRONMENT as well as the mock var, so a stray variable
in a real deployment can't fake completions and their billing. Production sets
neither, so this fails closed. `development` is allowed alongside `test`
because that is what base compose sets on the two workers that run the
injection -- proven by the mock working today with the var set on exactly
those two services -- and the overlay now pins ENVIRONMENT=test on them so the
tier can't lose its mock to a base-compose edit. A refusal warns rather than
passing silently: setting the var at all means someone expected mocking.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
The rig's `down -v` destroys the containers, so the workflow step that ran
`docker compose logs` afterwards always wrote an empty file — and overwrote
nothing useful only because there was nothing else to write. Capture inside
`ComposeRuntime.down()` while the containers still exist, and drop the step.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The bound was half a delay — a 1s window on per-file durations of ~12s, which
measures runner contention as much as batching. One delay is the midpoint
between the two outcomes the check separates (~0 spread fanned out, ~2 delays
serialised), and the delay itself has to be large enough for that midpoint to
sit above the noise.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The worker log settles what the test could not: `Arranging 3 files into 3
batches` — the fan-out happens. Yet the per-file durations it was judged by came
out 6.6 / 18.1 / 22.9. Three tool containers running at once on a CI runner cost
more in contention than they gain in overlap, so genuine fan-out lands further
apart than serialised files do, and no threshold separates the two. Raising the
delay does not help: the contention scales with the parallelism being measured.

So the proof is removed rather than retuned, and with it the mock delay that
existed only to feed it -- every mocked completion across the whole e2e tier was
stalling for it. `workflow-execution-fan-out` becomes a declared gap (visible in
the report, non-gating), and the test keeps asserting the rejoin, which is
directly observable. Closing the gap wants a batch id on WorkflowFileExecution,
which is execution observability worth having on its own.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@ritwik-g ritwik-g left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved — all review findings addressed, verified against head 6dd8d756

Re-reviewed the full delta since my first pass (293695cff..6dd8d756, 579 insertions / 19 files), not just the P1 line. Every finding is resolved, and I verified the fixes against the actual head + its green CI run (29812358973), not the diff alone.

P1 — CI gate hole (blocker): fixed. report's gate now hard-fails on needs.changes.result (ci-test.yaml:340), so a broken filter job can no longer skip both tiers to green. The related workflow_dispatch main-diff-against-itself hole I'd flagged unverified is also closed — relevant/e2e_relevant are forced true off-PR and the paths-filter step is gated to pull_request.

P2s: all fixed.

  • poll_delivered rewritten — sleeps on the transient path and records the exception; 406 → clear "200 lost in transit" failure instead of a misleading == 422 assert; non-422/406 → explicit "unexpected poll status" rather than a poll-protocol assert (the 500 tool-not-found case now reads correctly).
  • Frontend e2e coverage restored — !frontend/** removed from e2e_relevant, so a frontend change again exercises the image build (bun run build).
  • groups.py docstring corrected to describe the runtime skip-on-gate-failure this PR implements.

P3s: all fixed — MinIO secure= now derives from UNSTRACT_MINIO_SCHEME (one scheme drives both halves); README no longer claims 10/20/30 for the streaming path and documents the test_connection/adapter-register-llm gap; the api_deployment fixture docstring now says per-xdist-worker.

Beyond the findings, and welcome: the mock now fails closed unless ENVIRONMENT ∈ {test, development} — a real hardening of the billing-corruption risk, with parametrized refusal tests (production/staging/unset all pass in CI). workflow-execution-fan-out was honestly demoted to a covered_by: [] gap rather than proven by timing that can't discriminate contention from serialisation — the right call under "no false greens." Blocked-group state now round-trips through junit into CI's rig report, with a __post_init__ invariant and new cascade/round-trip tests.

Verified on head 6dd8d756: full CI green (unit 490, integration, e2e 10m); the six execute-path e2e tests all genuinely executed and passed (workflow 1, prompt-studio 1, etl 1, api-deployment 3); the new ENVIRONMENT-guard tests pass including the production-refusal cases; no unresolved review threads.

Nit, non-blocking: the PR description still reads as if the e2e tests and critical-path flips are not in this PR ("will land next") — worth a rewrite before merge so the squash commit matches what shipped.

Approving the review. Not merging — this is stacked on #2164, and merge is a separate call.

@sonarqubecloud

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown
Contributor

Unstract test results

Per-group results

Status Group Tier Passed Failed Errors Skipped Duration (s)
e2e-api-deployment e2e 3 0 0 0 20.4
e2e-coowners e2e 1 0 0 0 1.5
e2e-etl e2e 1 0 0 0 8.4
e2e-login e2e 2 0 0 0 1.2
e2e-prompt-studio e2e 1 0 0 0 5.0
e2e-smoke e2e 2 0 0 0 1.0
e2e-workflow e2e 1 0 0 0 16.3
integration-backend integration 124 0 0 27 63.6
integration-connectors integration 1 0 0 7 7.5
unit-backend unit 151 0 0 0 23.9
unit-connectors unit 63 0 0 0 10.7
unit-core unit 27 0 0 0 1.4
unit-platform-service unit 15 0 0 0 3.0
unit-rig unit 76 0 0 0 4.8
unit-sdk1 unit 490 0 0 0 25.6
unit-workers unit 723 0 0 0 43.7
TOTAL 1681 0 0 34 237.8

Critical paths

⚠️ Critical paths not yet covered

  • workflow-execution-fan-out — Multi-file workflow execution fans out to file-processing workers and rejoins. (declared coverage: no groups declared)
✅ Covered critical paths
  • auth-login — covered by e2e-login
  • adapter-register-llm — covered by integration-backend
  • workflow-author — covered by integration-backend
  • co-owner-manage — covered by integration-backend, e2e-coowners
  • workflow-create-execute — covered by e2e-workflow
  • api-deployment-provision — covered by integration-backend
  • api-deployment-auth — covered by integration-backend
  • api-deployment-run — covered by e2e-api-deployment
  • prompt-studio-author — covered by integration-backend
  • prompt-studio-fetch-response — covered by e2e-prompt-studio
  • connector-register-test — covered by integration-backend
  • pipeline-etl-execute — covered by e2e-etl
  • usage-aggregate-read — covered by integration-backend
  • usage-token-tracking — covered by e2e-api-deployment
  • callback-result-delivery — covered by e2e-api-deployment

@chandrasekharan-zipstack
chandrasekharan-zipstack merged commit 66bfaf1 into main Jul 21, 2026
11 checks passed
@chandrasekharan-zipstack
chandrasekharan-zipstack deleted the feat/un-3636-execute-path-llm-mock branch July 21, 2026 09:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants