Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 0 additions & 15 deletions docker/docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@ services:
- ./workflow_data:/data
- ${TOOL_REGISTRY_CONFIG_SRC_PATH}:/data/tool_registry_config
environment:
- ENVIRONMENT=development
- APPLICATION_NAME=unstract-backend
labels:
- traefik.enable=true
Expand All @@ -60,7 +59,6 @@ services:
- rabbitmq
- db
environment:
- ENVIRONMENT=development
- APPLICATION_NAME=unstract-worker-metrics

labels:
Expand All @@ -83,7 +81,6 @@ services:
- redis
- rabbitmq
environment:
- ENVIRONMENT=development
- APPLICATION_NAME=unstract-worker-ide-callback
- WORKER_TYPE=ide_callback
- WORKER_NAME=ide-callback-worker
Expand All @@ -107,7 +104,6 @@ services:
ports:
- "5555:5555"
environment:
- ENVIRONMENT=development
- APPLICATION_NAME=unstract-celery-flower
volumes:
- unstract_data:/data
Expand All @@ -129,7 +125,6 @@ services:
- db
- rabbitmq
environment:
- ENVIRONMENT=development
- APPLICATION_NAME=unstract-celery-beat

# Frontend React app
Expand All @@ -143,7 +138,6 @@ services:
- backend
- reverse-proxy
environment:
- ENVIRONMENT=development
# Running platform version, surfaced on the profile page. Reuses the same
# ${VERSION} that tags the image, so it always matches what's deployed
# (and updates on RC->stable promotion, which only retags — doesn't rebuild).
Expand Down Expand Up @@ -219,7 +213,6 @@ services:
- redis
- rabbitmq
environment:
- ENVIRONMENT=development
- APPLICATION_NAME=unstract-worker-api-deployment-v2
- WORKER_TYPE=api_deployment
- CELERY_QUEUES_API_DEPLOYMENT=${CELERY_QUEUES_API_DEPLOYMENT:-celery_api_deployments}
Expand Down Expand Up @@ -252,7 +245,6 @@ services:
- redis
- rabbitmq
environment:
- ENVIRONMENT=development
- APPLICATION_NAME=unstract-worker-callback-v2
- WORKER_TYPE=callback
- WORKER_NAME=callback-worker-v2
Expand Down Expand Up @@ -294,7 +286,6 @@ services:
- redis
- rabbitmq
environment:
- ENVIRONMENT=development
- APPLICATION_NAME=unstract-worker-file-processing-v2
- WORKER_TYPE=file_processing
- WORKER_MODE=oss
Expand Down Expand Up @@ -332,7 +323,6 @@ services:
- redis
- rabbitmq
environment:
- ENVIRONMENT=development
- APPLICATION_NAME=unstract-worker-general-v2
- WORKER_TYPE=general
- WORKER_NAME=general-worker-v2
Expand Down Expand Up @@ -360,7 +350,6 @@ services:
- redis
- rabbitmq
environment:
- ENVIRONMENT=development
- APPLICATION_NAME=unstract-worker-notification-v2
- WORKER_TYPE=notification
- WORKER_NAME=notification-worker-v2
Expand Down Expand Up @@ -409,7 +398,6 @@ services:
- redis
- rabbitmq
environment:
- ENVIRONMENT=development
- APPLICATION_NAME=unstract-worker-log-consumer-v2
- WORKER_TYPE=log_consumer
- WORKER_NAME=log-consumer-worker-v2
Expand Down Expand Up @@ -458,7 +446,6 @@ services:
- redis
- rabbitmq
environment:
- ENVIRONMENT=development
- APPLICATION_NAME=unstract-worker-log-history-scheduler-v2
# Scheduler interval in seconds
- LOG_HISTORY_CONSUMER_INTERVAL=${LOG_HISTORY_CONSUMER_INTERVAL:-5}
Expand All @@ -483,7 +470,6 @@ services:
- redis
- rabbitmq
environment:
- ENVIRONMENT=development
- APPLICATION_NAME=unstract-worker-scheduler-v2
- WORKER_TYPE=scheduler
- WORKER_NAME=scheduler-worker-v2
Expand Down Expand Up @@ -529,7 +515,6 @@ services:
- rabbitmq
- platform-service
environment:
- ENVIRONMENT=development
- APPLICATION_NAME=unstract-worker-executor-v2
- WORKER_TYPE=executor
- WORKER_NAME=executor-worker-v2
Expand Down
2 changes: 1 addition & 1 deletion tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ The `platform` pytest fixture in `tests/e2e/conftest.py` reads those env vars; e

Execute-path e2e tests must not call a real provider, so the rig sets `UNSTRACT_LLM_MOCK_RESPONSE` (default `MOCK_LLM_OK`) before boot, for any runtime and treating an exported empty string as unset. The test overlay forwards it into the workers, and `unstract.sdk1.llm` passes it to litellm as `mock_response`: for the non-streaming completion path litellm returns the string verbatim with fixed usage (10 prompt / 20 completion / 30 total), so both the answer and the token counts are exact-assertable. Streaming (`stream_complete`) goes through a different litellm path whose usage differs, so don't assume 10/20/30 there. Sentinels like `litellm.RateLimitError` force error paths. Unset (the production default) the hook is a no-op.

Mocking needs two conditions, not one: `ENVIRONMENT` must also be `test` or `development`. Production sets neither variable, so a stray mock var alone cannot fake completions and their billing — and a refusal is logged rather than silent, since setting the var at all means someone expected mocking. `development` is allowed because that is what base compose sets on the workers that run the injection; the test overlay pins `ENVIRONMENT=test` on those same two workers explicitly, so the tier can't lose its mock to a base-compose edit.
The variable being set is the only condition, deliberately. A second gate on `ENVIRONMENT` was tried and removed: it did not defend against the case it was written for — a worker env block copied out of this overlay carries both variables together — and compose declared `ENVIRONMENT` on every service anyway, so any deployment derived from it satisfied the gate. Nothing read that variable, so it was dropped everywhere along with the gate. What remains is that the hatch is off unless someone sets it, and that it warns once per process while active. Making fake spend safe to *detect* rather than trying to prevent the config wants the usage record itself to say it was mocked.

A CI/dev override wins (the rig only fills an unset value). Running these tests under the rig **fails** if the var is missing; running **without** the rig just skips the execute-path tests — export it on both sides (your shell and the workers) if you boot the stack yourself.

Expand Down
12 changes: 1 addition & 11 deletions tests/compose/docker-compose.test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,33 +7,23 @@ services:
# contributor run without setting it.
image: unstract/backend:${UNSTRACT_TEST_VERSION:-latest}
environment:
- ENVIRONMENT=test
# Workers ask the backend for this, so this is the value that normally
# takes effect for fan-out.
- MAX_PARALLEL_FILE_BATCHES=${MAX_PARALLEL_FILE_BATCHES:-3}

platform-service:
image: unstract/platform-service:${UNSTRACT_TEST_VERSION:-latest}
environment:
- ENVIRONMENT=test

runner:
image: unstract/runner:${UNSTRACT_TEST_VERSION:-latest}
environment:
- ENVIRONMENT=test

# Execute-path e2e must never reach a real provider. The delay gives each
# mocked completion a known cost so per-file durations stay comparable, and
# ENVIRONMENT is the mock's second condition — set here rather than inherited
# so the tier can't lose its mock to a base-compose edit.
# Execute-path e2e must never reach a real provider.
worker-executor-v2:
environment:
- ENVIRONMENT=test
- UNSTRACT_LLM_MOCK_RESPONSE=${UNSTRACT_LLM_MOCK_RESPONSE:-}

worker-file-processing-v2:
environment:
- ENVIRONMENT=test
- UNSTRACT_LLM_MOCK_RESPONSE=${UNSTRACT_LLM_MOCK_RESPONSE:-}

# Only the worker's fallback if it can't reach the backend; kept in step so it
Expand Down
21 changes: 0 additions & 21 deletions unstract/sdk1/src/unstract/sdk1/llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,6 @@
# Lets tests force a deterministic completion without a provider or a secret.
# Unset in production, where this is a no-op.
_MOCK_RESPONSE_ENV = "UNSTRACT_LLM_MOCK_RESPONSE"
# Second condition on the hatch, so a stray mock var alone can't fake
# completions and their billing. Deployments that set neither fail closed.
_ENVIRONMENT_ENV = "ENVIRONMENT"
_MOCK_ALLOWED_ENVIRONMENTS = frozenset({"test", "development"})


@lru_cache(maxsize=1)
Expand All @@ -52,27 +48,10 @@ def _warn_mock_active() -> None:
)


@lru_cache(maxsize=1)
def _warn_mock_refused(environment: str) -> None:
# Loud rather than silent: the var being set at all means someone expected
# mocking, and they need to know why the bill is real.
logger.warning(
"%s is set but %s=%r is not one of %s — calling the provider for real.",
_MOCK_RESPONSE_ENV,
_ENVIRONMENT_ENV,
environment,
sorted(_MOCK_ALLOWED_ENVIRONMENTS),
)


def _inject_mock_response(completion_kwargs: dict[str, object]) -> None:
mock = os.getenv(_MOCK_RESPONSE_ENV)
if not mock or "mock_response" in completion_kwargs:
return
environment = os.getenv(_ENVIRONMENT_ENV, "").strip().lower()
if environment not in _MOCK_ALLOWED_ENVIRONMENTS:
_warn_mock_refused(environment)
return
_warn_mock_active()
completion_kwargs["mock_response"] = mock
Comment thread
chandrasekharan-zipstack marked this conversation as resolved.

Expand Down
46 changes: 0 additions & 46 deletions unstract/sdk1/tests/test_mock_response.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,13 +38,6 @@ def _inject(kwargs: dict[str, object]) -> dict[str, object]:
@pytest.fixture(autouse=True)
def _reset_warn_cache() -> None:
_load_llm_module()._warn_mock_active.cache_clear()
_load_llm_module()._warn_mock_refused.cache_clear()


@pytest.fixture(autouse=True)
def _allowed_environment(monkeypatch: pytest.MonkeyPatch) -> None:
"""The hatch needs a permitted ENVIRONMENT; the guard itself is tested below."""
monkeypatch.setenv("ENVIRONMENT", "test")


def test_inject_is_noop_when_env_unset(monkeypatch: pytest.MonkeyPatch) -> None:
Expand All @@ -67,45 +60,6 @@ def test_inject_sets_mock_response_when_env_set(
assert _inject({"model": "gpt-4o"})["mock_response"] == "canned answer"


@pytest.mark.parametrize("environment", ["production", "staging", "", " "])
def test_mock_refused_outside_allowed_environments(
monkeypatch: pytest.MonkeyPatch, environment: str
) -> None:
# The whole point of the guard: a stray mock var in a real deployment must
# not fake completions and their billing.
monkeypatch.setenv("UNSTRACT_LLM_MOCK_RESPONSE", "canned")
monkeypatch.setenv("ENVIRONMENT", environment)
assert _inject({"model": "gpt-4o"}) == {"model": "gpt-4o"}


def test_mock_refused_when_environment_unset(monkeypatch: pytest.MonkeyPatch) -> None:
# Production k8s sets no ENVIRONMENT at all, so unset must fail closed.
monkeypatch.setenv("UNSTRACT_LLM_MOCK_RESPONSE", "canned")
monkeypatch.delenv("ENVIRONMENT", raising=False)
assert _inject({"model": "gpt-4o"}) == {"model": "gpt-4o"}


def test_refusal_is_warned_so_a_real_bill_is_never_silent(
monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
) -> None:
monkeypatch.setenv("UNSTRACT_LLM_MOCK_RESPONSE", "canned")
monkeypatch.setenv("ENVIRONMENT", "production")
with caplog.at_level("WARNING", logger="unstract.sdk1.llm"):
_inject({"model": "gpt-4o"})
assert any("not one of" in r.message for r in caplog.records), caplog.text


@pytest.mark.parametrize("environment", ["test", "development", "TEST", " Development "])
def test_mock_applies_in_allowed_environments(
monkeypatch: pytest.MonkeyPatch, environment: str
) -> None:
# `development` is what base compose sets on the workers that run the
# injection; a guard that only accepted `test` would kill the local stack.
monkeypatch.setenv("UNSTRACT_LLM_MOCK_RESPONSE", "canned")
monkeypatch.setenv("ENVIRONMENT", environment)
assert _inject({"model": "gpt-4o"})["mock_response"] == "canned"


def test_inject_does_not_clobber_explicit_mock_response(
monkeypatch: pytest.MonkeyPatch,
) -> None:
Expand Down
Loading