v0.7.13: PII redaction via Presidio sidecar#5183
Conversation
…e duplicate workflow-state cache, granular error boundaries (#5168) * refactor(session): migrate SessionProvider to React Query useSessionQuery Replace the hand-rolled useState/useEffect/loadSession session loading in SessionProvider with a useSessionQuery() React Query hook. The SessionContext shape is unchanged ({ data, isPending, error, refetch }) so no consumer changes. The 'upgraded' path still forces a fresh DB read via client.getSession({ query: { disableCookieCache: true } }) (refetch() cannot pass disableCookieCache) and writes the result via queryClient.setQueryData, then invalidates ['organizations']/['subscription'] as before. * refactor(workflows): collapse duplicate workflow-state cache The registry store fetched the GET /api/workflows/[id] envelope inline via requestJson while useWorkflowState cached the same endpoint's mapped state under workflowKeys.state(id) — two requests, two cache shapes, never reconciled. Collapse to one request + one cache entry keyed by workflowKeys.state(id): - Add hooks/queries/utils/fetch-workflow-envelope.ts: a standalone fetchWorkflowEnvelope(id, signal) returning the full GetWorkflowResponseData. Standalone (not in workflows.ts) to avoid a store -> query-hook import cycle. - useWorkflowState/useWorkflowStates now query the envelope and derive the mapped WorkflowState via select (mapWorkflowState), so consumers see the identical mapped shape from the shared entry. - The store's loadWorkflowState reads via getQueryClient().fetchQuery({ staleTime: 0 }) instead of raw requestJson — always-fresh (preserving the prior always-fetch boot/refresh semantics, incl. the socket handle-resource-event refresh path that has no separate state invalidation), in-flight deduped, writing into the same cache entry the hooks read. Request-id staleness guard, deployment-cache priming, cross-store projection, and the active-workflow-changed event are all preserved unchanged. * fix(workspace): add granular error boundaries to logs, knowledge, and files panels Scope a crash in one workspace panel to that panel instead of the whole workspace shell. Each boundary reuses the shared ErrorState component and mirrors the existing tables/settings error.tsx convention. * refactor(unsubscribe): migrate page to React Query Replace the hand-rolled useState+useEffect+requestJson server-state in the unsubscribe page with React Query hooks. Add useUnsubscribe (validation/load query, keyed by email+token, auto-runs on mount via enabled) and useUnsubscribeMutation (unsubscribe action, reconciles cached preferences on success) in hooks/queries/unsubscribe.ts with a hierarchical key factory. Export UnsubscribeData/UnsubscribeActionResponse/UnsubscribeType type aliases from the existing user contract; loading/error/success now derive from the query and mutation objects with no local server-state mirror. * test(frontend-arch): cover session race fix, workflow-state cache collapse, unsubscribe, error boundary Add targeted tests for the four frontend-architecture refactors: - session-provider: upgrade-path ordering — fresh disableCookieCache read wins over a late-resolving stale mount query (proves the cancelQueries guard) - fetch-workflow-envelope + registry store: single shared state(id) cache entry, always-refetch (staleTime 0), request-id staleness guard - unsubscribe: query enable-gating + mutation cache reconcile - logs error boundary: renders ErrorState + reset wiring (also first ErrorState coverage) * fix(session): harden upgrade path + address review feedback - Reconcile plan surfaces after upgrade even when the fresh disableCookieCache read fails: invalidate ['organizations']/['subscription'] regardless of the bypass-read outcome (they read server truth, not the cookie cache). The valid cookie-cached session is still served, so a transient failure no longer signs the user out or leaves the just-upgraded plan looking stale. Org-activate fallback stays gated on having a session. - Use a bare return in the cancelled branch of refreshAfterUpgrade (the caller discards the value) for clearer intent; caller coerces with ?? null. - Make the upgrade tests deterministic: the mount mock honors the abort signal like the real fetch-backed client, and assertions read the query cache (the state cancelQueries/setQueryData/invalidation actually govern) instead of the async-rendered context value. * refactor(session): break provider<->hook type cycle, fail-fast session query Address review feedback: - Move the AppSession type to lib/auth/session-response.ts (the module that produces it) so useSessionQuery and SessionProvider both import it from there, eliminating the provider <-> query-hook import cycle. - Add retry: false to useSessionQuery, restoring the prior fail-fast contract (the global QueryClient default is retry: 1; an auth failure should surface immediately rather than retry a request that won't succeed). - Return null (not the fetched value) from refreshAfterUpgrade's cancelled branch to make the cancellation contract explicit.
* feat(providers): add Sakana AI provider with Fugu models OpenAI-compatible provider at https://api.sakana.ai/v1 (bearer auth). Registers fugu (fast default) and fugu-ultra (reasoning flagship), both 1M context. BYOK-only, never hosted/auto-billed. Streaming, tool loop, and response_format supported; attachments mirror deepseek (unsupported in the current adapter). * fix(providers): defer Sakana structured output until after tool loop OpenAI-compatible backends reject a request carrying both response_format and active tools/tool_choice. Mirror the LiteLLM pattern: withhold the JSON schema while tools are active and apply it on a final tool-free call (tool_choice: none) for both streaming and non-streaming paths. * fix(providers): harden Sakana tool-loop error + final-stream tool_choice - Rethrow tool-loop failures instead of swallowing them, so a failed run surfaces as a ProviderError rather than a partial success (matches LiteLLM). - Force tool_choice: 'none' on the post-tool streaming pass so the model cannot emit fresh tool calls that the text-only stream adapter would drop. * fix(providers): Sakana streaming usage + filtered-tools stream guard - Pass stream_options: { include_usage: true } on both streaming calls so token/cost data is captured (the shared OpenAI-compatible stream helper only fills usage from chunk usage, which the API omits without the flag). - Include !hasActiveTools in the early-stream guard so requests whose tools are all filtered out (e.g. usageControl 'none') still take the fast streaming path instead of the tool-loop path. Mirrors LiteLLM. * fix(providers): answer every Sakana tool_call to keep message history valid An assistant message lists all tool_calls, so a call for an unconfigured tool must still get a matching `tool` response or the next request violates the OpenAI message contract. Emit an error tool-result for unknown tools instead of dropping them. * test(session): de-flake SessionProvider normal-load test flush() only drained microtasks, so the query->render update occasionally lost the race and ctx.data was still null after the flush budget. Yield one macrotask tick per flush so React Query's notifyManager and deferred renders settle deterministically. Verified across repeated local runs.
…l-1 (#5173) * feat(trigger): add trigger-eu-region flag to switch runs to eu-central-1 Global on/off feature flag routing every Trigger.dev run from the default us-east-1 to eu-central-1 via the per-trigger region option, resolved at each dispatch site through resolveTriggerRegion. * test(trigger): mock resolveTriggerRegion in delete-async route test The route now pulls in feature-flags (which imports isAppConfigEnabled from env-flags); the test's partial env-flags mock made that access throw. Stub the region module and assert the region option on the dispatch.
* feat(pi): add pi coding agent harness * formatting * update docs * change version num * guard to prevent prs on error * update param visibility * address security concerns * fix tests * reorder:
) * feat(presidio): build & own combined analyzer+anonymizer image Replace the stock mcr.microsoft.com/presidio-* sidecar images with a single image we build and push to ECR/GHCR. A thin FastAPI service constructs one AnalyzerEngine + one AnonymizerEngine at startup and serves both on port 3000 (/health, /supportedentities, /analyze, /anonymize) so the app needs one PRESIDIO_URL. English only; pinned presidio 2.2.362 + en_core_web_lg 3.8.0. Bakes in the native check-digit VIN recognizer and registers 12 English recognizers Presidio ships but does not load by default (UK_NINO, AU_*, IN_*, SG_*), taking the supported English set from 19 to 32. * feat(presidio): add multi-language support (es/it/pl/fi) Configure a multi-language spaCy NLP engine (en/es/it/pl/fi lg models) and explicitly register the national-id recognizers Presidio ships but does not load by default: ES_NIF/NIE, IT_FISCAL_CODE/DRIVER_LICENSE/VAT_CODE/PASSPORT/ IDENTITY_CARD, PL_PESEL, FI_PERSONAL_IDENTITY_CODE. Verified the NLP-engine + explicit-registration path detects in-language (Finnish id, score 1.0). * improvement(presidio): address review feedback - Register VIN under all served languages, not just en (Bugbot: VIN missed for non-English language routing). - Bump HEALTHCHECK start-period to 180s — five lg models load at import (Bugbot). - Drop --no-cache-dir so the pip cache mount actually works (Greptile). - Pydantic request models for /analyze + /anonymize so missing 'text' returns 422 not 500; default operator 'type' to 'replace' instead of KeyError->500 (Greptile). * refactor(pii): rename presidio image artifacts to pii Rename the image/repo/secret/files from 'presidio' to 'pii' for clarity — the service does PII detection + anonymization (and backs the guardrails block's block/mask), not just redaction, and 'pii' matches existing pii-* naming. docker/presidio.Dockerfile -> docker/pii.Dockerfile docker/presidio/ -> docker/pii/ ghcr.io/simstudioai/presidio -> .../pii ECR_PRESIDIO secret -> ECR_PII (infra side already renamed) No behavior change — paths/identifiers only. * refactor(pii): move service to apps/pii, make image ECR-only - Move server.py + requirements.txt from docker/pii/ to apps/pii/ (source belongs under apps/, matching app/realtime; Dockerfile stays in docker/). Add a minimal @sim/pii package.json so the apps/* bun workspace glob accepts the Python service. - Repoint docker/pii.Dockerfile COPY paths to apps/pii/; rename the container user presidio -> pii. - Drop GHCR for pii — it's a private ECS sidecar pulled from ECR, never published. Removed it from the arm64/manifest (GHCR-only) jobs and guarded the build-amd64 tag step to skip GHCR when no ghcr_image is set.
* fix(pii): bind a configurable $PORT to avoid app :3000 collision
The pii image hardcoded uvicorn --port 3000 and ignored env. In the app ECS
task (awsvpc) all containers share one network namespace, and the app owns
3000 — so the sidecar must listen elsewhere (the stock presidio images honored
PORT and ran on 5002/5001). Bind ${PORT} (shell-form CMD), default 5001, and
update EXPOSE/HEALTHCHECK accordingly so the taskdef can set PORT=5001.
Verified: default binds 5001; PORT=5002 override binds 5002; /analyze works on
the overridden port.
* fix(pii): hardcode port 5001 (drop $PORT indirection)
EXPOSE can't be parameterized, so the configurable-PORT approach left EXPOSE
showing 5001 regardless (Greptile P2). We own both the image and the taskdef
and only ever need 5001, so hardcode it: exec-form CMD on 5001, EXPOSE 5001,
healthcheck on 5001. Runtime cmdline is identical to the verified ${PORT}
default (uvicorn ... --port 5001).
…-rule language) (#5174) * fix(logs): run PII redaction over HTTP and fix Presidio provisioning - resolve the guardrails venv via candidate paths and fail fast instead of silently falling back to system python3 (the misleading "Presidio not installed" that broke redaction and the guardrails block in deployed runtimes) - install the en_core_web_lg spaCy model in setup.sh and app.Dockerfile - route log redaction through an internal /api/guardrails/mask-batch endpoint so Presidio always runs in the app container, including async executions that persist inside the trigger.dev runtime * fix(guardrails): chunk + time-bound internal PII mask requests - chunk maskPIIBatchViaHttp by count (2000) and bytes (256KB) so large executions split across requests and never hit the contract's 100k cap - add AbortSignal.timeout(45s) per request so a slow/unreachable app container aborts and the caller scrubs, instead of hanging the trigger.dev job - catch maskPIIBatch failures in the route: log and return a structured 500 (broken venv fails loudly server-side; caller still scrubs, no leak) - add mask-client tests (order across chunks, count split, non-2xx, empty) * fix(guardrails): mint internal token per mask request A single token (5min TTL) could expire mid-batch when a large execution fans out into many sequential chunk requests; mint one per request instead. * feat(guardrails): run PII via Presidio sidecars + TS recognizer registry - replace the per-call python3 subprocess (cold spaCy load every call) with two long-lived Presidio sidecars (analyzer + anonymizer) reached over HTTP; the app image no longer carries Python/Presidio/venv - add PRESIDIO_ANALYZER_URL / PRESIDIO_ANONYMIZER_URL - move VIN out of Python into a TS recognizer (check-digit validated) behind a CUSTOM_RECOGNIZERS registry so new custom detectors are one entry; masking is handled uniformly by the anonymizer - drive the guardrails block's PII type picker from the shared pii-entities catalog (adds VIN, fixes drift) so block + Data Retention never diverge - delete validate_pii.py, requirements.txt, setup.sh and the Dockerfile venv step * fix(guardrails): bound-parallelize mask batch; refresh stale comments - maskPIIBatch runs per-string sidecar calls with bounded concurrency (8) via mapWithConcurrency, so a chunk of many small leaves finishes within the 45s request timeout instead of aborting and scrubbing; order + fail-on-error kept - drop stale comments referencing the deleted Python venv / 30s subprocess timeout * refactor(guardrails): single Presidio image, native VIN, per-rule redaction language - collapse the analyzer/anonymizer URLs into one PRESIDIO_URL (combined image serves /analyze + /anonymize) - remove the TS VIN recognizer (vin.ts, recognizers.ts) — VIN is now native + multi-language in the image; validate_pii is a thin analyze→anonymize client - trim KR_RRN/TH_TNIN from the catalog (no Korean/Thai model in the image) - add per-rule redaction language: PII_LANGUAGES catalog drives the contract enum, the Data Retention rule modal, and the guardrails block dropdown; resolver + logger thread it through to maskPIIBatch (default en), so non-English entity rules (e.g. ES_NIF) actually fire instead of silently no-op'ing under en * fix(guardrails): correct sidecar port (5001) + README for combined image The combined Presidio image (docker/pii.Dockerfile) serves /analyze + /anonymize on a single port 5001 with native VIN + multi-language recognizers. Fix the PRESIDIO_URL default (was 5002) and rewrite the README, which still described two stock containers and a TS VIN recognizer. * fix(guardrails): coerce stored redaction language in the resolver The persist-path resolver accepted any stored language string, so a stale/invalid code (e.g. a dropped locale) would reach Presidio and scrub the log even though the admin UI shows English. Coerce against the supported set via a shared coercePiiLanguage helper (now reused by the data-retention route too), falling back to en for unknown values. * fix(guardrails): rename PRESIDIO_URL env var to PII_URL Match the infra taskdef, which sets PII_URL on the app container for the combined Presidio sidecar.
|
Too many files changed for review. ( |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 34197725 | Triggered | Basic Auth String | 6333919 | apps/sim/executor/handlers/pi/cloud-backend.test.ts | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secret safely. Learn here the best practices.
- Revoke and rotate this secret.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
PR SummaryHigh Risk Overview Log redaction can call a new internal Also ships the Pi Coding Agent workflow block (cloud E2B sandbox + PR flow with BYOK-only keys, or local SSH with Sim tools), docs, and executor backends. Smaller changes: session moves to React Query with safer post-upgrade refresh, Trigger.dev dispatches pass Reviewed by Cursor Bugbot for commit 8f312d2. Bugbot is set up for automated code reviews on this repo. Configure here. |
Uh oh!
There was an error while loading. Please reload this page.