Skip to content

v0.7.13: PII redaction via Presidio sidecar#5183

Open
TheodoreSpeaks wants to merge 8 commits into
mainfrom
staging
Open

v0.7.13: PII redaction via Presidio sidecar#5183
TheodoreSpeaks wants to merge 8 commits into
mainfrom
staging

Conversation

@TheodoreSpeaks

@TheodoreSpeaks TheodoreSpeaks commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

waleedlatif1 and others added 8 commits June 22, 2026 12:44
…e duplicate workflow-state cache, granular error boundaries (#5168)

* refactor(session): migrate SessionProvider to React Query useSessionQuery

Replace the hand-rolled useState/useEffect/loadSession session loading in
SessionProvider with a useSessionQuery() React Query hook. The SessionContext
shape is unchanged ({ data, isPending, error, refetch }) so no consumer changes.

The 'upgraded' path still forces a fresh DB read via
client.getSession({ query: { disableCookieCache: true } }) (refetch() cannot
pass disableCookieCache) and writes the result via queryClient.setQueryData,
then invalidates ['organizations']/['subscription'] as before.

* refactor(workflows): collapse duplicate workflow-state cache

The registry store fetched the GET /api/workflows/[id] envelope inline via
requestJson while useWorkflowState cached the same endpoint's mapped state
under workflowKeys.state(id) — two requests, two cache shapes, never
reconciled.

Collapse to one request + one cache entry keyed by workflowKeys.state(id):

- Add hooks/queries/utils/fetch-workflow-envelope.ts: a standalone
  fetchWorkflowEnvelope(id, signal) returning the full GetWorkflowResponseData.
  Standalone (not in workflows.ts) to avoid a store -> query-hook import cycle.
- useWorkflowState/useWorkflowStates now query the envelope and derive the
  mapped WorkflowState via select (mapWorkflowState), so consumers see the
  identical mapped shape from the shared entry.
- The store's loadWorkflowState reads via getQueryClient().fetchQuery({
  staleTime: 0 }) instead of raw requestJson — always-fresh (preserving the
  prior always-fetch boot/refresh semantics, incl. the socket
  handle-resource-event refresh path that has no separate state
  invalidation), in-flight deduped, writing into the same cache entry the
  hooks read.

Request-id staleness guard, deployment-cache priming, cross-store projection,
and the active-workflow-changed event are all preserved unchanged.

* fix(workspace): add granular error boundaries to logs, knowledge, and files panels

Scope a crash in one workspace panel to that panel instead of the whole
workspace shell. Each boundary reuses the shared ErrorState component and
mirrors the existing tables/settings error.tsx convention.

* refactor(unsubscribe): migrate page to React Query

Replace the hand-rolled useState+useEffect+requestJson server-state in the
unsubscribe page with React Query hooks. Add useUnsubscribe (validation/load
query, keyed by email+token, auto-runs on mount via enabled) and
useUnsubscribeMutation (unsubscribe action, reconciles cached preferences on
success) in hooks/queries/unsubscribe.ts with a hierarchical key factory.

Export UnsubscribeData/UnsubscribeActionResponse/UnsubscribeType type aliases
from the existing user contract; loading/error/success now derive from the
query and mutation objects with no local server-state mirror.

* test(frontend-arch): cover session race fix, workflow-state cache collapse, unsubscribe, error boundary

Add targeted tests for the four frontend-architecture refactors:
- session-provider: upgrade-path ordering — fresh disableCookieCache read wins
  over a late-resolving stale mount query (proves the cancelQueries guard)
- fetch-workflow-envelope + registry store: single shared state(id) cache entry,
  always-refetch (staleTime 0), request-id staleness guard
- unsubscribe: query enable-gating + mutation cache reconcile
- logs error boundary: renders ErrorState + reset wiring (also first ErrorState coverage)

* fix(session): harden upgrade path + address review feedback

- Reconcile plan surfaces after upgrade even when the fresh disableCookieCache
  read fails: invalidate ['organizations']/['subscription'] regardless of the
  bypass-read outcome (they read server truth, not the cookie cache). The valid
  cookie-cached session is still served, so a transient failure no longer signs
  the user out or leaves the just-upgraded plan looking stale. Org-activate
  fallback stays gated on having a session.
- Use a bare return in the cancelled branch of refreshAfterUpgrade (the caller
  discards the value) for clearer intent; caller coerces with ?? null.
- Make the upgrade tests deterministic: the mount mock honors the abort signal
  like the real fetch-backed client, and assertions read the query cache (the
  state cancelQueries/setQueryData/invalidation actually govern) instead of the
  async-rendered context value.

* refactor(session): break provider<->hook type cycle, fail-fast session query

Address review feedback:
- Move the AppSession type to lib/auth/session-response.ts (the module that
  produces it) so useSessionQuery and SessionProvider both import it from there,
  eliminating the provider <-> query-hook import cycle.
- Add retry: false to useSessionQuery, restoring the prior fail-fast contract
  (the global QueryClient default is retry: 1; an auth failure should surface
  immediately rather than retry a request that won't succeed).
- Return null (not the fetched value) from refreshAfterUpgrade's cancelled
  branch to make the cancellation contract explicit.
* feat(providers): add Sakana AI provider with Fugu models

OpenAI-compatible provider at https://api.sakana.ai/v1 (bearer auth).
Registers fugu (fast default) and fugu-ultra (reasoning flagship), both
1M context. BYOK-only, never hosted/auto-billed. Streaming, tool loop,
and response_format supported; attachments mirror deepseek (unsupported
in the current adapter).

* fix(providers): defer Sakana structured output until after tool loop

OpenAI-compatible backends reject a request carrying both response_format
and active tools/tool_choice. Mirror the LiteLLM pattern: withhold the
JSON schema while tools are active and apply it on a final tool-free call
(tool_choice: none) for both streaming and non-streaming paths.

* fix(providers): harden Sakana tool-loop error + final-stream tool_choice

- Rethrow tool-loop failures instead of swallowing them, so a failed run
  surfaces as a ProviderError rather than a partial success (matches LiteLLM).
- Force tool_choice: 'none' on the post-tool streaming pass so the model
  cannot emit fresh tool calls that the text-only stream adapter would drop.

* fix(providers): Sakana streaming usage + filtered-tools stream guard

- Pass stream_options: { include_usage: true } on both streaming calls so
  token/cost data is captured (the shared OpenAI-compatible stream helper
  only fills usage from chunk usage, which the API omits without the flag).
- Include !hasActiveTools in the early-stream guard so requests whose tools
  are all filtered out (e.g. usageControl 'none') still take the fast
  streaming path instead of the tool-loop path. Mirrors LiteLLM.

* fix(providers): answer every Sakana tool_call to keep message history valid

An assistant message lists all tool_calls, so a call for an unconfigured
tool must still get a matching `tool` response or the next request violates
the OpenAI message contract. Emit an error tool-result for unknown tools
instead of dropping them.

* test(session): de-flake SessionProvider normal-load test

flush() only drained microtasks, so the query->render update occasionally
lost the race and ctx.data was still null after the flush budget. Yield one
macrotask tick per flush so React Query's notifyManager and deferred renders
settle deterministically. Verified across repeated local runs.
…l-1 (#5173)

* feat(trigger): add trigger-eu-region flag to switch runs to eu-central-1

Global on/off feature flag routing every Trigger.dev run from the default
us-east-1 to eu-central-1 via the per-trigger region option, resolved at
each dispatch site through resolveTriggerRegion.

* test(trigger): mock resolveTriggerRegion in delete-async route test

The route now pulls in feature-flags (which imports isAppConfigEnabled from
env-flags); the test's partial env-flags mock made that access throw. Stub the
region module and assert the region option on the dispatch.
* feat(pi): add pi coding agent harness

* formatting

* update docs

* change version num

* guard to prevent prs on error

* update param visibility

* address security concerns

* fix tests

* reorder:
)

* feat(presidio): build & own combined analyzer+anonymizer image

Replace the stock mcr.microsoft.com/presidio-* sidecar images with a single
image we build and push to ECR/GHCR. A thin FastAPI service constructs one
AnalyzerEngine + one AnonymizerEngine at startup and serves both on port 3000
(/health, /supportedentities, /analyze, /anonymize) so the app needs one
PRESIDIO_URL. English only; pinned presidio 2.2.362 + en_core_web_lg 3.8.0.

Bakes in the native check-digit VIN recognizer and registers 12 English
recognizers Presidio ships but does not load by default (UK_NINO, AU_*, IN_*,
SG_*), taking the supported English set from 19 to 32.

* feat(presidio): add multi-language support (es/it/pl/fi)

Configure a multi-language spaCy NLP engine (en/es/it/pl/fi lg models) and
explicitly register the national-id recognizers Presidio ships but does not
load by default: ES_NIF/NIE, IT_FISCAL_CODE/DRIVER_LICENSE/VAT_CODE/PASSPORT/
IDENTITY_CARD, PL_PESEL, FI_PERSONAL_IDENTITY_CODE. Verified the NLP-engine +
explicit-registration path detects in-language (Finnish id, score 1.0).

* improvement(presidio): address review feedback

- Register VIN under all served languages, not just en (Bugbot: VIN missed for
  non-English language routing).
- Bump HEALTHCHECK start-period to 180s — five lg models load at import (Bugbot).
- Drop --no-cache-dir so the pip cache mount actually works (Greptile).
- Pydantic request models for /analyze + /anonymize so missing 'text' returns
  422 not 500; default operator 'type' to 'replace' instead of KeyError->500
  (Greptile).

* refactor(pii): rename presidio image artifacts to pii

Rename the image/repo/secret/files from 'presidio' to 'pii' for clarity — the
service does PII detection + anonymization (and backs the guardrails block's
block/mask), not just redaction, and 'pii' matches existing pii-* naming.

docker/presidio.Dockerfile -> docker/pii.Dockerfile
docker/presidio/ -> docker/pii/
ghcr.io/simstudioai/presidio -> .../pii
ECR_PRESIDIO secret -> ECR_PII (infra side already renamed)
No behavior change — paths/identifiers only.

* refactor(pii): move service to apps/pii, make image ECR-only

- Move server.py + requirements.txt from docker/pii/ to apps/pii/ (source belongs
  under apps/, matching app/realtime; Dockerfile stays in docker/). Add a minimal
  @sim/pii package.json so the apps/* bun workspace glob accepts the Python service.
- Repoint docker/pii.Dockerfile COPY paths to apps/pii/; rename the container user
  presidio -> pii.
- Drop GHCR for pii — it's a private ECS sidecar pulled from ECR, never published.
  Removed it from the arm64/manifest (GHCR-only) jobs and guarded the build-amd64
  tag step to skip GHCR when no ghcr_image is set.
* fix(pii): bind a configurable $PORT to avoid app :3000 collision

The pii image hardcoded uvicorn --port 3000 and ignored env. In the app ECS
task (awsvpc) all containers share one network namespace, and the app owns
3000 — so the sidecar must listen elsewhere (the stock presidio images honored
PORT and ran on 5002/5001). Bind ${PORT} (shell-form CMD), default 5001, and
update EXPOSE/HEALTHCHECK accordingly so the taskdef can set PORT=5001.

Verified: default binds 5001; PORT=5002 override binds 5002; /analyze works on
the overridden port.

* fix(pii): hardcode port 5001 (drop $PORT indirection)

EXPOSE can't be parameterized, so the configurable-PORT approach left EXPOSE
showing 5001 regardless (Greptile P2). We own both the image and the taskdef
and only ever need 5001, so hardcode it: exec-form CMD on 5001, EXPOSE 5001,
healthcheck on 5001. Runtime cmdline is identical to the verified ${PORT}
default (uvicorn ... --port 5001).
…-rule language) (#5174)

* fix(logs): run PII redaction over HTTP and fix Presidio provisioning

- resolve the guardrails venv via candidate paths and fail fast instead of
  silently falling back to system python3 (the misleading "Presidio not
  installed" that broke redaction and the guardrails block in deployed runtimes)
- install the en_core_web_lg spaCy model in setup.sh and app.Dockerfile
- route log redaction through an internal /api/guardrails/mask-batch endpoint
  so Presidio always runs in the app container, including async executions that
  persist inside the trigger.dev runtime

* fix(guardrails): chunk + time-bound internal PII mask requests

- chunk maskPIIBatchViaHttp by count (2000) and bytes (256KB) so large
  executions split across requests and never hit the contract's 100k cap
- add AbortSignal.timeout(45s) per request so a slow/unreachable app container
  aborts and the caller scrubs, instead of hanging the trigger.dev job
- catch maskPIIBatch failures in the route: log and return a structured 500
  (broken venv fails loudly server-side; caller still scrubs, no leak)
- add mask-client tests (order across chunks, count split, non-2xx, empty)

* fix(guardrails): mint internal token per mask request

A single token (5min TTL) could expire mid-batch when a large execution
fans out into many sequential chunk requests; mint one per request instead.

* feat(guardrails): run PII via Presidio sidecars + TS recognizer registry

- replace the per-call python3 subprocess (cold spaCy load every call) with
  two long-lived Presidio sidecars (analyzer + anonymizer) reached over HTTP;
  the app image no longer carries Python/Presidio/venv
- add PRESIDIO_ANALYZER_URL / PRESIDIO_ANONYMIZER_URL
- move VIN out of Python into a TS recognizer (check-digit validated) behind a
  CUSTOM_RECOGNIZERS registry so new custom detectors are one entry; masking is
  handled uniformly by the anonymizer
- drive the guardrails block's PII type picker from the shared pii-entities
  catalog (adds VIN, fixes drift) so block + Data Retention never diverge
- delete validate_pii.py, requirements.txt, setup.sh and the Dockerfile venv step

* fix(guardrails): bound-parallelize mask batch; refresh stale comments

- maskPIIBatch runs per-string sidecar calls with bounded concurrency (8) via
  mapWithConcurrency, so a chunk of many small leaves finishes within the 45s
  request timeout instead of aborting and scrubbing; order + fail-on-error kept
- drop stale comments referencing the deleted Python venv / 30s subprocess timeout

* refactor(guardrails): single Presidio image, native VIN, per-rule redaction language

- collapse the analyzer/anonymizer URLs into one PRESIDIO_URL (combined image
  serves /analyze + /anonymize)
- remove the TS VIN recognizer (vin.ts, recognizers.ts) — VIN is now native +
  multi-language in the image; validate_pii is a thin analyze→anonymize client
- trim KR_RRN/TH_TNIN from the catalog (no Korean/Thai model in the image)
- add per-rule redaction language: PII_LANGUAGES catalog drives the contract enum,
  the Data Retention rule modal, and the guardrails block dropdown; resolver +
  logger thread it through to maskPIIBatch (default en), so non-English entity
  rules (e.g. ES_NIF) actually fire instead of silently no-op'ing under en

* fix(guardrails): correct sidecar port (5001) + README for combined image

The combined Presidio image (docker/pii.Dockerfile) serves /analyze + /anonymize
on a single port 5001 with native VIN + multi-language recognizers. Fix the
PRESIDIO_URL default (was 5002) and rewrite the README, which still described two
stock containers and a TS VIN recognizer.

* fix(guardrails): coerce stored redaction language in the resolver

The persist-path resolver accepted any stored language string, so a stale/invalid
code (e.g. a dropped locale) would reach Presidio and scrub the log even though the
admin UI shows English. Coerce against the supported set via a shared
coercePiiLanguage helper (now reused by the data-retention route too), falling back
to en for unknown values.

* fix(guardrails): rename PRESIDIO_URL env var to PII_URL

Match the infra taskdef, which sets PII_URL on the app container for the
combined Presidio sidecar.
@TheodoreSpeaks TheodoreSpeaks requested a review from a team as a code owner June 23, 2026 09:57
@greptile-apps

greptile-apps Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Too many files changed for review. (111 files found, 100 file limit)

@vercel

vercel Bot commented Jun 23, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview, Comment Jun 23, 2026 9:57am

Request Review

@gitguardian

gitguardian Bot commented Jun 23, 2026

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
34197725 Triggered Basic Auth String 6333919 apps/sim/executor/handlers/pi/cloud-backend.test.ts View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@cursor

cursor Bot commented Jun 23, 2026

Copy link
Copy Markdown

PR Summary

High Risk
Touches PII detection/redaction, internal auth for batch masking, and high-risk execution paths (E2B sandbox with user keys, SSH to customer machines, session refresh after billing upgrades).

Overview
Adds a Presidio-based PII service (apps/pii) built as an ECR-only docker/pii.Dockerfile image (port 5001 in awsvpc). It exposes analyzer/anonymizer endpoints with multi-language spaCy models, extra national-ID recognizers, and a VIN check-digit recognizer. CI/image workflows build and push it without GHCR.

Log redaction can call a new internal POST /api/guardrails/mask-batch (internal JWT) so Trigger.dev and other runtimes mask via the app task where Presidio runs. Guardrails and enterprise data retention now share PII_ENTITY_GROUPS / PII_LANGUAGES, with per-rule language on retention rules.

Also ships the Pi Coding Agent workflow block (cloud E2B sandbox + PR flow with BYOK-only keys, or local SSH with Sim tools), docs, and executor backends. Smaller changes: session moves to React Query with safer post-upgrade refresh, Trigger.dev dispatches pass resolveTriggerRegion(), Pi’s tool picker greys out MCP/custom tools, and workspace error boundaries for files/knowledge/logs.

Reviewed by Cursor Bugbot for commit 8f312d2. Bugbot is set up for automated code reviews on this repo. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants