Add inline script environment persistence (PEP 723 PR 7/16) - #1697
Conversation
2719448 to
dbb5c72
Compare
|
AI agent review findings:
I validated the PR with Reviewed and posted by Eleanor's AI agent. |
dbb5c72 to
7c4f36e
Compare
Thanks Eleanor Boyd (@eleanorjboyd), both findings are valid.
|
105d9ca to
a39fd2e
Compare
|
will give a thumbs up to after the prior one merges as I assume that will create merge conflicts |
3072ce4 to
628c729
Compare
|
🔒 Automated review in progress — Rich Chiodo (@rchiodo) is auto-reviewing this PR. |
Rich Chiodo (rchiodo)
left a comment
There was a problem hiding this comment.
Approved via Review Center.
Persist and safely rehydrate per-script environment associations. Harden selection races, cache-lock handling, corrupt-state repair, scope validation, and central batch selection consistency without globally serializing environment operations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1a9f6ba1-9bd3-4664-bc25-a0d34d7a2e91
Keep inline-script manager settings scoped to exact script projects and prevent stale warm validation from superseding a newer selection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1a9f6ba1-9bd3-4664-bc25-a0d34d7a2e91
Keep exact script-project settings authoritative while routing active inline selections ahead of containing-project defaults, and retain strict PEP 440 validation for persisted associations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1a9f6ba1-9bd3-4664-bc25-a0d34d7a2e91
Prevent stale central selection publication, align inline environment identity with executable paths, normalize stale cleanup paths, and extract named association types. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1a9f6ba1-9bd3-4664-bc25-a0d34d7a2e91
Publish same-path version rebuilds, emit completed multi-manager unset groups consistently, and preserve cold associations across transient rehydration failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1a9f6ba1-9bd3-4664-bc25-a0d34d7a2e91
2e6ce79 to
1262b21
Compare
| } | ||
|
|
||
| const key = project ? project.uri.toString() : 'global'; | ||
| if (scope instanceof Uri) { |
There was a problem hiding this comment.
Issue · Please address or respond
Switching away from the inline manager only removes central in-memory routing; it does not unset the inline manager's persisted script association. A later restart or inline discovery can therefore resurrect the superseded selection. Clear the durable inline association after the replacement manager succeeds.
bde7cf8
into
microsoft:main
…1723) > Part of #1602 (PEP 723 inline script env support). Design doc: #1601. > > Builds on the merged creation, uv-fallback, and persistence work in #1656, #1696, and #1697. This branch is rebased on current `main`. ### Roadmap context This is **PR 15 of 16** in the PEP 723 inline-script roadmap. It adds the remaining lifecycle telemetry for the internal manager without changing routing, creation, or cache policy. | Phase 5: Lifecycle and polish | PR | Status | |---|---|---| | | PR 13: clear inline-script cache | separate | | | PR 14: opportunistic TTL eviction | follow-up | | | **PR 15: lifecycle telemetry** | **this PR** | | | PR 16: status-bar decision | resolved; no code PR | ### Why this PR The manager can create, reuse, and persist inline-script environments, including consent-gated uv/Python fallback, but those lifecycle outcomes are not observable as a coherent feature funnel. This PR adds low-cardinality telemetry that answers: - whether setup built a new environment or reused a validated cache entry; - how long actual environment construction took; - how many normalized dependencies were involved; and - which stable failure category prevented setup. The instrumentation is deliberately attached to the underlying coalesced operation rather than every caller, and it excludes script/package/interpreter content. ### What this PR does **Adds three typed lifecycle events** - `inlineScript.envCreated` - `inlineScript.envReuseHit` - `inlineScript.envError` The telemetry constants and GDPR declarations use typed event/property mappings so call sites cannot send undeclared fields. **Reports verified creation only** - Emits `envCreated` only after environment creation succeeds, ownership/version validation passes, and `.meta.json` is persisted. - Measures the actual build/rebuild interval rather than lock waiting or cache inspection. - Reports normalized, deduplicated dependency count rather than raw dependency values. - Emits one creation event for the underlying coalesced operation, even when multiple callers await it. **Reports validated cache reuse only** - Emits `envReuseHit` only after sidecar, interpreter, ownership, and compatibility checks complete successfully. - Does not report a hit for uncertain, stale, malformed, or partially validated entries. - Preserves the existing cache reuse and `lastUsedAt` behavior. **Classifies lifecycle failures without leaking details** `envError` uses stable low-cardinality categories for outcomes such as: - environment discovery failure; - no compatible Python; - declined compatible-Python installation; - uv/Python installation failure; - general setup, cache-validation, or metadata-persistence failure; - package-install cancellation; - cache-lock timeout; and - retained, orphaned, or otherwise unavailable lock state. Errors remain logged through the existing paths; telemetry sends no exception text. **Preserves uv consent and compatibility behavior** - Adds detailed internal uv lookup/install result types so the manager can distinguish `available`, `declined`, `failed`, and `installed` outcomes. - Keeps existing compatibility wrappers for callers that only need boolean/path results. - Does not change prompt text, consent requirements, install selectors, restart-required behavior, or uv's existing telemetry. **Keeps event emission coalesced and deterministic** - Same-key concurrent creation/reuse callers receive one lifecycle result for the shared operation. - Cache inspection, lock acquisition, fallback selection, cancellation, and cleanup retain their existing control flow. - Non-applicable `create()` calls and pre-validation exits do not emit success-shaped lifecycle events. ### Event payloads and privacy | Event | Data | |---|---| | `inlineScript.envCreated` | build duration; normalized dependency count | | `inlineScript.envReuseHit` | normalized dependency count | | `inlineScript.envError` | stable failure category | The events send no: - script URI or filesystem path; - requirement or dependency value; - package name; - Python/interpreter version; - cache key; - prompt text; or - exception/error-message content. Script-controlled metadata therefore cannot create unbounded telemetry dimensions. ### Lifecycle examples ```text validated cache hit → emit one inlineScript.envReuseHit ``` ```text cache miss or stale entry → start build timer → build + install dependencies → validate ownership/version → persist sidecar → emit one inlineScript.envCreated ``` ```text no compatible Python → user dismisses consent prompt → emit one inlineScript.envError with the declined category ``` ### Tests Coverage includes: - creation emitted only after verified sidecar persistence; - validated reuse and rebuild behavior; - one event for concurrent/coalesced callers; - duration boundaries excluding lock wait/cache inspection; - normalized/deduplicated dependency counts; - discovery, compatibility, consent, installation, cancellation, setup, and lock categories; - no lifecycle telemetry for non-applicable calls; - direct detailed uv results for available, declined, failed, and installed outcomes; and - preservation of existing uv wrapper behavior. Validation on the rebased branch: - `npm run compile-tests` - `npm run compile` - `npm run lint` - focused lifecycle/detailed-uv suites: 39 passing The full Windows unit run reaches 1612 passing and 5 pending; the existing concurrent `writeMetaJson` rename test can still intermittently fail with `EPERM` on Windows. That writer is unchanged by this PR and the same failure is reproducible on `main`. ### Performance - No activation work, scan, timer, watcher, or new filesystem operation is introduced. - Instrumentation performs constant-size event construction around operations that already occur. - Coalesced setup emits once rather than once per waiter. - No user-controlled strings are normalized or transmitted beyond the dependency count already needed for the cache operation. ### User impact **No default-path user impact.** The inline manager remains behind the undeclared, default-off `python-envs.inlineScripts.enabled` flag. When the internal flag is manually enabled, prompts, environment creation/reuse, cancellation, error propagation, and cache behavior remain unchanged. This PR only records privacy-safe lifecycle outcomes. ### Scope and follow-up This PR intentionally does **not** implement: - script detection or automatic routing; - activation-time cache discovery; - project registration or user-facing setup UX; - cache clearing or TTL eviction; or - status-bar behavior. The telemetry is ready for those later entry points to consume once the feature is intentionally exposed. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3cb82ae9-7424-40a4-9156-8c54ac6e0895 Copilot-Session: 6b12d843-8011-4bfc-9ba9-f75761eadee2
> Part of #1602 (PEP 723 inline script env support). Design doc: #1601. > > Builds on the merged persistence work in #1697 and is rebased on current `main`. ### Roadmap context This is **PR 13 of 16** in the PEP 723 inline-script roadmap. It adds the explicit, user-confirmed cache lifecycle operation that the later TTL work will reuse. | Phase 5: Lifecycle and polish | PR | Status | |---|---|---| | | PR 7: persistence (`get` / `set` + Memento) | merged (#1697) | | | **PR 13: clear inline-script cache** | **this PR** | | | PR 14: opportunistic 14-day TTL eviction | follow-up | | | PR 15: lifecycle telemetry | separate | | | PR 16: status-bar decision | resolved; no code PR | ### Why this PR The extension can create extension-owned inline-script environments and persist script associations, but it has no complete way to remove that state. Clearing only the files would leave Memento associations and `pythonProjects` entries pointing at deleted interpreters; clearing only settings would leave disk usage behind. This PR adds one coordinated lifecycle operation covering: - extension-owned cache entries; - persisted and in-memory script associations; - active selection events; and - generated inline-script project settings. Because this is destructive and the cache is shared by extension-host processes, the implementation is intentionally fail-closed around path ownership and locks. ### What this PR does **Adds an internal, confirmation-gated clear command** - Registers `python-envs.clearScriptEnvCache` only while the hidden inline-script feature flag is enabled. - Does not contribute the command to `package.json` or the Command Palette before rollout. - Shows a modal warning covering cached environments, associations, and project entries. - Cancelling the prompt performs no filesystem, state, or settings changes. - Runs cache cleanup before settings removal, so failed/partial cache cleanup does not silently rewrite project configuration. **Keeps generic cache clearing behavior safe** - The existing public `python-envs.clearCache` command continues to clear existing non-inline managers. - It skips the preview inline manager because the generic path has no inline-specific confirmation or project-settings lifecycle. - The dedicated command invokes the inline manager directly and performs the complete cleanup transaction. **Serializes in-process maintenance** - Adds a manager-local maintenance queue and barrier. - `create()`, `get()`, and `set()` cannot observe or mutate half-cleared state. - A clear request refuses to begin when creation already started. - A creation request that arrives after clear begins waits for maintenance to settle. - Multiple maintenance requests are serialized without globally serializing unrelated managers. **Coordinates deletion across extension hosts** - Acquires and holds each cache entry's cross-process lock through deletion. - Classifies locks as missing, held, retained, stale, orphaned, malformed, or unavailable. - Uses PID liveness to distinguish a live owner from a stale owner. - Makes retained markers generation-specific by preserving the owner's PID/nonce. - Reclaims only the exact stale/retained generation marker that was inspected. - If another process replaces that generation before the atomic claim, reclamation loses safely and touches nothing. - Ambiguous legacy fixed `retained` markers remain recognizable but are conservatively not reclaimed. **Validates every destructive path** Before deleting an entry, cleanup verifies that: - global storage and `script-envs-v1` are normal directories rather than symlinks/junctions; - the versioned cache root is the expected direct child of global storage; - neither path is a filesystem root or dangerously shallow; - physical `realpath` containment matches the expected ownership boundary; - the cache root has not changed since the cleanup snapshot; and - the target entry is a normal direct-child directory inside that same physical root. The entry lock is acquired first, then root and entry ownership are revalidated immediately before removal. **Keeps state consistent through partial failures** - Attempts cache entries independently and records successful removals. - Aggregates and surfaces deletion/persistence failures rather than returning success-shaped output. - Invalidates only associations whose environment was removed or is definitively missing. - Preserves associations for cache entries that could not safely be removed. - Cancels pending rehydration for invalidated scripts. - Clears warm environment and validation caches. - Advances association revisions so stale async work cannot restore removed selections. - Emits `onDidChangeEnvironment` only for selections actually invalidated. **Removes generated inline project settings safely** - Resolves `pythonProjects` entries independently from global, workspace, and workspace-folder sources. - Removes only entries whose manager is the inline-script manager. - Preserves non-inline duplicate entries and higher-precedence overrides. - Handles same relative paths across multiple workspace roots. - Aggregates global/workspace updates so each shared scope is written once. - Unloads only loaded projects that have no remaining configuration source. ### Cleanup semantics | Condition | Behavior | |---|---| | Entry is unlocked and physically owned | Lock, revalidate, delete | | Lock belongs to a live process | Refuse that deletion | | Exact stale/retained generation can be claimed | Reclaim, acquire a fresh lock, delete | | Lock is unavailable, malformed, orphaned, or legacy-ambiguous | Preserve entry and surface failure | | Root or entry is redirected/outside ownership boundary | Refuse deletion | | One entry fails after another was removed | Preserve valid survivors; invalidate removed associations; report aggregate failure | | Persistence update fails after disk cleanup | Keep in-memory state consistent and surface the persistence error | | Cache is already absent | Clear stale associations safely; remain idempotent | ### Example ```text Clear Script Environment Cache → modal confirmation → enter manager maintenance barrier → verify physical cache root → acquire exact per-entry lock → revalidate ownership immediately before deletion → delete safe entries → reconcile Memento + in-memory selections + events → remove generated inline pythonProjects settings ``` ### Tests Coverage includes: - prompt cancellation and command ordering; - generic clear behavior with the preview manager absent/present; - in-process create/clear ordering; - live, stale, retained, orphaned, malformed, unavailable, and legacy lock states; - exact-generation reclamation and delayed-reclaimer/new-creator races; - holding entry locks through deletion; - unsafe, shallow, redirected, symlinked, and root-swapped paths; - successful, missing-cache, idempotent, and partial-failure cleanup; - persistence failures and pending-rehydration races; - global/workspace/workspace-folder setting precedence; - multi-root projects and same-path entries; and - default-off command registration. Validation on the rebased branch: - `npm run compile-tests` - `npm run compile` - `npm run lint` - focused lock/cache-clear/settings/command suites: 38 passing The full Windows unit run reaches 1638 passing and 5 pending; the existing concurrent `writeMetaJson` rename test can still intermittently fail with `EPERM` on Windows. That writer is unchanged by this PR and the same failure is reproducible on `main`. ### Performance - No activation scan, timer, or background maintenance is added. - All work is initiated by the internal clear command. - Per-entry locks avoid globally serializing independent environment creation across extension hosts. - The maintenance barrier exists only inside the enabled inline manager and is active only during cleanup. ### User impact **No default-path user impact.** The manager and command remain behind the undeclared, default-off `python-envs.inlineScripts.enabled` flag, and the command is not publicly contributed. When the internal flag is manually enabled, the existing generic cache command still behaves as before for non-inline managers. Inline cleanup is available only through the dedicated confirmed lifecycle. ### Scope and follow-up This PR intentionally does **not** implement: - automatic routing or project registration; - activation-time discovery; - silent/opportunistic deletion; - TTL expiration; or - lifecycle telemetry. PR 14 will reuse this safety and state-cleanup foundation to remove entries whose `lastUsedAt` exceeds the planned 14-day TTL. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3cb82ae9-7424-40a4-9156-8c54ac6e0895 Copilot-Session: 6b12d843-8011-4bfc-9ba9-f75761eadee2
> Part of #1602 (PEP 723 inline script env support). Design doc: #1601. > > Builds on the merged persistence work in #1697 and is rebased on current `main`. ### Roadmap context This is **PR 8 of 16** in the PEP 723 inline-script roadmap. It makes extension-owned cached environments discoverable after activation; automatic script routing remains a separate follow-up. | Phase 2: Manager | PR | Status | |---|---|---| | | PR 4: `InlineScriptEnvManager` skeleton | merged (#1610) | | | PR 5a: generic env-creation utilities | merged (#1651) | | | PR 5b: inline-script cache + interpreter utilities | merged (#1655) | | | PR 5c: `create()` happy path | merged (#1656) | | | PR 6: `create()` uv-install fallback | merged (#1696) | | | PR 7: persistence (`get` / `set` + Memento) | merged (#1697) | | | **PR 8: activation-time discovery** | **this PR** | | | PR 9: route PEP 723 scripts to the inline manager | follow-up | ### Why this PR PR 7 persists and lazily rehydrates a specific script's selected inline environment when the manager is asked for that script. The manager still has no global inventory, however: `getEnvironments()` returns `[]`, and cached environments are not published after an extension-host restart unless a script-specific lookup happens to reconstruct one. This PR implements the activation-discovery portion of Q4 in the design: - walk the versioned global cache without blocking extension activation; - validate each cache entry before exposing it as a `PythonEnvironment`; - publish add/remove events as valid entries appear or become definitively invalid; - preserve previously known entries through locks and transient filesystem failures; - retry bounded transient work without creating a permanent watcher or polling loop. ### What this PR does **Defers discovery until after manager registration** - Starts activation discovery with `setImmediate()` after the manager is registered. - Leaves the feature-gate-off path unchanged: no manager registration, cache scan, timer, or filesystem work. - Keeps extension activation non-blocking. **Publishes a global discovered collection** - `getEnvironments('all')` returns a copy of the validated cache collection. - Other scopes remain empty because inline cache entries live in extension global storage rather than belonging to one workspace. - Results are sorted deterministically. - Collection reconciliation emits precise `EnvironmentChangeKind.add` and `remove` events only when manager identity, executable path, or Python version materially changes. **Validates every cache entry before publication** A candidate must: 1. be a normal directory rather than a symlink; 2. be unlocked and physically contained beneath the expected cache root; 3. have a valid `.meta.json` sidecar; 4. have an available cached launcher and base interpreter; 5. resolve into a real `PythonEnvironment`; 6. prove direct-child cache ownership through `sysPrefix`; and 7. match the Python release recorded in the sidecar. Discovery is read-only. It does not delete, rebuild, or rewrite invalid entries. **Distinguishes definitive invalidity from uncertainty** - Missing, malformed, unowned, or version-mismatched entries are omitted. - Locked entries and transient I/O/resolver/ownership failures preserve the previously published environment and request another activation pass. - A missing cache root is treated as an empty cache. - An unavailable cache root preserves the current collection and remains retryable. **Handles concurrent cache changes** - Activation scans compare an initial and final directory snapshot so an entry created during the scan triggers a follow-up pass. - Final cache-root absence publishes an empty collection immediately; transient final-read failures preserve the prior collection until a retry. - Per-entry filesystem fingerprints detect a cache directory rebuilt under the same key even when its name is unchanged. - Locks use the cache entry name/hash as their identity. - Published entries use that same cache-key identity rather than mixing lexical cache paths with canonical `sysPrefix` paths, avoiding false removal through symlink/junction path differences. - Lock probing uses `lstat`; only `ENOENT` means unlocked. `EIO`, access failures, and other uncertain states fail closed. **Coalesces refresh work without weakening activation discovery** - Concurrent compatible refreshes share one scan. - Activation joining an in-flight explicit refresh receives one snapshot-aware follow-up rather than accepting the explicit pass's weaker single-snapshot semantics. - Explicit `refresh()` cancels activation retries, performs one settled pass, and schedules no delayed work afterward. **Retries activation discovery only while useful** - Follow-up delays are bounded at 1 second, 5 seconds, and 30 seconds. - Retries are requested for locks, transient failures, or a changed cache snapshot. - No permanent filesystem watcher or unbounded polling loop is introduced. - Disposal cancels pending timers and prevents in-flight scans from publishing late results. **Strengthens the Windows usability guard** - Windows cache validation now checks both the cached environment launcher and the base interpreter referenced by `pyvenv.cfg`. - A surviving base interpreter no longer makes an environment with a missing `Scripts\python.exe` appear usable. ### Discovery semantics | Cache state | Behavior | |---|---| | Valid sidecar, launcher, interpreter, ownership, and version | Publish environment | | Entry is locked or being built | Preserve previous publication; retry | | Filesystem/resolver state is temporarily unavailable | Preserve previous publication; retry | | Entry appears during the scan | Schedule snapshot-aware follow-up | | Missing or malformed sidecar | Omit/remove from collection | | Missing launcher/base interpreter | Omit/remove from collection | | Ownership or recorded-version mismatch | Omit/remove from collection | | Cache root is absent | Publish empty collection | ### Example ```text extension activation → register InlineScriptEnvManager → defer one event-loop turn → scan <globalStorage>/script-envs-v1 → validate sidecar + launcher + ownership + version → publish valid cached environments through getEnvironments('all') → retry only if the scan observed a lock, transient state, or snapshot change ``` This inventory does not associate an environment with a script. PR 7 owns persisted script associations, and PR 9 will use those associations for automatic per-file routing. ### Tests Coverage includes: - deferred feature-gated activation startup; - valid cache discovery and `all`-scope publication; - missing, malformed, unavailable, non-directory, and symlinked entries; - missing and non-regular Windows launchers; - lock preservation, including unavailable (`EIO`) lock probes; - canonical `sysPrefix` versus lexical cache-root identities; - add/remove event reconciliation; - concurrent refresh coalescing; - activation joining an explicit refresh; - cache entries created during a scan; - builds completing after the short retry window; - bounded retry exhaustion; - explicit single-pass refresh behavior; and - disposal during in-flight scans and pending retries. Validation on the rebased branch: - `npm run compile-tests` - `npm run compile` - `npm run lint` - focused activation-discovery/cache-launcher/registration suites The full Windows unit run reaches 1613 passing and 5 pending; the existing concurrent `writeMetaJson` rename test can still intermittently fail with `EPERM` on Windows. That writer is unchanged by this PR and the same failure is reproducible on `main`. ### Performance - Activation is deferred and never waits for discovery. - Cache scans are coalesced. - Retries are bounded and stop after a stable pass. - Explicit refresh remains single-pass. - There is no persistent watcher, unbounded polling, or per-document work. ### User impact **No default-path user impact.** The manager and discovery remain behind the undeclared, default-off `python-envs.inlineScripts.enabled` flag. With the internal flag manually enabled, valid cached inline environments become available through the manager after restart. This PR does not automatically select one for a script and introduces no public command, setting, picker item, project registration, cache deletion, TTL cleanup, or telemetry. ### Scope and follow-up This PR intentionally does **not** implement: - automatic PEP 723 script routing (PR 9); - exact script project registration (PR 10); - CodeLens or bulk setup UX (PRs 11-12); - cache clearing or TTL eviction (PRs 13-14); or - lifecycle telemetry (PR 15). PR 7 (#1697) is merged, and this branch is rebased on current `main`. Automatic routing can follow independently after this manager-discovery layer lands. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3cb82ae9-7424-40a4-9156-8c54ac6e0895 Copilot-Session: 6b12d843-8011-4bfc-9ba9-f75761eadee2
### Roadmap context This is **PR 9 of 16** in the PEP 723 inline-script roadmap and the repository's intentional routing cutover. Earlier manager PRs create, discover, validate, and persist environments; this PR makes normal per-file environment lookup use them when a saved script association is proven current. | Phase 2/3 | PR | Status | |---|---|---| | | PR 7: per-script persistence | merged (#1697) | | | PR 8: activation-time cache discovery | merged (#1722) | | | **PR 9: automatic per-script routing** | **this PR** | | | PR 10: exact script project registration | follow-up | | | PRs 11-12: CodeLens and bulk setup UX | follow-up | | | PRs 17-19: Pylance/Python debugger integration | cross-repository follow-up | ### Why this PR The inline manager can already create/reuse environments, persist a script association, rediscover cache entries after restart, clear state safely, and report lifecycle telemetry. Nothing automatically selects that manager for a Python file, however. A script continues to use its normal project/default environment unless another component directly asks the inline manager. Automatic routing must be stricter than checking whether a file contains PEP 723 metadata or whether Memento contains an executable path. It must prove both: ```text current saved PEP 723 metadata exists + a persisted environment association is valid for that metadata and cache generation = route this file to InlineScriptEnvManager ``` If either proof is absent, dirty, stale, transiently unavailable, or superseded by an explicit choice, routing falls through to the existing project/default manager. ### What this PR does **Adds an activation-scoped routing registry** - Latches the hidden feature flag once per extension-host activation. - Creates one registry shared by the detector, inline manager, and central environment-manager router. - Tracks saved metadata, normalized metadata identity, metadata revision, and validated-association state independently per script. - Emits metadata and routeability transitions. - Requires both current metadata and a validated association before `shouldRoute(uri)` becomes true. **Turns the existing lazy detector into a routing input only when enabled** - Reads saved PEP 723 metadata on open and save. - Replays documents that were already open when activation occurred. - Tracks loose local `.py` files for routing while retaining the existing workspace-only telemetry population. - Withholds routing for restored dirty documents. - Invalidates routing immediately when an edit can touch the metadata block, while body-only edits retain routing. - Uses source-compatible offsets so BOM- and CRLF-prefixed metadata edits are classified correctly. - Uses per-URI read generations so an older open read cannot overwrite newer saved metadata. - Clears metadata and associations for deleted or renamed paths. When no routing registry exists, the detector retains its existing telemetry-only listeners, coalescing, workspace filter, and event behavior. **Defines a stable metadata identity** - Uses normalized dependencies plus trimmed `requires-python`. - Ignores dependency ordering, equivalent package-name spelling, and the unrelated `[tool]` table. - Does not include script paths, so equivalent scripts can share cache entries. **Adds durable cache provenance** - Sidecars store bounded SHA-256 hashes of metadata identities that were proven for the cache entry. - Same-key coalesced creators and cache-reuse callers merge their identities under the existing cache-entry lock. - The list is deduplicated and capped at 128 entries. - No package, requirement, script path, URI, or metadata content is stored in provenance. - Older sidecars without provenance remain usable through conservative cache-key and Python-constraint proof; additional-package environments remain non-routeable until explicitly proven. **Makes sidecar replacement recoverable** - Serializes same-sidecar writes in-process; production callers remain protected by the cross-process cache-entry lock. - Uses native rename with a unique backup on Windows replacement contention. - Restores the previous sidecar after failed replacement and retains the only known-good backup if restoration is uncertain. - Lock-held cache inspection can recover a strict, regular, size-bounded, schema-valid backup compatible with the selected base interpreter. - Invalid, temporary, symlinked, oversized, unsupported, or incompatible-only artifacts retain normal stale/uncertain behavior. **Upgrades persisted associations from path-only values** Each current record contains: ```text schemaVersion environmentPath metadataBinding: legacy | pending(sourceIdentity) | matched(sourceIdentity) ``` - `legacy`: old string association; remains retrievable but is not automatically routeable. - `pending`: the environment is proven, but saved metadata has not yet been durably matched (for example, selection while the document is dirty). - `matched`: saved metadata identity and cache provenance agree. - Future schema records are preserved rather than destructively rewritten. - Malformed requested entries can be repaired without discarding unrelated valid/future records. **Validates associations before routing** Validation requires: 1. current saved metadata; 2. an absolute persisted executable path; 3. no active cache build/lock; 4. a resolvable environment; 5. physical ownership by the expected extension cache entry; 6. compatible Python/requires-python state; 7. a matching persisted metadata binding; and 8. sidecar proof for the current metadata identity. Definitively stale associations are conditionally removed. Locked, transient, uncertain, or future-schema states are preserved but remain non-routeable. **Protects asynchronous validation from stale results** - Metadata revisions remain monotonic across cleared-state tombstones, preventing an old validator from winning after clear/restore of the same identity. - A changed saved metadata identity invalidates routeability synchronously before provenance validation begins. - Association revisions ensure unset/replacement wins over old rehydration or validation. - Same-script rehydration and metadata refreshes are coalesced. - Pending bindings are promoted to matched only if both metadata and association revisions are still current. - Failed persistence never publishes successful routeability. **Adds central manager routing with explicit precedence** | Priority | Manager source | |---|---| | 1 | Exact per-script project setting | | 2 | Explicit in-session non-inline override | | 3 | Validated inline-script association | | 4 | Existing project/default manager setting | | 5 | Existing cached project/global manager | - A user explicitly selecting a non-inline manager is not immediately overridden by automatic routing. - Clearing that override re-resolves and publishes through the newly effective manager, with reserved operation ordering so a newer selection still wins. - Inline selections are keyed by normalized script path, not containing project, so multiple scripts in one workspace can remain independent. - Inline selections are not published as active until routeability is validated. - Routeability changes refresh the appropriate manager and emit the existing active-environment transition. - Invalidated metadata falls back to the existing project/default environment and removes only the inline active-selection entry. - Existing revision checks continue preventing slow refreshes from overwriting newer selections. ### User flows **Previously configured script after restart** ```text open saved script → detector publishes metadata identity → manager loads persisted association → validate executable + ownership + sidecar provenance → registry marks association routeable → central environment lookup switches this file to the inline manager ``` **Metadata edit** ```text edit inside metadata block → routeability clears immediately → normal project/default manager remains active → save reads new metadata and revalidates → inline routing returns only if the existing association is still proven ``` **New unassociated script** Opening a new PEP 723 script does **not** silently select a matching cache entry. It remains on normal routing until the future explicit setup action (PR 11/12) creates or reuses an environment and persists the association. After that, this PR provides automatic routing. ### Routing and failure semantics | State | Behavior | |---|---| | Saved metadata + matched/proven association | Route inline | | Metadata but no association | Existing project/default routing | | Association but metadata unknown/dirty | Existing project/default routing | | Body-only edit | Preserve current routing | | Metadata edit or invalid saved block | Clear inline routing and fall back | | Explicit non-inline selection | Explicit override wins | | Legacy string association | Preserve/retrieve, but do not auto-route after restart | | Pending binding with matching saved proof | Atomically promote to matched | | Locked/transient cache state | Preserve association; remain non-routeable; retry later | | Same cache path rebuilt for different provenance | Reject old routing proof | | Rename/delete | Clear old path's association and routeability | ### Review guide The production changes are easiest to review in this order: 1. **Routing state and detection** - `routingRegistry.ts` - `activation.ts` - `metadata.ts` - `lazyDetector.ts` 2. **Cache provenance and durability** - `cacheLayout.ts` - creation/reuse sections of `inlineScript/envManager.ts` 3. **Persisted binding and validation state machine** - persistence/rehydration/metadata-refresh sections of `inlineScript/envManager.ts` 4. **Behavior cutover** - `envManagers.ts` - `extension.ts` - `inlineScript/main.ts` More than half of the diff is deterministic unit coverage for dirty/save/restart and async race behavior. ### Tests Coverage includes: - activation flag latching and default-off registration; - telemetry-only detector equivalence when routing is absent; - open/save ordering, restored dirty editors, loose files, CRLF edits, body-only edits, rename, and delete; - metadata-only, association-only, exact-setting, explicit-override, route-on, and fallback behavior; - independent same-workspace script selections and batch operations; - legacy, pending, matched, malformed, and future persisted records; - restart rehydration and delayed routeability; - same-path cache rebuilds and additional-package provenance; - same-key coalesced creation and one-event telemetry semantics; - stale metadata/association races against save, unset, and replacement; - sidecar write contention, restoration, backup recovery, and invalid artifact handling; - cache-clear partial failure with versioned associations; and - public API last-known fallback/event ordering. Validation on the final rebased tree: - `npm run compile-tests` - `npm run compile` - `npm run lint` - `npm run unittest`: **1,817 passing, 6 pending** ### Performance - The feature remains hidden and default-off. - With the flag off, no routing registry, manager, discovery timer, routing file listeners, Memento read, sidecar/cache work, or routing telemetry is added. - The existing telemetry detector keeps its prior listener/coalescing behavior; routing checks are optional branches. - With the flag on, metadata reads remain bounded to the first 8 KiB and occur only for opened/saved local Python files. - Same-script validation and same-key creation are coalesced. - Warm association validation is throttled. - Discovery retries remain bounded and cache maintenance remains serialized only within the inline manager. ### Privacy and safety - Telemetry remains count/boolean/category based and sends no URI, path, dependency, requirement, Python version, cache key, or error text. - Provenance uses local SHA-256 metadata identities rather than raw metadata. - Physical cache ownership and executable checks remain fail-closed. - Symlinked/unowned cache entries are never made routeable. - Destructive cache behavior remains exclusively in the merged confirmation-gated cleanup lifecycle. ### User impact **No default-path user impact.** `python-envs.inlineScripts.enabled` remains undeclared and defaults to false: - no visible setting or autocomplete; - no new command, menu, CodeLens, picker, or status-bar surface; - no inline manager registration; - no routing registry, discovery, persistence, or cache work; and - existing project/default routing remains unchanged. For developers manually enabling the hidden flag, existing proven script associations route automatically and fall back conservatively when proof is absent. New scripts still require the future explicit setup UX. ### Scope and follow-up This PR intentionally does **not** implement: - automatic setup for a newly encountered script; - exact generated script project registration (PR 10); - CodeLens or bulk setup UX (PRs 11-12); - Pylance per-file interpreter support (PRs 17-18); or - the Python debugger per-file resolver fix (PR 19). Those later PRs can build on this guarded routing layer without changing its validation contract. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6b12d843-8011-4bfc-9ba9-f75761eadee2
Roadmap context
This is PR 7 of 16 in the PEP 723 inline-script roadmap. It adds durable per-script environment associations to the internal manager.
InlineScriptEnvManagerskeletoncreate()happy pathcreate()uv-install fallbackget/set+ Memento)Why this PR
PR 5 can build or reuse an inline-script environment, but the manager does not remember that the resulting environment belongs to a particular script. After an extension-host restart, the in-memory association is gone.
This PR implements the persistence portion of Q4 in the design:
requires-pythonmetadata before returning an environment;What this PR does
Implements per-script
set()file:URIs and rejects invalid or mixed scopes atomically.onDidChangeEnvironmentonly for effective changes.create()behavior separate: creation alone does not implicitly establish a persisted association.Implements per-script
get()requires-pythonagainst the reconstructed Python version before returning it.Safely rehydrates persisted associations
PythonEnvironmentand confirms that it belongs to the expected extension-owned cache entry.Validates warm in-memory associations
Protects persistence and selection from races
Updates central active-environment tracking
Example
Given two scripts in the same workspace:
PR 7 stores and retrieves those associations independently. Selecting the environment for
import.pydoes not overwrite the last-known environment forreport.py.After restart:
If
report.pylater changes fromrequires-python = ">=3.11"to">=3.13", its persisted Python 3.12 environment is no longer returned as compatible.Persistence and failure semantics
Tests
Coverage includes:
npm run compile-tests,npm run lint, the full unit suite, and the focused persistence/central-manager suites are clean.Performance
User impact
No default-path user impact yet. This completes an internal Phase 2 manager capability. Automatic routing and user-facing entry points arrive in later roadmap PRs.
Once routing is wired, script-specific selections will survive extension-host restarts and remain independent even for multiple scripts in the same workspace.
Merge order
The core persistence behavior depends on the merged manager skeleton (#1610). This branch is rebased on current
main; PR 8 and PR 9 build on this capability.