Skip to content

Add inline script environment persistence (PEP 723 PR 7/16) - #1697

Merged
Stella Huang (StellaHuang95) merged 5 commits into
microsoft:mainfrom
StellaHuang95:pep723-pr7-persistence
Aug 14, 2026
Merged

Add inline script environment persistence (PEP 723 PR 7/16)#1697
Stella Huang (StellaHuang95) merged 5 commits into
microsoft:mainfrom
StellaHuang95:pep723-pr7-persistence

Conversation

@StellaHuang95

Copy link
Copy Markdown
Contributor

Part of #1602 (PEP 723 inline script env support). Design doc: #1601.

This replaces the earlier closed draft #1653 with the finalized implementation rebased on main.

Roadmap context

This is PR 7 of 16 in the PEP 723 inline-script roadmap. It adds durable per-script environment associations to the internal manager.

Phase 2: Manager PR Status
PR 4: InlineScriptEnvManager skeleton merged (#1610)
PR 5a: generic env-creation utilities merged (#1651)
PR 5b: inline-script cache + interpreter utilities merged (#1655)
PR 5c: create() happy path merged (#1656)
PR 6: create() uv-install fallback open (#1696)
PR 7: persistence (get / set + Memento) this PR
PR 8: activation-time discovery follow-up
PR 9: route PEP 723 scripts to the inline manager follow-up

Why this PR

PR 5 can build or reuse an inline-script environment, but the manager does not remember that the resulting environment belongs to a particular script. After an extension-host restart, the in-memory association is gone.

This PR implements the persistence portion of Q4 in the design:

  • maintain an independent environment association for each script;
  • persist script-to-environment executable paths in workspace Memento;
  • lazily and safely rehydrate those associations;
  • re-check current requires-python metadata before returning an environment;
  • report changes so the central environment API can update its last-known state.

What this PR does

Implements per-script set()

  • Accepts one or more local file: URIs and rejects invalid or mixed scopes atomically.
  • Validates that selected environments are owned inline-script cache entries.
  • Persists a normalized script path → environment executable path mapping under a dedicated Memento key.
  • Supports assigning and unassigning individual scripts or batches.
  • Updates in-memory state and emits onDidChangeEnvironment only for effective changes.
  • Leaves the existing create() behavior separate: creation alone does not implicitly establish a persisted association.

Implements per-script get()

  • Reads current PEP 723 metadata before returning an association.
  • Keeps unreadable or temporarily invalid script metadata from destructively clearing state.
  • Returns an in-memory association when valid.
  • Lazily reconstructs persisted environments after restart instead of resolving every script during activation.
  • Re-checks requires-python against the reconstructed Python version before returning it.

Safely rehydrates persisted associations

  • Requires an absolute executable path.
  • Preserves associations while their cache entry is locked or being created.
  • Verifies that the executable exists and is a regular file.
  • Resolves it into a PythonEnvironment and confirms that it belongs to the expected extension-owned cache entry.
  • Removes only definitively stale associations; transient filesystem or resolver failures remain retryable.
  • Emits a change event when a slow rehydration eventually succeeds, including when the public API's initial one-second wait has already elapsed.

Validates warm in-memory associations

  • Periodically revalidates cached associations without performing full resolution on every lookup.
  • Detects executables deleted while VS Code remains open.
  • Detects an environment rebuilt at the same cache path with a different Python version.
  • Preserves busy/locked entries instead of misclassifying them as stale.
  • Coalesces simultaneous validations for the same script.
  • Retains the existing environment object when resolution produces only a new generated ID for the same Python, avoiding false changes and duplicate ID-keyed resources.

Protects persistence and selection from races

  • Serializes Memento read-modify-write operations so concurrent script selections cannot lose one another.
  • Uses per-script association revisions so an older rehydration cannot overwrite a newer selection or unset.
  • Removes stale persisted values conditionally, only if the inspected path is still current.
  • Keeps failed persistence writes from changing in-memory state or emitting success-shaped events.
  • Does not globally serialize unrelated environment operations.

Updates central active-environment tracking

  • Keys inline-script selections by normalized script path rather than containing project, so two scripts in one workspace can retain different environments.
  • Uses per-scope revisions and manager identity checks so slow refreshes cannot overwrite newer selections.
  • Ensures failed selections and failed refreshes do not discard a valid in-flight refresh.
  • Groups same-manager batch unsets and calls the manager once with the complete URI array.
  • Updates central cache entries and events only after the manager operation succeeds.
  • Attributes inline-script change events to the script URI rather than the containing project URI.

Example

Given two scripts in the same workspace:

tools/report.py → Python 3.12 inline environment
tools/import.py → Python 3.13 inline environment

PR 7 stores and retrieves those associations independently. Selecting the environment for import.py does not overwrite the last-known environment for report.py.

After restart:

get(report.py)
→ read persisted executable
→ verify cache ownership and current metadata
→ resolve environment
→ cache and return it

If report.py later changes from requires-python = ">=3.11" to ">=3.13", its persisted Python 3.12 environment is no longer returned as compatible.

Persistence and failure semantics

Condition Behavior
Executable exists and cache ownership is valid Rehydrate and return
Cache entry is locked/in progress Preserve association; retry later
Resolver fails transiently Preserve association; retry later
Executable is definitively missing and unlocked Remove stale association and notify
A newer selection wins during rehydration Discard the stale result
Memento write fails Keep previous in-memory/persisted selection and propagate the error

Tests

Coverage includes:

  • assign, retrieve, unset, and batch persistence;
  • restart-time lazy rehydration and delayed success events;
  • metadata compatibility changes;
  • missing, malformed, unowned, busy, and transient cache states;
  • warm deletion and same-path rebuild detection;
  • concurrent persistence, rehydration, validation, selection, and unset races;
  • failed Memento writes;
  • strict URI-scope validation;
  • independent same-project script selections;
  • stale and failed central refresh ordering;
  • atomic same-manager batch unsets.

npm run compile-tests, npm run lint, the full unit suite, and the focused persistence/central-manager suites are clean.

Performance

  • Rehydration is lazy rather than activation-blocking.
  • Warm associations are cached and validation is throttled.
  • Same-script rehydration and validation work is coalesced.
  • Queues cover only shared persistence and mutation ordering; unrelated script reads and environment-manager operations remain independent.

User impact

No default-path user impact yet. This completes an internal Phase 2 manager capability. Automatic routing and user-facing entry points arrive in later roadmap PRs.

Once routing is wired, script-specific selections will survive extension-host restarts and remain independent even for multiple scripts in the same workspace.

Merge order

The core persistence behavior depends on the merged manager skeleton (#1610). This branch is rebased on current main; PR 8 and PR 9 build on this capability.

@eleanorjboyd

Copy link
Copy Markdown
Member

AI agent review findings:

  1. Selecting an inline-script environment currently uses the normal project settings persistence path, which can save inline-script as the containing project's default manager. Subsequent ordinary-file and unassociated-script lookups then route to InlineScriptEnvManager and can return no environment. Conversely, loose-script associations have no persisted manager routing and become unreachable after restart. This routing needs to remain per-script rather than changing the project-wide manager setting.

  2. validateCachedAssociation() captures the association revision only after awaiting the busy and filesystem checks, while still validating the earlier cached environment. An explicit selection during either await can therefore update the revision, after which the stale cached result is treated as current and may overwrite the newer selection. Please capture the revision with the original cached read, before any await, and add a regression test for that interleaving.

I validated the PR with npm run compile-tests, npm run compile, and the focused inline-script/last-known-environment tests (78 passing, 1 pending).

Reviewed and posted by Eleanor's AI agent.

@StellaHuang95

Copy link
Copy Markdown
Contributor Author

AI agent review findings:

  1. Selecting an inline-script environment currently uses the normal project settings persistence path, which can save inline-script as the containing project's default manager. Subsequent ordinary-file and unassociated-script lookups then route to InlineScriptEnvManager and can return no environment. Conversely, loose-script associations have no persisted manager routing and become unreachable after restart. This routing needs to remain per-script rather than changing the project-wide manager setting.
  2. validateCachedAssociation() captures the association revision only after awaiting the busy and filesystem checks, while still validating the earlier cached environment. An explicit selection during either await can therefore update the revision, after which the stale cached result is treated as current and may overwrite the newer selection. Please capture the revision with the original cached read, before any await, and add a regression test for that interleaving.

I validated the PR with npm run compile-tests, npm run compile, and the focused inline-script/last-known-environment tests (78 passing, 1 pending).

Reviewed and posted by Eleanor's AI agent.

Thanks Eleanor Boyd (@eleanorjboyd), both findings are valid.

  1. Per-script routing/settings:  pm.get(scriptUri)  can currently return the containing project, causing  setEnvironment()  to persist  inline-script  as that whole project’s manager. I’ll update both single and batch selection paths so an inline-script manager setting is persisted only when the resolved project URI exactly matches the script URI. Otherwise, only the inline manager’s per-script Memento association will be updated.
    The loose-script restart-routing gap is real, but the roadmap intentionally separates that work: PR7 persists and rehydrates the association inside the manager, PR9 routes known scripts to that manager, and PR10 registers scripts as individual projects so settings can be persisted per script. I’ll prevent the incorrect project-wide write in this PR without pulling all PR9/PR10 routing into it.
  2. Cached-association revision: Agreed. The revision is captured too late, after asynchronous lock/filesystem checks, so validation of an old cached environment can accidentally adopt a newer selection’s revision. I’ll capture the revision together with the cached environment before the first  await , pass it through validation and stale cleanup, and add a regression test where a new selection occurs while validation is paused.

@eleanorjboyd

Copy link
Copy Markdown
Member

will give a thumbs up to after the prior one merges as I assume that will create merge conflicts

@rchiodo

Rich Chiodo (rchiodo) commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

🔒 Automated review in progress — Rich Chiodo (@rchiodo) is auto-reviewing this PR.

Comment thread src/managers/builtin/inlineScript/envManager.ts Outdated
Comment thread src/features/envManagers.ts Outdated
Comment thread src/features/envManagers.ts Outdated
Comment thread src/managers/builtin/inlineScript/envManager.ts
Comment thread src/managers/builtin/inlineScript/envManager.ts
Comment thread src/managers/builtin/inlineScript/envManager.ts
Comment thread src/managers/builtin/inlineScript/envManager.ts
@rchiodo Rich Chiodo (rchiodo) added the review-auto:changes-requested Automated review: posted blocking findings to address. label Aug 14, 2026
Comment thread src/features/envManagers.ts
Comment thread src/features/envManagers.ts
Comment thread src/managers/builtin/inlineScript/envManager.ts
Comment thread src/managers/builtin/inlineScript/envManager.ts

@rchiodo Rich Chiodo (rchiodo) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

@rchiodo Rich Chiodo (rchiodo) removed the review-auto:changes-requested Automated review: posted blocking findings to address. label Aug 14, 2026
@rchiodo Rich Chiodo (rchiodo) added the review-auto:approved Automated review: no blocking findings (approval posted). label Aug 14, 2026
Persist and safely rehydrate per-script environment associations. Harden selection races, cache-lock handling, corrupt-state repair, scope validation, and central batch selection consistency without globally serializing environment operations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1a9f6ba1-9bd3-4664-bc25-a0d34d7a2e91
Keep inline-script manager settings scoped to exact script projects and prevent stale warm validation from superseding a newer selection.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 1a9f6ba1-9bd3-4664-bc25-a0d34d7a2e91
Keep exact script-project settings authoritative while routing active inline selections ahead of containing-project defaults, and retain strict PEP 440 validation for persisted associations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 1a9f6ba1-9bd3-4664-bc25-a0d34d7a2e91
Prevent stale central selection publication, align inline environment identity with executable paths, normalize stale cleanup paths, and extract named association types.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 1a9f6ba1-9bd3-4664-bc25-a0d34d7a2e91
Publish same-path version rebuilds, emit completed multi-manager unset groups consistently, and preserve cold associations across transient rehydration failures.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 1a9f6ba1-9bd3-4664-bc25-a0d34d7a2e91
}

const key = project ? project.uri.toString() : 'global';
if (scope instanceof Uri) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue · Please address or respond

Switching away from the inline manager only removes central in-memory routing; it does not unset the inline manager's persisted script association. A later restart or inline discovery can therefore resurrect the superseded selection. Clear the durable inline association after the replacement manager succeeds.

@rchiodo Rich Chiodo (rchiodo) added review-auto:changes-requested Automated review: posted blocking findings to address. and removed review-auto:approved Automated review: no blocking findings (approval posted). labels Aug 14, 2026
@StellaHuang95
Stella Huang (StellaHuang95) merged commit bde7cf8 into microsoft:main Aug 14, 2026
47 of 48 checks passed
@StellaHuang95
Stella Huang (StellaHuang95) deleted the pep723-pr7-persistence branch August 14, 2026 21:59
Stella Huang (StellaHuang95) added a commit that referenced this pull request Aug 19, 2026
…1723)

> Part of #1602 (PEP 723 inline script env support). Design doc: #1601.
>
> Builds on the merged creation, uv-fallback, and persistence work in
#1656, #1696, and #1697. This branch is rebased on current `main`.

### Roadmap context

This is **PR 15 of 16** in the PEP 723 inline-script roadmap. It adds
the remaining lifecycle telemetry for the internal manager without
changing routing, creation, or cache policy.

| Phase 5: Lifecycle and polish | PR | Status |
|---|---|---|
| | PR 13: clear inline-script cache | separate |
| | PR 14: opportunistic TTL eviction | follow-up |
| | **PR 15: lifecycle telemetry** | **this PR** |
| | PR 16: status-bar decision | resolved; no code PR |

### Why this PR

The manager can create, reuse, and persist inline-script environments,
including consent-gated uv/Python fallback, but those lifecycle outcomes
are not observable as a coherent feature funnel.

This PR adds low-cardinality telemetry that answers:

- whether setup built a new environment or reused a validated cache
entry;
- how long actual environment construction took;
- how many normalized dependencies were involved; and
- which stable failure category prevented setup.

The instrumentation is deliberately attached to the underlying coalesced
operation rather than every caller, and it excludes
script/package/interpreter content.

### What this PR does

**Adds three typed lifecycle events**

- `inlineScript.envCreated`
- `inlineScript.envReuseHit`
- `inlineScript.envError`

The telemetry constants and GDPR declarations use typed event/property
mappings so call sites cannot send undeclared fields.

**Reports verified creation only**

- Emits `envCreated` only after environment creation succeeds,
ownership/version validation passes, and `.meta.json` is persisted.
- Measures the actual build/rebuild interval rather than lock waiting or
cache inspection.
- Reports normalized, deduplicated dependency count rather than raw
dependency values.
- Emits one creation event for the underlying coalesced operation, even
when multiple callers await it.

**Reports validated cache reuse only**

- Emits `envReuseHit` only after sidecar, interpreter, ownership, and
compatibility checks complete successfully.
- Does not report a hit for uncertain, stale, malformed, or partially
validated entries.
- Preserves the existing cache reuse and `lastUsedAt` behavior.

**Classifies lifecycle failures without leaking details**

`envError` uses stable low-cardinality categories for outcomes such as:

- environment discovery failure;
- no compatible Python;
- declined compatible-Python installation;
- uv/Python installation failure;
- general setup, cache-validation, or metadata-persistence failure;
- package-install cancellation;
- cache-lock timeout; and
- retained, orphaned, or otherwise unavailable lock state.

Errors remain logged through the existing paths; telemetry sends no
exception text.

**Preserves uv consent and compatibility behavior**

- Adds detailed internal uv lookup/install result types so the manager
can distinguish `available`, `declined`, `failed`, and `installed`
outcomes.
- Keeps existing compatibility wrappers for callers that only need
boolean/path results.
- Does not change prompt text, consent requirements, install selectors,
restart-required behavior, or uv's existing telemetry.

**Keeps event emission coalesced and deterministic**

- Same-key concurrent creation/reuse callers receive one lifecycle
result for the shared operation.
- Cache inspection, lock acquisition, fallback selection, cancellation,
and cleanup retain their existing control flow.
- Non-applicable `create()` calls and pre-validation exits do not emit
success-shaped lifecycle events.

### Event payloads and privacy

| Event | Data |
|---|---|
| `inlineScript.envCreated` | build duration; normalized dependency
count |
| `inlineScript.envReuseHit` | normalized dependency count |
| `inlineScript.envError` | stable failure category |

The events send no:

- script URI or filesystem path;
- requirement or dependency value;
- package name;
- Python/interpreter version;
- cache key;
- prompt text; or
- exception/error-message content.

Script-controlled metadata therefore cannot create unbounded telemetry
dimensions.

### Lifecycle examples

```text
validated cache hit
→ emit one inlineScript.envReuseHit
```

```text
cache miss or stale entry
→ start build timer
→ build + install dependencies
→ validate ownership/version
→ persist sidecar
→ emit one inlineScript.envCreated
```

```text
no compatible Python
→ user dismisses consent prompt
→ emit one inlineScript.envError with the declined category
```

### Tests

Coverage includes:

- creation emitted only after verified sidecar persistence;
- validated reuse and rebuild behavior;
- one event for concurrent/coalesced callers;
- duration boundaries excluding lock wait/cache inspection;
- normalized/deduplicated dependency counts;
- discovery, compatibility, consent, installation, cancellation, setup,
and lock categories;
- no lifecycle telemetry for non-applicable calls;
- direct detailed uv results for available, declined, failed, and
installed outcomes; and
- preservation of existing uv wrapper behavior.

Validation on the rebased branch:

- `npm run compile-tests`
- `npm run compile`
- `npm run lint`
- focused lifecycle/detailed-uv suites: 39 passing

The full Windows unit run reaches 1612 passing and 5 pending; the
existing concurrent `writeMetaJson` rename test can still intermittently
fail with `EPERM` on Windows. That writer is unchanged by this PR and
the same failure is reproducible on `main`.

### Performance

- No activation work, scan, timer, watcher, or new filesystem operation
is introduced.
- Instrumentation performs constant-size event construction around
operations that already occur.
- Coalesced setup emits once rather than once per waiter.
- No user-controlled strings are normalized or transmitted beyond the
dependency count already needed for the cache operation.

### User impact

**No default-path user impact.** The inline manager remains behind the
undeclared, default-off `python-envs.inlineScripts.enabled` flag.

When the internal flag is manually enabled, prompts, environment
creation/reuse, cancellation, error propagation, and cache behavior
remain unchanged. This PR only records privacy-safe lifecycle outcomes.

### Scope and follow-up

This PR intentionally does **not** implement:

- script detection or automatic routing;
- activation-time cache discovery;
- project registration or user-facing setup UX;
- cache clearing or TTL eviction; or
- status-bar behavior.

The telemetry is ready for those later entry points to consume once the
feature is intentionally exposed.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3cb82ae9-7424-40a4-9156-8c54ac6e0895
Copilot-Session: 6b12d843-8011-4bfc-9ba9-f75761eadee2
Stella Huang (StellaHuang95) added a commit that referenced this pull request Aug 19, 2026
> Part of #1602 (PEP 723 inline script env support). Design doc: #1601.
>
> Builds on the merged persistence work in #1697 and is rebased on
current `main`.

### Roadmap context

This is **PR 13 of 16** in the PEP 723 inline-script roadmap. It adds
the explicit, user-confirmed cache lifecycle operation that the later
TTL work will reuse.

| Phase 5: Lifecycle and polish | PR | Status |
|---|---|---|
| | PR 7: persistence (`get` / `set` + Memento) | merged (#1697) |
| | **PR 13: clear inline-script cache** | **this PR** |
| | PR 14: opportunistic 14-day TTL eviction | follow-up |
| | PR 15: lifecycle telemetry | separate |
| | PR 16: status-bar decision | resolved; no code PR |

### Why this PR

The extension can create extension-owned inline-script environments and
persist script associations, but it has no complete way to remove that
state. Clearing only the files would leave Memento associations and
`pythonProjects` entries pointing at deleted interpreters; clearing only
settings would leave disk usage behind.

This PR adds one coordinated lifecycle operation covering:

- extension-owned cache entries;
- persisted and in-memory script associations;
- active selection events; and
- generated inline-script project settings.

Because this is destructive and the cache is shared by extension-host
processes, the implementation is intentionally fail-closed around path
ownership and locks.

### What this PR does

**Adds an internal, confirmation-gated clear command**

- Registers `python-envs.clearScriptEnvCache` only while the hidden
inline-script feature flag is enabled.
- Does not contribute the command to `package.json` or the Command
Palette before rollout.
- Shows a modal warning covering cached environments, associations, and
project entries.
- Cancelling the prompt performs no filesystem, state, or settings
changes.
- Runs cache cleanup before settings removal, so failed/partial cache
cleanup does not silently rewrite project configuration.

**Keeps generic cache clearing behavior safe**

- The existing public `python-envs.clearCache` command continues to
clear existing non-inline managers.
- It skips the preview inline manager because the generic path has no
inline-specific confirmation or project-settings lifecycle.
- The dedicated command invokes the inline manager directly and performs
the complete cleanup transaction.

**Serializes in-process maintenance**

- Adds a manager-local maintenance queue and barrier.
- `create()`, `get()`, and `set()` cannot observe or mutate half-cleared
state.
- A clear request refuses to begin when creation already started.
- A creation request that arrives after clear begins waits for
maintenance to settle.
- Multiple maintenance requests are serialized without globally
serializing unrelated managers.

**Coordinates deletion across extension hosts**

- Acquires and holds each cache entry's cross-process lock through
deletion.
- Classifies locks as missing, held, retained, stale, orphaned,
malformed, or unavailable.
- Uses PID liveness to distinguish a live owner from a stale owner.
- Makes retained markers generation-specific by preserving the owner's
PID/nonce.
- Reclaims only the exact stale/retained generation marker that was
inspected.
- If another process replaces that generation before the atomic claim,
reclamation loses safely and touches nothing.
- Ambiguous legacy fixed `retained` markers remain recognizable but are
conservatively not reclaimed.

**Validates every destructive path**

Before deleting an entry, cleanup verifies that:

- global storage and `script-envs-v1` are normal directories rather than
symlinks/junctions;
- the versioned cache root is the expected direct child of global
storage;
- neither path is a filesystem root or dangerously shallow;
- physical `realpath` containment matches the expected ownership
boundary;
- the cache root has not changed since the cleanup snapshot; and
- the target entry is a normal direct-child directory inside that same
physical root.

The entry lock is acquired first, then root and entry ownership are
revalidated immediately before removal.

**Keeps state consistent through partial failures**

- Attempts cache entries independently and records successful removals.
- Aggregates and surfaces deletion/persistence failures rather than
returning success-shaped output.
- Invalidates only associations whose environment was removed or is
definitively missing.
- Preserves associations for cache entries that could not safely be
removed.
- Cancels pending rehydration for invalidated scripts.
- Clears warm environment and validation caches.
- Advances association revisions so stale async work cannot restore
removed selections.
- Emits `onDidChangeEnvironment` only for selections actually
invalidated.

**Removes generated inline project settings safely**

- Resolves `pythonProjects` entries independently from global,
workspace, and workspace-folder sources.
- Removes only entries whose manager is the inline-script manager.
- Preserves non-inline duplicate entries and higher-precedence
overrides.
- Handles same relative paths across multiple workspace roots.
- Aggregates global/workspace updates so each shared scope is written
once.
- Unloads only loaded projects that have no remaining configuration
source.

### Cleanup semantics

| Condition | Behavior |
|---|---|
| Entry is unlocked and physically owned | Lock, revalidate, delete |
| Lock belongs to a live process | Refuse that deletion |
| Exact stale/retained generation can be claimed | Reclaim, acquire a
fresh lock, delete |
| Lock is unavailable, malformed, orphaned, or legacy-ambiguous |
Preserve entry and surface failure |
| Root or entry is redirected/outside ownership boundary | Refuse
deletion |
| One entry fails after another was removed | Preserve valid survivors;
invalidate removed associations; report aggregate failure |
| Persistence update fails after disk cleanup | Keep in-memory state
consistent and surface the persistence error |
| Cache is already absent | Clear stale associations safely; remain
idempotent |

### Example

```text
Clear Script Environment Cache
→ modal confirmation
→ enter manager maintenance barrier
→ verify physical cache root
→ acquire exact per-entry lock
→ revalidate ownership immediately before deletion
→ delete safe entries
→ reconcile Memento + in-memory selections + events
→ remove generated inline pythonProjects settings
```

### Tests

Coverage includes:

- prompt cancellation and command ordering;
- generic clear behavior with the preview manager absent/present;
- in-process create/clear ordering;
- live, stale, retained, orphaned, malformed, unavailable, and legacy
lock states;
- exact-generation reclamation and delayed-reclaimer/new-creator races;
- holding entry locks through deletion;
- unsafe, shallow, redirected, symlinked, and root-swapped paths;
- successful, missing-cache, idempotent, and partial-failure cleanup;
- persistence failures and pending-rehydration races;
- global/workspace/workspace-folder setting precedence;
- multi-root projects and same-path entries; and
- default-off command registration.

Validation on the rebased branch:

- `npm run compile-tests`
- `npm run compile`
- `npm run lint`
- focused lock/cache-clear/settings/command suites: 38 passing

The full Windows unit run reaches 1638 passing and 5 pending; the
existing concurrent `writeMetaJson` rename test can still intermittently
fail with `EPERM` on Windows. That writer is unchanged by this PR and
the same failure is reproducible on `main`.

### Performance

- No activation scan, timer, or background maintenance is added.
- All work is initiated by the internal clear command.
- Per-entry locks avoid globally serializing independent environment
creation across extension hosts.
- The maintenance barrier exists only inside the enabled inline manager
and is active only during cleanup.

### User impact

**No default-path user impact.** The manager and command remain behind
the undeclared, default-off `python-envs.inlineScripts.enabled` flag,
and the command is not publicly contributed.

When the internal flag is manually enabled, the existing generic cache
command still behaves as before for non-inline managers. Inline cleanup
is available only through the dedicated confirmed lifecycle.

### Scope and follow-up

This PR intentionally does **not** implement:

- automatic routing or project registration;
- activation-time discovery;
- silent/opportunistic deletion;
- TTL expiration; or
- lifecycle telemetry.

PR 14 will reuse this safety and state-cleanup foundation to remove
entries whose `lastUsedAt` exceeds the planned 14-day TTL.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3cb82ae9-7424-40a4-9156-8c54ac6e0895
Copilot-Session: 6b12d843-8011-4bfc-9ba9-f75761eadee2
Stella Huang (StellaHuang95) added a commit that referenced this pull request Aug 19, 2026
> Part of #1602 (PEP 723 inline script env support). Design doc: #1601.
>
> Builds on the merged persistence work in #1697 and is rebased on
current `main`.

### Roadmap context

This is **PR 8 of 16** in the PEP 723 inline-script roadmap. It makes
extension-owned cached environments discoverable after activation;
automatic script routing remains a separate follow-up.

| Phase 2: Manager | PR | Status |
|---|---|---|
| | PR 4: `InlineScriptEnvManager` skeleton | merged (#1610) |
| | PR 5a: generic env-creation utilities | merged (#1651) |
| | PR 5b: inline-script cache + interpreter utilities | merged (#1655)
|
| | PR 5c: `create()` happy path | merged (#1656) |
| | PR 6: `create()` uv-install fallback | merged (#1696) |
| | PR 7: persistence (`get` / `set` + Memento) | merged (#1697) |
| | **PR 8: activation-time discovery** | **this PR** |
| | PR 9: route PEP 723 scripts to the inline manager | follow-up |

### Why this PR

PR 7 persists and lazily rehydrates a specific script's selected inline
environment when the manager is asked for that script. The manager still
has no global inventory, however: `getEnvironments()` returns `[]`, and
cached environments are not published after an extension-host restart
unless a script-specific lookup happens to reconstruct one.

This PR implements the activation-discovery portion of Q4 in the design:

- walk the versioned global cache without blocking extension activation;
- validate each cache entry before exposing it as a `PythonEnvironment`;
- publish add/remove events as valid entries appear or become
definitively invalid;
- preserve previously known entries through locks and transient
filesystem failures;
- retry bounded transient work without creating a permanent watcher or
polling loop.

### What this PR does

**Defers discovery until after manager registration**

- Starts activation discovery with `setImmediate()` after the manager is
registered.
- Leaves the feature-gate-off path unchanged: no manager registration,
cache scan, timer, or filesystem work.
- Keeps extension activation non-blocking.

**Publishes a global discovered collection**

- `getEnvironments('all')` returns a copy of the validated cache
collection.
- Other scopes remain empty because inline cache entries live in
extension global storage rather than belonging to one workspace.
- Results are sorted deterministically.
- Collection reconciliation emits precise `EnvironmentChangeKind.add`
and `remove` events only when manager identity, executable path, or
Python version materially changes.

**Validates every cache entry before publication**

A candidate must:

1. be a normal directory rather than a symlink;
2. be unlocked and physically contained beneath the expected cache root;
3. have a valid `.meta.json` sidecar;
4. have an available cached launcher and base interpreter;
5. resolve into a real `PythonEnvironment`;
6. prove direct-child cache ownership through `sysPrefix`; and
7. match the Python release recorded in the sidecar.

Discovery is read-only. It does not delete, rebuild, or rewrite invalid
entries.

**Distinguishes definitive invalidity from uncertainty**

- Missing, malformed, unowned, or version-mismatched entries are
omitted.
- Locked entries and transient I/O/resolver/ownership failures preserve
the previously published environment and request another activation
pass.
- A missing cache root is treated as an empty cache.
- An unavailable cache root preserves the current collection and remains
retryable.

**Handles concurrent cache changes**

- Activation scans compare an initial and final directory snapshot so an
entry created during the scan triggers a follow-up pass.
- Final cache-root absence publishes an empty collection immediately;
transient final-read failures preserve the prior collection until a
retry.
- Per-entry filesystem fingerprints detect a cache directory rebuilt
under the same key even when its name is unchanged.
- Locks use the cache entry name/hash as their identity.
- Published entries use that same cache-key identity rather than mixing
lexical cache paths with canonical `sysPrefix` paths, avoiding false
removal through symlink/junction path differences.
- Lock probing uses `lstat`; only `ENOENT` means unlocked. `EIO`, access
failures, and other uncertain states fail closed.

**Coalesces refresh work without weakening activation discovery**

- Concurrent compatible refreshes share one scan.
- Activation joining an in-flight explicit refresh receives one
snapshot-aware follow-up rather than accepting the explicit pass's
weaker single-snapshot semantics.
- Explicit `refresh()` cancels activation retries, performs one settled
pass, and schedules no delayed work afterward.

**Retries activation discovery only while useful**

- Follow-up delays are bounded at 1 second, 5 seconds, and 30 seconds.
- Retries are requested for locks, transient failures, or a changed
cache snapshot.
- No permanent filesystem watcher or unbounded polling loop is
introduced.
- Disposal cancels pending timers and prevents in-flight scans from
publishing late results.

**Strengthens the Windows usability guard**

- Windows cache validation now checks both the cached environment
launcher and the base interpreter referenced by `pyvenv.cfg`.
- A surviving base interpreter no longer makes an environment with a
missing `Scripts\python.exe` appear usable.

### Discovery semantics

| Cache state | Behavior |
|---|---|
| Valid sidecar, launcher, interpreter, ownership, and version | Publish
environment |
| Entry is locked or being built | Preserve previous publication; retry
|
| Filesystem/resolver state is temporarily unavailable | Preserve
previous publication; retry |
| Entry appears during the scan | Schedule snapshot-aware follow-up |
| Missing or malformed sidecar | Omit/remove from collection |
| Missing launcher/base interpreter | Omit/remove from collection |
| Ownership or recorded-version mismatch | Omit/remove from collection |
| Cache root is absent | Publish empty collection |

### Example

```text
extension activation
→ register InlineScriptEnvManager
→ defer one event-loop turn
→ scan <globalStorage>/script-envs-v1
→ validate sidecar + launcher + ownership + version
→ publish valid cached environments through getEnvironments('all')
→ retry only if the scan observed a lock, transient state, or snapshot change
```

This inventory does not associate an environment with a script. PR 7
owns persisted script associations, and PR 9 will use those associations
for automatic per-file routing.

### Tests

Coverage includes:

- deferred feature-gated activation startup;
- valid cache discovery and `all`-scope publication;
- missing, malformed, unavailable, non-directory, and symlinked entries;
- missing and non-regular Windows launchers;
- lock preservation, including unavailable (`EIO`) lock probes;
- canonical `sysPrefix` versus lexical cache-root identities;
- add/remove event reconciliation;
- concurrent refresh coalescing;
- activation joining an explicit refresh;
- cache entries created during a scan;
- builds completing after the short retry window;
- bounded retry exhaustion;
- explicit single-pass refresh behavior; and
- disposal during in-flight scans and pending retries.

Validation on the rebased branch:

- `npm run compile-tests`
- `npm run compile`
- `npm run lint`
- focused activation-discovery/cache-launcher/registration suites

The full Windows unit run reaches 1613 passing and 5 pending; the
existing concurrent `writeMetaJson` rename test can still intermittently
fail with `EPERM` on Windows. That writer is unchanged by this PR and
the same failure is reproducible on `main`.

### Performance

- Activation is deferred and never waits for discovery.
- Cache scans are coalesced.
- Retries are bounded and stop after a stable pass.
- Explicit refresh remains single-pass.
- There is no persistent watcher, unbounded polling, or per-document
work.

### User impact

**No default-path user impact.** The manager and discovery remain behind
the undeclared, default-off `python-envs.inlineScripts.enabled` flag.

With the internal flag manually enabled, valid cached inline
environments become available through the manager after restart. This PR
does not automatically select one for a script and introduces no public
command, setting, picker item, project registration, cache deletion, TTL
cleanup, or telemetry.

### Scope and follow-up

This PR intentionally does **not** implement:

- automatic PEP 723 script routing (PR 9);
- exact script project registration (PR 10);
- CodeLens or bulk setup UX (PRs 11-12);
- cache clearing or TTL eviction (PRs 13-14); or
- lifecycle telemetry (PR 15).

PR 7 (#1697) is merged, and this branch is rebased on current `main`.
Automatic routing can follow independently after this manager-discovery
layer lands.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3cb82ae9-7424-40a4-9156-8c54ac6e0895
Copilot-Session: 6b12d843-8011-4bfc-9ba9-f75761eadee2
Stella Huang (StellaHuang95) added a commit that referenced this pull request Aug 24, 2026
### Roadmap context

This is **PR 9 of 16** in the PEP 723 inline-script roadmap and the
repository's intentional routing cutover. Earlier manager PRs create,
discover, validate, and persist environments; this PR makes normal
per-file environment lookup use them when a saved script association is
proven current.

| Phase 2/3 | PR | Status |
|---|---|---|
| | PR 7: per-script persistence | merged (#1697) |
| | PR 8: activation-time cache discovery | merged (#1722) |
| | **PR 9: automatic per-script routing** | **this PR** |
| | PR 10: exact script project registration | follow-up |
| | PRs 11-12: CodeLens and bulk setup UX | follow-up |
| | PRs 17-19: Pylance/Python debugger integration | cross-repository
follow-up |

### Why this PR

The inline manager can already create/reuse environments, persist a
script association, rediscover cache entries after restart, clear state
safely, and report lifecycle telemetry. Nothing automatically selects
that manager for a Python file, however. A script continues to use its
normal project/default environment unless another component directly
asks the inline manager.

Automatic routing must be stricter than checking whether a file contains
PEP 723 metadata or whether Memento contains an executable path. It must
prove both:

```text
current saved PEP 723 metadata exists
                  +
a persisted environment association is valid for that metadata and cache generation
                  =
route this file to InlineScriptEnvManager
```

If either proof is absent, dirty, stale, transiently unavailable, or
superseded by an explicit choice, routing falls through to the existing
project/default manager.

### What this PR does

**Adds an activation-scoped routing registry**

- Latches the hidden feature flag once per extension-host activation.
- Creates one registry shared by the detector, inline manager, and
central environment-manager router.
- Tracks saved metadata, normalized metadata identity, metadata
revision, and validated-association state independently per script.
- Emits metadata and routeability transitions.
- Requires both current metadata and a validated association before
`shouldRoute(uri)` becomes true.

**Turns the existing lazy detector into a routing input only when
enabled**

- Reads saved PEP 723 metadata on open and save.
- Replays documents that were already open when activation occurred.
- Tracks loose local `.py` files for routing while retaining the
existing workspace-only telemetry population.
- Withholds routing for restored dirty documents.
- Invalidates routing immediately when an edit can touch the metadata
block, while body-only edits retain routing.
- Uses source-compatible offsets so BOM- and CRLF-prefixed metadata
edits are classified correctly.
- Uses per-URI read generations so an older open read cannot overwrite
newer saved metadata.
- Clears metadata and associations for deleted or renamed paths.

When no routing registry exists, the detector retains its existing
telemetry-only listeners, coalescing, workspace filter, and event
behavior.

**Defines a stable metadata identity**

- Uses normalized dependencies plus trimmed `requires-python`.
- Ignores dependency ordering, equivalent package-name spelling, and the
unrelated `[tool]` table.
- Does not include script paths, so equivalent scripts can share cache
entries.

**Adds durable cache provenance**

- Sidecars store bounded SHA-256 hashes of metadata identities that were
proven for the cache entry.
- Same-key coalesced creators and cache-reuse callers merge their
identities under the existing cache-entry lock.
- The list is deduplicated and capped at 128 entries.
- No package, requirement, script path, URI, or metadata content is
stored in provenance.
- Older sidecars without provenance remain usable through conservative
cache-key and Python-constraint proof; additional-package environments
remain non-routeable until explicitly proven.

**Makes sidecar replacement recoverable**

- Serializes same-sidecar writes in-process; production callers remain
protected by the cross-process cache-entry lock.
- Uses native rename with a unique backup on Windows replacement
contention.
- Restores the previous sidecar after failed replacement and retains the
only known-good backup if restoration is uncertain.
- Lock-held cache inspection can recover a strict, regular,
size-bounded, schema-valid backup compatible with the selected base
interpreter.
- Invalid, temporary, symlinked, oversized, unsupported, or
incompatible-only artifacts retain normal stale/uncertain behavior.

**Upgrades persisted associations from path-only values**

Each current record contains:

```text
schemaVersion
environmentPath
metadataBinding: legacy | pending(sourceIdentity) | matched(sourceIdentity)
```

- `legacy`: old string association; remains retrievable but is not
automatically routeable.
- `pending`: the environment is proven, but saved metadata has not yet
been durably matched (for example, selection while the document is
dirty).
- `matched`: saved metadata identity and cache provenance agree.
- Future schema records are preserved rather than destructively
rewritten.
- Malformed requested entries can be repaired without discarding
unrelated valid/future records.

**Validates associations before routing**

Validation requires:

1. current saved metadata;
2. an absolute persisted executable path;
3. no active cache build/lock;
4. a resolvable environment;
5. physical ownership by the expected extension cache entry;
6. compatible Python/requires-python state;
7. a matching persisted metadata binding; and
8. sidecar proof for the current metadata identity.

Definitively stale associations are conditionally removed. Locked,
transient, uncertain, or future-schema states are preserved but remain
non-routeable.

**Protects asynchronous validation from stale results**

- Metadata revisions remain monotonic across cleared-state tombstones,
preventing an old validator from winning after clear/restore of the same
identity.
- A changed saved metadata identity invalidates routeability
synchronously before provenance validation begins.
- Association revisions ensure unset/replacement wins over old
rehydration or validation.
- Same-script rehydration and metadata refreshes are coalesced.
- Pending bindings are promoted to matched only if both metadata and
association revisions are still current.
- Failed persistence never publishes successful routeability.

**Adds central manager routing with explicit precedence**

| Priority | Manager source |
|---|---|
| 1 | Exact per-script project setting |
| 2 | Explicit in-session non-inline override |
| 3 | Validated inline-script association |
| 4 | Existing project/default manager setting |
| 5 | Existing cached project/global manager |

- A user explicitly selecting a non-inline manager is not immediately
overridden by automatic routing.
- Clearing that override re-resolves and publishes through the newly
effective manager, with reserved operation ordering so a newer selection
still wins.
- Inline selections are keyed by normalized script path, not containing
project, so multiple scripts in one workspace can remain independent.
- Inline selections are not published as active until routeability is
validated.
- Routeability changes refresh the appropriate manager and emit the
existing active-environment transition.
- Invalidated metadata falls back to the existing project/default
environment and removes only the inline active-selection entry.
- Existing revision checks continue preventing slow refreshes from
overwriting newer selections.

### User flows

**Previously configured script after restart**

```text
open saved script
→ detector publishes metadata identity
→ manager loads persisted association
→ validate executable + ownership + sidecar provenance
→ registry marks association routeable
→ central environment lookup switches this file to the inline manager
```

**Metadata edit**

```text
edit inside metadata block
→ routeability clears immediately
→ normal project/default manager remains active
→ save reads new metadata and revalidates
→ inline routing returns only if the existing association is still proven
```

**New unassociated script**

Opening a new PEP 723 script does **not** silently select a matching
cache entry. It remains on normal routing until the future explicit
setup action (PR 11/12) creates or reuses an environment and persists
the association. After that, this PR provides automatic routing.

### Routing and failure semantics

| State | Behavior |
|---|---|
| Saved metadata + matched/proven association | Route inline |
| Metadata but no association | Existing project/default routing |
| Association but metadata unknown/dirty | Existing project/default
routing |
| Body-only edit | Preserve current routing |
| Metadata edit or invalid saved block | Clear inline routing and fall
back |
| Explicit non-inline selection | Explicit override wins |
| Legacy string association | Preserve/retrieve, but do not auto-route
after restart |
| Pending binding with matching saved proof | Atomically promote to
matched |
| Locked/transient cache state | Preserve association; remain
non-routeable; retry later |
| Same cache path rebuilt for different provenance | Reject old routing
proof |
| Rename/delete | Clear old path's association and routeability |

### Review guide

The production changes are easiest to review in this order:

1. **Routing state and detection**
   - `routingRegistry.ts`
   - `activation.ts`
   - `metadata.ts`
   - `lazyDetector.ts`
2. **Cache provenance and durability**
   - `cacheLayout.ts`
   - creation/reuse sections of `inlineScript/envManager.ts`
3. **Persisted binding and validation state machine**
- persistence/rehydration/metadata-refresh sections of
`inlineScript/envManager.ts`
4. **Behavior cutover**
   - `envManagers.ts`
   - `extension.ts`
   - `inlineScript/main.ts`

More than half of the diff is deterministic unit coverage for
dirty/save/restart and async race behavior.

### Tests

Coverage includes:

- activation flag latching and default-off registration;
- telemetry-only detector equivalence when routing is absent;
- open/save ordering, restored dirty editors, loose files, CRLF edits,
body-only edits, rename, and delete;
- metadata-only, association-only, exact-setting, explicit-override,
route-on, and fallback behavior;
- independent same-workspace script selections and batch operations;
- legacy, pending, matched, malformed, and future persisted records;
- restart rehydration and delayed routeability;
- same-path cache rebuilds and additional-package provenance;
- same-key coalesced creation and one-event telemetry semantics;
- stale metadata/association races against save, unset, and replacement;
- sidecar write contention, restoration, backup recovery, and invalid
artifact handling;
- cache-clear partial failure with versioned associations; and
- public API last-known fallback/event ordering.

Validation on the final rebased tree:

- `npm run compile-tests`
- `npm run compile`
- `npm run lint`
- `npm run unittest`: **1,817 passing, 6 pending**

### Performance
- The feature remains hidden and default-off.
- With the flag off, no routing registry, manager, discovery timer,
routing file listeners, Memento read, sidecar/cache work, or routing
telemetry is added.
- The existing telemetry detector keeps its prior listener/coalescing
behavior; routing checks are optional branches.
- With the flag on, metadata reads remain bounded to the first 8 KiB and
occur only for opened/saved local Python files.
- Same-script validation and same-key creation are coalesced.
- Warm association validation is throttled.
- Discovery retries remain bounded and cache maintenance remains
serialized only within the inline manager.

### Privacy and safety

- Telemetry remains count/boolean/category based and sends no URI, path,
dependency, requirement, Python version, cache key, or error text.
- Provenance uses local SHA-256 metadata identities rather than raw
metadata.
- Physical cache ownership and executable checks remain fail-closed.
- Symlinked/unowned cache entries are never made routeable.
- Destructive cache behavior remains exclusively in the merged
confirmation-gated cleanup lifecycle.

### User impact

**No default-path user impact.** `python-envs.inlineScripts.enabled`
remains undeclared and defaults to false:

- no visible setting or autocomplete;
- no new command, menu, CodeLens, picker, or status-bar surface;
- no inline manager registration;
- no routing registry, discovery, persistence, or cache work; and
- existing project/default routing remains unchanged.

For developers manually enabling the hidden flag, existing proven script
associations route automatically and fall back conservatively when proof
is absent. New scripts still require the future explicit setup UX.

### Scope and follow-up

This PR intentionally does **not** implement:

- automatic setup for a newly encountered script;
- exact generated script project registration (PR 10);
- CodeLens or bulk setup UX (PRs 11-12);
- Pylance per-file interpreter support (PRs 17-18); or
- the Python debugger per-file resolver fix (PR 19).

Those later PRs can build on this guarded routing layer without changing
its validation contract.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6b12d843-8011-4bfc-9ba9-f75761eadee2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature-request Request for new features or functionality review-auto:changes-requested Automated review: posted blocking findings to address.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants