Route PEP 723 scripts to inline environments (PEP 723 PR 9/16) - #1729

Roadmap context

This is PR 9 of 16 in the PEP 723 inline-script roadmap and the repository's intentional routing cutover. Earlier manager PRs create, discover, validate, and persist environments; this PR makes normal per-file environment lookup use them when a saved script association is proven current.

Why this PR

The inline manager can already create/reuse environments, persist a script association, rediscover cache entries after restart, clear state safely, and report lifecycle telemetry. Nothing automatically selects that manager for a Python file, however. A script continues to use its normal project/default environment unless another component directly asks the inline manager.

Automatic routing must be stricter than checking whether a file contains PEP 723 metadata or whether Memento contains an executable path. It must prove both:

current saved PEP 723 metadata exists
                  +
a persisted environment association is valid for that metadata and cache generation
                  =
route this file to InlineScriptEnvManager

If either proof is absent, dirty, stale, transiently unavailable, or superseded by an explicit choice, routing falls through to the existing project/default manager.

What this PR does

Adds an activation-scoped routing registry

• Latches the hidden feature flag once per extension-host activation.
• Creates one registry shared by the detector, inline manager, and central environment-manager router.
• Tracks saved metadata, normalized metadata identity, metadata revision, and validated-association state independently per script.
• Emits metadata and routeability transitions.
• Requires both current metadata and a validated association before shouldRoute(uri) becomes true.

Turns the existing lazy detector into a routing input only when enabled

• Reads saved PEP 723 metadata on open and save.
• Replays documents that were already open when activation occurred.
• Tracks loose local .py files for routing while retaining the existing workspace-only telemetry population.
• Withholds routing for restored dirty documents.
• Invalidates routing immediately when an edit can touch the metadata block, while body-only edits retain routing.
• Uses source-compatible offsets so BOM- and CRLF-prefixed metadata edits are classified correctly.
• Uses per-URI read generations so an older open read cannot overwrite newer saved metadata.
• Clears metadata and associations for deleted or renamed paths.

When no routing registry exists, the detector retains its existing telemetry-only listeners, coalescing, workspace filter, and event behavior.

Defines a stable metadata identity

• Uses normalized dependencies plus trimmed requires-python.
• Ignores dependency ordering, equivalent package-name spelling, and the unrelated [tool] table.
• Does not include script paths, so equivalent scripts can share cache entries.

Adds durable cache provenance

• Sidecars store bounded SHA-256 hashes of metadata identities that were proven for the cache entry.
• Same-key coalesced creators and cache-reuse callers merge their identities under the existing cache-entry lock.
• The list is deduplicated and capped at 128 entries.
• No package, requirement, script path, URI, or metadata content is stored in provenance.
• Older sidecars without provenance remain usable through conservative cache-key and Python-constraint proof; additional-package environments remain non-routeable until explicitly proven.

Makes sidecar replacement recoverable

• Serializes same-sidecar writes in-process; production callers remain protected by the cross-process cache-entry lock.
• Uses native rename with a unique backup on Windows replacement contention.
• Restores the previous sidecar after failed replacement and retains the only known-good backup if restoration is uncertain.
• Lock-held cache inspection can recover a strict, regular, size-bounded, schema-valid backup compatible with the selected base interpreter.
• Invalid, temporary, symlinked, oversized, unsupported, or incompatible-only artifacts retain normal stale/uncertain behavior.

Upgrades persisted associations from path-only values

Each current record contains:

schemaVersion
environmentPath
metadataBinding: legacy | pending(sourceIdentity) | matched(sourceIdentity)

• legacy: old string association; remains retrievable but is not automatically routeable.
• pending: the environment is proven, but saved metadata has not yet been durably matched (for example, selection while the document is dirty).
• matched: saved metadata identity and cache provenance agree.
• Future schema records are preserved rather than destructively rewritten.
• Malformed requested entries can be repaired without discarding unrelated valid/future records.

Validates associations before routing

Validation requires:

1. current saved metadata;
2. an absolute persisted executable path;
3. no active cache build/lock;
4. a resolvable environment;
5. physical ownership by the expected extension cache entry;
6. compatible Python/requires-python state;
7. a matching persisted metadata binding; and
8. sidecar proof for the current metadata identity.

Definitively stale associations are conditionally removed. Locked, transient, uncertain, or future-schema states are preserved but remain non-routeable.

Protects asynchronous validation from stale results

• Metadata revisions remain monotonic across cleared-state tombstones, preventing an old validator from winning after clear/restore of the same identity.
• A changed saved metadata identity invalidates routeability synchronously before provenance validation begins.
• Association revisions ensure unset/replacement wins over old rehydration or validation.
• Same-script rehydration and metadata refreshes are coalesced.
• Pending bindings are promoted to matched only if both metadata and association revisions are still current.
• Failed persistence never publishes successful routeability.

Adds central manager routing with explicit precedence

• A user explicitly selecting a non-inline manager is not immediately overridden by automatic routing.
• Clearing that override re-resolves and publishes through the newly effective manager, with reserved operation ordering so a newer selection still wins.
• Inline selections are keyed by normalized script path, not containing project, so multiple scripts in one workspace can remain independent.
• Inline selections are not published as active until routeability is validated.
• Routeability changes refresh the appropriate manager and emit the existing active-environment transition.
• Invalidated metadata falls back to the existing project/default environment and removes only the inline active-selection entry.
• Existing revision checks continue preventing slow refreshes from overwriting newer selections.

User flows

Previously configured script after restart

open saved script
→ detector publishes metadata identity
→ manager loads persisted association
→ validate executable + ownership + sidecar provenance
→ registry marks association routeable
→ central environment lookup switches this file to the inline manager

Metadata edit

edit inside metadata block
→ routeability clears immediately
→ normal project/default manager remains active
→ save reads new metadata and revalidates
→ inline routing returns only if the existing association is still proven

New unassociated script

Opening a new PEP 723 script does not silently select a matching cache entry. It remains on normal routing until the future explicit setup action (PR 11/12) creates or reuses an environment and persists the association. After that, this PR provides automatic routing.

Routing and failure semantics

Review guide

The production changes are easiest to review in this order:

1. Routing state and detection
   • routingRegistry.ts
   • activation.ts
   • metadata.ts
   • lazyDetector.ts
2. Cache provenance and durability
   • cacheLayout.ts
   • creation/reuse sections of inlineScript/envManager.ts
3. Persisted binding and validation state machine
   • persistence/rehydration/metadata-refresh sections of inlineScript/envManager.ts
4. Behavior cutover
   • envManagers.ts
   • extension.ts
   • inlineScript/main.ts

More than half of the diff is deterministic unit coverage for dirty/save/restart and async race behavior.

Tests

Coverage includes:

• activation flag latching and default-off registration;
• telemetry-only detector equivalence when routing is absent;
• open/save ordering, restored dirty editors, loose files, CRLF edits, body-only edits, rename, and delete;
• metadata-only, association-only, exact-setting, explicit-override, route-on, and fallback behavior;
• independent same-workspace script selections and batch operations;
• legacy, pending, matched, malformed, and future persisted records;
• restart rehydration and delayed routeability;
• same-path cache rebuilds and additional-package provenance;
• same-key coalesced creation and one-event telemetry semantics;
• stale metadata/association races against save, unset, and replacement;
• sidecar write contention, restoration, backup recovery, and invalid artifact handling;
• cache-clear partial failure with versioned associations; and
• public API last-known fallback/event ordering.

Validation on the final rebased tree:

• npm run compile-tests
• npm run compile
• npm run lint
• npm run unittest: 1,817 passing, 6 pending

Performance

• The feature remains hidden and default-off.
• With the flag off, no routing registry, manager, discovery timer, routing file listeners, Memento read, sidecar/cache work, or routing telemetry is added.
• The existing telemetry detector keeps its prior listener/coalescing behavior; routing checks are optional branches.
• With the flag on, metadata reads remain bounded to the first 8 KiB and occur only for opened/saved local Python files.
• Same-script validation and same-key creation are coalesced.
• Warm association validation is throttled.
• Discovery retries remain bounded and cache maintenance remains serialized only within the inline manager.

Privacy and safety

• Telemetry remains count/boolean/category based and sends no URI, path, dependency, requirement, Python version, cache key, or error text.
• Provenance uses local SHA-256 metadata identities rather than raw metadata.
• Physical cache ownership and executable checks remain fail-closed.
• Symlinked/unowned cache entries are never made routeable.
• Destructive cache behavior remains exclusively in the merged confirmation-gated cleanup lifecycle.

User impact

No default-path user impact. python-envs.inlineScripts.enabled remains undeclared and defaults to false:

• no visible setting or autocomplete;
• no new command, menu, CodeLens, picker, or status-bar surface;
• no inline manager registration;
• no routing registry, discovery, persistence, or cache work; and
• existing project/default routing remains unchanged.

For developers manually enabling the hidden flag, existing proven script associations route automatically and fall back conservatively when proof is absent. New scripts still require the future explicit setup UX.

Scope and follow-up

This PR intentionally does not implement:

• automatic setup for a newly encountered script;
• exact generated script project registration (PR 10);
• CodeLens or bulk setup UX (PRs 11-12);
• Pylance per-file interpreter support (PRs 17-18); or
• the Python debugger per-file resolver fix (PR 19).

Those later PRs can build on this guarded routing layer without changing its validation contract.