Add PEP 723 inline script environment creation (PEP 723 PR 5c/16) - #1656
Conversation
0433271 to
95cb552
Compare
> Part of #1602 (PEP 723 inline script env support). Design doc: #1601. > **Split for review (3 PRs).** Reviewers flagged the original PR 5 as too large, so it is split into three stacked PRs grouped by dependency layer: > - **5a — generic env-creation utilities — this PR (#1651).** Based on `main`; independent; merges first. > - **5b — inline-script cache + interpreter utilities — #1655.** Stacked on 5a. > - **5c — `create()` happy path (manager + wiring) — #1656.** Stacked on 5b. > > Applied together the three PRs are byte-for-byte identical to the original single change. **Merge order: 5a → 5b → 5c.** ### Roadmap context This is the first slice of **PR 5 of 16** in the PEP 723 inline-script roadmap. The full plan lives in #1602. | Phase | PR | Status | |---|---|---| | **Phase 1: Foundation** | PR 1: cache key hash utility | merged (#1634) | | | PR 2: cache layout + `meta.json` sidecar | merged (#1635) | | | PR 3: `requires-python` to interpreter selection | merged (#1636) | | **Phase 2: Manager** | PR 4: `InlineScriptEnvManager` skeleton | merged (#1610) | | | **PR 5a: generic env-creation utilities** | **this PR (#1651)** | | | **PR 5b: inline-script cache + interpreter utilities** | **#1655** | | | **PR 5c: `create()` happy path (manager + wiring)** | **#1656** | | | PR 6: `create()` uv-install fallback | not started (needs 3, 5) | | | PR 7: persistence with `get`, `set`, and Memento | not started (needs 4) | | | PR 8: activation-time discovery | not started (needs 2, 4, 7) | | **Phase 3: Routing** | PR 9: route PEP 723 scripts to the inline manager | not started (needs 4, 7) | | | PR 10: per-script project registration | not started (needs 9) | | **Phase 4+: UX / lifecycle** | PRs 11-16 | not started | ### Why this PR PR 5c implements `InlineScriptEnvManager.create()`. Before touching the manager, this PR lands the **generic, reusable primitives** it relies on — a cross-process file lock, a venv Python-path helper, a cancellation-hardened process runner, and two small `createWithProgress` options. None of this code is inline-script-specific, so it is reviewed on its own. ### What this PR adds **Cross-process file lock** (`src/common/lockfile.apis.ts`, new): `acquireFileLock` uses an atomic `mkdir` of a `<path>.lock` directory plus a per-owner marker file, returning `AcquiredFileLock { release, retain }`. `retain()` writes a `retained` marker so a later acquirer **fails fast with `ELOCKRETAINED`** instead of waiting out the 5-minute timeout — used when a build is cancelled mid-flight. Distinct error codes (`ELOCKED`, `ELOCKRETAINED`, `ELOCKORPHANED`, `ECOMPROMISED`, `ERETAINFAILED`) separate contention from corruption. **Shared `getVenvPythonPath`** (`src/common/utils/virtualEnvironment.ts`, new): returns `Scripts\python.exe` on Windows, else `bin/python`. Replaces an inline copy in `venvUtils` and is reused by 5b/5c. **Hardened process helper** (`src/managers/builtin/helpers.ts`): `runUV` and `runPython` now share one `runProcess` implementation whose cancellation guards `kill()` in `try/catch` and still emits a clean `CancellationError` if the process errors after a cancel. Per-caller options preserve existing behavior (`collectStderr`, `logPrefix`). **`venvUtils.ts`:** `createWithProgress` gains `CreateWithProgressOptions { trackUvEnvironment }`, and `CreateEnvironmentResult` gains `pkgInstallationCancelled` so a caller can tell cancellation apart from a real install failure. Existing callers are unaffected (both are optional / additive). ### Tests - **`lockfile.apis.unit.test.ts`** — 9 tests: contention, retain/fail-fast, orphaned and compromised locks, and timeout. - **`virtualEnvironment.unit.test.ts`** — 2 tests for `getVenvPythonPath` on Windows and POSIX. - **`helpers.cancellation.unit.test.ts`** — 4 tests for `runProcess` cancellation safety. - **`venvUtils.createWithProgress.unit.test.ts`** — 3 tests for `trackUvEnvironment` and `pkgInstallationCancelled`. On this branch alone `npm run compile-tests` is clean and `npm run unittest` reports **1447 passing, 0 failing, 4 pending**. ### User impact **None.** These are internal primitives with no new user-visible behavior. The refactors to `helpers.ts` and `venvUtils.ts` are behavior-preserving for existing callers. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 39dcc6a3-0fbd-4f36-9d0f-68677de49c27
Test verification reportCI clarification: all checks are green for this commit. The TypeScript unit-test job runs only on Ubuntu and Windows; macOS CI runs smoke/E2E/integration suites, not these unit tests. Must fix
Warnings
Summary
Trace checks confirmed the coalescing, cache-reuse, and rollback tests execute real manager paths. Mutations showed coalescing and sidecar rollback tests correctly catch regressions. All temporary repairs, traces, and mutations were reverted; the worktree is clean. |
…1655) > Part of #1602 (PEP 723 inline script env support). Design doc: #1601. > **Split for review (3 PRs).** Reviewers flagged the original PR 5 as too large, so it is split into three PRs grouped by dependency layer: > - **5a — generic env-creation utilities — #1651.** Merged. > - **5b — inline-script cache + interpreter utilities — this PR (#1655).** Rebased on `main`. > - **5c — `create()` happy path (manager + wiring) — #1656.** Stacked on 5b. > > #1651 has merged and this branch has been rebased. The diff now contains only this PR's seven files. **Remaining merge order: 5b → 5c.** ### Roadmap context This is the second slice of **PR 5 of 16**. See #1651 for the full roadmap table. | Phase 2: Manager | PR | Status | |---|---|---| | | PR 4: `InlineScriptEnvManager` skeleton | merged (#1610) | | | PR 5a: generic env-creation utilities | merged (#1651) | | | **PR 5b: inline-script cache + interpreter utilities** | **this PR (#1655)** | | | PR 5c: `create()` happy path (manager + wiring) | #1656 | ### Why this PR With the generic primitives from 5a in place, this PR lands the **inline-script-specific utilities** that `create()` (5c) composes: a normalized dependency cache key, cache-layout ownership/status checks, and interpreter-constraint handling. These are pure functions with no manager wiring yet, so they are reviewed on their own. ### What this PR adds **Cache-key tail normalization** (`src/common/inlineScriptCacheKey.ts`): adds `normalizeRequirementTail`, a quote-aware scanner that collapses whitespace and tightens comparator spacing (`>= 1.0` → `>=1.0`) in a requirement's version/marker tail while **preserving quoted PEP 508 marker literals verbatim** (e.g. `python_version >= "3.11"`). Direct-reference requirements (`pkg @ https://…`) are kept verbatim after the name and extras. The effect is that semantically identical dependency strings normalize to the same cache key, so they reuse the same cached environment. **Cache-layout additions** (`src/common/inlineScriptCacheLayout.ts`): `resolveCacheEntryPath` (containment under the cache root), `inspectOwnedCacheEntry` (realpath ownership), `getBaseInterpreterStatus` (`available | missing | unavailable`), `inspectMetaJson` (typed sidecar read), and a stricter `validateMeta`. The `.meta.json` sidecar schema is `{ schemaVersion, baseInterpreterPath, baseInterpreterVersion, lastUsedAt }`. Uses `getVenvPythonPath` from merged PR #1651. **Interpreter-constraint trimming** (`src/common/inlineScriptInterpreter.ts`): `pickCompatibleInterpreter` now trims `requires-python`, so a whitespace-only constraint is treated as no constraint. **Manager ID constants** (`src/common/constants.ts`): centralizes the conda and inline-script manager IDs used by interpreter filtering. ### Tests - **`inlineScriptCacheKey.unit.test.ts`** — canonicalization cases including marker literals and direct references. - **`inlineScriptCacheLayout.unit.test.ts`** — the new containment, ownership, base-interpreter-status, and typed sidecar-read helpers. - **`inlineScriptInterpreter.unit.test.ts`** — constraint trimming / selection. On this rebased branch `npm run compile-tests` is clean and `npm run unittest` reports **1467 passing, 0 failing, 5 pending**. ### User impact **None.** These are pure utilities. Nothing calls the new code paths until the manager lands in 5c (#1656). ### Merge order #1651 has merged. Merge this PR next, then #1656. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 39dcc6a3-0fbd-4f36-9d0f-68677de49c27
95cb552 to
b4c915d
Compare
|
Will do a separate PR to put all the inline related files into a "inline" directory. Don't want to make this PR too messy and difficult to review. |
feedback addressed. |
Low — Consolidate the duplicated file-not-found helper
No Critical, High, or Medium findings. Lock retention, rollback, cache ownership, symlink containment, interpreter selection, cross-platform paths, constants placement, and comments all look sound. |
Eduardo Villalpando Mello (edvilme)
left a comment
There was a problem hiding this comment.
Will take a look at the rest throughout the day, but left some comments in the meantime :)
Addressed both of your test verification report and consolidate the helper. |
Eduardo Villalpando Mello (edvilme)
left a comment
There was a problem hiding this comment.
Couple of comments about maybe extracting some of the helper functions off of the class since they are not bound to a specific instance, but otherwise LGTM :)
### Why VS Code 1.131 removed the legacy `Contents/MacOS/Electron` compatibility path from its macOS application bundle. `@vscode/test-electron` 2.5.2 still launches that hard-coded path, so all macOS smoke jobs fail with `ENOENT` before VS Code or the extension starts. This currently blocks the macOS checks on #1656. The upstream issue and fix are microsoft/vscode-test#349 and microsoft/vscode-test#350. ### What Upgrade `@vscode/test-electron` from 2.5.2 to 3.1.0 and refresh the lockfile. Version 3.1.0 resolves the macOS executable from `CFBundleExecutable` in `Info.plist`, with the old `Electron` path retained as a fallback for older VS Code builds. This is a development-only dependency change. Version 3.1.0 requires Node 22, which is already used by CI. ### Validation - `npm run compile-tests` - `npm run unittest` — 1469 passing, 0 failing, 5 pending Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 39dcc6a3-0fbd-4f36-9d0f-68677de49c27
Implement InlineScriptEnvManager.create(): select a compatible base interpreter and build or reuse a dependency-keyed virtual environment, with cache-ownership validation, cross-process locking, and cancellation-safe creation. Wire the manager's collaborators. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 39dcc6a3-0fbd-4f36-9d0f-68677de49c27
Use the merged createWithProgress boolean parameter while preserving that inline-script cache entries are not tracked as workspace uv environments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 39dcc6a3-0fbd-4f36-9d0f-68677de49c27
Centralize the pyenv manager ID, simplify inline manager data flow and release comparisons, and share filesystem not-found classification across cache components. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 39dcc6a3-0fbd-4f36-9d0f-68677de49c27
00386ec to
c75660a
Compare
c0f89be
into
microsoft:main
…oft#1651) > Part of microsoft#1602 (PEP 723 inline script env support). Design doc: microsoft#1601. > **Split for review (3 PRs).** Reviewers flagged the original PR 5 as too large, so it is split into three stacked PRs grouped by dependency layer: > - **5a — generic env-creation utilities — this PR (microsoft#1651).** Based on `main`; independent; merges first. > - **5b — inline-script cache + interpreter utilities — microsoft#1655.** Stacked on 5a. > - **5c — `create()` happy path (manager + wiring) — microsoft#1656.** Stacked on 5b. > > Applied together the three PRs are byte-for-byte identical to the original single change. **Merge order: 5a → 5b → 5c.** ### Roadmap context This is the first slice of **PR 5 of 16** in the PEP 723 inline-script roadmap. The full plan lives in microsoft#1602. | Phase | PR | Status | |---|---|---| | **Phase 1: Foundation** | PR 1: cache key hash utility | merged (microsoft#1634) | | | PR 2: cache layout + `meta.json` sidecar | merged (microsoft#1635) | | | PR 3: `requires-python` to interpreter selection | merged (microsoft#1636) | | **Phase 2: Manager** | PR 4: `InlineScriptEnvManager` skeleton | merged (microsoft#1610) | | | **PR 5a: generic env-creation utilities** | **this PR (microsoft#1651)** | | | **PR 5b: inline-script cache + interpreter utilities** | **microsoft#1655** | | | **PR 5c: `create()` happy path (manager + wiring)** | **microsoft#1656** | | | PR 6: `create()` uv-install fallback | not started (needs 3, 5) | | | PR 7: persistence with `get`, `set`, and Memento | not started (needs 4) | | | PR 8: activation-time discovery | not started (needs 2, 4, 7) | | **Phase 3: Routing** | PR 9: route PEP 723 scripts to the inline manager | not started (needs 4, 7) | | | PR 10: per-script project registration | not started (needs 9) | | **Phase 4+: UX / lifecycle** | PRs 11-16 | not started | ### Why this PR PR 5c implements `InlineScriptEnvManager.create()`. Before touching the manager, this PR lands the **generic, reusable primitives** it relies on — a cross-process file lock, a venv Python-path helper, a cancellation-hardened process runner, and two small `createWithProgress` options. None of this code is inline-script-specific, so it is reviewed on its own. ### What this PR adds **Cross-process file lock** (`src/common/lockfile.apis.ts`, new): `acquireFileLock` uses an atomic `mkdir` of a `<path>.lock` directory plus a per-owner marker file, returning `AcquiredFileLock { release, retain }`. `retain()` writes a `retained` marker so a later acquirer **fails fast with `ELOCKRETAINED`** instead of waiting out the 5-minute timeout — used when a build is cancelled mid-flight. Distinct error codes (`ELOCKED`, `ELOCKRETAINED`, `ELOCKORPHANED`, `ECOMPROMISED`, `ERETAINFAILED`) separate contention from corruption. **Shared `getVenvPythonPath`** (`src/common/utils/virtualEnvironment.ts`, new): returns `Scripts\python.exe` on Windows, else `bin/python`. Replaces an inline copy in `venvUtils` and is reused by 5b/5c. **Hardened process helper** (`src/managers/builtin/helpers.ts`): `runUV` and `runPython` now share one `runProcess` implementation whose cancellation guards `kill()` in `try/catch` and still emits a clean `CancellationError` if the process errors after a cancel. Per-caller options preserve existing behavior (`collectStderr`, `logPrefix`). **`venvUtils.ts`:** `createWithProgress` gains `CreateWithProgressOptions { trackUvEnvironment }`, and `CreateEnvironmentResult` gains `pkgInstallationCancelled` so a caller can tell cancellation apart from a real install failure. Existing callers are unaffected (both are optional / additive). ### Tests - **`lockfile.apis.unit.test.ts`** — 9 tests: contention, retain/fail-fast, orphaned and compromised locks, and timeout. - **`virtualEnvironment.unit.test.ts`** — 2 tests for `getVenvPythonPath` on Windows and POSIX. - **`helpers.cancellation.unit.test.ts`** — 4 tests for `runProcess` cancellation safety. - **`venvUtils.createWithProgress.unit.test.ts`** — 3 tests for `trackUvEnvironment` and `pkgInstallationCancelled`. On this branch alone `npm run compile-tests` is clean and `npm run unittest` reports **1447 passing, 0 failing, 4 pending**. ### User impact **None.** These are internal primitives with no new user-visible behavior. The refactors to `helpers.ts` and `venvUtils.ts` are behavior-preserving for existing callers. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 39dcc6a3-0fbd-4f36-9d0f-68677de49c27
> Part of #1602 (PEP 723 inline script env support). Design doc: #1601. > > This replaces the earlier closed draft #1652 with the finalized implementation rebased on `main`. ### Roadmap context This is **PR 6 of 16** in the PEP 723 inline-script roadmap. It extends the PR 5 `create()` happy path with the missing-compatible-interpreter fallback. | Phase 2: Manager | PR | Status | |---|---|---| | | PR 4: `InlineScriptEnvManager` skeleton | merged (#1610) | | | PR 5a: generic env-creation utilities | merged (#1651) | | | PR 5b: inline-script cache + interpreter utilities | merged (#1655) | | | PR 5c: `create()` happy path | merged (#1656) | | | **PR 6: `create()` uv-install fallback** | **this PR** | | | PR 7: persistence (`get` / `set` + Memento) | separate follow-up | ### Why this PR PR 5 can create or reuse an inline-script environment when an installed base interpreter already satisfies the script's `requires-python`. It deliberately stops when no compatible interpreter exists. This PR adds the consent-gated fallback for that case: 1. Re-check installed interpreters after entering a narrow fallback queue. 2. Select a safe Python version satisfying the complete PEP 440 requirement. 3. Ask the user before installing uv or Python. 4. Refresh environment discovery after installation. 5. If discovery is stale or unavailable, resolve and validate uv's returned executable directly. 6. Continue through the existing PR 5 cache create/reuse path. ### What this PR does **Adds the inline-script fallback to `InlineScriptEnvManager.create()`** - Keeps the installed-interpreter path unchanged and only enters the fallback when no compatible base exists. - Skips installation prompts for quick-create/noninteractive calls. - Serializes only fallback installations, then re-checks discovery before prompting so concurrent compatible requests reuse one installation. - Retains successfully direct-resolved uv interpreters so queued requests can reuse them even while discovery remains stale. **Selects a safe uv target from `requires-python`** - Uses simple safe selectors directly, such as `>=3.13` → `3.13` and `==3.13.1` → `3.13.1`. - Queries uv's advertised versions for bounded or exclusion-heavy constraints. - Restricts catalog candidates to default CPython 3 builds satisfying the complete specifier. - Handles exclusions such as `>=3.13.2,!=3.13.2` without installing the excluded floor. - Uses an advertised release for bounded ranges rather than fabricating a potentially unavailable patch-zero release. - Applies full PEP 440 prerelease semantics consistently across discovery, catalog selection, direct resolution, and cache validation. - Normalizes accepted prerelease aliases (for example, `c1` → `rc1`) before passing a version to uv. **Extends the uv installer's consent flow** - Adds an inline-script-specific prompt that shows both the script requirement and selected Python version. - Sanitizes and caps script-controlled prompt details. - Validates install selectors before forwarding them to uv. - If catalog lookup is required and uv is missing, asks for consent to install uv first. - Re-checks whether a newly installed uv is usable by the current extension host and surfaces the existing restart-required message when needed. **Handles stale discovery after installation** - Refreshes environment discovery after uv installs Python. - Treats refresh/discovery failures as recoverable. - Resolves the executable returned by uv directly, verifies that it satisfies `requires-python`, and canonicalizes its path before creating the cached environment. ### Examples | `requires-python` | Fallback behavior | |---|---| | `>=3.13` | Request uv's `3.13` selector | | `==3.13.1` | Request exactly `3.13.1` without requiring a catalog lookup | | `>=3.11,<3.12` | Choose an advertised compatible `3.11.x` release | | `>=3.13.2,!=3.13.2` | Skip `3.13.2` and choose a compatible advertised release | | `>=3.15.0a1,<3.16` | Permit an explicitly requested prerelease | | `>=3.14,<3.16` | Do not select a prerelease implicitly | ### Safety and concurrency - No uv or Python installation occurs without explicit user consent. - Script-derived values cannot inject arbitrary uv arguments. - Declined, cancelled, or failed installations do not mutate the script-environment cache. - The fallback queue does not globally serialize environment selection or normal cache creation. - Existing cross-process cache locking and rollback behavior from PR 5 remains unchanged. ### Tests Coverage includes: - simple, exact, bounded, exclusion, and prerelease requirements; - uv catalog filtering and consented uv bootstrap; - declined and failed installations; - refresh and discovery failures with direct resolution; - simultaneous same- and different-constraint requests; - quick-create prompt suppression; - strict post-install and cache compatibility checks; - prompt sanitization and install-selector validation. `npm run compile-tests`, `npm run lint`, the full unit suite, and the focused inline-script/uv suites are clean. ### User impact **No default-path user impact yet.** This completes an internal Phase 2 manager capability. Automatic routing and user-facing entry points arrive in later roadmap PRs. When those entry points are wired, users whose scripts require an unavailable Python will be able to approve installing a compatible interpreter rather than having environment creation stop. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1a9f6ba1-9bd3-4664-bc25-a0d34d7a2e91
> Part of #1602 (PEP 723 inline script env support). Design doc: #1601. > > This replaces the earlier closed draft #1653 with the finalized implementation rebased on `main`. ### Roadmap context This is **PR 7 of 16** in the PEP 723 inline-script roadmap. It adds durable per-script environment associations to the internal manager. | Phase 2: Manager | PR | Status | |---|---|---| | | PR 4: `InlineScriptEnvManager` skeleton | merged (#1610) | | | PR 5a: generic env-creation utilities | merged (#1651) | | | PR 5b: inline-script cache + interpreter utilities | merged (#1655) | | | PR 5c: `create()` happy path | merged (#1656) | | | PR 6: `create()` uv-install fallback | open (#1696) | | | **PR 7: persistence (`get` / `set` + Memento)** | **this PR** | | | PR 8: activation-time discovery | follow-up | | | PR 9: route PEP 723 scripts to the inline manager | follow-up | ### Why this PR PR 5 can build or reuse an inline-script environment, but the manager does not remember that the resulting environment belongs to a particular script. After an extension-host restart, the in-memory association is gone. This PR implements the persistence portion of Q4 in the design: - maintain an independent environment association for each script; - persist script-to-environment executable paths in workspace Memento; - lazily and safely rehydrate those associations; - re-check current `requires-python` metadata before returning an environment; - report changes so the central environment API can update its last-known state. ### What this PR does **Implements per-script `set()`** - Accepts one or more local `file:` URIs and rejects invalid or mixed scopes atomically. - Validates that selected environments are owned inline-script cache entries. - Persists a normalized script path → environment executable path mapping under a dedicated Memento key. - Supports assigning and unassigning individual scripts or batches. - Updates in-memory state and emits `onDidChangeEnvironment` only for effective changes. - Leaves the existing `create()` behavior separate: creation alone does not implicitly establish a persisted association. **Implements per-script `get()`** - Reads current PEP 723 metadata before returning an association. - Keeps unreadable or temporarily invalid script metadata from destructively clearing state. - Returns an in-memory association when valid. - Lazily reconstructs persisted environments after restart instead of resolving every script during activation. - Re-checks `requires-python` against the reconstructed Python version before returning it. **Safely rehydrates persisted associations** - Requires an absolute executable path. - Preserves associations while their cache entry is locked or being created. - Verifies that the executable exists and is a regular file. - Resolves it into a `PythonEnvironment` and confirms that it belongs to the expected extension-owned cache entry. - Removes only definitively stale associations; transient filesystem or resolver failures remain retryable. - Emits a change event when a slow rehydration eventually succeeds, including when the public API's initial one-second wait has already elapsed. **Validates warm in-memory associations** - Periodically revalidates cached associations without performing full resolution on every lookup. - Detects executables deleted while VS Code remains open. - Detects an environment rebuilt at the same cache path with a different Python version. - Preserves busy/locked entries instead of misclassifying them as stale. - Coalesces simultaneous validations for the same script. - Retains the existing environment object when resolution produces only a new generated ID for the same Python, avoiding false changes and duplicate ID-keyed resources. **Protects persistence and selection from races** - Serializes Memento read-modify-write operations so concurrent script selections cannot lose one another. - Uses per-script association revisions so an older rehydration cannot overwrite a newer selection or unset. - Removes stale persisted values conditionally, only if the inspected path is still current. - Keeps failed persistence writes from changing in-memory state or emitting success-shaped events. - Does not globally serialize unrelated environment operations. **Updates central active-environment tracking** - Keys inline-script selections by normalized script path rather than containing project, so two scripts in one workspace can retain different environments. - Uses per-scope revisions and manager identity checks so slow refreshes cannot overwrite newer selections. - Ensures failed selections and failed refreshes do not discard a valid in-flight refresh. - Groups same-manager batch unsets and calls the manager once with the complete URI array. - Updates central cache entries and events only after the manager operation succeeds. - Attributes inline-script change events to the script URI rather than the containing project URI. ### Example Given two scripts in the same workspace: ```text tools/report.py → Python 3.12 inline environment tools/import.py → Python 3.13 inline environment ``` PR 7 stores and retrieves those associations independently. Selecting the environment for `import.py` does not overwrite the last-known environment for `report.py`. After restart: ```text get(report.py) → read persisted executable → verify cache ownership and current metadata → resolve environment → cache and return it ``` If `report.py` later changes from `requires-python = ">=3.11"` to `">=3.13"`, its persisted Python 3.12 environment is no longer returned as compatible. ### Persistence and failure semantics | Condition | Behavior | |---|---| | Executable exists and cache ownership is valid | Rehydrate and return | | Cache entry is locked/in progress | Preserve association; retry later | | Resolver fails transiently | Preserve association; retry later | | Executable is definitively missing and unlocked | Remove stale association and notify | | A newer selection wins during rehydration | Discard the stale result | | Memento write fails | Keep previous in-memory/persisted selection and propagate the error | ### Tests Coverage includes: - assign, retrieve, unset, and batch persistence; - restart-time lazy rehydration and delayed success events; - metadata compatibility changes; - missing, malformed, unowned, busy, and transient cache states; - warm deletion and same-path rebuild detection; - concurrent persistence, rehydration, validation, selection, and unset races; - failed Memento writes; - strict URI-scope validation; - independent same-project script selections; - stale and failed central refresh ordering; - atomic same-manager batch unsets. `npm run compile-tests`, `npm run lint`, the full unit suite, and the focused persistence/central-manager suites are clean. ### Performance - Rehydration is lazy rather than activation-blocking. - Warm associations are cached and validation is throttled. - Same-script rehydration and validation work is coalesced. - Queues cover only shared persistence and mutation ordering; unrelated script reads and environment-manager operations remain independent. ### User impact **No default-path user impact yet.** This completes an internal Phase 2 manager capability. Automatic routing and user-facing entry points arrive in later roadmap PRs. Once routing is wired, script-specific selections will survive extension-host restarts and remain independent even for multiple scripts in the same workspace. ### Merge order The core persistence behavior depends on the merged manager skeleton (#1610). This branch is rebased on current `main`; PR 8 and PR 9 build on this capability. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1a9f6ba1-9bd3-4664-bc25-a0d34d7a2e91
…1723) > Part of #1602 (PEP 723 inline script env support). Design doc: #1601. > > Builds on the merged creation, uv-fallback, and persistence work in #1656, #1696, and #1697. This branch is rebased on current `main`. ### Roadmap context This is **PR 15 of 16** in the PEP 723 inline-script roadmap. It adds the remaining lifecycle telemetry for the internal manager without changing routing, creation, or cache policy. | Phase 5: Lifecycle and polish | PR | Status | |---|---|---| | | PR 13: clear inline-script cache | separate | | | PR 14: opportunistic TTL eviction | follow-up | | | **PR 15: lifecycle telemetry** | **this PR** | | | PR 16: status-bar decision | resolved; no code PR | ### Why this PR The manager can create, reuse, and persist inline-script environments, including consent-gated uv/Python fallback, but those lifecycle outcomes are not observable as a coherent feature funnel. This PR adds low-cardinality telemetry that answers: - whether setup built a new environment or reused a validated cache entry; - how long actual environment construction took; - how many normalized dependencies were involved; and - which stable failure category prevented setup. The instrumentation is deliberately attached to the underlying coalesced operation rather than every caller, and it excludes script/package/interpreter content. ### What this PR does **Adds three typed lifecycle events** - `inlineScript.envCreated` - `inlineScript.envReuseHit` - `inlineScript.envError` The telemetry constants and GDPR declarations use typed event/property mappings so call sites cannot send undeclared fields. **Reports verified creation only** - Emits `envCreated` only after environment creation succeeds, ownership/version validation passes, and `.meta.json` is persisted. - Measures the actual build/rebuild interval rather than lock waiting or cache inspection. - Reports normalized, deduplicated dependency count rather than raw dependency values. - Emits one creation event for the underlying coalesced operation, even when multiple callers await it. **Reports validated cache reuse only** - Emits `envReuseHit` only after sidecar, interpreter, ownership, and compatibility checks complete successfully. - Does not report a hit for uncertain, stale, malformed, or partially validated entries. - Preserves the existing cache reuse and `lastUsedAt` behavior. **Classifies lifecycle failures without leaking details** `envError` uses stable low-cardinality categories for outcomes such as: - environment discovery failure; - no compatible Python; - declined compatible-Python installation; - uv/Python installation failure; - general setup, cache-validation, or metadata-persistence failure; - package-install cancellation; - cache-lock timeout; and - retained, orphaned, or otherwise unavailable lock state. Errors remain logged through the existing paths; telemetry sends no exception text. **Preserves uv consent and compatibility behavior** - Adds detailed internal uv lookup/install result types so the manager can distinguish `available`, `declined`, `failed`, and `installed` outcomes. - Keeps existing compatibility wrappers for callers that only need boolean/path results. - Does not change prompt text, consent requirements, install selectors, restart-required behavior, or uv's existing telemetry. **Keeps event emission coalesced and deterministic** - Same-key concurrent creation/reuse callers receive one lifecycle result for the shared operation. - Cache inspection, lock acquisition, fallback selection, cancellation, and cleanup retain their existing control flow. - Non-applicable `create()` calls and pre-validation exits do not emit success-shaped lifecycle events. ### Event payloads and privacy | Event | Data | |---|---| | `inlineScript.envCreated` | build duration; normalized dependency count | | `inlineScript.envReuseHit` | normalized dependency count | | `inlineScript.envError` | stable failure category | The events send no: - script URI or filesystem path; - requirement or dependency value; - package name; - Python/interpreter version; - cache key; - prompt text; or - exception/error-message content. Script-controlled metadata therefore cannot create unbounded telemetry dimensions. ### Lifecycle examples ```text validated cache hit → emit one inlineScript.envReuseHit ``` ```text cache miss or stale entry → start build timer → build + install dependencies → validate ownership/version → persist sidecar → emit one inlineScript.envCreated ``` ```text no compatible Python → user dismisses consent prompt → emit one inlineScript.envError with the declined category ``` ### Tests Coverage includes: - creation emitted only after verified sidecar persistence; - validated reuse and rebuild behavior; - one event for concurrent/coalesced callers; - duration boundaries excluding lock wait/cache inspection; - normalized/deduplicated dependency counts; - discovery, compatibility, consent, installation, cancellation, setup, and lock categories; - no lifecycle telemetry for non-applicable calls; - direct detailed uv results for available, declined, failed, and installed outcomes; and - preservation of existing uv wrapper behavior. Validation on the rebased branch: - `npm run compile-tests` - `npm run compile` - `npm run lint` - focused lifecycle/detailed-uv suites: 39 passing The full Windows unit run reaches 1612 passing and 5 pending; the existing concurrent `writeMetaJson` rename test can still intermittently fail with `EPERM` on Windows. That writer is unchanged by this PR and the same failure is reproducible on `main`. ### Performance - No activation work, scan, timer, watcher, or new filesystem operation is introduced. - Instrumentation performs constant-size event construction around operations that already occur. - Coalesced setup emits once rather than once per waiter. - No user-controlled strings are normalized or transmitted beyond the dependency count already needed for the cache operation. ### User impact **No default-path user impact.** The inline manager remains behind the undeclared, default-off `python-envs.inlineScripts.enabled` flag. When the internal flag is manually enabled, prompts, environment creation/reuse, cancellation, error propagation, and cache behavior remain unchanged. This PR only records privacy-safe lifecycle outcomes. ### Scope and follow-up This PR intentionally does **not** implement: - script detection or automatic routing; - activation-time cache discovery; - project registration or user-facing setup UX; - cache clearing or TTL eviction; or - status-bar behavior. The telemetry is ready for those later entry points to consume once the feature is intentionally exposed. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3cb82ae9-7424-40a4-9156-8c54ac6e0895 Copilot-Session: 6b12d843-8011-4bfc-9ba9-f75761eadee2
> Part of #1602 (PEP 723 inline script env support). Design doc: #1601. > > Builds on the merged persistence work in #1697 and is rebased on current `main`. ### Roadmap context This is **PR 8 of 16** in the PEP 723 inline-script roadmap. It makes extension-owned cached environments discoverable after activation; automatic script routing remains a separate follow-up. | Phase 2: Manager | PR | Status | |---|---|---| | | PR 4: `InlineScriptEnvManager` skeleton | merged (#1610) | | | PR 5a: generic env-creation utilities | merged (#1651) | | | PR 5b: inline-script cache + interpreter utilities | merged (#1655) | | | PR 5c: `create()` happy path | merged (#1656) | | | PR 6: `create()` uv-install fallback | merged (#1696) | | | PR 7: persistence (`get` / `set` + Memento) | merged (#1697) | | | **PR 8: activation-time discovery** | **this PR** | | | PR 9: route PEP 723 scripts to the inline manager | follow-up | ### Why this PR PR 7 persists and lazily rehydrates a specific script's selected inline environment when the manager is asked for that script. The manager still has no global inventory, however: `getEnvironments()` returns `[]`, and cached environments are not published after an extension-host restart unless a script-specific lookup happens to reconstruct one. This PR implements the activation-discovery portion of Q4 in the design: - walk the versioned global cache without blocking extension activation; - validate each cache entry before exposing it as a `PythonEnvironment`; - publish add/remove events as valid entries appear or become definitively invalid; - preserve previously known entries through locks and transient filesystem failures; - retry bounded transient work without creating a permanent watcher or polling loop. ### What this PR does **Defers discovery until after manager registration** - Starts activation discovery with `setImmediate()` after the manager is registered. - Leaves the feature-gate-off path unchanged: no manager registration, cache scan, timer, or filesystem work. - Keeps extension activation non-blocking. **Publishes a global discovered collection** - `getEnvironments('all')` returns a copy of the validated cache collection. - Other scopes remain empty because inline cache entries live in extension global storage rather than belonging to one workspace. - Results are sorted deterministically. - Collection reconciliation emits precise `EnvironmentChangeKind.add` and `remove` events only when manager identity, executable path, or Python version materially changes. **Validates every cache entry before publication** A candidate must: 1. be a normal directory rather than a symlink; 2. be unlocked and physically contained beneath the expected cache root; 3. have a valid `.meta.json` sidecar; 4. have an available cached launcher and base interpreter; 5. resolve into a real `PythonEnvironment`; 6. prove direct-child cache ownership through `sysPrefix`; and 7. match the Python release recorded in the sidecar. Discovery is read-only. It does not delete, rebuild, or rewrite invalid entries. **Distinguishes definitive invalidity from uncertainty** - Missing, malformed, unowned, or version-mismatched entries are omitted. - Locked entries and transient I/O/resolver/ownership failures preserve the previously published environment and request another activation pass. - A missing cache root is treated as an empty cache. - An unavailable cache root preserves the current collection and remains retryable. **Handles concurrent cache changes** - Activation scans compare an initial and final directory snapshot so an entry created during the scan triggers a follow-up pass. - Final cache-root absence publishes an empty collection immediately; transient final-read failures preserve the prior collection until a retry. - Per-entry filesystem fingerprints detect a cache directory rebuilt under the same key even when its name is unchanged. - Locks use the cache entry name/hash as their identity. - Published entries use that same cache-key identity rather than mixing lexical cache paths with canonical `sysPrefix` paths, avoiding false removal through symlink/junction path differences. - Lock probing uses `lstat`; only `ENOENT` means unlocked. `EIO`, access failures, and other uncertain states fail closed. **Coalesces refresh work without weakening activation discovery** - Concurrent compatible refreshes share one scan. - Activation joining an in-flight explicit refresh receives one snapshot-aware follow-up rather than accepting the explicit pass's weaker single-snapshot semantics. - Explicit `refresh()` cancels activation retries, performs one settled pass, and schedules no delayed work afterward. **Retries activation discovery only while useful** - Follow-up delays are bounded at 1 second, 5 seconds, and 30 seconds. - Retries are requested for locks, transient failures, or a changed cache snapshot. - No permanent filesystem watcher or unbounded polling loop is introduced. - Disposal cancels pending timers and prevents in-flight scans from publishing late results. **Strengthens the Windows usability guard** - Windows cache validation now checks both the cached environment launcher and the base interpreter referenced by `pyvenv.cfg`. - A surviving base interpreter no longer makes an environment with a missing `Scripts\python.exe` appear usable. ### Discovery semantics | Cache state | Behavior | |---|---| | Valid sidecar, launcher, interpreter, ownership, and version | Publish environment | | Entry is locked or being built | Preserve previous publication; retry | | Filesystem/resolver state is temporarily unavailable | Preserve previous publication; retry | | Entry appears during the scan | Schedule snapshot-aware follow-up | | Missing or malformed sidecar | Omit/remove from collection | | Missing launcher/base interpreter | Omit/remove from collection | | Ownership or recorded-version mismatch | Omit/remove from collection | | Cache root is absent | Publish empty collection | ### Example ```text extension activation → register InlineScriptEnvManager → defer one event-loop turn → scan <globalStorage>/script-envs-v1 → validate sidecar + launcher + ownership + version → publish valid cached environments through getEnvironments('all') → retry only if the scan observed a lock, transient state, or snapshot change ``` This inventory does not associate an environment with a script. PR 7 owns persisted script associations, and PR 9 will use those associations for automatic per-file routing. ### Tests Coverage includes: - deferred feature-gated activation startup; - valid cache discovery and `all`-scope publication; - missing, malformed, unavailable, non-directory, and symlinked entries; - missing and non-regular Windows launchers; - lock preservation, including unavailable (`EIO`) lock probes; - canonical `sysPrefix` versus lexical cache-root identities; - add/remove event reconciliation; - concurrent refresh coalescing; - activation joining an explicit refresh; - cache entries created during a scan; - builds completing after the short retry window; - bounded retry exhaustion; - explicit single-pass refresh behavior; and - disposal during in-flight scans and pending retries. Validation on the rebased branch: - `npm run compile-tests` - `npm run compile` - `npm run lint` - focused activation-discovery/cache-launcher/registration suites The full Windows unit run reaches 1613 passing and 5 pending; the existing concurrent `writeMetaJson` rename test can still intermittently fail with `EPERM` on Windows. That writer is unchanged by this PR and the same failure is reproducible on `main`. ### Performance - Activation is deferred and never waits for discovery. - Cache scans are coalesced. - Retries are bounded and stop after a stable pass. - Explicit refresh remains single-pass. - There is no persistent watcher, unbounded polling, or per-document work. ### User impact **No default-path user impact.** The manager and discovery remain behind the undeclared, default-off `python-envs.inlineScripts.enabled` flag. With the internal flag manually enabled, valid cached inline environments become available through the manager after restart. This PR does not automatically select one for a script and introduces no public command, setting, picker item, project registration, cache deletion, TTL cleanup, or telemetry. ### Scope and follow-up This PR intentionally does **not** implement: - automatic PEP 723 script routing (PR 9); - exact script project registration (PR 10); - CodeLens or bulk setup UX (PRs 11-12); - cache clearing or TTL eviction (PRs 13-14); or - lifecycle telemetry (PR 15). PR 7 (#1697) is merged, and this branch is rebased on current `main`. Automatic routing can follow independently after this manager-discovery layer lands. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3cb82ae9-7424-40a4-9156-8c54ac6e0895 Copilot-Session: 6b12d843-8011-4bfc-9ba9-f75761eadee2
Roadmap context
This is the final slice of PR 5 of 16 — the actual
create()happy path. See #1651 for the full roadmap table.InlineScriptEnvManagerskeletoncreate()happy path (manager + wiring)create()uv-install fallbackWhy this PR
InlineScriptEnvManager.create()was a deliberately empty no-op after PR 4. This PR implements its happy path: the case where the machine already has a base interpreter that satisfies the script'srequires-python, so no uv Python install is required. Given a PEP 723 script, it builds — or reuses — a dependency-keyed virtual environment under the extension's global storage, following the pipx-style cache design from Q4 of #1601. The uv-install fallback (no compatible interpreter present) is deferred to PR 6.It composes the primitives from 5a (#1651) and the inline-script utilities from 5b (#1655); this PR adds only the manager and its wiring.
What this PR does
Wires the manager's collaborators (
extension.ts,inlineScriptMain.ts):registerInlineScriptFeaturesand theInlineScriptEnvManagerconstructor now receive theNativePythonFinder, thePythonEnvironmentApi, the base (system) environment manager, andglobalStorageUri.Implements
create(scope)(inlineScriptEnvManager.ts):file:URI (a bareUrior single-element array). Anything else —'global', a folder, or multiple URIs — logs a warning and returnsundefined.undefined.metadata.dependencieswithoptions.additionalPackages, trims each, and rejects empty entries.create()calls for the same key via apendingCreationsmap.Base-interpreter selection (
selectBaseInterpreter): starts fromgetEnvironments('global'), keeps only true base managers (system, pyenv, condabase), and excludes derived environments by rejecting a non-absolutesysPrefixor the presence ofpyvenv.cfg. It then picks the newest compatible interpreter withpickCompatibleInterpreterand resolves the executable throughfs.realpathso the cache key is canonical. If a candidate cannot be resolved it falls through to the next.Create-or-reuse under a cross-process lock (
createOrReuseEnvironment): acquires a directory lock (5a), inspects the existing cache entry, and reuses / rebuilds / preserves accordingly, always releasing the lock infinally.Fail-closed cache inspection (
inspectCacheEntry) returnsabsent | stale | uncertain | reusable:resolveCacheEntryPath)..meta.jsonsidecar and confirms the recorded base-interpreter path and version still match the selected base.getBaseInterpreterStatus).PythonEnvironment, confirms it is genuinely ours via realpath containment (inspectOwnedCacheEntry), compares Python release segments, and re-checksrequires-pythonwith the existingmatchesPythonVersion.stale(rebuild); any doubt yieldsuncertain, and an uncertain entry is preserved, never deleted. A reused entry has itslastUsedAtrefreshed.Environment build (
buildCacheEntry): delegates to the existingcreateWithProgressvenv flow withtrackUvEnvironmentset tofalseso cached script environments are not registered as workspace venvs. On success it writes the sidecar and re-validates that the built environment matches the requested release and is owned by this entry. On failure it removes the directory and returns empty. On cancellation it retains the lock so a half-built environment is not silently reused later.Tests
inlineScriptEnvManager.unit.test.ts— 40 tests across scope/metadata validation, base-interpreter selection, cache creation, cache reuse, transaction rollback, and events/disposal.inlineScriptMain.unit.test.ts— updated for the newregisterInlineScriptFeaturessignature.On this rebased branch
npm run compile-testsis clean andnpm run unittestreports 1491 passing, 0 failing, 5 pending.User impact
None on the default path. The manager is still registered only when the undeclared
python-envs.inlineScripts.enabledflag is on, so default users see nothing.create()is now a declared method (PR 4 omitted it), so with the flag on the inline manager can appear as a create target. But it acts only on a single local script URI and no-ops on every other scope, and nothing in the extension routes a script URI to it yet. Wiring the trigger is later work: routing in PR 9, and the "Set up env for this script" picker item and bulk command in PR 11/12.Merge order
#1651 and #1655 have merged. This PR is the remaining final slice.