Align startup paths and sysconfig/venv with Python 3.15 by youknowdot · Pull Request #9174 · RustPython/RustPython

youknowdot · 2026-10-17T10:09:27Z

Align startup paths and the CPython 3.15 sysconfig/venv closure

Split from #8954.

This remains a draft. Head edb0540 repairs a copied completion-lock invariant exposed by the macOS fork failure; fresh Linux release test_threading and native/lint gates pass, and hosted CI is now terminal success with all 25 jobs passing. Previous head bc27c61 fixed the Windows symlink-helper failures and passed Windows/Linux Python CI, with 24 passing jobs and one macOS thread/fork failure. The earlier CodSpeed performance-analysis result remains unresolved; a passing benchmark workflow alone does not resolve it.

Changes

Honor both PYTHONHOME prefix halves, including empty-half fallback and -E/-I handling.
Recognize readable pyvenv.cfg without a home key, separate venv public prefixes from base-library paths, and preserve launcher overrides.
Preserve a discovered standard library. Place compiled fallback paths after configured paths and the default ZIP; retain customized embedding path order.
Supply bounded Windows platform/source-directory/runtime-module metadata and LIBDEST.
Freeze zipimport as part of the core bootstrap, matching CPython's bootstrap design and avoiding a negative path-importer cache entry before the ZIP hook is registered. Track the canonical source explicitly for incremental builds and Windows text-symlink checkouts.
Migrate the associated sysconfig, venv, _osx_support, zipimport and test closure through scripts/update_lib from CPython 3.15.0 d220409a13. Preserve genuine existing guards and the implementation-name customization.
Retire only the two trace coverage guards proved successful on hosted Windows and the repaired Linux binary. Keep the other two inherited trace guards.

This branch is based directly on main and excludes the separate lazy-import integration.

Symlink-helper repair at `bc27c61`

Import only the PythonSymlink class from pinned CPython 3.15.0 d220409a13 through scripts/update_lib. Keep its canonical PYTHONHOME/VPATH handling; add PYTHONPATH=STDLIB_DIR only for RustPython's Windows child, whose Cargo layout does not use CPython Modules/Setup markers.
The narrow updater mode verifies source/destination hashes and exact class/anchor shapes, rejects drift or shadowing, and reproduces identical output on repeated migration. All unrelated test.support content and existing guards remain unchanged. No runtime PYTHONHOME fallback or test skip was added.
All 269 updater unit tests pass, including canonical/legacy import, repeated runs, changed anchors, hash mismatches, conflicting options and CPython/RustPython environment behavior. Normal commit hooks pass.
Fresh Linux test_sysconfig: 41 tests, 28 pass, 7 existing skips and 6 existing expected failures. Fresh test_platform: 39 tests, 34 pass and 5 existing skips.
These Linux checks reuse the sealed release binary built from 5be5a82 (SHA-256 1775a7d647d758214972176931e4ce9483b40d2c11f05fed169899a9d0164a18). All native/build inputs were reverified unchanged; the updated external Lib/test closure was hashed separately. Bounded supervisors verified exit status and process-group cleanup. The workspace/C-API builds and native gates below are previous-head evidence, not fresh rebuilds for this helper-only change.
Hosted CI: https://github.com/RustPython/RustPython/actions/runs/39089432286. Benchmark workflow: https://github.com/RustPython/RustPython/actions/runs/39089432289. The benchmark workflow passes. Windows and Linux Python jobs pass; the Windows log records test_platform and test_sysconfig passing with no new test skips. macOS failed in test_threading: the first pass reported a fork-child SIGSEGV, and its automatic rerun timed out waiting for a forked child in test_main_thread_after_fork_from_nonmain_thread. All other 24 CI jobs passed. A cause for this macOS failure is not established; no thread/fork runtime code changed in the helper repair. Latest main 924e342 passed its macOS Python job. Re-running the failed PR job through GitHub Actions is blocked for this account (HTTP 403: repository admin rights required); a maintainer rerun is needed. This does not establish that the underlying fork issue is resolved.

Fork completion-lock repair at `edb0540`

Reset every live thread handle's copied completion mutex and condition variable before the survivor/NotStarted early return in the post-fork child. Keep the original completion value and handle state for those handles, and preserve the existing cleanup for threads that did not survive.
The old ordering could leave the surviving handle's completion mutex owned by a vanished parent joiner. CPython's completion event is atomic and its parking lot is reset after fork; RustPython's mutex-backed event requires this explicit per-handle reset. No Python test skips or weakened assertions were added.
A deterministic native invariant test models the inherited locked completion event and checks the surviving, not-started, dead-running and already-complete states without forking the Rust test harness. The focused test and all 151 VM native tests pass in the normal test profile.
A fresh, source-bound release interpreter (SHA-256 440cd9a2a994e41f160ea9df3c82fe491e7704ef3568e6373fb2bb7fb15ada31) passes both exact canonical fork tests and full test_threading: 245 tests, 18 inherited skips and one inherited expected failure. Its build banner identifies edb0540.
Normal commit hooks and strict workspace/C-API Clippy (-D warnings) pass.
Fresh workspace release native tests and doctests: 1,391 pass, 18 inherited ignored. Standard-profile C-API tests: 123 pass, four inherited ignored; C-API doctests complete with zero tests.
Validation attempts are distinguished: the debug interpreter's default recursion limit prevented importing the Python test harness (zero tests run); the C-API release attempt used an unsuitable profile for existing debug-assertion-dependent NULL-pointer panic tests and terminated abnormally. These are not counted as passes. Final Python validation used the fresh release interpreter and final C-API validation used the standard test profile. Two earlier C-API build attempts stopped at the 1 GiB disk reserve; their incomplete receipts were retained.
Hosted CI run 39102011793 is terminal SUCCESS: all 25 jobs pass, including Linux/macOS/Windows Python and native tests, strict Clippy and Miri. macOS test_threading passes in 42.0 seconds; Windows test_threading, test_platform and test_sysconfig also pass. The previous fork-child SIGSEGV and timeout did not recur in this run, although this does not isolate the old SIGSEGV root cause. The earlier job-rerun HTTP 403 remains recorded; this was a substantive source repair, not an empty rerun commit.
The final head has 28 successful checks and one intentionally skipped pyperformance check. CodSpeed benchmark execution passes, but no separate CodSpeed performance-analysis check was reported on this head; the historical performance result remains unresolved.

Previous repair validation

Source checkpoint 66e80ef. The sealed release binary was built from 5be5a82; the only subsequent source change is the updater-generated test_trace guard retirement. All native inputs are identical.

Offline release build and normal hooks pass.
Twelve CPython-first startup observations: all 108 explicit semantic checks match; six cases match fully after fixture/binary normalization and six retain documented installation/build-layout differences.
Both exact package/subpackage-in-ZIP startup tests pass.
Canonical test_zipimport: 81 pass, 52 skip. Optional zstd/C-API, resource and inherited sparse-file guards remain; the skipped traceback test is not claimed as a pass.
Full test_trace with CPU coverage enabled: 25 pass and two inherited expected failures; no skips. Both retired coverage tests execute successfully.
Workspace native: 1,371 pass, including both new structural path-order tests.
C-API native: 123 pass and four existing ignored.
Strict workspace and C-API Clippy pass.
Final-binary test_sysconfig: 28 pass, 7 skip and 6 inherited expected failures.
Final-binary test_venv: 26 pass, 9 skip and 4 inherited expected failures, with the same three explicit local exclusions.
test__osx_support: 16 Linux skips.
Workspace docs: 24 pass and 18 existing ignored; C-API docs complete with zero tests.
Compiler-only nonthreading check passes.
Source/artifact hashes reconcile after every gate; generated bytecode is preserved separately. Final path/ZIP repeats and all later runtime launches pin the absence of the default ZIP inputs.

Previous hosted repair results

At head 66e80ef, all checks are terminal: 27 successful, two failed and one skipped.

Linux and macOS Python jobs pass, as do the native/Rust checks.
Windows passes test_cmd_line_script, test_zipimport, test_zipimport_support and test_trace. The only failing test modules are test_platform and test_sysconfig: test_architecture_via_symlink and test_symlink respectively. Both errors reproduce on the workflow's automatic rerun.
The CodSpeed build/benchmark workflow succeeds. Its separate analysis flags rustpython[frozenset.py]: 361.0 to 416.6 microseconds, a 13.34% efficiency decrease (15.39% longer simulated time), against main fc9f4a9. No cause is established.
CodSpeed records matching CPU, OS and Rust/LLVM settings, but its head environment record includes libffi, libm and libpython entries absent from the base record. This metadata difference does not establish the cause of the result. Existing exposed records do not supply repeat variance; the result has not been dismissed or acknowledged.

Remaining platform and hosted limitations

The narrow Windows helper integration is now implemented at bc27c61. Windows hosted Python execution now passes, including test_platform and test_sysconfig. Explicit PYTHONHOME remains authoritative in the runtime.
The first CodSpeed benchmark-build job exited 143 without a compiler diagnostic. The repair workflow subsequently completed successfully; the original exit cause remains unclassified and the separate performance-analysis failure remains open.
Full frozen-backend startup parity, zip-only stdlib installation and native Windows/macOS local execution are not established by these Linux checks.
The same two multiprocessing selections and pip-install venv scenario remain excluded locally for their previously documented effects boundaries.

AI assistance

OpenAI Codex assisted with extraction, implementation, canonical-library migration, validation and this description. Model version was unavailable. Independent source/input reviews and the explicit Linux checks above were performed; platform and environment gaps remain disclosed.

Assisted-by: OpenAI Codex:model version unavailable

coderabbitai · 2026-10-17T10:09:34Z

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: RustPython/RustPython/.coderabbit.yml
Review profile: CHILL
Plan: Advanced
Run ID: 828ab72a-5242-47c1-8169-3a145aa551e8

📥 Commits

Reviewing files that changed from the base of the PR and between fc9f4a9 and edb0540.

⛔ Files ignored due to path filters: 14

Lib/_osx_support.py is excluded by !Lib/**
Lib/sysconfig/__init__.py is excluded by !Lib/**
Lib/sysconfig/__main__.py is excluded by !Lib/**
Lib/test/support/__init__.py is excluded by !Lib/**
Lib/test/test__osx_support.py is excluded by !Lib/**
Lib/test/test_sysconfig.py is excluded by !Lib/**
Lib/test/test_trace.py is excluded by !Lib/**
Lib/test/test_venv.py is excluded by !Lib/**
Lib/test/test_zipimport.py is excluded by !Lib/**
Lib/venv/__init__.py is excluded by !Lib/**
Lib/venv/scripts/nt/activate.bat is excluded by !Lib/**
Lib/venv/scripts/nt/deactivate.bat is excluded by !Lib/**
Lib/venv/scripts/posix/activate.csh is excluded by !Lib/**
Lib/zipimport.py is excluded by !Lib/**

📒 Files selected for processing: 13

crates/host_env/src/windows.rs
crates/vm/Lib/python_builtins/zipimport.py
crates/vm/build.rs
crates/vm/src/getpath.rs
crates/vm/src/stdlib/_sysconfig.rs
crates/vm/src/stdlib/_sysconfigdata.rs
crates/vm/src/stdlib/_thread.rs
crates/vm/src/stdlib/sys.rs
crates/vm/src/vm/setting.rs
scripts/update_lib/cmd_migrate.py
scripts/update_lib/tests/test_migrate.py
src/interpreter.rs
src/settings.rs

+-------------------------+
| Think twice, code once. |
+-------------------------+
  \
   \   (\__/)
       (•ㅅ•)
       / 　 づ

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-10-17T10:10:25Z

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
:4: SyntaxWarning: invalid decimal literal
:7: SyntaxWarning: "\z" is an invalid escape sequence. Such sequences will not work in the future. Did you mean "\z"? A raw string is also an option.
:10: SyntaxWarning: "\400" is an invalid octal escape sequence. Such sequences will not work in the future. Did you mean "\400"? A raw string is also an option.
[x] lib: cpython/Lib/venv
[x] test: cpython/Lib/test/test_venv.py (TODO: 18)

dependencies:

venv

dependent tests: (1 tests)

venv: test_venv

[ ] lib: cpython/Lib/sysconfig
[x] lib: cpython/Lib/_aix_support.py
[x] lib: cpython/Lib/_osx_support.py
[x] test: cpython/Lib/test/test_sysconfig.py (TODO: 7)
[x] test: cpython/Lib/test/test__osx_support.py

dependencies:

sysconfig (native: _sysconfig, _winapi, importlib.machinery, importlib.util, os.path, sys)
- _aix_support
- _osx_support
- contextlib (native: sys)
- re (native: _casefix, _constants, _sre, sys, unicodedata)
- subprocess (native: _posixsubprocess, _winapi, builtins, errno, fcntl, grp, msvcrt, pwd, select, sys, time)
- tempfile (native: _thread, errno, sys)
- _aix_support, _osx_support, importlib, json, os, threading, types, warnings

dependent tests: (58 tests)

sysconfig: test_asdl_parser test_c_locale_coercion test_cmd_line test_dtrace test_embed test_gc test_hashlib test_importlib test_inspect test_launcher test_os test_osx_env test_peg_generator test_pyexpat test_regrtest test_site test_ssl test_subprocess test_support test_sys test_sysconfig test_time test_tools test_urllib2net test_venv
- ctypes: test_android test_buffer test_bytes test_code test_codecs test_ctypes test_genericalias test_io test_ntpath test_os
  - platform: test__locale test__osx_support test_asyncio test_baseexception test_builtin test_cmath test_fcntl test_math test_mimetypes test_platform test_shutil test_socket test_strptime test_winreg test_wsgiref
  - webbrowser: test_webbrowser
- ensurepip: test_ensurepip
- pydoc: test_enum test_pydoc
  - pdb: test_pdb
  - xmlrpc.server: test_docxmlrpc test_xmlrpc
- trace: test_trace

[x] lib: cpython/Lib/zipimport.py
[x] test: cpython/Lib/test/test_zipimport.py (TODO: 2)
[ ] test: cpython/Lib/test/test_zipimport_support.py (TODO: 2)

dependencies:

zipimport

dependent tests: (9 tests)

zipimport: test_cmd_line_script test_importlib test_pkgutil test_zipimport test_zipimport_support
- pkgutil: test_pydoc test_pyrepl test_runpy
  - pydoc: test_enum

[ ] lib: cpython/Lib/trace.py
[x] test: cpython/Lib/test/test_trace.py (TODO: 2)

dependencies:

trace (native: main, gc, sys, time)
- argparse (native: sys)
- io (native: _io, _thread, errno, msvcrt, sys)
- pickle (native: _pickle, itertools, sys)
- runpy (native: importlib.machinery, importlib.util, sys)
- tokenize (native: _tokenize, builtins, itertools, sys)
- sysconfig
- dis, inspect, linecache, os, threading, token

dependent tests: (1 tests)

trace: test_trace

Legend:

[+] path exists in CPython
[x] up-to-date, [ ] outdated

Align zipimport and its tests with CPython 3.15.0 via update_lib, preserving inherited test guards. Keep build-time fallback paths after configured paths and the default ZIP, while respecting customized embedding paths. Assisted-by: OpenAI Codex:model version unavailable

Both canonical coverage-file tests execute successfully after preserving startup search order. Preserve the other two inherited guards and restore canonical Python 3.15 JIT optimizer decorators. Assisted-by: OpenAI Codex:model version unavailable

codspeed · 2026-10-17T12:06:01Z

Merging this PR will degrade performance by 13.34%

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 1 regressed benchmark
✅ 61 untouched benchmarks
⏩ 4 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
❌	`rustpython[frozenset.py]`	361 µs	416.6 µs	-13.34%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing youknowdot:fix/startup-paths-sysconfig-315 (66e80ef) with main (fc9f4a9)}

4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Import the pinned CPython 3.15 PythonSymlink class through a narrow, hash-pinned update_lib mode. Preserve canonical home/VPATH behavior and explicitly supply RustPython build stdlib to the Windows child without changing runtime PYTHONHOME semantics or skipping tests. Assisted-by: OpenAI Codex:model version unavailable

Reinitialize every live thread handle completion mutex and condition variable before skipping the surviving or not-yet-started handle. Preserve their completion payload and state, and retain the existing dead-thread cleanup. Assisted-by: OpenAI Codex:model version unavailable

chatgpt-codex-connector · 2026-10-18T06:53:14Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-18T07:01:25.567387Z	`edb0540`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: edb05403b5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-18T07:01:23Z

+    let search_dir = if venv_prefix.is_some()
+        && home_dir.is_none()
+        && explicit_home.is_none()
+        && paths.base_executable.is_empty()


Resolve no-home venvs from the base executable

When PYTHONEXECUTABLE or __PYVENV_LAUNCHER__ points at a venv whose readable pyvenv.cfg has no home key, the override has already made paths.base_executable nonempty, so this guard skips the target-installation search and calculates base_prefix from the venv's bin/Scripts directory (typically falling back to /usr/local on Unix) instead of the actual executable's installation. This gives the newly supported no-home/launcher case incorrect base prefixes and stdlib search paths; use the parent of paths.base_executable as the landmark search directory rather than disabling this branch.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-10-18T07:01:23Z

-            ('_base_executable', sys._base_executable),
+            # Usually compare to sys.prefix, but if we're running in our own
+            # venv then we really need to compare to our base prefix
+            ('base_prefix', sys.base_prefix),


Preserve the venv base-executable assertion

When test_sysconfig launches the newly created venv, replacing the _base_executable case with this base_prefix case removes the general regression check for the base-interpreter path even though this patch substantially changes calculate_base_executable(). Keep the former assertion and add the prefix assertion separately so an incorrect launcher/base executable cannot pass this suite; replacing or deleting test assertions is also explicitly prohibited by the repository test policy.

AGENTS.md reference: AGENTS.md:L277-L278

Useful? React with 👍 / 👎.

youknowdot added 3 commits October 17, 2026 17:47

Align startup home and venv paths with canonical sysconfig closure

dd682f7

Assisted-by: OpenAI Codex:model version unavailable

Preserve selected stdlib and empty home prefix fallback

4065936

Assisted-by: OpenAI Codex:model version unavailable

Collapse startup exec-prefix fallback condition

11d88b1

Assisted-by: OpenAI Codex:model version unavailable

youknowdot added 2 commits October 17, 2026 20:12

youknowdot added 2 commits October 18, 2026 11:14

youknowone marked this pull request as ready for review October 18, 2026 06:53

youknowone merged commit 6e8e7e0 into RustPython:main Oct 18, 2026
29 of 30 checks passed

chatgpt-codex-connector Bot reviewed Oct 18, 2026

View reviewed changes

Navigation Menu

Repository navigation

Align startup paths and sysconfig/venv with Python 3.15 - #9174

Align startup paths and sysconfig/venv with Python 3.15#9174
youknowone merged 7 commits into
RustPython:mainfrom
youknowdot:fix/startup-paths-sysconfig-315

youknowdot commented Oct 17, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 17, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 17, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Oct 17, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 18, 2026 •

edited

Loading

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Oct 18, 2026

Uh oh!

chatgpt-codex-connector Bot Oct 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

youknowdot commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Align startup paths and the CPython 3.15 sysconfig/venv closure

Changes

Symlink-helper repair at bc27c61

Fork completion-lock repair at edb0540

Previous repair validation

Previous hosted repair results

Remaining platform and hosted limitations

AI assistance

Uh oh!

coderabbitai Bot commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 Library Dependencies

Uh oh!

codspeed Bot commented Oct 17, 2026

Merging this PR will degrade performance by 13.34%

Performance Changes

Footnotes

Uh oh!

chatgpt-codex-connector Bot commented Oct 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 18, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Oct 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

youknowdot commented Oct 17, 2026 •

edited

Loading

Symlink-helper repair at `bc27c61`

Fork completion-lock repair at `edb0540`

coderabbitai Bot commented Oct 17, 2026 •

edited

Loading

github-actions Bot commented Oct 17, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 18, 2026 •

edited

Loading