Optimize startup performance by deferring non-critical imports - #15353
Draft
Carreau wants to merge 9 commits into
Draft
Optimize startup performance by deferring non-critical imports#15353Carreau wants to merge 9 commits into
Carreau wants to merge 9 commits into
Conversation
Resolving a theme's base style went through `pygments.styles`, whose module body imports `pygments.plugin` and with it `importlib.metadata`, `email` and `zipfile` -- about 10ms and 36 modules on every `ipython` start, all of it so that third party *style plugins* can be found by name. No theme IPython ships needs that. Both call sites only ever want the base style's `styles` mapping, so add `PyColorize._pygments_base_styles`, which for the handful of builtin styles IPython's own themes are based on executes the single `pygments/styles/*.py` module that defines them and reads the mapping out of it. Anything else -- an unknown name, or a style that has moved within pygments -- still goes through `pygments.styles.get_style_by_name`, plugins included, so this is a shortcut and never the source of truth. The style module is deliberately not registered in `sys.modules`: it is pure data, and registering a submodule of a package that is not itself imported breaks later `import pygments.styles.<mod>` statements. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR
`refresh_style()` turns a whole pygments style into prompt_toolkit style rules, and is called several times while a shell is set up: once from `init_syntax_highlighting` when `colors` is first set, again from `init_magics`, again when the prompt_toolkit application is created. Only the last of those states is ever observed, and for a run that never draws a prompt -- `ipython -c ...`, `--simple-prompt`, a kernel, the test suite -- none of them are. Make `refresh_style()` invalidate a cache instead, and build the style behind a `_style` property. `_style` is only ever read through the `DynamicStyle` the prompt session renders with, so the rebuild lands on the first render. The property keeps a setter so that assigning to `_style` still works. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR
`IPython.terminal.shortcuts.filters` evaluates the filter expressions of the built-in key bindings at import time, and needed three names from `IPython.core.guarded_eval` to do it: `BINARY_OP_DUNDERS`, `UNARY_OP_DUNDERS` and `_find_dunder`. Importing `guarded_eval` for them cost ~5ms and pulled in `typing_extensions` on every terminal start, for two dicts and a six-line helper that only need `ast`. Move those (and `COMP_OP_DUNDERS`, which belongs with them) to `IPython.core._dunder_ops`, and re-export them from `guarded_eval`, which stays their documented home. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR
Four more stdlib imports that every `import IPython` paid for and almost no session used: - `platform` in `IPython.utils.sysinfo`, needed only by `pkg_info()`; - `pprint` in the same module, needed only by `sys_info()`; - `textwrap` in `IPython.core.interactiveshell`, needed only to dedent the banner used when `SOURCE_DATE_EPOCH` is set; - `html` (and with it `html.entities`) in `IPython.core.oinspect`, needed only to build the html mime representation of an inspected object; - `platform` in `IPython.lib.pretty`, needed only to special-case PyPy when pretty-printing a `super` object. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR
The kitty graphics autodetection walks the process tree looking for a terminal that speaks the protocol. It runs on every interactive start -- a headless `ipython -c ...` returns early on the isatty check, which is why this never showed up in startup profiles -- and importing psutil for it costs ~13ms and 6 modules. On Linux, ``/proc/<pid>/stat`` holds both the name psutil would report and the parent pid, so one read per ancestor is all the walk needs. Everywhere else -- macOS, or a Linux without ``/proc`` -- fall back to psutil as before. `comm` in that file is unescaped and may contain spaces and parentheses, so the name is taken from the last ``)``; it is also truncated to 15 characters, which cannot affect the outcome since every terminal name matched against is far shorter. Also replace `platform.system()` with `sys.platform`, which answers the same question without importing `platform` (another ~1.5ms). Together this takes 48 modules off an interactive start. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR
Four more single-use stdlib imports on the `import IPython` path: - `mimetypes` (and the `winreg`/`_winapi` probing it does) in `IPython.core.display`, used only to guess the type of an embedded video; - `locale` in `IPython.utils.encoding`, consulted only when the stream encoding is missing or ascii -- which it usually is not, so this was an import for a call that mostly never happened; - `glob` in `IPython.utils.path`, used only by `shellglob()`; - `runpy` in `IPython.core.interactiveshell`, used only by `safe_run_module()`. Making `mimetypes` lazy surfaced that `tests/test_path.py` registers a stub `winreg` module in `sys.modules` and never removes it, so any stdlib module importing `winreg` later in the session -- `mimetypes` does, to decide whether to read the Windows registry -- found the stub and tried to use it. The only test that needs `winreg` runs on Windows only, so import it there and leave `sys.modules` alone everywhere else. The encoding tests reached `locale` through `IPython.utils.encoding.locale`; patch the `locale` module itself, which is the same object either way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR
`test_supports_kitty_graphics_does_not_import_psutil_on_linux` set `sys.platform = "linux"` and asserted the walk answers without psutil, but on the Windows and macOS runners there is no `/proc`, so detection correctly fell back to psutil and the test failed -- the only failure in the six red jobs. Skip it unless `/proc/self` is really there, and stop faking `sys.platform`: what the test is for is that a real Linux answers without psutil. The psutil branch is still covered on every platform by `test_supports_kitty_graphics_handles_psutil_access_denied`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR continues the effort to reduce IPython's startup time by moving more work off the critical path. The changes focus on deferring imports and computations that are not needed until they're actually used.
Key Changes
Kitty graphics detection optimization: On Linux, the process tree walk now reads
/proc/<pid>/statdirectly instead of importing psutil, avoiding ~10ms of startup overhead. The psutil fallback is retained for macOS and systems without/proc.Pygments style resolution optimization: Resolving a theme's base pygments style no longer imports
pygments.stylesand its plugin machinery for builtin styles. The style modules are read directly for the handful of builtin styles IPython uses, while plugin styles still resolve through the normal path.Lazy prompt style building: The prompt style is now built on first use rather than being rebuilt multiple times during shell initialization. This is particularly beneficial for non-interactive runs that never draw a prompt.
Operator dunder mapping extraction: Moved
BINARY_OP_DUNDERS,COMP_OP_DUNDERS,UNARY_OP_DUNDERS, and_find_dunderfromguarded_eval.pyto a new_dunder_ops.pymodule. This allows terminal shortcut filters to resolve operators without importing the entire guarded evaluation machinery and its dependencies (typing_extensions,dataclasses,inspect).Single-use imports deferred: Moved imports of
platform,pprint,textwrap,html,mimetypes,locale,glob, andrunpyto their actual use sites rather than module level.Test improvements: Enhanced test coverage for the new
/proc-based process detection with comprehensive tests for edge cases (spaces in process names, unreadable ancestors, etc.). Added a test verifying psutil is not imported on Linux.Performance Impact
These changes collectively reduce startup time by approximately 13% for interactive
ipythonin a real terminal, 9% foripython -c pass, and eliminate ~43 modules from an interactive start, building on previous optimization rounds.Implementation Details
/procreading implementation handles process names with spaces and parentheses by parsing from the last)character in/proc/<pid>/stat/proc(fromhidepidon multi-user systems) are handled gracefully by stopping the ancestor walksys.modulesto avoid breaking later import statementshttps://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR