Skip to content

Optimize startup performance by deferring non-critical imports - #15353

Draft
Carreau wants to merge 9 commits into
mainfrom
claude/ipython-startup-perf-25ge56
Draft

Optimize startup performance by deferring non-critical imports#15353
Carreau wants to merge 9 commits into
mainfrom
claude/ipython-startup-perf-25ge56

Conversation

@Carreau

@Carreau Carreau commented Aug 4, 2026

Copy link
Copy Markdown
Member

This PR continues the effort to reduce IPython's startup time by moving more work off the critical path. The changes focus on deferring imports and computations that are not needed until they're actually used.

Key Changes

  • Kitty graphics detection optimization: On Linux, the process tree walk now reads /proc/<pid>/stat directly instead of importing psutil, avoiding ~10ms of startup overhead. The psutil fallback is retained for macOS and systems without /proc.

  • Pygments style resolution optimization: Resolving a theme's base pygments style no longer imports pygments.styles and its plugin machinery for builtin styles. The style modules are read directly for the handful of builtin styles IPython uses, while plugin styles still resolve through the normal path.

  • Lazy prompt style building: The prompt style is now built on first use rather than being rebuilt multiple times during shell initialization. This is particularly beneficial for non-interactive runs that never draw a prompt.

  • Operator dunder mapping extraction: Moved BINARY_OP_DUNDERS, COMP_OP_DUNDERS, UNARY_OP_DUNDERS, and _find_dunder from guarded_eval.py to a new _dunder_ops.py module. This allows terminal shortcut filters to resolve operators without importing the entire guarded evaluation machinery and its dependencies (typing_extensions, dataclasses, inspect).

  • Single-use imports deferred: Moved imports of platform, pprint, textwrap, html, mimetypes, locale, glob, and runpy to their actual use sites rather than module level.

  • Test improvements: Enhanced test coverage for the new /proc-based process detection with comprehensive tests for edge cases (spaces in process names, unreadable ancestors, etc.). Added a test verifying psutil is not imported on Linux.

Performance Impact

These changes collectively reduce startup time by approximately 13% for interactive ipython in a real terminal, 9% for ipython -c pass, and eliminate ~43 modules from an interactive start, building on previous optimization rounds.

Implementation Details

  • The /proc reading implementation handles process names with spaces and parentheses by parsing from the last ) character in /proc/<pid>/stat
  • Permission errors when reading /proc (from hidepid on multi-user systems) are handled gracefully by stopping the ancestor walk
  • Pygments style module execution is isolated and not registered in sys.modules to avoid breaking later import statements
  • The prompt style cache uses a private attribute with property accessors to ensure lazy initialization

https://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR

claude added 9 commits August 3, 2026 18:38
Resolving a theme's base style went through `pygments.styles`, whose module
body imports `pygments.plugin` and with it `importlib.metadata`, `email` and
`zipfile` -- about 10ms and 36 modules on every `ipython` start, all of it so
that third party *style plugins* can be found by name. No theme IPython ships
needs that.

Both call sites only ever want the base style's `styles` mapping, so add
`PyColorize._pygments_base_styles`, which for the handful of builtin styles
IPython's own themes are based on executes the single `pygments/styles/*.py`
module that defines them and reads the mapping out of it. Anything else --
an unknown name, or a style that has moved within pygments -- still goes
through `pygments.styles.get_style_by_name`, plugins included, so this is a
shortcut and never the source of truth.

The style module is deliberately not registered in `sys.modules`: it is pure
data, and registering a submodule of a package that is not itself imported
breaks later `import pygments.styles.<mod>` statements.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR
`refresh_style()` turns a whole pygments style into prompt_toolkit style
rules, and is called several times while a shell is set up: once from
`init_syntax_highlighting` when `colors` is first set, again from
`init_magics`, again when the prompt_toolkit application is created. Only the
last of those states is ever observed, and for a run that never draws a
prompt -- `ipython -c ...`, `--simple-prompt`, a kernel, the test suite --
none of them are.

Make `refresh_style()` invalidate a cache instead, and build the style behind
a `_style` property. `_style` is only ever read through the `DynamicStyle`
the prompt session renders with, so the rebuild lands on the first render.
The property keeps a setter so that assigning to `_style` still works.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR
`IPython.terminal.shortcuts.filters` evaluates the filter expressions of the
built-in key bindings at import time, and needed three names from
`IPython.core.guarded_eval` to do it: `BINARY_OP_DUNDERS`, `UNARY_OP_DUNDERS`
and `_find_dunder`. Importing `guarded_eval` for them cost ~5ms and pulled in
`typing_extensions` on every terminal start, for two dicts and a six-line
helper that only need `ast`.

Move those (and `COMP_OP_DUNDERS`, which belongs with them) to
`IPython.core._dunder_ops`, and re-export them from `guarded_eval`, which
stays their documented home.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR
Four more stdlib imports that every `import IPython` paid for and almost no
session used:

- `platform` in `IPython.utils.sysinfo`, needed only by `pkg_info()`;
- `pprint` in the same module, needed only by `sys_info()`;
- `textwrap` in `IPython.core.interactiveshell`, needed only to dedent the
  banner used when `SOURCE_DATE_EPOCH` is set;
- `html` (and with it `html.entities`) in `IPython.core.oinspect`, needed
  only to build the html mime representation of an inspected object;
- `platform` in `IPython.lib.pretty`, needed only to special-case PyPy when
  pretty-printing a `super` object.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR
The kitty graphics autodetection walks the process tree looking for a
terminal that speaks the protocol. It runs on every interactive start -- a
headless `ipython -c ...` returns early on the isatty check, which is why
this never showed up in startup profiles -- and importing psutil for it costs
~13ms and 6 modules.

On Linux, ``/proc/<pid>/stat`` holds both the name psutil would report and
the parent pid, so one read per ancestor is all the walk needs. Everywhere
else -- macOS, or a Linux without ``/proc`` -- fall back to psutil as before.
`comm` in that file is unescaped and may contain spaces and parentheses, so
the name is taken from the last ``)``; it is also truncated to 15 characters,
which cannot affect the outcome since every terminal name matched against is
far shorter.

Also replace `platform.system()` with `sys.platform`, which answers the same
question without importing `platform` (another ~1.5ms).

Together this takes 48 modules off an interactive start.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR
Four more single-use stdlib imports on the `import IPython` path:

- `mimetypes` (and the `winreg`/`_winapi` probing it does) in
  `IPython.core.display`, used only to guess the type of an embedded video;
- `locale` in `IPython.utils.encoding`, consulted only when the stream
  encoding is missing or ascii -- which it usually is not, so this was an
  import for a call that mostly never happened;
- `glob` in `IPython.utils.path`, used only by `shellglob()`;
- `runpy` in `IPython.core.interactiveshell`, used only by
  `safe_run_module()`.

Making `mimetypes` lazy surfaced that `tests/test_path.py` registers a stub
`winreg` module in `sys.modules` and never removes it, so any stdlib module
importing `winreg` later in the session -- `mimetypes` does, to decide
whether to read the Windows registry -- found the stub and tried to use it.
The only test that needs `winreg` runs on Windows only, so import it there
and leave `sys.modules` alone everywhere else.

The encoding tests reached `locale` through `IPython.utils.encoding.locale`;
patch the `locale` module itself, which is the same object either way.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR
`test_supports_kitty_graphics_does_not_import_psutil_on_linux` set
`sys.platform = "linux"` and asserted the walk answers without psutil, but on
the Windows and macOS runners there is no `/proc`, so detection correctly fell
back to psutil and the test failed -- the only failure in the six red jobs.

Skip it unless `/proc/self` is really there, and stop faking `sys.platform`:
what the test is for is that a real Linux answers without psutil. The psutil
branch is still covered on every platform by
`test_supports_kitty_graphics_handles_psutil_access_denied`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9XqGGqj9CZhSXWJ49t7fR
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants