add stub-driven lazy imports using lazy_loader - #3459
Conversation
Replace DeepLabCut's hand-maintained lazy export maps (_API_EXPORTS_MAP, _OPTIONAL_EXPORTS, and the __all__ export lists) with Scientific Python's `lazy_loader.attach_stub`. `deeplabcut/__init__.pyi` is now the single declarative source of truth for the public API: static tools read it for discoverability and signatures, while lazy_loader parses it at runtime to install `__getattr__`, `__dir__`, and `__all__`, importing each implementation module only when its attribute is first accessed. This keeps the flat API intact and avoids eagerly importing heavy or optional modules such as `pose_estimation_pytorch`, `gui`, and `pose_tracking_pytorch` during `import deeplabcut`. - add `lazy_loader` as a runtime dependency, - add `__init__.pyi` and `py.typed` - add tests for the new loading behavior.
| # ----------------------------------------------------------------------------- | ||
| # Public API | ||
| # ----------------------------------------------------------------------------- | ||
| _TORCH_EXPORTS = frozenset({"transformer_reID"}) |
There was a problem hiding this comment.
How do you rate this in terms of maintainability?
Could we parse from the pyproject somehow?
Or make a shared helper that is called by all e.g. GUI modules that tries to import gui deps,
and raises directly an import error asking to install the optional dep? This way each module with optional deps has clear guidance messages, we have a centralized file with all of those at a glance, and it only couples it to local source code imports without risking drift with the pyproject
| "VERSION", | ||
| "DEBUG", | ||
| ] | ||
| _GUI_DEPENDENCY_MODULES = frozenset({"PySide6", "napari", "qdarkstyle"}) |
There was a problem hiding this comment.
Should this be broader? What is napari-deeplabcut is missing?
There was a problem hiding this comment.
Will this ever run in CI or is it meant as local only?
|
|
||
|
|
||
| @pytest.mark.skipif( | ||
| not (_module_available("torch") and _module_available("PySide6")), |
There was a problem hiding this comment.
The skip condition does not represent the full optional dependency set, right? With PySide6 installed but missing another GUI dep, this would run. Can we mark this test in the CI test selector maybe? Or maybe there are other minimal solutions that don't require copying the deps from pyproject at all
| def __getattr__(name: str): | ||
| try: | ||
| return _lazy_getattr(name) | ||
| except ModuleNotFoundError as exc: | ||
| if name in _GUI_EXPORTS and _is_missing_gui_dependency(exc): | ||
| raise ImportError( | ||
| f"{name!r} requires the DeepLabCut GUI dependencies. Install the supported GUI extra." | ||
| ) from exc | ||
|
|
||
| _OPTIONAL_API_EXPORTS = list(_OPTIONAL_EXPORTS) | ||
| if name in _TORCH_EXPORTS and _is_missing_torch_dependency(exc): | ||
| raise ImportError(f"{name!r} requires the PyTorch tracking dependencies.") from exc | ||
|
|
||
| __all__ = ( | ||
| _VERSION_EXPORTS | ||
| + _CORE_EXPORTS | ||
| + _PROJECT_EXPORTS | ||
| + _DATASET_EXPORTS | ||
| + _API_EXPORTS | ||
| + _UTIL_EXPORTS | ||
| + _THREE_D_EXPORTS | ||
| + _OPTIONAL_API_EXPORTS | ||
| ) | ||
| raise |
There was a problem hiding this comment.
Maybe good to mention: this changes e.g. hasattr(deeplabcut, "launch_dlc") from returning False to raising.
I would assume this is fine in most cases but still a bit of a sneaky change
| from .utils.auxfun_videos import DownSampleVideo as DownSampleVideo | ||
| from .utils.auxfun_videos import ShortenVideo as ShortenVideo | ||
| from .utils.auxfun_videos import check_video_integrity as check_video_integrity | ||
| from .utils.auxfun_videos import collect_video_paths as collect_video_paths |
There was a problem hiding this comment.
This was not present before, is adding intentional?
Summary
Replace DeepLabCut's hand-maintained lazy export maps (_API_EXPORTS_MAP, _OPTIONAL_EXPORTS, and the all export lists) with Scientific Python's
lazy_loader.attach_stub.deeplabcut/__init__.pyiis now the source of truth for public API. Type checkers can use use it, and at runtimelazy_loaderparses it to install__getattr__,__dir__, and__all__, as well.Each implementation module is only imported when its attribute is first accessed. This avoids eagerly importing heavy or optional modules such as
pose_estimation_pytorch,gui, andpose_tracking_pytorchduringimport deeplabcut.Changes
lazy_loaderas a runtime dependency,__init__.pyiandpy.typed