Skip to content

Synthesize __getattr__ on untyped parents of typed subpackages#21552

Open
Hnasar wants to merge 1 commit into
python:masterfrom
Hnasar:fix-untyped-parent-pkg
Open

Synthesize __getattr__ on untyped parents of typed subpackages#21552
Hnasar wants to merge 1 commit into
python:masterfrom
Hnasar:fix-untyped-parent-pkg

Conversation

@Hnasar
Copy link
Copy Markdown
Contributor

@Hnasar Hnasar commented May 28, 2026

Fixes #16149.

When mypy follows a third-party import pkg.typed and pkg itself has no py.typed, pkg is loaded as an empty namespace package -- its __init__.py is never read. Subsequent pkg.name access from any other file then raises attr-defined, even when __init__.py re-exports name at runtime.

The motivating case: numba ships numba/typed/py.typed but not numba/py.typed. Any source that does import numba.typed makes every @numba.jit(...) site elsewhere fail with

Module has no attribute "jit"

The existing workarounds (follow_imports = "skip" or per-module follow_untyped_imports = True) both require the user to know the shape of every dependency they pull in.

Call chain for an example repro of the bug:

  1. foo.py does import numba.typed.
  2. FindModuleCache._find_module("numba.typed") calls _find_module_non_stub_helper, which walks the components and finds numba/typed/py.typed at iteration 1 -- so the helper returns the parent path (pkg_dir/numba, False) (the typed sub is reachable).
  3. Back in _find_module, line 496 calls _update_ns_ancestors(["numba", "typed"], (pkg_dir/numba, False)). The loop's first iteration sets ns_ancestors["numba"] = pkg_dir/numba -- even though numba/__init__.py exists, i.e. numba is a regular package, not a namespace package.
  4. Loading numba.typed requires its parent numba as an ancestor (State.add_ancestors). mypy calls find_module("numba").
  5. _find_module_non_stub_helper("numba", pkg_dir) returns FOUND_WITHOUT_TYPE_HINTS (no numba/py.typed). That should be the final answer -- but at line 595 the fallback ancestor = self.ns_ancestors.get("numba") hits the entry written in step 3 and returns the directory path instead. numba is now "found" with a real path.
  6. mypy parses numba/__init__.py (silenced, because it was found by following imports into site-packages). The line from numba.core import jit triggers find_module("numba.core") -> FOUND_WITHOUT_TYPE_HINTS. Because the parent is being processed under silenced follow-imports, the resulting ModuleNotFound doesn't surface and numba.core is never analyzed; jit never enters numba's symbol table.
  7. main.py does import numba. Same find result as step 5 (cached). The symbol table from step 6 is reused -- it lacks jit. The access numba.jit raises attr-defined.

My initial fix (not the one in this commit) had been to avoid adding a package to ns_ancestors in step 3 above, but that has the downside that usages of import numba.typed become Any, compared with the __getattr__ fix.

Instead, this more robust fix injects a module-level __getattr__: (str) -> Any into the parent's symbol table, which allows the typed submodule to stay typed.

This synthesized annotation is added when:

  1. the State isn't a stub, has no definitions, and no other gettattr
  2. the module was previously only found as a parent module of a py.typed
  3. self.path is a directory (treated as a namespace package), but there's actually a __init__.py[i] file there.

The synthetic annotation is added with module_public=False and module_hidden=True so direct user accesses like pkg.__getattr__ and from pkg import __getattr__ fall through to
types.ModuleType.__getattr__ from typeshed -- the same answer a plain untyped module would give. mypy's internal getattr-fallback path reads tree.names["__getattr__"] directly and bypasses both flags, so the synthetic is still consulted for pkg.X lookups.

lookup_module_name in semanal keeps its existing priority order: a real submodule already loaded into self.modules wins first, and only unresolved attributes fall back to __getattr__. So:

  • numba.jit -- resolves through __getattr__ to Any instead of raising attr-defined.
  • numba.typed resolves to the typed submodule.
  • Direct binding forms (from numba.typed import X, from numba import typed, import numba.typed as ts) remain typed.

The helper needs find_module_cache.ns_ancestors and two cached FS reads via fscache, both owned by BuildManager, so doing the work in State reuses self.manager natively. Since semantic_analysis_pass1 runs once per module and uses a number of early exits. When analyzing the State per-file --timing-stats the change very minimal (20us).

(Claude 4.7 used brainstorming different iterations of this fix.)

(Explain how this PR changes mypy.)

@github-actions

This comment has been minimized.

Fixes python#16149.

When mypy follows a third-party `import pkg.typed` and `pkg` itself
has no `py.typed`, `pkg` is loaded as an empty namespace package -- its
`__init__.py` is never read. Subsequent `pkg.name` access from any other
file then raises `attr-defined`, even when `__init__.py` re-exports
`name` at runtime.

The motivating case: numba ships `numba/typed/py.typed` but not
`numba/py.typed`. Any source that does `import numba.typed` makes
every `@numba.jit(...)` site elsewhere fail with

    Module has no attribute "jit"

The existing workarounds (`follow_imports = "skip"` or per-module
`follow_untyped_imports = True`) both require the user to know the
shape of every dependency they pull in.

Call chain for an example repro of the bug:

1. `foo.py` does `import numba.typed`.
2. `FindModuleCache._find_module("numba.typed")` calls
   `_find_module_non_stub_helper`, which walks the components and finds
   `numba/typed/py.typed` at iteration 1 -- so the helper returns
   the parent path `(pkg_dir/numba, False)` (the typed sub is reachable).
3. Back in `_find_module`, line 496 calls
   `_update_ns_ancestors(["numba", "typed"], (pkg_dir/numba, False))`.
   The loop's first iteration sets `ns_ancestors["numba"] = pkg_dir/numba`
   -- even though `numba/__init__.py` exists, i.e. `numba` is a regular
   package, not a namespace package.
4. Loading `numba.typed` requires its parent `numba` as an ancestor
   (`State.add_ancestors`). mypy calls `find_module("numba")`.
5. `_find_module_non_stub_helper("numba", pkg_dir)` returns
   `FOUND_WITHOUT_TYPE_HINTS` (no `numba/py.typed`). That should be the
   final answer -- but at line 595 the fallback
   `ancestor = self.ns_ancestors.get("numba")` hits the entry written
   in step 3 and returns the directory path instead. `numba` is now
   "found" with a real path.
6. mypy parses `numba/__init__.py` (silenced, because it was found by
   following imports into site-packages). The line
   `from numba.core import jit` triggers
   `find_module("numba.core")` -> `FOUND_WITHOUT_TYPE_HINTS`. Because
   the parent is being processed under silenced follow-imports, the
   resulting `ModuleNotFound` doesn't surface and `numba.core` is never
   analyzed; `jit` never enters `numba`'s symbol table.
7. `main.py` does `import numba`. Same find result as step 5 (cached).
   The symbol table from step 6 is reused -- it lacks `jit`. The access
   `numba.jit` raises `attr-defined`.

My initial fix (not the one in this commit) had been to avoid adding a
package to ns_ancestors in step 3 above, but that has the downside that
usages of `import numba.typed` become `Any`, compared with the
`__getattr__` fix.

Instead, this more robust fix injects a module-level
`__getattr__: (str) -> Any` into the parent's symbol table, which allows
the typed submodule to stay typed.

This synthesized annotation is added when:

1. the State isn't a stub, has no definitions, and no other gettattr
2. the module was previously only found as a parent module of a py.typed
3. self.path is a directory (treated as a namespace package), but
   there's actually a `__init__.py[i]` file there.

The synthetic annotation is added with `module_public=False` and
`module_hidden=True` so direct user accesses like `pkg.__getattr__`
and `from pkg import __getattr__` fall through to
`types.ModuleType.__getattr__` from typeshed -- the same answer a
plain untyped module would give. mypy's internal __getattr__-fallback
path reads `tree.names["__getattr__"]` directly and bypasses both
flags, so the synthetic is still consulted for `pkg.X` lookups.

`lookup_module_name` in semanal keeps its existing priority order: a
real submodule already loaded into `self.modules` wins first, and
only unresolved attributes fall back to `__getattr__`. So:

- `numba.jit` -- resolves through `__getattr__` to `Any` instead of
  raising `attr-defined`.
- `numba.typed` resolves to the typed submodule.
- Direct binding forms (`from numba.typed import X`,
  `from numba import typed`, `import numba.typed as ts`)
  remain typed.

The helper needs `find_module_cache.ns_ancestors` and two cached FS
reads via `fscache`, both owned by `BuildManager`, so doing the work in
`State` reuses `self.manager` natively. Since `semantic_analysis_pass1`
runs once per module and uses a number of early exits. When analyzing
the State per-file --timing-stats the change very minimal (20us).

(Claude 4.7 used brainstorming different iterations of this fix.)
@Hnasar Hnasar force-pushed the fix-untyped-parent-pkg branch from 12dd987 to 02767a9 Compare May 28, 2026 04:57
@github-actions
Copy link
Copy Markdown
Contributor

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unintuitive behaviour when only subpackage provides py.typed

1 participant