BUG, TYP: add the remaining _core.multiarray function runtime signatures#30147
Conversation
_core.multiarray function runtime signatures_core.multiarray function runtime signatures
|
Hmm apparently the array api testsuite doesn't understand that |
This comment has been minimized.
This comment has been minimized.
|
Ah woops, |
_core.multiarray function runtime signatures_core.multiarray function runtime signatures
This comment has been minimized.
This comment has been minimized.
|
In order to be compatible with the Array API spec for This won't cause any backwards-incompatibily issues at runtime, as you can still call But as mypy_primer shows, calling |
|
re: the primer diff (#30147 (comment)): The errors for Jax and Colour look correct to me; The xarray errors are a consequence of the |
21d7edc to
fa476b5
Compare
|
Diff from mypy_primer, showing the effect of this PR on type check results on a corpus of open source code: jax (https://github.com/google/jax)
+ jax/_src/numpy/lax_numpy.py:5994: error: Argument 1 to "arange" has incompatible type "Array | ndarray[tuple[Any, ...], dtype[Any]] | numpy.bool[builtins.bool] | number[Any, int | float | complex] | builtins.bool | int | float | complex | TypedNdArray | int | Any"; expected "integer[Any] | floating[Any] | datetime64[date | int | None] | timedelta64[timedelta | int | None] | float" [arg-type]
+ jax/_src/scipy/special.py:1211: error: Incompatible return value type (got "signedinteger[_64Bit]", expected "ndarray[tuple[Any, ...], dtype[Any]]") [return-value]
xarray (https://github.com/pydata/xarray)
+ xarray/tests/test_plot.py: note: In member "setUp" of class "TestDiscreteColorMap":
+ xarray/tests/test_plot.py:1167: error: No overload variant of "arange" matches argument types "int", "int", "int" [call-overload]
+ xarray/tests/test_plot.py:1167: note: Possible overload variants:
+ xarray/tests/test_plot.py:1167: note: def [_ArangeScalarT: integer[Any] | floating[Any] | datetime64[date | int | None] | timedelta64[timedelta | int | None]] arange(integer[Any] | floating[Any] | datetime64[date | int | None] | timedelta64[timedelta | int | None] | float, /, stop: integer[Any] | floating[Any] | datetime64[date | int | None] | timedelta64[timedelta | int | None] | float | None = ..., step: integer[Any] | floating[Any] | datetime64[date | int | None] | timedelta64[timedelta | int | None] | float | None = ..., *, dtype: type[_ArangeScalarT] | dtype[_ArangeScalarT] | _SupportsDType[dtype[_ArangeScalarT]], device: Literal['cpu'] | None = ..., like: _SupportsArrayFunc | None = ...) -> ndarray[tuple[int], dtype[_ArangeScalarT]]
+ xarray/tests/test_plot.py:1167: note: def arange(int | integer[Any] | numpy.bool[builtins.bool], /, stop: int | integer[Any] | numpy.bool[builtins.bool] | None = ..., step: int | integer[Any] | numpy.bool[builtins.bool] | None = ..., *, dtype: type[int] | type[signedinteger[_32Bit | _64Bit]] | dtype[signedinteger[_32Bit | _64Bit]] | _SupportsDType[dtype[signedinteger[_32Bit | _64Bit]]] | None = ..., device: Literal['cpu'] | None = ..., like: _SupportsArrayFunc | None = ...) -> ndarray[tuple[int], dtype[signedinteger[_32Bit | _64Bit]]]
+ xarray/tests/test_plot.py:1167: note: def arange(float | floating[Any], /, stop: float | floating[Any] | integer[Any] | numpy.bool[builtins.bool] | None = ..., step: float | floating[Any] | integer[Any] | numpy.bool[builtins.bool] | None = ..., *, dtype: type[float] | type[float64] | dtype[float64] | _SupportsDType[dtype[float64]] | None = ..., device: Literal['cpu'] | None = ..., like: _SupportsArrayFunc | None = ...) -> ndarray[tuple[int], dtype[float64 | Any]]
+ xarray/tests/test_plot.py:1167: note: def arange(float | floating[Any] | integer[Any] | numpy.bool[builtins.bool], /, stop: float | floating[Any], step: float | floating[Any] | integer[Any] | numpy.bool[builtins.bool] | None = ..., *, dtype: type[float] | type[float64] | dtype[float64] | _SupportsDType[dtype[float64]] | None = ..., device: Literal['cpu'] | None = ..., like: _SupportsArrayFunc | None = ...) -> ndarray[tuple[int], dtype[float64 | Any]]
+ xarray/tests/test_plot.py:1167: note: def arange(timedelta64[timedelta | int | None], /, stop: int | timedelta64[timedelta | int | None] | integer[Any] | numpy.bool[builtins.bool] | None = ..., step: int | timedelta64[timedelta | int | None] | integer[Any] | numpy.bool[builtins.bool] | None = ..., *, dtype: type[timedelta64[timedelta | int | None]] | dtype[timedelta64[timedelta | int | None]] | _SupportsDType[dtype[timedelta64[timedelta | int | None]]] | None = ..., device: Literal['cpu'] | None = ..., like: _SupportsArrayFunc | None = ...) -> ndarray[tuple[int], dtype[timedelta64[Any]]]
+ xarray/tests/test_plot.py:1167: note: def arange(int | timedelta64[timedelta | int | None] | integer[Any] | numpy.bool[builtins.bool], /, stop: timedelta64[timedelta | int | None], step: int | timedelta64[timedelta | int | None] | integer[Any] | numpy.bool[builtins.bool] | None = ..., *, dtype: type[timedelta64[timedelta | int | None]] | dtype[timedelta64[timedelta | int | None]] | _SupportsDType[dtype[timedelta64[timedelta | int | None]]] | None = ..., device: Literal['cpu'] | None = ..., like: _SupportsArrayFunc | None = ...) -> ndarray[tuple[int], dtype[timedelta64[Any]]]
+ xarray/tests/test_plot.py:1167: note: def arange(datetime64[date | int | None], /, stop: datetime64[date | int | None], step: int | timedelta64[timedelta | int | None] | integer[Any] | numpy.bool[builtins.bool] | None = ..., *, dtype: type[datetime64[date | int | None]] | dtype[datetime64[date | int | None]] | _SupportsDType[dtype[datetime64[date | int | None]]] | None = ..., device: Literal['cpu'] | None = ..., like: _SupportsArrayFunc | None = ...) -> ndarray[tuple[int], dtype[datetime64[Any]]]
+ xarray/tests/test_plot.py:1167: note: def arange(integer[Any] | floating[Any] | datetime64[date | int | None] | timedelta64[timedelta | int | None] | float, /, stop: integer[Any] | floating[Any] | datetime64[date | int | None] | timedelta64[timedelta | int | None] | float | None = ..., step: integer[Any] | floating[Any] | datetime64[date | int | None] | timedelta64[timedelta | int | None] | float | None = ..., *, dtype: type[Any] | dtype[Any] | _SupportsDType[dtype[Any]] | tuple[Any, Any] | list[Any] | _DTypeDict | str | None = ..., device: Literal['cpu'] | None = ..., like: _SupportsArrayFunc | None = ...) -> ndarray[tuple[int], dtype[Any]]
+ xarray/tests/test_plot.py:1168: error: No overload variant of "arange" matches argument types "int", "int", "int" [call-overload]
+ xarray/tests/test_plot.py:1168: note: Possible overload variants:
+ xarray/tests/test_plot.py:1168: note: def [_ArangeScalarT: integer[Any] | floating[Any] | datetime64[date | int | None] | timedelta64[timedelta | int | None]] arange(integer[Any] | floating[Any] | datetime64[date | int | None] | timedelta64[timedelta | int | None] | float, /, stop: integer[Any] | floating[Any] | datetime64[date | int | None] | timedelta64[timedelta | int | None] | float | None = ..., step: integer[Any] | floating[Any] | datetime64[date | int | None] | timedelta64[timedelta | int | None] | float | None = ..., *, dtype: type[_ArangeScalarT] | dtype[_ArangeScalarT] | _SupportsDType[dtype[_ArangeScalarT]], device: Literal['cpu'] | None = ..., like: _SupportsArrayFunc | None = ...) -> ndarray[tuple[int], dtype[_ArangeScalarT]]
+ xarray/tests/test_plot.py:1168: note: def arange(int | integer[Any] | numpy.bool[builtins.bool], /, stop: int | integer[Any] | numpy.bool[builtins.bool] | None = ..., step: int | integer[Any] | numpy.bool[builtins.bool] | None = ..., *, dtype: type[int] | type[signedinteger[_32Bit | _64Bit]] | dtype[signedinteger[_32Bit | _64Bit]] | _SupportsDType[dtype[signedinteger[_32Bit | _64Bit]]] | None = ..., device: Literal['cpu'] | None = ..., like: _SupportsArrayFunc | None = ...) -> ndarray[tuple[int], dtype[signedinteger[_32Bit | _64Bit]]]
+ xarray/tests/test_plot.py:1168: note: def arange(float | floating[Any], /, stop: float | floating[Any] | integer[Any] | numpy.bool[builtins.bool] | None = ..., step: float | floating[Any] | integer[Any] | numpy.bool[builtins.bool] | None = ..., *, dtype: type[float] | type[float64] | dtype[float64] | _SupportsDType[dtype[float64]] | None = ..., device: Literal['cpu'] | None = ..., like: _SupportsArrayFunc | None = ...) -> ndarray[tuple[int], dtype[float64 | Any]]
+ xarray/tests/test_plot.py:1168: note: def arange(float | floating[Any] | integer[Any] | numpy.bool[builtins.bool], /, stop: float | floating[Any], step: float | floating[Any] | integer[Any] | numpy.bool[builtins.bool] | None = ..., *, dtype: type[float] | type[float64] | dtype[float64] | _SupportsDType[dtype[float64]] | None = ..., device: Literal['cpu'] | None = ..., like: _SupportsArrayFunc | None = ...) -> ndarray[tuple[int], dtype[float64 | Any]]
+ xarray/tests/test_plot.py:1168: note: def arange(timedelta64[timedelta | int | None], /, stop: int | timedelta64[timedelta | int | None] | integer[Any] | numpy.bool[builtins.bool] | None = ..., step: int | timedelta64[timedelta | int | None] | integer[Any] | numpy.bool[builtins.bool] | None = ..., *, dtype: type[timedelta64[timedelta | int | None]] | dtype[timedelta64[timedelta | int | None]] | _SupportsDType[dtype[timedelta64[timedelta | int | None]]] | None = ..., device: Literal['cpu'] | None = ..., like: _SupportsArrayFunc | None = ...) -> ndarray[tuple[int], dtype[timedelta64[Any]]]
+ xarray/tests/test_plot.py:1168: note: def arange(int | timedelta64[timedelta | int | None] | integer[Any] | numpy.bool[builtins.bool], /, stop: timedelta64[timedelta | int | None], step: int | timedelta64[timedelta | int | None] | integer[Any] | numpy.bool[builtins.bool] | None = ..., *, dtype: type[timedelta64[timedelta | int | None]] | dtype[timedelta64[timedelta | int | None]] | _SupportsDType[dtype[timedelta64[timedelta | int | None]]] | None = ..., device: Literal['cpu'] | None = ..., like: _SupportsArrayFunc | None = ...) -> ndarray[tuple[int], dtype[timedelta64[Any]]]
+ xarray/tests/test_plot.py:1168: note: def arange(datetime64[date | int | None], /, stop: datetime64[date | int | None], step: int | timedelta64[timedelta | int | None] | integer[Any] | numpy.bool[builtins.bool] | None = ..., *, dtype: type[datetime64[date | int | None]] | dtype[datetime64[date | int | None]] | _SupportsDType[dtype[datetime64[date | int | None]]] | None = ..., device: Literal['cpu'] | None = ..., like: _SupportsArrayFunc | None = ...) -> ndarray[tuple[int], dtype[datetime64[Any]]]
+ xarray/tests/test_plot.py:1168: note: def arange(integer[Any] | floating[Any] | datetime64[date | int | None] | timedelta64[timedelta | int | None] | float, /, stop: integer[Any] | floating[Any] | datetime64[date | int | None] | timedelta64[timedelta | int | None] | float | None = ..., step: integer[Any] | floating[Any] | datetime64[date | int | None] | timedelta64[timedelta | int | None] | float | None = ..., *, dtype: type[Any] | dtype[Any] | _SupportsDType[dtype[Any]] | tuple[Any, Any] | list[Any] | _DTypeDict | str | None = ..., device: Literal['cpu'] | None = ..., like: _SupportsArrayFunc | None = ...) -> ndarray[tuple[int], dtype[Any]]
+ xarray/tests/test_plot.py: note: At top level:
colour (https://github.com/colour-science/colour)
+ colour/plotting/temperature.py:150: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]]", variable has type "ndarray[tuple[Any, ...], dtype[float64]]") [assignment]
- colour/plotting/quality.py:214: note: def __sub__(self, int | numpy.bool[builtins.bool], /) -> ndarray[tuple[int], dtype[generic[Any]]]
+ colour/plotting/quality.py:214: note: def __sub__(self, int | numpy.bool[builtins.bool], /) -> ndarray[tuple[int], dtype[integer[Any] | floating[Any] | datetime64[date | int | None] | timedelta64[timedelta | int | None]]]
- colour/plotting/quality.py:214: note: def __sub__(self, _SupportsArray[dtype[numpy.bool[builtins.bool]]] | _NestedSequence[_SupportsArray[dtype[numpy.bool[builtins.bool]]]] | builtins.bool | _NestedSequence[builtins.bool], /) -> ndarray[tuple[Any, ...], dtype[generic[Any]]]
+ colour/plotting/quality.py:214: note: def __sub__(self, _SupportsArray[dtype[numpy.bool[builtins.bool]]] | _NestedSequence[_SupportsArray[dtype[numpy.bool[builtins.bool]]]] | builtins.bool | _NestedSequence[builtins.bool], /) -> ndarray[tuple[Any, ...], dtype[integer[Any] | floating[Any] | datetime64[date | int | None] | timedelta64[timedelta | int | None]]]
- colour/utilities/tests/test_array.py:1604: error: Incompatible types in assignment (expression has type "ndarray[tuple[int], dtype[signedinteger[Any]]]", variable has type "int") [assignment]
+ colour/utilities/tests/test_array.py:1604: error: Incompatible types in assignment (expression has type "ndarray[tuple[int], dtype[signedinteger[_32Bit | _64Bit]]]", variable has type "int") [assignment]
|
|
Thanks Joren. |
This fixes `inspect.signature` for - `np.random.BitGenerator` - `np.random.Generator` - `np.random.MT19937` - `np.random.PCG64` - `np.random.PCG64DXSM` - `np.random.Philox` - `np.random.RandomState` - `np.random.SFC64` - `np.random.SeedSequence` - `np.random.bit_generator.SeedlessSeedSequence` This also fixes a typo in `bit_generator.pxd` that accidentally defined an empty unused class `np.random.bit_generator.SeedlessSequence`. Related to #30104, #30114, #30121, #30124, #30126, #30137, #30138, #30140, #30143, #30146, #30147, and #30155
|
The blast radius for this PR, and the whole "add more signatures" effort in general, seems quite large (the |
So far, the reactions have been mostly positive. But the impact that this seems to have is pretty surprising to me.
The mypy_primer output looks pretty scary at first glance, but I'm pretty sure that the new mypy_errors that this will introduce for jax and colour, are justified; i.e. that they're either true positives. And in case of xarray, related to the If you consider the reported line numbers in the mypy_primer diff, you'll see that we're talking about 5 lines of code here; so most what you're seeing is just irrelevant noise :)
That's also what I was thinking, yea. I wasn't sure yet what the best way to go about that was. Open a separate PR for the release note (highlight?) and link that PR, or use the PR number of one these actual PR's? |
#30208 :) |
This reverts a very small part of numpygh-30147, because it broken Pythran (see pythran#2368). While a new Pythran 0.18.1 release is available, that doesn't help for the SciPy 1.15.x releases which allow numpy<2.5 and pythran<0.18, see https://github.com/scipy/scipy/blob/maintenance/1.15.x/pyproject.toml#L30-L41 So introducing this signature change only for 2.5.0 is much better. In addition, doing so doesn't leave us only a very narrow window with compatible NumPy and Pythran releases which would otherwise bite distro packagers as well.
This reverts a very small part of numpygh-30147, because it broken Pythran (see pythran#2368). While a new Pythran 0.18.1 release is available, that doesn't help for the SciPy 1.15.x releases which allow numpy<2.5 and pythran<0.18, see https://github.com/scipy/scipy/blob/maintenance/1.15.x/pyproject.toml#L30-L41 So introducing this signature change only for 2.5.0 is much better. In addition, doing so doesn't leave us only a very narrow window with compatible NumPy and Pythran releases which would otherwise bite distro packagers as well.
This fixes `inspect.signature` for - `np.random.BitGenerator` - `np.random.Generator` - `np.random.MT19937` - `np.random.PCG64` - `np.random.PCG64DXSM` - `np.random.Philox` - `np.random.RandomState` - `np.random.SFC64` - `np.random.SeedSequence` - `np.random.bit_generator.SeedlessSeedSequence` This also fixes a typo in `bit_generator.pxd` that accidentally defined an empty unused class `np.random.bit_generator.SeedlessSequence`. Related to numpy#30104, numpy#30114, numpy#30121, numpy#30124, numpy#30126, numpy#30137, numpy#30138, numpy#30140, numpy#30143, numpy#30146, numpy#30147, and numpy#30155
This fixes `inspect.signature` for - `np.random.BitGenerator` - `np.random.Generator` - `np.random.MT19937` - `np.random.PCG64` - `np.random.PCG64DXSM` - `np.random.Philox` - `np.random.RandomState` - `np.random.SFC64` - `np.random.SeedSequence` - `np.random.bit_generator.SeedlessSeedSequence` This also fixes a typo in `bit_generator.pxd` that accidentally defined an empty unused class `np.random.bit_generator.SeedlessSequence`. Related to numpy#30104, numpy#30114, numpy#30121, numpy#30124, numpy#30126, numpy#30137, numpy#30138, numpy#30140, numpy#30143, numpy#30146, numpy#30147, and numpy#30155
This reverts a very small part of numpygh-30147, because it broken Pythran (see pythran#2368). While a new Pythran 0.18.1 release is available, that doesn't help for the SciPy 1.15.x releases which allow numpy<2.5 and pythran<0.18, see https://github.com/scipy/scipy/blob/maintenance/1.15.x/pyproject.toml#L30-L41 So introducing this signature change only for 2.5.0 is much better. In addition, doing so doesn't leave us only a very narrow window with compatible NumPy and Pythran releases which would otherwise bite distro packagers as well.
This fixes
inspect.signaturefrom raising aValueErrorfor the following functions:numpy.arangenumpy.char.compare_chararraysnumpy.busdaycalendarnumpy.datetime_datanumpy.from_dlpacknumpy.frombuffernumpy.fromfilenumpy.fromiternumpy.frompyfuncnumpy.fromstringnumpy.nested_itersnumpy.promote_typesThe tests were getting a bit messy, so I cleaned them up a bit, and grouped the
signature-related onces into a class-bsaed test.I'll try and improve thearangesignature in a follow-up, as that'll require quite a few changes in the stubs.Related to #30104, #30114, #30121, #30124, #30126, #30137, #30138, #30140, #30143, and #30146.
FYI, still on my
inspect.signatureto-do list are the classes innumpy.random,ndarray.to_device(which is missing docs), and 53 of thenp.genericmethods.