Skip to content

MAINT: use pointers to dtype dtypes instead of direct object access #31100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mattip wants to merge 2 commits into
base: main
Choose a base branch
from
Open

MAINT: use pointers to dtype dtypes instead of direct object access #31100

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
100acfa
MAINT: use pointers to dtype dtypes instead of direct object access
mattip Mar 29, 2026
52702db
DOC: tweak documentation
mattip Mar 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions doc/source/reference/c-api/array.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3533,14 +3533,16 @@ Also see :ref:`dtypemeta` for documentation on ``PyArray_DTypeMeta`` and
.. c:function:: int PyArrayInitDTypeMeta_FromSpec( \
PyArray_DTypeMeta *Dtype, PyArrayDTypeMeta_Spec *spec)

Initialize a new DType. It must currently be a static Python C type that is
declared as :c:type:`PyArray_DTypeMeta` and not :c:type:`PyTypeObject`.
Initialize a new DType. It must currently be a Python C type that is
declared as :c:type:`PyArray_DTypeMetaPtr` and not :c:type:`PyTypeObject`.
Further, it must subclass `np.dtype` and set its type to
:c:type:`PyArrayDTypeMeta_Type` (before calling :c:func:`PyType_Ready()`),
:c:type:`PyArrayDTypeMeta_TypePtr` (before calling :c:func:`PyType_Ready()`),
which has additional fields compared to a normal :c:type:`PyTypeObject`. See
the examples in the ``numpy-user-dtypes`` repository for usage with both
parametric and non-parametric data types.

The type can be either a static type or a heap type.

.. _dtype-flags:

Flags
Expand Down
20 changes: 11 additions & 9 deletions doc/source/reference/c-api/types-and-structures.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -807,9 +807,9 @@ PyArrayMethod_Context and PyArrayMethod_Spec
PyArray_DTypeMeta and PyArrayDTypeMeta_Spec
-------------------------------------------

.. c:var:: PyTypeObject PyArrayDTypeMeta_Type
.. c:var:: PyTypeObject PyArrayDTypeMeta_TypePtr

The python type object corresponding to :c:type:`PyArray_DTypeMeta`.
A pointer to the python type object corresponding to :c:type:`PyArray_DTypeMeta`.

.. c:type:: PyArray_DTypeMeta

Expand Down Expand Up @@ -919,25 +919,27 @@ Exposed DTypes classes (``PyArray_DTypeMeta`` objects)
------------------------------------------------------

For use with promoters, NumPy exposes a number of Dtypes following the
pattern ``PyArray_<Name>DType`` corresponding to those found in `np.dtypes`.
pattern ``PyArray_<Name>DTypePtr`` corresponding to those found in `np.dtypes`.

Additionally, the three DTypes, ``PyArray_PyLongDType``,
``PyArray_PyFloatDType``, ``PyArray_PyComplexDType`` correspond to the
Additionally, the three DTypes, ``PyArray_PyLongDTypePtr``,
``PyArray_PyFloatDTypePtr``, ``PyArray_PyComplexDTypePtr`` correspond to the
Python scalar values. These cannot be used in all places, but do allow
for example the common dtype operation and implementing promotion with them
may be necessary.

Further, the following abstract DTypes are defined which cover both the
builtin NumPy ones and the python ones, and users can in principle subclass
from them (this does not inherit any DType specific functionality):
* ``PyArray_IntAbstractDType``
* ``PyArray_FloatAbstractDType``
* ``PyArray_ComplexAbstractDType``
* ``PyArray_IntAbstractDTypePtr``
* ``PyArray_FloatAbstractDTypePtr``
* ``PyArray_ComplexAbstractDTypePtr``

.. warning::
As of NumPy 2.0, the *only* valid use for these DTypes is registering a
promoter conveniently to e.g. match "any integers" (and subclass checks).
Because of this, they are not exposed to Python.
Because of this, they are not exposed to Python. From NumPy 2.0 to 2.5 they
were exposed without the `Ptr` suffix, at some point the non-pointer version
will be removed.

@ngoldbaum ngoldbaum Mar 30, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is enough for someone who is implementing a user DType to understand what to do about this change.

To make that concrete - how should e.g. numpy_quaddtype adapt to this API change? It's not completely clear to me what to do based on these docs changes. The text you've added here also isn't very greppable for someone who is searching for the old name in the docs.



PyUFunc_Type and PyUFuncObject
Expand Down
71 changes: 70 additions & 1 deletion numpy/_core/include/numpy/_public_dtype_api_table.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@
#if NPY_FEATURE_VERSION >= NPY_2_0_API_VERSION

/*
* The type of the DType metaclass
* The type of the DType metaclass. At some point deprecate these
* non-pointer definitions
*/
#define PyArrayDTypeMeta_Type (*(PyTypeObject *)(PyArray_API + 320)[0])
/*
Expand Down Expand Up @@ -82,5 +83,73 @@

#endif /* NPY_FEATURE_VERSION >= NPY_2_0_API_VERSION */

#if NPY_FEATURE_VERSION >= NPY_2_5_API_VERSION

/*
* Pointer to the type of the DType metaclass
*/
#define PyArrayDTypeMeta_TypePtr (PyTypeObject *)(PyArray_API + 320)[0]
/*
* NumPy's builtin DTypes:
*/
#define PyArray_BoolDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[1]
/* Integers */
#define PyArray_ByteDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[2]
#define PyArray_UByteDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[3]
#define PyArray_ShortDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[4]
#define PyArray_UShortDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[5]
#define PyArray_IntDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[6]
#define PyArray_UIntDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[7]
#define PyArray_LongDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[8]
#define PyArray_ULongDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[9]
#define PyArray_LongLongDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[10]
#define PyArray_ULongLongDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[11]
/* Integer aliases */
#define PyArray_Int8DTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[12]
#define PyArray_UInt8DTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[13]
#define PyArray_Int16DTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[14]
#define PyArray_UInt16DTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[15]
#define PyArray_Int32DTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[16]
#define PyArray_UInt32DTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[17]
#define PyArray_Int64DTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[18]
#define PyArray_UInt64DTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[19]
#define PyArray_IntpDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[20]
#define PyArray_UIntpDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[21]
/* Floats */
#define PyArray_HalfDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[22]
#define PyArray_FloatDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[23]
#define PyArray_DoubleDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[24]
#define PyArray_LongDoubleDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[25]
/* Complex */
#define PyArray_CFloatDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[26]
#define PyArray_CDoubleDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[27]
#define PyArray_CLongDoubleDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[28]
/* String/Bytes */
#define PyArray_BytesDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[29]
#define PyArray_UnicodeDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[30]
/* Datetime/Timedelta */
#define PyArray_DatetimeDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[31]
#define PyArray_TimedeltaDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[32]
/* Object/Void */
#define PyArray_ObjectDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[33]
#define PyArray_VoidDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[34]
/* Python types (used as markers for scalars) */
#define PyArray_PyLongDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[35]
#define PyArray_PyFloatDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[36]
#define PyArray_PyComplexDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[37]
/* Default integer type */
#define PyArray_DefaultIntDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[38]
/* New non-legacy DTypes follow in the order they were added */
#define PyArray_StringDTypePtr (PyArray_DTypeMeta *)(PyArray_API + 320)[39]

/* NOTE: offset 40 is free */

/* Need to start with a larger offset again for the abstract classes: */
#define PyArray_IntAbstractDTypePtr (PyArray_DTypeMeta *)PyArray_API[366]
#define PyArray_FloatAbstractDTypePtr (PyArray_DTypeMeta *)PyArray_API[367]
#define PyArray_ComplexAbstractDTypePtr (PyArray_DTypeMeta *)PyArray_API[368]

#endif /* NPY_FEATURE_VERSION >= NPY_2_5_API_VERSION */

#endif /* NPY_INTERNAL_BUILD */
#endif /* NUMPY_CORE_INCLUDE_NUMPY__PUBLIC_DTYPE_API_TABLE_H_ */
6 changes: 5 additions & 1 deletion numpy/_core/include/numpy/ndarraytypes.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1887,7 +1887,11 @@ typedef struct {
* Part of this (at least the size) is expected to be public API without
* further modifications.
*/
/* TODO: Make this definition public in the API, as soon as its settled */
NPY_NO_EXPORT extern PyTypeObject *PyArrayDTypeMeta_TypePtr;
/*
* This is made public in _public_dtype_api_table.h via the PyArray_API table,
* where it is a dereferenced pointer. Maybe deprecate at some point.

@ngoldbaum ngoldbaum Mar 30, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

given that isn't not deprecated, we probably still need to document it...

*/
NPY_NO_EXPORT extern PyTypeObject PyArrayDTypeMeta_Type;

/*
Expand Down
68 changes: 37 additions & 31 deletions numpy/_core/src/multiarray/abstractdtypes.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,49 +85,49 @@ discover_descriptor_from_pycomplex(
NPY_NO_EXPORT int
initialize_and_map_pytypes_to_dtypes()
{
if (PyType_Ready((PyTypeObject *)&PyArray_IntAbstractDType) < 0) {
if (PyType_Ready((PyTypeObject *)PyArray_IntAbstractDTypePtr) < 0) {
return -1;
}
if (PyType_Ready((PyTypeObject *)&PyArray_FloatAbstractDType) < 0) {
if (PyType_Ready((PyTypeObject *)PyArray_FloatAbstractDTypePtr) < 0) {
return -1;
}
if (PyType_Ready((PyTypeObject *)&PyArray_ComplexAbstractDType) < 0) {
if (PyType_Ready((PyTypeObject *)PyArray_ComplexAbstractDTypePtr) < 0) {
return -1;
}
/*
* Delayed assignments to avoid "error C2099: initializer is not a constant"
* in windows compilers. Can hopefully be done in structs in the future.
*/
((PyTypeObject *)&PyArray_PyLongDType)->tp_base =
(PyTypeObject *)&PyArray_IntAbstractDType;
PyArray_PyLongDType.scalar_type = &PyLong_Type;
if (PyType_Ready((PyTypeObject *)&PyArray_PyLongDType) < 0) {
((PyTypeObject *)PyArray_PyLongDTypePtr)->tp_base =
(PyTypeObject *)PyArray_IntAbstractDTypePtr;
PyArray_PyLongDTypePtr->scalar_type = &PyLong_Type;
if (PyType_Ready((PyTypeObject *)PyArray_PyLongDTypePtr) < 0) {
return -1;
}
((PyTypeObject *)&PyArray_PyFloatDType)->tp_base =
(PyTypeObject *)&PyArray_FloatAbstractDType;
PyArray_PyFloatDType.scalar_type = &PyFloat_Type;
if (PyType_Ready((PyTypeObject *)&PyArray_PyFloatDType) < 0) {
((PyTypeObject *)PyArray_PyFloatDTypePtr)->tp_base =
(PyTypeObject *)PyArray_FloatAbstractDTypePtr;
PyArray_PyFloatDTypePtr->scalar_type = &PyFloat_Type;
if (PyType_Ready((PyTypeObject *)PyArray_PyFloatDTypePtr) < 0) {
return -1;
}
((PyTypeObject *)&PyArray_PyComplexDType)->tp_base =
(PyTypeObject *)&PyArray_ComplexAbstractDType;
PyArray_PyComplexDType.scalar_type = &PyComplex_Type;
if (PyType_Ready((PyTypeObject *)&PyArray_PyComplexDType) < 0) {
((PyTypeObject *)PyArray_PyComplexDTypePtr)->tp_base =
(PyTypeObject *)PyArray_ComplexAbstractDTypePtr;
PyArray_PyComplexDTypePtr->scalar_type = &PyComplex_Type;
if (PyType_Ready((PyTypeObject *)PyArray_PyComplexDTypePtr) < 0) {
return -1;
}

/* Register the new DTypes for discovery */
if (_PyArray_MapPyTypeToDType(
&PyArray_PyLongDType, &PyLong_Type, NPY_FALSE) < 0) {
PyArray_PyLongDTypePtr, &PyLong_Type, NPY_FALSE) < 0) {
return -1;
}
if (_PyArray_MapPyTypeToDType(
&PyArray_PyFloatDType, &PyFloat_Type, NPY_FALSE) < 0) {
PyArray_PyFloatDTypePtr, &PyFloat_Type, NPY_FALSE) < 0) {
return -1;
}
if (_PyArray_MapPyTypeToDType(
&PyArray_PyComplexDType, &PyComplex_Type, NPY_FALSE) < 0) {
PyArray_PyComplexDTypePtr, &PyComplex_Type, NPY_FALSE) < 0) {
return -1;
}

Expand Down Expand Up @@ -170,12 +170,12 @@ int_common_dtype(PyArray_DTypeMeta *NPY_UNUSED(cls), PyArray_DTypeMeta *other)
if (NPY_DT_is_legacy(other) && other->type_num < NPY_NTYPES_LEGACY) {
if (other->type_num == NPY_BOOL) {
/* Use the default integer for bools: */
return NPY_DT_NewRef(&PyArray_IntpDType);
return NPY_DT_NewRef(PyArray_IntpDTypePtr);
}
}
else if (NPY_DT_is_legacy(other)) {
/* This is a back-compat fallback to usually do the right thing... */
PyArray_DTypeMeta *uint8_dt = &PyArray_UInt8DType;
PyArray_DTypeMeta *uint8_dt = PyArray_UInt8DTypePtr;
PyArray_DTypeMeta *res = NPY_DT_CALL_common_dtype(other, uint8_dt);
if (res == NULL) {
PyErr_Clear();
Expand All @@ -187,7 +187,7 @@ int_common_dtype(PyArray_DTypeMeta *NPY_UNUSED(cls), PyArray_DTypeMeta *other)
return res;
}
/* Try again with `int8`, an error may have been set, though */
PyArray_DTypeMeta *int8_dt = &PyArray_Int8DType;
PyArray_DTypeMeta *int8_dt = PyArray_Int8DTypePtr;
res = NPY_DT_CALL_common_dtype(other, int8_dt);
if (res == NULL) {
PyErr_Clear();
Expand All @@ -199,7 +199,7 @@ int_common_dtype(PyArray_DTypeMeta *NPY_UNUSED(cls), PyArray_DTypeMeta *other)
return res;
}
/* And finally, we will try the default integer, just for sports... */
PyArray_DTypeMeta *default_int = &PyArray_IntpDType;
PyArray_DTypeMeta *default_int = PyArray_IntpDTypePtr;
res = NPY_DT_CALL_common_dtype(other, default_int);
if (res == NULL) {
PyErr_Clear();
Expand All @@ -217,16 +217,16 @@ float_common_dtype(PyArray_DTypeMeta *cls, PyArray_DTypeMeta *other)
if (NPY_DT_is_legacy(other) && other->type_num < NPY_NTYPES_LEGACY) {
if (other->type_num == NPY_BOOL || PyTypeNum_ISINTEGER(other->type_num)) {
/* Use the default integer for bools and ints: */
return NPY_DT_NewRef(&PyArray_DoubleDType);
return NPY_DT_NewRef(PyArray_DoubleDTypePtr);
}
}
else if (other == &PyArray_PyLongDType) {
else if (other == PyArray_PyLongDTypePtr) {
Py_INCREF(cls);
return cls;
}
else if (NPY_DT_is_legacy(other)) {
/* This is a back-compat fallback to usually do the right thing... */
PyArray_DTypeMeta *half_dt = &PyArray_HalfDType;
PyArray_DTypeMeta *half_dt = PyArray_HalfDTypePtr;
PyArray_DTypeMeta *res = NPY_DT_CALL_common_dtype(other, half_dt);
if (res == NULL) {
PyErr_Clear();
Expand All @@ -238,7 +238,7 @@ float_common_dtype(PyArray_DTypeMeta *cls, PyArray_DTypeMeta *other)
return res;
}
/* Retry with double (the default float) */
PyArray_DTypeMeta *double_dt = &PyArray_DoubleDType;
PyArray_DTypeMeta *double_dt = PyArray_DoubleDTypePtr;
res = NPY_DT_CALL_common_dtype(other, double_dt);
return res;
}
Expand All @@ -254,12 +254,12 @@ complex_common_dtype(PyArray_DTypeMeta *cls, PyArray_DTypeMeta *other)
if (other->type_num == NPY_BOOL ||
PyTypeNum_ISINTEGER(other->type_num)) {
/* Use the default integer for bools and ints: */
return NPY_DT_NewRef(&PyArray_CDoubleDType);
return NPY_DT_NewRef(PyArray_CDoubleDTypePtr);
}
}
else if (NPY_DT_is_legacy(other)) {
/* This is a back-compat fallback to usually do the right thing... */
PyArray_DTypeMeta *cfloat_dt = &PyArray_CFloatDType;
PyArray_DTypeMeta *cfloat_dt = PyArray_CFloatDTypePtr;
PyArray_DTypeMeta *res = NPY_DT_CALL_common_dtype(other, cfloat_dt);
if (res == NULL) {
PyErr_Clear();
Expand All @@ -271,13 +271,13 @@ complex_common_dtype(PyArray_DTypeMeta *cls, PyArray_DTypeMeta *other)
return res;
}
/* Retry with cdouble (the default complex) */
PyArray_DTypeMeta *cdouble_dt = &PyArray_CDoubleDType;
PyArray_DTypeMeta *cdouble_dt = PyArray_CDoubleDTypePtr;
res = NPY_DT_CALL_common_dtype(other, cdouble_dt);
return res;

}
else if (other == &PyArray_PyLongDType ||
other == &PyArray_PyFloatDType) {
else if (other == PyArray_PyLongDTypePtr ||
other == PyArray_PyFloatDTypePtr) {
Py_INCREF(cls);
return cls;
}
Expand All @@ -301,6 +301,7 @@ NPY_NO_EXPORT PyArray_DTypeMeta PyArray_IntAbstractDType = {{{
.type_num = -1,
.flags = NPY_DT_ABSTRACT,
};
NPY_NO_EXPORT PyArray_DTypeMeta *PyArray_IntAbstractDTypePtr = &PyArray_IntAbstractDType;

NPY_DType_Slots pylongdtype_slots = {
.discover_descr_from_pyobject = discover_descriptor_from_pylong,
Expand All @@ -319,6 +320,7 @@ NPY_NO_EXPORT PyArray_DTypeMeta PyArray_PyLongDType = {{{
.dt_slots = &pylongdtype_slots,
.scalar_type = NULL, /* set in initialize_and_map_pytypes_to_dtypes */
};
NPY_NO_EXPORT PyArray_DTypeMeta *PyArray_PyLongDTypePtr = &PyArray_PyLongDType;

NPY_NO_EXPORT PyArray_DTypeMeta PyArray_FloatAbstractDType = {{{
PyVarObject_HEAD_INIT(&PyArrayDTypeMeta_Type, 0)
Expand All @@ -330,6 +332,7 @@ NPY_NO_EXPORT PyArray_DTypeMeta PyArray_FloatAbstractDType = {{{
.type_num = -1,
.flags = NPY_DT_ABSTRACT,
};
NPY_NO_EXPORT PyArray_DTypeMeta *PyArray_FloatAbstractDTypePtr = &PyArray_FloatAbstractDType;

NPY_DType_Slots pyfloatdtype_slots = {
.discover_descr_from_pyobject = discover_descriptor_from_pyfloat,
Expand All @@ -348,6 +351,7 @@ NPY_NO_EXPORT PyArray_DTypeMeta PyArray_PyFloatDType = {{{
.dt_slots = &pyfloatdtype_slots,
.scalar_type = NULL, /* set in initialize_and_map_pytypes_to_dtypes */
};
NPY_NO_EXPORT PyArray_DTypeMeta *PyArray_PyFloatDTypePtr = &PyArray_PyFloatDType;

NPY_NO_EXPORT PyArray_DTypeMeta PyArray_ComplexAbstractDType = {{{
PyVarObject_HEAD_INIT(&PyArrayDTypeMeta_Type, 0)
Expand All @@ -359,6 +363,7 @@ NPY_NO_EXPORT PyArray_DTypeMeta PyArray_ComplexAbstractDType = {{{
.type_num = -1,
.flags = NPY_DT_ABSTRACT,
};
NPY_NO_EXPORT PyArray_DTypeMeta *PyArray_ComplexAbstractDTypePtr = &PyArray_ComplexAbstractDType;

NPY_DType_Slots pycomplexdtype_slots = {
.discover_descr_from_pyobject = discover_descriptor_from_pycomplex,
Expand All @@ -377,6 +382,7 @@ NPY_NO_EXPORT PyArray_DTypeMeta PyArray_PyComplexDType = {{{
.dt_slots = &pycomplexdtype_slots,
.scalar_type = NULL, /* set in initialize_and_map_pytypes_to_dtypes */
};
NPY_NO_EXPORT PyArray_DTypeMeta *PyArray_PyComplexDTypePtr = &PyArray_PyComplexDType;


/*
Expand Down
Loading
Loading