gh-31364 will users to use heaptypes as DTypes, but it would still be nice to create a newer API, see below:
The user DTypes were created at a time when the stable ABI was not ready for them. But with Python 3.12 the basis should exist now: there should be a path forward around this (i.e. even if it might require hacks similar to what we have, I suspect there is more prior art around that, e.g. maybe in nanobind, at the time NumPy was relatively early to the game, it's not anymore).
This goes hand-in-hand with any effort to make NumPy use heap-types mostly/everywhere, but I think we'll want a new API call that merges:
PyType_FromMetaclass (or some variation of that)PyArrayInitDTypeMeta_FromSpec
to do both in one go. Now there are a couple of things to figure out here, for one, DTypeMeta currently has a tp_new to prevent creation from Python (this can just be removed in principle, although direct __new__ calls are then unsafe from Python) -- this collides with PyType_FromMetaclass().
Another thing is, that PyArrayInitDTypeMeta_FromSpec was based on the stable API at the time without thinking much about improving the pattern. But Python has had some improvements, I believe (i.e. now using only slots and a new PEP https://discuss.python.org/t/pep-820-pyslot-unified-slot-system-for-the-c-api/105552 to revamp the slot system. If NumPy has API that wraps type creation from Python, then we need to check that out.).
gh-31364 will users to use heaptypes as DTypes, but it would still be nice to create a newer API, see below:
The user DTypes were created at a time when the stable ABI was not ready for them. But with Python 3.12 the basis should exist now: there should be a path forward around this (i.e. even if it might require hacks similar to what we have, I suspect there is more prior art around that, e.g. maybe in
nanobind, at the time NumPy was relatively early to the game, it's not anymore).This goes hand-in-hand with any effort to make NumPy use heap-types mostly/everywhere, but I think we'll want a new API call that merges:
PyType_FromMetaclass(or some variation of that)PyArrayInitDTypeMeta_FromSpecto do both in one go. Now there are a couple of things to figure out here, for one,
DTypeMetacurrently has atp_newto prevent creation from Python (this can just be removed in principle, although direct__new__calls are then unsafe from Python) -- this collides withPyType_FromMetaclass().Another thing is, that
PyArrayInitDTypeMeta_FromSpecwas based on the stable API at the time without thinking much about improving the pattern. But Python has had some improvements, I believe (i.e. now using only slots and a new PEP https://discuss.python.org/t/pep-820-pyslot-unified-slot-system-for-the-c-api/105552 to revamp the slot system. If NumPy has API that wraps type creation from Python, then we need to check that out.).