ENH: Build _reduction_loop_tests against the Limited API - #32256

Hmm, here and below I don't think this is correct. Too bad we have to take special care here - these functions were never added to the limited API.

See the docs for Py_SETREF. This is exactly "the obvious ... deadly" code.

We could also add something like this to an internal compat header or maybe to pythoncapi-compat itself (suggested by an AI model, please double-check and validate it by comparing with CPython's implementation):

#ifndef Py_SETREF
#if defined(__GNUC__) || defined(__clang__)
#define NPY_SETREF_IMPL_(dst, src, decref) \
    do { \
        __typeof__(dst) *_tmp_dst_ptr = &(dst); \
        __typeof__(dst) _tmp_old_dst = (*_tmp_dst_ptr); \
        *_tmp_dst_ptr = (src); \
        decref(_tmp_old_dst); \
    } while (0)
#else
#define NPY_SETREF_IMPL_(dst, src, decref) \
    do { \
        PyObject **_tmp_dst_ptr = (PyObject **)&(dst); \
        PyObject *_tmp_old_dst = (*_tmp_dst_ptr); \
        PyObject *_tmp_src = (PyObject *)(src); \
        memcpy(_tmp_dst_ptr, &_tmp_src, sizeof(PyObject *)); \
        decref(_tmp_old_dst); \
    } while (0)
#endif
#define Py_SETREF(dst, src)  NPY_SETREF_IMPL_(dst, src, Py_DECREF)
#define Py_XSETREF(dst, src) NPY_SETREF_IMPL_(dst, src, Py_XDECREF)
#endif

If you want a more minimal fix, this is also safe:

PyObject *tmp = dst;
dst = src;
Py_DECREF(tmp);

But doing it without storing a reference in the temporary before re-assigning the reference may cause a use-after-free.

I opened capi-workgroup/decisions#110 about this. But of course that will only impact Python 3.16 and newer, if at all.

Thanks for catching the blunder you are right that is the unsafe ordering. I have gone with the minimal fix, using a temporary at each site:

PyObject *old1 = *o1, *old2 = *o2;
*o1 = lo;
*o2 = hi;
Py_XDECREF(old1);
Py_XDECREF(old2);

Both slots are published before either decref, so neither is stale while a __del__ runs.

I am also happy to do the shared-macro version instead if you had prefer but it felt like it belongs in a seperate PR rather than this one. And thanks for opening the capi-workgroup issue!