test_pickling_large (and the rest of the full testsuite) now pass locally on this branch, thanks to a small type fix in arraydescr_reduce.

A few notes here:

• on an ARM M3 Max mac laptop, that test takes a long time to pass, more than 6 minutes: 1 passed in 402.71s (0:06:42); I think that means that the slow() marker doesn't cut it, and I guess NumPy doesn't have xslow? What do you want me to do here then? Do we really need to iterate over 2 x > C int cases x 5 pickle protocol versions? If so, how should we selectively run this only once in a while? Even a single protocol version is over a minute to run.
• should we also stack in a requires_memory decoration here? I believe these arrays are pretty massive like in the other test I added that materializes such arrays.

Maybe just skip the "dtype is functional" test, or make the array empty np.zeros(0, dtype=dtype).
I am not 100% sure if that covers everything, but it seems fair to me to do that.

There isn't really much to check anyway, the only thing I can think of is whether dtype.flags == pickled.flags, because the dtype == pickled should check almost everything else (metadata is already checked).

I left the initial materialization in, y = np.zeros(3, dtype=pickled), but gave check_pickling a default-on argument that optionally runs the actually assertion on the materialized arrays, which I've turned off for the test_pickling_large calls.

I believe this is probably robust, but may have some risk, if some operating system did actually fully allocate the RAM on a zeros call, but I believe that's rare? CI may tells us--I just flushed CI for this time.

With that change, the test would no longer fail without the source patches here though, so I added an extra check that assert_equal(pickled.itemsize, dtype.itemsize), which does fail without the work in this PR. I was slightly surprised that assert_equal(pickled.descr, dtype.descr) wasn't sufficient to enforce proper reconstitution of itemsize, since that reads as "they have the same descriptors," but apparently it doesn't...

This also allowed me to remove the slow marker.

Alright, CI is failing on the first try, IS_64BIT may not be a sufficient guard for some Windows cases?

test_gh_31308 needs a 64-bit guard. Some of the CI failures are not related--bit messy.

There is one MKL job generally failing right now, just ignore that one.

The latest commit suppresses CI again for now. There is one remaining failure that I'm going to need to debug directly on a Windows box, below the fold. The new itemsize check that I added to check_pickling seems to show dtype = np.dtype(f"({2**31},)i") getting set to dtype(('<i4', (-2147483648,))), independent of serialization. The intp size on that Windows CI runner should be just fine since the test is guarded by IS_64BIT, which comes from np.dtype(np.intp).itemsize. git grep -E -i "int elsize" still has several hits on this branch, but way easier to debug directly on Win machine, so I'll have to circle back.

>           assert_equal(pickled.itemsize, dtype.itemsize)
E           AssertionError: 
E           Items are not equal:
E            ACTUAL: 0
E            DESIRED: 8589934592

arr_assert = False
buf        = b'cnumpy\ndtype\np0\n(VV0\np1\nI00\nI01\ntp2\nRp3\n(I3\nV|\np4\n(g0\n(Vi4\np5\nI00\nI01\ntp6\nRp7\n(I3\nV<\np8\nNNNI-1\nI-1\nI0\ntp9\nb(I-2147483648\ntp10\ntp11\nNNI0\nI4\nI0\ntp12\nb.'
dtype      = dtype(('<i4', (-2147483648,)))
pickled    = dtype(('<i4', (-2147483648,)))

You can probably get away without trying a windows box. If this diverges on windows, you can be sure there is some use of long along the way. Either a C long or PyLong_FromLong or so call somewhere along the branch.
And just grepping sees things like:

PyObject *size_obj = PyLong_FromLong((long) totalsize);

There are a lot more in that file... Got to just replace them all with FromSsizet (or whatever it is). And most likely that will be enough.

(There might be corresponding PyLong_AsLong somewhere, but from a grep those seem more likely to be fixed than the reverse.)

Ok, the CI is passing apart from two clearly unrelated failures.

I did end up having to use a Windows box because fork CI debugging was too painful. I also added an additional assertion to one of my new tests (test_gh_31308()), since it should have picked up the descr problem that was causing the pickle test to fail (it now does).

Ohh, fun. OTOH, the error here would be that we are not reading everything (i.e. the last bit of the result not being -1). But I think there is a fun little thing happening here:

• This is a signed int overflow 2**31 -> -2**31
• Cast to size_t for reading, it'll actually just go back to 2**31 as that is unsigned.

So, no problem until we reach 2**32 at which point the error would be reading nothing at all.

Possible we could employ a funny trick here: Just try to read 1 element of a 2**32+1 sized dtype from a short file (not empty with the +1).
With fromfile(...., count=1) that should try to read one element, fail and return an empty array. But with the bug I think it'll actually return an array of length 1.
(That said, clear that is more of a regression test designed for the specific error in this code...)