Ohh, fun. OTOH, the error here would be that we are not reading everything (i.e. the last bit of the result not being -1). But I think there is a fun little thing happening here:

• This is a signed int overflow 2**31 -> -2**31
• Cast to size_t for reading, it'll actually just go back to 2**31 as that is unsigned.

So, no problem until we reach 2**32 at which point the error would be reading nothing at all.

Possible we could employ a funny trick here: Just try to read 1 element of a 2**32+1 sized dtype from a short file (not empty with the +1).
With fromfile(...., count=1) that should try to read one element, fail and return an empty array. But with the bug I think it'll actually return an array of length 1.
(That said, clear that is more of a regression test designed for the specific error in this code...)

Playing with this a bit, something really strange/bad happens somewhere--I can retrieve items from the reconstituted array quickly, but attempting to perform assertions on even those individual elements is incredibly slow.

For example,

item = actual["x"][0][1].item() # fast
assert item > 0 # prohibitively slow

It is only slow if it is destined to fail (!), with or without the npy_intp patch applied. How strange, I wonder if something is getting corrupted, or if I'm missing something obvious getting expanded too large (item prints out innocently enough as either 0.0 or 1.0 depending on application of the patch). I spent a few hours poking at that because it was so strange looking. It will hang on the mismatch of the tobytes patterns as well, which might suggest a corruption somewhere I suppose (since the same bytes pattern comparison is fine in raw interpreter).

I didn't find any advantage to using count=1. Maybe I misunderstood, but our problem is that we have a single massive "item" so it doesn't cut that down/discretize? The test works just fine with it, but runtime/result is the same. Perhaps you meant reading back in with a separate smaller dtype itemsize and looking at a slice of the data that way, not sure.

For now, I've pushed in a revision to test_recarray_fromfile_massive() such that it uses a larger itemsize (that part of your suggestion was indeed needed), and passes within 1-10 seconds locally with the previous patch applied to array_fromfile_binary. Without that small patch, the test will basically hang for reasons I don't fully understand. I suppose that's at least a bit closer to what I want re: a test that doesn't really reasonably pass without the patch, though still a bit annoying.

Maybe I misunderstood, but our problem is that we have a single massive "item" so it doesn't cut that down/discretize?

I think I just skipped over that it seemed to me that we pass count=1 it'll assume it can read 1 item, while otherwise it checks how big the file actually is.
I.e. my thought was we can just put a few bytes into the file, but NumPy will incorrectly attempt to read an actual element.

assert item > 0 # prohibitively slow

In general, or in pytest? Because if it is in pytest, I could imagine it wants to do something like print actual or so?

Ah yeah, thanks, looks like suppressing the traceback allows the failure scenario to happen in a second or two (when type change is reverted): spin test -t numpy/_core/tests/test_records.py::TestFromrecords::test_recarray_fromfile_massive -- --tb=no

That's annoying that it doesn't auto truncate/summarize.

This also "works" though is not very nice:

--- a/numpy/_core/tests/test_records.py
+++ b/numpy/_core/tests/test_records.py
@@ -122,7 +122,9 @@ def test_recarray_fromfile_massive(self, tmpdir):
             actual = np.fromfile("f.data", dtype=kind_dtype)
             assert actual.itemsize == 2 ** 29 * 8
             item = actual["x"][0][1]
-            assert_allclose(item, 1)
+            if item != 1:
+                # avoid hang from pytest traceback dumping massive array
+                pytest.fail("fromfile elsize error", pytrace=False)

I'll leave this one alone for now--seems to be getting closer to something sensible perhaps...