PR summary

This was originally part of #31682 but @seberg asked me to split it off. I also took his comments on that version about how I handled complex values into account.

Currently, we only support correctly parsing NaN values for float64 for casting and setitem. This leads to this (IMO) strange asymmetry:

>>> import numpy as np
>>> sdt = np.dtypes.StringDType(na_object=np.nan)
>>> arr = np.array([1, np.nan, 4.5], dtype=np.float32)
>>> arr.astype(sdt)
array(['1.0', 'nan', '4.5'], dtype=StringDType(na_object=nan))
>>> isinstance(arr.astype(sdt)[1], str)
True
>>> arr = np.array([1, np.nan, 4.5], dtype=np.float64)
>>> arr.astype(sdt)
array(['1.0', nan, '4.5'], dtype=StringDType(na_object=nan))
>>> arr.astype(sdt)[1] is np.nan
True

IMO this is just a straight-up bug in the implementation, which relies on coercing the value to a Python scalar, and fails to catch e.g. np.float32('nan') because float32 scalars aren't subtypes of PyFloat. That said, it is an observable behavior change.

We don't parse any complex-valued NaN as a NaN missing data value, leading to behavior like this:

>>> arr.astype(np.dtypes.StringDType(na_object=np.nan))
array(['(nan+0j)', 'nanj'], dtype=StringDType(na_object=nan))

This behavior is a bit more arguable to change, but under the principle that missing values should round-trip correctly via casting and that setitem should behave identically to casting, I also added behavior for complex values with nan real values to coerce to a null string if the na_object is NaN-like. This followes this existing behavior in NumPy:

>>> arr = np.array([complex(np.nan, 3.5), complex(0, np.nan), complex(3.5, np.nan), complex(np.nan, np.nan)])
>>> arr
array([nan+3.5j, 0. +nanj, 3.5+nanj, nan+nanj])
>>> arr.astype('float64')
<python-input-3>:1: ComplexWarning: Casting complex values to real discards the imaginary part
array([nan, 0. , 3.5, nan])

Note that the ComplexWarning happens unconditionally here for all complex values, even if the real part is zero.

I refactored StringDType internals to reflect this behavior. This refactoring included correcting some restructuring to reflect that the float to string casts might now produce null strings where has_nan_na was ignored before.

Added tests for this new behavior for floats and complex dtypes. Also adds a test that round-trips from float64 to float64 via complex, float64, and stringdtype, including all the permutations of that should round-trip.

AI Disclosure

An AI model initially noticed this issue. I also used an AI to brainstorm and review the code changes and tests.