ENH: add NEP-50 style semantics for string scalars and StringDType by ngoldbaum · Pull Request #32040 · numpy/numpy

ngoldbaum · 2026-07-19T12:48:22Z

PR summary

While working on ByteStringDType, I noticed that trailing nulls are currently stripped when StringDType operations use python scalars. For example:

>>> np.array(["hello", "world"], dtype="T") + "hello\0\0"
array(['hellohello', 'worldhello'], dtype=StringDType())

This is happening because "hello\0\0" is coerced to a fixed-width unicode array, which strips the trailing NULLs.

@seberg let me know that the infrastructure in NumPy to do this is already in NumPy to support NEP-50. Somewhat differently from the other NEP-50 cases, I don't think it's necessary to add a new abstract dtype to represent python strings because we don't want string scalars and string arrays to have different semantics.

While working on this I realized that partition and rpartition need a promoter to be able to work with this new functionality, so this includes that as well.

AI Disclosure

I iterated on the implementation with an AI model.

seberg

Thanks, this looks nice and straight-forward enough. As you said, dunno if that one test has much use, but I don't mind keeping it.

May make sense to do that (yes annoying) ternary to avoid double-conversion, I think, but otherwise this seems good and pragmatic.

ngoldbaum

I think I responded to all your comments. I also fixed some additional issues I missed on the first pass for unary ufuncs, ufunc.at, and ufunc.outer.

ngoldbaum · 2026-07-21T21:32:08Z

+    result = np.strings.replace(a, old, new)
+    assert_array_equal(result, np.array(["a+b"], dtype=dt))
+    assert a.array_calls == old.array_calls == new.array_calls == 1
+    assert a.ufunc_calls == old.ufunc_calls == new.ufunc_calls == 0


let me know if you think this test is over-engineered

I quite like it, though perhaps would just raise NotImplementedError in __array_ufunc__, in which case you do not need ufunc_calls and its checks.

mhvk

@ngoldbaum - this is rather nice! But while reviewing the special-casing for .outer, I realized there was a bigger problem there (see #32076).

I also somewhat wonder whether it would not be better to treat the python str scalars more similar to the numerical ones (see comment about the tests).

mhvk · 2026-07-23T15:46:07Z

        out_op_DTypes[i] = NPY_DTYPE(PyArray_DESCR(out_op[i]));
        Py_INCREF(out_op_DTypes[i]);

+        /* Does not affect promotion, only conversion after resolution. */


I think this can be moved inside the scalar-case if block on original l.688, since it would be a scalar and does not seem relevant for a ufunc with nin=1. Has the advantage of not causing any delay for non-string inputs.

Unary ufuncs have an early exit in convert_ufunc_arguments so implementing this also requires getting rid of that early exit. Which is more correct anyway, it just wasn't necessary for NEP-50 semantics.

Hmm, but I htink we can still move it in the else branch of PyArray_Check -- since this obviously wasn't an array. (I'm not sure what "line l.688" I referred to in my initial comment...).

Done, thanks for clarifying. This is simpler.

mhvk · 2026-07-23T16:18:21Z

+    result = np.strings.replace(a, old, new)
+    assert_array_equal(result, np.array(["a+b"], dtype=dt))
+    assert a.array_calls == old.array_calls == new.array_calls == 1
+    assert a.ufunc_calls == old.ufunc_calls == new.ufunc_calls == 0


I quite like it, though perhaps would just raise NotImplementedError in __array_ufunc__, in which case you do not need ufunc_calls and its checks.

mhvk · 2026-07-23T16:35:34Z


-    if np.result_type(arr, old, new).char == "T":
-        return _replace(arr, old, new, count)
+    if np.result_type(arr, old_arr, new_arr).char == "T":


Hmm, it would be nice if one could avoid asanyarray in case of str input and then let the rest still work. But right now,

np.result_type(np.array(123, "f4"), 1.0) # dtype('float32') np.result_type(np.array("123", "T"), "1") # TypeError: data type '' not understood

Is this because of not introducing an abstract dtype class for python strings?

Or could this be fixed more easily? (Anyway, not the biggest deal...)

I don't think an abstract string dtype is needed. Python strings and string arrays have the same promotion semantics. We're only adding the scalar tracking so that we don't lose trailing NULL bytes in python strings because there's no way to override the detour via the U dtype default.

Hmm, I was just hoping to avoid this whole "create an array and then only use it if not str" dance, which would be possible if result_type just worked with str just like it does with float.

And looking at the code in multiarraymodule.c, this is actually not hard in principle, but in practice strings get interpreted as dtypes, and one cannot break that easily.

An alternative would be to use _array_converter, which is easier to adjust (I just tried) and would help to remove boilerplate.

But arguably also beyond what is needed here, especially as it still needlessly converts the string into an array, so maybe better done as follow-up....

OK, let's do that in a followup.

mhvk

Only a small suggestion left... (and a larger comment with no action required).

mhvk · 2026-08-08T00:14:07Z

        out_op_DTypes[i] = NPY_DTYPE(PyArray_DESCR(out_op[i]));
        Py_INCREF(out_op_DTypes[i]);

+        /* Does not affect promotion, only conversion after resolution. */


Hmm, but I htink we can still move it in the else branch of PyArray_Check -- since this obviously wasn't an array. (I'm not sure what "line l.688" I referred to in my initial comment...).

mhvk · 2026-08-08T02:14:08Z


-    if np.result_type(arr, old, new).char == "T":
-        return _replace(arr, old, new, count)
+    if np.result_type(arr, old_arr, new_arr).char == "T":


Hmm, I was just hoping to avoid this whole "create an array and then only use it if not str" dance, which would be possible if result_type just worked with str just like it does with float.

And looking at the code in multiarraymodule.c, this is actually not hard in principle, but in practice strings get interpreted as dtypes, and one cannot break that easily.

An alternative would be to use _array_converter, which is easier to adjust (I just tried) and would help to remove boilerplate.

But arguably also beyond what is needed here, especially as it still needlessly converts the string into an array, so maybe better done as follow-up....

ngoldbaum · 2026-08-13T14:47:27Z

Marten approved the bulk of this and Sebastian was also OK with it, so I'm bringing this in to unblock my big ByteStringDType PR and ByteStringDType NEP PR I'd like to get done this week.

Thanks for looking this over both of you!

ngoldbaum added the 01 - Enhancement label Jul 19, 2026

ngoldbaum commented Jul 19, 2026

View reviewed changes

Comment thread numpy/_core/src/umath/ufunc_object.c Outdated

seberg reviewed Jul 19, 2026

View reviewed changes

Comment thread numpy/_core/src/umath/ufunc_object.c Outdated

Comment thread numpy/_core/tests/test_stringdtype.py Outdated

Comment thread numpy/_core/strings.py

ngoldbaum commented Jul 21, 2026

View reviewed changes

mhvk mentioned this pull request Jul 23, 2026

BUG: NEP 50 promotion for python scalars is not followed for <ufunc>.outer and <ufunc>.at #32076

Closed

mhvk reviewed Jul 23, 2026

View reviewed changes

ngoldbaum mentioned this pull request Jul 29, 2026

ENH: NEP 50 scalar promotion semantics in ufunc.at #32094

Merged

ngoldbaum added 2 commits August 5, 2026 16:03

ENH: preserve Python string scalars in ufuncs

f23a35f

DOC: document Python string scalar ufunc behavior

2a45797

ngoldbaum force-pushed the fix-trailing-null-scalars branch from 2ce5efc to 2a45797 Compare August 5, 2026 22:09

mhvk reviewed Aug 8, 2026

View reviewed changes

MAINT: respond to review comments

a2838b7

ngoldbaum merged commit 620518d into numpy:main Aug 13, 2026
91 checks passed

mhvk mentioned this pull request Aug 14, 2026

MAINT: treat str special in _array_converter and use it in strings.py #32297

Draft

Uh oh!

Uh oh!

Conversation

ngoldbaum commented Jul 19, 2026

PR summary

AI Disclosure

Uh oh!

Uh oh!

seberg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ngoldbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhvk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhvk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngoldbaum commented Aug 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants