Skip to content

BUG: fix np.linalg.norm overflow for representable results (gh-8775)#31927

Open
vismaytiwari wants to merge 8 commits into
numpy:mainfrom
vismaytiwari:fix-linalg-norm-overflow
Open

BUG: fix np.linalg.norm overflow for representable results (gh-8775)#31927
vismaytiwari wants to merge 8 commits into
numpy:mainfrom
vismaytiwari:fix-linalg-norm-overflow

Conversation

@vismaytiwari

@vismaytiwari vismaytiwari commented Jul 9, 2026

Copy link
Copy Markdown

Fixes #8775. Fixes #19097.

np.linalg.norm's 2-norm fast path accumulates the sum of squares in the input dtype, so it overflows to inf once that sum exceeds the dtype maximum — even when the norm itself is representable:

>>> np.linalg.norm(np.float16([600, 800]))
inf          # should be 1000.0

As @stevengj pointed out on the issue, this isn't float16-specific — it happens for any float dtype past sqrt(max), e.g. norm([1e200, 1e200]) in float64.

This detects a non-finite result from all-finite input and recomputes the sum of squares after scaling by max(|x|), which cannot overflow (the classic nrm2 scaling):

>>> np.linalg.norm(np.float16([600, 800]))
1000.0
>>> np.linalg.norm(np.array([1e200, 1e200]))
1.4142135623730951e+200

The common (non-overflowing) path is unchanged — a single vdot(x, x).real, with the scaled recompute only when that result is non-finite or zero. Genuine inf/nan inputs and zeros are preserved.

The same recompute is applied to the reductions that square (or raise to ord>1) their input: the axis=None fast path (1-D 2-norm and 2-D Frobenius), the per-axis vector 2-norm and arbitrary ord>1, and the Frobenius norm reduced over an axis pair. The axis cases rescale per slice via a small helper, so only the rows/columns that actually over/underflow are recomputed. The matrix 2-norm/nuclear norm (SVD) and the 1-/inf-norms don't square, so they're unaffected.

The transient overflow warning from the naive step is suppressed only in the test (via np.errstate), so the hot path stays free of any per-call context.

AI Disclosure:

  • The tool: Codex
  • What it did for me — reproduced it, and wrote the code fix .
  • My part — I reviewed it, understand it, and take responsibility for it.

)

The 2-norm fast path accumulates the sum of squares in the input dtype,
so it overflows to inf once that sum exceeds the dtype maximum - even
when the norm itself is representable. This happens for float16
(norm([600, 800]) returned inf instead of 1000) and, as noted in the
issue, for any float dtype past sqrt(max) (e.g. norm([1e200, 1e200]) in
float64).

Detect a non-finite result from all-finite input and recompute the sum
of squares after scaling by max(|x|), which cannot overflow. The common
non-overflowing path is unchanged apart from an errstate guard that
suppresses the transient overflow warning. Genuine inf/nan inputs and
zeros are preserved.
@ikrommyd

ikrommyd commented Jul 9, 2026

Copy link
Copy Markdown
Member

I have a feeling that doing the calculation, checking every element of the output with isfinite, and then rerunning the calculation if an overflow occurred adds overhead in the nominal case. Have you benchmarked this?

To maintainers: not 100% sure if I'd call this a bug. It's most likely a "bug?" with a question mark like the issue says but I added the bug label anyways. Somebody change it if you feel otherwise

@vismaytiwari

Copy link
Copy Markdown
Author

Hey @ikrommyd
so I benchmarked it, Two things first:

  1. the check is on the scalar result (isfinite(ret) — the norm is a single number, so it's O(1), not element-wise), and the isfinite(x).all() + rescale are short-circuited to run only when the result is non-finite. So the nominal path doesn't do any O(n) scan or recompute.
  2. But you're right there was real per-call overhead on small arrays, and it came from two places — the errstate guard (~390 ns) and the np.isfinite ufunc dispatch. Dropping errstate only got about halfway; switching the scalar check to math.isfinite(float(ret)) closes the rest. With both, size-3 goes 312 → ~319 ns (within noise) and it's negligible on larger arrays. Overflow-case warning behavior is unchanged vs main, and the float16 / large-float64 / complex / inf-nan cases stay covered by the test.
    Pushing that now. Thanks for the nudge.
    Happy to address any other feedback.

Following review: the errstate context manager plus the np.isfinite
ufunc dispatch added measurable per-call overhead in the common
(non-overflowing) path (~2-3x on small arrays). Drop the errstate
wrapper and use math.isfinite(float(ret)) for the scalar overflow
check. The nominal path now adds no measurable overhead (size-3 within
noise) and the recompute path is unchanged; overflow-case warning
behavior matches the prior code.
Comment thread numpy/linalg/_linalg.py Outdated
Comment on lines +2751 to +2759
if not math.isfinite(float(ret)) and isfinite(x).all():
max_abs = abs(x).max()
scaled = x / max_abs
if isComplexType(x.dtype.type):
sqnorm = (scaled.real.dot(scaled.real)
+ scaled.imag.dot(scaled.imag))
else:
sqnorm = scaled.dot(scaled)
ret = max_abs * sqrt(sqnorm)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You also need to check for underflow, e.g. [1e-200, 1e-200] should not have a norm of 0.0. Something like

Suggested change
if not math.isfinite(float(ret)) and isfinite(x).all():
max_abs = abs(x).max()
scaled = x / max_abs
if isComplexType(x.dtype.type):
sqnorm = (scaled.real.dot(scaled.real)
+ scaled.imag.dot(scaled.imag))
else:
sqnorm = scaled.dot(scaled)
ret = max_abs * sqrt(sqnorm)
if not math.isfinite(float(ret)) and ret != 0: # possible over/underflow
max_abs = abs(x).max()
if not math.isfinite(float(max_abs)) or max_abs == 0:
return ret
scaled = x / max_abs
if isComplexType(x.dtype.type):
sqnorm = (scaled.real.dot(scaled.real)
+ scaled.imag.dot(scaled.imag))
else:
sqnorm = scaled.dot(scaled)
ret = max_abs * sqrt(sqnorm)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks — you're right, it was only handling overflow. Pushed a fix so it rescales on underflow too.
One small thing on the suggested condition: not math.isfinite(ret) and ret != 0 doesn't actually catch the underflow case, since [1e-200, 1e-200] gives ret == 0 (which is finite) — so I went with not math.isfinite(ret) or ret == 0. Kept your max_abs guard so inf/nan still propagate and an all-zero vector stays 0. norm([1e-200, 1e-200]) now returns √2·1e-200 instead of 0, and I added float64/float32/complex underflow cases to the test.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I meant not math.isfinite(float(ret)) or ret == 0

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that's what's in the current version — not math.isfinite(float(ret)) or ret == 0. 👍

Comment thread numpy/linalg/_linalg.py Outdated
if isComplexType(x.dtype.type):
x_real = x.real
x_imag = x.imag
sqnorm = x_real.dot(x_real) + x_imag.dot(x_imag)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Seems like it should be faster to do numpy.vdot(x, x).real, so that it can do a single pass over the complex array?)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's cleaner — switched the complex sum of squares to vdot(x, x).real so it's a single pass, both in the main path and the rescaled one. Thanks!

@stevengj

stevengj commented Jul 9, 2026

Copy link
Copy Markdown

See also JuliaLang/LinearAlgebra.jl#1650 for some benchmarks and exploration of various algorithms (in Julia, but the underlying principles should be broadly applicable).

then rerunning the calculation if an overflow occurred adds overhead in the nominal case.

I'm not sure what you mean by the "nominal" case — it only slows down the uncommon case of over/underflow.

As mentioned in the above Julia issue, it is also desirable to do scaling if the naive algorithm is merely subnormal, even if it doesn't underflow to 0.0, since with subnormal values the naive algorithm has poor accuracy. But that's a pretty rare corner case.

Extend the numpygh-8775 2-norm fast-path fix to also handle underflow: a tiny
vector whose sum of squares underflows to zero (e.g. norm([1e-200, 1e-200]))
now recomputes with the max-scaled sum of squares instead of returning 0.
The rescale is triggered on a non-finite *or* zero result and guarded so
that inf/nan inputs still propagate and an all-zero vector stays 0.

Also compute the complex sum of squares with vdot(x, x).real, a single pass
over the array. Add underflow regression tests (float64, float32, complex).
@vismaytiwari

Copy link
Copy Markdown
Author

See also JuliaLang/LinearAlgebra.jl#1650 for some benchmarks and exploration of various algorithms (in Julia, but the underlying principles should be broadly applicable).

then rerunning the calculation if an overflow occurred adds overhead in the nominal case.

I'm not sure what you mean by the "nominal" case — it only slows down the uncommon case of over/underflow.

As mentioned in the above Julia issue, it is also desirable to do scaling if the naive algorithm is merely subnormal, even if it doesn't underflow to 0.0, since with subnormal values the naive algorithm has poor accuracy. But that's a pretty rare corner case.

Thanks for the Julia link, good reference. On the subnormal case — agreed it'd be nice to scale there too for accuracy, but since it's a pretty rare corner case I've left it out to keep this PR focused; happy to do it as a follow-up if you'd rather. And thanks for confirming the perf side — only the uncommon over/underflow path pays anything.

Comment thread numpy/linalg/_linalg.py Outdated
else:
sqnorm = x.dot(x)
ret = sqrt(sqnorm)
if not math.isfinite(float(ret)) or ret == 0:

@ikrommyd ikrommyd Jul 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will not work. ret does not have to be a scalar. You need to check if ANY of the elements are such that the scaling fallback is needed, it's not 1 element. I do not have permissions to run the CI but please test locally. You can do so with spin test or spin test numpy/linalg to run only the linear algebra testing. I get the following with your branch:

FAILED numpy/linalg/tests/test_linalg.py::TestNorm_NonSystematic::test_overflow - RuntimeWarning: overflow encountered in dot
FAILED numpy/linalg/tests/test_linalg.py::TestNormDouble::test_empty - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED numpy/linalg/tests/test_linalg.py::TestNormDouble::test_matrix_empty - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED numpy/linalg/tests/test_linalg.py::TestNormSingle::test_empty - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED numpy/linalg/tests/test_linalg.py::TestNormSingle::test_matrix_empty - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED numpy/linalg/tests/test_linalg.py::TestNormInt64::test_empty - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED numpy/linalg/tests/test_linalg.py::TestNormInt64::test_matrix_empty - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED numpy/linalg/tests/test_regression.py::TestRegression::test_norm_object_array - TypeError: only 0-dimensional arrays can be converted to Python scalars
FAILED numpy/matrixlib/tests/test_matrix_linalg.py::TestNormDoubleMatrix::test_matrix_empty - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED numpy/matrixlib/tests/test_matrix_linalg.py::TestNormSingleMatrix::test_matrix_empty - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED numpy/matrixlib/tests/test_matrix_linalg.py::TestNormInt64Matrix::test_matrix_empty - ValueError: zero-size array to reduction operation maximum which has no identity

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right on all three — thanks for actually running it, and sorry for the churn (I was expecting CI/CD to catch this) .
Fix: I've restricted the rescale to non-empty real/complex float input (x.size and issubclass(x.dtype.type, inexact)). So empty arrays no longer hit abs(x).max() (the ret == 0 trigger I added was firing on them), and object dtype no longer hits float(ret) — that's your "ret isn't necessarily a scalar" point; object and empty now just fall through to the naive result. (In this fast path the input is ravel'd to 1-D and dotted, so the norm itself is a single scalar — the guard just keeps us on that scalar path only where it's valid.)
For the overflow RuntimeWarning, I wrapped the extreme-input assertions in np.errstate(...) in the test rather than putting errstate back in the hot path (keeping your earlier perf point) — the code recovers to the correct value and the warning just matches main's behaviour.
Built locally under 3.12 and ran spin test numpy/linalg: 458 passed, 0 failed, including test_overflow, test_empty, test_matrix_empty, and test_regression.py::test_norm_object_array. Pushed as a4c83d0.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't ret always be a scalar in this branch (axis is None)?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes — in this axis is None fast path it's a scalar (x is ravel'd to 1-D and dotted). The guard is for the two degenerate cases that also reach here: object dtype (where norm returns a non-0-d result — what tripped float(ret) earlier) and empty input (where abs(x).max() has no identity). x.size and issubclass(..., inexact) just keeps us on the scalar path where it holds.

@ikrommyd ikrommyd Jul 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoopsie I did not notice that this was all in an axis=None block. FYI you can provide an initial value to max using the initial= argument if it's helpful in any way.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Used initial=0 in the per-axis version so empty slices avoid a no-identity max.

…loat input

The numpygh-8775 fast-path rescale wrongly ran for empty arrays (abs(x).max() on a
zero-size array raised ValueError) and for object-dtype arrays (float(ret) on
a non-scalar raised TypeError), because the `ret == 0` trigger also fires for
those. Restrict the rescale to non-empty real/complex float input via
`x.size and issubclass(x.dtype.type, inexact)`, so empty and object-dtype
inputs keep the naive result unchanged. Wrap the extreme-input assertions in
the regression test in np.errstate so the expected over/underflow warnings
from the naive step (which the code recovers from) don't fail the test.
Comment thread numpy/linalg/tests/test_linalg.py Outdated
# gh-8775 also covers underflow: a tiny vector whose sum of squares
# underflows to 0 must still return the representable norm, not 0.
assert_allclose(norm(np.array([1e-200, 1e-200])), np.sqrt(2) * 1e-200)
assert norm(np.array([1e-200, 1e-200])) != 0.0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would check that it is approximately 1e-200 * np.sqrt(2).

(Since assert_allclose correctly has a default absolute tolerance atol=0, its default rtol=1e-07 will work fine for very small values.)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, you are already checking that above. Then I don't see the purpose of the assertion, which is redundant with the preceding allclose check.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — removed it; the assert_allclose right above already pins the value.

Comment thread numpy/linalg/tests/test_linalg.py Outdated
# float32 underflows sooner, but the scaled norm still fits.
x32 = np.array([1e-30, 1e-30], dtype=np.float32)
assert_allclose(norm(x32), np.sqrt(2) * 1e-30, rtol=1e-3)
assert norm(x32) != 0.0 and norm(x32).dtype == np.float32

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again, the != 0.0 check seems redundant

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped the != 0.0 here too — kept the norm(x32).dtype == np.float32 part, since that checks the float32 dtype survives the rescale (not redundant with the assert_allclose). Happy to drop it entirely if you'd prefer.

Comment thread numpy/linalg/_linalg.py Outdated
x_real = x.real
x_imag = x.imag
sqnorm = x_real.dot(x_real) + x_imag.dot(x_imag)
sqnorm = vdot(x, x).real

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't you just use vdot unconditionally for both real and complex arrays? For real arrays it is equivalent to dot, no?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — confirmed vdot(x, x).real matches x.dot(x) for real dtypes (float16 still overflows to inf as float16, float32 still underflows to 0), so I dropped the branch and use vdot for both the main and rescaled paths.

@stevengj

stevengj commented Jul 9, 2026

Copy link
Copy Markdown

Should also fix #19097 — be sure to add a test for the Frobenius norm case.

@stevengj

stevengj commented Jul 9, 2026

Copy link
Copy Markdown

Note that the axis case for ord == 2 has the same bug: https://github.com/vismaytiwari/numpy/blob/a4c83d02ec26340eb27008d68a2b4db1b6be9afa/numpy/linalg/_linalg.py#L2798-L2801

Similarly, it supports arbitrary ord > 1 and the code for that has the same bug: https://github.com/vismaytiwari/numpy/blob/a4c83d02ec26340eb27008d68a2b4db1b6be9afa/numpy/linalg/_linalg.py#L2807-L2811

…serts

Address review on numpygh-8775. Compute the sum of squares with vdot(x, x).real
unconditionally: for real input it is equivalent to x.dot(x), and it is a
single pass for complex, so the isComplexType branches can go. Drop the
norm != 0 assertions in the regression test that are redundant with the
preceding assert_allclose value checks (keeping the float32 dtype check), and
the empty/object assertions that duplicate the existing test_empty and
test_norm_object_array coverage.
@vismaytiwari

Copy link
Copy Markdown
Author

Should also fix #19097

Nice — #19097 is the same overflow (2-norm of a vector + Frobenius of a matrix), and both go through the fast path this PR fixes, so I've added Fixes #19097. There's already a Frobenius case in the test (norm([[1e200, 1e200]])) — happy to expand it to a full 2-D matrix.

the axis case for ord == 2 has the same bug ... arbitrary ord > 1

You're right. The fix there is different, though — those return an array, so it needs per-slice max-scaling rather than the scalar check here. I'd rather keep this PR focused on the fast path and do the axis/ord>1 cases in a follow-up PR.

@ikrommyd

ikrommyd commented Jul 9, 2026

Copy link
Copy Markdown
Member

Note that the axis case for ord == 2 has the same bug: https://github.com/vismaytiwari/numpy/blob/a4c83d02ec26340eb27008d68a2b4db1b6be9afa/numpy/linalg/_linalg.py#L2798-L2801

Similarly, it supports arbitrary ord > 1 and the code for that has the same bug: https://github.com/vismaytiwari/numpy/blob/a4c83d02ec26340eb27008d68a2b4db1b6be9afa/numpy/linalg/_linalg.py#L2807-L2811

Yeah I was just about to ask that cause I assumed that the problem can also occur in the case where axis is not None.
I'd prefer it if we fixed all the cases of the same bug of norm here @vismaytiwari . It's easier to keep the review focused.

Comment thread numpy/linalg/_linalg.py Outdated
Comment on lines +2739 to +2749
# For floating-point input the naive sum of squares can overflow
# to inf (values past sqrt(dtype max)) or underflow to 0 (tiny
# values) even though the norm itself is representable - e.g.
# float16([600, 800]) or float64([1e-200, 1e-200]). Recompute those
# (rare) cases with a max-scaled sum of squares, which is immune to
# both. See gh-8775. Restricted to non-empty real/complex float
# input: empty or object-dtype arrays don't have this problem and
# don't support the scalar float()/max() checks, so they fall
# through with the naive result.
# vdot(x, x).real is the sum of squared magnitudes in a single
# pass; for real input it is equivalent to x.dot(x).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is massive, please make it concise

@vismaytiwari vismaytiwari Jul 9, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

Comment thread numpy/linalg/_linalg.py Outdated
Comment on lines +2755 to +2757
# A non-finite or zero max means the input itself is
# non-finite (inf/nan should propagate) or all-zero (norm is
# genuinely 0); leave the naive result alone in those cases.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also a but big IMO but whatever. If it can be reduced to 1 or 2 lines sure

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reduced to one line.

…s norms

numpygh-8775 also affects the axis reductions that square (or raise to ord>1)
their input: the 1-D vector 2-norm and arbitrary ord>1, and the 2-D
Frobenius norm. Extend the max-scaled recompute to them via a shared
_rescale_axis_norm helper that rescales only the slices whose naive
power-sum over- or underflowed, leaving the rest (and inf/nan and genuine
zeros) untouched. abs(x).max(..., initial=0) keeps empty slices from
tripping a no-identity reduction. Also trims the overflow-fix comments in
the axis=None fast path per review.
@vismaytiwari

Copy link
Copy Markdown
Author

I'd prefer it if we fixed all the cases of the same bug of norm here

@ikrommyd done — all three are in this PR now: the per-axis vector 2-norm, arbitrary ord>1, and the 2-axis Frobenius norm, each rescaled per slice. Pushed with a test covering the mixed/overflow/underflow cases.

@ikrommyd

Copy link
Copy Markdown
Member

@stevengj I'd greatly appreciate if you could do another review pass for the math here 😃

Comment thread numpy/linalg/_linalg.py
# x.size/inexact guard skips empty and object arrays).
sqnorm = vdot(x, x).real
ret = sqrt(sqnorm)
if (x.size and issubclass(x.dtype.type, inexact)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The inexact check will always be True, no? x is converted to float at the beginning of the function: https://github.com/vismaytiwari/numpy/blob/c02eb25d9831615c29c322e4ece4d45dc9f3081e/numpy/linalg/_linalg.py#L2758-L2759

Otherwise you would have trouble with integer overflow, e.g.

>>> x = np.array([10000000000000, 10000000000000])
>>> np.vdot(x,x)
np.int64(-5075528580230807552)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Integers can't reach it — they're cast to float at the top, which is what avoids that int64 wraparound. The check is for object dtype: it stays un-cast and can return a non-scalar (norm([array([0,1]), 0, 0])[0, 1]), where float(ret) would raise — the test_norm_object_array case.

@stevengj

stevengj commented Jul 10, 2026

Copy link
Copy Markdown

The math looks good to me now, modulo the inaccuracy in the subnormal case (see the discussion in JuliaLang/LinearAlgebra.jl#1650 (comment)), which is kind of an obscure corner case that a lot of implementations seem to get wrong (even some BLAS dnrm2 implementations mess up IIRC). e.g. this example is only accurate to two digits:

>>> x = np.full(10, 1e-161)
>>> np.linalg.norm(x) # gets only 2 digits right
np.float64(3.1434555694052575e-161)
>>> np.sqrt(10) * 1e-161 # correct answer
np.float64(3.1622776601683798e-161)
>>> scipy.linalg.blas.dnrm2(x) # openblas 0.3.33 gets it right
3.1622776601683798e-161

To fix this, replace the ret == 0 check with sqnorm < numpy.finfo(ret.dtype).smallest_normal (replace sqnorm with ret**ord for other orders).

By the way, essentially the same spurious overflow/underflow problem (with the same fix) occurs in numpy.std — Julia and R also get std wrong in the same way, as described in JuliaStats/Statistics.jl#195 … but that should probably a separate issue/PR.

Per review on numpygh-8775: a sum of squares that is subnormal (but non-zero)
has already lost precision, so the norm is inaccurate even though the naive
result is finite and non-zero (e.g. norm(np.full(10, 1e-161)) was only
~2 digits). Trigger the max-scaled recompute on
sqnorm < finfo(ret.dtype).smallest_normal (ret**ord for the axis / ord>1
paths) instead of ret == 0. This still catches plain underflow (0 is below
smallest_normal) and does not fire on normal inputs. Tests added for the
scalar and per-axis cases.
@vismaytiwari

Copy link
Copy Markdown
Author

The math looks good to me now, modulo the inaccuracy in the subnormal case

Folded it in — I'd floated it as a follow-up, but it's a one-liner on top of the max-scaling already here: the trigger is now sqnorm < finfo(ret.dtype).smallest_normal (ret**ord for the axis / ord>1 paths). norm(np.full(10, 1e-161)) is full-precision now, with a test. Agree numpy.std should be its own issue/PR.

Comment thread numpy/linalg/_linalg.py
# (== x.dot(x) for real). gh-8775: it can overflow/underflow/go
# subnormal while the norm is representable, so rescale those rare
# float cases (x.size/inexact guard skips empty and object arrays).
sqnorm = vdot(x, x).real

@stevengj stevengj Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that vdot does twice as much arithmetic as the previous code, because it computes the (≈ zero) imaginary part only to discard it. However, since this is a BLAS1 operation I would expect it to be memory bound, so a single pass would still be faster for a sufficiently large array.

Would be good to do a little benchmark, however, to see how the performance compares as a function of x.size.

(Unfortunately, there doesn't seem to be a good primitive in numpy to compute just vdot(x,x).real in a single pass. If x is a contiguous complex array, you could reinterpret as a real array of twice the length and then just take the real dot product with itself, but that will fail if it x is not contiguous, e.g. a strided slice of some other array.)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Benchmarked it (numpy 2.4.6, Accelerate BLAS; contiguous complex128, since the code ravels x first). vdot(x, x).real vs the old x.real.dot(x.real) + x.imag.dot(x.imag):

size vdot 2-pass vdot/2-pass
1e5 22 µs 41 µs 0.53
1e6 213 µs 417 µs 0.51
1e7 2.1 ms 4.2 ms 0.51
1e8 21 ms 42 ms 0.50

So vdot is ~2× faster across the board — memory-bound as you expected, the extra imaginary flops don't show. And for real arrays vdot(x, x) == x.dot(x) (ratio ~1.0), so no overhead there.

One thing worth flagging: since we ravel(order='K') first, x is always contiguous at this point, so the reinterpret trick you mentioned does apply here — viewing the contiguous complex array as a length-2N real array and taking a real dot is another ~2–9× faster than vdot (e.g. 1e6: ~22 µs vs 213 µs), since it never computes the discarded imaginary part.

So two options: keep the simpler vdot (already ~2× over the previous code), or use the view-as-real reinterpret for the extra speed (a few more lines, guarded on applicability). Happy to go either way — any preference?

@stevengj stevengj Jul 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what the maintainers prefer here? Maybe postpone further performance optimization to another PR?

@vismaytiwari

Copy link
Copy Markdown
Author

@stevengj if everything looks good, can you approve?
Thanks

@stevengj

Copy link
Copy Markdown

LGTM. I can't formally click "approve" since I don't have write access to this repo.

@ikrommyd

Copy link
Copy Markdown
Member

I also only got triage rights so I can't run CI (needs merge rights). A maintainer needs to show up to do so. I'll do a pass though for any potential code (not math) changes

@vismaytiwari

Copy link
Copy Markdown
Author

I will for wait for maintainer but thanks for detailed review .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2-norm of vector and Frobenius norm of matrix, both suffer from unnecessary overflow BUG? np.linalg.norm overflows for float16

3 participants