BUG: fix np.linalg.norm overflow for representable results (gh-8775)#31927
BUG: fix np.linalg.norm overflow for representable results (gh-8775)#31927vismaytiwari wants to merge 8 commits into
Conversation
) The 2-norm fast path accumulates the sum of squares in the input dtype, so it overflows to inf once that sum exceeds the dtype maximum - even when the norm itself is representable. This happens for float16 (norm([600, 800]) returned inf instead of 1000) and, as noted in the issue, for any float dtype past sqrt(max) (e.g. norm([1e200, 1e200]) in float64). Detect a non-finite result from all-finite input and recompute the sum of squares after scaling by max(|x|), which cannot overflow. The common non-overflowing path is unchanged apart from an errstate guard that suppresses the transient overflow warning. Genuine inf/nan inputs and zeros are preserved.
|
I have a feeling that doing the calculation, checking every element of the output with To maintainers: not 100% sure if I'd call this a bug. It's most likely a "bug?" with a question mark like the issue says but I added the bug label anyways. Somebody change it if you feel otherwise |
|
Hey @ikrommyd —
|
Following review: the errstate context manager plus the np.isfinite ufunc dispatch added measurable per-call overhead in the common (non-overflowing) path (~2-3x on small arrays). Drop the errstate wrapper and use math.isfinite(float(ret)) for the scalar overflow check. The nominal path now adds no measurable overhead (size-3 within noise) and the recompute path is unchanged; overflow-case warning behavior matches the prior code.
| if not math.isfinite(float(ret)) and isfinite(x).all(): | ||
| max_abs = abs(x).max() | ||
| scaled = x / max_abs | ||
| if isComplexType(x.dtype.type): | ||
| sqnorm = (scaled.real.dot(scaled.real) | ||
| + scaled.imag.dot(scaled.imag)) | ||
| else: | ||
| sqnorm = scaled.dot(scaled) | ||
| ret = max_abs * sqrt(sqnorm) |
There was a problem hiding this comment.
You also need to check for underflow, e.g. [1e-200, 1e-200] should not have a norm of 0.0. Something like
| if not math.isfinite(float(ret)) and isfinite(x).all(): | |
| max_abs = abs(x).max() | |
| scaled = x / max_abs | |
| if isComplexType(x.dtype.type): | |
| sqnorm = (scaled.real.dot(scaled.real) | |
| + scaled.imag.dot(scaled.imag)) | |
| else: | |
| sqnorm = scaled.dot(scaled) | |
| ret = max_abs * sqrt(sqnorm) | |
| if not math.isfinite(float(ret)) and ret != 0: # possible over/underflow | |
| max_abs = abs(x).max() | |
| if not math.isfinite(float(max_abs)) or max_abs == 0: | |
| return ret | |
| scaled = x / max_abs | |
| if isComplexType(x.dtype.type): | |
| sqnorm = (scaled.real.dot(scaled.real) | |
| + scaled.imag.dot(scaled.imag)) | |
| else: | |
| sqnorm = scaled.dot(scaled) | |
| ret = max_abs * sqrt(sqnorm) |
There was a problem hiding this comment.
thanks — you're right, it was only handling overflow. Pushed a fix so it rescales on underflow too.
One small thing on the suggested condition: not math.isfinite(ret) and ret != 0 doesn't actually catch the underflow case, since [1e-200, 1e-200] gives ret == 0 (which is finite) — so I went with not math.isfinite(ret) or ret == 0. Kept your max_abs guard so inf/nan still propagate and an all-zero vector stays 0. norm([1e-200, 1e-200]) now returns √2·1e-200 instead of 0, and I added float64/float32/complex underflow cases to the test.
There was a problem hiding this comment.
Sorry, I meant not math.isfinite(float(ret)) or ret == 0
There was a problem hiding this comment.
Yep, that's what's in the current version — not math.isfinite(float(ret)) or ret == 0. 👍
| if isComplexType(x.dtype.type): | ||
| x_real = x.real | ||
| x_imag = x.imag | ||
| sqnorm = x_real.dot(x_real) + x_imag.dot(x_imag) |
There was a problem hiding this comment.
(Seems like it should be faster to do numpy.vdot(x, x).real, so that it can do a single pass over the complex array?)
There was a problem hiding this comment.
Yeah, that's cleaner — switched the complex sum of squares to vdot(x, x).real so it's a single pass, both in the main path and the rescaled one. Thanks!
|
See also JuliaLang/LinearAlgebra.jl#1650 for some benchmarks and exploration of various algorithms (in Julia, but the underlying principles should be broadly applicable).
I'm not sure what you mean by the "nominal" case — it only slows down the uncommon case of over/underflow. As mentioned in the above Julia issue, it is also desirable to do scaling if the naive algorithm is merely subnormal, even if it doesn't underflow to |
Extend the numpygh-8775 2-norm fast-path fix to also handle underflow: a tiny vector whose sum of squares underflows to zero (e.g. norm([1e-200, 1e-200])) now recomputes with the max-scaled sum of squares instead of returning 0. The rescale is triggered on a non-finite *or* zero result and guarded so that inf/nan inputs still propagate and an all-zero vector stays 0. Also compute the complex sum of squares with vdot(x, x).real, a single pass over the array. Add underflow regression tests (float64, float32, complex).
Thanks for the Julia link, good reference. On the subnormal case — agreed it'd be nice to scale there too for accuracy, but since it's a pretty rare corner case I've left it out to keep this PR focused; happy to do it as a follow-up if you'd rather. And thanks for confirming the perf side — only the uncommon over/underflow path pays anything. |
| else: | ||
| sqnorm = x.dot(x) | ||
| ret = sqrt(sqnorm) | ||
| if not math.isfinite(float(ret)) or ret == 0: |
There was a problem hiding this comment.
This will not work. ret does not have to be a scalar. You need to check if ANY of the elements are such that the scaling fallback is needed, it's not 1 element. I do not have permissions to run the CI but please test locally. You can do so with spin test or spin test numpy/linalg to run only the linear algebra testing. I get the following with your branch:
FAILED numpy/linalg/tests/test_linalg.py::TestNorm_NonSystematic::test_overflow - RuntimeWarning: overflow encountered in dot
FAILED numpy/linalg/tests/test_linalg.py::TestNormDouble::test_empty - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED numpy/linalg/tests/test_linalg.py::TestNormDouble::test_matrix_empty - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED numpy/linalg/tests/test_linalg.py::TestNormSingle::test_empty - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED numpy/linalg/tests/test_linalg.py::TestNormSingle::test_matrix_empty - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED numpy/linalg/tests/test_linalg.py::TestNormInt64::test_empty - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED numpy/linalg/tests/test_linalg.py::TestNormInt64::test_matrix_empty - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED numpy/linalg/tests/test_regression.py::TestRegression::test_norm_object_array - TypeError: only 0-dimensional arrays can be converted to Python scalars
FAILED numpy/matrixlib/tests/test_matrix_linalg.py::TestNormDoubleMatrix::test_matrix_empty - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED numpy/matrixlib/tests/test_matrix_linalg.py::TestNormSingleMatrix::test_matrix_empty - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED numpy/matrixlib/tests/test_matrix_linalg.py::TestNormInt64Matrix::test_matrix_empty - ValueError: zero-size array to reduction operation maximum which has no identity
There was a problem hiding this comment.
You're right on all three — thanks for actually running it, and sorry for the churn (I was expecting CI/CD to catch this) .
Fix: I've restricted the rescale to non-empty real/complex float input (x.size and issubclass(x.dtype.type, inexact)). So empty arrays no longer hit abs(x).max() (the ret == 0 trigger I added was firing on them), and object dtype no longer hits float(ret) — that's your "ret isn't necessarily a scalar" point; object and empty now just fall through to the naive result. (In this fast path the input is ravel'd to 1-D and dotted, so the norm itself is a single scalar — the guard just keeps us on that scalar path only where it's valid.)
For the overflow RuntimeWarning, I wrapped the extreme-input assertions in np.errstate(...) in the test rather than putting errstate back in the hot path (keeping your earlier perf point) — the code recovers to the correct value and the warning just matches main's behaviour.
Built locally under 3.12 and ran spin test numpy/linalg: 458 passed, 0 failed, including test_overflow, test_empty, test_matrix_empty, and test_regression.py::test_norm_object_array. Pushed as a4c83d0.
There was a problem hiding this comment.
Shouldn't ret always be a scalar in this branch (axis is None)?
There was a problem hiding this comment.
Yes — in this axis is None fast path it's a scalar (x is ravel'd to 1-D and dotted). The guard is for the two degenerate cases that also reach here: object dtype (where norm returns a non-0-d result — what tripped float(ret) earlier) and empty input (where abs(x).max() has no identity). x.size and issubclass(..., inexact) just keeps us on the scalar path where it holds.
There was a problem hiding this comment.
Whoopsie I did not notice that this was all in an axis=None block. FYI you can provide an initial value to max using the initial= argument if it's helpful in any way.
There was a problem hiding this comment.
Used initial=0 in the per-axis version so empty slices avoid a no-identity max.
…loat input The numpygh-8775 fast-path rescale wrongly ran for empty arrays (abs(x).max() on a zero-size array raised ValueError) and for object-dtype arrays (float(ret) on a non-scalar raised TypeError), because the `ret == 0` trigger also fires for those. Restrict the rescale to non-empty real/complex float input via `x.size and issubclass(x.dtype.type, inexact)`, so empty and object-dtype inputs keep the naive result unchanged. Wrap the extreme-input assertions in the regression test in np.errstate so the expected over/underflow warnings from the naive step (which the code recovers from) don't fail the test.
| # gh-8775 also covers underflow: a tiny vector whose sum of squares | ||
| # underflows to 0 must still return the representable norm, not 0. | ||
| assert_allclose(norm(np.array([1e-200, 1e-200])), np.sqrt(2) * 1e-200) | ||
| assert norm(np.array([1e-200, 1e-200])) != 0.0 |
There was a problem hiding this comment.
I would check that it is approximately 1e-200 * np.sqrt(2).
(Since assert_allclose correctly has a default absolute tolerance atol=0, its default rtol=1e-07 will work fine for very small values.)
There was a problem hiding this comment.
Oh, you are already checking that above. Then I don't see the purpose of the assertion, which is redundant with the preceding allclose check.
There was a problem hiding this comment.
Agreed — removed it; the assert_allclose right above already pins the value.
| # float32 underflows sooner, but the scaled norm still fits. | ||
| x32 = np.array([1e-30, 1e-30], dtype=np.float32) | ||
| assert_allclose(norm(x32), np.sqrt(2) * 1e-30, rtol=1e-3) | ||
| assert norm(x32) != 0.0 and norm(x32).dtype == np.float32 |
There was a problem hiding this comment.
again, the != 0.0 check seems redundant
There was a problem hiding this comment.
Dropped the != 0.0 here too — kept the norm(x32).dtype == np.float32 part, since that checks the float32 dtype survives the rescale (not redundant with the assert_allclose). Happy to drop it entirely if you'd prefer.
| x_real = x.real | ||
| x_imag = x.imag | ||
| sqnorm = x_real.dot(x_real) + x_imag.dot(x_imag) | ||
| sqnorm = vdot(x, x).real |
There was a problem hiding this comment.
Can't you just use vdot unconditionally for both real and complex arrays? For real arrays it is equivalent to dot, no?
There was a problem hiding this comment.
Done — confirmed vdot(x, x).real matches x.dot(x) for real dtypes (float16 still overflows to inf as float16, float32 still underflows to 0), so I dropped the branch and use vdot for both the main and rescaled paths.
|
Should also fix #19097 — be sure to add a test for the Frobenius norm case. |
|
Note that the Similarly, it supports arbitrary |
…serts Address review on numpygh-8775. Compute the sum of squares with vdot(x, x).real unconditionally: for real input it is equivalent to x.dot(x), and it is a single pass for complex, so the isComplexType branches can go. Drop the norm != 0 assertions in the regression test that are redundant with the preceding assert_allclose value checks (keeping the float32 dtype check), and the empty/object assertions that duplicate the existing test_empty and test_norm_object_array coverage.
Nice — #19097 is the same overflow (2-norm of a vector + Frobenius of a matrix), and both go through the fast path this PR fixes, so I've added
You're right. The fix there is different, though — those return an array, so it needs per-slice max-scaling rather than the scalar check here. I'd rather keep this PR focused on the fast path and do the axis/ |
Yeah I was just about to ask that cause I assumed that the problem can also occur in the case where axis is not None. |
| # For floating-point input the naive sum of squares can overflow | ||
| # to inf (values past sqrt(dtype max)) or underflow to 0 (tiny | ||
| # values) even though the norm itself is representable - e.g. | ||
| # float16([600, 800]) or float64([1e-200, 1e-200]). Recompute those | ||
| # (rare) cases with a max-scaled sum of squares, which is immune to | ||
| # both. See gh-8775. Restricted to non-empty real/complex float | ||
| # input: empty or object-dtype arrays don't have this problem and | ||
| # don't support the scalar float()/max() checks, so they fall | ||
| # through with the naive result. | ||
| # vdot(x, x).real is the sum of squared magnitudes in a single | ||
| # pass; for real input it is equivalent to x.dot(x). |
There was a problem hiding this comment.
this is massive, please make it concise
| # A non-finite or zero max means the input itself is | ||
| # non-finite (inf/nan should propagate) or all-zero (norm is | ||
| # genuinely 0); leave the naive result alone in those cases. |
There was a problem hiding this comment.
This is also a but big IMO but whatever. If it can be reduced to 1 or 2 lines sure
…s norms numpygh-8775 also affects the axis reductions that square (or raise to ord>1) their input: the 1-D vector 2-norm and arbitrary ord>1, and the 2-D Frobenius norm. Extend the max-scaled recompute to them via a shared _rescale_axis_norm helper that rescales only the slices whose naive power-sum over- or underflowed, leaving the rest (and inf/nan and genuine zeros) untouched. abs(x).max(..., initial=0) keeps empty slices from tripping a no-identity reduction. Also trims the overflow-fix comments in the axis=None fast path per review.
@ikrommyd done — all three are in this PR now: the per-axis vector 2-norm, arbitrary |
|
@stevengj I'd greatly appreciate if you could do another review pass for the math here 😃 |
| # x.size/inexact guard skips empty and object arrays). | ||
| sqnorm = vdot(x, x).real | ||
| ret = sqrt(sqnorm) | ||
| if (x.size and issubclass(x.dtype.type, inexact) |
There was a problem hiding this comment.
The inexact check will always be True, no? x is converted to float at the beginning of the function: https://github.com/vismaytiwari/numpy/blob/c02eb25d9831615c29c322e4ece4d45dc9f3081e/numpy/linalg/_linalg.py#L2758-L2759
Otherwise you would have trouble with integer overflow, e.g.
>>> x = np.array([10000000000000, 10000000000000])
>>> np.vdot(x,x)
np.int64(-5075528580230807552)There was a problem hiding this comment.
Integers can't reach it — they're cast to float at the top, which is what avoids that int64 wraparound. The check is for object dtype: it stays un-cast and can return a non-scalar (norm([array([0,1]), 0, 0]) → [0, 1]), where float(ret) would raise — the test_norm_object_array case.
|
The math looks good to me now, modulo the inaccuracy in the subnormal case (see the discussion in JuliaLang/LinearAlgebra.jl#1650 (comment)), which is kind of an obscure corner case that a lot of implementations seem to get wrong (even some BLAS >>> x = np.full(10, 1e-161)
>>> np.linalg.norm(x) # gets only 2 digits right
np.float64(3.1434555694052575e-161)
>>> np.sqrt(10) * 1e-161 # correct answer
np.float64(3.1622776601683798e-161)
>>> scipy.linalg.blas.dnrm2(x) # openblas 0.3.33 gets it right
3.1622776601683798e-161To fix this, replace the By the way, essentially the same spurious overflow/underflow problem (with the same fix) occurs in |
Per review on numpygh-8775: a sum of squares that is subnormal (but non-zero) has already lost precision, so the norm is inaccurate even though the naive result is finite and non-zero (e.g. norm(np.full(10, 1e-161)) was only ~2 digits). Trigger the max-scaled recompute on sqnorm < finfo(ret.dtype).smallest_normal (ret**ord for the axis / ord>1 paths) instead of ret == 0. This still catches plain underflow (0 is below smallest_normal) and does not fire on normal inputs. Tests added for the scalar and per-axis cases.
Folded it in — I'd floated it as a follow-up, but it's a one-liner on top of the max-scaling already here: the trigger is now |
| # (== x.dot(x) for real). gh-8775: it can overflow/underflow/go | ||
| # subnormal while the norm is representable, so rescale those rare | ||
| # float cases (x.size/inexact guard skips empty and object arrays). | ||
| sqnorm = vdot(x, x).real |
There was a problem hiding this comment.
Note that vdot does twice as much arithmetic as the previous code, because it computes the (≈ zero) imaginary part only to discard it. However, since this is a BLAS1 operation I would expect it to be memory bound, so a single pass would still be faster for a sufficiently large array.
Would be good to do a little benchmark, however, to see how the performance compares as a function of x.size.
(Unfortunately, there doesn't seem to be a good primitive in numpy to compute just vdot(x,x).real in a single pass. If x is a contiguous complex array, you could reinterpret as a real array of twice the length and then just take the real dot product with itself, but that will fail if it x is not contiguous, e.g. a strided slice of some other array.)
There was a problem hiding this comment.
Benchmarked it (numpy 2.4.6, Accelerate BLAS; contiguous complex128, since the code ravels x first). vdot(x, x).real vs the old x.real.dot(x.real) + x.imag.dot(x.imag):
| size | vdot | 2-pass | vdot/2-pass |
|---|---|---|---|
| 1e5 | 22 µs | 41 µs | 0.53 |
| 1e6 | 213 µs | 417 µs | 0.51 |
| 1e7 | 2.1 ms | 4.2 ms | 0.51 |
| 1e8 | 21 ms | 42 ms | 0.50 |
So vdot is ~2× faster across the board — memory-bound as you expected, the extra imaginary flops don't show. And for real arrays vdot(x, x) == x.dot(x) (ratio ~1.0), so no overhead there.
One thing worth flagging: since we ravel(order='K') first, x is always contiguous at this point, so the reinterpret trick you mentioned does apply here — viewing the contiguous complex array as a length-2N real array and taking a real dot is another ~2–9× faster than vdot (e.g. 1e6: ~22 µs vs 213 µs), since it never computes the discarded imaginary part.
So two options: keep the simpler vdot (already ~2× over the previous code), or use the view-as-real reinterpret for the extra speed (a few more lines, guarded on applicability). Happy to go either way — any preference?
There was a problem hiding this comment.
I'm not sure what the maintainers prefer here? Maybe postpone further performance optimization to another PR?
|
@stevengj if everything looks good, can you approve? |
|
LGTM. I can't formally click "approve" since I don't have write access to this repo. |
|
I also only got triage rights so I can't run CI (needs merge rights). A maintainer needs to show up to do so. I'll do a pass though for any potential code (not math) changes |
|
I will for wait for maintainer but thanks for detailed review . |
Fixes #8775. Fixes #19097.
np.linalg.norm's 2-norm fast path accumulates the sum of squares in the input dtype, so it overflows toinfonce that sum exceeds the dtype maximum — even when the norm itself is representable:As @stevengj pointed out on the issue, this isn't float16-specific — it happens for any float dtype past
sqrt(max), e.g.norm([1e200, 1e200])in float64.This detects a non-finite result from all-finite input and recomputes the sum of squares after scaling by
max(|x|), which cannot overflow (the classicnrm2scaling):The common (non-overflowing) path is unchanged — a single
vdot(x, x).real, with the scaled recompute only when that result is non-finite or zero. Genuineinf/naninputs and zeros are preserved.The same recompute is applied to the reductions that square (or raise to
ord>1) their input: theaxis=Nonefast path (1-D 2-norm and 2-D Frobenius), the per-axis vector 2-norm and arbitraryord>1, and the Frobenius norm reduced over an axis pair. The axis cases rescale per slice via a small helper, so only the rows/columns that actually over/underflow are recomputed. The matrix 2-norm/nuclear norm (SVD) and the 1-/inf-norms don't square, so they're unaffected.The transient overflow warning from the naive step is suppressed only in the test (via
np.errstate), so the hot path stays free of any per-call context.AI Disclosure: