Skip to content

Pins numpy<2, fix svd for scipy>=1.11.0 - #2827

Merged
MMathisLab merged 5 commits into
mainfrom
niels/pin_numpy
Jan 7, 2025
Merged

Pins numpy<2, fix svd for scipy>=1.11.0#2827
MMathisLab merged 5 commits into
mainfrom
niels/pin_numpy

Conversation

@n-poulsen

@n-poulsen n-poulsen commented Jan 7, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

This pull request pins the numpy version required for DeepLabCut to numpy<2.0.0 and fixes SVD computation in Tracklet.estimate_rank for scipy>=1.11.0.

scipy SVD computation

With scipy<1.11.0, computation of the SVD of an all-zero matrix would be successful, returning an all-zero array for the singular values. With scipy>=1.11.0, this fails with a ValueError. Hence, we first check if the matrice is the zero matrix before computing the SVD. If it is, we return a zero-vector to match the behavior of scipy<1.11.0.

This can be verified with the following script:

import numpy as np
from scipy.linalg.interpolative import svd

mat = np.ones((100, 100))
u, s, v = svd(mat, 10)
print(s.shape)
print(s)

mat = np.zeros((100, 100))
u, s, v = svd(mat, 10)
print(s.shape)
print(s)

With scipy==1.10.1 this succeeds with the output:

# Installed packages
# 
# Package    Version
# ---------- -------
# numpy      1.26.4
# pip        24.2
# scipy      1.10.1
# setuptools 75.1.0
# wheel      0.44.0

(10,)
[1.00000000e+002 2.00894856e-014 9.74976675e-030 1.87711978e-044
 3.36274201e-059 7.64750134e-074 6.78987388e-089 1.85753577e-103
 5.94236148e-118 5.94807826e-133]
(10,)
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]

While with scipy==1.15.0 this fails with the output:

# Installed packages
# 
# Package    Version
# ---------- -------
# numpy      1.26.4
# pip        24.2
# scipy      1.15.0
# setuptools 75.1.0
# wheel      0.44.0

(10,)
[1.00000000e+002 1.97841743e-014 2.33093502e-029 3.08938377e-044
 4.24604188e-059 6.43087871e-074 1.13626229e-088 8.36470319e-104
 1.62555162e-118 3.13271339e-133]
Traceback (most recent call last):
  File "sktest.py", line 11, in <module>
    u, s, v = svd(mat, 10)
  File "/miniconda/envs/sk15/lib/python3.10/site-packages/scipy/linalg/interpolative.py", line 905, in svd
    U, S, V = _backend.iddr_asvd(A, k, rng=rng)
  File "_decomp_interpolative.pyx", line 933, in scipy.linalg._decomp_interpolative.iddr_asvd
  File "/miniconda/envs/sk15/lib/python3.10/site-packages/scipy/linalg/_decomp_svd.py", line 106, in svd
    a1 = _asarray_validated(a, check_finite=check_finite)
  File "/miniconda/envs/sk15/lib/python3.10/site-packages/scipy/_lib/_util.py", line 537, in _asarray_validated
    a = toarray(a)
  File "/miniconda/envs/sk15/lib/python3.10/site-packages/numpy/lib/function_base.py", line 630, in asarray_chkfinite
    raise ValueError(
ValueError: array must not contain infs or NaNs

@n-poulsen n-poulsen added the dependencies Pull requests that update a dependency file label Jan 7, 2025
@n-poulsen
n-poulsen requested a review from MMathisLab January 7, 2025 14:00
@n-poulsen n-poulsen changed the title Pins numpy<2 Pins numpy<2, fix stitching for scipy>=1.11.0 Jan 7, 2025
@n-poulsen n-poulsen changed the title Pins numpy<2, fix stitching for scipy>=1.11.0 Pins numpy<2, fix svd for scipy>=1.11.0 Jan 7, 2025
@n-poulsen
n-poulsen requested a review from maximpavliv January 7, 2025 15:22
@MMathisLab
MMathisLab merged commit d4da23a into main Jan 7, 2025
@n-poulsen
n-poulsen deleted the niels/pin_numpy branch January 7, 2025 15:28
@n-poulsen n-poulsen mentioned this pull request Jan 7, 2025
Closed
2 tasks
n-poulsen added a commit that referenced this pull request Jan 8, 2025
@n-poulsen
Pins numpy<2, fix svd for scipy>=1.11.0 (#2827)
20b6e49 
* pin numpy to < 2.0

* update numpy pin in requirements.txt

* fix SVD in stitch with scipy>=1.11.0

* speedup check if matrix contains only zeros

* improve readability
@maximpavliv maximpavliv mentioned this pull request Jan 10, 2025
Closed
@hmaarrfk

hmaarrfk commented Feb 17, 2025

Copy link
Copy Markdown
Contributor

As a user and downstream packager, I would just like to ask why numpy is being pinned to less than 2?

In my experience it has been quite "smooth" all things considered to transition to numpy2 for python packages. Tensorflow also just released a version compatible with numpy2 these last few months (tensorflow 2.18)

@natsukium natsukium mentioned this pull request Jul 20, 2026
Open
natsukium added a commit to natsukium/DeepLabCut that referenced this pull request Jul 25, 2026
@natsukium
Estimate tracklet rank with a dense SVD instead of interpolative SVD
4b977b4 
Tracklet stitching estimated a Hankelet's rank from its singular values via
scipy.linalg.interpolative.svd. On NumPy 2 that randomized routine raises
"array must not contain infs or NaNs" on finite input, so any stitching that
computes tracklet affinities fails at runtime.

Only the singular values matter, so use np.linalg.svd(compute_uv=False):
exact, deterministic, identical on NumPy 1.22.4 and 2. It also returns zeros
for an all-zero matrix instead of raising, making the DeepLabCut#2827 guard against
scipy>=1.11's all-zero ValueError redundant; it is removed.

scipy is removing the backend's noncompliant randomization:
scipy/scipy#18367
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MMathisLab MMathisLab MMathisLab approved these changes
@maximpavliv maximpavliv Awaiting requested review from maximpavliv

Assignees

No one assigned

Labels

dependencies Pull requests that update a dependency file

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@n-poulsen @hmaarrfk @MMathisLab