[ty] Benchmark NumPy and its tests in CI - #30262

Stacked on #30241.

Add NumPy to the ty walltime benchmarks alongside pandas and colour-science. The benchmark checks the entire numpy package at the revision from astral-sh/ty#4990. This includes the parametrized test whose list of NumPy functions exposed the slowdown. The issue reduces that list to a single entry, [np.inner].

This benchmark would have caught the severe performance regression fixed by #30241. On CodSpeed, the median time for a multithreaded NumPy check was 5m05s on main without #30241 (9d2a566, one run). On this branch with #30241 (1477022), the median of six per-run medians was 17.07s. Each run took two samples on the same type of runner. That makes #30241 about 18× faster than main in this comparison¹.

• Baseline job on main with NumPy added
• Stacked PR runs

CI cost

The separate CI runs below compare the existing suite on main with the suite including NumPy on this PR.

The ty walltime step took 36 seconds longer in the PR run, but the instrumented ty shard simulation: micro-2 finished last in both runs.

Run-to-run variability

Across six CodSpeed runs of the same benchmark binary, NumPy's per-run medians ranged from 15.17s to 19.02s, a 25% increase from fastest to slowest. By relative standard deviation, it was about 14× more variable than pandas, the next most variable benchmark in the ty walltime suite. Fastest NumPy run · Slowest NumPy run

The other five benchmarks each had a relative standard deviation below 0.46%. Even on the same physical runner, NumPy's median was 19.02s, 17.43s, and 16.71s in three separate runs. Run 1 · Run 2 · Run 6

With this much variation, a single 10% change is hard to interpret. We'll probably want a relatively high threshold for CodSpeed to comment on NumPy regressions or improvements. The benchmark is still worth running in CI to catch pathological regressions like the one fixed in #30241.