Add an asv benchmark suite - #596
Open
eriknw wants to merge 1 commit into
Open
Conversation
eriknw
marked this pull request as ready for review
August 4, 2026 16:07
eriknw
force-pushed
the
16-asv-benchmarks
branch
from
August 4, 2026 16:12
0610f1d to
8120ea8
Compare
eriknw
force-pushed
the
16-asv-benchmarks
branch
2 times, most recently
from
August 5, 2026 03:18
a9630c5 to
871707e
Compare
eriknw
force-pushed
the
16-asv-benchmarks
branch
from
August 5, 2026 17:44
871707e to
48c0a37
Compare
eriknw
force-pushed
the
16-asv-benchmarks
branch
from
August 5, 2026 18:03
48c0a37 to
0b7a63a
Compare
eriknw
force-pushed
the
16-asv-benchmarks
branch
2 times, most recently
from
August 6, 2026 07:59
2241c0d to
f28bd3e
Compare
eriknw
force-pushed
the
16-asv-benchmarks
branch
from
August 6, 2026 15:39
f28bd3e to
0a586c9
Compare
eriknw
force-pushed
the
16-asv-benchmarks
branch
3 times, most recently
from
August 6, 2026 20:41
e469e6c to
ced8b66
Compare
eriknw
force-pushed
the
16-asv-benchmarks
branch
from
August 7, 2026 02:48
ced8b66 to
80d325e
Compare
101 benchmarks over the library's hot paths: scalar access (getitem, get and contains on hit and miss, setitem), index parsing, small-op overhead on 10/100/1000-element objects, large kernels at ~1e6 nnz (mxm, mxv, ewise, reduce, single-threaded via OMP_NUM_THREADS=1), coo/dense/scipy conversions, cold import and first-use timings in fresh subprocesses, and repr costs. `benchmarks/_verify.py` is a standalone checker rather than an asv benchmark: it imports every module the way asv does and calls each benchmark once across all parameter combinations. It reports 101 passed, 0 failed, 0 skipped. No timing numbers are claimed; what this commit establishes is that the benchmarks exist and run. `asv.conf.json` uses `//` comments. That is asv's own documented config format and its loader strips them, but the repo's `check-json` hook is a strict JSON parser and rejects the file, so that one path is excluded from the hook and the hook's stale "no JSON files yet" comment is replaced. Verified with `pre-commit run check-json --all-files`, which passes. Two alternatives were considered and not taken. Renaming to `asv.conf.jsonc` needs no exclude and asv 0.6.5 does resolve it (`Config.load` accepts `.json` and `.jsonc`), but asv is not a pinned dependency here and older versions look only for the `.json` name, which would fail as "No `asv.conf` file found". Stripping the comments would delete the only explanation of why each setting is set as it is. The exclude names the single path, so every other JSON file is still checked. asv is deliberately not added to `dev-requirements.txt` or `environment.yml`. Nothing in the test suite or CI invokes it, and this repo ties dependency changes to `scripts/check_versions.sh` and the CI version pools, which a benchmarks-only change should not be editing. The README carries the `pip install asv` line and the reasoning. CI wiring is also left alone: a weekly cron on a fixed machine suits asv far better than per-PR runs on shared runners. The lint configuration for benchmarks/ rides in this commit because the directory it configures arrives here: flake8 and ruff ignore B015/B018 (a bare expression IS what an asv benchmark measures) and T201 for the benchmark CLI, and codespell learns "bu".
eriknw
force-pushed
the
16-asv-benchmarks
branch
from
August 7, 2026 05:09
80d325e to
dfa59f3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
101 benchmarks over the library's hot paths: scalar access (getitem, get and
contains on hit and miss, setitem), index parsing, small-op overhead on
10/100/1000-element objects, large kernels at ~1e6 nnz (mxm, mxv, ewise,
reduce, single-threaded via OMP_NUM_THREADS=1), coo/dense/scipy conversions,
cold import and first-use timings in fresh subprocesses, and repr costs.
benchmarks/_verify.pyis a standalone checker rather than an asv benchmark:it imports every module the way asv does and calls each benchmark once across
all parameter combinations. It reports 101 passed, 0 failed, 0 skipped. No
timing numbers are claimed; what this commit establishes is that the
benchmarks exist and run.
asv.conf.jsonuses//comments. That is asv's own documented configformat and its loader strips them, but the repo's
check-jsonhook is astrict JSON parser and rejects the file, so that one path is excluded from
the hook and the hook's stale "no JSON files yet" comment is replaced.
Verified with
pre-commit run check-json --all-files, which passes.Two alternatives were considered and not taken. Renaming to
asv.conf.jsoncneeds no exclude and asv 0.6.5 does resolve it (
Config.loadaccepts.jsonand.jsonc), but asv is not a pinned dependency here and olderversions look only for the
.jsonname, which would fail as "Noasv.conffile found". Stripping the comments would delete the only explanation of why
each setting is set as it is. The exclude names the single path, so every
other JSON file is still checked.
asv is deliberately not added to
dev-requirements.txtorenvironment.yml.Nothing in the test suite or CI invokes it, and this repo ties dependency
changes to
scripts/check_versions.shand the CI version pools, which abenchmarks-only change should not be editing. The README carries the
pip install asvline and the reasoning. CI wiring is also left alone: aweekly cron on a fixed machine suits asv far better than per-PR runs on
shared runners.
The lint configuration for benchmarks/ rides in this commit because the
directory it configures arrives here: flake8 and ruff ignore B015/B018
(a bare expression IS what an asv benchmark measures) and T201 for the
benchmark CLI, and codespell learns "bu".
Stack created with GitHub Stacks CLI • Give Feedback 💬