Skip to content

Add an asv benchmark suite - #596

Open
eriknw wants to merge 1 commit into
15-udt-division-semanticsfrom
16-asv-benchmarks
Open

Add an asv benchmark suite#596
eriknw wants to merge 1 commit into
15-udt-division-semanticsfrom
16-asv-benchmarks

Conversation

@eriknw

@eriknw eriknw commented Aug 4, 2026

Copy link
Copy Markdown
Member

101 benchmarks over the library's hot paths: scalar access (getitem, get and
contains on hit and miss, setitem), index parsing, small-op overhead on
10/100/1000-element objects, large kernels at ~1e6 nnz (mxm, mxv, ewise,
reduce, single-threaded via OMP_NUM_THREADS=1), coo/dense/scipy conversions,
cold import and first-use timings in fresh subprocesses, and repr costs.

benchmarks/_verify.py is a standalone checker rather than an asv benchmark:
it imports every module the way asv does and calls each benchmark once across
all parameter combinations. It reports 101 passed, 0 failed, 0 skipped. No
timing numbers are claimed; what this commit establishes is that the
benchmarks exist and run.

asv.conf.json uses // comments. That is asv's own documented config
format and its loader strips them, but the repo's check-json hook is a
strict JSON parser and rejects the file, so that one path is excluded from
the hook and the hook's stale "no JSON files yet" comment is replaced.
Verified with pre-commit run check-json --all-files, which passes.

Two alternatives were considered and not taken. Renaming to asv.conf.jsonc
needs no exclude and asv 0.6.5 does resolve it (Config.load accepts
.json and .jsonc), but asv is not a pinned dependency here and older
versions look only for the .json name, which would fail as "No asv.conf
file found". Stripping the comments would delete the only explanation of why
each setting is set as it is. The exclude names the single path, so every
other JSON file is still checked.

asv is deliberately not added to dev-requirements.txt or environment.yml.
Nothing in the test suite or CI invokes it, and this repo ties dependency
changes to scripts/check_versions.sh and the CI version pools, which a
benchmarks-only change should not be editing. The README carries the
pip install asv line and the reasoning. CI wiring is also left alone: a
weekly cron on a fixed machine suits asv far better than per-PR runs on
shared runners.

The lint configuration for benchmarks/ rides in this commit because the
directory it configures arrives here: flake8 and ruff ignore B015/B018
(a bare expression IS what an asv benchmark measures) and T201 for the
benchmark CLI, and codespell learns "bu".


Stack created with GitHub Stacks CLIGive Feedback 💬

@eriknw
eriknw marked this pull request as ready for review August 4, 2026 16:07
@eriknw
eriknw force-pushed the 16-asv-benchmarks branch from 0610f1d to 8120ea8 Compare August 4, 2026 16:12
@eriknw
eriknw force-pushed the 16-asv-benchmarks branch 2 times, most recently from a9630c5 to 871707e Compare August 5, 2026 03:18
@eriknw
eriknw force-pushed the 16-asv-benchmarks branch from 871707e to 48c0a37 Compare August 5, 2026 17:44
@eriknw
eriknw force-pushed the 16-asv-benchmarks branch from 48c0a37 to 0b7a63a Compare August 5, 2026 18:03
@eriknw
eriknw force-pushed the 16-asv-benchmarks branch 2 times, most recently from 2241c0d to f28bd3e Compare August 6, 2026 07:59
@eriknw
eriknw force-pushed the 16-asv-benchmarks branch from f28bd3e to 0a586c9 Compare August 6, 2026 15:39
@eriknw
eriknw force-pushed the 16-asv-benchmarks branch 3 times, most recently from e469e6c to ced8b66 Compare August 6, 2026 20:41
@eriknw
eriknw force-pushed the 16-asv-benchmarks branch from ced8b66 to 80d325e Compare August 7, 2026 02:48
101 benchmarks over the library's hot paths: scalar access (getitem, get and
contains on hit and miss, setitem), index parsing, small-op overhead on
10/100/1000-element objects, large kernels at ~1e6 nnz (mxm, mxv, ewise,
reduce, single-threaded via OMP_NUM_THREADS=1), coo/dense/scipy conversions,
cold import and first-use timings in fresh subprocesses, and repr costs.

`benchmarks/_verify.py` is a standalone checker rather than an asv benchmark:
it imports every module the way asv does and calls each benchmark once across
all parameter combinations. It reports 101 passed, 0 failed, 0 skipped. No
timing numbers are claimed; what this commit establishes is that the
benchmarks exist and run.

`asv.conf.json` uses `//` comments. That is asv's own documented config
format and its loader strips them, but the repo's `check-json` hook is a
strict JSON parser and rejects the file, so that one path is excluded from
the hook and the hook's stale "no JSON files yet" comment is replaced.
Verified with `pre-commit run check-json --all-files`, which passes.

Two alternatives were considered and not taken. Renaming to `asv.conf.jsonc`
needs no exclude and asv 0.6.5 does resolve it (`Config.load` accepts
`.json` and `.jsonc`), but asv is not a pinned dependency here and older
versions look only for the `.json` name, which would fail as "No `asv.conf`
file found". Stripping the comments would delete the only explanation of why
each setting is set as it is. The exclude names the single path, so every
other JSON file is still checked.

asv is deliberately not added to `dev-requirements.txt` or `environment.yml`.
Nothing in the test suite or CI invokes it, and this repo ties dependency
changes to `scripts/check_versions.sh` and the CI version pools, which a
benchmarks-only change should not be editing. The README carries the
`pip install asv` line and the reasoning. CI wiring is also left alone: a
weekly cron on a fixed machine suits asv far better than per-PR runs on
shared runners.

The lint configuration for benchmarks/ rides in this commit because the
directory it configures arrives here: flake8 and ruff ignore B015/B018
(a bare expression IS what an asv benchmark measures) and T201 for the
benchmark CLI, and codespell learns "bu".
@eriknw
eriknw force-pushed the 16-asv-benchmarks branch from 80d325e to dfa59f3 Compare August 7, 2026 05:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant