Repository navigation
Pin the hash seed in the microbenchmark harness - #9126
Conversation
Settings::default() leaves hash_seed = None, so every benchmark process draws a random str hash secret. That changes globals-dict probe lengths between runs and makes dict/loop/string microbenchmarks swing by 10-17% with no code change. Use a fixed seed so results only move when the code does.
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@fanninpm Thank you! |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Requested in #9010.
Why
benches/microbenchmarks.rsbuilds the interpreter fromSettings::default(), which leaveshash_seed = None. Each benchmark process then picks a random str hash secret. Depending on the seed, the names in the module globals dict collide more or less, so dict, loop, string and compare microbenchmarks move by 10–17% between runs even when the code is the same. Examples:loop_append.py: 2.3 ms ↔ 2.7 ms (build(deps): bump libc from 0.2.189 to 0.2.190 #9010, a libc-only bump)cmp.py: 62.5 ms ↔ 69.6 ms (build(deps): bump libc from 0.2.189 to 0.2.190 #9010, Implement Python 3.15 type alias and variadic metadata #9113)loop_string.py/strings.py: about 843 ↔ 995 µs and 40.7 ↔ 47.6 ms (Reject unsupported native keywords before argument binding #8957, build(deps): bump which from 8.0.5 to 8.0.6 #8978, build(deps): bump bitflags from 2.13.1 to 2.13.2 #8979, build(deps): bump https://github.com/streetsidesoftware/cspell-cli from v10.2.0 to 10.3.0 #9103)Change
Set
settings.hash_seed = Some(0)inbench_rustpython_code. The hash secret is set once per process by the first interpreter, and every RustPython interpreter in this harness uses the same settings, so all microbenchmarks now hash with the same secret on every run.Benchmark code and CPython comparisons are not affected. The first CodSpeed run on this PR can move the affected benchmarks once, since the fixed seed may give different collisions than the random seed in the base run. After that, results should only change when the code does.
benches/execution.rsuses the sameSettings::default()pattern and could be pinned the same way if needed.Comment
@codspeedbot pin the hash seed in benches/execution.rs tooand I'll do it.