Skip unit-test harnesses when building benchmarks by codspeed[bot] · Pull Request #9158 · RustPython/RustPython

codspeed · 2026-10-15T20:00:56Z

Follow-up to #9151, where the CodSpeed job failed in a way unrelated to that PR.

Problem

The CodSpeed workflow often dies in Build the benchmark targets with The runner has received a shutdown signal (exit 143). Over the last ~200 runs, 18 runs failed this way, mostly main pushes, including #9151 and the two most recent main runs. In each case the build was killed while compiling the final rustpython targets.

cargo codspeed build (like cargo bench) selects every target that has bench = true, and both the library and the binary have it by default. As a result, the build makes 4 LTO-linked copies of the interpreter with full debug info, about 350 MB each:

artifact	needed?
`execution` bench	yes
`microbenchmarks` bench	yes
`rustpython` binary (Cargo builds it automatically for benches)	yes
`rustpython` library unit-test harness (`--test`)	no, never run

Each of these LTO compilations takes about 5 GB RSS and several minutes on the 2-core, 16 GB ubuntu-latest runner.

Change

bench = false on [lib] and [[bin]]. Only the criterion benches and the binary they depend on are built now. cargo test and the measured benchmarks are unchanged.

Local measurement

cargo-codspeed 5.0.3, -j 2 (same as CI), dependencies cached, rebuilding the rustpython crate's targets:

	before	after
`cargo codspeed build` wall time	9m 57s	6m 58s (−30%)
LTO interpreter builds	4	3
peak system memory	~10.0 GB	~11.9 GB

The build is now about 3 minutes shorter, so it spends less time near its memory peak. The peak itself did not drop, because the two bench binaries still compile and link at the same time (the difference above is sampling noise). If the shutdowns continue after this change, the next step would be to stop those two links from overlapping, or to move to a larger runner.

coderabbitai · 2026-10-15T20:01:06Z

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: RustPython/RustPython/.coderabbit.yml
Review profile: CHILL
Plan: Advanced
Run ID: bcdfa53b-2c10-4713-a4d1-2100fe6d19d0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

fanninpm · 2026-10-15T20:25:58Z

@youknowone @ShaharNaveh does this make sense?

fanninpm · 2026-10-15T21:03:10Z

NOTE: I think CI should fail if the CodSpeed action didn't roll the same machine as baseline.

codspeed · 2026-10-15T21:04:00Z

NOTE: I think CI should fail if the CodSpeed action didn't roll the same machine as baseline.

I've made that change and pushed it to this PR in 8c41217. I checked the workflow's YAML syntax but haven't run it in CI yet.

What changed in .github/workflows/codspeed.yaml:

A new step right after Name the machine this run drew fails the job when steps.environment.outputs.match == 'false'. It posts an error saying that this runner is not the machine main's baseline was measured on, and that re-running the job will draw another runner.
Because it fails early, the later steps (cache restore, cargo-codspeed install, build, run) are skipped automatically. Their match != 'false' conditions did nothing anymore, so I removed them.
main pushes never fail this way: main records the machine and isn't compared against anything.
I updated the docstring in scripts/codspeed-environment.py to match. The script still exits 0 and only reports match, so the workflow decides what a mismatch costs.

Trade-offs to keep in mind:

PRs can go red by chance. ubuntu-latest assigns runners at random, so some PRs will fail only because they drew different hardware and will need a re-run. The CodSpeed check is probably not a required status, but it will still show a red ❌.
A missing baseline recording also fails the job. If main's machine recording can't be read (cache evicted, or no successful main run yet), PRs will fail until main records it again. Before, these runs were skipped with a warning.
Changes on main itself fail every PR. When main records a different machine, for example after a runner image update with a new rustc or Python, every open PR fails until it's rebased or re-run on matching hardware.

This is unrelated to the build-memory change, so if you'd prefer to keep the PR focused, I can move it to its own PR. Comment @codspeedbot move the runner-mismatch check to a separate PR and I'll do it.

Summary

Warming up the performance wizard...
Make the CodSpeed job fail when the runner differs from main's baseline machine — Make the CodSpeed job fail when the runner differs from main's baseline machine
Validate, commit and push — Pushed 8c41217
Handling comment — Comment handling finished successfully

_{Was this response helpful? React with 👍 or 👎 to help us improve CodSpeedBot.}

fanninpm · 2026-10-15T21:11:50Z

Revert the previous commit. I will iterate on my idea later.

codspeed · 2026-10-15T21:12:45Z

Revert the previous commit. I will iterate on my idea later.

I reverted 8c41217 in f5379b2 and pushed it to this PR. .github/workflows/codspeed.yaml and scripts/codspeed-environment.py are back to how they were in 3cb22c4: a runner that doesn't match main's baseline machine skips the run with a warning again, instead of failing the job. The branch's files are now the same as 3cb22c4, so the only change left in this PR is bench = false on [lib] and [[bin]].

Summary

Warming up the performance wizard...
Handling comment — Comment handling finished successfully

_{Was this response helpful? React with 👍 or 👎 to help us improve CodSpeedBot.}

fanninpm · 2026-10-16T16:07:17Z

Could you please rebase this branch onto a fresh copy of main?

codspeed · 2026-10-16T16:08:19Z

Could you please rebase this branch onto a fresh copy of main?

I rebased the branch onto the latest main (8720306, #9162) and force-pushed it. The new head is e933b95. The rebase had no conflicts.

All three commits are still there: d2d86b8 (bench = false), d55eabe (runner-mismatch check) and e933b95 (its revert). The last two cancel each other out, so compared with main the PR still only changes Cargo.toml, adding bench = false on [lib] and [[bin]].
I checked that the manifest still parses with cargo metadata. I haven't run a build or the benchmarks; CI will run on the new head.

If you'd like a cleaner history, comment @codspeedbot squash this PR into a single commit and I'll drop the commit and its revert.

Summary

Warming up the performance wizard...
Handling comment — Comment handling finished successfully

_{Was this response helpful? React with 👍 or 👎 to help us improve CodSpeedBot.}

fanninpm · 2026-10-16T18:03:21Z

@codspeedbot squash this PR into a single commit

cargo bench and cargo codspeed build select every target with bench = true. That includes the library and binary targets, so the build also compiled the library unit-test harness as an extra LTO-linked copy of the interpreter that no benchmark runs. Mark them bench = false so only the criterion benchmarks (and the binary they depend on) are built.

codspeed · 2026-10-16T18:04:16Z

@codspeedbot squash this PR into a single commit

I squashed the PR into one commit, 3636128 (Skip unit-test harnesses when building benchmarks), and force-pushed it.

The runner-mismatch commit and its revert are gone. The new commit sits directly on main (8720306, Preserve function metadata and annotation callback lifetimes #9162).
The files are exactly the same as the previous head (e933b95). The PR still only adds bench = false to [lib] and [[bin]] in Cargo.toml.
The commit message is the original one from the bench = false commit.

CI will run on the new head.

Summary

Warming up the performance wizard...
Handling comment — Comment handling finished successfully

_{Was this response helpful? React with 👍 or 👎 to help us improve CodSpeedBot.}

chatgpt-codex-connector · 2026-10-17T14:37:23Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-17T14:38:58.835260Z	`3636128`	Draft marked ready
🔒 Security Review	✅ Completed 2026-10-17T14:39:27.385495Z	`3636128`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

codspeed Bot mentioned this pull request Oct 15, 2026

Release SQLite callbacks and expanded trace SQL safely #9151

Merged

1 task done

codspeed Bot force-pushed the codspeed/bench-build-memory branch from f5379b2 to e933b95 Compare October 16, 2026 16:08

codspeed Bot force-pushed the codspeed/bench-build-memory branch from e933b95 to 3636128 Compare October 16, 2026 18:04

fanninpm requested a review from youknowone October 16, 2026 19:45

youknowone marked this pull request as ready for review October 17, 2026 14:37

youknowone merged commit 008b8e1 into main Oct 17, 2026
23 checks passed

youknowone deleted the codspeed/bench-build-memory branch October 17, 2026 14:37

Navigation Menu

Repository navigation

Skip unit-test harnesses when building benchmarks - #9158

Skip unit-test harnesses when building benchmarks#9158
youknowone merged 1 commit into
mainfrom
codspeed/bench-build-memory

codspeed Bot commented Oct 15, 2026

Uh oh!

coderabbitai Bot commented Oct 15, 2026 •

edited

Loading

Review skipped

Uh oh!

fanninpm commented Oct 15, 2026

Uh oh!

fanninpm commented Oct 15, 2026

Uh oh!

codspeed Bot commented Oct 15, 2026 •

edited

Loading

Uh oh!

fanninpm commented Oct 15, 2026

Uh oh!

codspeed Bot commented Oct 15, 2026

Uh oh!

fanninpm commented Oct 16, 2026

Uh oh!

codspeed Bot commented Oct 16, 2026

Uh oh!

fanninpm commented Oct 16, 2026

Uh oh!

codspeed Bot commented Oct 16, 2026

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 17, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

codspeed Bot commented Oct 15, 2026

Problem

Change

Local measurement

Uh oh!

coderabbitai Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

fanninpm commented Oct 15, 2026

Uh oh!

fanninpm commented Oct 15, 2026

Uh oh!

codspeed Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fanninpm commented Oct 15, 2026

Uh oh!

codspeed Bot commented Oct 15, 2026

Uh oh!

fanninpm commented Oct 16, 2026

Uh oh!

codspeed Bot commented Oct 16, 2026

Uh oh!

fanninpm commented Oct 16, 2026

Uh oh!

codspeed Bot commented Oct 16, 2026

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

coderabbitai Bot commented Oct 15, 2026 •

edited

Loading

codspeed Bot commented Oct 15, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 17, 2026 •

edited

Loading