While we would double-check, I'm pretty sure that the PR checks (see .github/workflows/main.yml) can pass even when the tests fail. If this is the case then we need to fix it.

Example: https://github.com/python/pyperformance/pull/163/checks