Skip to content

Auto-retry flaky MP tests#7603

Open
ShaharNaveh wants to merge 3 commits intoRustPython:mainfrom
ShaharNaveh:flaky-mp-tests-retry
Open

Auto-retry flaky MP tests#7603
ShaharNaveh wants to merge 3 commits intoRustPython:mainfrom
ShaharNaveh:flaky-mp-tests-retry

Conversation

@ShaharNaveh
Copy link
Copy Markdown
Contributor

@ShaharNaveh ShaharNaveh commented Apr 14, 2026

This is pretty much what we were doing but manually until now:/

Summary by CodeRabbit

  • Chores
    • Improved CI test reliability by adding retry logic for flaky tests (up to 5 attempts), grouping per-attempt logs for easier debugging, and exiting early on a successful attempt.
    • Explicitly set the step shell to bash to ensure consistent execution behavior.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Apr 14, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: c4885f27-d5e7-4a44-bcaf-fd3e6a55c538

📥 Commits

Reviewing files that changed from the base of the PR and between f77e21b and 631fb45.

📒 Files selected for processing (1)
  • .github/workflows/ci.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
  • .github/workflows/ci.yaml

📝 Walkthrough

Walkthrough

The CI workflow step "Run flaky MP CPython tests" was changed to retry the test command up to 5 times in a bash loop, grouping logs per attempt and exiting early on success; the step fails only after all attempts return non‑zero.

Changes

Cohort / File(s) Summary
CI Workflow Retry Logic
​.github/workflows/ci.yaml
Replaced single test invocation with a bash loop that runs the test command up to 5 attempts, wraps each attempt with ::group::.../::endgroup, captures each attempt's exit status, exits early on success, and explicitly sets shell: bash.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Poem

🐰 Hop, hop, I try once more,
Five little runs across the floor,
Grouped logs shining, tidy and bright,
If one succeeds — we end the night,
Flakes beware, I'll test you right!

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Auto-retry flaky MP tests' clearly and concisely summarizes the main change: automating retries for flaky multiprocessing tests, matching the PR's core objective.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/workflows/ci.yaml:
- Line 295: The CI job contains a malformed GitHub Actions group terminator
string (currently echo "::endgroup:) — fix the shell command to use a properly
quoted, correctly-formed terminator (echo "::endgroup::") so Bash parsing and
GitHub Actions group closing work; locate the echo line that references
::endgroup and replace it with a balanced-quoted, double-colon terminator
(::endgroup::) to restore correct behavior.
- Around line 289-300: Fix the shell loop by closing the stray quote on the echo
that ends the group and by fixing the indentation of the command inside the
loop: change the broken echo "::endgroup: to echo "::endgroup::" and ensure the
command invoking target/release/rustpython -m test ... (the line starting with
target/release/rustpython) is indented consistently inside the for-attempt block
so the loop executes correctly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: 7c4ba548-2d50-4bfb-9480-7ad79c1201af

📥 Commits

Reviewing files that changed from the base of the PR and between 2b1b0ba and f77e21b.

📒 Files selected for processing (1)
  • .github/workflows/ci.yaml

target/release/rustpython -m test -j 1 ${{ join(matrix.extra_test_args, ' ') }} --slowest --fail-env-changed --timeout 600 -v ${{ env.FLAKY_MP_TESTS }}
status=$?

echo "::endgroup:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail
python - <<'PY'
from pathlib import Path
line = Path(".github/workflows/ci.yaml").read_text().splitlines()[294]  # Line 295
print(f"Line 295: {line!r}")
if 'echo "::endgroup::"' not in line:
    raise SystemExit("Malformed GitHub Actions group terminator at Line 295.")
print("Line 295 looks valid.")
PY

Repository: RustPython/RustPython

Length of output: 178


Fix malformed GitHub Actions group terminator on Line 295.

Line 295 contains echo "::endgroup: which is syntactically broken. The string is missing a closing quote and the terminator is missing its trailing colon, breaking both Bash parsing and GitHub Actions log group handling.

Suggested fix
-            echo "::endgroup:
+            echo "::endgroup::"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
echo "::endgroup:
echo "::endgroup::"
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/ci.yaml at line 295, The CI job contains a malformed
GitHub Actions group terminator string (currently echo "::endgroup:) — fix the
shell command to use a properly quoted, correctly-formed terminator (echo
"::endgroup::") so Bash parsing and GitHub Actions group closing work; locate
the echo line that references ::endgroup and replace it with a balanced-quoted,
double-colon terminator (::endgroup::) to restore correct behavior.

@youknowone
Copy link
Copy Markdown
Member

@fanninpm what do you think about? Would there be a better idea?

@fanninpm
Copy link
Copy Markdown
Contributor

Have you tried looking at libregrtest's --rerun flag?

@ShaharNaveh
Copy link
Copy Markdown
Contributor Author

ShaharNaveh commented Apr 14, 2026

Have you tried looking at libregrtest's --rerun flag?

I wasn't aware of that flag, that's cool. but from what I can tell it only retry it once? Haven't seen an option to configure the retry count, I go to failed CIs quite a lot tbh, sometimes it takes 3 attempts for the tests to pass:/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants