Skip to content

Added a test updater tool #6709

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
terryluan12 wants to merge 12 commits into from
Closed

Added a test updater tool #6709

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
0a6ec87
Added initial update_tests.sh script
terryluan12 Jan 11, 2026
0fe8ac2
Updated update_tests.sh
terryluan12 Jan 11, 2026
0a2ea1b
Added conditioning for erroring tests + moved some logic into separat…
terryluan12 Jan 11, 2026
f8ff03e
Added logic for hanging tests
terryluan12 Jan 11, 2026
965c27a
Added checking existing skipped tests + setting custom timeout + chec…
terryluan12 Jan 11, 2026
f6b69b7
Some quality of life changes + bug fixes
terryluan12 Jan 11, 2026
e9265bd
Added local annotations + added initial concurrency + added a limit o…
terryluan12 Jan 11, 2026
28ef3b4
Added a semaphore + limited number of jobs at the same time
terryluan12 Jan 11, 2026
61e485b
Added backing up and reapplying skips
terryluan12 Jan 11, 2026
fdb476d
Added ignored_libraries for multiprocessing and concurrent libraries
terryluan12 Jan 12, 2026
5b9ae5c
When checking skips skipping tests with no skip in them + updated doc…
terryluan12 Jan 12, 2026
db005d4
Rearranged the update_tests.sh to be easier to read + fixed minor bug…
terryluan12 Jan 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Added backing up and reapplying skips
  • Loading branch information
@terryluan12
terryluan12 committed Jan 11, 2026
commit 61e485bdccbc11cfe26243d66a935b6d98f3f628
55 changes: 51 additions & 4 deletions scripts/update_tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,17 @@ Options:
Example Usage:
$0 -c ~/cpython -r .
$0 -r . --check-skipped

** Notes:
* When using the update skip functionality
* Updating only looks for files with the format "test_*.py". Everything else (including __init__.py and __main__.py files are ignored)
* Care needs to be taken when updating. All tests with the same name will be updated at the same time
EOF
exit 1
}

if [[ $# -eq 0 ]]; then
usage
usage
fi


Expand Down Expand Up @@ -167,9 +172,11 @@ annotate_lib() {
if grep -q "\.\.\.$" <<< "$output"; then
crashing_test=$(echo "$output" | grep '\.\.\.$' | head -n 1 | awk '{print $1}')
if grep -q "Timeout" <<< "$output"; then
message=" hanging"
hanging=true
else
hanging=false
fi
add_above_test $rlib_path $crashing_test "@unittest.skip('TODO: RUSTPYTHON;$message')"
apply_skip "$rlib_path" "$crashing_test" $hanging
fi

output=$(rustpython $lib 2>&1)
Expand All @@ -179,6 +186,8 @@ annotate_lib() {
break;
fi
done

unset SKIP_BACKUP
}
Comment on lines +201 to +250
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Jan 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Declare and assign separately to avoid masking return values.

The combined declaration and assignment on line 204 masks the exit status of rustpython. If rustpython fails catastrophically, you won't detect it.

Also, the 15-attempt limit is reasonable but the logic could still get stuck if tests alternate between different failure modes.

Suggested fix for SC2155 
 annotate_lib() {
     local lib=${1//\//.}
     local rlib_path=$2
-    local output=$(rustpython $lib 2>&1)
+    local output
+    output=$(rustpython "$lib" 2>&1)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
annotate_lib() {
local lib=${1//\//.}
local rlib_path=$2
local output=$(rustpython $lib 2>&1)
if grep -q "NO TESTS RAN" <<< "$output"; then
echo "No tests ran in $lib. skipping annotation"
return
fi
echo "Annotating $lib"
local attempts=0
while ! grep -q "Tests result: SUCCESS" <<< "$output"; do
((attempts++))
# echo "$lib failing, annotating..."
readarray -t failed_tests <<< "$(echo "$output" | awk '/^(FAIL:|ERROR:)/ {print $2}' | sort -u)"
# If the test fails/errors, then expectedFailure it
for test in "${failed_tests[@]}"
do
if already_failed $rlib_path $test; then
replace_expected_with_skip $rlib_path $test
else
add_above_test $rlib_path $test "$RUSTPYTHON_CANONICAL_EX_FAILURE"
fi
done
# If the test crashes/hangs, then skip it
if grep -q "\.\.\.$" <<< "$output"; then
crashing_test=$(echo "$output" | grep '\.\.\.$' | head -n 1 | awk '{print $1}')
if grep -q "Timeout" <<< "$output"; then
hanging=true
else
hanging=false
fi
apply_skip "$rlib_path" "$crashing_test" $hanging
fi
output=$(rustpython $lib 2>&1)
if [[ $attempts -gt 15 ]]; then
echo "Issue annotating $lib" >&2
return;
fi
done
echo "Successfully updated $lib"
unset SKIP_BACKUP
}
annotate_lib() {
local lib=${1//\//.}
local rlib_path=$2
local output
output=$(rustpython "$lib" 2>&1)
if grep -q "NO TESTS RAN" <<< "$output"; then
echo "No tests ran in $lib. skipping annotation"
return
fi
echo "Annotating $lib"
local attempts=0
while ! grep -q "Tests result: SUCCESS" <<< "$output"; do
((attempts++))
# echo "$lib failing, annotating..."
readarray -t failed_tests <<< "$(echo "$output" | awk '/^(FAIL:|ERROR:)/ {print $2}' | sort -u)"
# If the test fails/errors, then expectedFailure it
for test in "${failed_tests[@]}"
do
if already_failed $rlib_path $test; then
replace_expected_with_skip $rlib_path $test
else
add_above_test $rlib_path $test "$RUSTPYTHON_CANONICAL_EX_FAILURE"
fi
done
# If the test crashes/hangs, then skip it
if grep -q "\.\.\.$" <<< "$output"; then
crashing_test=$(echo "$output" | grep '\.\.\.$' | head -n 1 | awk '{print $1}')
if grep -q "Timeout" <<< "$output"; then
hanging=true
else
hanging=false
fi
apply_skip "$rlib_path" "$crashing_test" $hanging
fi
output=$(rustpython $lib 2>&1)
if [[ $attempts -gt 15 ]]; then
echo "Issue annotating $lib" >&2
return;
fi
done
echo "Successfully updated $lib"
unset SKIP_BACKUP
}
🧰 Tools
🪛 Shellcheck (0.11.0)

[warning] 204-204: Declare and assign separately to avoid masking return values.

(SC2155)

🤖 Prompt for AI Agents 
In @scripts/update_tests.sh around lines 201 - 250, In annotate_lib, avoid
combining declaration and assignment for output (currently local
output=$(rustpython ...)) so the rustpython exit code isn’t masked; instead
declare local output first, run output=$(rustpython $lib 2>&1) and capture its
exit status ($?) immediately, logging and returning on a non-zero catastrophic
failure; also harden the retry loop around attempts by detecting unchanged or
oscillating output (e.g., keep previous_output and if output == previous_output
or it alternates between two states for multiple iterations then abort with an
error) so the while loop can’t hang indefinitely even within the 15-attempt cap.


files_equal() {
Expand Down Expand Up @@ -207,9 +216,45 @@ remove_skips() {

echo "Removing all skips from $rlib_path"

backup_skips "$rlib_path"

sed -i -E '/^[[:space:]]*@unittest\.skip.*\(["'\'']TODO\s?:\s?RUSTPYTHON.*["'\'']\)/Id' $rlib_path
}

apply_skip() {
local rlib_path=$1
local test_name=$2
local hanging=$3
message="unknown"

# Check if the test has a backup skip
if [[ -n "${SKIP_BACKUP[$test_name]}" ]]; then
message="${SKIP_BACKUP[$test_name]//\'/\"}"
echo "Message is $message"
elif $hanging; then
message="hanging"
fi

add_above_test "$rlib_path" "$test_name" "@unittest.skip('TODO: RUSTPYTHON; $message')"
}

backup_skips() {
local rlib_path=$1
declare -gA SKIP_BACKUP=() # global associative array
readarray -t skips < <(grep -E -n "^[[:space:]]*@unittest\.skip.*TODO\s?:\s?RUSTPYTHON" "$rlib_path" | sort -u)

for line in "${skips[@]}"; do
line_num="${line%%:*}"
line_text=$(echo "$line" | grep -oPi "(?<=RUSTPYTHON)\s*[;:]\s*\K(.*)?(?=[\"'])")
next_line=$(sed -n "$((line_num + 1))p" "$rlib_path")

if [[ "$next_line" =~ def[[:space:]]+([a-zA-Z0-9_]+)\( ]]; then
test_name="${BASH_REMATCH[1]}"
SKIP_BACKUP[$test_name]="$line_text"
fi
done
}

if ! $check_skip_flag; then
echo "Updating Tests"

Expand All @@ -220,6 +265,8 @@ if ! $check_skip_flag; then
else
echo "Checking Skips"

readarray -t libraries <<< $(find ${rpython_path} -iname "test_*.py" -type f -printf "%P\n")
if [[ ${#libraries[@]} -eq 0 ]]; then
readarray -t libraries <<< $(find ${rpython_path} -iname "test_*.py" -type f -printf "%P\n")
fi
check_skips "${libraries[@]}"
fi