Skip to content

Added a test updater tool #6709

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
terryluan12 wants to merge 12 commits into from
Closed

Added a test updater tool #6709

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
0a6ec87
Added initial update_tests.sh script
terryluan12 Jan 11, 2026
0fe8ac2
Updated update_tests.sh
terryluan12 Jan 11, 2026
0a2ea1b
Added conditioning for erroring tests + moved some logic into separat…
terryluan12 Jan 11, 2026
f8ff03e
Added logic for hanging tests
terryluan12 Jan 11, 2026
965c27a
Added checking existing skipped tests + setting custom timeout + chec…
terryluan12 Jan 11, 2026
f6b69b7
Some quality of life changes + bug fixes
terryluan12 Jan 11, 2026
e9265bd
Added local annotations + added initial concurrency + added a limit o…
terryluan12 Jan 11, 2026
28ef3b4
Added a semaphore + limited number of jobs at the same time
terryluan12 Jan 11, 2026
61e485b
Added backing up and reapplying skips
terryluan12 Jan 11, 2026
fdb476d
Added ignored_libraries for multiprocessing and concurrent libraries
terryluan12 Jan 12, 2026
5b9ae5c
When checking skips skipping tests with no skip in them + updated doc…
terryluan12 Jan 12, 2026
db005d4
Rearranged the update_tests.sh to be easier to read + fixed minor bug…
terryluan12 Jan 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Added checking existing skipped tests + setting custom timeout + chec… 
…king erroring but not crashing tests
  • Loading branch information
@terryluan12
terryluan12 committed Jan 11, 2026
commit 965c27a85421fce70d9bfbf3ac27d81c6fc864bf
99 changes: 77 additions & 22 deletions scripts/update_tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,14 @@ Options:
-c/--cpython-path <path> Path to the CPython source tree (older version)
-r/--rpython-path <path> Path to the RustPython source tree (newer version)
-u/--copy-untracked Copy untracked tests only
-s/--check-skipped Check existing skipped tests (must be run separate from updating the tests)
-t/--timeout Set a timeout for a test
-a/--annotate While copying tests, run them and annotate failures dynamically
-h/--help Show this help message and exit

Example Usage: $0 -c ~/cpython -r .
Example Usage:
$0 -c ~/cpython -r .
$0 -r . --check-skipped
EOF
exit 1
}
Expand All @@ -29,6 +33,8 @@ cpython_path=""
rpython_path=""
copy_untracked=false
annotate=false
timeout=300
check_skip_flag=false
libraries=()

while [[ $# -gt 0 ]]; do
Expand All @@ -49,6 +55,14 @@ while [[ $# -gt 0 ]]; do
usage
return
;;
-s|--check-skipped)
check_skip_flag=true
shift
;;
-t|--timeout)
timout="$2"
shift 2
;;
-a|--annotate)
annotate=true
shift
Expand All @@ -63,11 +77,13 @@ done
cpython_path="$cpython_path/Lib/test"
rpython_path="$rpython_path/Lib/test"

update_tests() {
if [[ ${#libraries[@]} -eq 0 ]]; then
libraries=$(find ${cpython_path} -type f -printf "%P\n")
fi
if [[ ${#libraries[@]} -eq 0 ]]; then
libraries=$(find ${cpython_path} -type f -printf "%P\n")
fi


update_tests() {
libraries=$1
for lib in "${libraries[@]}"
do
update_test "$lib"
Expand All @@ -76,39 +92,63 @@ update_tests() {

update_test() {
lib=$1
cpython_file="$cpython_path/$lib"
rpython_file="$rpython_path/$lib"

filename=${lib##*/}
basename=${filename%.py}
clib_path="$cpython_path/$lib"
rlib_path="$rpython_path/$lib"

if files_equal "$cpython_file" "$rpython_file"; then
if files_equal "$clib_path" "$rlib_path"; then
echo "No changes in $lib. Skipping..."
continue
fi

if [[ ! -f "$rpython_file" ]]; then
if [[ ! -f "$rlib_path" ]]; then
echo "Test file $lib missing"
if $copy_untracked; then
echo "Copying $lib ..."
cp "$cpython_file" "$rpython_file"
cp "$clib_path" "$rlib_path"
fi
else
echo "Using lib_updater to update $lib"
./scripts/lib_updater.py --from $cpython_file --to $rpython_file -o $rpython_file
./scripts/lib_updater.py --from $clib_path --to $rlib_path -o $rlib_path
fi


output=$(rustpython $lib 2>&1)
if $annotate; then
annotate_lib $lib $rlib_path
fi
Comment on lines +132 to +138
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Jan 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Missing error handling and variable quoting.

  1. No error handling if lib_updater.py fails - the script continues silently.
  2. Variables passed to annotate_lib should be quoted to handle paths with spaces.
  3. The boolean check $annotate works but [[ $annotate == true ]] is clearer.
Suggested improvements 
     else
         echo "Using lib_updater to update $lib"
-        ./scripts/lib_updater.py --from $clib_path --to $rlib_path -o $rlib_path
+        if ! ./scripts/lib_updater.py --from "$clib_path" --to "$rlib_path" -o "$rlib_path"; then
+            echo "Warning: lib_updater.py failed for $lib" >&2
+        fi
     fi


-    if [[ $annotate && -f "$rlib_path" && $(basename -- "$rlib_path") == test_*.py ]]; then
-        annotate_lib $lib $rlib_path
+    if [[ $annotate == true && -f "$rlib_path" && $(basename -- "$rlib_path") == test_*.py ]]; then
+        annotate_lib "$lib" "$rlib_path"
     fi
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
./scripts/lib_updater.py --from $clib_path --to $rlib_path -o $rlib_path
fi
if [[ $annotate && -f "$rlib_path" && $(basename -- "$rlib_path") == test_*.py ]]; then
annotate_lib $lib $rlib_path
fi
if ! ./scripts/lib_updater.py --from "$clib_path" --to "$rlib_path" -o "$rlib_path"; then
echo "Warning: lib_updater.py failed for $lib" >&2
fi
fi
if [[ $annotate == true && -f "$rlib_path" && $(basename -- "$rlib_path") == test_*.py ]]; then
annotate_lib "$lib" "$rlib_path"
fi
🤖 Prompt for AI Agents 
In @scripts/update_tests.sh around lines 132 - 138, The block calling
lib_updater.py should check its exit status and fail fast: after running
./scripts/lib_updater.py --from $clib_path --to $rlib_path -o $rlib_path, test
the command's return code and log/exit (e.g., non-zero -> echo error and exit 1)
instead of continuing silently; also change the annotate conditional to
explicitly test [[ $annotate == true ]] and quote variables when calling
annotate_lib (use annotate_lib "$lib" "$rlib_path") so paths with spaces are
handled safely.

}

check_skips() {
libraries=$1
for lib in "${libraries[@]}"
do
check_skip "$lib"
done
}

check_skip() {
lib=$1
rlib_path="$rpython_path/$lib"

remove_skips $rlib_path

annotate_lib $lib $rlib_path
}

annotate_lib() {
lib=$1
rlib_path=$2
output=$(rustpython $lib 2>&1)

echo "Annotating $lib"

while ! grep -q "Tests result: SUCCESS" <<< "$output"; do
echo "$lib failing, annotating..."
failed_tests=$(echo "$output" | grep '^FAIL: ' | awk '{print $2}' | sort -u)
readarray -t failed_tests <<< $(echo "$output" | awk '/^(FAIL:|ERROR:)/ {print $2}' | sort -u)

# If the test fails/errors, then expectedFailure it
for test in "${failed_tests[@]}"
do
sed -i "s/^\([[:space:]]*\)def $test(/\1@unittest.expectedFailure # TODO: RUSTPYTHON\n\1def $test(/" "$rpython_file"
add_above_test $rlib_path $test "@unittest.expectedFailure # TODO: RUSTPYTHON"
done

# If the test crashes/hangs, then skip it
Expand All @@ -117,16 +157,13 @@ update_test() {
if grep -q "Timeout" <<< "$output"; then
message="; hanging"
fi
sed -i "s/^\([[:space:]]*\)def $crashing_test(/\1@unittest.skip('TODO: RUSTPYTHON$message')\n\1def $crashing_test(/" "$rpython_file"
add_above_test $rlib_path $crashing_test "@unittest.skip('TODO: RUSTPYTHON$message')"
fi

output=$(rustpython $lib 2>&1)
done
fi
}



files_equal() {
file1=$1
file2=$2
Expand All @@ -138,4 +175,22 @@ rustpython() {
cargo run --release --features encodings,sqlite -- -m test -j 1 -u all --fail-env-changed --timeout 300 -v "$@"
}
Comment on lines +271 to +275
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Jan 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Missing variable quoting and lost skip reason.

  1. $file should be quoted to handle paths with spaces.
  2. The replacement loses any specific skip reason, converting to a generic message. Consider preserving the reason if one exists.
Suggested fix 
 replace_expected_with_skip() {
-    file=$1
-    test_name=$2
-    sed -E "/$RUSTPYTHON_CANONICAL_EX_FAILURE_RE/ { N; /\n\s*def $test_name/ { s/^(\s*)@unittest\.expectedFailure\s+# TODO: RUSTPYTHON/\1@unittest.skip\('TODO: RUSTPYTHON'\)/ } }" -i $file
+    local file=$1
+    local test_name=$2
+    sed -E "/$RUSTPYTHON_CANONICAL_EX_FAILURE_RE/ { N; /\n\s*def $test_name/ { s/^(\s*)@unittest\.expectedFailure\s+# TODO: RUSTPYTHON/\1@unittest.skip('TODO: RUSTPYTHON')/ } }" -i "$file"
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
replace_expected_with_skip() {
file=$1
test_name=$2
sed -E "/$RUSTPYTHON_CANONICAL_EX_FAILURE_RE/ { N; /\n\s*def $test_name/ { s/^(\s*)@unittest\.expectedFailure\s+# TODO: RUSTPYTHON/\1@unittest.skip\('TODO: RUSTPYTHON'\)/ } }" -i $file
}
replace_expected_with_skip() {
local file=$1
local test_name=$2
sed -E "/$RUSTPYTHON_CANONICAL_EX_FAILURE_RE/ { N; /\n\s*def $test_name/ { s/^(\s*)@unittest\.expectedFailure\s+# TODO: RUSTPYTHON/\1@unittest.skip('TODO: RUSTPYTHON')/ } }" -i "$file"
}
🤖 Prompt for AI Agents 
In @scripts/update_tests.sh around lines 271 - 275, The helper function
replace_expected_with_skip should quote the $file argument and preserve any
existing skip reason instead of injecting a generic one; update
replace_expected_with_skip to call sed on "$file" (quoted) and change the
substitution to capture the original inline comment/reason (e.g., the text after
the # marker or after "TODO: RUSTPYTHON") and reuse that captured reason inside
the @unittest.skip(...) replacement so the original reason is preserved while
converting @unittest.expectedFailure to @unittest.skip.


update_tests
add_above_test() {
file=$1
test=$2
line=$3
sed -i "s/^\([[:space:]]*\)def $test(/\1$line\n\1def $test(/" "$file"
}

remove_skips() {
rlib_path=$1
sed -i -E '/^[[:space:]]*@unittest\.skip.*\(["'\'']TODO\s?:\s?RUSTPYTHON.*["'\'']\)/Id' $rlib_path
}

if ! $check_skip_flag; then
echo "Updating Tests"
update_tests $libraries
else
echo "Checking Skips"
check_skips $libraries
fi