Skip to content

[MNT] Diagnose and address long test runtimes (#1633) #1692

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Abhishek9639 wants to merge 3 commits into
base: main
Choose a base branch
from
Open

[MNT] Diagnose and address long test runtimes (#1633) #1692

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
149d23c
[MNT] Diagnose and address long test runtimes (#1633)
Feb 26, 2026
c8fd9a9
Address review feedback: revert CI/conftest changes, improve profile …
Mar 1, 2026
144cee9
Update profile_tests.sh: add -n workers, --dist=load, remove -q
Mar 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address review feedback: revert CI/conftest changes, improve profile … 
…script

- Revert CI workflow to original --durations=20 (no timeout)
- Remove global timeout from pyproject.toml
- Revert conftest.py verify_cache_state scope to function
- Update profile_tests.sh: accept CLI args (-m, -d, -t, -o) with defaults
  • Loading branch information
@Abhishek9639
Abhishek authored and Abhishek9639 committed Apr 7, 2026
commit c8fd9a9c6a81f498c24c8d977c6087078f769538
6 changes: 3 additions & 3 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ jobs:
marks="not production_server"
fi

pytest -n 4 --durations=0 --timeout=600 --dist load -sv $codecov -o log_cli=true -m "$marks"
pytest -n 4 --durations=20 --dist load -sv $codecov -o log_cli=true -m "$marks"

- name: Run tests on Ubuntu Production
if: matrix.os == 'ubuntu-latest'
Expand All @@ -171,14 +171,14 @@ jobs:
marks="production_server"
fi

pytest -n 4 --durations=0 --timeout=600 --dist load -sv $codecov -o log_cli=true -m "$marks"
pytest -n 4 --durations=20 --dist load -sv $codecov -o log_cli=true -m "$marks"

- name: Run tests on Windows
if: matrix.os == 'windows-latest'
env:
OPENML_TEST_SERVER_ADMIN_KEY: ${{ secrets.OPENML_TEST_SERVER_ADMIN_KEY }}
run: | # we need a separate step because of the bash-specific if-statement in the previous one.
pytest -n 4 --durations=0 --timeout=600 --dist load -sv --reruns 5 --reruns-delay 1 -m "not test_server"
pytest -n 4 --durations=20 --dist load -sv --reruns 5 --reruns-delay 1 -m "not test_server"

- name: Upload coverage
if: matrix.code-cov && always()
Expand Down
1 change: 0 additions & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,6 @@ log_level="DEBUG"
testpaths = ["tests"]
minversion = "7.0"
xfail_strict = true
timeout = 600
filterwarnings=[
"ignore:the matrix subclass:PendingDeprecationWarning"
]
Expand Down
44 changes: 34 additions & 10 deletions scripts/profile_tests.sh
View file
Open in desktop
Comment thread
Abhishek9639 marked this conversation as resolved.
Original file line number Diff line number Diff line change
@@ -1,27 +1,51 @@
#!/bin/bash
# Profile test durations to diagnose slow tests (Issue #1633)
# Usage: ./scripts/profile_tests.sh [marker_filter]
#
# Usage: ./scripts/profile_tests.sh [options]
#
# Options:
# -m MARKER Pytest marker filter (default: "not production_server and not test_server")
# -d DURATION Number of slowest durations to show, 0 for all (default: 20)
# -t TIMEOUT Per-test timeout in seconds (default: 300)
# -o OUTPUT Output file path for the report (default: test_durations_report.txt)
#
# Examples:
# ./scripts/profile_tests.sh # non-server tests
# ./scripts/profile_tests.sh "production_server" # production server tests only
# ./scripts/profile_tests.sh "sklearn" # sklearn tests only
# ./scripts/profile_tests.sh
# ./scripts/profile_tests.sh -m "production_server" -d 0 -t 600
# ./scripts/profile_tests.sh -m "sklearn" -o sklearn_report.txt

set -euo pipefail

MARKER_FILTER="${1:-not production_server and not test_server}"
# Default values
MARKER_FILTER="not production_server and not test_server"
DURATIONS=20
TIMEOUT=300
OUTPUT_FILE="test_durations_report.txt"

# Parse command line arguments
while getopts "m:d:t:o:" opt; do
case $opt in
m) MARKER_FILTER="$OPTARG" ;;
d) DURATIONS="$OPTARG" ;;
t) TIMEOUT="$OPTARG" ;;
o) OUTPUT_FILE="$OPTARG" ;;
*) echo "Usage: $0 [-m marker] [-d durations] [-t timeout] [-o output_file]" && exit 1 ;;
esac
done

echo "=== OpenML Test Duration Profiler ==="
echo "Marker filter: $MARKER_FILTER"
echo "Timeout per test: 300s"
echo "Durations to show: $DURATIONS"
echo "Timeout per test: ${TIMEOUT}s"
echo "Output file: $OUTPUT_FILE"
echo ""

pytest \
--durations=0 \
--timeout=300 \
--durations="$DURATIONS" \
--timeout="$TIMEOUT" \
-q \
-m "$MARKER_FILTER" \
2>&1 | tee test_durations_report.txt
2>&1 | tee "$OUTPUT_FILE"
Comment thread
Abhishek9639 marked this conversation as resolved.

echo ""
echo "=== Report saved to test_durations_report.txt ==="
echo "=== Report saved to $OUTPUT_FILE ==="
2 changes: 1 addition & 1 deletion tests/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -276,7 +276,7 @@ def test_apikey_v2() -> str:
return openml.config.get_test_servers()[APIVersion.V2]["apikey"]


@pytest.fixture(autouse=True, scope="module")
@pytest.fixture(autouse=True, scope="function")
def verify_cache_state(test_files_directory) -> Iterator[None]:
assert_static_test_cache_correct(test_files_directory)
yield
Expand Down