Skip to content

Speed up tests #977

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
PGijsbers merged 15 commits into from
Oct 29, 2020
Merged

Speed up tests #977

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
9e3fb88
Cache _list_all we don't need the latest list
PGijsbers Oct 27, 2020
bfb5304
Reduce the amount of verified runs
PGijsbers Oct 27, 2020
deddc04
Increase the batch size to avoid more than 2 pages
PGijsbers Oct 27, 2020
480cee0
Mark as test_get_run_trace as skip
PGijsbers Oct 27, 2020
402a2ef
Filter on dataset id serverside
PGijsbers Oct 27, 2020
fd02a9d
Reduce the amount of pages loaded
PGijsbers Oct 27, 2020
a9fd67b
Simplify model tested in swapped parameter test
PGijsbers Oct 27, 2020
7904ef6
Add a cli flag to toggle short/long scenarios
PGijsbers Oct 27, 2020
db26dc9
Skip time measurement on any Windows machine
PGijsbers Oct 27, 2020
bbf76f4
Invoke the --long versions on the COVERAGE job
PGijsbers Oct 27, 2020
6186121
Add long/short versions for some long tests
PGijsbers Oct 27, 2020
63e103b
Check the trace can be retrieved individually
PGijsbers Oct 28, 2020
a7e6b73
Remove old test
PGijsbers Oct 28, 2020
e7f6b3d
Use patch isolate list_all caching to one test
PGijsbers Oct 28, 2020
d85b77a
Fix decorator call
PGijsbers Oct 29, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Reduce the amount of verified runs 
Downloading a run takes a non-significant amount of time (est. 300ms on
my current setup). It is unnecessary to compare against all >=100 runs,
while a handful should do fine (perhaps even just one should do).
  • Loading branch information
@PGijsbers
PGijsbers committed Oct 27, 2020
commit bfb53048a07490095c61af035d1125fdb21bb242
2 changes: 1 addition & 1 deletion tests/test_evaluations/test_evaluation_functions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ def _check_list_evaluation_setups(self, **kwargs):
# Check if output and order of list_evaluations is preserved
self.assertSequenceEqual(evals_setups["run_id"].tolist(), evals["run_id"].tolist())
# Check if the hyper-parameter column is as accurate and flow_id
for index, row in evals_setups.iterrows():
for index, row in evals_setups.head(3).iterrows():
Comment thread
PGijsbers marked this conversation as resolved.
Outdated
params = openml.runs.get_run(row["run_id"]).parameter_settings
list1 = [param["oml:value"] for param in params]
list2 = list(row["parameters"].values())
Expand Down