Skip to content

Commit 4853d7c

Browse files
mfeurermfeurer
authored
Example for study and suite (openml#810)
* fill in simple studies tutorial * add benchmark suites tutorial * add study tutorial * Take Pieter's feedback into account * [skip ci] missed on remark * Take into account Arlind's comments
1 parent ef3e4d1 commit 4853d7c

4 files changed

Lines changed: 275 additions & 7 deletions

File tree

‎examples/20_basic/simple_studies_tutorial.py‎

Lines changed: 0 additions & 7 deletions
This file was deleted.

‎examples/20_basic/simple_suites_tutorial.py‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
"""
2+
================
3+
Benchmark suites
4+
================
5+
6+
This is a brief showcase of OpenML benchmark suites, which were introduced by
7+
`Bischl et al. (2019) <https://arxiv.org/abs/1708.03731v2>`_. Benchmark suites standardize the
8+
datasets and splits to be used in an experiment or paper. They are fully integrated into OpenML
9+
and simplify both the sharing of the setup and the results.
10+
"""
11+
12+
import openml
13+
14+
####################################################################################################
15+
# OpenML-CC18
16+
# ===========
17+
#
18+
# As an example we have a look at the OpenML-CC18, which is a suite of 72 classification datasets
19+
# from OpenML which were carefully selected to be usable by many algorithms and also represent
20+
# datasets commonly used in machine learning research. These are all datasets from mid-2018 that
21+
# satisfy a large set of clear requirements for thorough yet practical benchmarking:
22+
#
23+
# 1. the number of observations are between 500 and 100,000 to focus on medium-sized datasets,
24+
# 2. the number of features does not exceed 5,000 features to keep the runtime of the algorithms
25+
# low
26+
# 3. the target attribute has at least two classes with no class having less than 20 observations
27+
# 4. the ratio of the minority class and the majority class is above 0.05 (to eliminate highly
28+
# imbalanced datasets which require special treatment for both algorithms and evaluation
29+
# measures).
30+
#
31+
# A full description can be found in the `OpenML benchmarking docs
32+
# <https://docs.openml.org/benchmark/#openml-cc18>`_.
33+
#
34+
# In this example we'll focus on how to use benchmark suites in practice.
35+
36+
####################################################################################################
37+
# Downloading benchmark suites
38+
# ============================
39+
40+
suite = openml.study.get_suite(99)
41+
print(suite)
42+
43+
####################################################################################################
44+
# The benchmark suite does not download the included tasks and datasets itself, but only contains
45+
# a list of which tasks constitute the study.
46+
#
47+
# Tasks can then be accessed via
48+
49+
tasks = suite.tasks
50+
print(tasks)
51+
52+
####################################################################################################
53+
# and iterated for benchmarking. For speed reasons we'll only iterate over the first three tasks:
54+
55+
for task_id in tasks[:3]:
56+
task = openml.tasks.get_task(task_id)
57+
print(task)
58+
59+
####################################################################################################
60+
# Further examples
61+
# ================
62+
#
63+
# * `Advanced benchmarking suites tutorial <../30_extended/suites_tutorial.html>`_
64+
# * `Benchmarking studies tutorial <../30_extended/study_tutorial.html>`_
65+
# * `Using studies to compare linear and non-linear classifiers
66+
# <../40_paper/2018_ida_strang_example.html>`_

‎examples/30_extended/study_tutorial.py‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
"""
2+
=================
3+
Benchmark studies
4+
=================
5+
6+
How to list, download and upload benchmark studies.
7+
8+
In contrast to `benchmark suites <https://docs.openml.org/benchmark/#benchmarking-suites>`_ which
9+
hold a list of tasks, studies hold a list of runs. As runs contain all information on flows and
10+
tasks, all required information about a study can be retrieved.
11+
"""
12+
############################################################################
13+
import uuid
14+
15+
import numpy as np
16+
import sklearn.tree
17+
import sklearn.pipeline
18+
import sklearn.impute
19+
20+
import openml
21+
22+
23+
############################################################################
24+
# .. warning:: This example uploads data. For that reason, this example
25+
# connects to the test server at test.openml.org before doing so.
26+
# This prevents the crowding of the main server with example datasets,
27+
# tasks, runs, and so on.
28+
############################################################################
29+
30+
31+
############################################################################
32+
# Listing studies
33+
# ***************
34+
#
35+
# * Use the output_format parameter to select output type
36+
# * Default gives ``dict``, but we'll use ``dataframe`` to obtain an
37+
# easier-to-work-with data structure
38+
39+
studies = openml.study.list_studies(output_format='dataframe', status='all')
40+
print(studies.head(n=10))
41+
42+
43+
############################################################################
44+
# Downloading studies
45+
# ===================
46+
47+
############################################################################
48+
# This is done based on the study ID.
49+
study = openml.study.get_study(123)
50+
print(study)
51+
52+
############################################################################
53+
# Studies also features a description:
54+
print(study.description)
55+
56+
############################################################################
57+
# Studies are a container for runs:
58+
print(study.runs)
59+
60+
############################################################################
61+
# And we can use the evaluation listing functionality to learn more about
62+
# the evaluations available for the conducted runs:
63+
evaluations = openml.evaluations.list_evaluations(
64+
function='predictive_accuracy',
65+
output_format='dataframe',
66+
study=study.study_id,
67+
)
68+
print(evaluations.head())
69+
70+
############################################################################
71+
# Uploading studies
72+
# =================
73+
#
74+
# Creating a study is as simple as creating any kind of other OpenML entity.
75+
# In this examples we'll create a few runs for the OpenML-100 benchmark
76+
# suite which is available on the OpenML test server.
77+
78+
openml.config.start_using_configuration_for_example()
79+
80+
# Very simple classifier which ignores the feature type
81+
clf = sklearn.pipeline.Pipeline(steps=[
82+
('imputer', sklearn.impute.SimpleImputer()),
83+
('estimator', sklearn.tree.DecisionTreeClassifier(max_depth=5)),
84+
])
85+
86+
suite = openml.study.get_suite(1)
87+
# We'll create a study with one run on three random datasets each
88+
tasks = np.random.choice(suite.tasks, size=3, replace=False)
89+
run_ids = []
90+
for task_id in tasks:
91+
task = openml.tasks.get_task(task_id)
92+
run = openml.runs.run_model_on_task(clf, task)
93+
run.publish()
94+
run_ids.append(run.run_id)
95+
96+
# The study needs a machine-readable and unique alias. To obtain this,
97+
# we simply generate a random uuid.
98+
alias = uuid.uuid4().hex
99+
100+
new_study = openml.study.create_study(
101+
name='Test-Study',
102+
description='Test study for the Python tutorial on studies',
103+
run_ids=run_ids,
104+
alias=alias,
105+
benchmark_suite=suite.study_id,
106+
)
107+
new_study.publish()
108+
print(new_study)
109+
110+
111+
############################################################################
112+
openml.config.stop_using_configuration_for_example()

‎examples/30_extended/suites_tutorial.py‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
"""
2+
================
3+
Benchmark suites
4+
================
5+
6+
How to list, download and upload benchmark suites.
7+
8+
If you want to learn more about benchmark suites, check out our
9+
`brief introductory tutorial <../20_basic/simple_suites_tutorial.html>`_ or the
10+
`OpenML benchmark docs <https://docs.openml.org/benchmark/#benchmarking-suites>`_.
11+
"""
12+
############################################################################
13+
import uuid
14+
15+
import numpy as np
16+
17+
import openml
18+
19+
20+
############################################################################
21+
# .. warning:: This example uploads data. For that reason, this example
22+
# connects to the test server at test.openml.org before doing so.
23+
# This prevents the main server from crowding with example datasets,
24+
# tasks, runs, and so on.
25+
############################################################################
26+
27+
28+
############################################################################
29+
# Listing suites
30+
# **************
31+
#
32+
# * Use the output_format parameter to select output type
33+
# * Default gives ``dict``, but we'll use ``dataframe`` to obtain an
34+
# easier-to-work-with data structure
35+
36+
suites = openml.study.list_suites(output_format='dataframe', status='all')
37+
print(suites.head(n=10))
38+
39+
############################################################################
40+
# Downloading suites
41+
# ==================
42+
43+
############################################################################
44+
# This is done based on the dataset ID.
45+
suite = openml.study.get_suite(99)
46+
print(suite)
47+
48+
############################################################################
49+
# Suites also feature a description:
50+
print(suite.description)
51+
52+
############################################################################
53+
# Suites are a container for tasks:
54+
print(suite.tasks)
55+
56+
############################################################################
57+
# And we can use the task listing functionality to learn more about them:
58+
tasks = openml.tasks.list_tasks(output_format='dataframe')
59+
60+
# Using ``@`` in `pd.DataFrame.query <
61+
# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.query.html>`_
62+
# accesses variables outside of the current dataframe.
63+
tasks = tasks.query('tid in @suite.tasks')
64+
print(tasks.describe().transpose())
65+
66+
############################################################################
67+
# Uploading suites
68+
# ================
69+
#
70+
# Uploading suites is as simple as uploading any kind of other OpenML
71+
# entity - the only reason why we need so much code in this example is
72+
# because we upload some random data.
73+
74+
openml.config.start_using_configuration_for_example()
75+
76+
# We'll take a random subset of at least ten tasks of all available tasks on
77+
# the test server:
78+
all_tasks = list(openml.tasks.list_tasks().keys())
79+
task_ids_for_suite = sorted(np.random.choice(all_tasks, replace=False, size=20))
80+
81+
# The study needs a machine-readable and unique alias. To obtain this,
82+
# we simply generate a random uuid.
83+
84+
alias = uuid.uuid4().hex
85+
86+
new_suite = openml.study.create_benchmark_suite(
87+
name='Test-Suite',
88+
description='Test suite for the Python tutorial on benchmark suites',
89+
task_ids=task_ids_for_suite,
90+
alias=alias,
91+
)
92+
new_suite.publish()
93+
print(new_suite)
94+
95+
96+
############################################################################
97+
openml.config.stop_using_configuration_for_example()

0 commit comments

Comments
 (0)