Skip to content

Parquet Support #1029

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
PGijsbers merged 18 commits into from
Mar 4, 2021
Merged

Parquet Support #1029

Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
f9f50ac
Store the minio_url from description xml
PGijsbers Feb 18, 2021
049196c
Add minio dependency
PGijsbers Feb 18, 2021
fe3dbe2
Add call for downloading file from minio bucket
PGijsbers Feb 18, 2021
1a5ed88
Allow objects to be located in directories
PGijsbers Feb 18, 2021
3905d66
add parquet equivalent of _get_dataset_arff
PGijsbers Feb 18, 2021
ca27f18
Store parquet alongside arff, if available
PGijsbers Feb 18, 2021
b4ed955
Deal with unknown buckets, fix path expectation
PGijsbers Feb 18, 2021
7cecfb9
Update test to reflect parquet file is downloaded
PGijsbers Feb 18, 2021
a0ab074
Download parquet file through lazy loading
PGijsbers Feb 19, 2021
36ede4d
Load data from parquet if available
PGijsbers Feb 19, 2021
f02eb0f
Update (doc) strings
PGijsbers Feb 22, 2021
7ccbe5e
Cast to signify url is str
PGijsbers Feb 22, 2021
5099c2b
Make cache file path generation extension agnostic
PGijsbers Feb 25, 2021
8d86c2b
Remove return argument
PGijsbers Feb 25, 2021
117c671
Add clear test messages, update minio urls
PGijsbers Mar 1, 2021
56137a1
Debugging on CI with print
PGijsbers Mar 1, 2021
d64257d
Add pyarrow dependency for loading parquet
PGijsbers Mar 1, 2021
2eeb8bf
Remove print
PGijsbers Mar 2, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update test to reflect parquet file is downloaded
  • Loading branch information
@PGijsbers
PGijsbers committed Feb 18, 2021
commit 7cecfb91ba3b2037918d68573482771128274881
8 changes: 2 additions & 6 deletions tests/test_datasets/test_dataset_functions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1498,12 +1498,8 @@ def test_data_fork(self):
def test_get_dataset_parquet(self):
dataset = openml.datasets.get_dataset(20)
self.assertIsNotNone(dataset._minio_url)
self.assertIsNone(dataset.data_parquet_file, "The file should not yet be cached")

data, categorical, attribute_names = dataset._load_data()
self.assertIsNotNone(
dataset.data_parquet_file, "Loading data should have stored the parquet file"
)
self.assertIsNotNone(dataset.parquet_file)
self.assertTrue(os.path.isfile(dataset.parquet_file))


@pytest.mark.parametrize(
Expand Down