Skip to content

Parquet Support #1029

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
PGijsbers merged 18 commits into from
Mar 4, 2021
Merged

Parquet Support #1029

Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
f9f50ac
Store the minio_url from description xml
PGijsbers Feb 18, 2021
049196c
Add minio dependency
PGijsbers Feb 18, 2021
fe3dbe2
Add call for downloading file from minio bucket
PGijsbers Feb 18, 2021
1a5ed88
Allow objects to be located in directories
PGijsbers Feb 18, 2021
3905d66
add parquet equivalent of _get_dataset_arff
PGijsbers Feb 18, 2021
ca27f18
Store parquet alongside arff, if available
PGijsbers Feb 18, 2021
b4ed955
Deal with unknown buckets, fix path expectation
PGijsbers Feb 18, 2021
7cecfb9
Update test to reflect parquet file is downloaded
PGijsbers Feb 18, 2021
a0ab074
Download parquet file through lazy loading
PGijsbers Feb 19, 2021
36ede4d
Load data from parquet if available
PGijsbers Feb 19, 2021
f02eb0f
Update (doc) strings
PGijsbers Feb 22, 2021
7ccbe5e
Cast to signify url is str
PGijsbers Feb 22, 2021
5099c2b
Make cache file path generation extension agnostic
PGijsbers Feb 25, 2021
8d86c2b
Remove return argument
PGijsbers Feb 25, 2021
117c671
Add clear test messages, update minio urls
PGijsbers Mar 1, 2021
56137a1
Debugging on CI with print
PGijsbers Mar 1, 2021
d64257d
Add pyarrow dependency for loading parquet
PGijsbers Mar 1, 2021
2eeb8bf
Remove print
PGijsbers Mar 2, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Deal with unknown buckets, fix path expectation
  • Loading branch information
@PGijsbers
PGijsbers committed Feb 18, 2021
commit b4ed9558fb1d3a6108a8955bc20c31eb6412d09a
20 changes: 12 additions & 8 deletions openml/_api_calls.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,21 +89,25 @@ def _download_minio_file(
destination = pathlib.Path(destination)
parsed_url = urllib.parse.urlparse(source)

# expect path format: /minio/BUCKET/path/to/file.ext
if not str(parsed_url.path).startswith("/minio/"):
raise ValueError(f"Expect start of path of source is '/minio/', found '{parsed_url.path}'")

bucket, object_name = parsed_url.path[len("/minio/") :].split("/", maxsplit=1)
# expect path format: /BUCKET/path/to/file.ext
bucket, object_name = parsed_url.path[1:].split("/", maxsplit=1)
if destination.is_dir():
destination = pathlib.Path(destination, object_name)
if destination.is_file() and not exists_ok:
raise FileExistsError(f"File already exists in {destination}.")

client = minio.Minio(endpoint=parsed_url.netloc, secure=False,)
Comment thread
mfeurer marked this conversation as resolved.
Outdated

file_meta = client.fget_object(
bucket_name=bucket, object_name=object_name, file_path=str(destination),
)
try:
file_meta = client.fget_object(
bucket_name=bucket, object_name=object_name, file_path=str(destination),
)
except minio.error.S3Error as e:
if e.message.startswith("Object does not exist"):
raise FileNotFoundError(f"Object at '{source}' does not exist.") from e
# e.g. permission error, or a bucket does not exist (which is also interpreted as a
# permission error on minio level).
raise FileNotFoundError("Bucket does not exist or is private.") from e
return file_meta
Comment thread
mfeurer marked this conversation as resolved.
Outdated


Expand Down