Expected Behavior
If FileSource.offline_write_batch, write correctly on S3 storage if use s3:// path.
Current Behavior
file_options = feature_view.batch_source.file_options
filesystem, path = FileSource.create_filesystem_and_path(
file_options.uri, file_options.s3_endpoint_override
)
prev_table = pyarrow.parquet.read_table(path, memory_map=True)When read prev_table, it doens't use "filesystem" kwargs
Steps to reproduce
Always happened
Specifications
- Version: 0.29.0
- Platform:
- Subsystem:
Possible Solution
# default filesystem is None, so it works local & s3fs both
prev_table = pyarrow.parquet.read_table(path, filesystem=filesystem, memory_map=True)
Expected Behavior
If FileSource.offline_write_batch, write correctly on S3 storage if use
s3://path.Current Behavior
When read prev_table, it doens't use "filesystem" kwargs
Steps to reproduce
Always happened
Specifications
Possible Solution