Is your feature request related to a problem? Please describe.
Parquet/ORC files for S3 destination are recommended for Athena/other serverless access patterns due to their compression and columnar format.
CSV and JSON file types require Athena to open the ENTIRE file, parse it all, and then query it to return results. this is expensive and slow.
Describe the solution you'd like
have the output file type parquet or orc available for the S3 destination
Additional context
https://docs.aws.amazon.com/athena/latest/ug/columnar-storage.html
https://pkg.go.dev/github.com/apache/arrow/go/parquet
Is your feature request related to a problem? Please describe.
Parquet/ORC files for S3 destination are recommended for Athena/other serverless access patterns due to their compression and columnar format.
CSV and JSON file types require Athena to open the ENTIRE file, parse it all, and then query it to return results. this is expensive and slow.
Describe the solution you'd like
have the output file type
parquetororcavailable for the S3 destinationAdditional context
https://docs.aws.amazon.com/athena/latest/ug/columnar-storage.html
https://pkg.go.dev/github.com/apache/arrow/go/parquet