Skip to content

Parquet/ORC files for S3 destination #7220

@Barneyjm

Description

@Barneyjm

Is your feature request related to a problem? Please describe.

Parquet/ORC files for S3 destination are recommended for Athena/other serverless access patterns due to their compression and columnar format.

CSV and JSON file types require Athena to open the ENTIRE file, parse it all, and then query it to return results. this is expensive and slow.

Describe the solution you'd like

have the output file type parquet or orc available for the S3 destination

Additional context

https://docs.aws.amazon.com/athena/latest/ug/columnar-storage.html

https://pkg.go.dev/github.com/apache/arrow/go/parquet

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions