Skip to content

Modular Encryption #1884

@gatesn

Description

@gatesn

This isn't a priority at the moment, but filing a tracking issue so users can upvote if it's important to them, as well as for us to remain aware of the requirements so as to leave the API holes required to implement this in the future.

See Parquet's modular encryption for more information: https://parquet.apache.org/docs/file-format/data-pages/encryption/

The general implementation in Vortex would be to:

  • Add a set of encryption schemes (algorithm + key metadata) to the FileLayout flatbuffer.
  • Add a pointer into the encryption schemes vector to the Segment flatbuffer, allowing each segment to leverage a different encryption scheme.
  • Layout writers to be configured to store metadata / data with a specific encryption scheme, e.g. StructLayout could be configured to have different encryption per column.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions