This isn't a priority at the moment, but filing a tracking issue so users can upvote if it's important to them, as well as for us to remain aware of the requirements so as to leave the API holes required to implement this in the future.
See Parquet's modular encryption for more information: https://parquet.apache.org/docs/file-format/data-pages/encryption/
The general implementation in Vortex would be to:
- Add a set of encryption schemes (algorithm + key metadata) to the
FileLayout flatbuffer.
- Add a pointer into the encryption schemes vector to the
Segment flatbuffer, allowing each segment to leverage a different encryption scheme.
- Layout writers to be configured to store metadata / data with a specific encryption scheme, e.g. StructLayout could be configured to have different encryption per column.
This isn't a priority at the moment, but filing a tracking issue so users can upvote if it's important to them, as well as for us to remain aware of the requirements so as to leave the API holes required to implement this in the future.
See Parquet's modular encryption for more information: https://parquet.apache.org/docs/file-format/data-pages/encryption/
The general implementation in Vortex would be to:
FileLayoutflatbuffer.Segmentflatbuffer, allowing each segment to leverage a different encryption scheme.