Skip to content

Add an OptimizeFn to re-pack arrays prior to compression #1798

@gatesn

Description

@gatesn

Examples where this is useful typically involve sliced arrays:

  • DictArray - remove unused values from the dictionary, and also sort values by frequency (better cache locality for decompression)
  • BooleanArray - make sure bools are zero-aligned, instead of starting half way through a byte
  • VarBinView - remove unused strings from the data buffers

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions