VarBinViewBuilder: support decoding Dict and Sparse arrays directly into the builder - #9236
Open
robert3005 wants to merge 2 commits into
Open
VarBinViewBuilder: support decoding Dict and Sparse arrays directly into the builder#9236robert3005 wants to merge 2 commits into
robert3005 wants to merge 2 commits into
Conversation
Merging this PR will regress 2 benchmarks
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
robert3005
force-pushed
the
rk/varbinview-gather-scatter
branch
from
August 6, 2026 15:36
fa8eda8 to
4b3b54f
Compare
onursatici
reviewed
Aug 6, 2026
robert3005
force-pushed
the
rk/varbinview-gather-scatter
branch
from
August 6, 2026 22:48
4b3b54f to
470b635
Compare
Dict was the one string encoding with no direct VarBinViewBuilder path: it took the dictionary to full logical length (allocating an intermediate views buffer proportional to the row count) and then append_varbinview_array walked all those views a second time to rebase their buffer indices. Sparse had no append_to_builder override at all and paid the same double pass over its scattered views. - VarBinViewBuilder grows push_buffers (flush-aware, dedup-aware buffer adoption returning the index each buffer landed at) and two bulk view appends built on it: append_views_gathered (one rebased view write per row through an index lookup, null rows skip the lookup) and append_views_scattered (one bulk fill-view write plus one write per patch). - Dict gathers views through its codes straight into the builder: the dictionary's buffers are adopted once — deduplicated across chunks sharing the dictionary — with no byte copy and no intermediate array. - Sparse overrides append_to_builder for strings: the view builder gets the scatter directly, and VarBinBuilder gets an in-order walk of fill runs (append_n_values) and patches, preserving last-wins semantics for duplicate patch indices. Non-string dtypes keep the canonicalize fallback. - execute_varbin_inner no longer pushes a data buffer for a fill value short enough to inline — the view never referenced it, so it was pure dead weight in every canonical decode of a short-filled sparse array. Checks: cargo nextest -p vortex-array -p vortex-sparse; cargo +nightly fmt --all; cargo clippy --all-targets on the touched crates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uet46qdrbQcF61HXRG7EKc Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
force-pushed
the
rk/varbinview-gather-scatter
branch
from
August 6, 2026 23:22
470b635 to
ffeb8fd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We can avoid copies if we support scatter and gather operations on builder which
is essentially what dict and sparse array append_to_builder should be
Stack created with GitHub Stacks CLI • Give Feedback 💬