Skip to content

Add list length min-max statistics#8886

Draft
mhk197 wants to merge 2 commits into
developfrom
mk/list-shape-stats
Draft

Add list length min-max statistics#8886
mhk197 wants to merge 2 commits into
developfrom
mk/list-shape-stats

Conversation

@mhk197

@mhk197 mhk197 commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Rationale for this change

Predicates over list_length() currently lack zone-level length bounds, so they cannot eliminate zones without evaluating list offsets or sizes. Zoned statistics bind aggregates directly to root columns, so this adds one fused adapter for the minimum and maximum of the derived list lengths.

What changes are included in this PR?

  • Add ListLengthMinMax, returning nullable {min_length, max_length} bounds for variable-length lists.
  • Ignore null outer lists while allowing valid empty lists to contribute length zero.
  • Register ListLengthMinMax as a default zoned aggregate for variable-length lists.
  • Rewrite list_length() minimum and maximum statistics to the corresponding fields.
  • Derive fixed-size-list bounds directly from the static list size without storing an aggregate.
  • Add aggregate, rewrite, default-selection, and zone-pruning tests.

Validation:

  • cargo nextest run -p vortex-array -p vortex-layout (3,203 passed, 1 skipped)
  • cargo clippy --all-targets --all-features
  • cargo +nightly fmt --all
  • git diff --check

What APIs are changed? Are there any user-facing changes?

This adds the public ListLengthMinMax aggregate and its min_length and max_length result-field constants. Newly written zoned layouts for variable-length lists include these length bounds, enabling list_length() predicate pruning. Existing APIs are otherwise unchanged.

@mhk197
Add list shape statistics
dbba71d 
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
@mhk197 mhk197 added the changelog/performance A performance improvement label Jul 21, 2026
@codspeed-hq

codspeed-hq Bot commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown

Merging this PR will degrade performance by 19.09%

❌ 4 regressed benchmarks
✅ 1831 untouched benchmarks
⏩ 46 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Simulation copy_nullable[65536] 1 ms 1.4 ms -24.17%
❌ Simulation decode_varbin[(1000, 8)] 26.3 µs 34 µs -22.75%
❌ Simulation copy_non_nullable[65536] 913.5 µs 1,094.5 µs -16.53%
❌ Simulation fsl_large 87.5 µs 99.8 µs -12.35%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

Comparing mk/list-shape-stats (4c3a191) with develop (b7ee873)

Open in CodSpeed

Footnotes

  1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@github-actions

github-actions Bot commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done 4c3a191 1 Explore Profiling Data
Previous Runs (1)
Status Commit Job Attempt Link
🟢 Done dbba71d 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

github-actions Bot commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +4.9%
Engines: DataFusion No clear signal (+3.4%, low confidence) · DuckDB No clear signal (+6.3%, low confidence)
Vortex (geomean): 1.064x ➖
Parquet (geomean): 1.039x ➖
Shifts: Parquet (control) +3.9% · Median polish +6.9%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.
datafusion / vortex-file-compressed (1.088x ➖, 0↑ 0↓)
name PR 4c3a191 (ns) base b7ee873 (ns) ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed 10265526 9472638 1.08
vortex_q01/datafusion:vortex-file-compressed 6492790 5947344 1.09
datafusion / parquet (1.051x ➖, 0↑ 0↓)
name PR 4c3a191 (ns) base b7ee873 (ns) ratio (PR/base)
vortex_q00/datafusion:parquet 20888925 20526611 1.02
vortex_q01/datafusion:parquet 4994100 4597130 1.09
duckdb / vortex-file-compressed (1.092x ➖, 0↑ 1↓)
name PR 4c3a191 (ns) base b7ee873 (ns) ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed 10782426 10002024 1.08
vortex_q01/duckdb:vortex-file-compressed 🚨 6452661 5833329 1.11
duckdb / parquet (1.027x ➖, 0↑ 0↓)
name PR 4c3a191 (ns) base b7ee873 (ns) ratio (PR/base)
vortex_q00/duckdb:parquet 23871002 23225002 1.03
vortex_q01/duckdb:parquet 9569102 9318548 1.03

No file size changes detected.

@mhk197
Use a fused list length min-max statistic
4c3a191 
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
@mhk197 mhk197 changed the title Add list shape statistics Add list length min-max statistics Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

changelog/performance A performance improvement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mhk197