List layout: list-aligned element chunking#8641
Conversation
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Merging this PR will not alter performance
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | bitwise_not_vortex_buffer_mut[128] |
215.3 ns | 244.4 ns | -11.93% |
| ❌ | Simulation | chunked_varbinview_into_canonical[(100, 100)] |
271.6 µs | 306.4 µs | -11.37% |
| ⚡ | Simulation | chunked_varbinview_opt_canonical_into[(1000, 10)] |
206.2 µs | 169.6 µs | +21.57% |
| ⚡ | Simulation | chunked_varbinview_into_canonical[(1000, 10)] |
205.5 µs | 169.3 µs | +21.38% |
| ⚡ | Simulation | chunked_varbinview_opt_into_canonical[(1000, 10)] |
219.6 µs | 183.4 µs | +19.75% |
| ⚡ | Simulation | slice_empty_vortex |
397.8 ns | 339.4 ns | +17.18% |
| ⚡ | Simulation | encode_varbin[(1000, 8)] |
155.4 µs | 140.1 µs | +10.88% |
| ⚡ | Simulation | encode_varbin[(1000, 4)] |
153.7 µs | 139.2 µs | +10.43% |
| ⚡ | Simulation | encode_varbin[(1000, 2)] |
153.2 µs | 138.7 µs | +10.41% |
| ⚡ | Simulation | encode_varbin[(1000, 32)] |
159.1 µs | 144.4 µs | +10.16% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing mk/list-element-chunk-index (3df5ed6) with mk/add-list-layout (ebbac66)
Footnotes
-
4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals ProfilingVortex (geomean): 1.076x ➖ How to read Verdict and Engines
datafusion / vortex-file-compressed (1.076x ➖, 0↑ 2↓)
File Size Changes (1 files changed, +0.0% overall, 1↑ 0↓)
Totals:
|
Benchmarks: TPC-H SF=1 on NVMEVerdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.996x ➖, 0↑ 0↓)
datafusion / parquet (0.975x ➖, 2↑ 0↓)
datafusion / arrow (0.983x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed (0.987x ➖, 0↑ 0↓)
duckdb / parquet (0.985x ➖, 2↑ 1↓)
File Size Changes (17 files changed, -44.3% overall, 6↑ 11↓)
Totals:
|
Benchmarks: FineWeb NVMeVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.894x ✅, 6↑ 0↓)
datafusion / parquet (0.896x ✅, 6↑ 0↓)
duckdb / vortex-file-compressed (0.917x ➖, 3↑ 0↓)
duckdb / parquet (0.926x ➖, 2↑ 0↓)
File Size Changes (3 files changed, -46.3% overall, 1↑ 2↓)
Totals:
|
Benchmarks: TPC-DS SF=1 on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.008x ➖, 0↑ 1↓)
datafusion / parquet (1.008x ➖, 0↑ 5↓)
duckdb / vortex-file-compressed (1.019x ➖, 0↑ 5↓)
duckdb / parquet (1.008x ➖, 1↑ 2↓)
File Size Changes (31 files changed, -43.5% overall, 2↑ 29↓)
Totals:
|
Benchmarks: Clickbench Sorted on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.016x ➖, 0↑ 0↓)
datafusion / parquet (1.012x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.997x ➖, 1↑ 0↓)
duckdb / parquet (1.013x ➖, 0↑ 0↓)
File Size Changes (201 files changed, -42.6% overall, 45↑ 156↓)
Totals:
|
Benchmarks: Statistical and Population GeneticsVerdict: No clear signal (low confidence) How to read Verdict and Engines
duckdb / vortex-file-compressed (0.936x ➖, 2↑ 0↓)
duckdb / parquet (0.990x ➖, 0↑ 0↓)
File Size Changes (3 files changed, -32.3% overall, 1↑ 2↓)
Totals:
|
Benchmarks: FineWeb S3Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.969x ➖, 0↑ 0↓)
datafusion / parquet (1.058x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.963x ➖, 1↑ 0↓)
duckdb / parquet (1.024x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.931x ➖, 6↑ 0↓)
datafusion / parquet (0.941x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed (0.923x ➖, 14↑ 0↓)
duckdb / parquet (0.961x ➖, 0↑ 0↓)
File Size Changes (201 files changed, -39.1% overall, 42↑ 159↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.001x ➖, 0↑ 0↓)
datafusion / parquet (0.999x ➖, 0↑ 0↓)
datafusion / arrow (0.859x ✅, 16↑ 0↓)
duckdb / vortex-file-compressed (0.998x ➖, 0↑ 0↓)
duckdb / parquet (0.997x ➖, 0↑ 0↓)
File Size Changes (47 files changed, -44.4% overall, 15↑ 32↓)
Totals:
|
Benchmarks: TPC-H SF=1 on S3Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.137x ➖, 0↑ 6↓)
datafusion / parquet (1.154x ➖, 1↑ 9↓)
duckdb / vortex-file-compressed (0.974x ➖, 0↑ 0↓)
duckdb / parquet (1.031x ➖, 0↑ 1↓)
|
Adds element chunking to
ListLayoutStrategyviaListLayoutStrategy::with_element_chunk_len(target).Chunks only on list boundaries so no single list straddles two chunks. A selective read then fetches only the element chunks its rows reference:
Stacked on
mk/add-list-layout.