Feature Request / Improvement
When scanning a table with row filter, _open_manifest currently deserializes all live manifest entries via ManifestEntry#fetch_manifest_entry() and then applies partition and metrics filters as a second pass. This materialises the full list of entries before discarding non-matching ones.
The intermediate list and second iteration pass add overhead that scales with total entry count, not matched entry count.
Proposal:
Add ManifestFile#prune_manifest_entry() that applies the partition/metrics evaluation as the entries are deserialized, avoiding the intermediate list allocation for non-matching entries. Update _open_manifest() to use it.
Benchmark results (across multiple tables and filter expression):
| Total Manifest Entries |
Entries Matching Filter |
Two-pass Time |
Fused Time |
Improvement |
| 116k |
78 (0.07%) |
9.13s |
7.18s |
-21.3% |
| 438k |
1 (~0%) |
35.12s |
27.81s |
-20.8% |
| 823k |
4,962 (0.6%) |
54.38s |
41.38s |
-23.9% |
| 18k |
269 (1.5%) |
1.23s |
0.90s |
-27.0% |
| 8k |
185 (2.2%) |
0.57s |
0.40s |
-30.2% |
No behavioral change - correctness validated across all runs.
Feature Request / Improvement
When scanning a table with row filter,
_open_manifestcurrently deserializes all live manifest entries viaManifestEntry#fetch_manifest_entry()and then applies partition and metrics filters as a second pass. This materialises the full list of entries before discarding non-matching ones.The intermediate list and second iteration pass add overhead that scales with total entry count, not matched entry count.
Proposal:
Add
ManifestFile#prune_manifest_entry()that applies the partition/metrics evaluation as the entries are deserialized, avoiding the intermediate list allocation for non-matching entries. Update_open_manifest()to use it.Benchmark results (across multiple tables and filter expression):
No behavioral change - correctness validated across all runs.