perf(fs): avoid redundant object-store metadata requests for known-size files - #189

I am not sure whether introducing a new FileSystem API is acceptable upstream, but...

Summary

This PR adds an Open(path, file_size) path to the object store file system API for callers that already have trusted file metadata.

Paimon split reads already carry the data-file size from manifest metadata. Passing that size through avoids a per-file object metadata request, while retaining Open(path) as the default path for callers without a known size.

This is a generic object store optimization. My WIP OSS(v2) filesystem uses the supplied size to skip its per-file HeadObject request, and the S3 filesystem can use the same known-size opening path.

Benchmark

The benchmark exercises My WIP OSS(v2) filesystem, using the same real Paimon table throughout:

• Location: oss://paimon-cpp-fs/paimon-cpp-demo-20260710-220136/demo.db/events
• 96 ORC data files, about 4.27 MiB each
• Result validation: count(*) = 786,435, sum(quantity) = 39,713,738

The only difference is how data files are opened:

• Baseline: Open(path), which requires one HeadObject request per file.
• Optimized: Open(path, file_size), which uses the file size already present in the Paimon manifest and skips that request.

The largest gain occurs when many known-size files are opened sequentially with little or no reading, which isolates the cost of the per-file metadata request. In an end-to-end DuckDB scan, file reads, decoding, and concurrency reduce the relative impact, but the aggregate query still shows a stable 19.7%–28.4% mean latency reduction on this 96-file table.

End-to-end query

SET threads = <1 or 8>;
ATTACH oss://paimon-cpp-fs/paimon-cpp-demo-20260710-220136/ 
  AS paimon_demo (TYPE paimon, READ_ONLY);

SELECT count(*) AS row_count, sum(quantity) AS quantity_sum
FROM paimon_demo.demo.events;

Raw timings from five independent processes: