turbo-persistence: cover block lengths in CRCs - #100322

Summary

Stacked on #100316.

The 8-byte headers of SST blocks and blob files store [uncompressed_length][crc32], but the CRC only covered the payload. A corrupted length therefore passed the checksum check and could:

• make a reader allocate a decompression buffer of up to 4 GB;
• flip whether a block is treated as compressed (0 means uncompressed).

The CRC now covers the four big-endian length bytes followed by the on-disk payload. checksum_block(uncompressed_length, data) hashes both incrementally with crc32fast::Hasher, so no extra copy is made.

Readers verify this checksum before using the length to allocate decompression output or to choose the decompression path. This applies to the mmap and file SST readers (lookups, decoded iteration and the index block), to blob reads and to sst_inspect.

The index block's checksum is now verified before its sentinel check, so a corrupted index length is reported as a checksum error.

The on-disk layout is unchanged, but the meaning of the checksum changes. Caches written by older builds fail validation and are treated as cache corruption.

Iteration still passes medium value blocks to compaction as raw bytes without decompressing them. Their original length and CRC are preserved through the copy, so a corrupted block is detected when the copied value is read. The README documents the new checksum coverage for SST blocks and blob files.

Not covered: a CRC detects accidental corruption, not tampering. There is no cap on the output size, and no compatibility reader for old caches.

Verification

Tests that corrupt only the length header, with offsets left valid:

• SST blocks: lookups and decoded iteration in mmap and file mode, including compressed-to-uncompressed and uncompressed-to-compressed sentinel corruption.
• The compaction raw-copy path.
• Blob reads in mmap and file mode.
• sst_inspect.

Commands run:

• cargo test -p turbo-persistence --all-targets
• cargo test -p turbo-persistence --features strict_checks
• cargo test -p turbo-persistence --no-default-features
• cargo clippy -p turbo-persistence --all-targets, run with the default features, with --no-default-features and with --features strict_checks
• cargo check -p turbo-tasks-backend
• Miri, using the CI flags, on the new length, raw-copy and checksum tests and the existing offset-corruption test.

Before this change, small length corruptions in blob files were caught only by the decompressor, not by the checksum check.