Skip to content

Read manifests with V3 projection#3690

Open
KaiqiJinWow wants to merge 3 commits into
apache:mainfrom
KaiqiJinWow:fix-v3-manifest-read-projection
Open

Read manifests with V3 projection#3690
KaiqiJinWow wants to merge 3 commits into
apache:mainfrom
KaiqiJinWow:fix-v3-manifest-read-projection

Conversation

@KaiqiJinWow

@KaiqiJinWow KaiqiJinWow commented Jul 21, 2026

Copy link
Copy Markdown

Rationale for this change

V3 manifest schemas are defined, but manifest entries and manifest lists are still read with the V2 projection. Avro schema resolution therefore drops V3-only fields such as first_row_id, referenced_data_file, content_offset, and content_size_in_bytes before they reach DataFile and ManifestFile.

Read manifests with the latest supported projection while retaining the V2 layout as the default for constructors and writers. V1 and V2 manifests remain compatible because their missing V3 fields resolve to null.

This isolates the read-side concern raised in #3624 from its V3 writer work and is a prerequisite for #3478 to consume deletion-vector content ranges from catalog manifests.

Are these changes tested?

Yes.

  • Added V3 manifest-entry round trips covering all four V3 data-file fields.
  • Added a V3 manifest-list round trip covering first_row_id.
  • Extended the V1 and V2 compatibility tests to verify V3 fields resolve to null and are not written into older manifest formats.
  • Ran pytest tests/avro/test_file.py tests/utils/test_manifest.py: 50 passed.
  • Ran the available table test suite: 293 passed; tests requiring unavailable optional datafusion and pyiceberg-core extras were excluded.
  • Ran Ruff format and lint checks for the changed files.
  • Manually validated the read path against an Iceberg 1.10 V3 manifest containing a deletion-vector content range.

Are there any user-facing changes?

No. This fixes internal manifest deserialization and prepares V3 read support without changing behavior for currently supported V1 and V2 tables.

Read manifest entries and manifest lists with the latest supported schema so V3-only fields are retained while older manifests resolve missing fields to null.
Exclude V3-only properties from V1 and V2 fastavro comparisons and clarify the latest read-version constant name.
Exclude V3-only properties from the reflected V2 manifest dictionary used by the REST integration test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant