Skip to content

Support writing V3 manifests and manifest lists#3624

Open
moomindani wants to merge 6 commits into
apache:mainfrom
moomindani:moomindani/v3-manifest-write
Open

Support writing V3 manifests and manifest lists#3624
moomindani wants to merge 6 commits into
apache:mainfrom
moomindani:moomindani/v3-manifest-write

Conversation

@moomindani

Copy link
Copy Markdown

Closes #3620 (part of #1551)

Rationale for this change

Adds the writer side for V3 manifests and manifest lists (the V3 read schemas already exist):

  • ManifestWriterV3: writes manifest entries using the V3 record layout so the V3-only data file fields (first_row_id, referenced_data_file, content_offset, content_size_in_bytes) are actually written — with the default V2 record layout they would silently serialize as null. Data files bound to the V2 layout are rebound to V3.
  • ManifestListWriterV3: implements the spec's First Row ID Assignment: existing first_row_id values are preserved, delete manifests are never assigned one, and unassigned data manifests get a running value starting at the snapshot's first-row-id, advanced by existing_rows_count + added_rows_count. The writer exposes next_row_id so the commit path can update the table's next-row-id (wiring is Support row lineage assignment on commit #3621). Metadata carries first-row-id and format-version: 3, matching the Java ManifestListWriter.V3Writer.

This is writer-level only: nothing produces V3 snapshots yet (that's #3621, with table-level enablement in #3551 / #3622).

Are these changes tested?

Yes, five new tests: a V3 manifest round-trip verifying the V3 data file fields are written (including the V2-to-V3 rebinding path), first-row-id assignment across unassigned/preserved/delete/second-unassigned manifests (asserting [1000, 77, null, 1125] and next_row_id == 1130), and validation that first_row_id is required. All fail without the implementation and pass with it. No new failures in tests/table, tests/catalog, tests/avro.

Are there any user-facing changes?

No user-facing behavior changes; this adds writer capabilities used by upcoming V3 write support.

This pull request and its description were written by Claude Fable 5.

Add ManifestWriterV3 and ManifestListWriterV3. The manifest writer
uses the V3 record layout so the V3-only data file fields
(first_row_id, referenced_data_file, content_offset,
content_size_in_bytes) are written, rebinding V2-layout records when
needed. The manifest list writer implements the spec's First Row ID
Assignment: preserving existing first_row_id values, never assigning
one to delete manifests, and assigning a running value advanced by
existing and added row counts to unassigned data manifests. It also
exposes next_row_id for the commit path to update the table's
next-row-id.

Closes apache#3620
@moomindani
moomindani force-pushed the moomindani/v3-manifest-write branch from 36f5b56 to d161e78 Compare July 8, 2026 12:05
Carry an optional first_row_id through ManifestWriterV3 into the
produced manifest file (used when rewriting manifests whose
first_row_id is known), matching ManifestFiles.newWriter in Java.
Expand tests for parity with TestManifestWriterVersions and
TestManifestListVersions: reader compatibility of V3-written files,
metrics field round-trips, null first_row_id when reading V2 manifest
lists with the V3 layout, and preservation of writer-carried
first_row_id without advancing next_row_id.
Assigning a first-row-id while treating unknown existing/added row
counts as zero would let subsequent manifests overlap row ID ranges.
Raise a clear error instead, matching the reference implementation
which requires the counts. Also drop the always-None value from the
first-row-id-required error message and document the record layout
invariant behind the V3 rebinding.

@abnobdoss abnobdoss left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking up the v3 writer work! The first-row-id assignment lines up with the Java writer nicely; I left a few comments, mostly on the read version handling.

Comment thread pyiceberg/manifest.py Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per the comment on _wrap_data_file: we likely need to go through all the uses of DEFAULT_READ_VERSION, including this one. Reading a v3 manifest list here drops first_row_id, which is what makes a round trip fail. fetch_manifest_entry has the same issue for the new data file fields.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed by adding LATEST_MANIFEST_ENTRY_READ_VERSION / LATEST_MANIFEST_LIST_READ_VERSION (derived from max() of the respective schema dicts, so they stay in sync as new format versions are added) and switching fetch_manifest_entry / read_manifest_list to read with these instead of DEFAULT_READ_VERSION. Reading an older file with the latest schema is safe since the added fields are all optional, so Avro schema resolution fills them with None. DEFAULT_READ_VERSION itself is unchanged and still used for constructing/writing records, so this only touches the two read call sites. Also added the corresponding V3 properties on DataFile (first_row_id, referenced_data_file, content_offset, content_size_in_bytes) so the values are actually exposed to callers, and extended the round-trip test to assert on their actual values rather than just presence.

Comment thread pyiceberg/manifest.py Outdated
so that is the layout to zip the positional data against.
"""
if len(manifest_file._data) >= len(MANIFEST_LIST_FILE_SCHEMAS[3].fields):
return copy(manifest_file)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copy(manifest_file) doesn't seem to work in its current form: the copy and the original share the same underlying _data list, so writing to one writes to both.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — ManifestListWriterV3._wrap now always rebinds through from_args instead of copy(), so mutating the returned manifest file's first_row_id never leaks back into the caller's record.

One thing to flag separately: ManifestListWriterV2.prepare_manifest has the same shallow-copy sharing bug (copy(manifest_file) there also shares _data). I didn't touch it since it's pre-existing and out of scope for this PR, but wanted to make sure it's on your radar — happy to file a follow-up issue if useful.

Comment thread pyiceberg/manifest.py Outdated
if len(data_file._data) >= len(DATA_FILE_TYPE[3].fields):
return data_file
args = {
field.name: value for field, value in zip(DATA_FILE_TYPE[DEFAULT_READ_VERSION].fields, data_file._data, strict=True)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be DEFAULT_READ_VERSION? It assumes any record that isn't v3 is v2, but DATA_FILE_TYPE has 15 fields for v1, 16 for v2 and 20 for v3, so a v1-bound DataFile would fail the strict zip with a confusing error.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed by adding _layout_version_from_field_count, which recovers the source layout from the record's field count (each version's layout currently has a distinct field count) instead of assuming V2.

I did think about how this holds up for a future V4: if V4 only adds fields (the pattern so far), field-count inference keeps working automatically — no code change needed beyond adding the V4 schema entry, same as how LATEST_MANIFEST_ENTRY_READ_VERSION / LATEST_MANIFEST_LIST_READ_VERSION (see the other thread) are derived via max() rather than hardcoded. But if V4 ever lands with the same field count as an existing version, or changes a field non-additively, field-count inference can no longer tell them apart — so I added an explicit check that raises Ambiguous layout in that case instead of silently binding to the wrong one. That failure mode felt important to guard against explicitly rather than leave implicit.

One thing I'd like your take on: this is still inference, not the record actually knowing its own version. The more principled fix would have Record/DataFile/ManifestFile carry the format version they were bound with, so rebinding wouldn't need to guess at all — that's how Java's manifest records work, since they're self-describing rather than positional. I considered that, but went with field-count inference for now since it's consistent with the existing code's approach (the pre-existing early-return already used a field-count threshold) and, after switching reads to the latest schema, most records reaching this rebind path are already V3-shaped — it's now mostly a defensive path for records constructed programmatically at an older version, not a hot path. Changing Record's core design to carry its own version felt like the right call for a dedicated follow-up, not something to fold into this PR's scope, but let me know if you'd weigh that differently.

Reading V2/V3 manifests and manifest lists used a fixed V2 read schema,
so V3-only fields (first_row_id, referenced_data_file, content_offset,
content_size_in_bytes) were silently dropped by Avro schema resolution,
breaking V3 round trips. Read with the latest known schema instead,
keeping the V2 default for constructing and writing records unchanged.

Also fixes ManifestListWriterV3._wrap sharing _data with the source
record via a shallow copy, and rebinding a record to the V3 layout
against a fixed V2 field count, which raised on V1-bound records.
max() and list indexing on the version-keyed schema dicts widen the
Literal[1, 2, 3] TableVersion back to int, so mypy flags the results
as incompatible with the declared TableVersion return types.
test_rest_manifest.py has its own todict helper (separate from
tests/avro/test_file.py's), which also enumerates every DataFile
property via reflection. Adding V3-only properties to DataFile in
the previous commit caused this helper to include them for a
V2-shaped record too, diverging from the V2 dict fastavro writes and
failing the REST integration round-trip test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support writing V3 manifests and manifest lists

2 participants