Support writing V3 manifests and manifest lists#3624
Conversation
Add ManifestWriterV3 and ManifestListWriterV3. The manifest writer uses the V3 record layout so the V3-only data file fields (first_row_id, referenced_data_file, content_offset, content_size_in_bytes) are written, rebinding V2-layout records when needed. The manifest list writer implements the spec's First Row ID Assignment: preserving existing first_row_id values, never assigning one to delete manifests, and assigning a running value advanced by existing and added row counts to unassigned data manifests. It also exposes next_row_id for the commit path to update the table's next-row-id. Closes apache#3620
36f5b56 to
d161e78
Compare
Carry an optional first_row_id through ManifestWriterV3 into the produced manifest file (used when rewriting manifests whose first_row_id is known), matching ManifestFiles.newWriter in Java. Expand tests for parity with TestManifestWriterVersions and TestManifestListVersions: reader compatibility of V3-written files, metrics field round-trips, null first_row_id when reading V2 manifest lists with the V3 layout, and preservation of writer-carried first_row_id without advancing next_row_id.
Assigning a first-row-id while treating unknown existing/added row counts as zero would let subsequent manifests overlap row ID ranges. Raise a clear error instead, matching the reference implementation which requires the counts. Also drop the always-None value from the first-row-id-required error message and document the record layout invariant behind the V3 rebinding.
abnobdoss
left a comment
There was a problem hiding this comment.
Thanks for picking up the v3 writer work! The first-row-id assignment lines up with the Java writer nicely; I left a few comments, mostly on the read version handling.
There was a problem hiding this comment.
Per the comment on _wrap_data_file: we likely need to go through all the uses of DEFAULT_READ_VERSION, including this one. Reading a v3 manifest list here drops first_row_id, which is what makes a round trip fail. fetch_manifest_entry has the same issue for the new data file fields.
There was a problem hiding this comment.
Fixed by adding LATEST_MANIFEST_ENTRY_READ_VERSION / LATEST_MANIFEST_LIST_READ_VERSION (derived from max() of the respective schema dicts, so they stay in sync as new format versions are added) and switching fetch_manifest_entry / read_manifest_list to read with these instead of DEFAULT_READ_VERSION. Reading an older file with the latest schema is safe since the added fields are all optional, so Avro schema resolution fills them with None. DEFAULT_READ_VERSION itself is unchanged and still used for constructing/writing records, so this only touches the two read call sites. Also added the corresponding V3 properties on DataFile (first_row_id, referenced_data_file, content_offset, content_size_in_bytes) so the values are actually exposed to callers, and extended the round-trip test to assert on their actual values rather than just presence.
| so that is the layout to zip the positional data against. | ||
| """ | ||
| if len(manifest_file._data) >= len(MANIFEST_LIST_FILE_SCHEMAS[3].fields): | ||
| return copy(manifest_file) |
There was a problem hiding this comment.
copy(manifest_file) doesn't seem to work in its current form: the copy and the original share the same underlying _data list, so writing to one writes to both.
There was a problem hiding this comment.
Fixed — ManifestListWriterV3._wrap now always rebinds through from_args instead of copy(), so mutating the returned manifest file's first_row_id never leaks back into the caller's record.
One thing to flag separately: ManifestListWriterV2.prepare_manifest has the same shallow-copy sharing bug (copy(manifest_file) there also shares _data). I didn't touch it since it's pre-existing and out of scope for this PR, but wanted to make sure it's on your radar — happy to file a follow-up issue if useful.
| if len(data_file._data) >= len(DATA_FILE_TYPE[3].fields): | ||
| return data_file | ||
| args = { | ||
| field.name: value for field, value in zip(DATA_FILE_TYPE[DEFAULT_READ_VERSION].fields, data_file._data, strict=True) |
There was a problem hiding this comment.
Should this be DEFAULT_READ_VERSION? It assumes any record that isn't v3 is v2, but DATA_FILE_TYPE has 15 fields for v1, 16 for v2 and 20 for v3, so a v1-bound DataFile would fail the strict zip with a confusing error.
There was a problem hiding this comment.
Fixed by adding _layout_version_from_field_count, which recovers the source layout from the record's field count (each version's layout currently has a distinct field count) instead of assuming V2.
I did think about how this holds up for a future V4: if V4 only adds fields (the pattern so far), field-count inference keeps working automatically — no code change needed beyond adding the V4 schema entry, same as how LATEST_MANIFEST_ENTRY_READ_VERSION / LATEST_MANIFEST_LIST_READ_VERSION (see the other thread) are derived via max() rather than hardcoded. But if V4 ever lands with the same field count as an existing version, or changes a field non-additively, field-count inference can no longer tell them apart — so I added an explicit check that raises Ambiguous layout in that case instead of silently binding to the wrong one. That failure mode felt important to guard against explicitly rather than leave implicit.
One thing I'd like your take on: this is still inference, not the record actually knowing its own version. The more principled fix would have Record/DataFile/ManifestFile carry the format version they were bound with, so rebinding wouldn't need to guess at all — that's how Java's manifest records work, since they're self-describing rather than positional. I considered that, but went with field-count inference for now since it's consistent with the existing code's approach (the pre-existing early-return already used a field-count threshold) and, after switching reads to the latest schema, most records reaching this rebind path are already V3-shaped — it's now mostly a defensive path for records constructed programmatically at an older version, not a hot path. Changing Record's core design to carry its own version felt like the right call for a dedicated follow-up, not something to fold into this PR's scope, but let me know if you'd weigh that differently.
Reading V2/V3 manifests and manifest lists used a fixed V2 read schema, so V3-only fields (first_row_id, referenced_data_file, content_offset, content_size_in_bytes) were silently dropped by Avro schema resolution, breaking V3 round trips. Read with the latest known schema instead, keeping the V2 default for constructing and writing records unchanged. Also fixes ManifestListWriterV3._wrap sharing _data with the source record via a shallow copy, and rebinding a record to the V3 layout against a fixed V2 field count, which raised on V1-bound records.
max() and list indexing on the version-keyed schema dicts widen the Literal[1, 2, 3] TableVersion back to int, so mypy flags the results as incompatible with the declared TableVersion return types.
test_rest_manifest.py has its own todict helper (separate from tests/avro/test_file.py's), which also enumerates every DataFile property via reflection. Adding V3-only properties to DataFile in the previous commit caused this helper to include them for a V2-shaped record too, diverging from the V2 dict fastavro writes and failing the REST integration round-trip test.
Closes #3620 (part of #1551)
Rationale for this change
Adds the writer side for V3 manifests and manifest lists (the V3 read schemas already exist):
ManifestWriterV3: writes manifest entries using the V3 record layout so the V3-only data file fields (first_row_id,referenced_data_file,content_offset,content_size_in_bytes) are actually written — with the default V2 record layout they would silently serialize as null. Data files bound to the V2 layout are rebound to V3.ManifestListWriterV3: implements the spec's First Row ID Assignment: existingfirst_row_idvalues are preserved, delete manifests are never assigned one, and unassigned data manifests get a running value starting at the snapshot'sfirst-row-id, advanced byexisting_rows_count + added_rows_count. The writer exposesnext_row_idso the commit path can update the table'snext-row-id(wiring is Support row lineage assignment on commit #3621). Metadata carriesfirst-row-idandformat-version: 3, matching the JavaManifestListWriter.V3Writer.This is writer-level only: nothing produces V3 snapshots yet (that's #3621, with table-level enablement in #3551 / #3622).
Are these changes tested?
Yes, five new tests: a V3 manifest round-trip verifying the V3 data file fields are written (including the V2-to-V3 rebinding path), first-row-id assignment across unassigned/preserved/delete/second-unassigned manifests (asserting
[1000, 77, null, 1125]andnext_row_id == 1130), and validation thatfirst_row_idis required. All fail without the implementation and pass with it. No new failures intests/table,tests/catalog,tests/avro.Are there any user-facing changes?
No user-facing behavior changes; this adds writer capabilities used by upcoming V3 write support.
This pull request and its description were written by Claude Fable 5.