Skip to content

[connectors] Delta snapshot read fails on UC-Uniform-over-Iceberg tables #6557

Description

@swanandx

Describe the bug

Reading a Unity Catalog Uniform table (Iceberg files exposed as Delta, columnMapping.mode=id) fails the initial snapshot read:

error retrieving initial snapshot query 'select "after","op" from snapshot':
Execution("Non-nullable column 'col-102' is missing from the physical schema")

Root cause

Under mode=id, Delta refers to each column by physicalName = col-. But data files written by a native Iceberg writer (Flink/pyiceberg) use logical names (op, after) + a Parquet field_id. delta-rs's "next" scan provider resolves columns by physical name (DefaultPhysicalExprAdapter), so col-102 is absent from every file -> missing -> hard error (non-nullable) or silent NULL-fill (nullable). The field_id that would resolve it is present on both sides but unused. No schema evolution involved.

Confirmed (file inspection of a UC-Uniform/pyiceberg table)

  • Parquet: required binary field_id=8 op — logical name, nullable=false.
  • UC-generated Delta log (engineInfo: Kernel-…/Iceberg REST Catalog) stats address it as col-8.
  • Feldera -> Non-nullable column 'col-8' is missing from the physical schema. (field_id 8 → col-8 mirrors customer's 102 → col-102.)
  • Databricks-engine (USING iceberg) tables write col-N on disk and are unaffected — the writer is the only variable.

Impact

  • Non-nullable missing column -> hard read failure. Nullable -> silently NULL (data-correctness risk).
  • Affects UC Uniform over Flink/pyiceberg/Spark-Iceberg. Native Delta + Databricks-managed Iceberg unaffected.

Fix ( required in delta-rs, not in feldera )

Resolve column-mapped reads by field-id when present, fall back to name (fork patch on delta-rs, worth upstreaming):

  1. When column_mapping_mode == Id, propagate delta.columnMapping.id onto the Arrow read schema as PARQUET:field_id (currently dropped by try_into_arrow()).
  2. Field-id-aware PhysicalExprAdapterFactory: match by PARQUET:field_id when both sides have it, else by name (delegates unchanged when no field-ids -> native Delta untouched).

Repro: native Iceberg writer (pyiceberg via UC Iceberg REST) + op NOT NULL; or zero-dep local fixture (hand-written _delta_log over logical-name Parquet) for regression.

Metadata

Metadata

Assignees

Labels

connectorsIssues related to the adapters/connectors crate

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions