Describe the bug
Reading a Unity Catalog Uniform table (Iceberg files exposed as Delta, columnMapping.mode=id) fails the initial snapshot read:
error retrieving initial snapshot query 'select "after","op" from snapshot':
Execution("Non-nullable column 'col-102' is missing from the physical schema")
Root cause
Under mode=id, Delta refers to each column by physicalName = col-. But data files written by a native Iceberg writer (Flink/pyiceberg) use logical names (op, after) + a Parquet field_id. delta-rs's "next" scan provider resolves columns by physical name (DefaultPhysicalExprAdapter), so col-102 is absent from every file -> missing -> hard error (non-nullable) or silent NULL-fill (nullable). The field_id that would resolve it is present on both sides but unused. No schema evolution involved.
Confirmed (file inspection of a UC-Uniform/pyiceberg table)
- Parquet: required binary field_id=8 op — logical name, nullable=false.
- UC-generated Delta log (engineInfo: Kernel-…/Iceberg REST Catalog) stats address it as col-8.
- Feldera -> Non-nullable column 'col-8' is missing from the physical schema. (field_id 8 → col-8 mirrors customer's 102 → col-102.)
- Databricks-engine (USING iceberg) tables write col-N on disk and are unaffected — the writer is the only variable.
Impact
- Non-nullable missing column -> hard read failure. Nullable -> silently NULL (data-correctness risk).
- Affects UC Uniform over Flink/pyiceberg/Spark-Iceberg. Native Delta + Databricks-managed Iceberg unaffected.
Fix ( required in delta-rs, not in feldera )
Resolve column-mapped reads by field-id when present, fall back to name (fork patch on delta-rs, worth upstreaming):
- When column_mapping_mode == Id, propagate delta.columnMapping.id onto the Arrow read schema as PARQUET:field_id (currently dropped by try_into_arrow()).
- Field-id-aware PhysicalExprAdapterFactory: match by PARQUET:field_id when both sides have it, else by name (delegates unchanged when no field-ids -> native Delta untouched).
Repro: native Iceberg writer (pyiceberg via UC Iceberg REST) + op NOT NULL; or zero-dep local fixture (hand-written _delta_log over logical-name Parquet) for regression.
Describe the bug
Reading a Unity Catalog Uniform table (Iceberg files exposed as Delta, columnMapping.mode=id) fails the initial snapshot read:
Root cause
Under mode=id, Delta refers to each column by physicalName = col-. But data files written by a native Iceberg writer (Flink/pyiceberg) use logical names (op, after) + a Parquet field_id. delta-rs's "next" scan provider resolves columns by physical name (DefaultPhysicalExprAdapter), so col-102 is absent from every file -> missing -> hard error (non-nullable) or silent NULL-fill (nullable). The field_id that would resolve it is present on both sides but unused. No schema evolution involved.
Confirmed (file inspection of a UC-Uniform/pyiceberg table)
Impact
Fix ( required in delta-rs, not in feldera )
Resolve column-mapped reads by field-id when present, fall back to name (fork patch on delta-rs, worth upstreaming):
Repro: native Iceberg writer (pyiceberg via UC Iceberg REST) + op NOT NULL; or zero-dep local fixture (hand-written _delta_log over logical-name Parquet) for regression.