Skip to content

[connectors] Iceberg connector modernization. #6165

Description

@ryzhyk

This is a meta-issue for Iceberg connector improvements. The initial implementation of the connector mirrored the delta connector at the time (for the snapshot mode only), but the delta connector has received many improvements since. We need to apply these improvements to the Iceberg connector, as well as add support for incremental ingest:

  • Retry loops. No idea if the iceberg crate is better than delta-rs at handling intermittent failures, but it likely requires retries for similar reasons.
  • Parallel parsing. Spawn a configurable number of parallel parser tasks
  • Input buffer staging (stage input buffers before pushing them to the input queue to reduce latency)
  • Transaction support
  • follow and snapshot-and-follow modes
  • Support the skip_unused_columns attribute (currently ignored).

Metadata

Metadata

Assignees

Labels

connectorsIssues related to the adapters/connectors cratemarketingRelevant for marketing content

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions