Skip to content

[connectors] retry loops and parallel parsing for Iceberg snapshots#6623

Open
swanandx wants to merge 9 commits into
mainfrom
iceberg-6165-modernize-snapshot
Open

[connectors] retry loops and parallel parsing for Iceberg snapshots#6623
swanandx wants to merge 9 commits into
mainfrom
iceberg-6165-modernize-snapshot

Conversation

@swanandx

Copy link
Copy Markdown
Member

making progress on #6165

Describe Manual Test Plan

tested locally by generated table:

python3 crates/iceberg/src/test/create_test_table_s3.py \
  --catalog=sql --warehouse-path=<DIR> --rows=5000000

for failure i did chmod 000 on some files of iceberg data, it did retry as expected:

[iceberg] 2026-07-13T09:42:47.494407Z  WARN feldera_iceberg::input:  iceberg iceberg_test.unnamed-0: error retrieving initial snapshot query 'select * from snapshot' after 1 attempts: error retrieving batch 4079: ..
[iceberg] 2026-07-13T09:43:00.176396Z  WARN feldera_iceberg::input:  iceberg iceberg_test.unnamed-0: error retrieving initial snapshot query 'select * from snapshot' after 2 attempts: error retrieving batch 4079: ..
[iceberg] 2026-07-13T09:43:13.483484Z  WARN feldera_iceberg::input:  iceberg iceberg_test.unnamed-0: error retrieving initial snapshot query 'select * from snapshot' after 3 attempts: error retrieving batch 4079: ..
[iceberg] 2026-07-13T09:43:27.892297Z ERROR dbsp_adapters::server:  FATAL error on input endpoint 'iceberg_test.unnamed-0': error retrieving initial snapshot query 'select * from snapshot' after 4 attempts: error retrieving batch 4079: External(DataInvalid => Failed to open file /tmp/ice/test_table/data/date=2023-02-01/00000-53-6e192283-2721-4667-9234-3447e980d7cc.parquet: Permission denied (os error 13)
)
[iceberg] 2026-07-13T09:43:27.892377Z  INFO feldera_iceberg::input:  iceberg iceberg_test.unnamed-0: finished reading initial snapshot log.target="feldera_iceberg::input" log.module_path="feldera_iceberg::input" log.file="/Users/swanx/work/feldera/crates/iceberg/src/input.rs" log.line=320

one this to note is that, as it retries whole snapshot, Records Processed were 16,580,608; which is > records in table. That is expected given current fault tolerance of connector.

and for num parsers, i ran with diff config options

num_parsers time speedup
1 115.7s 1.00×
2 67.2s 1.72×
4 42.7s 2.71×
8 26.2s 4.42×
16 18.7s 6.19×

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

Should not be breaking change. And moving some utils from adapter -> adapterlib should not be breaking change either ( hopefully, please correct me if wrong )

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice bring-up of the Iceberg snapshot read path to Delta parity — the retry-around-execute_df + parallel-parse-via-JobQueue split is clean, and hoisting both helpers into adapterlib is the right call now that two connectors need them (the crate-graph note in the commit message is on point). The parser speedup table (1.72× / 2.71× / 4.42× / 6.19× at 2/4/8/16 workers) is convincing, and the manual chmod 000 retry trace matches the code path. Approve. A few non-blocking things worth a look before you merge:

  1. max_retries doc overstates the scope. In feldera-types/src/transport/iceberg.rs (also mirrored into the docs page and OpenAPI) the field is described as covering "opening the table or reading a Parquet file", but in iceberg/src/input.rs the retry loop only wraps execute_dfopen_table() at line 462 is called exactly once with no wrapper. That is defensible (opening the table is fail-fast and returns a ControllerError, not a stream failure), but the doc should say so. Either extend the retry to open_table too or narrow the wording to "reading the snapshot data-frame".

  2. Backoff sleep is not cancellable. With max_retries defaulting to u32::MAX and a 32s cap, a stuck object store can leave the reader in sleep(backoff_delay).await while the controller is trying to shut the pipeline down. Wrap the sleep in a select! with receiver.changed() (or check for PipelineState::Terminated) so shutdown does not stall for up to 32s per attempt. Delta has the same shape, so if you fix it here, worth doing there too.

  3. OpenAPI minimum for num_parsers. The generated schema emits "minimum": 0, but num_parsers == 0 is rejected at construction (correctly). Consider #[schema(minimum = 1)] or equivalent so the API doc matches runtime validation — same for the recommended upper bound if you want to advertise it.

  4. Double fork in the parser closure. In execute_df_inner (input.rs ~1030), worker_builder calls input_stream.fork() once per worker, and then the inner FnMut calls input_stream.fork() again on every job. If parse_record_batch drains state via take_all() per call, the inner fork looks redundant. Might be defensive, but worth a comment saying so — or drop it if it truly is redundant.

  5. Plural mismatch in the retry log. "error retrieving {descr} after 1 attempts" — cosmetic, but easy to fix ("attempt" for retry_count == 1).

  6. Changelog checkbox is unchecked in the PR body. A one-line entry under Unreleased for the new num_parsers / max_retries options would help downstream users notice.

None of these block. Once #1 and #2 are addressed I would ship it.

Comment thread crates/feldera-types/src/transport/iceberg.rs
Comment thread crates/iceberg/src/input.rs
Comment thread crates/iceberg/src/input.rs
Comment thread crates/iceberg/src/input.rs Outdated
Comment thread crates/iceberg/src/input.rs
@swanandx
swanandx requested a review from ryzhyk July 13, 2026 15:04
Comment thread crates/adapterlib/src/utils/job_queue.rs
Comment thread crates/adapterlib/src/utils/backoff.rs
Comment thread crates/iceberg/src/input.rs Outdated
Comment thread crates/iceberg/src/input.rs
Comment thread crates/iceberg/src/input.rs Outdated
@swanandx
swanandx force-pushed the iceberg-6165-modernize-snapshot branch from 514e304 to 278a65c Compare July 13, 2026 19:23
@swanandx
swanandx requested a review from mythical-fred July 13, 2026 19:23

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at tip 278a65cc.

Additive delta since my last review addresses the nits I raised plus mihaibudiu's:

  • max_retries doc scope narrowed to "reading the table snapshot" (was "opening the table or reading a Parquet file", which oversold coverage since open_table() is called once outside the retry loop) — mirrored in the config doc, docs page, and OpenAPI description.
  • #[schema(minimum = 1)] added on num_parsers so the generated OpenAPI schema matches the runtime num_parsers == 0 rejection.
  • Double-fork in the JobQueue worker closure documented with a comment explaining the outer-worker vs per-job fork rationale.
  • "after N attempts" -> "after N attempt(s)" for the single-attempt case.
  • Parse error message narrowed to "records read from the Iceberg table" (drops the Parquet-only assumption; Iceberg supports Avro/ORC too — good catch by mihaibudiu).

On the cancellable sleep(backoff_delay) question: reasonable to defer. worker_task at input.rs:295 already wraps worker_task_inner in select! against receiver.wait_for(Terminated), so a stuck sleep inside execute_df is cancelled at the outer level on shutdown. Latency to react is bounded by the outer termination cancelling the whole future tree, not by the sleep itself — the exponential backoff cap of ~32s isn't observable to the controller. I'd still consider making it explicit some day (the pattern-of-two makes reasoning about shutdown a hair harder), but it's not a blocker.

Approving. Nice work turning the review around quickly.

.update_connector_health(ConnectorHealth::healthy());
return Ok(total_records);
}
Err(e) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know the delta connector works the same way, but we will need to distinguish retryable errors here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's create a separate issue for this? ( as this needs to be fixed in delta as well right? )

@swanandx
swanandx force-pushed the iceberg-6165-modernize-snapshot branch from 327bfa2 to 9f8c37d Compare July 15, 2026 10:12

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice restructure. The refactor is clean, the retry loop now returns healthy on success and unhealthy on terminal failure, and the transaction commit fires exactly once per successful execute_df regardless of retries. Two small notes inline, neither blocking.

Comment thread crates/iceberg/src/input.rs
Comment thread crates/iceberg/src/input.rs

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — metrics addition is clean. IcebergPhase enum with repr(u64) + Atomic is a nice pattern, Ordering::Relaxed is correct for independent gauges/counters, tests assert on the new metrics, and docs table entries match the code descriptions exactly.

@swanandx
swanandx force-pushed the iceberg-6165-modernize-snapshot branch from d443caf to 7e05cd7 Compare July 17, 2026 13:43

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Metrics wiring is clean, ordering is Relaxed throughout (correct for counters/gauges), docs and tests updated.

@swanandx
swanandx force-pushed the iceberg-6165-modernize-snapshot branch from 6809309 to e00b519 Compare July 22, 2026 17:52

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 854b7a1 — one new commit [connectors] retry Iceberg table opens.

Directly addresses the very first nit I raised on the initial review ("extend retry to open_table"). Implementation is careful:

  • is_retryable_open_error matches on lower-cased full rate-limit codes/messages (throttlingexception, rate exceeded, slowdown, too many requests, requestlimitexceeded, provisionedthroughputexceeded). No stems, so prose like "not throttled" or "throttling is disabled" won't match.
  • open_table_with_retries reuses the existing calculate_backoff_delay + max_retries() accessor for behavior parity with the mid-stream retry loop.
  • Permanent errors (missing table, bad credentials, wrong region) fail fast — critical because max_retries defaults to unlimited; otherwise a typo in the table name would loop forever.
  • New unit test retryable_open_error_matches_only_throttling covers a real Glue ServiceError payload and four "must not retry" strings including the tricky false-positive prose.

Signed-off, no AI trailer.

APPROVE.

swanandx added 9 commits July 23, 2026 02:53
The Iceberg crate can't depend on dbsp_adapters (adapters already depends
on feldera-iceberg, so the reverse edge would be a cycle). Move JobQueue
and calculate_backoff_delay into feldera_adapterlib::utils so the Delta
and Iceberg connectors can share them.

No behavior change: the Delta connector re-imports both from adapterlib.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
Bring the Iceberg snapshot read path to parity with Delta:

- Retry: execute_df now retries the whole dataframe with exponential
  backoff on transient object-store failures and reports connector health.
  Bounded by the new max_retries option (unlimited by default, 0 disables).
  A mid-stream Parquet read error now triggers a retry instead of being
  silently skipped, matching Delta.
- Parallel parsing: record batches are parsed by a pool of num_parsers
  tasks (default 4) via the shared JobQueue, which preserves ordering, so
  buffers reach the input queue in read order.

The input queue is wrapped in an Arc so the parser pool's consumer task can
push to it. num_parsers == 0 is rejected at config time.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
Add a transaction_mode option (none | snapshot) to the Iceberg
connector. In snapshot mode the initial snapshot is
ingested as Feldera transactions: the whole snapshot in one transaction
when timestamp_column is unset, or one per lateness range when it is set.

Each execute_df call maps to one transaction. Parser entries carry the
start label (the input queue starts the transaction on the first flushed
entry) and a commit entry is pushed once the dataframe drains. This rides
the existing non-fault-tolerant queue() path, which already honors the
transaction fields on queue entries.

Input-buffer staging is deferred: it needs the flush-based FT queue path
that follow mode will introduce.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
Add IcebergMetrics (ConnectorMetrics) for the snapshot phase:
current phase, snapshot completion time, snapshot record
transaction counts, etc.

Metrics are registered in IcebergInputReader::new (the open path), not
at endpoint construction: the controller inserts the endpoint's status
entry after constructing the endpoint but before open, so registering
earlier is silently dropped.

Extend the iceberg-tests-fs tests to assert the snapshot reaches the
completed phase and that transaction_mode=snapshot starts exactly one
transaction (unordered) or at least one per lateness range (ordered).

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
Report AtLeastOnce and store a seekable resume point in each checkpoint,
following the Delta connector.

- Pin the snapshot at the first read, so a resumed read sees the same
  data even if the table advanced.
- Ordered reads (timestamp_column) commit one transaction per lateness
  range and record the timestamp bound, so a restart re-reads at most the
  range in flight.
- A completed snapshot resumes straight into completion, re-reading
  nothing.
- Stage parsed buffers so parse cost is paid ahead of circuit demand.

At-least-once: a retry, or a resume from a range boundary, may re-emit
rows.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
Drive the connector through a live pipeline manager, covering what the
in-crate tests cannot:

- ingest every Iceberg-representable SQL type and check the rows.
- checkpoint, suspend and resume; a completed snapshot is not re-read.
- ordered snapshot (timestamp_column + LATENESS) ingests as several
  transactions and every row lands once.

Tables are written with pyiceberg and read via metadata_location: local
runs use a filesystem warehouse, CI uses the MinIO S3 bucket.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
Iceberg stores UUID as a 16-byte fixed binary. The input deserializer
defaulted to the string UUID format, so it failed to decode the bytes
and silently dropped every row. Set UuidFormat::Binary in the Iceberg
serde config so a UUID column decodes from its fixed binary.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
Cover nested list/map/struct columns (with NULL nested fields), UUID,
and nullable scalars, and assert every column round-trips by value
instead of spot-checking a few. Fix the connector docs: the type table
wrongly listed nested types, uuid, and tz timestamps as unsupported.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
opening table under load can lead to ThrottlingException or other
rate limiting failures. we retry open_table in such cases

Retry only rate-limit errors, not permanent ones.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
@swanandx
swanandx force-pushed the iceberg-6165-modernize-snapshot branch from 854b7a1 to e052663 Compare July 22, 2026 21:23
@swanandx
swanandx enabled auto-merge July 22, 2026 21:23
@swanandx
swanandx added this pull request to the merge queue Jul 22, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants