[connectors] retry loops and parallel parsing for Iceberg snapshots#6623
[connectors] retry loops and parallel parsing for Iceberg snapshots#6623swanandx wants to merge 9 commits into
Conversation
mythical-fred
left a comment
There was a problem hiding this comment.
Nice bring-up of the Iceberg snapshot read path to Delta parity — the retry-around-execute_df + parallel-parse-via-JobQueue split is clean, and hoisting both helpers into adapterlib is the right call now that two connectors need them (the crate-graph note in the commit message is on point). The parser speedup table (1.72× / 2.71× / 4.42× / 6.19× at 2/4/8/16 workers) is convincing, and the manual chmod 000 retry trace matches the code path. Approve. A few non-blocking things worth a look before you merge:
-
max_retriesdoc overstates the scope. Infeldera-types/src/transport/iceberg.rs(also mirrored into the docs page and OpenAPI) the field is described as covering "opening the table or reading a Parquet file", but iniceberg/src/input.rsthe retry loop only wrapsexecute_df—open_table()at line 462 is called exactly once with no wrapper. That is defensible (opening the table is fail-fast and returns aControllerError, not a stream failure), but the doc should say so. Either extend the retry toopen_tabletoo or narrow the wording to "reading the snapshot data-frame". -
Backoff sleep is not cancellable. With
max_retriesdefaulting tou32::MAXand a 32s cap, a stuck object store can leave the reader insleep(backoff_delay).awaitwhile the controller is trying to shut the pipeline down. Wrap the sleep in aselect!withreceiver.changed()(or check forPipelineState::Terminated) so shutdown does not stall for up to 32s per attempt. Delta has the same shape, so if you fix it here, worth doing there too. -
OpenAPI
minimumfornum_parsers. The generated schema emits"minimum": 0, butnum_parsers == 0is rejected at construction (correctly). Consider#[schema(minimum = 1)]or equivalent so the API doc matches runtime validation — same for the recommended upper bound if you want to advertise it. -
Double fork in the parser closure. In
execute_df_inner(input.rs ~1030),worker_buildercallsinput_stream.fork()once per worker, and then the innerFnMutcallsinput_stream.fork()again on every job. Ifparse_record_batchdrains state viatake_all()per call, the inner fork looks redundant. Might be defensive, but worth a comment saying so — or drop it if it truly is redundant. -
Plural mismatch in the retry log.
"error retrieving {descr} after 1 attempts"— cosmetic, but easy to fix ("attempt"forretry_count == 1). -
Changelog checkbox is unchecked in the PR body. A one-line entry under Unreleased for the new
num_parsers/max_retriesoptions would help downstream users notice.
None of these block. Once #1 and #2 are addressed I would ship it.
514e304 to
278a65c
Compare
mythical-fred
left a comment
There was a problem hiding this comment.
Re-review at tip 278a65cc.
Additive delta since my last review addresses the nits I raised plus mihaibudiu's:
max_retriesdoc scope narrowed to "reading the table snapshot" (was "opening the table or reading a Parquet file", which oversold coverage sinceopen_table()is called once outside the retry loop) — mirrored in the config doc, docs page, and OpenAPI description.#[schema(minimum = 1)]added onnum_parsersso the generated OpenAPI schema matches the runtimenum_parsers == 0rejection.- Double-fork in the
JobQueueworker closure documented with a comment explaining the outer-worker vs per-job fork rationale. "after N attempts"->"after N attempt(s)"for the single-attempt case.- Parse error message narrowed to "records read from the Iceberg table" (drops the Parquet-only assumption; Iceberg supports Avro/ORC too — good catch by mihaibudiu).
On the cancellable sleep(backoff_delay) question: reasonable to defer. worker_task at input.rs:295 already wraps worker_task_inner in select! against receiver.wait_for(Terminated), so a stuck sleep inside execute_df is cancelled at the outer level on shutdown. Latency to react is bounded by the outer termination cancelling the whole future tree, not by the sleep itself — the exponential backoff cap of ~32s isn't observable to the controller. I'd still consider making it explicit some day (the pattern-of-two makes reasoning about shutdown a hair harder), but it's not a blocker.
Approving. Nice work turning the review around quickly.
| .update_connector_health(ConnectorHealth::healthy()); | ||
| return Ok(total_records); | ||
| } | ||
| Err(e) => { |
There was a problem hiding this comment.
I know the delta connector works the same way, but we will need to distinguish retryable errors here.
There was a problem hiding this comment.
let's create a separate issue for this? ( as this needs to be fixed in delta as well right? )
327bfa2 to
9f8c37d
Compare
mythical-fred
left a comment
There was a problem hiding this comment.
Nice restructure. The refactor is clean, the retry loop now returns healthy on success and unhealthy on terminal failure, and the transaction commit fires exactly once per successful execute_df regardless of retries. Two small notes inline, neither blocking.
mythical-fred
left a comment
There was a problem hiding this comment.
LGTM — metrics addition is clean. IcebergPhase enum with repr(u64) + Atomic is a nice pattern, Ordering::Relaxed is correct for independent gauges/counters, tests assert on the new metrics, and docs table entries match the code descriptions exactly.
d443caf to
7e05cd7
Compare
mythical-fred
left a comment
There was a problem hiding this comment.
LGTM. Metrics wiring is clean, ordering is Relaxed throughout (correct for counters/gauges), docs and tests updated.
6809309 to
e00b519
Compare
mythical-fred
left a comment
There was a problem hiding this comment.
Re-review at 854b7a1 — one new commit [connectors] retry Iceberg table opens.
Directly addresses the very first nit I raised on the initial review ("extend retry to open_table"). Implementation is careful:
is_retryable_open_errormatches on lower-cased full rate-limit codes/messages (throttlingexception,rate exceeded,slowdown,too many requests,requestlimitexceeded,provisionedthroughputexceeded). No stems, so prose like "not throttled" or "throttling is disabled" won't match.open_table_with_retriesreuses the existingcalculate_backoff_delay+max_retries()accessor for behavior parity with the mid-stream retry loop.- Permanent errors (missing table, bad credentials, wrong region) fail fast — critical because
max_retriesdefaults to unlimited; otherwise a typo in the table name would loop forever. - New unit test
retryable_open_error_matches_only_throttlingcovers a real GlueServiceErrorpayload and four "must not retry" strings including the tricky false-positive prose.
Signed-off, no AI trailer.
APPROVE.
The Iceberg crate can't depend on dbsp_adapters (adapters already depends on feldera-iceberg, so the reverse edge would be a cycle). Move JobQueue and calculate_backoff_delay into feldera_adapterlib::utils so the Delta and Iceberg connectors can share them. No behavior change: the Delta connector re-imports both from adapterlib. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
Bring the Iceberg snapshot read path to parity with Delta: - Retry: execute_df now retries the whole dataframe with exponential backoff on transient object-store failures and reports connector health. Bounded by the new max_retries option (unlimited by default, 0 disables). A mid-stream Parquet read error now triggers a retry instead of being silently skipped, matching Delta. - Parallel parsing: record batches are parsed by a pool of num_parsers tasks (default 4) via the shared JobQueue, which preserves ordering, so buffers reach the input queue in read order. The input queue is wrapped in an Arc so the parser pool's consumer task can push to it. num_parsers == 0 is rejected at config time. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
Add a transaction_mode option (none | snapshot) to the Iceberg connector. In snapshot mode the initial snapshot is ingested as Feldera transactions: the whole snapshot in one transaction when timestamp_column is unset, or one per lateness range when it is set. Each execute_df call maps to one transaction. Parser entries carry the start label (the input queue starts the transaction on the first flushed entry) and a commit entry is pushed once the dataframe drains. This rides the existing non-fault-tolerant queue() path, which already honors the transaction fields on queue entries. Input-buffer staging is deferred: it needs the flush-based FT queue path that follow mode will introduce. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
Add IcebergMetrics (ConnectorMetrics) for the snapshot phase: current phase, snapshot completion time, snapshot record transaction counts, etc. Metrics are registered in IcebergInputReader::new (the open path), not at endpoint construction: the controller inserts the endpoint's status entry after constructing the endpoint but before open, so registering earlier is silently dropped. Extend the iceberg-tests-fs tests to assert the snapshot reaches the completed phase and that transaction_mode=snapshot starts exactly one transaction (unordered) or at least one per lateness range (ordered). Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
Report AtLeastOnce and store a seekable resume point in each checkpoint, following the Delta connector. - Pin the snapshot at the first read, so a resumed read sees the same data even if the table advanced. - Ordered reads (timestamp_column) commit one transaction per lateness range and record the timestamp bound, so a restart re-reads at most the range in flight. - A completed snapshot resumes straight into completion, re-reading nothing. - Stage parsed buffers so parse cost is paid ahead of circuit demand. At-least-once: a retry, or a resume from a range boundary, may re-emit rows. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
Drive the connector through a live pipeline manager, covering what the in-crate tests cannot: - ingest every Iceberg-representable SQL type and check the rows. - checkpoint, suspend and resume; a completed snapshot is not re-read. - ordered snapshot (timestamp_column + LATENESS) ingests as several transactions and every row lands once. Tables are written with pyiceberg and read via metadata_location: local runs use a filesystem warehouse, CI uses the MinIO S3 bucket. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
Iceberg stores UUID as a 16-byte fixed binary. The input deserializer defaulted to the string UUID format, so it failed to decode the bytes and silently dropped every row. Set UuidFormat::Binary in the Iceberg serde config so a UUID column decodes from its fixed binary. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
Cover nested list/map/struct columns (with NULL nested fields), UUID, and nullable scalars, and assert every column round-trips by value instead of spot-checking a few. Fix the connector docs: the type table wrongly listed nested types, uuid, and tz timestamps as unsupported. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
opening table under load can lead to ThrottlingException or other rate limiting failures. we retry open_table in such cases Retry only rate-limit errors, not permanent ones. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
854b7a1 to
e052663
Compare
making progress on #6165
Describe Manual Test Plan
tested locally by generated table:
for failure i did
chmod 000on some files of iceberg data, it did retry as expected:one this to note is that, as it retries whole snapshot, Records Processed were 16,580,608; which is > records in table. That is expected given current fault tolerance of connector.
and for num parsers, i ran with diff config options
Checklist
Breaking Changes?
Should not be breaking change. And moving some utils from adapter -> adapterlib should not be breaking change either ( hopefully, please correct me if wrong )