fix(postgres-cdc): gate acks on durability; fix shutdown#6652
Conversation
mythical-fred
left a comment
There was a problem hiding this comment.
Careful, correctness-focused rework of the CDC ack machinery. The DeferredAck split (SnapshotCopyBarrier / Stream{MayDefer,RequireDurable}) and WatcherReceiver::{Fast,Strict} cleanly encode the durability contract that was missing before. The unreachable! guards on the snapshot barrier — plus the explicit #[should_panic] test — are the kind of belt-and-braces I like to see on a fix of this class.
Verdict: COMMENT — the design is right and the tests are substantial. Three items should be resolved before merge (see below); once they are, this is an approve.
Blocking-ish
1. CI is red on main check-run. 5937a719 fails the pre-commit run --all-files step (clippy/fmt). Needs a green build before merge.
2. New integration tests are not #[ignore]d, but the older CDC tests are. Every pre-existing test_cdc_* in crates/adapters/src/integrated/postgres/test.rs has #[ignore] — the new ones (test_cdc_ft_streaming_ack_latency_and_slot_advance, test_cdc_ft_stream_ack_flushes_below_min_batch, test_cdc_ft_accepted_streaming_events_replay_after_restart, both test_cdc_discard_*_on_startup, test_cdc_ft_terminal_catchup_ack_waits_for_checkpoint, test_cdc_ft_snapshot_barrier_covers_buffered_tail) do not. Two concerns:
- These tests hit a live Postgres with
wal_level=logical(the CI job setsPOSTGRES_URL, but pre-existing tests are still ignored — was that just historical, or islogicalnot guaranteed?). If Postgres isn't logical everywhere, this breaks non-CIcargo test. - The tests carry bounded sleeps up to 5s and a
NEGATIVE_OBSERVATION_MS = 10 * STREAMING_ACK_HOLD_MSwindow, run on a shared runner with sidecars.PR_REVIEW_RULES.md§4 (flakiness zero tolerance): please confirm these are stable across many CI runs, or make them#[ignore]like their siblings and run them in a separate manual/nightly lane.
3. Unbounded rollback loop in discard_matching_table_errors.
for table_id in errored_table_ids {
loop { // no iteration bound
let Some(state) = store.get_table_state(table_id).await? else { break; };
if !should_discard_table_error(&state, discard_all) { break; }
store.rollback_table_state(table_id).await ...
}
}The comment says "Errored states can stack, so keep rolling back until something else surfaces." Fair intent, but if a race or bug re-inserts the sentinel error faster than we can drain it, startup blocks forever with no diagnostic. NASA Power of Ten #2 (bounded loops). Cap the inner loop (say, 32) and log a warning if the cap trips.
Design / correctness — verified
- Snapshot terminal barrier is queued behind all snapshot data before being completed (
FelderaDestination::ack_snapshot_copy+DeferredAck::complete()), and cannot beAccepted(enforced statically).test_cdc_ft_snapshot_barrier_covers_buffered_tailrestarts withsend_snapshot: trueand asserts the whole snapshot re-emerges from checkpoint — exactly the property the PR title promises. Good. - Streaming acks:
MayDefer→Acceptedafterstreaming_ack_hold_ms;RequireDurablenever times out. Terminal catchup batch isRequireDurable, so etl cannot advancesync_done/readyuntil Feldera checkpoints.test_cdc_ft_terminal_catchup_ack_waits_for_checkpointcovers this. - FT strict mode watches
total_checkpointed_steps, non-FT fast mode watchestotal_completed_steps. Slot never advances past durable data under FT. Correct. - Shutdown ordering:
shutdown_etl_pipeline()fires beforeabort_completion_watcher()on Terminated and TableErrorMonitor paths, so etl sees shutdown before it sees dropped acks and won't persist the sentinel error. That's the "fix shutdown" half of the title — it holds. - Backwards compatibility: ack semantics change silently, but the old behavior was unsafe (ack could race checkpoint). A compat flag would perpetuate the bug; the right knob is
streaming_ack_hold_ms, which is well-documented.
Config surface — good
streaming_ack_hold_ms = 2000, discard_shutdown_errors = true, discard_table_errors = false. Defaults are durability-first. Validator rejects streaming_ack_hold_ms = 0 with a clear message. All three appear in openapi.json and the docs page. discard_table_errors doc calls out the primary-key requirement — nice touch.
Docs
postgresql-cdc.md is clear and mostly follows ENGLISH.md. Two small suggestions:
- The
streaming_ack_hold_mstable cell is very long. Consider moving the terminal-batches / snapshot-catchup nuance into the "Resume behavior" section and leaving a one-sentence summary in the table. - "Do not manually delete only the replication slot or only the persisted etl state" — good warning, but consider linking it to a Troubleshooting section (even a stub) since operators tend to reach for exactly that when things wedge.
Commits
5ad7d13c fix(postgres-cdc): gate acks on durability; fix shutdown— imperative, signed-off, body explains the why. No AI trailers. Good.5937a719 [ci] apply automatic fixes— signed-off by feldera-bot. Fine.
Nits
PendingAcks::step_at_flushcomment: "The flushed data lands in the next step" — could be explicit that "next" means the completion frontier must exceedstep_at_flush, i.e.total_completed_steps ≥ step_at_flush + 1.- The
#[cfg(test)] DeferredAck::Testvariant is a nice testability affordance, but itsstream: bool, may_defer: boolcombo can encode illegal states (stream=false, may_defer=true). Not user-facing, so fine, but a smallTestAckKindenum would be tidier.
Nothing here is a hard block on the fix itself — it's the CI red, the ignore/flakiness question, and the bounded loop. Address those and I'll approve.
mihaibudiu
left a comment
There was a problem hiding this comment.
Maybe some of this is standard terminology, but I don't know this terminology. I think you need to explain better and have some sections with definitions and a description of the entities involved and how the ensemble works.
| @@ -1,28 +1,94 @@ | |||
| //! PostgreSQL change-data-capture input connector. | |||
| //! | |||
| //! This module embeds an `etl` pipeline that snapshots a PostgreSQL publication and then | |||
There was a problem hiding this comment.
where is the pipeline embedded?
There was a problem hiding this comment.
The Postgres CDC connector uses etl's Pipeline abstraction. We create it here:
| #[schema(default = default_streaming_ack_hold_ms)] | ||
| pub streaming_ack_hold_ms: u64, | ||
|
|
||
| /// Retry tables whose pending destination acknowledgment was dropped by a |
There was a problem hiding this comment.
This comment suggests that this function returns a list of tables.
Also, "retry" expects a verb. Maybe "True if the connector will retry to read data for tables"
| #[schema(default = default_discard_shutdown_errors)] | ||
| pub discard_shutdown_errors: bool, | ||
|
|
||
| /// Retry tables with persisted etl errors when the connector starts. |
|
|
||
| if self.streaming_ack_hold_ms == 0 { | ||
| return Err( | ||
| "streaming_ack_hold_ms must be greater than zero: with a zero hold every \ |
There was a problem hiding this comment.
If 0 is illegal maybe you don't need to give such a long explanation. Just say it's illegal.
| assert!(config.validate().is_ok()); | ||
| } | ||
|
|
||
| #[test] |
There was a problem hiding this comment.
I am not sure how useful this test is.
| database host, port, database, publication, and source table resumes from the | ||
| existing replication state. Changing any of those values creates a different | ||
| replication identity and can cause a new snapshot. | ||
| The connector stores replication state in PostgreSQL and manages its own logical |
There was a problem hiding this comment.
I think you need to start with a little section that describes how this work and establishes terminology. This is one important bit that people may need to understand. Is this postgres the same as the input database, or is it the internal database on the control plane?
| existing replication state. Changing any of those values creates a different | ||
| replication identity and can cause a new snapshot. | ||
| The connector stores replication state in PostgreSQL and manages its own logical | ||
| replication slots. After the initial table copy becomes durable, restarting with |
There was a problem hiding this comment.
when does it become durable?
| The connector stores replication state in PostgreSQL and manages its own logical | ||
| replication slots. After the initial table copy becomes durable, restarting with | ||
| the same database host, port, database, publication, and source table resumes | ||
| from that state instead of repeating the completed copy. A copy interrupted |
There was a problem hiding this comment.
that state? which state? what is the state saved? you seem to have both the data, and some metadata about the data (i.e. which snapshot).
| from that state instead of repeating the completed copy. A copy interrupted | ||
| before its terminal durability barrier completes can be retried. | ||
|
|
||
| Changing any identity field creates a new replication identity and starts a new |
There was a problem hiding this comment.
what is an identity field?
what is a replication identity?
| before its terminal durability barrier completes can be retried. | ||
|
|
||
| Changing any identity field creates a new replication identity and starts a new | ||
| snapshot. Do not manually delete only the replication slot or only the persisted |
There was a problem hiding this comment.
how can one delete stuff manually? You don't say where it is.
Previously, etl could mark the initial snapshot as complete before the snapshot rows were included in a Feldera checkpoint. This meant a restart could skip the snapshot even though Feldera resumed from a checkpoint that didn't contain those rows. Make the final snapshot ack wait until all snapshot rows are durable. Streaming acks now wait for step completion, or for a checkpoint when fault tolerance is enabled. Regular streaming batches can return Accepted after `streaming_ack_hold_ms`, which lets etl continue reading WAL without advancing the replication slot. Also signal etl shutdown before dropping pending acks. Otherwise, a clean stop could persist an errored table state and prevent the connector from starting again. Upgrade etl to 8762c0fc, which contains the shutdown fix, and add tests for snapshot retries, streaming acks and replication slot advancement. Signed-off-by: Abhinav Gyawali <22275402+abhizer@users.noreply.github.com>
mythical-fred
left a comment
There was a problem hiding this comment.
Re-review of 19b8d33 (squashed to a single commit since my 2026-07-16 COMMENT).
Blocker follow-up:
- Unbounded rollback loop — addressed.
discard_matching_table_errorsnow bounds each table atMAX_ERROR_ROLLBACKS_PER_TABLEand falls back toupdate_table_state(TableState::Init)(re-copy from scratch) with awarn!when the cap is hit. Exactly the shape I asked for. Good. #[ignore]inconsistency on new CDC integration tests — not addressed here (the newtest_cdc_ft_streaming_ack_latency_and_slot_advance,test_cdc_ft_stream_ack_flushes_below_min_batch,test_cdc_ft_accepted_streaming_events_replay_after_restart,test_cdc_ft_terminal_catchup_ack_waits_for_checkpoint,test_cdc_ft_snapshot_barrier_covers_buffered_tailall run in CI while the older sibling tests are#[ignore]d). If this is intentional because CI's Postgres is guaranteedwal_level=logicalnow, please add a one-line comment where the ignored tests live noting the divergence. Not a merge blocker on its own — deferring to yours + swanandx's judgement on flakiness risk.- CI still red on 19b8d33. The
maincheck job (run 29598068886, job 87943212985) fails at step 11 ("Don't build the webconsole for rust checks"). That was already red on 5937a71 and the squash didn't clear it. Please rebase / fix the lint or fmt before merge — CI green is a hard rule.
The durability contract itself looks unchanged from the version I already read through carefully: DeferredAck::{SnapshotCopyBarrier, Stream {durability}, Test}, WatcherReceiver::{Fast,Strict}, queued_acks: AtomicUsize gating request_step(), shutdown_etl_pipeline() before dropping pending acks, discard_shutdown_errors (default true) vs discard_table_errors (default false, opt-in destructive resnap). The crown-jewel test_cdc_ft_snapshot_barrier_covers_buffered_tail is still there. Config surface (streaming_ack_hold_ms: u64 = 2000, validator rejects 0) is documented in postgresql-cdc.md and lands in openapi.json.
Commit message imperative + explains why + signed-off, no AI trailers. Good.
Verdict: COMMENT — approving as soon as CI is green. The rollback-loop fix is exactly what I wanted; the only remaining hard blocker is CI red. Item (2) can ride if you're confident.
Previously, etl could mark the initial snapshot as complete before the snapshot rows were included in a Feldera checkpoint. This meant a restart could skip the snapshot even though Feldera resumed from a checkpoint that didn't contain those rows.
Make the final snapshot ack wait until all snapshot rows are durable. Streaming acks now wait for step completion, or for a checkpoint when fault tolerance is enabled. Regular streaming batches can return Accepted after
streaming_ack_hold_ms, which lets etl continue reading WAL without advancing the replication slot.Also signal etl shutdown before dropping pending acks. Otherwise, a clean stop could persist an errored table state and prevent the connector from starting again.
Upgrade etl to 8762c0fc, which contains the shutdown fix, and add tests for snapshot retries, streaming acks and replication slot advancement.
Describe Manual Test Plan
Checklist