Skip to content

adapters: pg-cdc: add snapshot transaction support#6670

Open
abhizer wants to merge 2 commits into
pg-cdc-ft-fixesfrom
pg-cdc-transactions
Open

adapters: pg-cdc: add snapshot transaction support#6670
abhizer wants to merge 2 commits into
pg-cdc-ft-fixesfrom
pg-cdc-transactions

Conversation

@abhizer

@abhizer abhizer commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Add support for transactions during snapshot phase.

Group the initial COPY and WAL catchup into separate transactions during
the snapshot phase.

Signed-off-by: Abhinav Gyawali <22275402+abhizer@users.noreply.github.com>
@abhizer
abhizer requested review from ryzhyk and swanandx July 17, 2026 17:27
Signed-off-by: feldera-bot <feldera-bot@feldera.com>

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE. Nice piece of work: the initial-sync transaction split cleanly matches etl's copy/catchup lifecycle, and SnapshotPhase keeps the state machine small and testable.

What I checked:

  • State machine (SnapshotPhase). Copy → Catchup → Inactive, all guarded by a single Mutex<SnapshotPhase>; the four public transitions (start_copy, finish_copy, start_catchup, finish_catchup) each hold the lock only across trivial checks, so blocking a Tokio worker on the std::sync::Mutex is not a concern. finish_catchup_after_etl_completion holds the guard while it also calls phase.finish_catchup() — that's a method on the inner SnapshotPhase (not a re-lock), so no double-lock. Good.
  • Startup path. SnapshotTransactions::initialize() takes the store out of startup_store once and derives the initial phase from etl's persisted TableState. The lock_phase() assert!(not Uninitialized) documents the invariant "etl calls startup() before any write", which matches the destination trait contract. The Errored branch mapping to Inactive is the right call — restart-then-recover is the discard_*_errors path's job, not this state machine's.
  • Copy/catchup commit atomicity. test_cdc_snapshot_transactions_cover_copy_and_catchup is the important test. It asserts (a) COPY rows are not visible until commit, (b) catchup rows are not visible until its commit, (c) steady-state CDC stays outside a connector transaction. The catchup row-count assertion allows SNAPSHOT + CATCHUP or SNAPSHOT + CATCHUP + 1 — mildly loose since etl may or may not have folded the anchor row into the same commit, but that's fine because the atomicity property is what's being checked.
  • Refactor of discard_matching_table_errors. Went from "iterate every errored table id" to "look up the one target table". Correct simplification since this connector replicates exactly one table; any other errored table in the publication was irrelevant to us. Rollback loop unchanged in shape.
  • TableStateMonitor rename + expanded duty. Now polls once per second for both (a) fatal errors and (b) idle catchup completion. Both branches are cheap and read-only against the store on the happy path. Reasonable.
  • Commit hygiene. Two commits, both signed off, no AI trailers.
  • CI. Pending at post time; nothing to block on yet.

Two tiny nits, none blocking:

  1. docs.feldera.com/docs/connectors/sources/postgresql-cdc.md — the new Transactions section calls the tool ETL (uppercase), but the code, module docs, and connector logs consistently use etl (lowercase, matching the crate name). Worth aligning for docs consistency.
  2. SnapshotPhase::start_catchup(None, true) correctly returns false (target table id unknown), but the reason is subtle enough that a one-line doc comment on start_catchup (or a debug_assert! explaining why None cannot legitimately reach here after catchup has begun) would help the next reader.

Otherwise this is good to land once the CI you're waiting on clears.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants