Skip to content

Handle the end of a replication stream at timeline switches and shutdown - #6715

Open
Brar wants to merge 4 commits into
npgsql:mainfrom
Brar:repl/stream-termination
Open

Brar wants to merge 4 commits into
npgsql:mainfrom
Brar:repl/stream-termination

Conversation

@Brar

@Brar Brar commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

Depends on #6689, #6690 and #6709, which aren't merged yet; their commits show up here until they are.

  • End of a historic timeline. When a physical stream reaches the end of a timeline that is historic or becomes so (e.g. after the standby we stream from got promoted), the server ends copy mode with CopyDone. Npgsql expected only CopyData and broke the connection ("Received backend message CopyDone while expecting CopyDataMessage"). It now answers with CopyDone, reads the next timeline that the server sends, both CommandCompletes and ReadyForQuery, and ends the enumeration normally, leaving the connection usable. Status updates stop before our CopyDone, since the protocol doesn't allow CopyData after it. The next timeline is logged (event 50006), and the StartReplication docs explain how to continue: IDENTIFY_SYSTEM returns the current timeline and TIMELINE_HISTORY the position where it branched off, which can be before the end of the WAL received on the old timeline.
  • Start at the end of a historic timeline. The server doesn't enter copy mode then but sends the next timeline right away, which Npgsql treated as an unexpected message. The enumeration now ends right away.
  • Server shutdown (Misleading exception when a replication connection is aborted due to server shutdown. #5752). A walsender that exits after everything has been replicated sends CommandComplete, which broke the connection with a misleading "Please file a bug" error. It now logs event 50007 and throws an NpgsqlException saying that the server ended the replication stream.
  • Unknown CopyData message types. In logical replication they are skipped instead of breaking the connection, like the logical replication worker, which ignores them on purpose. Physical replication keeps rejecting them, like walreceiver, since it must not lose WAL. Keepalive fields that newer servers may add are skipped in both.
  • The write position (Physical replication reports the server's WAL end as the write position #6697). In physical replication, the received position reported to the server was the server's own WAL end, not the end of the WAL received, so pg_stat_replication.write_lsn and synchronous_commit = remote_write treated WAL as written before it arrived. It is now the end of the data the consumer is done with, as walreceiver reports what it wrote. LastReceivedLsn also starts over at the start position of each stream instead of keeping the maximum of all previous streams, which was wrong when continuing on a new timeline at a lower position.
  • The XLogData stream is disposed after each message instead of later in the cleanup, where it could skip data based on a stale position: a server error that ended the stream after a keepalive (e.g. after pg_cancel_backend) turned into a timeout and a broken connection.

How a real server gets there

On PostgreSQL 18.6 unless noted, main against this PR:

  • Timeline end: a client streams from a standby that gets promoted (an ordinary failover) or that follows its upstream onto a new timeline, or it asks for an older timeline of the server's history (START_REPLICATION … TIMELINE n-1, which is deterministic). With a promotion, main broke the connection at the CopyDone. With this PR the wire showed both CopyDones, the result set and ReadyForQuery, the enumeration ended normally, and IDENTIFY_SYSTEM on the same connection returned timeline 2. Asking for the older timeline gave the same results on 18.6 and 9.6.24.
  • A keepalive after the server's CopyDone: a consumer that holds the timeline's last XLogData for longer than half of wal_sender_timeout gets a reply-requesting keepalive after the server's CopyDone, since the walsender keeps pinging there. With wal_sender_timeout = 3 s and the message held for 2.2 s, it came 1.5 s after the server's CopyDone; this PR sent its CopyDone without a status update and read the result set. Leaving the keepalive unanswered is protocol hygiene only, as PostgreSQL ignores CopyData outside of COPY.
  • Starting at the end of a historic timeline: a client that resumes where it stopped, on a timeline that has become historic meanwhile, lands exactly at the switch point. Main broke the connection with "Received unexpected backend message RowDescription"; with this PR the enumeration was empty, and IDENTIFY_SYSTEM returned timeline 2 (also on 9.6.24).
  • Shutdown: pg_ctl stop -m fast while a client streams. Main reported "Received backend message CommandComplete while expecting CopyDataMessage. Please file a bug."; this PR logs event 50007 and throws "The server ended the replication stream, e.g. because it is shutting down." Both break the connection, since the server is gone.
  • The write position: an XLogData message's WalEnd is the server's send position, so on main any physical consumer that is more than one 128-kB message behind reported WAL as written that it hadn't received yet, e.g. while catching up after a reconnect or during a bulk load. With a 5.4-MB backlog, main reported write 0/95714A8 while sent_lsn was 0/90D4000. With the client in synchronous_standby_names and a session using synchronous_commit = remote_write, main's status update released a commit after 6 ms, about 1 MB before the client had the commit record; with this PR the commit waited until the consumer had passed it.
  • LastReceivedLsn across streams: pg_receivewal's pattern (continue on the next timeline on the same connection, from a lower position), or a second START_REPLICATION at a lower position, e.g. from the last flushed position after a cancellation. With test_decoding, main's status update reported the end of the first stream while the second one started lower; this PR reported the second stream's own start.
  • The stale skip: a physical or test_decoding stream has received a keepalive since its last data message and then ends with a server error that Npgsql didn't request, e.g. after pg_cancel_backend on the walsender (a job that cancels long-running statements). Main waited for WalReceiverTimeout (60 s by default) and then broke the connection with a TimeoutException instead of the server's error. With this PR the PostgresException (57014) came after 25 ms, and IDENTIFY_SYSTEM worked on the same connection. The mock test for this is Stream_ended_by_a_server_error.
  • Mock only, kept as defensive hardening:
    • Unknown CopyData types: no server sends one to Npgsql. The logical skip mirrors PostgreSQL's apply worker ("other message types are purposefully ignored", worker.c on master), and physical replication errors out like walreceiver.
    • Extra keepalive fields: every server version sends keepalives of exactly 18 bytes, so skipping more is forward compatibility.
    • A physical keepalive's WalEnd: it is the end of the WAL the server sent, which the client already has, so not counting it as received changes nothing on real servers; it is the same rule as for XLogData.

The mock tests' standard stop, ErrorResponse 57014 and ReadyForQuery after CopyBothResponse, is what a real walsender sends for a cancellation (checked on the wire).

Tests

  • ReplicationStreamEndMockTests (11 cases, mock): on the code before this PR, 9 of them fail. The other two pass there too: a guard (physical replication still breaks on an unknown type) and a control (the logical received position).
  • PhysicalReplicationClusterTests (live): streaming from a standby across its promotion and on from the new timeline, and streaming through a fast shutdown. They use the throwaway clusters of Add a throwaway primary/standby cluster for replication tests #6690 and need NPGSQL_TEST_PG_BIN, which CI doesn't set; they passed with the binaries of PostgreSQL 10 to 18.

For the reviewer

  • New public constants in NpgsqlEventId: 50006 (ReplicationTimelineEnded) and 50007 (ReplicationStreamEndedByServer). The class suppresses RS0016, so PublicAPI.Unshipped.txt doesn't change; there is no other API change.
  • Behavior changes, also in the 11.0 release notes: the end of a historic timeline ends the enumeration normally, a server shutdown throws NpgsqlException, and the physical write position is the end of the WAL received, not the server's WAL end.
  • The logical skip assumes an aligned stream. Before Stop replication streams properly when they are disposed early #6716, a message longer than the read buffer mid-stream makes the loop read from a stale buffer, and skipping then makes a misread worse. That's why this PR must land together with that one.
  • The write position fix is meant to go to the 10.0 hotfix branch on its own once this is merged; Physical replication reports the server's WAL end as the write position #6697 is its reference.

Dependencies and merge order

Fixes #5752
Fixes #6697

Brar and others added 4 commits October 25, 2026 13:26
PgServerMock can now play a walsender: CopyBothResponse, CopyData,
XLogData and keepalive messages, CopyDone, CopyOutResponse and
NoticeResponse writers, NULL DataRow columns, a configurable
server_version, the parsed startup parameters (e.g. "replication") and
Expect helpers for CopyDone, Terminate and standby status updates.

This also fixes length bugs that were hidden because the client parses
these messages by content (CopyInResponse, ParameterDescription and
ErrorResponse sent wrong lengths), and ExpectMessage skipped
synchronously without ensuring the message was fully received.

ReplicationMock opens replication connections against the mock with the
feedback timers disabled, and ReplicationMockTests smoke-tests startup,
IDENTIFY_SYSTEM, physical streaming, keepalive replies and cancellation,
with timeouts on both the client and the mock side.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ections

- TimelineHistory threw an InvalidCastException on PostgreSQL 14 and
  later, which declare the content column as text instead of bytea. The
  column carries the raw bytes of the history file either way, so it is
  now read as raw bytes. That also stops bytea unescaping from mangling
  backslashes in the content on older servers, and leaves the bytea
  parsing without a use, so it is removed.
- Show("all") returns many rows, which made the command helper expect
  CommandComplete, get another DataRow and break the connection. Further
  rows are now skipped, and Show throws NotSupportedException for results
  that aren't a single value, leaving the connection usable.
- The ReplicationConnection.CommandTimeout property was ignored for
  replication commands, since every command reset the timeouts to the
  connection string's command timeout. It now applies to all commands,
  and setting it before Open no longer throws. Timeouts that don't fit
  into an int of milliseconds (e.g. TimeSpan.MaxValue) mean no timeout.
- StartReplication(PhysicalReplicationSlot, CancellationToken) throws
  ArgumentNullException for a null slot instead of a
  NullReferenceException.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PgTestCluster creates PostgreSQL clusters in temporary directories with
the binaries that NPGSQL_TEST_PG_BIN points to (initdb, pg_ctl,
pg_basebackup): a primary that allows physical and logical replication,
streaming standbys of it, promotion and shutdown in any mode. This
covers tests that can't run against the shared test server, like
timeline switches and server shutdowns. Tests using it are ignored when
the variable isn't set, so CI is unaffected.

The clusters listen on 127.0.0.1 only (no Unix sockets, whose default
directory isn't writable for unprivileged users on Debian-based
builds), pick a free port right before each start, retry when another
process took it, and limit every step to well below CI's 30 seconds per
test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- At the end of a historic timeline (e.g. after the standby we stream
  from got promoted), the server ends copy mode with CopyDone. Npgsql
  expected only CopyData and broke the connection ("Received backend
  message CopyDone while expecting CopyDataMessage"). It now answers
  with CopyDone, reads the next timeline the server sends, both
  CommandCompletes and ReadyForQuery, and ends the enumeration normally,
  leaving the connection usable. Status updates stop before our
  CopyDone, since the protocol doesn't allow CopyData after it. The next
  timeline is logged, and the StartReplication docs explain how to
  continue: IDENTIFY_SYSTEM returns the current timeline and
  TIMELINE_HISTORY the position where it branched off, which can be
  before the end of the WAL received on the old timeline.
- Asked to start at the end of a historic timeline, the server doesn't
  enter copy mode but sends the next timeline right away, which Npgsql
  treated as an unexpected message. The enumeration now ends right away.
- A walsender that exits after everything has been replicated (e.g. at
  server shutdown, npgsql#5752) sends CommandComplete, which also broke the
  connection with a misleading error. It now throws an NpgsqlException
  saying that the server ended the replication stream.
- In logical replication, unknown CopyData message types are skipped
  instead of breaking the connection, like the logical replication
  worker, which ignores them on purpose. Physical replication keeps
  rejecting them, like walreceiver, since it must not lose WAL.
  Keepalive fields that newer servers may add are skipped in both.
- In physical replication, the received position reported to the server
  was the server's own WAL end, not the end of the WAL received, so
  pg_stat_replication.write_lsn and synchronous_commit = remote_write
  treated WAL as written before it arrived. It is now the end of the
  data the consumer is done with, like walreceiver reports what it
  wrote.
- LastReceivedLsn starts over at the start position of each stream
  instead of keeping the maximum of all previous streams, which was
  wrong when continuing on a new timeline at a lower position.
- The XLogData stream is disposed after each message instead of later in
  cleanup, where it could skip data based on a stale position: a server
  error that ended the stream after a keepalive (e.g. after
  pg_cancel_backend) turned into a timeout and a broken connection.

The new cluster tests stream physically across a standby's promotion
and through a fast shutdown, using the throwaway clusters from
NPGSQL_TEST_PG_BIN.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant