Repository navigation
Conversation
PgServerMock can now play a walsender: CopyBothResponse, CopyData, XLogData and keepalive messages, CopyDone, CopyOutResponse and NoticeResponse writers, NULL DataRow columns, a configurable server_version, the parsed startup parameters (e.g. "replication") and Expect helpers for CopyDone, Terminate and standby status updates. This also fixes length bugs that were hidden because the client parses these messages by content (CopyInResponse, ParameterDescription and ErrorResponse sent wrong lengths), and ExpectMessage skipped synchronously without ensuring the message was fully received. ReplicationMock opens replication connections against the mock with the feedback timers disabled, and ReplicationMockTests smoke-tests startup, IDENTIFY_SYSTEM, physical streaming, keepalive replies and cancellation, with timeouts on both the client and the mock side. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ections
- TimelineHistory threw an InvalidCastException on PostgreSQL 14 and
later, which declare the content column as text instead of bytea. The
column carries the raw bytes of the history file either way, so it is
now read as raw bytes. That also stops bytea unescaping from mangling
backslashes in the content on older servers, and leaves the bytea
parsing without a use, so it is removed.
- Show("all") returns many rows, which made the command helper expect
CommandComplete, get another DataRow and break the connection. Further
rows are now skipped, and Show throws NotSupportedException for results
that aren't a single value, leaving the connection usable.
- The ReplicationConnection.CommandTimeout property was ignored for
replication commands, since every command reset the timeouts to the
connection string's command timeout. It now applies to all commands,
and setting it before Open no longer throws. Timeouts that don't fit
into an int of milliseconds (e.g. TimeSpan.MaxValue) mean no timeout.
- StartReplication(PhysicalReplicationSlot, CancellationToken) throws
ArgumentNullException for a null slot instead of a
NullReferenceException.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PgTestCluster creates PostgreSQL clusters in temporary directories with the binaries that NPGSQL_TEST_PG_BIN points to (initdb, pg_ctl, pg_basebackup): a primary that allows physical and logical replication, streaming standbys of it, promotion and shutdown in any mode. This covers tests that can't run against the shared test server, like timeline switches and server shutdowns. Tests using it are ignored when the variable isn't set, so CI is unaffected. The clusters listen on 127.0.0.1 only (no Unix sockets, whose default directory isn't writable for unprivileged users on Debian-based builds), pick a free port right before each start, retry when another process took it, and limit every step to well below CI's 30 seconds per test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- At the end of a historic timeline (e.g. after the standby we stream
from got promoted), the server ends copy mode with CopyDone. Npgsql
expected only CopyData and broke the connection ("Received backend
message CopyDone while expecting CopyDataMessage"). It now answers
with CopyDone, reads the next timeline the server sends, both
CommandCompletes and ReadyForQuery, and ends the enumeration normally,
leaving the connection usable. Status updates stop before our
CopyDone, since the protocol doesn't allow CopyData after it. The next
timeline is logged, and the StartReplication docs explain how to
continue: IDENTIFY_SYSTEM returns the current timeline and
TIMELINE_HISTORY the position where it branched off, which can be
before the end of the WAL received on the old timeline.
- Asked to start at the end of a historic timeline, the server doesn't
enter copy mode but sends the next timeline right away, which Npgsql
treated as an unexpected message. The enumeration now ends right away.
- A walsender that exits after everything has been replicated (e.g. at
server shutdown, npgsql#5752) sends CommandComplete, which also broke the
connection with a misleading error. It now throws an NpgsqlException
saying that the server ended the replication stream.
- In logical replication, unknown CopyData message types are skipped
instead of breaking the connection, like the logical replication
worker, which ignores them on purpose. Physical replication keeps
rejecting them, like walreceiver, since it must not lose WAL.
Keepalive fields that newer servers may add are skipped in both.
- In physical replication, the received position reported to the server
was the server's own WAL end, not the end of the WAL received, so
pg_stat_replication.write_lsn and synchronous_commit = remote_write
treated WAL as written before it arrived. It is now the end of the
data the consumer is done with, like walreceiver reports what it
wrote.
- LastReceivedLsn starts over at the start position of each stream
instead of keeping the maximum of all previous streams, which was
wrong when continuing on a new timeline at a lower position.
- The XLogData stream is disposed after each message instead of later in
cleanup, where it could skip data based on a stale position: a server
error that ended the stream after a keepalive (e.g. after
pg_cancel_backend) turned into a timeout and a broken connection.
The new cluster tests stream physically across a standby's promotion
and through a fast shutdown, using the throwaway clusters from
NPGSQL_TEST_PG_BIN.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Breaking out of a replication enumeration, or an exception in the consumer, left the inner enumerator undisposed. The connection stayed in replication mode with its feedback timers running, and every other command failed with NpgsqlOperationInProgressException. The enumerators are now disposed, which stops streaming: physical replication ends copy mode with CopyDone, which keeps the walsender ready for the next START_REPLICATION and keeps temporary slots; logical replication cancels the command, since a logical walsender can't stream again after CopyDone. Like any error, the cancellation drops the temporary slots of the session (documented on the logical StartReplication methods). Servers before PostgreSQL 10 ignore cancellation requests on replication connections, so with them the logical stop breaks the connection instead. A final status update reports the processed position first; for logical replication this is best effort, since the walsender may handle the cancellation before it reads the update. If stopping fails, the connection is broken; the exception that made the consumer stop is never replaced. On PostgreSQL 14-17, the server may not send pgoutput's Relation messages again in the next stream on the same connection after such a cancellation (on 14 after any error, on 15-17 if the connection created the slot with an exported snapshot; fixed in PostgreSQL 18 by 80b727eb9de). This is documented on the pgoutput StartReplication, and the test avoids it. - Status updates that were still on their way when a stream ended (e.g. from a timer or another thread) could be written after the end of the command, interleaved with the next one. They are now dropped. - Whatever the consumer didn't read of the message it held is skipped first, also when a plugin like pgoutput reads the data directly from the buffer. Reading beyond the end of a message breaks the connection. - After cancellation, messages that were already on their way were still handed out until the server's error arrived. They are now skipped. This makes the physical replication tests reliable, so they run again (some of them also used colliding slot names). - Starting a second stream while one was in progress overwrote the state of the first one before failing, so disposing the connection didn't stop the first stream anymore. It now fails before touching anything, and a start that fails leaves no state behind. - A message bigger than the read buffer (e.g. a long notice) made the connector switch to a bigger buffer while the stream kept reading from the old one. This is fixed for the replication stream itself; the pgoutput decoder still reads from the old buffer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A WalReceiverStatusInterval or WalReceiverTimeout of zero made the feedback timers fire over and over, which flooded the server with status updates (in the case of the timeout with reply requests, which the server answered every time). Like wal_receiver_status_interval and wal_receiver_timeout in PostgreSQL, zero now disables them, and so do intervals longer than a timer supports (~49 days), which failed once the server had switched to copy mode. Positive intervals below a millisecond fire every millisecond. Waiting for the end of START_REPLICATION after CopyDone is bounded for such long timeouts too. - Negative values other than Timeout.InfiniteTimeSpan are now rejected by the setters. Before, starting replication failed with them once the server had switched to copy mode, which left the connection unusable. - SendStatusUpdate logged write failures without reporting them, so callers that push their flush position (e.g. for synchronous replication) couldn't tell that it never reached the server. It now throws them; like any failed write, they break the connection. A status update that waited for one that broke the connection fails with InvalidOperationException instead of a NullReferenceException, and so does one after the stream ended. The reply to a keepalive also reports its failure, through the stream, instead of a NullReferenceException at the next read. - Add a missing space to a log message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a status update of a feedback timer failed, the timer only logged the failure, and breaking the connection disposed its buffers. If the consumer held a message at that time, its next MoveNextAsync failed with a NullReferenceException: when reading the next message, or, for plugins like pgoutput that read from the buffer directly, when skipping the rest of the held message. Any other failure that broke the connection while the consumer held a message, e.g. a SendStatusUpdate that failed, had the same effect. The stream now checks the connection when the consumer asks for the next message. If it broke, it throws an NpgsqlException with the exception that broke it as the inner exception, with the message NpgsqlConnector.StartUserAction uses for a broken connection when keepalive is enabled. NpgsqlConnector gets an internal BreakReason accessor for this. pgoutput's Insert, Update and Delete messages whose values weren't read to the end still fail with a misleading exception in this case, since pgoutput consumes the rest of the row before it asks for the next message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #6689, #6690, #6709, #6715, #6716 and #6719, which aren't merged yet; their commits show up here until they are.
When a status update of a feedback timer failed, the timer only logged the failure (event 50103), and breaking the connection disposed its buffers. If the consumer held a message at that time, its next
MoveNextAsyncfailed with a NullReferenceException: when reading the next message, or, for plugins like pgoutput that read from the buffer directly, when skipping the rest of the held message. Any other failure that broke the connection while the consumer held a message, e.g. aSendStatusUpdatethat failed, had the same effect.The stream now checks the connection when the consumer asks for the next message. If it broke, it throws an NpgsqlException with the exception that broke it as the inner exception, with the message that
NpgsqlConnector.StartUserActionuses for a broken connection when keepalive is enabled ("The connection was previously broken because of the following exception").NpgsqlConnectorgets an internalBreakReasonaccessor for this.Not covered: pgoutput's Insert, Update and Delete messages whose values weren't read to the end still fail with a misleading exception in this case, since pgoutput consumes the rest of the row before it asks for the next message.
How a real server gets there
A consumer is still working on a message (a slow sink, retries) when the server side ends the connection:
pg_terminate_backendby an administrator or a failover tool, a restart or a failover, the walsender'swal_sender_timeout(nobody answers its keepalive requests while the consumer holds the message, ifWalReceiverStatusIntervalis at leastwal_sender_timeout), or a reset by a NAT or a load balancer. The timer's next status update then hits the dead socket and breaks the connector.Live on PostgreSQL 18.6, with a terminated walsender (physical and pgoutput streams), the replication timeout and a TCP reset:
MoveNextAsyncthrew NullReferenceException (inNpgsqlConnector.ReadMessage), in Release and Debug.SetReplicationStatusandSendStatusUpdateon a connection that died meanwhile: on main the firstSendStatusUpdatereturned normally and the nextMoveNextAsyncthrew NullReferenceException. With Fix zero feedback intervals and swallowed status update failures #6719 and this change,SendStatusUpdatethrows the write failure, andMoveNextAsyncthrows it again as the break reason.Tests
ReplicationFeedbackMockTests.Timer_status_update_failure_ends_the_stream, reading through the message's stream and bypassing it as pgoutput does: the consumer holds a message while the timer's write fails. Red first: NullReferenceException in both cases.SendStatusUpdate_reports_write_failures(from Fix zero feedback intervals and swallowed status update failures #6719) gets an assertion for the nextMoveNextAsync, which fails without this change.For the reviewer
NpgsqlConnector; no public API change.yield return, not beforeReadMessage: with pgoutput's direct reads the NullReferenceException comes from skipping the rest of the payload, before the loop reads again, and a check beforeReadMessagealone left that case red.Dependencies and merge order
ReplicationFeedbackMockTests); merge those first. Only the last commit is this PR's own; the others drop out when the branch is rebased onto main after they land.SendStatusUpdatethrows its write failure, the consumer's nextMoveNextAsyncruns into the NullReferenceException that this PR fixes.Fixes #6700