From 8b5b05568c3eed00ac03228903015bfcce20a40a Mon Sep 17 00:00:00 2001 From: Peter Lamut Date: Tue, 23 Jun 2020 15:54:05 +0200 Subject: [PATCH 1/3] docs: add flow control section to publish overview --- docs/publisher/index.rst | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/docs/publisher/index.rst b/docs/publisher/index.rst index 2a785359c..fdf056a6f 100644 --- a/docs/publisher/index.rst +++ b/docs/publisher/index.rst @@ -128,6 +128,35 @@ You can also attach a callback to the future: future.add_done_callback(callback) +Publish Flow Control +-------------------- + +If publishing large amounts of messages or very large messages in quick +succession, some of the publish requests might time out, especially if the +bandwidth available is limited. To mitigate this the client can be +configured with custom :class:`~.pubsub_v1.types.PublishFlowControl` settings. + +You can configure the maximum desired number of messages and their maximum total +size, as well as the action that should be taken when the threshold is reached. +If you choose ``BLOCK``, for instance, the client will internally queue +excessive messages before their corresponding publish requests are made, +reducing the chance of network timeouts. + +.. code-block:: python + + from google.cloud import pubsub_v1 + + client = pubsub_v1.PublisherClient( + publisher_options=pubsub_v1.types.PublisherOptions( + flow_control=pubsub_v1.types.PublishFlowControl( + message_limit=500, + byte_limit=2 * 1024 * 1024, + limit_exceeded_behavior=pubsub_v1.types.LimitExceededBehavior.BLOCK, + ), + ), + ) + + API Reference ------------- From e1d98f86ae56eee3d3edf34a11fe3026dc03506e Mon Sep 17 00:00:00 2001 From: Peter Lamut Date: Tue, 30 Jun 2020 08:50:29 +0200 Subject: [PATCH 2/3] Explain options for limit exceeded behavior --- docs/publisher/index.rst | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/docs/publisher/index.rst b/docs/publisher/index.rst index fdf056a6f..664d61736 100644 --- a/docs/publisher/index.rst +++ b/docs/publisher/index.rst @@ -138,9 +138,6 @@ configured with custom :class:`~.pubsub_v1.types.PublishFlowControl` settings. You can configure the maximum desired number of messages and their maximum total size, as well as the action that should be taken when the threshold is reached. -If you choose ``BLOCK``, for instance, the client will internally queue -excessive messages before their corresponding publish requests are made, -reducing the chance of network timeouts. .. code-block:: python @@ -156,6 +153,18 @@ reducing the chance of network timeouts. ), ) +The action to be taken on overflow can be one of the following: + +* :attr:`~.pubsub_v1.types.LimitExceededBehavior.IGNORE` (default): Ignore the + overflow and continue publishing the messages as normal. +* :attr:`~.pubsub_v1.types.LimitExceededBehavior.ERROR`: Raise + :exc:`~.pubsub_v1.publisher.exceptions.FlowControlLimitError` and reject the message. +* :attr:`~.pubsub_v1.types.LimitExceededBehavior.BLOCK`: Temporarily block in the + :meth:`~.pubsub_v1.publisher.client.Client.publish` method until there is + enough capacity available. Excessive messages are queued internally before + their corresponding publish requests are actually made, reducing the chance of + network timeouts. + API Reference ------------- From 751b5cf0a7160db04f244d72fae084f27230968f Mon Sep 17 00:00:00 2001 From: Peter Lamut Date: Fri, 10 Jul 2020 16:36:25 +0200 Subject: [PATCH 3/3] Omit a sentence that might cause confusion. --- docs/publisher/index.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/publisher/index.rst b/docs/publisher/index.rst index 664d61736..cd2e5cbea 100644 --- a/docs/publisher/index.rst +++ b/docs/publisher/index.rst @@ -161,9 +161,7 @@ The action to be taken on overflow can be one of the following: :exc:`~.pubsub_v1.publisher.exceptions.FlowControlLimitError` and reject the message. * :attr:`~.pubsub_v1.types.LimitExceededBehavior.BLOCK`: Temporarily block in the :meth:`~.pubsub_v1.publisher.client.Client.publish` method until there is - enough capacity available. Excessive messages are queued internally before - their corresponding publish requests are actually made, reducing the chance of - network timeouts. + enough capacity available. API Reference