Skip to content
This repository was archived by the owner on Mar 31, 2026. It is now read-only.

Commit 78b2eba

Browse files
tseavertseaver
authored
docs: streamline 'timeout' / 'retry' docs in docstrings (#461)
* Add 'requests' intersphinx refs * Add narrative docs for timeouts and retries * Include API docs for the 'retry' module, as well, to unbreak links. * Replace boilerplate docstring entries for 'timeout' / 'retry' with links to new narrative docs. * Add docstrings for default conditional policies. Closes #455.
1 parent 2c87f2f commit 78b2eba

13 files changed

Lines changed: 450 additions & 851 deletions

File tree

‎docs/conf.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -364,6 +364,7 @@
364364
"grpc": ("https://grpc.github.io/grpc/python/", None),
365365
"proto-plus": ("https://proto-plus-python.readthedocs.io/en/latest/", None),
366366
"protobuf": ("https://googleapis.dev/python/protobuf/latest/", None),
367+
"requests": ("https://docs.python-requests.org/en/master/", None),
367368
}
368369

369370

‎docs/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ API Reference
2121
constants
2222
hmac_key
2323
notification
24+
retry_timeout
2425

2526
Changelog
2627
---------

‎docs/retry_timeout.rst‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
Configuring Timeouts and Retries
2+
================================
3+
4+
When using object methods which invoke Google Cloud Storage API methods,
5+
you have several options for how the library handles timeouts and
6+
how it retries transient errors.
7+
8+
9+
.. _configuring_timeouts:
10+
11+
Configuring Timeouts
12+
--------------------
13+
14+
For a number of reasons, methods which invoke API methods may take
15+
longer than expected or desired. By default, such methods all time out
16+
after a default interval, 60.0 seconds. Rather than blocking your application
17+
code for that interval, you may choose to configure explicit timeouts
18+
in your code, using one of three forms:
19+
20+
- You can pass a single integer or float which functions as the timeout for the
21+
entire request. E.g.:
22+
23+
.. code-block:: python
24+
25+
bucket = client.get_bucket(BUCKET_NAME, timeout=300.0) # five minutes
26+
27+
- You can also be passed as a two-tuple, ``(connect_timeout, read_timeout)``,
28+
where the ``connect_timeout`` sets the maximum time required to establish
29+
the connection to the server, and the ``read_timeout`` sets the maximum
30+
time to wait for a completed response. E.g.:
31+
32+
.. code-block:: python
33+
34+
bucket = client.get_bucket(BUCKET_NAME, timeout=(3, 10))
35+
36+
37+
- You can also pass ``None`` as the timeout value: in this case, the library
38+
will block indefinitely for a response. E.g.:
39+
40+
.. code-block:: python
41+
42+
bucket = client.get_bucket(BUCKET_NAME, timeout=None)
43+
44+
.. note::
45+
Depending on the retry strategy, a request may be
46+
repeated several times using the same timeout each time.
47+
48+
See also:
49+
50+
:ref:`Timeouts in requests <requests:timeouts>`
51+
52+
53+
.. _configuring_retries:
54+
55+
Configuring Retries
56+
--------------------
57+
58+
.. note::
59+
60+
For more background on retries, see also the
61+
`GCS Retry Strategies Document <https://cloud.google.com/storage/docs/retry-strategy#python>`_
62+
63+
Methods which invoke API methods may fail for a number of reasons, some of
64+
which represent "transient" conditions, and thus can be retried
65+
automatically. The library tries to provide a sensible default retry policy
66+
for each method, base on its semantics:
67+
68+
- For API requests which are always idempotent, the library uses its
69+
:data:`~google.cloud.storage.retry.DEFAULT_RETRY` policy, which
70+
retries any API request which returns a "transient" error.
71+
72+
- For API requests which are idempotent only if the blob has
73+
the same "generation", the library uses its
74+
:data:`~google.cloud.storage.retry.DEFAULT_RETRY_IF_GENERATION_SPECIFIED`
75+
policy, which retries API requests which returns a "transient" error,
76+
but only if the original request includes an ``ifGenerationMatch`` header.
77+
78+
- For API requests which are idempotent only if the bucket or blob has
79+
the same "metageneration", the library uses its
80+
:data:`~google.cloud.storage.retry.DEFAULT_RETRY_IF_METAGENERATION_SPECIFIED`
81+
policy, which retries API requests which returns a "transient" error,
82+
but only if the original request includes an ``ifMetagenerationMatch`` header.
83+
84+
- For API requests which are idempotent only if the bucket or blob has
85+
the same "etag", the library uses its
86+
:data:`~google.cloud.storage.retry.DEFAULT_RETRY_IF_ETAG_IN_JSON`
87+
policy, which retries API requests which returns a "transient" error,
88+
but only if the original request includes an ``ETAG`` in its payload.
89+
90+
- For those API requests which are never idempotent, the library passes
91+
``retry=None`` by default, suppressing any retries.
92+
93+
Rather than using one of the default policies, you may choose to configure an
94+
explicit policy in your code.
95+
96+
- You can pass ``None`` as a retry policy to disable retries. E.g.:
97+
98+
.. code-block:: python
99+
100+
bucket = client.get_bucket(BUCKET_NAME, retry=None)
101+
102+
- You can pass an instance of :class:`google.api_core.retry.Retry` to enable
103+
retries; the passed object will define retriable response codes and errors,
104+
as well as configuring backoff and retry interval options. E.g.:
105+
106+
.. code-block:: python
107+
108+
from google.api_core import exceptions
109+
from google.api_core.retry import Retry
110+
111+
_MY_RETRIABLE_TYPES = [
112+
exceptions.TooManyRequests, # 429
113+
exceptions.InternalServerError, # 500
114+
exceptions.BadGateway, # 502
115+
exceptions.ServiceUnavailable, # 503
116+
]
117+
118+
def is_retryable(exc):
119+
return isinstance(exc, _MY_RETRIABLE_TYPES)
120+
121+
my_retry_policy = Retry(predicate=is_retryable)
122+
bucket = client.get_bucket(BUCKET_NAME, retry=my_retry_policy)
123+
124+
- You can pass an instance of
125+
:class:`google.cloud.storage.retry.ConditionalRetryPolicy`, which wraps a
126+
:class:`~google.cloud.storage.retry.RetryPolicy`, activating it only if
127+
certain conditions are met. This class exists to provide safe defaults
128+
for RPC calls that are not technically safe to retry normally (due to
129+
potential data duplication or other side-effects) but become safe to retry
130+
if a condition such as if_metageneration_match is set. E.g.:
131+
132+
.. code-block:: python
133+
134+
from google.api_core.retry import Retry
135+
from google.cloud.storage.retry import ConditionalRetryPolicy
136+
from google.cloud.storage.retry import is_etag_in_json
137+
138+
def is_retryable(exc):
139+
... # as above
140+
141+
my_retry_policy = Retry(predicate=is_retryable)
142+
my_cond_policy = ConditionalRetryPolicy(
143+
my_retry_policy, conditional_predicate=is_etag_in_json)
144+
bucket = client.get_bucket(BUCKET_NAME, retry=my_cond_policy)
145+
146+
147+
Retry Module API
148+
----------------
149+
150+
.. automodule:: google.cloud.storage.retry
151+
:members:
152+
:show-inheritance:

‎google/cloud/storage/_helpers.py‎

Lines changed: 15 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -162,11 +162,9 @@ def reload(
162162
properties to return.
163163
164164
:type timeout: float or tuple
165-
:param timeout: (Optional) The amount of time, in seconds, to wait
166-
for the server response.
167-
168-
Can also be passed as a tuple (connect_timeout, read_timeout).
169-
See :meth:`requests.Session.request` documentation for details.
165+
:param timeout:
166+
(Optional) The amount of time, in seconds, to wait
167+
for the server response. See: :ref:`configuring_timeouts`
170168
171169
:type if_generation_match: long
172170
:param if_generation_match: (Optional) Make the operation conditional on whether
@@ -190,18 +188,8 @@ def reload(
190188
blob's current metageneration does not match the given value.
191189
192190
:type retry: google.api_core.retry.Retry or google.cloud.storage.retry.ConditionalRetryPolicy
193-
:param retry: (Optional) How to retry the RPC. A None value will disable retries.
194-
A google.api_core.retry.Retry value will enable retries, and the object will
195-
define retriable response codes and errors and configure backoff and timeout options.
196-
197-
A google.cloud.storage.retry.ConditionalRetryPolicy value wraps a Retry object and
198-
activates it only if certain conditions are met. This class exists to provide safe defaults
199-
for RPC calls that are not technically safe to retry normally (due to potential data
200-
duplication or other side-effects) but become safe to retry if a condition such as
201-
if_metageneration_match is set.
202-
203-
See the retry.py source code and docstrings in this package (google.cloud.storage.retry) for
204-
information on retry types and how to configure them.
191+
:param retry:
192+
(Optional) How to retry the RPC. See: :ref:`configuring_retries`
205193
"""
206194
client = self._require_client(client)
207195
query_params = self._query_params
@@ -275,11 +263,9 @@ def patch(
275263
``client`` stored on the current object.
276264
277265
:type timeout: float or tuple
278-
:param timeout: (Optional) The amount of time, in seconds, to wait
279-
for the server response.
280-
281-
Can also be passed as a tuple (connect_timeout, read_timeout).
282-
See :meth:`requests.Session.request` documentation for details.
266+
:param timeout:
267+
(Optional) The amount of time, in seconds, to wait
268+
for the server response. See: :ref:`configuring_timeouts`
283269
284270
:type if_generation_match: long
285271
:param if_generation_match: (Optional) Make the operation conditional on whether
@@ -303,18 +289,8 @@ def patch(
303289
blob's current metageneration does not match the given value.
304290
305291
:type retry: google.api_core.retry.Retry or google.cloud.storage.retry.ConditionalRetryPolicy
306-
:param retry: (Optional) How to retry the RPC. A None value will disable retries.
307-
A google.api_core.retry.Retry value will enable retries, and the object will
308-
define retriable response codes and errors and configure backoff and timeout options.
309-
310-
A google.cloud.storage.retry.ConditionalRetryPolicy value wraps a Retry object and
311-
activates it only if certain conditions are met. This class exists to provide safe defaults
312-
for RPC calls that are not technically safe to retry normally (due to potential data
313-
duplication or other side-effects) but become safe to retry if a condition such as
314-
if_metageneration_match is set.
315-
316-
See the retry.py source code and docstrings in this package (google.cloud.storage.retry) for
317-
information on retry types and how to configure them.
292+
:param retry:
293+
(Optional) How to retry the RPC. See: :ref:`configuring_retries`
318294
"""
319295
client = self._require_client(client)
320296
query_params = self._query_params
@@ -363,11 +339,9 @@ def update(
363339
``client`` stored on the current object.
364340
365341
:type timeout: float or tuple
366-
:param timeout: (Optional) The amount of time, in seconds, to wait
367-
for the server response.
368-
369-
Can also be passed as a tuple (connect_timeout, read_timeout).
370-
See :meth:`requests.Session.request` documentation for details.
342+
:param timeout:
343+
(Optional) The amount of time, in seconds, to wait
344+
for the server response. See: :ref:`configuring_timeouts`
371345
372346
:type if_generation_match: long
373347
:param if_generation_match: (Optional) Make the operation conditional on whether
@@ -391,18 +365,8 @@ def update(
391365
blob's current metageneration does not match the given value.
392366
393367
:type retry: google.api_core.retry.Retry or google.cloud.storage.retry.ConditionalRetryPolicy
394-
:param retry: (Optional) How to retry the RPC. A None value will disable retries.
395-
A google.api_core.retry.Retry value will enable retries, and the object will
396-
define retriable response codes and errors and configure backoff and timeout options.
397-
398-
A google.cloud.storage.retry.ConditionalRetryPolicy value wraps a Retry object and
399-
activates it only if certain conditions are met. This class exists to provide safe defaults
400-
for RPC calls that are not technically safe to retry normally (due to potential data
401-
duplication or other side-effects) but become safe to retry if a condition such as
402-
if_metageneration_match is set.
403-
404-
See the retry.py source code and docstrings in this package (google.cloud.storage.retry) for
405-
information on retry types and how to configure them.
368+
:param retry:
369+
(Optional) How to retry the RPC. See: :ref:`configuring_retries`
406370
"""
407371
client = self._require_client(client)
408372

‎google/cloud/storage/acl.py‎

Lines changed: 26 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -219,11 +219,9 @@ def _ensure_loaded(self, timeout=_DEFAULT_TIMEOUT):
219219
"""Load if not already loaded.
220220
221221
:type timeout: float or tuple
222-
:param timeout: (Optional) The amount of time, in seconds, to wait
223-
for the server response.
224-
225-
Can also be passed as a tuple (connect_timeout, read_timeout).
226-
See :meth:`requests.Session.request` documentation for details.
222+
:param timeout:
223+
(Optional) The amount of time, in seconds, to wait
224+
for the server response. See: :ref:`configuring_timeouts`
227225
"""
228226
if not self.loaded:
229227
self.reload(timeout=timeout)
@@ -442,20 +440,13 @@ def reload(self, client=None, timeout=_DEFAULT_TIMEOUT, retry=DEFAULT_RETRY):
442440
:param client: (Optional) The client to use. If not passed, falls back
443441
to the ``client`` stored on the ACL's parent.
444442
:type timeout: float or tuple
445-
:param timeout: (Optional) The amount of time, in seconds, to wait
446-
for the server response.
447-
448-
Can also be passed as a tuple (connect_timeout, read_timeout).
449-
See :meth:`requests.Session.request` documentation for details.
443+
:param timeout:
444+
(Optional) The amount of time, in seconds, to wait
445+
for the server response. See: :ref:`configuring_timeouts`
450446
451447
:type retry: :class:`~google.api_core.retry.Retry`
452-
:param retry: (Optional) How to retry the RPC.
453-
454-
A None value will disable retries.
455-
456-
A google.api_core.retry.Retry value will enable retries,
457-
and the object will define retriable response codes and errors
458-
and configure backoff and timeout options.
448+
:param retry:
449+
(Optional) How to retry the RPC. See: :ref:`configuring_retries`
459450
"""
460451
path = self.reload_path
461452
client = self._require_client(client)
@@ -489,21 +480,15 @@ def _save(self, acl, predefined, client, timeout=_DEFAULT_TIMEOUT):
489480
``NoneType``
490481
:param client: (Optional) The client to use. If not passed, falls back
491482
to the ``client`` stored on the ACL's parent.
492-
:type timeout: float or tuple
493-
:param timeout: (Optional) The amount of time, in seconds, to wait
494-
for the server response.
495483
496-
Can also be passed as a tuple (connect_timeout, read_timeout).
497-
See :meth:`requests.Session.request` documentation for details.
484+
:type timeout: float or tuple
485+
:param timeout:
486+
(Optional) The amount of time, in seconds, to wait
487+
for the server response. See: :ref:`configuring_timeouts`
498488
499489
:type retry: :class:`~google.api_core.retry.Retry`
500-
:param retry: (Optional) How to retry the RPC.
501-
502-
A None value will disable retries.
503-
504-
A google.api_core.retry.Retry value will enable retries,
505-
and the object will define retriable response codes and errors
506-
and configure backoff and timeout options.
490+
:param retry:
491+
(Optional) How to retry the RPC. See: :ref:`configuring_retries`
507492
"""
508493
client = self._require_client(client)
509494
query_params = {"projection": "full"}
@@ -545,12 +530,11 @@ def save(self, acl=None, client=None, timeout=_DEFAULT_TIMEOUT):
545530
``NoneType``
546531
:param client: (Optional) The client to use. If not passed, falls back
547532
to the ``client`` stored on the ACL's parent.
548-
:type timeout: float or tuple
549-
:param timeout: (Optional) The amount of time, in seconds, to wait
550-
for the server response.
551533
552-
Can also be passed as a tuple (connect_timeout, read_timeout).
553-
See :meth:`requests.Session.request` documentation for details.
534+
:type timeout: float or tuple
535+
:param timeout:
536+
(Optional) The amount of time, in seconds, to wait
537+
for the server response. See: :ref:`configuring_timeouts`
554538
"""
555539
if acl is None:
556540
acl = self
@@ -577,12 +561,11 @@ def save_predefined(self, predefined, client=None, timeout=_DEFAULT_TIMEOUT):
577561
``NoneType``
578562
:param client: (Optional) The client to use. If not passed, falls back
579563
to the ``client`` stored on the ACL's parent.
580-
:type timeout: float or tuple
581-
:param timeout: (Optional) The amount of time, in seconds, to wait
582-
for the server response.
583564
584-
Can also be passed as a tuple (connect_timeout, read_timeout).
585-
See :meth:`requests.Session.request` documentation for details.
565+
:type timeout: float or tuple
566+
:param timeout:
567+
(Optional) The amount of time, in seconds, to wait
568+
for the server response. See: :ref:`configuring_timeouts`
586569
"""
587570
predefined = self.validate_predefined(predefined)
588571
self._save(None, predefined, client, timeout=timeout)
@@ -601,12 +584,11 @@ def clear(self, client=None, timeout=_DEFAULT_TIMEOUT):
601584
``NoneType``
602585
:param client: (Optional) The client to use. If not passed, falls back
603586
to the ``client`` stored on the ACL's parent.
604-
:type timeout: float or tuple
605-
:param timeout: (Optional) The amount of time, in seconds, to wait
606-
for the server response.
607587
608-
Can also be passed as a tuple (connect_timeout, read_timeout).
609-
See :meth:`requests.Session.request` documentation for details.
588+
:type timeout: float or tuple
589+
:param timeout:
590+
(Optional) The amount of time, in seconds, to wait
591+
for the server response. See: :ref:`configuring_timeouts`
610592
"""
611593
self.save([], client=client, timeout=timeout)
612594

0 commit comments

Comments
 (0)