Repository navigation
Core: add retry on 503 errors. #5410
Description
The google-api-python-client and gcloud CLI both retry on 503 SERVICE UNAVAILABLE errors yet the current version of google-cloud-python doesn't.
https://github.com/google/google-api-python-client/blob/master/googleapiclient/http.py#L94
We seem to have coded a retry on 500 and 429 , but not 503.
if_transient_error = if_exception_type((
exceptions.InternalServerError,
exceptions.TooManyRequests))Please consider adding 503 as retryable.
I'm running 2.7.11 but I expect same behavior with 3
503 The service is currently unavailable.
File "./create_policies.py", line 48, in <module>
<class 'google.api_core.exceptions.ServiceUnavailable'>
print('Created ' + future.result().name)
File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/concurrent/futures/_base.py", line 462, in result
return self.__get_result()
File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/concurrent/futures/thread.py", line 63, in run
result = self.fn(*self.args, **self.kwargs)
File "./create_policies.py", line 11, in create_policy
policy = alert_client.create_alert_policy(project, policy)
File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/cloud/monitoring_v3/gapic/alert_policy_service_client.py", line 399, in create_alert_policy
request, retry=retry, timeout=timeout, metadata=metadata)
File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/gapic_v1/method.py", line 139, in __call__
return wrapped_func(*args, **kwargs)
File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/retry.py", line 263, in retry_wrapped_func
on_error=on_error,
File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/retry.py", line 178, in retry_target
return target()
File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/timeout.py", line 206, in func_with_timeout
return func(*args, **kwargs)
File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/grpc_helpers.py", line 56, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/six.py", line 737, in raise_from
raise value
google.api_core.exceptions.ServiceUnavailable: 503 The service is currently unavailable.---
Metadata-Version: 2.0
Name: google-cloud
Version: 0.22.0
Summary: API Client library for Google Cloud
Home-page: https://github.com/GoogleCloudPlatform/google-cloud-python
Author: Google Cloud Platform
Author-email: jjg+google-cloud-python@google.com
License: Apache 2.0
Location: /home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages
Requires: google-cloud-resource-manager, google-cloud-monitoring, google-cloud-happybase, google-cloud-language, google-cloud-storage, google-cloud-translate, google-cloud-error-reporting, google-cloud-vision, google-cloud-runtimeconfig, google-cloud-core, google-cloud-bigtable, google-cloud-bigquery, google-cloud-pubsub, google-cloud-datastore, google-cloud-logging, google-cloud-dns
Activity
tseaver commentedon May 31, 2018
@bmenasha Thanks for the report! @theacodes retrying 503 responses seems reasonable to me.
One quick note: the
google-cloudumbrella package is deprecated, and will stop including any of the API-specific packages as dependencies in its next release (about three weeks from now). After that point, you will need to install the API-specific packages directly.xmedeko commentedon May 27, 2019
A storage copy operation fenced by Retry just failed for me by 503, too. Also, the
polling.RETRY_PREDICATEcontainsBadGatewayHTTP 502 exception.Please, fix this simple issue, add
ServiceUnavailableandBadGatewaytoif_transient_errorand uniry it withpolling.RETRY_PREDICATE.Ugly hack until the issue is solved:
[-]Should retry on 503 errors (Core)[/-][+]Core: add retry on 503 errors.[/+]Core: Classify 503 Service Unavailable errors as transient. (#8182)
xmedeko commentedon May 28, 2019
@tseaver Thanks for the fix. What about
BadGatewayHTTP 502 exception? It's contained in thepolling.RETRY_PREDICATEtoo.Core: Classify 503 Service Unavailable errors as transient. (#8182)
Core: Classify 503 Service Unavailable errors as transient. (#8182)
fix(gcp): Add Retry Logic for GCP API Transient HTTP Errors (#2234)
fix(gcp): Add Retry Logic for GCP API Transient HTTP Errors (cartogra…