Skip to content

socket.timeout with will cause api client to become unuseable #563

Closed
Closed
socket.timeout with will cause api client to become unuseable#563
Assignees
busunkim96
Labels
🚨This issue needs some love.priority: p2Moderately-important priority. Fix may not be included in next release.status: will not fixInvalid (untrue/unsound/erroneous), inconsistent with product, not on roadmap.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Description

@thehesiod
thehesiod
opened on Aug 23, 2018

We recently ran into a socket.timeout issue using the google api client, and noticed that then all our requests would fail in the same way. I later find that connections are cached here: https://github.com/httplib2/httplib2/blob/master/python3/httplib2/__init__.py#L1759

however there is no code to remove cached connections that are bad. This is a major oversight.

Here's the callstack we hit:

    batch_entry.batch.execute(http=http, num_retries=num_retries)
File "/pyenv/lib/python3.6/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
File "/pyenv/lib/python3.6/site-packages/googleapiclient/http.py", line 1465, in execute
    self._execute(http, self._order, self._requests, num_retries)
File "/pyenv/lib/python3.6/site-packages/googleapiclient/http.py", line 1394, in _execute
    body=body, headers=headers)
File "/pyenv/lib/python3.6/site-packages/googleapiclient/http.py", line 185, in _retry_request
    raise exception
File "/pyenv/lib/python3.6/site-packages/googleapiclient/http.py", line 165, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
File "/pyenv/lib/python3.6/site-packages/oauth2client/transport.py", line 175, in new_request
    redirections, connection_type)
File "/pyenv/lib/python3.6/site-packages/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
File "/pyenv/lib/python3.6/site-packages/httplib2/__init__.py", line 1514, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/pyenv/lib/python3.6/site-packages/httplib2/__init__.py", line 1264, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/pyenv/lib/python3.6/site-packages/httplib2/__init__.py", line 1217, in _conn_request
    response = conn.getresponse()
File "/usr/local/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
File "/usr/local/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
File "/usr/local/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
File "/usr/local/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
File "/usr/local/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
File "/usr/local/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

Activity

  1. thehesiod commented on Aug 23, 2018

    @thehesiod
    thehesiod
    on Aug 23, 2018
    ContributorAuthor

    looks like this should do something similar to what's being done in the cloud-sdk retry code: http://testcompany.info/google-cloud-sdk/lib/third_party/apitools/base/py/http_wrapper.py

  2. thehesiod commented on Aug 23, 2018

    @thehesiod
    thehesiod
    on Aug 23, 2018
    ContributorAuthor

    note: a separate callstack happens via discovery.build

  3. thehesiod commented on Aug 23, 2018

    @thehesiod
    thehesiod
    on Aug 23, 2018
    ContributorAuthor

    I think a better solution would be for httplib2 to handle most of these cases, like socket.*. Higher level errors can be handled by the google client

  4. thehesiod commented on Aug 23, 2018

    @thehesiod
    thehesiod
    on Aug 23, 2018
    ContributorAuthor

    see: httplib2/httplib2#18

  5. JustinBeckwith
    added
    triage meI really want to be triaged.
    🚨This issue needs some love.
    on Aug 23, 2018

  6. busunkim96 commented on Dec 6, 2018

    @busunkim96
    busunkim96
    on Dec 6, 2018
    Contributor

    This has been resolved as of httplib2 v0.12.0.

  7. busunkim96
    closed this as completedon Dec 6, 2018
  8. busunkim96
    added
    status: will not fixInvalid (untrue/unsound/erroneous), inconsistent with product, not on roadmap.
    and removed
    🚨This issue needs some love.
    triage meI really want to be triaged.
    on Dec 6, 2018

  9. thehesiod commented on Dec 6, 2018

    @thehesiod
    thehesiod
    on Dec 6, 2018
    ContributorAuthor

    actually no, it's post 0.12.0 (806de52a2939d95800e43b1ab96be45a33b882d1 vs 806de52a2939d95800e43b1ab96be45a33b882d1 for 0.12.0), supposedly 12.1. This should be re-opened

  10. mvrozanti commented on Dec 6, 2018

    @mvrozanti
    mvrozanti
    on Dec 6, 2018

    Can confirm.

    
Traceback (most recent call last):
  File "./query.py", line 88, in <module>
    if __name__ == '__main__': get_front_page(1,1)
  File "./query.py", line 67, in get_front_page
    service = get_authenticated_service()
  File "./query.py", line 54, in get_authenticated_service
    service = build(API_SERVICE_NAME, API_VERSION, credentials=credentials)
  File "/home/nexor/.pyenv/versions/3.6.0/lib/python3.6/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/nexor/.pyenv/versions/3.6.0/lib/python3.6/site-packages/googleapiclient/discovery.py", line 222, in build
    requested_url, discovery_http, cache_discovery, cache, developerKey)
  File "/home/nexor/.pyenv/versions/3.6.0/lib/python3.6/site-packages/googleapiclient/discovery.py", line 272, in _retrieve_discovery_doc
    resp, content = http.request(actual_url)
  File "/home/nexor/.local/lib/python3.6/site-packages/httplib2/__init__.py", line 1924, in request
    cachekey,
  File "/home/nexor/.local/lib/python3.6/site-packages/httplib2/__init__.py", line 1595, in _request
    conn, request_uri, method, body, headers
  File "/home/nexor/.local/lib/python3.6/site-packages/httplib2/__init__.py", line 1501, in _conn_request
    conn.connect()
  File "/home/nexor/.local/lib/python3.6/site-packages/httplib2/__init__.py", line 1289, in connect
    sock.connect((self.host, self.port))
socket.timeout: timed out

    Checked to see if credentials were expired but it's fine. Not sure why this is happening

    Requirement already up-to-date: httplib2 in /home/nexor/.local/lib/python3.6/site-packages (0.12.0)

  11. busunkim96
    reopened this on Dec 6, 2018
  12. JustinBeckwith
    added
    triage meI really want to be triaged.
    on Dec 6, 2018

  13. 6 remaining items

  14. sduskis
    added
    priority: p2Moderately-important priority. Fix may not be included in next release.
    and removed
    🚨This issue needs some love.
    triage meI really want to be triaged.
    on Dec 7, 2018

  15. mvrozanti commented on Dec 7, 2018

    @mvrozanti
    mvrozanti
    on Dec 7, 2018

    Anyone got any quickfix in the meantime? I don't seem to be able to get past authentication, ever. But the same script was working a couple days ago. quickstart.py sample also stopped working.

  16. mvrozanti commented on Dec 11, 2018

    @mvrozanti
    mvrozanti
    on Dec 11, 2018

    Magically started working again

  17. yoshi-automation
    added
    🚨This issue needs some love.
    on Feb 19, 2019

  18. busunkim96 commented on Jun 10, 2019

    @busunkim96
    busunkim96
    on Jun 10, 2019
    Contributor

    v0.12.1 of httplib2 has been released and contains the update to clear dead connections.

  19. busunkim96
    closed this as completedon Jun 10, 2019

  20. echan00 commented on Oct 7, 2019

    @echan00
    echan00
    on Oct 7, 2019

    #480

  21. topoleov commented on Dec 7, 2019

    @topoleov
    topoleov
    on Dec 7, 2019

    httplib2 v.0.14.0 error still happen

  22. busunkim96 commented on Dec 9, 2019

    @busunkim96
    busunkim96
    on Dec 9, 2019
    Contributor

    @soogly If you're experiencing this issue please open a new issue with more details on what's happening. Repository maintainers have a harder time keeping track of responses to closed issues.

    Thanks!

  23. Vmarkelov commented on Feb 25, 2020

    @Vmarkelov
    Vmarkelov
    on Feb 25, 2020

    Hello, this error is happening multiple times a day.

    The read operation timed out
    Traceback (most recent call last):
    File "", line 197, in
    File "", line 34, in get_results
    File "/usr/local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
    File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 849, in execute
    resp, content = _retry_request(
    File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 184, in _retry_request
    raise exception
    File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 165, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
    File "/usr/local/lib/python3.8/site-packages/oauth2client/transport.py", line 173, in new_request
    resp, content = request(orig_request_method, uri, method, body,
    File "/usr/local/lib/python3.8/site-packages/oauth2client/transport.py", line 280, in request
    return http_callable(uri, method=method, body=body, headers=headers,
    File "/usr/local/lib/python3.8/site-packages/httplib2/init.py", line 1982, in request
    (response, content) = self._request(
    File "/usr/local/lib/python3.8/site-packages/httplib2/init.py", line 1650, in _request
    (response, content) = self._conn_request(
    File "/usr/local/lib/python3.8/site-packages/httplib2/init.py", line 1589, in _conn_request
    response = conn.getresponse()
    File "/usr/local/lib/python3.8/http/client.py", line 1322, in getresponse
    response.begin()
    File "/usr/local/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
    File "/usr/local/lib/python3.8/http/client.py", line 264, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
    File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
    File "/usr/local/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
    File "/usr/local/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
    socket.timeout: The read operation timed out
    Script failed.

    Any suggestions how to get rid of it

    Thanks

  24. busunkim96 commented on Feb 25, 2020

    @busunkim96
    busunkim96
    on Feb 25, 2020
    Contributor

    Please reference #709 to track socket timeout issues.

  25. nulld commented on Dec 3, 2020

    @nulld
    nulld
    on Dec 3, 2020

    Hello, this error is happening multiple times a day.

    The read operation timed out
    Traceback (most recent call last):
    File "", line 197, in
    File "", line 34, in get_results
    File "/usr/local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
    File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 849, in execute
    resp, content = _retry_request(
    File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 184, in _retry_request
    raise exception
    File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 165, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
    File "/usr/local/lib/python3.8/site-packages/oauth2client/transport.py", line 173, in new_request
    resp, content = request(orig_request_method, uri, method, body,
    File "/usr/local/lib/python3.8/site-packages/oauth2client/transport.py", line 280, in request
    return http_callable(uri, method=method, body=body, headers=headers,
    File "/usr/local/lib/python3.8/site-packages/httplib2/init.py", line 1982, in request
    (response, content) = self._request(
    File "/usr/local/lib/python3.8/site-packages/httplib2/init.py", line 1650, in _request
    (response, content) = self._conn_request(
    File "/usr/local/lib/python3.8/site-packages/httplib2/init.py", line 1589, in _conn_request
    response = conn.getresponse()
    File "/usr/local/lib/python3.8/http/client.py", line 1322, in getresponse
    response.begin()
    File "/usr/local/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
    File "/usr/local/lib/python3.8/http/client.py", line 264, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
    File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
    File "/usr/local/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
    File "/usr/local/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
    socket.timeout: The read operation timed out
    Script failed.

    Any suggestions how to get rid of it

    Thanks

    After hours of hacking api code, found the solution.

    import socket
socket.setdefaulttimeout(4000) 


import argparse
import googleapiclient.discovery
from google.oauth2 import service_account
import mimetypes
import json
import sys
import os

    Make note the order - socket.setdefaulttimeout(timeoutInSec) needs to be before any import of googleapiclient.
    No magic happens instead ;)

  26. JustinBeckwith
    assigned
    busunkim96
    on Feb 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

🚨This issue needs some love.priority: p2Moderately-important priority. Fix may not be included in next release.status: will not fixInvalid (untrue/unsound/erroneous), inconsistent with product, not on roadmap.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions