socket.timeout: timed out when using full IPv6 stack.

Environment details

OS: macOS 10.14.5 (18F132) Mojave
Python version: 3.7.3
pip version: 19.1.1
google-api-python-client version: 1.7.9

Steps to reproduce

Pre-steps:

Have a full IPv6 stack with an IPv6 address. In my case, I have the following from Spectrum, through a Linksys WRT3200ACM router.

inet6 fe80::4ab:197d:b439:8d97%en0 prefixlen 64 secured scopeid 0x9 
inet6 2604:6000:1013:a0eb:cca:87bb:d7ed:91b6 prefixlen 64 autoconf secured 
inet6 2604:6000:1013:a0eb:91e1:1128:7728:8e1 prefixlen 64 autoconf temporary

Error:

Follow the steps for the Python Quickstart (https://developers.google.com/docs/api/quickstart/python)
run python quickstart.py
python exits with socket.timeout: timed out error

If I open System Preferences > Network > Wi-FI > Advanced > TCP/IP and change the IPv6 configuration from Automatically to Link-local only, the quickstart.py properly runs.

A few folks have filed issues about socket timeouts recently - thank you for the additional info!

Socket problem is still here, in 2020.

Environment details

OS: Alpine (docker image https://hub.docker.com/_/python python:3.8.0-alpine)

Container runs on AWS EC2 instance.

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/rq/worker.py", line 886, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.8/site-packages/rq/job.py", line 664, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.8/site-packages/rq/job.py", line 670, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/usr/src/app/searchconsole/data.py", line 90, in load_data_for_property
    save_data_batch(property, batch)
  File "/usr/src/app/searchconsole/data.py", line 103, in save_data_batch
    for data_batch in data_batches:
  File "/usr/src/app/searchconsole/google/reports.py", line 103, in generate_lazy
    response = request.execute()
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 849, in execute
    resp, content = _retry_request(
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 184, in _retry_request
    raise exception
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 165, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/google_auth_httplib2.py", line 197, in request
    response, content = self.http.request(
  File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1982, in request
    (response, content) = self._request(
  File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1650, in _request
    (response, content) = self._conn_request(
  File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1589, in _conn_request
    response = conn.getresponse()
  File "/usr/local/lib/python3.8/http/client.py", line 1322, in getresponse
    response.begin()
  File "/usr/local/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.8/http/client.py", line 264, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

@vladignatyev

@vladignatyev - I had this error and was able to get around it by updating /etc/gai.conf to prefer IPV4 over IPV6 (uncomment the line precedence ::ffff:0:0/96 100 ).

I have this issue too

I'm experiencing this right now, the whole weekend, a code that was previoudly fine now i can't get pass authentication:

.../python3.8/site-packages/httplib2/init.py, line 1324, in connect
sock.connect((self.host.self.port__
socket.timetou: time out

Any advices how to solve this ?

I also started experiencing this issue just a day ago, with no modification to the code:

    service = build('sheets', 'v4', credentials=get_creds())
  File "/home/me/anaconda3/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/me/anaconda3/lib/python3.7/site-packages/googleapiclient/discovery.py", line 225, in build
    requested_url, discovery_http, cache_discovery, cache, developerKey
  File "/home/me/anaconda3/lib/python3.7/site-packages/googleapiclient/discovery.py", line 282, in _retrieve_discovery_doc
    resp, content = http.request(actual_url)
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1991, in request
    cachekey,
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1651, in _request
    conn, request_uri, method, body, headers
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1557, in _conn_request
    conn.connect()
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1324, in connect
    sock.connect((self.host, self.port))

OS: Ubuntu 18.04.5 LTS
Python: 3.7.6
pip: 20.0.2
googleapiclient: 1.8.0

Uncommenting the line precedence ::ffff:0:0/96 100 in /etc/gai.conf fixed the problem.

My first attempt to reproduce on a Windows machine did not manifest the bug.
Trying Linux....

Tried on Linux, haven't manifested the bug yet. Still working on it...

How often do you see the socket timeout? With every request? What fraction of your requests?

Had the same issue.
I can see from a nmap that TCP ports 80 and 443 are not open on the IPv6 addresses for www.googleapis.com but they are open for IPv4 addresses:

$ nmap -Pn -p80,443 -6 www.googleapis.com

Starting Nmap 7.60 ( https://nmap.org ) at 2021-06-04 17:57 AEST
Nmap scan report for www.googleapis.com (2404:6800:4015:802::200a)
Host is up.
Other addresses for www.googleapis.com (not scanned): 2404:6800:4015:801::200a 2404:6800:4015:800::200a 2404:6800:4015:803::200a 142.250.70.234 142.250.70.170 142.250.70.138 142.250.70.202
rDNS record for 2404:6800:4015:802::200a: mel05s01-in-x0a.1e100.net

PORT    STATE    SERVICE
80/tcp  filtered http
443/tcp filtered https

Nmap done: 1 IP address (1 host up) scanned in 3.07 seconds

$ nmap -4 www.googleapis.com

Starting Nmap 7.60 ( https://nmap.org ) at 2021-06-04 17:56 AEST
Nmap scan report for www.googleapis.com (142.250.70.202)
Host is up (0.034s latency).
Other addresses for www.googleapis.com (not scanned): 2404:6800:4015:802::200a 2404:6800:4015:803::200a 2404:6800:4015:800::200a 2404:6800:4015:801::200a 142.250.70.234 142.250.70.138 142.250.70.170
rDNS record for 142.250.70.202: mel05s01-in-f10.1e100.net
Not shown: 998 filtered ports
PORT    STATE SERVICE
80/tcp  open  http
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 5.16 seconds

I used the quick hack from here in my code to force the underlying socket to connect over IPv4:
https://stackoverflow.com/a/50044152

It looks like a Google firewall issue

Add timeouts

Hi there! Just came to say that this issue (IPv6 connections that remain stuck until they timeout) still happens from time to time.

Last time I experienced it was on Thursday 7 November morning from ~9:00 to 10:15 (UTC+1) trying to connect to the address resolved by accounts.google.com.

Looks like the issue still exists as a few developers from my organization experienced it now.

File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/googleapiclient/http.py", line 920, in execute
    resp, content = _retry_request(
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/googleapiclient/http.py", line 222, in _retry_request
    raise exception
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/googleapiclient/http.py", line 191, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/google_auth_httplib2.py", line 218, in request
    response, content = self.http.request(
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/httplib2/__init__.py", line 1708, in request
    (response, content) = self._request(
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/httplib2/__init__.py", line 1424, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/site-packages/httplib2/__init__.py", line 1376, in _conn_request
    response = conn.getresponse()
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/http/client.py", line 1345, in getresponse
    response.begin()
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "<user home>/.pyenv/versions/3.9.5/lib/python3.9/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

have been experiencing this intermittently as well while using the gmail client, will try catching and retrying for now to see if that gets through it

Issue still there for me.

@nulld

@nulld mentioned a workaround in #563 where he had to move the socket.setdefaulttimeout() function before import googleapiclient. If it solves the problem, then credit goes to @nulld ! Either way, we can use this data point to better understand whether there are multiple issues going on. Please can someone who is experiencing the issue try the workaround and report back?

@parthea this workaround is working for me for downloading files using drive API.

Also getting this same issue... working code using the Sheets API just stopped. and now getting the socket timeout error every single time. I've tried oauth and API key and am seeing identical results.

I tried the workaround linked above and was unable to make progress.

Even in places where the code make it through (like authorizing the credentials), it take 2-5 minutes where before it was happening almost instantly (note: before meaning 'when the code worked' not 'before the workaround').

Experiencing the same issue with the upload of 8.394 KB to a spreadsheet with BatchUpdate (20000 rows, 40 columns) even after preferring ipv4 in /etc/gai.conf
The docs suggest to stay below 2MB for operations but state that otherwise there are no hard limits:
https://developers.google.com/sheets/api/limits#quota
Initiating the request from Germany
The num_retries parameter is ignored in case of a socket timeout:

The httplib2 _conn_request method raises directly for a socket.timeout
The num_retries is used by the googleapiclient.discovery build inside _retrieve_discovery_doc but the returned discovery doc makes no mention of "retry" or "retries" in the response (at least for sheets v4).
Do I have to pass num_retries to every execute() ?

Let me know if any other information can be useful for troubleshooting

Closing due to inactivity. If this is still a problem, please re-open and we will take a look

Navigation Menu

Repository navigation

`socket.timeout: timed out` when using full IPv6 stack. #709

Environment details

Steps to reproduce

33 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

socket.timeout: timed out when using full IPv6 stack. #709

Description

Environment details

Steps to reproduce

Activity

busunkim96 commented on Jul 10, 2019

vladignatyev commented on Apr 20, 2020

gselva commented on Jun 2, 2020

cheptsov commented on Jun 14, 2020

tuliocgon commented on Aug 2, 2020

vlad-oles commented on Sep 11, 2020

SurferJeffAtGoogle commented on Sep 18, 2020

SurferJeffAtGoogle commented on Sep 18, 2020

SurferJeffAtGoogle commented on Sep 18, 2020

33 remaining items

clmcavaney commented on Jun 4, 2021

glumia commented on Nov 7, 2021

huahouzwift commented on Feb 18, 2022

sourdzl commented on Apr 18, 2022

valentin-panalyt commented on May 31, 2022

rashivkp commented on Jun 29, 2022

bradyjensen1115 commented on Dec 23, 2022

xEverth commented on May 2, 2024

daniel-sanche commented on Feb 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`socket.timeout: timed out` when using full IPv6 stack. #709