httplib2.Http is not thread-safe

This is from a StackOverflow post. I've done some debugging from the Datastore side but I don't think these requests ever make it into the Datastore part of the stack. I'd appreciate if someone look at this from the gcloud-python team. Note that this issue was originally reported via gcd-discuss@google.com In July.

I have a Python Django application running on a Google Compute instance. It is using gcloudoem to interface from Django to Google Datastore. gcloudoem uses the same underlying code to communicate with Datastore as gcloud-python 0.5.x

At what seems to be completely random times, I will get SSL errors happening when trying to talk to Datastore. There is no pattern in where in my application code these happen. It's just during a random call to Datastore. Here are the two flavours of errors:

ERROR:django.request:Internal Server Error: /complete/google-oauth2/
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/cache.py", line 52, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/csrf.py", line 57, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/apps/django_app/utils.py", line 51, in wrapper
    return func(request, backend, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/apps/django_app/views.py", line 28, in complete
    redirect_name=REDIRECT_FIELD_NAME, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/actions.py", line 43, in do_complete
    user = backend.complete(user=user, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/base.py", line 41, in complete
    return self.auth_complete(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/utils.py", line 229, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/oauth.py", line 387, in auth_complete
    *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/utils.py", line 229, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/oauth.py", line 396, in do_auth
    return self.strategy.authenticate(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/strategies/django_strategy.py", line 96, in authenticate
    return authenticate(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/__init__.py", line 60, in authenticate
    user = backend.authenticate(**credentials)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/base.py", line 82, in authenticate
    return self.pipeline(pipeline, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/base.py", line 85, in pipeline
    out = self.run_pipeline(pipeline, pipeline_index, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/base.py", line 112, in run_pipeline
    result = func(*args, **out) or {}
  File "/usr/local/lib/python2.7/dist-packages/social/pipeline/social_auth.py", line 20, in social_user
    social = backend.strategy.storage.user.get_social_auth(provider, uid)
  File "./social_gc/storage.py", line 105, in get_social_auth
    return cls.objects.get(provider=provider, uid=uid)
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/queryset/__init__.py", line 162, in get
    num = len(clone)
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/queryset/__init__.py", line 126, in __len__
    self._fetch_all()
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/queryset/__init__.py", line 370, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/query.py", line 480, in __iter__
    self.next_page()
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/query.py", line 452, in next_page
    transaction_id=transaction and transaction.id,
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/connection.py", line 249, in run_query
    response = self._rpc('runQuery', request, datastore_pb.RunQueryResponse)
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/connection.py", line 159, in _rpc
    data=request_pb.SerializeToString()
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/connection.py", line 134, in _request
    body=data
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 589, in new_request
    redirections, connection_type)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1609, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1351, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1307, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1127, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 453, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 734, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 621, in read
    v = self._sslobj.read(len or 1024)
SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1752)

Unfortunately, for the second, I don't have a full stacktrace handy:

[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:1752)

These errors don't happen when I am using the GCD tool. Does anyone have any idea what is happening here? Is this some sort of networking problem?

Also -- this is using gcloudoem, which most (all?) of the datastore code was taken from gcloud-python. If you do think this is only an issue with gcloudoem and not gcloud-python, let's close this issue and reopen it for the gcloudoem owner.

Unfortunately, even if we did find and fix a bug in gcloud-python, gcloud-datastore-oem / gcloudoem copied, pasted and modified the code, so our changes won't fix anything for this particular situation.

I think it's worth a quick shot at trying to reproduce on our side, but we still need to open a bug at gcloud-datastore-oem.

True -- but based on the type of error I'd expect it to also be an error in gcloud-python (unless we've already fixed it). If we are fairly confident it isn't, we can just open a bug against gcloud-datastore-oem and forget about it(may be worthwhile to do this anyways! -- the bug, not the forgetting)

Can you provide code to repro? Like, loop 5,000 times and then it happens probably?

I originally reported this problem and I can provide a little more context.

The problem happens quite regularly when I am developing code from my local machine using the production datastore. However, if I use the gcd tool locally instead of actual datastore, it never happens. It happens less regularly in production from a compute instance. It might also be worth noting that the code is run in a docker container.

I've tried installing pyOpenSSL to see if the problem is rectified but it appears to have no affect.

I will write some code using only gcloud-python to see if I can reproduce the problem. I will set it running locally first because I'm slightly concerned about the potential cost it could accumulate on gcloud. I'll report back with results.

Here is the code I am running. I will report back when something goes wrong.

@rstuart85

Thanks @rstuart85! It looks like a very simple script 👍

The script has been running for a couple of days with no errors. So there appears to be something related to my setup that is causing the SSL errors. I'll keep investigating.

Another update. It isn't just Datastore that gets the problem. Here is a stacktrace from trying to access a bucket using gcloud-python.

ERROR:django.request:Internal Server Error: /shared_image/ded1e2fa794e425b866ffdcf5ffbccda/
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/http.py", line 41, in inner
    return func(request, *args, **kwargs)
  File "./research_portal/views.py", line 623, in shared
    blob = gcloud_utils.get_object(shared_id, bucket=storage.get_bucket(settings.SHARED_BUCKET))
  File "./research_portal/gcloud_utils.py", line 31, in get_object
    return bucket.get_blob(name)
  File "/usr/local/lib/python2.7/dist-packages/gcloud/storage/bucket.py", line 190, in get_blob
    path=blob.path)
  File "/usr/local/lib/python2.7/dist-packages/gcloud/storage/connection.py", line 229, in api_request
    method=method, url=url, data=data, content_type=content_type)
  File "/usr/local/lib/python2.7/dist-packages/gcloud/storage/connection.py", line 141, in _make_request
    return self._do_request(method, url, headers, data)
  File "/usr/local/lib/python2.7/dist-packages/gcloud/storage/connection.py", line 166, in _do_request
    body=data)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 589, in new_request
    redirections, connection_type)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1609, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1351, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1307, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1127, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 453, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 734, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 621, in read
    v = self._sslobj.read(len or 1024)
SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1752)

And here is the code:

def get_object(name, bucket=None):
    retries = 0
    while True:
        try:
            if not bucket:
                bucket = storage.get_bucket(settings.UPLOADS_BUCKET)
            return bucket.get_blob(name)
        except InternalServerError as e:  # Retry when Google has an error
            logger.exception("Got a Google Storage error trying to upload a file.")
            if retries < 3:
                sleep = random.random() * 2
                time.sleep(sleep)
                retries += 1
                continue
            raise e

As you can see, I've tried to sidestep the errors using retires.

Just incase it is worth while, here is the Dockerfile I use to build the environment this runs from.

FROM ubuntu:15.04
MAINTAINER Ryan Stuart <ryan@kapiche.com>

# Install packages
RUN apt-get update && apt-get install -y \
                                build-essential \ 
                                git \
                                libffi-dev \
                                libjpeg-dev \
                                libsqlite3-dev \
                                libssl-dev \
                                libyaml-dev \
                                libxml2-dev \
                                libxslt1-dev \
                                python \
                                python-dev \
                                python-setuptools \
                                sqlite3 \
                                zlib1g-dev

# Setup ssh acces to private repos
RUN mkdir /root/.ssh/
ADD id_rsa /root/.ssh/id_rsa
RUN chmod 500 /root/.ssh/id_rsa && echo "    IdentityFile ~/.ssh/id_rsa" >> /etc/ssh/ssh_config
RUN touch /root/.ssh/known_hosts && ssh-keyscan github.com >> /root/.ssh/known_hosts

# Setup the application
RUN easy_install -U pip
RUN pip install apsw==3.8.7.3.post1
RUN mkdir /research_portal
ADD . /research_portal
WORKDIR /research_portal
RUN pip install -r requirements.txt
RUN python -m nltk.downloader punkt
ENV PYTHONUNBUFFERED=1 \
    STATIC_URL='https://storage.googleapis.com/static.kapiche.com/research/b0fa/'

And the requirements.txt:

caterpillar>=1.0.0.dev7
-e git+ssh://git@github.com/Kapiche/caterpillar-influence.git@master#egg=caterpillar-influence
-e git+ssh://git@github.com/Kapiche/caterpillar-topics.git@master#egg=caterpillar-topics
celery~=3.1.17
Django~=1.7.9
django-debug-toolbar~=1.3.2
django-gravatar2
gcloud~=0.4.2
gcloudoem>=0.1.0rc10
goose-extractor~=1.0.25
markdown~=2.6.2
opbeat
pdfminer==20140328
pycrypto~=2.6.1
pyopenssl~=0.15.1
python-social-auth~=0.2.7
requests~=2.6.0
stripe~=1.22.2
wsgiref==0.1.2
xlrd==0.9.3
uwsgi

@jcgregorio

It's worrisome. I'm curious if the error is

In the ssl module from the stdlib
In your docker environ
In some resource leak in httplib2

Maybe we should call in @jcgregorio to see if he has encountered this anywhere else with httplib2

I'm curious if turning off stdin buffering (PYTHONUNBUFFERED=1) makes a difference. I highly doubt it, but it's interesting to see.

Have you tried catching the ssl.SSLError exception and entering a debugger (I prefer ipdb over pdb but either will work).

No, but I'll give that a try. It happens both inside and outside of docker (on the production server which uses docker and on my local laptop which doesn't), so I doubt it is docker related. My local machine is OSX and the server is created using docker-machine.

Very good to know! Local laptop is running which OS?

OSX

@eric-optimizely

@eric-optimizely Just in case it's useful: I fixed this for my stuff a while ago by making the connection object threadsafe. You can see the changes here. Hope it's useful.

@rstuart85

@rstuart85 Cool, thanks! This looks like it's based on #1274 which should probably just be merged by @dhermes and @jonparrott , unless it's still incomplete or there's some other blocker. I'm not particularly interested in maintaining a fork of this library -- I just want it to work correctly on Google's own cloud platform.

@eric-optimizely

@eric-optimizely I think the outcome of the discussion here was that it should be fixed upstream. Maybe @dhermes can confirm or deny that.

Yes I am taking this on right now, the goal is to get a divorce from httplib2 and allow BYO transport with some nice defaults / examples.

I just re-deployed my app and I went from never seeing this issue, to getting thousands of them. Did anything recently change? They come in a number of flavors:

[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:1750)
[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1750)
Content purported to be compressed with gzip but failed to decompress.
[SSL: BLOCK_CIPHER_PAD_IS_WRONG] block cipher pad is wrong (_ssl.c:1750)

I was previously sharing http connections across threads. I'll make them theadsafe and see if that helps.

Making my use of httplib thread-safe fixed the issue. Not sure what I did to cause this error to pop up, but I was tinkering with some related code, so I may have caused it.

If it helps anyone else, below is the code that I use to get the authentication token. It relies on a dictionary PRIVATE_KEY to lookup the service account email and the PEM file (which should probably just be inputs into the function).

PRIVATE_KEY = {'my-project' : {'email':'serviceaccount@foo.com', 'key':'my_key.pem'}}
auth_cache = threading.local()
auth_cache_lock = threading.Lock()
def get_auth(project, force = False, scopes = None):
    global auth_cache, auth_cache_lock
    # At some point oauth2client had breaking changes. There are different 
    # copies of this library floating around, so handle whichever default imports.  
    try:
        from oauth2client.client import SignedJwtAssertionCredentials
    except ImportError as e:
        SignedJwtAssertionCredentials = None
        from oauth2client import crypt as oauth_crypt
        from oauth2client.service_account import ServiceAccountCredentials
    if project not in PRIVATE_KEY:
        raise Exception("Cannot connect to %s"%project)
    with auth_cache_lock:
        if not hasattr(auth_cache, 'auths'):
            auth_cache.auths = {}

        if not force and project in auth_cache.auths:
            return auth_cache.auths[project]
        f = None
        key = PRIVATE_KEY[project]['key']
        for path in ('updater/', '../updater/', ''):
            pfile = path + key
            try:
                f = file(pfile, 'rb')
            except IOError:
                continue
            else:
                break
        if not f:
            raise Exception("Could not find key file.")
        pem_contents = f.read()
        f.close()
        if scopes == None:
            scopes = DEFAULT_SCOPES
        # The first parameter, service_account_name, is the Email address created 
        # for the Service account. It must be the email address associated with 
        # the key that was created.
        if SignedJwtAssertionCredentials:
            # The old way of doing it.
            credential = SignedJwtAssertionCredentials(
              PRIVATE_KEY[project]['email'], pem_contents, scope = scopes)
        else:
            # The new way (improvement?)
            signer = oauth_crypt.Signer.from_string(pem_contents)
            credential = ServiceAccountCredentials(
                PRIVATE_KEY[project]['email'], signer, scopes = scopes)
            credential._private_key_pkcs8_pem = pem_contents

        http = httplib2.Http()
        http = credential.authorize(http)
        auth_cache.auths[project] = http
        return http

Superseded by the discussion in #1346

For those still following this issue, #3674 is putting the final nail in our usage of httplib2

I have the issues with WRONG_VERSION_NUMBER even when I have a single client per thread (just doing two requests one after another in the same function).

The workaround for me was to clear the connections before making each request: http.connection = {}.

chore(deps): update dependency setuptools to v60.9.3 (#1214)

chore: update sys test creds (#1214)

chore(deps): update dependency google-auth to v2.6.4 (#1214)

docs: migrate code from googleapis/python-bigquery (#18303)

Navigation Menu

Repository navigation

httplib2.Http is not thread-safe #1214

61 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

httplib2.Http is not thread-safe #1214

Description

Activity

pcostell commented on Nov 8, 2015

jgeewax commented on Nov 8, 2015

pcostell commented on Nov 8, 2015

dhermes commented on Nov 8, 2015

rstuart85 commented on Nov 10, 2015

rstuart85 commented on Nov 11, 2015

dhermes commented on Nov 11, 2015

rstuart85 commented on Nov 13, 2015

rstuart85 commented on Nov 17, 2015

rstuart85 commented on Nov 17, 2015

dhermes commented on Nov 17, 2015

dhermes commented on Nov 17, 2015

rstuart85 commented on Nov 17, 2015

dhermes commented on Nov 17, 2015

rstuart85 commented on Nov 17, 2015

61 remaining items

rstuart85 commented on Jul 19, 2016

eric-optimizely commented on Jul 19, 2016

rstuart85 commented on Jul 20, 2016

dhermes commented on Jul 20, 2016

speedplane commented on Oct 7, 2016

speedplane commented on Oct 8, 2016

theacodes commented on Dec 2, 2016

dhermes commented on Jul 27, 2017

Ark-kun commented on Jan 15, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions