You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Magically take care of threading issues, please. #926
I don't want to worry about connections and threads. I want to be able to pass API classes from thread to thread and have the classes take care of threading issues.
Yes, when I'm calling the code on app engine, my http handlers will be called from multiple threads. I can't share a connection/http object between them all because the underlying http object is not thread safe. I don't want to create a new http object for each request because then I lose the benefit of connection: keep-alive.
Given that we are moving to re-bind domain objects to clients (where clients encapsulate a connection), satisfying this goal is going to be tricky, unless the "foreign" thread is willing to pass in its client to methods which need a client.
A bigger issue may also be that httplib2 is not thread-safe. @tseaver You should check out how easily @mwitkow-io was able to make requests work in our library in a threaded environment in #908.
I do know a little bit about this. For one, httplib2 is not thread safe. I have been advocating for a "divorce" from httplib2 or at least a way to give people other options (see googleapis/oauth2client#128). I feel like large structural changes should wait until googleapis/oauth2client#212 is complete though.
We already enable this with our library (#908) but it doesn't solve the auth problem for people until oauth2client catches up.
@SurferJeffAtGoogle It may be "faster" to avoid calling credentials = oauth2client.GoogleCredentials.get_application_default() in every single thread, but I think it's not so bad an approach.
We could manually add a connection or client pool that just binds to a thread local for the user?
Activity
tmatsuo commentedon Jun 17, 2015
+1 I really like this idea
dhermes commentedon Jun 17, 2015
I like the idea too, but it is pretty abstract. Can you give an example where you have to deal with threading that you'd like the library to handle?
SurferJeffAtGoogle commentedon Jun 17, 2015
Yes, when I'm calling the code on app engine, my http handlers will be called from multiple threads. I can't share a connection/http object between them all because the underlying http object is not thread safe. I don't want to create a new http object for each request because then I lose the benefit of connection: keep-alive.
SurferJeffAtGoogle commentedon Jun 17, 2015
Also, I don't want to create a new http object for each request because then it has to re-authenticate.
dhermes commentedon Jun 17, 2015
In the short term I'd like to point to #908 and the custom code there using
requests.We'd like to address this, but if your needs are pressing, you may be happy with a short-term hack in the style described in that issue.
tseaver commentedon Jun 23, 2015
Given that we are moving to re-bind domain objects to clients (where clients encapsulate a connection), satisfying this goal is going to be tricky, unless the "foreign" thread is willing to pass in its client to methods which need a client.
dhermes commentedon Jun 23, 2015
A bigger issue may also be that
httplib2is not thread-safe. @tseaver You should check out how easily @mwitkow-io was able to makerequestswork in our library in a threaded environment in #908.theacodes commentedon Sep 25, 2015
Pinging this issue as threadsafety for gcloud clients recently came up in discussion.
What's the status on this? Are client objects not threadsafe? If not, what needs to be done to allow that?
Also, if not, can we explicitly document that?
dhermes commentedon Sep 25, 2015
I think the best way to know is to write threaded code and see what happens. Anyone want to volunteer?
theacodes commentedon Sep 25, 2015
tentatively raises hand
I'll give a shot when I get a bit of free time.
dhermes commentedon Sep 25, 2015
I do know a little bit about this. For one,
httplib2is not thread safe. I have been advocating for a "divorce" fromhttplib2or at least a way to give people other options (see googleapis/oauth2client#128). I feel like large structural changes should wait until googleapis/oauth2client#212 is complete though.We already enable this with our library (#908) but it doesn't solve the auth problem for people until
oauth2clientcatches up.SurferJeffAtGoogle commentedon Sep 25, 2015
I coped with this issue by using thread-local instances of httplib2.Http.
https://github.com/GoogleCloudPlatform/pubsub-shout-csharp/blob/master/appengine-python-flask/pubsub.py
dhermes commentedon Sep 25, 2015
@SurferJeffAtGoogle It may be "faster" to avoid calling
credentials = oauth2client.GoogleCredentials.get_application_default()in every single thread, but I think it's not so bad an approach.We could manually add a connection or client pool that just binds to a thread local for the user?
25 remaining items
Fix typo [(#926)](GoogleCloudPlatform/python-docs-samples#926)
Fix typo [(#926)](GoogleCloudPlatform/python-docs-samples#926)
Fix typo [(#926)](GoogleCloudPlatform/python-docs-samples#926)
Fix typo [(#926)](GoogleCloudPlatform/python-docs-samples#926)
chore(main): release 0.29.0 (#926)
chore(docs): add preview docstrings to v3 client (#926)
fix(bazel): Re-enable Python µgen integration tests post monolith rul…
chore: release 0.50.1 (#928)
feat: use generator for stream results (#926)
chore: rename rst files to avoid conflict with service names (#926)
chore(docs): add preview docstrings to v3 client (#926)
chore(main): release 2.18.0 (#926)
chore(deps): update all dependencies (#926)
chore(deps): update all dependencies (#926)
docs: migrate code from googleapis/python-bigquery (#18303)