Can't import google.protobuf.timestamp_pb2 from App Engine

I'm getting ImportError: No module named protobuf from the line from google.protobuf import timestamp_pb2 because the google module refers to the AppEngine SDK.

This is a well-known issue that's been around for years. Maybe you can bundle protobuf directly inside gcloud-python instead of an external dependency?

Thanks for reporting and for upgrading!

I am curious about a few things:

How do you vendor in gcloud-python for App Engine?
Do you vendor in protobuf? (AFAIK it isn't part of the GAE runtime)
Which services / gcloud-python subpackages do you use? (I'm curious which imports / import paths your code has been using.)
Are you modify the gcloud module object so that it acts like a namespace package? If no, this is the solution to your problem. I made an app that shows how to do this. The app is now out-of-date but the file that handles protobuf imports is still valid.

Are you modify the gcloud module object so that it acts like a namespace package? If no, this is the solution to your problem. I made an app that shows how to do this. The app is now out-of-date but the file that handles protobuf imports is still valid.

I don't think this shouldn't be necessary anymore, I'm interested to see if that's somehow changed.

@jonparrott

Me too. Thanks for chiming in @jonparrott I was going to ping you.

I should really write an App Engine test case.

@SaMnCo

@SaMnCo also reported this import error, it may be

we don't understand our requirements like we think we do.
pip doesn't upgrade deps in setup.py as I expect
a PYTHONPATH issue

@SaMnCo

@SaMnCo can you do a pip show protobuf, pip show gcloud in the environment where this is failing and then make sure the same versions are imported:

$ python -c 'import google.protobuf; print(google.protobuf.__version__)'
$ python -c 'import gcloud; print(gcloud.__version__)'

@dhermes

@dhermes I think I've got a similar problem as the reporter of the issue. Perhaps I can help by providing my answers to your questions:

How do you vendor in gcloud-python for App Engine?

In appengine_config.py:

from google.appengine.ext import vendor                                         
vendor.add('libs')                                                              
import google.protobuf; print(google.protobuf.__version__)                      
import gcloud; print(gcloud.__version__)

Results in:

3.0.0b2
0.10.0

Do you vendor in protobuf? (AFAIK it isn't part of the GAE runtime)

It's in libs, so yes.

Which services / gcloud-python subpackages do you use? (I'm curious which imports / import paths your code has been using.)

  from gcloud import storage
File "/x/libs/gcloud/storage/__init__.py", line 43, in <module>
  from gcloud.storage.blob import Blob
File "/x/libs/gcloud/storage/blob.py", line 27, in <module>
  from gcloud._helpers import _rfc3339_to_datetime
File "/x/libs/gcloud/_helpers.py", line 26, in <module>
  from google.protobuf import timestamp_pb2
ImportError: No module named protobuf

@kvdb

@kvdb Thanks a lot. I'm wondering now if it may be related to Darth-Vendor and namespace packages not being the best of friends. I need to dig in and reproduce. Can you confirm that's what you used to vendor in your package?

Darth is part of the SDK (from google.appengine.ext import vendor) so that's definitely what's being used. It explicitly has support for namespaces. So I'm also unsure what's going on.

Maybe it's an issue with old versions of pip? The module certainly shows up:

$ mkdir lib
$ pip install -t lib gcloud
$ cd lib/google/protobuf/
$ ls -1 *pb2.py | egrep -v unittest
any_pb2.py
api_pb2.py
descriptor_pb2.py
duration_pb2.py
empty_pb2.py
field_mask_pb2.py
source_context_pb2.py
struct_pb2.py
timestamp_pb2.py
type_pb2.py
wrappers_pb2.py

Going to run a test app in dev_appserver to verify import works.

@kvdb

OK I just confirmed this works in dev_appserver as well.

@kvdb Have you tried clearing out lib and starting fresh? What version of pip are you using?

pip 8.0.2
Yes, I cleared my virtualenv/libs before reporting.

Very strange! Does the timestamp_pb2.py file exist and just the import fails or the file never shows up?

@tseaver

@tseaver, @jonparrott installing it with pip put the package in the wrong directory.
And appengine expects it in its own directory google_appengine/google.

@yanikkoval can you give us more details? What exact pip command are you running? What's in your appengine_config.py?

It is also happening on local machine using Ubuntu 14.04 LTS - from pip freze:

gapic-google-cloud-datastore-v1==0.14.1
gapic-google-cloud-logging-v2==0.90.1
gapic-google-cloud-pubsub-v1==0.14.1
gcloud==0.18.3
google-cloud==0.22.0
google-cloud-bigquery==0.22.1
google-cloud-bigtable==0.22.0
google-cloud-core==0.22.1
google-cloud-datastore==0.22.1
google-cloud-dns==0.22.0
google-cloud-error-reporting==0.22.0
google-cloud-happybase==0.22.0
google-cloud-language==0.22.2
google-cloud-logging==0.22.0
google-cloud-monitoring==0.22.0
google-cloud-pubsub==0.22.0
google-cloud-resource-manager==0.22.0
google-cloud-runtimeconfig==0.22.0
google-cloud-storage==0.22.0
google-cloud-translate==0.22.0
google-cloud-vision==0.22.0
grpc-google-cloud-datastore-v1==0.14.0
grpc-google-cloud-logging-v2==0.90.0
grpc-google-cloud-pubsub-v1==0.14.0

Traceback (most recent call last):
  File "./src/main.py", line 9, in <module>
    from google.cloud import logging, pubsub
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/logging/__init__.py", line 18, in <module>
    from google.cloud.logging.client import Client
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/logging/client.py", line 32, in <module>
    from google.cloud.client import JSONClient
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/client.py", line 21, in <module>
    from google.cloud._helpers import _determine_default_project
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/_helpers.py", line 30, in <module>
    from google.protobuf import timestamp_pb2
ImportError: cannot import name timestamp_pb2

@ensonic

@ensonic Run import google; print(google.__path__) so see what the problem is. The usual suspect here is that you have a "bad" package breaking the google namespace. (The fact that you have the very old gcloud co-tenant with google-cloud isn't a good sign.)

@dhermes

@dhermes

python -c "import google; print(google.__path__)"
['/usr/local/lib/python2.7/dist-packages/google']

which ones should I get rid of - the 'grpc-google-' or 'gapic-google-'

Neither need be gotten rid of AFAIK. From there you'd want to poke around in /usr/local/lib/python2.7/dist-packages/google/__init__.py and see if it is a namespace package or otherwise (you also need to look around for (maybe bad?) .nspkg files in your site-packages

Speech API: enhanced model and recognition metadata [(#1436)](GoogleC…

fix(rest): Use strict encoding to lowercase query string bools (#1436)

fix: fix user cred universe domain issue (#1436)

docs: created samples for load table and create table from schema file (

feat: Support dry_run in `to_pandas()` (#1436)

docs: migrate code from googleapis/python-bigquery (#18303)

Navigation Menu

Repository navigation

Can't import google.protobuf.timestamp_pb2 from App Engine #1436

62 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Can't import google.protobuf.timestamp_pb2 from App Engine #1436

Description

Activity

dhermes commented on Feb 1, 2016

theacodes commented on Feb 1, 2016

dhermes commented on Feb 1, 2016

theacodes commented on Feb 1, 2016

dhermes commented on Feb 9, 2016

dhermes commented on Feb 11, 2016

kvdb commented on Feb 18, 2016

dhermes commented on Feb 18, 2016

theacodes commented on Feb 18, 2016

dhermes commented on Feb 19, 2016

dhermes commented on Feb 19, 2016

kvdb commented on Feb 21, 2016

dhermes commented on Feb 21, 2016

62 remaining items

yanik-ai commented on Sep 5, 2016

theacodes commented on Sep 6, 2016

ensonic commented on Feb 21, 2017

dhermes commented on Feb 21, 2017

ensonic commented on Feb 21, 2017

dhermes commented on Feb 21, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions