Skip to content

Error running "from google.cloud import storage" command #4039

Closed
Closed
Error running "from google.cloud import storage" command#4039
Assignees
dhermes
Labels
packaging

Description

@ep-tpat
ep-tpat
opened on Sep 22, 2017

Thank you for reporting an issue to google-cloud-python!

If you are reporting an issue or requesting a feature, please search the
existing open and closed issues to see if there is already work being done.

If you can provide us with as much of the following information as possible it
will help us identify the cause of your issue more quickly.

Hello, I'm hoping you can help. I'm trying to run the Python Client Library for Google Compute and running into issues. See below for my information. Thanks for any help you can provide.

1. OS type and version
$ cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)

2. Python version and virtual environment information python --version
$ python3 -V
Python 3.4.5

3. google-cloud-python version pip show google-cloud, pip show google-<service> or pip freeze
$ pip3 freeze | grep ^google
google-auth==1.1.1
google-cloud==0.27.0
google-cloud-bigquery==0.26.0
google-cloud-bigtable==0.26.0
google-cloud-core==0.26.0
google-cloud-datastore==1.2.0
google-cloud-dns==0.26.0
google-cloud-error-reporting==0.26.0
google-cloud-language==0.27.0
google-cloud-logging==1.2.0
google-cloud-monitoring==0.26.0
google-cloud-pubsub==0.27.0
google-cloud-resource-manager==0.26.0
google-cloud-runtimeconfig==0.26.0
google-cloud-spanner==0.26.0
google-cloud-speech==0.28.0
google-cloud-storage==1.3.2
google-cloud-translate==1.1.0
google-cloud-videointelligence==0.25.0
google-cloud-vision==0.26.0
google-gax==0.15.15
google-resumable-media==0.2.3
googleapis-common-protos==1.5.2

4. Stacktrace if available
- See below

5. Steps to reproduce
- from a fresh CentOS 7 VM in Google Compute, I ran the following commands:
sudo yum -y install epel-release
sudo yum -y install python34
sudo yum -y install python34-devel
sudo yum -y install python34-setuptools
sudo python3 /usr/lib/python3.4/site-packages/easy_install.py pip
sudo pip3 install --upgrade pip
pip3 install --upgrade google-cloud

- from python IDLE, ran the following command:
from google.cloud import storage
- error received:
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3.4/site-packages/google/cloud/storage/init.py", line 37, in
from google.cloud.storage.batch import Batch
File "/usr/lib/python3.4/site-packages/google/cloud/storage/batch.py", line 29, in
from google.cloud import _helpers
File "/usr/lib/python3.4/site-packages/google/cloud/_helpers.py", line 33, in
from google.protobuf import duration_pb2
ImportError: No module named 'google.protobuf'

6. Code example
N/A
Using GitHub flavored markdown can help make your request clearer.
See: https://guides.github.com/features/mastering-markdown/

Activity

  1. lukesneeringer commented on Sep 23, 2017

    @lukesneeringer
    lukesneeringer
    on Sep 23, 2017
    Contributor

    Is protobuf installed? (Your pip freeze command obscures this.)

  2. ep-tpat commented on Sep 25, 2017

    @ep-tpat
    ep-tpat
    on Sep 25, 2017
    Author

    Hi Luke. Yes, protobuf seems to be installed:

    $ pip3 freeze | egrep '^google|protobuf'
    google-auth==1.1.1
    google-cloud==0.27.0
    google-cloud-bigquery==0.26.0
    google-cloud-bigtable==0.26.0
    google-cloud-core==0.26.0
    google-cloud-datastore==1.2.0
    google-cloud-dns==0.26.0
    google-cloud-error-reporting==0.26.0
    google-cloud-language==0.27.0
    google-cloud-logging==1.2.0
    google-cloud-monitoring==0.26.0
    google-cloud-pubsub==0.27.0
    google-cloud-resource-manager==0.26.0
    google-cloud-runtimeconfig==0.26.0
    google-cloud-spanner==0.26.0
    google-cloud-speech==0.28.0
    google-cloud-storage==1.3.2
    google-cloud-translate==1.1.0
    google-cloud-videointelligence==0.25.0
    google-cloud-vision==0.26.0
    google-gax==0.15.15
    google-resumable-media==0.2.3
    googleapis-common-protos==1.5.2
    protobuf==3.4.0
    $ python3
    Python 3.4.5 (default, May 29 2017, 15:17:55)
    [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
    Type "help", "copyright", "credits" or "license" for more information.

    from google.cloud import storage
    Traceback (most recent call last):
    File "", line 1, in
    File "/usr/lib/python3.4/site-packages/google/cloud/storage/init.py", line 37, in
    from google.cloud.storage.batch import Batch
    File "/usr/lib/python3.4/site-packages/google/cloud/storage/batch.py", line 29, in
    from google.cloud import _helpers
    File "/usr/lib/python3.4/site-packages/google/cloud/_helpers.py", line 33, in
    from google.protobuf import duration_pb2
    ImportError: No module named 'google.protobuf'

  3. lukesneeringer commented on Sep 27, 2017

    @lukesneeringer
    lukesneeringer
    on Sep 27, 2017
    Contributor

    Hmm. I just tried on my machine with a new virtualenv and was successful.
    Are you able to import google.protobuf directly? (Just trying to think of things to eliminate.)

  4. dhermes commented on Sep 27, 2017

    @dhermes
    dhermes
    on Sep 27, 2017
    Contributor

    We've seen this issue quite a lot and it's almost always caused by a "bad" install, i.e. one where the correct .pth files don't get created.

    @ep-tpat Try installing into a virtualenv, so you can verify the install works. Then from your global install (i.e. /usr/lib/python3.4/site-packages) you can work on fixing the issue. (Usually just pip install --ignore-installed protobuf will work.)

    I am pre-emptively closing this since it's not an issue in our code (i.e. we can install it on lots of different machines) and it's a common issue we've seen with pip usage in the wild. I'm happy to re-open if this turns out to be something else.

    Feel free to keep asking questions in this issue though, we're happy to help.

  5. dhermes
    closed this as completedon Sep 27, 2017
  6. dhermes
    added
    packaging
    on Sep 27, 2017

  7. ep-tpat commented on Sep 28, 2017

    @ep-tpat
    ep-tpat
    on Sep 28, 2017
    Author

    Thanks for the help, guys. After installing virtualenv, it worked perfectly. So....any chance you can point me in the right direction to get the issue ironed out not using virtualenv? This was a freshly spun up Google VM running CentOS 7. I didn't do anything else to it other than install Python3, Pip and Google Cloud. Also note, I'm new to Python (only about 2-3 months). Commands below:

    sudo yum -y install epel-release
    sudo yum -y install python34
    sudo yum -y install python34-devel
    sudo yum -y install python34-setuptools
    sudo python3 /usr/lib/python3.4/site-packages/easy_install.py pip
    sudo pip3 install --upgrade pip
    pip3 install --upgrade google-cloud

    Thanks for any help you can offer.

  8. dhermes commented on Sep 28, 2017

    @dhermes
    dhermes
    on Sep 28, 2017
    Contributor

    @ep-tpat

    1. System Python isn't really a great place to set up an environment, pyenv / virtualenv really are good options
    2. Did you try pip install --ignore-installed protobuf (will need to be with sudo if you use the yum installed Python)?
    3. In an interpreter import google and print out google.__path__, it should give you an idea of where it is installed.
    4. Look for "bad" namespace files in your site-packages (e.g. I see .../site-packages/protobuf-3.4.0-py3.4-nspkg.pth, but I want that there, i.e. it isn't bad / dangling from an uninstalled package)
    5. Look for directories in your site-packages that start with google, but you didn't expect to be there (just ``.../site-packages/google/`).

  9. ep-tpat commented on Sep 28, 2017

    @ep-tpat
    ep-tpat
    on Sep 28, 2017
    Author

    @dhermes

    I'll stick with your first point. If virtualenv is a good option then I'll just move forward with that. Thanks for everyone's help!

  10. rsmith49 commented on Oct 31, 2017

    @rsmith49
    rsmith49
    on Oct 31, 2017

    @dhermes I'm also running into this problem (but with google-cloud-language), not sure if I should make a new issue or just continue on this thread.

    OS Type and Version
    Using AWS Elastic Beanstalk, EC2 instance with:
    Amazon Linux AMI release 2017.03

    Python Version and Virtual Environment Info
    Python 3.4.3
    Not sure what virtualenv version the EC2 instance is running, Elastic Beanstalk creates its own virtual environments on application deployment

    google-cloud-python version
    Current pip freeze output:

    airbrake==2.1.0
appnope==0.1.0
APScheduler==3.3.1
awscli==1.11.126
awsebcli==3.10.4
backports-abc==0.5
backports.weakref==1.0rc1
bleach==1.5.0
blessed==1.14.2
boto==2.48.0
boto3==1.4.4
botocore==1.5.89
bz2file==0.98
cachetools==2.0.1
cement==2.8.2
certifi==2017.7.27.1
chardet==3.0.4
click==6.7
colorama==0.3.7
cycler==0.10.0
decorator==4.1.2
dill==0.2.7.1
docker-py==1.7.2
dockerpty==0.4.1
docopt==0.6.2
docutils==0.13.1
entrypoints==0.2.3
flasgger==0.6.5
Flask==0.12
Flask-API==0.7.1
future==0.16.0
gensim==2.0.0
google-api-python-client==1.6.2
google-auth==1.1.1
google-auth-httplib2==0.0.2
google-cloud-core==0.27.1
google-cloud-language==0.29.0
google-cloud-storage==1.0.0
google-gax==0.15.15
googleapis-common-protos==1.5.3
grpcio==1.4.0
gunicorn==19.7.0
h5py==2.7.0
html5lib==0.9999999
httplib2==0.10.3
idna==2.6
ipykernel==4.6.1
ipython==6.1.0
ipython-genutils==0.2.0
ipywidgets==7.0.0
itsdangerous==0.24
jedi==0.10.2
Jinja2==2.9.5
jmespath==0.9.3
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.1.0
jupyter-console==5.2.0
jupyter-core==4.3.0
Keras==2.0.6
Markdown==2.6.8
MarkupSafe==1.0
matplotlib==2.0.0
mistune==0.7.4
nbconvert==5.2.1
nbformat==4.4.0
nltk==3.2.4
notebook==5.0.0
numpy==1.13.1
oauth2client==3.0.0
pandas==0.20.3
pandocfilters==1.4.2
pathspec==0.5.0
pexpect==4.2.1
pickleshare==0.7.4
ply==3.8
prompt-toolkit==1.0.15
protobuf==3.4.0
ptyprocess==0.5.2
pyasn1==0.3.7
pyasn1-modules==0.1.5
pyenchant==1.6.8
Pygments==2.2.0
pyparsing==2.2.0
python-dateutil==2.6.0
pytz==2016.10
PyYAML==3.12
pyzmq==16.0.2
qtconsole==4.3.1
requests==2.18.4
rsa==3.4.2
s3transfer==0.1.10
scikit-learn==0.18.1
scipy==0.19.1
semantic-version==2.5.0
simplegeneric==0.8.1
six==1.11.0
smart-open==1.5.3
tabulate==0.7.5
tensorflow==1.2.1
termcolor==1.1.0
terminado==0.6
testpath==0.3.1
Theano==0.9.0
tornado==4.5.1
traitlets==4.3.2
typing==3.6.2
tzlocal==1.4
uritemplate==3.0.0
urllib3==1.22
wcwidth==0.1.7
websocket-client==0.44.0
Werkzeug==0.11.15
widgetsnbextension==3.0.1

    Previously had been using:

    google-api-python-client==1.6.2
google-auth==1.0.1
google-auth-httplib2==0.0.2
google-cloud-core==0.24.1
google-cloud-language==0.24.1
google-cloud-storage==1.0.0
googleapis-common-protos==1.5.2
protobuf==3.0.0b2.post2

    which worked fine, but now need to upgrade to the latest google-cloud-language version, and the protobuf==3.0.0b2.post2 workaround is too outdated to work with the latest version.

    Stacktrace

    Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/google/cloud/language.py", line 17, in <module>
    from google.cloud.language_v1 import LanguageServiceClient
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/google/cloud/language_v1/__init__.py", line 17, in <module>
    from google.cloud.language_v1 import types
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/google/cloud/language_v1/types.py", line 18, in <module>
    from google.gax.utils.messages import get_messages
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/google/gax/__init__.py", line 44, in <module>
    from google.rpc import code_pb2
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/google/rpc/code_pb2.py", line 6, in <module>
    from google.protobuf.internal import enum_type_wrapper
ImportError: No module named 'google.protobuf'

    Steps to Reproduce

    Since Elastic Beanstalk automatically installs packages on deployment, I have been ssh'ing into the EC2 instance, running source /opt/python/run/venv/lib/activate, and playing around with different combinations/versions of installing protobuf and google-cloud-language.

    eb ssh
cd /opt/python/
source run/venv/bin/activate
python
>>> from google.cloud import language

    I have also tried uninstalling all packages and reinstalling using --ignore-installed, which also did not work (same error):

    pip freeze | xargs pip uninstall -y
pip install --ignore-installed protobuf
pip install google-cloud-lanuguage
python
>>> from google.cloud import language

  11. dhermes commented on Oct 31, 2017

    @dhermes
    dhermes
    on Oct 31, 2017
    Contributor

    @ep-tpat That is quite a lot of packages! Part of the problem may also be that we don't support Python 3.4, so there may be some unknown issues. What exactly does protobuf==3.0.0b2.post2 work around?

    In addition to uninstalling and reinstalling using --ignore-installed, you should for path files (/opt/python/run/venv/local/lib/python3.4/site-packages/*pth) that seem out of place. This google package confusion can happen if there is a mix of installed eggs (you don't want any when dealing with namespace packages) and "normal" (i.e. pip install-ed) packages.

  12. rsmith49 commented on Oct 31, 2017

    @rsmith49
    rsmith49
    on Oct 31, 2017

    @dhermes Not sure what Issue I was reading, but I came across a potential fix where setting the protobuf version to 3.0.0b2.post2 solved the problem, so I had been using that for the past few months. However, I guess the latest google-cloud-language version needs a later version of protobuf.

    As far as files that seem out of place, do you have any suggestions on what to look out for? I am not that familiar with the package dependency structure used by pip/python

  13. rsmith49 commented on Oct 31, 2017

    @rsmith49
    rsmith49
    on Oct 31, 2017

    Here is the output of ls /opt/python/run/venv/local/lib/python3.4/site-packages/*pth

    run/venv/local/lib/python3.4/site-packages/googleapis_common_protos-1.5.3-py3.4-nspkg.pth
run/venv/local/lib/python3.4/site-packages/google_auth-1.1.1-py2.7-nspkg.pth
run/venv/local/lib/python3.4/site-packages/google_cloud_core-0.27.1-py3.6-nspkg.pth
run/venv/local/lib/python3.4/site-packages/google_cloud_language-0.29.0-py3.6-nspkg.pth
run/venv/local/lib/python3.4/site-packages/google_cloud_storage-1.0.0-py3.6-nspkg.pth
run/venv/local/lib/python3.4/site-packages/google_gax-0.15.15-py3.4-nspkg.pth

  14. dhermes commented on Oct 31, 2017

    @dhermes
    dhermes
    on Oct 31, 2017
    Contributor

    When you uninstall, just make sure all the .pth files are deleted too. (The "bad" one is usually just easy-install.pth when a namespace package comes in as part of an egg.) Similarly, you may want to delete any "dist info" or .egg files that don't seem to belong.

    I can confirm the issue isn't Python 3.4 (at least for installing and importing):

    >>> # After: 
>>> # $ python3.4 -m virtualenv venv
>>> # $ venv/bin/pip install google-cloud-language
>>> # $ venv/bin/python
>>> import sys
>>> sys.version
'3.4.7 (default, Oct 10 2017, 10:51:21) \n[GCC 5.4.0 20160609]'
>>> sys.executable
'.../venv/bin/python'
>>> import google.cloud.language
>>> google.cloud.language
<module 'google.cloud.language' from '.../venv/lib/python3.4/site-packages/google/cloud/language.py'>
>>> import google.protobuf
>>> google.protobuf
<module 'google.protobuf' from '.../venv/lib/python3.4/site-packages/google/protobuf/__init__.py'>

  15. rsmith49 commented on Nov 8, 2017

    @rsmith49
    rsmith49
    on Nov 8, 2017

    Apparently the issue had to do with Amazon Linux AMI causing pip to import some packages to venv/lib/python3.4/site-packages/ and some to venv/lib64/python3.4/site-packages, which I expect then caused problems because the google/cloud modules were under lib and the google/protobuf module was under lib64.

    Here is a relevant link tracking the issue for pip, just in case anyone else runs into this issue with AWS EC2 instances: pypa/pip#4464, and I solved this by including:

    /opt/python/run/venv/bin/pip install --target=/opt/python/run/venv/lib/python3.4/site-packages/ --upgrade protobuf
/opt/python/run/venv/bin/pip install --ignore-installed google-cloud

    as container commands in .ebextensions

  16. harshit0605 commented on Jun 14, 2018

    @harshit0605
    harshit0605
    on Jun 14, 2018

    Hi,
    I am not able to import google.cloud.vision package in python. The stack trace is given below:

    Traceback (most recent call last):
  File "E:\ANACONDA\lib\site-packages\IPython\core\interactiveshell.py", line 2862, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-417-db2467e60f89>", line 1, in <module>
    import google.cloud.vision
  File "E:\PYCHARM\PyCharm Community Edition 2017.2.3\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "E:\ANACONDA\lib\site-packages\google\cloud\vision.py", line 18, in <module>
    __version__ = get_distribution('google-cloud-vision').version
  File "E:\ANACONDA\lib\site-packages\pkg_resources\__init__.py", line 564, in get_distribution
    except ImportError:
  File "E:\ANACONDA\lib\site-packages\pkg_resources\__init__.py", line 436, in get_provider
    if provMac.group(1) != reqMac.group(1) or \
  File "E:\ANACONDA\lib\site-packages\pkg_resources\__init__.py", line 984, in require
    or dist.py_version == self.python
  File "E:\ANACONDA\lib\site-packages\pkg_resources\__init__.py", line 875, in resolve
    distributions.sort()
pkg_resources.ContextualVersionConflict: (requests 2.14.2 (e:\anaconda\lib\site-packages), Requirement.parse('requests<3.0.0dev,>=2.18.0'), {'google-api-core'})

    Please help me out.

  17. tseaver commented on Jun 14, 2018

    @tseaver
    tseaver
    on Jun 14, 2018
    Contributor

    @harshitHadoop You've tagged onto a closed (and not exactly related) issue here. Please open a new issue, including showing the version of Python, how you've installed google-cloud-vision, etc.

  18. JustinBeckwith
    assigned
    dhermes
    on Feb 1, 2021
  19. parthea
    added a commit that references this issue on Sep 22, 2023

    chore: move samples from python-docs-sample (#33)

    69c4425
  20. parthea
    added 2 commits that reference this issue on Oct 21, 2023

    Update dependency google-cloud-pubsub to v1.6.0 [(#4039)](GoogleCloud…

    a294267

    Update dependency google-cloud-pubsub to v1.6.0 [(#4039)](GoogleCloud…

    7db5035
  21. parthea
    added 2 commits that reference this issue on Mar 2, 2026

    samples: add samples from pubsub/cloud-client (#134)

    8c90f22

    Update dependency google-cloud-pubsub to v1.6.0 [(#4039)](GoogleCloud…

    1bc8778
  22. parthea
    added a commit that references this issue on Mar 9, 2026

    samples: add cloud client samples from python-docs-samples (#626)

    2b99c9d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

packaging

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions