Thanks for stopping by to let us know something could be better!
PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.
I'm going to do both, thanks.
Please run down the following list and make sure you've tried the usual "quick fixes":
Done and done.
If you are still having issues, please be sure to include as much information as possible:
Environment details
-
Specify the API at the beginning of the title (for example, "BigQuery: ...")
General, Core, and Other are also allowed as types
-
OS type and version
Works on MacOS and debian linux locally and in GCB, does not work in debian linux on GKE
-
Python version and virtual environment information:
3.7.6
- google-cloud- version:
google-api-core==1.22.0 google-auth==1.20.1 google-cloud-pubsub==1.7.0 googleapis-common-protos==1.52.0 grpc-google-iam-v1==0.12.3 grpcio==1.31.0
Steps to reproduce
- Run python code locally in MacOS, confirm that it works and tests pass.
- Run python code locally in docker container, confirm that it works and tests pass.
- Run python code on GCB as part of CICD process, confirm that tests pass and build succeeds.
- Deploy python code in docker container on GKE, watch it fail!
- Shell into the kubernetes pod, run python code there, watch it fail!
Code example
def async_pull(self) -> None:
"""
Asynchronously streams messages from pubsub and calls the callback() on them
"""
streaming_pull_future = self.subscriber.subscribe(
self.subscription_path, callback=self.callback
)
log.info(f"Listening for messages on {self.subscription_path}...")
with self.subscriber:
try:
streaming_pull_future.result(timeout=self.pubsub_timeout)
except TimeoutError:
streaming_pull_future.cancel()
logging.error(
f"Streaming_pull_future timed out while listening for messages on {self.subscription_path}"
)Stack trace
08-13 17:32:27 __main__: INFO Listening for messages on subscriber...
ERROR:root:Streaming_pull_future timed out while listening for messages on subscriber
08-13 17:32:32 __main__: INFO Listening for messages on subscriber...
ERROR:google.api_core.bidi:Thread-ConsumeBidirectionalStream caught unexpected exception Cannot invoke RPC: Channel closed! and will exit.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 637, in _thread_main
self._bidi_rpc.open()
File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 280, in open
call = self._start_rpc(iter(request_generator), metadata=self._rpc_metadata)
File "/usr/local/lib/python3.7/site-packages/google/cloud/pubsub_v1/gapic/subscriber_client.py", line 1185, in streaming_pull
requests, retry=retry, timeout=timeout, metadata=metadata
File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/usr/local/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 144, in error_remapped_callable
result = callable_(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 1089, in __call__
operationses, event_handler, self._context)
File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 1183, in create
operationses_and_tags, context)
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 476, in grpc._cython.cygrpc.Channel.integrated_call
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 294, in grpc._cython.cygrpc._integrated_call
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 218, in grpc._cython.cygrpc._call
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 258, in grpc._cython.cygrpc._call
ValueError: Cannot invoke RPC: Channel closed!
Making sure to follow these steps will guarantee the quickest resolution possible.
Thanks!
Thanks for stopping by to let us know something could be better!
PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.
I'm going to do both, thanks.Please run down the following list and make sure you've tried the usual "quick fixes":
Done and done.If you are still having issues, please be sure to include as much information as possible:
Environment details
Specify the API at the beginning of the title (for example, "BigQuery: ...")
General, Core, and Other are also allowed as types
OS type and version
Works on MacOS and debian linux locally and in GCB, does not work in debian linux on GKEPython version and virtual environment information:
3.7.6google-api-core==1.22.0 google-auth==1.20.1 google-cloud-pubsub==1.7.0 googleapis-common-protos==1.52.0 grpc-google-iam-v1==0.12.3 grpcio==1.31.0Steps to reproduce
Code example
Stack trace
Making sure to follow these steps will guarantee the quickest resolution possible.
Thanks!