I'm running google-cloud-pubsub 0.37.2 on python 3.6.6. This code is running on the google cloud container OS. The code runs fine for days at a time, pulling messages from PubSub and processing them, but it occasionally crashes as follows:
First I see the following logging messages:
Call to retryable <bound method BidiRpc.recv of <google.cloud.pubsub_v1.subscriber._protocol.bidi.ResumableBidiRpc object at 0x7fba5c24b908>> caused Can not recv() on an RPC that has never been open()ed..
Observed non-recoverable stream error Can not recv() on an RPC that has never been open()ed.
Not retrying <bound method BidiRpc.recv of <google.cloud.pubsub_v1.subscriber._protocol.bidi.ResumableBidiRpc object at 0x7fba5c24b908>> due to Can not recv() on an RPC that has never been open()ed..
RPC termination has signaled streaming pull manager shutdown.
Stopping consumer.
Then I see this stack trace:
Thread-ConsumeBidirectionalStream caught unexpected exception Can not recv() on an RPC that has never been open()ed. and will exit.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 491, in _thread_main
response = self._bidi_rpc.recv()
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 404, in recv
super(ResumableBidiRpc, self).recv)
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 393, in _recoverable
raise exc
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 385, in _recoverable
return method(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 258, in recv
'Can not recv() on an RPC that has never been open()ed.')
ValueError: Can not recv() on an RPC that has never been open()ed."
This is followed by the following logging messages:
Thread-ConsumeBidirectionalStream exiting
Stopping scheduler.
Stopping leaser.
Thread-LeaseMaintainer exiting.
Stopping dispatcher.
Exiting the QueueCallbackWorker.
Stopping heartbeater.
Thread-Heartbeater exiting.
Finished stopping manager.
After this point the application continues to run but it receives no new PubSub messages.
Please advise.
I'm running google-cloud-pubsub 0.37.2 on python 3.6.6. This code is running on the google cloud container OS. The code runs fine for days at a time, pulling messages from PubSub and processing them, but it occasionally crashes as follows:
First I see the following logging messages:
Then I see this stack trace:
This is followed by the following logging messages:
After this point the application continues to run but it receives no new PubSub messages.
Please advise.