Repository navigation
Number of acked messages does not correspond to metrics in Stackdriver #4214
Description
I have this trivial script that just consumes messages from PubSub
from google.cloud import pubsub
subscriber = pubsub.SubscriberClient()
subscription_path = 'projects/myproject/subscriptions/mysub'
def callback(msg):
msg.ack()
subscriber.subscribe(subscription_path, callback=callback)I'd expect that all messages would be acked without any problem, but when I look at Stackdriver, I see a huge number of Deadline Operations and very few StreamingPull Acknowledge Operations. See screenshot of Stackdriver https://imgur.com/Z4ZK2l1 (couldn't upload it here for unknown reasons).
Is this expected? Is it fault of Stackdriver or can there be a problem with acking?
Tested on Google Dataproc with google-cloud-pubsub-0.28.4.
Thanks!
Activity
Marigold commentedon Oct 18, 2017
Forgot to mention that this happens for topic that has 7 days old messages, could that be an issue? I tried it on queue that is not full and the difference between acked and pulled operations were about 10% (way lower than in previous case, but still unexpected).
lukesneeringer commentedon Oct 19, 2017
This looks like it might be related to this question on Stack Overflow.
I will look into this further.
Marigold commentedon Oct 19, 2017
Thanks. We have a similar use case to batching #4141 and weren't able to get even your sample code working (ratio of acked / pulled messages was extremely low). Had to switch to 0.27.0 with sync pulls which work just fine.
dhermes commentedon Dec 15, 2017
I am going to close this, since it seems very similar to other reported issue and the
Consumer.pause()andConsumer.resume(), which was resolved by #4558.@Marigold If I'm closing this in error, I'm happy to work with you to get a case we can reproduce. (Having to wait until messages are 7 days queued will obviously take longer to reproduce 😀.)