sqs can return empty messages if the message number is low (<1000), maybe this explains that why the last few hundred tasks takes longer time. this package set the wait time to be 0, and could potentially get empty messages. https://github.com/seung-lab/python-task-queue/blob/master/taskqueue/aws_queue_api.py#L60 here is the documentation of the long polling: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/working-with-messages.html#setting-up-long-polling as suggested, simply setting the waittime to 20 seconds should fix this.
Activity
william-silversmith commentedon Jul 24, 2020
Wait time has been set to 20 sec since a355d04