Skip to content

debugging issue with redis and queue #390

@bilsch

Description

@bilsch

Not sure if this is the best place to ask about this - feel free to bounce me to a better place.

I'm debugging an issue in the run_queue.py and have some questions.

What is the significance of transports and how does it relate to the queue size? Is this the same thing or somehow different?

[2016-02-01 10:35:07,545] DEBUG   Number of transports: 1000
[2016-02-01 10:35:07,545] DEBUG   Main consumer queue Size is: 0

I'm debugging an issue related to timeouts and redis. We were tripping the queue timeout which was previously at a very long 12 hours. I noticed large surges in log ingestion to our elk stack and tracked it back to something going on in how beaver is slurping the data in. When I shrank the timeout down to 600 seconds we ended up with this:

[2016-02-01 09:05:21,076] INFO    Starting queue consumer
[2016-02-01 09:15:24,076] INFO    Queue timeout of "600" seconds exceeded, stopping queue
[2016-02-01 09:15:26,127] INFO    Starting queue consumer
[2016-02-01 09:25:29,042] INFO    Queue timeout of "600" seconds exceeded, stopping queue
[2016-02-01 09:25:31,181] INFO    Starting queue consumer
[2016-02-01 09:35:34,050] INFO    Queue timeout of "600" seconds exceeded, stopping queue
[2016-02-01 09:35:36,241] INFO    Starting queue consumer
[2016-02-01 09:45:39,097] INFO    Queue timeout of "600" seconds exceeded, stopping queue
[2016-02-01 09:45:41,299] INFO    Starting queue consumer
[2016-02-01 09:55:44,029] INFO    Queue timeout of "600" seconds exceeded, stopping queue
[2016-02-01 09:55:46,351] INFO    Starting queue consumer

Where the queue is constantly restarting.

Any ideas on what to look at?

Beaver version: 36.0.0 and 36.0.1 - both show this problem
redis version: 3.0.5

beaver.conf:

[beaver]
redis_url = redis://xxxx.internal:6379/0
redis_namespace: logstash:beaver
queue_timeout: 600
logstash_version: 1
sincedb_path: /tmp/beaver_since.db

Not sure if the other input config sections would be related - I can post / describe those. Basically the typical tail a file pattern and ship it off with a few tags/fields.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions