Not sure if this is the best place to ask about this - feel free to bounce me to a better place.
I'm debugging an issue in the run_queue.py and have some questions.
What is the significance of transports and how does it relate to the queue size? Is this the same thing or somehow different?
[2016-02-01 10:35:07,545] DEBUG Number of transports: 1000
[2016-02-01 10:35:07,545] DEBUG Main consumer queue Size is: 0
I'm debugging an issue related to timeouts and redis. We were tripping the queue timeout which was previously at a very long 12 hours. I noticed large surges in log ingestion to our elk stack and tracked it back to something going on in how beaver is slurping the data in. When I shrank the timeout down to 600 seconds we ended up with this:
[2016-02-01 09:05:21,076] INFO Starting queue consumer
[2016-02-01 09:15:24,076] INFO Queue timeout of "600" seconds exceeded, stopping queue
[2016-02-01 09:15:26,127] INFO Starting queue consumer
[2016-02-01 09:25:29,042] INFO Queue timeout of "600" seconds exceeded, stopping queue
[2016-02-01 09:25:31,181] INFO Starting queue consumer
[2016-02-01 09:35:34,050] INFO Queue timeout of "600" seconds exceeded, stopping queue
[2016-02-01 09:35:36,241] INFO Starting queue consumer
[2016-02-01 09:45:39,097] INFO Queue timeout of "600" seconds exceeded, stopping queue
[2016-02-01 09:45:41,299] INFO Starting queue consumer
[2016-02-01 09:55:44,029] INFO Queue timeout of "600" seconds exceeded, stopping queue
[2016-02-01 09:55:46,351] INFO Starting queue consumer
Where the queue is constantly restarting.
Any ideas on what to look at?
Beaver version: 36.0.0 and 36.0.1 - both show this problem
redis version: 3.0.5
beaver.conf:
[beaver]
redis_url = redis://xxxx.internal:6379/0
redis_namespace: logstash:beaver
queue_timeout: 600
logstash_version: 1
sincedb_path: /tmp/beaver_since.db
Not sure if the other input config sections would be related - I can post / describe those. Basically the typical tail a file pattern and ship it off with a few tags/fields.
Not sure if this is the best place to ask about this - feel free to bounce me to a better place.
I'm debugging an issue in the run_queue.py and have some questions.
What is the significance of transports and how does it relate to the queue size? Is this the same thing or somehow different?
I'm debugging an issue related to timeouts and redis. We were tripping the queue timeout which was previously at a very long 12 hours. I noticed large surges in log ingestion to our elk stack and tracked it back to something going on in how beaver is slurping the data in. When I shrank the timeout down to 600 seconds we ended up with this:
Where the queue is constantly restarting.
Any ideas on what to look at?
Beaver version: 36.0.0 and 36.0.1 - both show this problem
redis version: 3.0.5
beaver.conf:
Not sure if the other input config sections would be related - I can post / describe those. Basically the typical tail a file pattern and ship it off with a few tags/fields.