Skip to content

Using the SinceDB Greatly increases server I/O Wait #229

@tewner

Description

@tewner

Hey guys!
We've started investigating using beaver as a light-weight log shipper. We've found that enabling "sincedb_path: /opt/beaver/var/since.db" GREATLY increases I/O Wait on our EC2 instances:

Environment:
Python 2.6.8
beaver=30

---- WITHOUT SinceDB ----
iostat:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.31    0.00    2.54    0.00    0.00   93.15

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
xvdap1            0.00     2.50    0.00    2.50     0.00    20.00    16.00     0.01    4.00   1.60   0.40

top (PEAK values): 
Cpu(s):  3.8%us,  0.9%sy,  0.0%ni, 95.2%id,  0.0%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:   7633616k total,  6705172k used,   928444k free,    81316k buffers
Swap:        0k total,        0k used,        0k free,  4186472k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                    
 4789 beaver    20   0  270m  10m 3728 S  2.3  0.1   1:22.76 beaver  

---- USING SinceDB ----
iostat: 
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.63    0.00    3.08    4.11    0.00   88.17

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
xvdap1            0.00    91.00    0.00   74.00     0.00   662.00    17.89     0.12    1.68   1.38  10.20

top (typical value):
Cpu(s):  4.2%us,  1.9%sy,  0.0%ni, 89.7%id,  4.1%wa,  0.0%hi,  0.0%si,  0.2%st
Mem:   7633616k total,  6709708k used,   923908k free,    81316k buffers
Swap:        0k total,        0k used,        0k free,  4189104k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                    
 5053 beaver    20   0  271m  12m 4132 S  4.0  0.2   1:11.89 beaver 

Is this a known issue? Has it been fixed in later versions of beaver?

Keep up the great work!
-Mike

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions