I am just working on getting beaver set up to import data from my remote hosts into a logstash server. I am currently working on apache access logs. My intent is to write out apache logs in the json_event format to ease importing into logstash (http://cookbook.logstash.net/recipes/apache-json-logs/), but I'm not able to get those raw events into logstash correctly.
My beaver.conf:
redis_url = redis://127.0.0.1:6379/0
redis_namespace: logstash:beaver
[/var/log/httpd/kibana_access.json]
type: apache
tags: access
format: rawjson
vhost config in case it is needed:
LogFormat "{ \"@timestamp\": \"%{%Y-%m-%dT%H:%M:%S%z}t\", \"@fields\": { \"proxied_for\": \"%{X-Forwarded-For}i\", \"client\": \"%a\", \"duration_usec\": %D, \"status\": %>s, \"request\": \"%U%q\", \"method\": \"%m\", \"referrer\": \"%{Referer}i\", \"response_size\": \"%b\", \"user_agent\": \"%{User-agent}i\" } }" logstash_json
CustomLog /var/log/httpd/kibana_access.json logstash_json
logstash has a simple redis input and elasticsearch output for the type apache. Logstash is recording the json document as the @message rather than the fields specified in the log message.
Sample logfile:
{ "@timestamp": "2013-04-01T17:47:54+0000", "@fields": { "proxied_for": "-", "client": "172.31.127.202", "duration_usec": 29058, "status": 200, "request": "/api/graph/count/1800000/eyJzZWFyY2giOiJAdHlwZT1hcGFjaGUiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjE3MjgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzNjQ4Mzg0NzMwNjJ9/2?_=1364838474512", "method": "GET", "referrer": "http://ue1a-logstash02/", "response_size": "1096", "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.43 Safari/537.31" } }
Any help as to where I am going wrong would be greatly appreciated.
I am just working on getting beaver set up to import data from my remote hosts into a logstash server. I am currently working on apache access logs. My intent is to write out apache logs in the json_event format to ease importing into logstash (http://cookbook.logstash.net/recipes/apache-json-logs/), but I'm not able to get those raw events into logstash correctly.
My beaver.conf:
vhost config in case it is needed:
logstash has a simple redis input and elasticsearch output for the type apache. Logstash is recording the json document as the @message rather than the fields specified in the log message.
Sample logfile:
Any help as to where I am going wrong would be greatly appreciated.