2019独角兽企业重金招聘Python工程师标准>>> 
Logstash
使用logstash收集apache日志
apache日志的格式：
83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png
HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"1、input { }
input {
file {
path => '/var/log/apache.log'
start_position => 'beginning'
}
}2、filter { }
使用 '%{COMBINEDAPACHELOG}' grok 模式：
filter {
grok {
match => {"message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => 'clientip'
}
date {
match => ["timestamp","dd/MMM/yyyy:HH:mm:ss Z"]
}
}IP Address --> clientip User ID -->ident User Authentication -->auth timestamp-->timestamp
HTTP Verb --> verb Request body --> request HTTP Version --> httpversion Http Status Code -->response Bytes served --> bytes Referrer URL --> referrer User Agent -- > agent
经过grok处理后的，具有如下JSON格式：
{
"clientip" : "83.149.9.216",
"ident" : ,
"auth" : ,
"timestamp" : "04/Jan/2015:05:13:42 +0000",
"verb" : "GET",
"request" : "/presentations/logstash-monitorama-2013/images/kibana-search.png",
"httpversion" : "HTTP/1.1",
"response" : "200",
"bytes" : "203023",
"referrer" : "http://semicomplete.com/presentations/logstash-monitorama-2013/",
"agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
}3、output { }
output {
elasticsearch {
protocol => 'http'
}
stdout {}
}使用如下命令去验证你的配置：
bin/logstash -f logstash.conf --configtest如果验证没有问题，使用如下命令启动logstash:
bin/logstash -f logstash.conf使用基于grok的字段在ES里面查询：
curl -XGET 'curl -XGET 'localhost:9200/logstash-2015.08.25/_search?q=geoip.city_name=Beijing'