elk分析nginx日志

在生产环境中，nginx日志格式往往使用的是自定义的格式，我们需要把logstash中的message结构化后再存储，方便kibana的搜索和统计，因此需要对message进行解析。

logstash自带的grok正则中有Apache的标准日志格式：

COMMONAPACHELOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

vim /developer/java/logstash-6.3.2/config/logstash.conf

input {
    file {
        path => ["/usr/local/nginx/logs/access.log"]
        start_position => beginning
    }
}
filter {
    grok {
        match => {"message" => "%{COMBINEDAPACHELOG} %{QS:x_forwarded_for}"}
    }
    date {
        match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
    }
    geoip {
        source => "clientip"
    }
}
output {
    elasticsearch {
    hosts => "xxx.xxx.xxx.xxx:9200"
    }
    stdout { codec => rubydebug }
}

最后执行./logstash -f ../config/logstash.conf

查看结果