Logstash -- Nginx 日志收集处理!

Nginx 日志收集示例一：

Logstash 默认自带了 apache 标准日志的 grok 正则:

COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{NOTSPACE:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

对于 nginx 标准日志格式，可以发现只是最后多了一个 $http_x_forwarded_for 变量。所以 nginx 标准日志的 grok 正则定义是：

MAINNGINXLOG %{COMBINEDAPACHELOG} %{QS:x_forwarded_for}

• logstash 配置文件01[直接使用nginx默认的logformat]：

[root@ ~]# cat /logstash/config/nginx-test01.conf
input {
    file {
        path => "/tmp/nginx/*access*.log"
        start_position => beginning
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG} %{QS:x_forwarded_for}"}
    }
    date {
        match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
    }
    geoip {
        source => "clientip"
    }
}
output {
    elasticsearch { 
    hosts => "192.168.11.10:9200" 
    }
    stdout { codec => rubydebug }
}
# ./bin/logstash -f config/nginx-test01.conf

Nginx 日志收集示例二：

• 需要对Nginx的配置文件中的日志格式进行定制

# cat /etc/nginx/nginx.conf
............
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    upstream elasticsearch {
           server 192.168.11.10:9200;
           keepalive 15;
       }

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    log_format    json '{"@timestamp":"$time_iso8601",
                      "@version":"1","host":"$server_addr",
                      "client":"$remote_addr", "size":"$body_bytes_sent,
                      "responsetime":"$request_time",
                      "domain":"$host","url":"$uri","status":"$status"}'

    access_log  /var/log/access_json.log  json;

    sendfile        on;
    #tcp_nopush     on;
............ 省略
# cat /opt/logstash/config/nginx-test02.conf 
input {
    file {
        path => "/var/log/nginx/*access*"
        codec => "json"
    }
}
output {
  elasticsearch { hosts => ["192.168.11.10:9200"] }
  stdout { codec => rubydebug }
}
# ./bin/logstash -f config/nginx-test02.conf

在实际应用中 选取其中一种方式收集即可！

最后查看kibana展示出来的信息即可！

---

参考链接

elastic 权威指南
logstash 官方示例