elk分析nginx日志

日志格式示例

1. nginx日志示例

18.18.18.18 - - [04/May/2017:18:46:45 +0800] POST /weixin/api/order/check HTTP/1.1 200 836 "https://www.baidu.com/weixin/pay/order?showwxpaytitle=1&merchant_id=96&gzid=10013&nofetch=1&sitno=31&take_mode=1" "Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Mobile/14E304 MicroMessenger/6.5.7 NetType/WIFI Language/zh_CN" 0.035 0.037

配置logstash

logstash 配置文件

input {
  beats {
    port => 5044
  }
}
filter {
    grok {
      patterns_dir => ["/etc/logstash/patterns"]
      match => {
          "message" => "%{NGINX_COMMONLOG}"
      }
    }
    #将日志中的时间替换成@timestamp
    date {
      match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
    }
}
output {
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
      }
}

添加pattern_dir /etc/logstash/patterns/httpd 文件

HTTPDUSER %{EMAILADDRESS}|%{USER}
HTTPDERROR_DATE %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}
HTTPD_COMMONLOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:%{NUMBER:res_time}|-)
HTTPD_COMBINEDLOG %{HTTPD_COMMONLOG} %{QS:referrer} %{QS:agent}
HTTPD20_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:message}
HTTPD24_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}:tid %{NUMBER:tid}\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_message}:)?( \[client %{IPORHOST:clientip}:%{POSINT:clientport}\])? %{DATA:errorcode}: %{GREEDYDATA:message}
HTTPD_ERRORLOG %{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}
COMMONAPACHELOG %{HTTPD_COMMONLOG}
COMBINEDAPACHELOG %{HTTPD_COMBINEDLOG}
NGINX_COMMONLOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} \[%{HTTPDATE:timestamp}\] (?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest}) %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:upstream_response_time:float} %{NUMBER:request_time:float}

配置filebeat

filebeat.prospectors:
      - input_type: log
      #修改为tomcat配置的日志路径
      paths:
      - /var/logs/nginx/access.log
output.logstash:
  hosts: ["18.18.18.18:5046"]

配置完成后可以分析类似格式的nginx日志。

使用docker快速搭建elk
elk分析tomcat 业务日志
elk分析tomcat access日志