ELK分析tomcat access日志

日志格式示例

1. tomcat access log示例

18.18.18.18 - - [04/May/2017:15:37:48 +0800] "POST /api/cashie/module/delivery HTTP/1.1" 200 244 75

配置logstash

logstash 配置文件

input {
  beats {
    port => 5044
  }
}
filter {
    grok {
      patterns_dir => ["/etc/logstash/patterns"]
      match => {
          "message" => "%{COMMONAPACHELOG}"
      }
    }
    #将日志中的时间替换成@timestamp
    date {
      match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
    }
}
output {
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
      }
}

添加pattern_dir /etc/logstash/patterns/httpd 文件

HTTPDUSER %{EMAILADDRESS}|%{USER}
HTTPDERROR_DATE %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}
HTTPD_COMMONLOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:%{NUMBER:res_time}|-)
HTTPD_COMBINEDLOG %{HTTPD_COMMONLOG} %{QS:referrer} %{QS:agent}
HTTPD20_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:message}
HTTPD24_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}:tid %{NUMBER:tid}\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_message}:)?( \[client %{IPORHOST:clientip}:%{POSINT:clientport}\])? %{DATA:errorcode}: %{GREEDYDATA:message}
HTTPD_ERRORLOG %{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}
COMMONAPACHELOG %{HTTPD_COMMONLOG}
COMBINEDAPACHELOG %{HTTPD_COMBINEDLOG}

配置filebeat

tomcat日志多行日志合并

filebeat.prospectors:
      - input_type: log
      #修改为tomcat配置的日志路径
      paths:
      - /opt/tomcat7/logs/access.log
output.logstash:
  hosts: ["18.18.18.18:5046"]

配置完成后可以分析类似格式的tomcat日志。

使用docker快速搭建elk
elk分析tomcat 业务日志