Logstash6 and grok 轻巧强大

已经验证的几个方法：

https://www.elastic.co/guide/en/logstash/current/first-event.html

1. filebeat

client:

[elastic@cms104 filebeat-6.0.0-linux-x86_64]$ vi filebeat.yml filebeat.prospectors: - type: log paths: - /opt/glassfish/domains/domain1/logs/server.log output.logstash: hosts: ["10.2.28.8:5044"]

server:

[elastic@localhost bin]$ vi first-pipeline.conf

 input {

beats { port => "5044" } } # The filter part of this file is commented out to indicate that it is # optional.

# filter { # # }

output { stdout { codec => rubydebug } }

可以成功收到log:

2. elasticsearch config

server

input { file { id => "my_file_plugin" #Default value is "plain" codec => json #type => "log" path => "/home/elastic/kibana-6.0.0-linux-x86_64/nohu*.out" #/var/log/**/*.log, a recursive search of /var/log will be done for all *.log files discover_interval => 10 #start_position => "beginning" #defalut end } }

filter { if ([type] != "response") { drop {} } if ([req][remoteAddress] == "10.2.1.66") { drop {} } mutate { add_field => { "real_ip" => "79.1.14.87" } add_field => { "new_ip" => "%{[req][remoteAddress]}" } remove_field => ["[req][referer]"] } geoip { source => "real_ip" } # grok { match => ["message", "%{USERNAME:username}"]} #date { # match => [ "logdate", "MMM dd yyyy HH:mm:ss" ] #} }

output { stdout { codec => rubydebug } elasticsearch { # Indexes may not contain uppercase characters index => "log-kibana-%{+YYYY.MM.dd}" hosts => ["10.2.28.8:9200"] #, "10.2.135.104:9200"] user => "elastic" password => "password" } }

Grok是个比较强大的小工具

https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns

字符必须一一对应

------------------------------------------------------

123,456 firstStr secStr thridStr fourthStr

%{NUMBER:duration}\,%{NUMBER:speed} %{DATA:method} %{GREEDYDATA:logContent} %{WORD:lastFld}

{

"duration": "123",

"method": "firstStr",

"logContent": "secStr thridStr",

"lastFld": "fourthStr",

"speed": "456"

}

官方sample

log:

192.168.10.97 - - [19/Jul/2016:16:28:52 +0800] "GET / HTTP/1.1" 200 23 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36"

Grok:

%{IPORHOST:addre} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:http_method} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:status} (?:%{NUMBER:bytes}|-) \"(?:%{URI:http_referer}|-)\" \"%{GREEDYDATA:User_Agent}\"

result:

{ "request": "/", "auth": "-", "ident": "-", "User_Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36", "addre": "192.168.10.97", "http_method": "GET", "bytes": "23", "httpversion": "1.1", "timestamp": "19/Jul/2016:16:28:52 +0800", "status": "200" }