Logstash入门教程

前言

在线调试工具：http://grokdebug.herokuapp.com/
一个例子

input {
	file {
		path => "/data/mosh/logstash-6.2.4/logs/test.log"
		# 下面2个配置是为了从头开始读
		start_position => "beginning"
		sincedb_path => "/dev/null"
	}
}
filter {
	grok {
		match => ["message", "%{TIMESTAMP_ISO8601:tmp_log_date} \|- %{WORD} %{JAVACLASS} \[%{JAVAFILE}\]  - taskId %{BASE10NUM:tmp_task_id} step %{INT:tmp_task_step}"]
		add_field => {
			"log_date" => "%{tmp_log_date}"
			"task_id" => "%{tmp_task_id}"
			"task_step" => "%{tmp_task_step}"
		}
	}
}
output {
    # 不满足筛选条件的就不写入数据库了
	if "_grokparsefailure" not in [tags] {
		mongodb {
			uri => "mongodb://username:userpassword@mad134:27019"
			database => "cis-ws-monitor"
			collection => "task"
		}
	}
}

输出ERROR级别的日志

input {
	file {
		path => "/data/mosh/logstash-6.2.4/logs/server.log"
		start_position => "beginning"
		sincedb_path => "/dev/null"
		codec => multiline {
			pattern => "%{TIMESTAMP_ISO8601:logdate} \|-\s*%{LOGLEVEL}"
			negate => true
			what => "previous"
			auto_flush_interval => 30  # 如果在规定时候内没有新的日志事件就不等待后面的日志事件
		}
	}
}
filter {
    grok {
        match => ["message", "%{TIMESTAMP_ISO8601:logdate} \|-\s*ERROR"]
    }
}
output { 
    if "_grokparsefailure" not in [tags] {
		stdout {
			codec => rubydebug
		}
    }
}

读取多个路径下的日志

input {
	file {
		path => ["/data/server.log","/data/server2.log"]
	}
}

参考博客

每次从头读日志
[1]https://blog.csdn.net/jiao_fuyou/article/details/50777816
默认类型
[2]https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns
查看插件和安装插件
[3]https://blog.csdn.net/laoyang360/article/details/65448962
清除解析失败的数据
[4]https://blog.csdn.net/qq1032355091/article/details/52953837?locationNum=3&fps=1
[5]http://www.mamicode.com/info-detail-1693015.html
自定义输出的json格式
[6]https://www.cnblogs.com/qq27271609/p/4762562.html
Logstash的简单使用
[7]https://blog.csdn.net/qq_33689414/article/details/80365029