Elasticsearch、Logstash、Kibana、Filebeat的使用总结
ELK是什么？
ELK Stack是软件集合Elasticsearch、Logstash、Kibana的简称，由这三个软件及其相关的组件可以打造大规模日志实时处理系统。
ElasticSearch：是一个基于Lucene的搜索服务器。它提供了一个分布式多用户能力的全文搜索引擎，基于RESTful web接口。Elasticsearch是用Java开发的，并作为Apache许可条款下的开放源码发布，是当前流行的企业级搜索引擎。设计用于云计算中，能够达到实时搜索，稳定，可靠，快速，安装使用方便。
Logstash：是一款强大的数据处理工具，它可以实现数据传输，格式处理，格式化输出，还有强大的插件功能，常用于日志处理。
Kibana：是一个可视化工具，主要负责查询 Elasticsearch 的数据并以可视化的方式展现给业务方，比如各类饼图、直方图、区域图等。
关于Elasticsearch的版本，我是从5.x版本开始学起的，听说之前的版本跳跃2.x--->5.x,貌似变化还挺大的。关于head插件在5.x之后变成独立服务了。
关于ES集群的安装配置以及head插件的安装参考：http://blog.csdn.net/chenxun_2010/article/details/78437852
logstash最佳实践：https://doc.yonyoucloud.com/doc/logstash-best-practice-cn/get_start/hello_world.html
Logstash使用：
https://www.elastic.co/guide/en/logstash/current/pipeline.html
Logstash 工作的三个阶段： input、filter、output
安装logstash： 其安装非常简单，只需要下载安装包解压开箱即用。 只要写配置文件即可。
启动方式：
./bin/logstash -f your_config.file配置文件的写法格式：参考http://blog.csdn.net/chenxun_2010/article/details/78605934
input {
file {
path => ["/home/elk_test/logstash-5.6.3/logfile"]
codec => json {
charset => "UTF-8"
}
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => "192.168.0.153:9200"
}
}Kibana的安装也非常简单，下载安装包解压修改一下配置文件就可以：
server.host: "192.168.2.181"
#elasticsearch.username: "elastic"
#elasticsearch.password: "changeme"
elasticsearch.url: "http://192.168.2.181:9200"
kibana.index: ".kibana"Filebeat：的使用
filebeat：部署在具体的业务机器上，通过定时监控的方式获取增量的日志，并转发到logstash、elasticsearch、kafka等等。
配置vim filebeat.yml
#=========================== Filebeat prospectors =============================
filebeat.prospectors:
# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.
- input_type: log
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /home/elk/logfile
#- c:\programdata\elasticsearch\logs\*输出到logstash：
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["192.168.2.181:5044"]logstash的配置
input {
beats {
host => "192.168.2.181"
port => 5044
codec => json
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => "192.168.2.181:9200"
}
}启动filebeat：
./filebeat启动logstash：
./bin/logstash -f beat.config把准备好的json数据输入到filebeat监控的文件中去：
{ "firstName": "1", "lastName":"McLaughlin", "email": "aaaa" }
{ "firstName": "2", "lastName":"Hunter", "email": "bbbb"}
{ "firstName": "3", "lastName":"Harold", "email": "cccc" }在logstash端我们看到输出：成功解析json字段
{
"firstName" => "1",
"lastName" => "McLaughlin",
"@timestamp" => 2017-12-01T08:38:37.480Z,
"offset" => 63,
"@version" => "1",
"beat" => {
"name" => "Ubuntu-20170424",
"hostname" => "Ubuntu-20170424",
"version" => "5.6.3"
},
"input_type" => "log",
"host" => "Ubuntu-20170424",
"source" => "/home/elk/filebeat-5.6.3-linux-x86_64/request",
"type" => "log",
"email" => "aaaa",
"tags" => [
[0] "beats_input_codec_json_applied"
]
}
{
"firstName" => "2",
"lastName" => "Hunter",
"@timestamp" => 2017-12-01T08:39:02.482Z,
"offset" => 121,
"@version" => "1",
"beat" => {
"name" => "Ubuntu-20170424",
"hostname" => "Ubuntu-20170424",
"version" => "5.6.3"
},
"input_type" => "log",
"host" => "Ubuntu-20170424",
"source" => "/home/elk/filebeat-5.6.3-linux-x86_64/request",
"type" => "log",
"email" => "bbbb",
"tags" => [
[0] "beats_input_codec_json_applied"
]
}
{
"firstName" => "3",
"lastName" => "Harold",
"@timestamp" => 2017-12-01T08:39:02.482Z,
"offset" => 180,
"@version" => "1",
"beat" => {
"name" => "Ubuntu-20170424",
"hostname" => "Ubuntu-20170424",
"version" => "5.6.3"
},
"input_type" => "log",
"host" => "Ubuntu-20170424",
"source" => "/home/elk/filebeat-5.6.3-linux-x86_64/request",
"type" => "log",
"email" => "cccc",
"tags" => [
[0] "beats_input_codec_json_applied"
]
}在head插件管理页面看到数据：
