一、概述
Elasticsearch是个开源分布式搜索引擎,它的特点有:分布式,零配置,自动发现,索引自动分片,索引副本机制,restful风格接口,多数据源,自动搜索负载等。
Logstash是一个开源的用于收集,分析和存储日志的工具。
Kibana 也是一个开源和免费的工具,Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面,可以汇总、分析和搜索重要数据日志。
Beats是elasticsearch公司开源的一款采集系统监控数据的代理agent,是在被监控服务器上以客户端形式运行的数据收集器的统称,可以直接把数据发送给Elasticsearch或者通过Logstash发送给Elasticsearch,然后进行后续的数据分析活动。Beats由如下组成:
:是一个网络数据包分析器,用于监控、收集网络流量信息,
Packetbeat嗅探服务器之间的流量,解析应用层协议,并关联到消息的处理, 其支 持ICMP (v4 and v6)、DNS、HTTP、Mysql、PostgreSQL、Redis、
MongoDB、Memcache等协议;
2. Filebeat:用于监控、收集服务器日志文件,其已取代 logstash forwarder;
3. Metricbeat:可定期获取外部系统的监控指标信息,其可以监控、收集
Apache、HAProxy、MongoDB、MySQL、Nginx、PostgreSQL、
Redis、System、Zookeeper等服务;
4. Winlogbeat:用于监控、收集Windows系统的日志信息;
5. Create your own Beat:自定义beat ,如果上面的指标不能满足需求,elasticsarch鼓励开发者 使用go语言,扩展实现自定义的beats,只需要按照模板,实现监控的输入,日志,输出等即可。
Beats 将搜集到的数据发送到 Logstash,经 Logstash 解析、过滤后,将其发送到 Elasticsearch 存储,并由 Kibana 呈现给用户。
Beats 作为日志搜集器没有Logstash 作为日志搜集器消耗资源,解决了 Logstash 在各服务器节点上占用系统资源高的问题。
Elastic Stack官方下载地址:/downloads。
二、开源实时日志分析系统Elastic Stack 部署:
A.安装依赖包JDK
①关闭防火墙
# cat /etc/selinux/config |grep -v "#" SELINUX=disabled ##关闭selinux SELINUXTYPE=targeted # systemctl stop firewalld ##关闭防火墙
②下载安装JDK
# wget /pub/epel/epel-release-latest- # yum install jre ##这里我只安装的是jre环境,一样可用 # java -version openjdk version "_111" OpenJDK Runtime Environment (build _111-b15) OpenJDK 64-Bit Server VM (build -b15, mixed mode
B.安装Elasticsearch
①下载安装elasticsearch
# rpm --import /GPG-KEY-elasticsearch # cat > /etc// <
注:安装后出现如下问题,致使elasticsearch无法启动
# systemctl status elasticsearch OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should =N OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000085330000, 2060255232, 0) failed; error='Cannot a ...'(errno=12) # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 2060255232 bytes for committing reserved memory. # An error report file with more information is saved as: # /tmp/hs_err_解决:
配置elasticsearch下的:
# vi /etc/elasticsearch/ -Xms4g ##启用如下两项 -Xmx4g ##-Xms2g ##关闭如下两项 ##-Xmx2g
②ElasticSearch默认的对外服务的HTTP端口是9200,节点间交互的TCP端口是9300。
# ss -tlnp |grep -E '9200|9300'
③测试服务
# curl -X GET localhost:9200 { "name" : "XVY0Ovb", "cluster_name" : "elasticsearch", "cluster_uuid" : "tR_H9avzT6Kf4hXWTIfWyA", "version" : { "number" : "", "build_hash" : "080bb47", "build_date" : "2016-11-11T22:08:", "build_snapshot" : false, "lucene_version" : "" }, "tagline" : "You Know, for Search" }也可以使用如下命令测试
# curl -i -XGET 'localhost:9200/' HTTP/ 200 OK content-type: application/json; charset=UTF-8 content-length: 327 { "name" : "XVY0Ovb", "cluster_name" : "elasticsearch", "cluster_uuid" : "tR_H9avzT6Kf4hXWTIfWyA", "version" : { "number" : "", "build_hash" : "080bb47", "build_date" : "2016-11-11T22:08:", "build_snapshot" : false, "lucene_version" : "" }, "tagline" : "You Know, for Search" }
C.安装Logstash
# rpm --import /GPG-KEY-elasticsearch # cat > /etc// <
D.安装Kibana
①安装Kibana
# rpm --import /GPG-KEY-elasticsearch # cat > /etc// <
②只需更改如下配置
# cat /etc/kibana/ |grep -v "#" : ""
③检测服务
# ss -tlnp|grep 5601 ##Kibana默认进程名:node ,端口5601
浏览器输入localhost:5601
E、Beats安装部署
a.安装部署Filebeat
①安装Filebeat
# curl -L -O /downloads/beats/filebeat/filebeat--x86_ # rpm -ivh filebeat--x86_ # systemctl start filebeat # systemctl status filebeat # -version ##查看版本 filebeat version (amd64), libbeat
②配置Filebeat
# cp /etc/filebeat # cp # vi /etc/filebeat/ ##配置filebeat #============= Filebeat prospectors =============== : - input_type: log paths: - /var/log/*.log #==================== Outputs ===================== #------------- Elasticsearch output --------------- : # Array of hosts to connect to. hosts: ["localhost:9200"] #---------------- Logstash output ----------------- : # The Logstash hosts hosts: ["localhost:5043"] ##只需配置该处,其他默认即可 # -configtest -e ##验证配置文件
③配置Logstash
# cat /etc/logstash// input { beats { port => "5043" } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}"} } geoip { source => "clientip" } } output { elasticsearch { hosts => [ "localhost:9200" ] } } # systemctl restart logstash ##重启logstash # /usr/share/logstash/bin/logstash -f /etc/logstash// --_and_exit ##验证配置文件 Sending Logstash logs to /var/log/logstash which is now configured via Configuration OK # ss -tlnp|grep -E '5043|9600'
④配置kibana
浏览器输入localhost:5601,配置filebeat的索引(只需输入filebeat-*即可)。
在第一个框里输入filebeat-*后稍等片刻,kibana会自动识别,OK后下面的按钮会由灰色变为可操控的按钮"Create",如上图所示。点击该按钮后,最后就会呈现如下图所示:
我们再会过头新建logstash的索引,浏览器输入localhost:5601,点击左边栏的”Management”===>然后点击“index Patterns”===>
===>然后点击“Add New”===>
===>点击“Crete”按钮创建logstash索引,创建完成后即会展现如下图所示:
b.安装部署Packetbeat
①安装配置Packetbeat
# yum install libpcap # curl -L -O /downloads/beats/packetbeat/packetbeat--x86_ # rpm -ivh packetbeat--x86_ # cat /etc/packetbeat/ ==================== Network device =================== : any ##捕获所有消息发送或接收的网络接口 ======================== Flows ======================== : timeout: 30s period: 10s ================== Transaction protocols ============== ##如下是packetbeat默认支持的主要协议及端口 : enabled: true : ports: [5672] : ports: [9042] : ports: [53] include_authorities: true include_additionals: true : ports: [80, 8080, 8000, 5000, 8002] : ports: [11211] : ports: [3306] : ports: [5432] : ports: [6379] : ports: [9090] : ports: [27017] : ports: [2049] ========================= General ========================= ========================= Outputs ========================= ------------------- Elasticsearch output ------------------ : hosts: ["localhost:9200"] --------------------- Logstash output --------------------- : hosts: ["localhost:5043"] ##只需配置该处,其他默认即可 ============================= Logging =====================
②验证配置并启动packetbeat# -version ##查看packetbeat版本 packetbeat version (amd64), libbeat # -configtest -e ##测试配置文件 ...... Config OK # systemctl start packetbeat # systemctl status packetbeat
③配置Kibana(新建packetbeat索引)在localhost:5601下新建索引页面输入“packetbeat-*”,之后kibana会自动更新,在“Time-field name”下面的三个选项中选择“@timestamp”,最后点击“Create”创建即可。
创建完成后,kibana显示如下:
c.安装部署Metricbeat
①安装配置metricbeat
# curl -L -O /downloads/beats/metricbeat/metricbeat--x86_ # rpm -ivh metricbeat--x86_ # cat /etc/metricbeat/ ================= Modules configuration ================= : ---------------------- System Module --------------------- - module: system metricsets: - cpu - load - filesystem - fsstat - memory - network - process enabled: true period: 10s processes: ['.*'] ========================= General ======================= ========================= Outputs ======================= ------------------- Elasticsearch output ---------------- : hosts: ["localhost:9200"] --------------------- Logstash output ------------------- : hosts: ["localhost:5043"] ##只需配置该项,其他默认即可 ======================= Logging =========================
②验证配置并启动metricbeat# -version ##查看版本信息 metricbeat version (amd64), libbeat # -configtest -e ##验证配置文件 # systemctl start metricbeat # systemctl status metricbeat
③配置kibana(新建metricbeat索引)
在localhost:5601下新建索引页面输入“metricbeat-*”,之后kibana会自动更新,在“Time-field name”下面的选项中选择“@timestamp”,最后点击“Create”创建即可。
最后呈现如下图所示:
注1:
关于ELK Stack的一些查询语句:
①查询filebeat
# curl -XGET 'localhost:9200/filebeat-*/_search?pretty'②查询packetbeat
# curl -XGET 'localhost:9200/packetbeat-*/_search?pretty'③查询metricbeat
# curl -XGET 'localhost:9200/metricbeat-*/_search?pretty'④查询集群健康度
# curl 'localhost:9200/_cat/health?v'⑤查看节点列表
# curl 'localhost:9200/_cat/nodes?v' ip cpu load_1m load_5m load_15m master name 37 93 3 mdi * XVY0Ovb⑥列出所有索引
# curl 'localhost:9200/_cat/indices?v' health status index uuid pri rep yellow open filebeat- Mn4MzxdTRaCj9iseutcmqA 5 1 2 0 12kb 12kb yellow open filebeat- iMrr710mT42mApxdV62k-A 5 1 159 0 yellow open packetbeat- wkTcIwD6RgiiCFwlWBIILA 5 1 5652 0 yellow open customer NvxXLgHoREefJLRhot13Ug 5 1 0 0 800b 800b yellow open packetbeat- Beoe07S7QB-dntNV4nxJNQ 5 1 2446 0 yellow open test M7WbkYq2QNmeJ9NOyMfMZA 5 1 0 0 800b 800b yellow open logstash- pcb_84ChSBe9A7VRd-SQNw 5 1 161 0 yellow open metricbeat- AmVeT1xCQGCnxlAFXUxhYw 5 1 94459 0 yellow open logstash- 6PCKMYKCSVmPfdg-Sx2ARA 5 1 85772 0 yellow open .kibana QYTg0I5KS-yc3d7GSey3Zw 1 1 5 0 102kb 102kb
注2:
如果搭建期间有什么不清楚或不了解的,建议去看官方文档,文档地址如下:
/guide/