需求背景：
业务发展越来越庞大，服务器越来越多 各种访问日志、应用日志、错误日志量越来越多，导致运维人员无法很好的去管理日志 开发人员排查问题，需要到服务器上查日志，不方便 运营人员需要一些数据，需要我们运维到服务器上分析日志
为什么要用到ELK：
一般我们需要进行日志分析场景：直接在日志文件中 grep、awk 就可以获得自己想要的信息。但在规模较大也就是日志量多而复杂的场景中， 此方法效率低下，面临问题包括日志量太大如何归档、文本搜索太慢怎么办、如何多维度查询。需要集中化的日志管理，所有服务器上的日志收集汇总。 常见解决思路是建立集中式日志收集系统，将所有节点上的日志统一收集，管理，访问。 大型系统通常都是一个分布式部署的架构，不同的服务模块部署在不同的服务器上，问题出现时，大部分情况需要根据问题暴露的关键信息， 定位到具体的服务器和服务模块，构建一套集中式日志系统，可以提高定位问题的效率。 一个完整的集中式日志系统，需要包含以下几个主要特点： 收集－能够采集多种来源的日志数据 传输－能够稳定的把日志数据传输到中央系统 存储－如何存储日志数据 分析－可以支持 UI 分析 警告－能够提供错误报告，监控机制 而ELK则提供了一整套解决方案，并且都是开源软件，之间互相配合使用，完美衔接，高效的满足了很多场合的应用。是目前主流的一种日志系统。
ELK简介：
ELK是三个开源软件的缩写，分别为：Elasticsearch 、 Logstash以及Kibana , 它们都是开源软件。不过现在还新增了一个Beats， 它是一个轻量级的日志收集处理工具(Agent)，Beats占用资源少，适合于在各个服务器上搜集日志后传输给Logstash，官方也推荐此工具， 目前由于原本的ELK Stack成员中加入了 Beats 工具所以已改名为Elastic Stack。
Elastic Stack包含：
Elasticsearch是个开源分布式搜索引擎，提供搜集、分析、存储数据三大功能。它的特点有：分布式，零配置，自动发现，索引自动分片， 索引副本机制，restful风格接口，多数据源，自动搜索负载等。详细可参考Elasticsearch权威指南。 Logstash 主要是用来日志的搜集、分析、过滤日志的工具，支持大量的数据获取方式。一般工作方式为c/s架构，client端安装在需要收集日志的主机上， server端负责将收到的各节点日志进行过滤、修改等操作在一并发往elasticsearch上去。 Kibana 也是一个开源和免费的工具，Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助汇总、分析和搜索重要数据日志。 Beats在这里是一个轻量级日志采集器，其实Beats家族有6个成员，早期的ELK架构中使用Logstash收集、解析日志，但是Logstash对内存、cpu、io等资源消耗比较高。 相比 Logstash，Beats所占系统的CPU和内存几乎可以忽略不计
ELK一些组件：
openjdk 1.8.0_191 ElasticSearch Logstash Kibana Filebeat
ELK Stack （5.0版本之后）--> Elastic Stack == （ELK Stack + Beats）。目前Beats包含六种工具：
Packetbeat： 网络数据（收集网络流量数据） Metricbeat： 指标 （收集系统、进程和文件系统级别的 CPU 和内存使用情况等数据） Filebeat： 日志文件（收集文件数据） Winlogbeat： windows事件日志（收集 Windows 事件日志数据） Auditbeat：审计数据 （收集审计日志） Heartbeat：运行时间监控 （收集系统运行时的数据）
关于x-pack工具：
x-pack对Elastic Stack提供了安全、警报、监控、报表、图表于一身的扩展包，是收费的，所以本文不涉及x-pack的安装。
ELK相关链接：
ELK官网：https://www.elastic.co/cn/ 中文指南：https://www.gitbook.com/book/chenryn/elk-stack-guide-cn/details
备注：以上内容来自网络。
一、准备
ELK注意点：
1、主要掌握logstash的原理、ElasticSearch原理、Kibana可视化技术（这里我采用Grafana可视化） 2、logstash的时候output先输出到命令控制台，如果没问题再写入配置文件。 3、要引入redis或者kanfa，有很多优点。一般情况下redis都能胜任，kanfa太重型。
1、关闭防火墙和selinux
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl stop firewalld [root@iZj6c7bp1a289r04izstw4Z ~]# systemctl disable firewalld [root@iZj6c7bp1a289r04izstw4Z ~]# vi /etc/sysconfig/selinux SELINUX=disabled
2、安装常用工具、同步时间
[root@iZj6c7bp1a289r04izstw4Z ~]# yum -y install lrzsz rdate wget vim net-tools ntpdate [root@iZj6c7bp1a289r04izstw4Z ~]# rdate -s time-b.nist.gov
3、更改最大文件描述符、打开数、进程数
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/security/limits.conf * hard nofile 65536 * soft nofile 65536 * soft nproc 65536 * hard nproc 65536
4、修改最大打开进程数，最低2048
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/security/limits.d/20-nproc.conf * soft nproc 4096
5、其他设置
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/sysctl.conf vm.max_map_count=655360 net.core.somaxconn=65535 net.ipv4.ip_forward = 1 [root@iZj6c7bp1a289r04izstw4Z ~]# sysctl -p
二、安装JDK环境
可以yum、rpm、tar.gz等方式安装。为了简单和省事，我直接yum方式安装（推荐yum方式安装，因为它会将java程序放在/usr/bin等目录下。 其它两种方式会造成服务启动不了，需要设置环境变量、elk配置文件或做软链接）。 [root@iZj6c7bp1a289r04izstw4Z ~]# yum install java-1.8.0-openjdk -y [root@localhost ~]# java -version openjdk version "1.8.0_191" OpenJDK Runtime Environment (build 1.8.0_191-b12) OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
三、安装ELK
1、配置yum源
[root@iZj6c7bp1a289r04izstw4Z ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch [root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/yum.repos.d/elk.repo [elk-6] name=elk-6 baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 [root@iZj6c7bp1a289r04izstw4Z ~]# yum clean all [root@iZj6c7bp1a289r04izstw4Z ~]# yum makecache
2、安装ELK
[root@iZj6c7bp1a289r04izstw4Z ~]# yum install elasticsearch logstash kibana filebeat -y 备注：当然elk除了采用yum方式安装，还可以rpm、tar.gz等方式安装。由于默认yum方式安装和其它方式安装的版本是一样，所以我选择采用yum方式。 rpm包链接如下： https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.1.rpm https://artifacts.elastic.co/downloads/logstash/logstash-6.6.1.rpm https://artifacts.elastic.co/downloads/kibana/kibana-6.6.1-x86_64.rpm https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.6.1-x86_64.rpm
四、配置ELK
1、配置elasticsearch
[root@iZj6c7bp1a289r04izstw4Z ~]# rpm -qc elasticsearch
/etc/elasticsearch/elasticsearch.yml
/etc/elasticsearch/jvm.options
/etc/elasticsearch/log4j2.properties
/etc/elasticsearch/role_mapping.yml
/etc/elasticsearch/roles.yml
/etc/elasticsearch/users
/etc/elasticsearch/users_roles
/etc/init.d/elasticsearch
/etc/sysconfig/elasticsearch
/usr/lib/sysctl.d/elasticsearch.conf
/usr/lib/systemd/system/elasticsearch.service
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/elasticsearch/elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
cluster.name: ELK
node.name: node-1
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.1.104:9300"]
discovery.zen.minimum_master_nodes: 1
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl daemon-reload
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl enable elasticsearch
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl restart elasticsearch
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl status elasticsearch
[root@iZj6c7bp1a289r04izstw4Z ~]# ps aux |grep elasticsearch
[root@iZj6c7bp1a289r04izstw4Z ~]# netstat -nltp | grep java
tcp 0 0 0.0.0.0:9200 0.0.0.0:* LISTEN 1776/java
tcp 0 0 0.0.0.0:9300 0.0.0.0:* LISTEN 1776/java
[root@iZj6c7bp1a289r04izstw4Z ~]# curl -X GET http://localhost:9200
{
"name" : "node-1",
"cluster_name" : "ELK",
"cluster_uuid" : "egGlhWnESzeeUalJ2FXUuA",
"version" : {
"number" : "6.6.1",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "1fd8f69",
"build_date" : "2019-02-13T17:10:04.160291Z",
"build_snapshot" : false,
"lucene_version" : "7.6.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
2、配置kibana
[root@iZj6c7bp1a289r04izstw4Z ~]# rpm -qc kibana /etc/kibana/kibana.yml [root@iZj6c7bp1a289r04izstw4Z ~]# vi /etc/kibana/kibana.yml server.port: 5601 server.host: "0.0.0.0" elasticsearch.url: "http://localhost:9200" logging.dest: /var/log/kibana.log #创建日志文件： [root@master-node ~]# touch /var/log/kibana.log; chmod 777 /var/log/kibana.log [root@iZj6c7bp1a289r04izstw4Z ~]# systemctl start kibana [root@iZj6c7bp1a289r04izstw4Z ~]# systemctl status kibana [root@iZj6c7bp1a289r04izstw4Z ~]# ps aux |grep kibana [root@iZj6c7bp1a289r04izstw4Z ~]# netstat -lntp |grep 5601 tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 6030/node 注：由于kibana是使用node.js开发的，所以进程名称为node 然后在浏览器里进行访问，如：http://172.31.116.244:5601/ ，由于我们并没有安装x-pack，所以此时是没有用户名和密码的，可以直接访问的：
如果需要账号密码认证的，需要安装插件，但是插件好像是收费的。
不过可以安装nginx和认证模块，然后代理kibana就可以，具体步骤如下：
[root@iZj6c7bp1a289r04izstw4Z ~]# yum -y install epel-release #安装epel源
[root@iZj6c7bp1a289r04izstw4Z ~]# yum -y install nginx httpd-tools #安装Nginx、httpd用户认证工具
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/nginx/nginx.conf #注释如下图
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/nginx/conf.d/kibana.conf
server {
listen 5666;
server_name kibana;
auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/kibana-user;
location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
[root@iZj6c7bp1a289r04izstw4Z ~]# htpasswd -cm /etc/nginx/kibana-user gxm
New password:
Re-type new password:
Adding password for user gxm
[root@iZj6c7bp1a289r04izstw4Z ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl start nginx
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl status nginx
3、配置logstash
[root@iZj6c7bp1a289r04izstw4Z ~]# rpm -qc logstash /etc/logstash/jvm.options /etc/logstash/log4j2.properties /etc/logstash/logstash-sample.conf /etc/logstash/logstash.yml /etc/logstash/pipelines.yml /etc/logstash/startup.options [root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/logstash/logstash.yml path.config: /etc/logstash/conf.d
4、测试logstash场景1（系统日志作为输入，直接输出到终端）。
#设置syslog日志配置文件 [root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/rsyslog.conf #### RULES #### *.* @172.31.116.244:10514 [root@data-node1 ~]# systemctl restart rsyslog
#设置logstash配置文件
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/logstash/conf.d/syslog-01.conf
input { # 定义日志源
syslog {
type => "system-syslog" # 定义类型
port => 10514 # 定义监听端口
}
}
output { # 定义日志输出
stdout {
codec => rubydebug # 将日志输出到当前的终端上显示
}
}
备注：安装完之后，先不要启动服务，先配置logstash收集syslog日志（只做测试，后期这个文件会改输出）。
# 检测配置文件是否有错，为ok则代表配置文件没有问题。
# 命令说明：
--path.settings 用于指定logstash的配置文件所在的目录
-f 指定需要被检测的配置文件的路径
--config.test_and_exit 指定检测完之后就退出，不然就会直接启动了
[root@iZj6c7bp1a289r04izstw4Z ~]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog-01.conf --config.test_and_exit
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-03-02T23:54:57,560][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2019-03-02T23:55:06,025][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
#指定刚才的配置文件，启动logstash：
[root@iZj6c7bp1a289r04izstw4Z ~]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog-01.conf
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
#这时终端会停留在这里，因为我们在配置文件中定义的是将信息输出到当前终端
#打开新终端检查一下10514端口是否已被监听（logstash配置文件定义的监听端口）：
[root@iZj6c7bp1a289r04izstw4Z ~]# netstat -lntp |grep 10514
tcp6 0 0 :::10514 :::* LISTEN 4312/java
#远程登录到服务器，让系统产生日志，观察上述命令输出：
可以看到终端中以JSON的格式打印了收集到的日志，测试成功。
[root@iZj6c7bp1a289r04izstw4Z ~]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog-01.conf
{
"@timestamp" => 2019-03-02T16:14:30.000Z,
"@version" => "1",
"host" => "172.31.116.243",
"severity" => 6,
"facility" => 3,
"facility_label" => "system",
"severity_label" => "Informational",
"message" => "Starting logstash...",
"timestamp" => "Mar 3 00:14:30",
"type" => "system-syslog",
"priority" => 30,
"logsource" => "iZj6c7bp1a289r04izstw4Z",
"program" => "systemd"
}
5、测试logstash场景2（系统日志作为输入，输出到es服务器）
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/logstash/conf.d/syslog-02.conf
input {
syslog {
type => "system-syslog-to-es"
port => 10514
}
}
output {
elasticsearch {
hosts => ["localhost:9200"] # 定义es服务器的ip
index => "system-syslog-toes-%{+YYYY.MM}" # 定义索引，传递给es，在kibana展示
}
}
# 检测配置文件是否有错，为ok则代表配置文件没有问题。 # 命令说明： --path.settings 用于指定logstash的配置文件所在的目录 -f 指定需要被检测的配置文件的路径 --config.test_and_exit 指定检测完之后就退出，不然就会直接启动了 [root@iZj6c7bp1a289r04izstw4Z ~]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog-02.conf --config.test_and_exit Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties [2019-03-02T23:54:57,560][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [2019-03-02T23:55:06,025][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [root@iZj6c7bp1a289r04izstw4Z ~]# systemctl start logstash [root@iZj6c7bp1a289r04izstw4Z ~]# systemctl enable logstash [root@iZj6c7bp1a289r04izstw4Z ~]# ps aux |grep logstash [root@iZj6c7bp1a289r04izstw4Z ~]# netstat -lntp |grep 10514 #我这里启动logstash后，进程是正常存在的，但是9600以及10514端口却没有被监听。于是查看logstash的日志看看有没有错误信息的输出，但是发现没有记录日志信息，那就只能转而去查看messages的日志，发现错误信息如下
报错1：
[root@iZj6c7bp1a289r04izstw4Z ~]# cat /var/log/messages | grep "ERROR"
Mar 3 00:28:32 iZj6c7bp1a289r04izstw4Z logstash: 2019-03-03 00:28:32,731 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
Mar 3 00:28:32 iZj6c7bp1a289r04izstw4Z logstash: 2019-03-03 00:28:32,755 main ERROR RollingFileManager (/var/log/logstash/logstash-slowlog-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-slowlog-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-slowlog-plain.log (Permission denied)
Mar 3 00:28:32 iZj6c7bp1a289r04izstw4Z logstash: 2019-03-03 00:28:32,778 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@43fa1646] unable to create manager for [/var/log/logstash/logstash-slowlog-plain.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@53460794[pattern=/var/log/logstash/logstash-slowlog-plain-%d{yyyy-MM-dd}-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)]), strategy=DefaultRolloverStrategy(min=1, max=7, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c] %.10000m%n, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@43fa1646] unable to create manager for [/var/log/logstash/logstash-slowlog-plain.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@53460794[pattern=/var/log/logstash/logstash-slowlog-plain-%d{yyyy-MM-dd}-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=104857600)]), strategy=DefaultRolloverStrategy(min=1, max=7, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c] %.10000m%n, filePermissions=null, fileOwner=null]]
Mar 3 00:28:32 iZj6c7bp1a289r04izstw4Z logstash: 2019-03-03 00:28:32,801 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
Mar 3 00:28:32 iZj6c7bp1a289r04izstw4Z logstash: 2019-03-03 00:28:32,825 main ERROR RollingFileManager (/var/log/logstash/logstash-slowlog-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-slowlog-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-slowlog-plain.log (Permission denied)
报错2：
Mar 3 00:34:36 iZj6c7bp1a289r04izstw4Z logstash: [2019-03-03T00:34:36,440][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:447:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:229:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:140:in `block in validate_all'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:139:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:278:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:237:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `'"]}
Mar 3 00:34:36 iZj6c7bp1a289r04izstw4Z logstash: [2019-03-03T00:34:36,462][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
Mar 3 00:34:36 iZj6c7bp1a289r04izstw4Z systemd: logstash.service: main process exited, code=exited, status=1/FAILURE
解决办法：
[root@iZj6c7bp1a289r04izstw4Z ~]# ll /var/log/logstash/
total 8
-rw-r--r-- 1 root root 1756 Mar 3 00:19 logstash-plain-2019-03-02-1.log.gz
-rw-r--r-- 1 root root 1706 Mar 3 00:31 logstash-plain.log
-rw-r--r-- 1 root root 0 Mar 2 23:36 logstash-slowlog-plain.log
[root@iZj6c7bp1a289r04izstw4Z ~]# chown logstash /var/log/logstash/logstash-slowlog-plain.log
[root@iZj6c7bp1a289r04izstw4Z ~]# chown logstash /var/log/logstash/logstash-plain.log
[root@iZj6c7bp1a289r04izstw4Z ~]# ll /var/log/logstash/
total 8
-rw-r--r-- 1 root root 1756 Mar 3 00:19 logstash-plain-2019-03-02-1.log.gz
-rw-r--r-- 1 logstash root 1706 Mar 3 00:31 logstash-plain.log
-rw-r--r-- 1 logstash root 0 Mar 2 23:36 logstash-slowlog-plain.log
[root@iZj6c7bp1a289r04izstw4Z ~]# ll /var/lib/logstash/
total 16
drwxr-xr-x 2 root root 4096 Mar 2 23:36 dead_letter_queue
drwxr-xr-x 3 root root 4096 Mar 2 23:41 plugins
drwxr-xr-x 2 root root 4096 Mar 2 23:36 queue
-rw-r--r-- 1 root root 36 Mar 2 23:40 uuid
[root@iZj6c7bp1a289r04izstw4Z ~]# chown -R logstash /var/lib/logstash/
[root@iZj6c7bp1a289r04izstw4Z ~]# ll /var/lib/logstash/
total 16
drwxr-xr-x 2 logstash root 4096 Mar 2 23:36 dead_letter_queue
drwxr-xr-x 3 logstash root 4096 Mar 2 23:41 plugins
drwxr-xr-x 2 logstash root 4096 Mar 2 23:36 queue
-rw-r--r-- 1 logstash root 36 Mar 2 23:40 uuid
端口监听正常：
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl restart logstash
[root@iZj6c7bp1a289r04izstw4Z ~]# netstat -lntp |grep 10514
tcp 0 0 0.0.0.0:10514 0.0.0.0:* LISTEN 5945/java
[root@iZj6c7bp1a289r04izstw4Z ~]# netstat -lntp |grep 9600
tcp 0 0 127.0.0.1:9600 0.0.0.0:* LISTEN 5945/java
[root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/logstash/logstash.yml
http.host: "172.31.116.244"
[root@iZj6c7bp1a289r04izstw4Z ~]# systemctl restart logstash
[root@iZj6c7bp1a289r04izstw4Z ~]# netstat -lntp |grep 9600
tcp 0 0 172.31.116.244:9600 0.0.0.0:* LISTEN 6132/java
#yellow状态解决（不建议按照如下方法解决，建议esasticsearch做集群）
[root@iZj6c7bp1a289r04izstw4Z ~]# curl 'localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .kibana_1 iWQ-ElLnRXaq0mJtsCXe-Q 1 0 2 0 8.6kb 8.6kb
yellow open system-syslog-toes-2019.03 ZBfgIsFQRnuXIDU1PKYkkg 5 1 451 0 202.5kb 202.5kb
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# curl 'http://localhost:9200/_cat/health?v'
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1551578321 01:58:41 ELK yellow 1 1 6 6 0 0 5 0 - 54.5%
#正常情况下，集群（（单机版称为空集群））得健康状态分为三种：
green：最健康得状态，说明所有的分片包括备份都可用
yellow：基本的分片可用，但是备份不可用（或者是没有备份）
red：部分的分片可用，表明分片有一部分损坏。此时执行查询部分数据仍然可以查到，遇到这种情况，还是赶快解决比较好
从上边查看的集群状态中还包括一些其他信息：集群名字时ELK，集群的节点个数为1，分片数量为5，活动分片百分比为 54.5%
#索引的健康状态也为yellow，索引的健康状态也有三种，yellow、green、red与集群的健康状态解释是一样的。
#单节点部署elasticsearch，而默认的分片副本数目配置为1，而相同的分片不能在一个节点上，所以就存在副本分片指定不明确的问题，
所以显示为yellow，我们可以通过在elasticsearch集群上添加一个节点来解决问题，如果你不想这么做，
你可以删除那些指定不明确的副本分片（当然这不是一个好办法）但是作为测试和解决办法还是可以尝试的，下面我们试一下删除副本分片的办法。
[root@iZj6c7bp1a289r04izstw4Z ~]# curl -XGET http://localhost:9200/_cluster/health?pretty=true
{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 6,
"active_shards" : 6,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 5,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 54.54545454545454
}
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# curl -H "Content-Type: application/json" -XPUT 'http://localhost:9200/_settings' -d '
> {
> "index" : {
> "number_of_replicas" : 0
> }
> }'
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# curl -XGET http://localhost:9200/_cluster/health?pretty=true
{
"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 6,
"active_shards" : 6,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# curl 'localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open system-syslog-toes-2019.03 NMhafFEzT9-w6RWFCP0flw 5 0 13251 0 2.4mb 2.4mb
green open .kibana_1 iWQ-ElLnRXaq0mJtsCXe-Q 1 0 2 0 8.6kb 8.6kb
如上，可以看到，在logstash配置文件中定义的system-syslog索引成功获取到了，证明配置没问题，logstash与es通信正常。
获取指定索引详细信息，有数据表示正常：
[root@iZj6c7bp1a289r04izstw4Z ~]# curl -XGET 'localhost:9200/system-syslog-toes-2019.03?pretty'
如果日后需要删除索引的话，使用以下命令可以删除指定索引：
[root@iZj6c7bp1a289r04izstw4Z ~]# curl -XDELETE 'localhost:9200/.kibana_1'
6、测试logstash场景3（nginx日志文件作为输入，经过过滤后，输出到es服务器）
#安装nginx [root@iZj6c23v8zk8gxxvjdylt8Z ~]# rpm -Uvh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm [root@iZj6c23v8zk8gxxvjdylt8Z ~]# yum install -y nginx [root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl start nginx [root@iZj6c23v8zk8gxxvjdylt8Z ~]# curl http://127.0.0.1 [root@iZj6c23v8zk8gxxvjdylt8Z ~]# tail -f /var/log/nginx/access.log
#nginx日志有main和log_json两种，默认为main普通文本格式 ELK存储为json格式，文本格式华，拆分出如ip地址、访问agent等，便于后续使用 [root@iZj6c23v8zk8gxxvjdylt8Z ~]# vim /etc/nginx/nginx.conf log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; [root@iZj6c23v8zk8gxxvjdylt8Z ~]# tail -f /var/log/nginx/access.log 127.0.0.1 - - [03/Mar/2019:10:22:10 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
#logstash配置
#创建nginx正则表达式，引用grok正则
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# vim /etc/logstash/conf.d/nginx-access
#nginx-access
WZ ([^ ]*)
NGINXACCESS %{IP:remote_ip} \- \- \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{WZ:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:status} %{NUMBER:bytes} %{QS:referer} %{QS:agent} %{QS:xforward}
备注：grok使用的正则表达式在grok-patterns文件
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# find / -name grok-patterns
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# cat /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# vim /etc/logstash/conf.d/nginx.conf
input {
file { # 指定一个文件作为输入源
path => "/var/log/nginx/access.log" # 指定文件的路径
start_position => "beginning" # 指定何时开始收集
type => "nginx" # 定义日志类型，可自定义
}
}
# #筛选
filter {
# 如果是nginx访问日志
if ( [type] == "nginx" ) {
#按字段切割
grok {
patterns_dir=>"/etc/logstash/conf.d/nginx-access"
match => { "message" => "%{NGINXACCESS}" }
}
# 时间格式转换
date {
match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
}
# 删除不需要的字段
mutate {
remove_field => [ "offset", "@version", "beat", "input_type", "tags","id"]
}
}
}
output {
#标准输出，调试使用#
stdout {
codec => rubydebug { }
}
# 输出到es#
elasticsearch {
hosts => ["localhost:9200"]
index => "%{type}-%{+YYYY.MM.dd}"
}
}
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx.conf --config.test_and_exit
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-03-03T10:27:56,895][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2019-03-03T10:28:04,969][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl stop logstash
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx.conf
然后访问nginx网站，如下表示成功。
{
"xforward" => "\"-\"",
"message" => "127.0.0.1 - - [03/Mar/2019:10:50:55 +0800] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.29.0\" \"-\"",
"host" => "iZj6c23v8zk8gxxvjdylt8Z",
"type" => "nginx",
"agent" => "\"curl/7.29.0\"",
"timestamp" => "03/Mar/2019:10:50:55 +0800",
"httpversion" => "1.1",
"remote_ip" => "127.0.0.1",
"referer" => "\"-\"",
"path" => "/var/log/nginx/access.log",
"method" => "GET",
"request" => "/",
"status" => "200",
"bytes" => "612",
"@timestamp" => 2019-03-03T02:50:55.000Z
}
{
"xforward" => "\"-\"",
"message" => "127.0.0.1 - - [03/Mar/2019:10:22:10 +0800] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.29.0\" \"-\"",
"host" => "iZj6c23v8zk8gxxvjdylt8Z",
"type" => "nginx",
"agent" => "\"curl/7.29.0\"",
"timestamp" => "03/Mar/2019:10:22:10 +0800",
"httpversion" => "1.1",
"remote_ip" => "127.0.0.1",
"referer" => "\"-\"",
"path" => "/var/log/nginx/access.log",
"method" => "GET",
"request" => "/",
"status" => "200",
"bytes" => "612",
"@timestamp" => 2019-03-03T02:22:10.000Z
}
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl start logstash
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# curl 'localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .kibana_1 iWQ-ElLnRXaq0mJtsCXe-Q 1 0 2 0 8.6kb 8.6kb
yellow open system-syslog-toes-2019.03 ZBfgIsFQRnuXIDU1PKYkkg 5 1 68178 0 5.6mb 5.6mb
yellow open nginx-2019.03.03 lQdtD7pqRFyerEWHfMapvQ 5 1 2 0 24.3kb 24.3kb
7、测试logstash场景4（filebeat收集nginx日志并作为输入，经过过滤后，输出到es服务器）
#filebeat收集nginx日志，然后传递到logstash，logstash过滤后输出到es服务器。filebeat比logstash直接收取要占用资源少。 #安装nginx [root@iZj6c23v8zk8gxxvjdylt8Z ~]# rpm -Uvh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm [root@iZj6c23v8zk8gxxvjdylt8Z ~]# yum install -y nginx [root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl start nginx [root@iZj6c23v8zk8gxxvjdylt8Z ~]# curl http://127.0.0.1 [root@iZj6c23v8zk8gxxvjdylt8Z ~]# tail -f /var/log/nginx/access.log
#nginx日志有main和log_json两种，默认为main普通文本格式 ELK存储为json格式，文本格式华，拆分出如ip地址、访问agent等，便于后续使用 [root@iZj6c23v8zk8gxxvjdylt8Z ~]# vim /etc/nginx/nginx.conf log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; [root@iZj6c23v8zk8gxxvjdylt8Z ~]# tail -f /var/log/nginx/access.log 127.0.0.1 - - [03/Mar/2019:10:22:10 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
filebeat配置： [root@iZj6c23v8zk8gxxvjdylt8Z ~]# vim /etc/filebeat/filebeat.yml - input_type: log enable: yes #tags: nginx-access paths: - /var/log/nginx/access.log exclude_lines: ["^$"] #output.elasticsearch: # Array of hosts to connect to. # hosts: ["localhost:9200"] output.logstash: hosts: ["localhost:5044"] #hosts: ["172.16.50.32:5044"] #index: filebeat [root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl enable filebeat [root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl restart filebeat [root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl status filebeat
#logstash配置
#创建nginx正则表达式，引用grok正则
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# vim /etc/logstash/conf.d/nginx-access
#nginx-access
WZ ([^ ]*)
NGINXACCESS %{IP:remote_ip} \- \- \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{WZ:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:status} %{NUMBER:bytes} %{QS:referer} %{QS:agent} %{QS:xforward}
备注：grok使用的正则表达式在grok-patterns文件
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# find / -name grok-patterns
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# cat /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# vim /etc/logstash/conf.d/nginx-filebeat.conf
input {
#filebeat客户端#
beats {
port => 5044
}
}
# #筛选
filter {
# 如果是nginx访问日志
if ( [type] == "nginx-access-test" ) {
#按字段切割
grok {
patterns_dir=>"/etc/logstash/conf.d/nginx-access"
match => { "message" => "%{NGINXACCESS}" }
}
# 时间格式转换
date {
match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
}
# 删除不需要的字段
mutate {
remove_field => [ "offset", "@version", "beat", "input_type", "tags","id"]
}
}
}
output {
#标准输出，调试使用#
stdout {
codec => rubydebug { }
}
# 输出到es#
elasticsearch {
hosts => ["localhost:9200"]
index => "%{type}-%{+YYYY.MM.dd}"
}
}
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx-filebeat.conf --config.test_and_exit
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-03-03T10:27:56,895][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK
[2019-03-03T10:28:04,969][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl stop logstash
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx-filebeat.conf
然后访问nginx网站，如下表示成功。
{
"source" => "/var/log/nginx/access.log",
"remote_ip" => "127.0.0.1",
"httpversion" => "1.1",
"bytes" => "612",
"meta" => {
"cloud" => {
"instance_id" => "i-j6c23v8zk8gxxvjdylt8",
"availability_zone" => "cn-hongkong-b",
"provider" => "ecs",
"region" => "cn-hongkong"
}
},
"method" => "GET",
"xforward" => "\"-\"",
"agent" => "\"curl/7.29.0\"",
"referer" => "\"-\"",
"timestamp" => "03/Mar/2019:11:11:03 +0800",
"message" => "127.0.0.1 - - [03/Mar/2019:11:11:03 +0800] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.29.0\" \"-\"",
"prospector" => {
"type" => "log"
},
"type" => "nginx-access-test",
"status" => "200",
"input" => {
"type" => "log"
},
"host" => {
"os" => {
"platform" => "centos",
"version" => "7 (Core)",
"family" => "redhat",
"codename" => "Core",
"name" => "CentOS Linux"
},
"id" => "20181129113200424400422638950048",
"name" => "iZj6c23v8zk8gxxvjdylt8Z",
"architecture" => "x86_64",
"containerized" => true
},
"request" => "/",
"@timestamp" => 2019-03-03T03:11:03.000Z,
"log" => {
"file" => {
"path" => "/var/log/nginx/access.log"
}
}
}
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl start logstash [root@iZj6c23v8zk8gxxvjdylt8Z ~]# curl 'localhost:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open .kibana_1 iWQ-ElLnRXaq0mJtsCXe-Q 1 0 2 0 8.6kb 8.6kb yellow open system-syslog-toes-2019.03 ZBfgIsFQRnuXIDU1PKYkkg 5 1 68178 0 5.6mb 5.6mb yellow open nginx-access-test-2019.03.03 YCOqQhpjSMKkqHmvDzxfOA 5 1 8 0 87.4kb 87.4kb yellow open nginx-2019.03.03 lQdtD7pqRFyerEWHfMapvQ 5 1 2 0 24.6kb 24.6kb 备注：上述logstash配置里面的调试输出在调试完成后需要关掉，logstash可以根据input、filter、output和优先级进行设计子文件。
8、配置kibana
9、收集腾讯云CDN日志，安装grafana5.3可视化日志
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# yum install https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-5.3.2-1.x86_64.rpm [root@iZj6c23v8zk8gxxvjdylt8Z ~]# yum install fontconfig freetype* urw-fonts -y [root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl enable grafana-server [root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl start grafana-server [root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl status grafana-server [root@iZj6c23v8zk8gxxvjdylt8Z ~]# grafana-cli plugins install grafana-piechart-panel [root@iZj6c23v8zk8gxxvjdylt8Z ~]# grafana-cli plugins install grafana-clock-panel [root@iZj6c23v8zk8gxxvjdylt8Z ~]# grafana-cli plugins uninstall blackmirror1-singlestat-math-panel [root@iZj6c23v8zk8gxxvjdylt8Z ~]# service grafana-server restart [root@iZj6c23v8zk8gxxvjdylt8Z ~]# rpm -qc grafana /etc/init.d/grafana-server /etc/sysconfig/grafana-server /usr/lib/systemd/system/grafana-server.service 备注：这里介绍安装其它插件，本教程用不到。 #列出可安装查看 [root@iZj6c23v8zk8gxxvjdylt8Z ~]# grafana-cli plugins list-remote #安装完成重启服务 [root@iZj6c23v8zk8gxxvjdylt8Z ~]# service grafana-server restart #默认插件安装目录 [root@iZj6c23v8zk8gxxvjdylt8Z ~]# ll /var/lib/grafana/plugins/
添加如下两行： [root@iZj6cdt6ht4gae9swlcd87Z cdn]# vim /etc/elasticsearch/elasticsearch.yml http.cors.enabled: true http.cors.allow-origin: "*" [root@iZj6c23v8zk8gxxvjdylt8Z ~]# mkdir -p /data/logs/cdn/ [root@iZj6c23v8zk8gxxvjdylt8Z ~]# vim /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /data/logs/cdn/*.com exclude_lines: ["^$"] fields: type: "cdn-log" fields_under_root: true #output.elasticsearch: # Array of hosts to connect to. # hosts: ["localhost:9200"] output.logstash: # The Logstash hosts hosts: ["localhost:5044"] [root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl enable filebeat [root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl restart filebeat [root@iZj6c23v8zk8gxxvjdylt8Z ~]# systemctl status filebeat #filter中alter插件logstash6版本默认没安装，如果安装请按照如下方法： #我们需要用到alter插件。 参考链接：https://www.elastic.co/guide/en/logstash/current/plugins-filters-alter.html [root@iZj6c23v8zk8gxxvjdylt8Z cdn]# cd /usr/share/logstash/bin/ [root@iZj6c23v8zk8gxxvjdylt8Z bin]# ./logstash-plugin install logstash-filter-alter Validating logstash-filter-alter Installing logstash-filter-alter Installation successful
[root@iZj6c23v8zk8gxxvjdylt8Z Kibana_Hanization]# vim /etc/logstash/conf.d/cdn.conf
input {
beats {
port => 5044
}
}
filter {
if [type] == "cdn-log" {
grok {
patterns_dir => ["./patterns"]
match => { "message" => "%{DATESTAMP_EVENTLOG:timestamp} %{IPORHOST:client_ip} %{IPORHOST:server_name} %{NOTSPACE:request} %{NUMBER:bytes} %{NUMBER:province} %{NUMBER:operator} %{NUMBER:status} (?:%{URI:referrer}|%{WORD:referrer}) %{NUMBER:request_time} %{QS:agent} \"\(%{WORD:range}\)\" %{WORD:method} (?:HTTP/%{NUMBER:protocol}|HTTPS) %{WORD:cache}" }
}
date {
match => [ "timestamp", "yyyyMMddHHmmss"]
target => "@timestamp"
}
alter {
condrewrite => [
"province", "22", "北京",
"province", "86", "内蒙古",
"province", "146", "山西",
"province", "1069", "河北",
"province", "1177", "天津",
"province", "119", "宁夏",
"province", "152", "陕西",
"province", "1208", "甘肃",
"province", "1467", "青海",
"province", "1468", "新疆",
"province", "145", "黑龙江",
"province", "1445", "吉林",
"province", "1464", "辽宁",
"province", "2", "福建",
"province", "120", "江苏",
"province", "121", "安徽",
"province", "122", "山东",
"province", "1050", "上海",
"province", "1442", "浙江",
"province", "182", "河南",
"province", "1135", "湖北",
"province", "1465", "江西",
"province", "1466", "湖南",
"province", "118", "贵州",
"province", "153", "云南",
"province", "1051", "重庆",
"province", "1068", "四川",
"province", "1155", "西藏",
"province", "4", "广东",
"province", "173", "广西",
"province", "1441", "海南",
"province", "0", "其他",
"province", "1", "港澳台",
"province", "-1", "境外",
"operator", "2", "中国电信",
"operator", "26", "中国联通",
"operator", "38", "教育网",
"operator", "43", "长城宽带",
"operator", "1046", "中国移动",
"operator", "3947", "中国铁通",
"operator", "-1", "海外运营商",
"operator", "0", "其他运营商"
]
}
}
}
output {
stdout {
codec => rubydebug { }
}
elasticsearch {
hosts => ["localhost:9200"]
index => "%{[type]}-%{+YYYY.MM.dd}"
}
}
[root@iZj6c23v8zk8gxxvjdylt8Z ~]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/cdn.conf --config.test_and_exit Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties [2019-03-03T10:27:56,895][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [2019-03-03T10:28:04,969][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [root@iZj6c7bp1a289r04izstw4Z ~]# chown logstash /var/log/logstash/logstash-slowlog-plain.log [root@iZj6c7bp1a289r04izstw4Z ~]# chown logstash /var/log/logstash/logstash-plain.log [root@iZj6c7bp1a289r04izstw4Z ~]# chown -R logstash /var/lib/logstash/ [root@iZj6c7bp1a289r04izstw4Z ~]# vim /etc/logstash/logstash.yml http.host: "172.31.116.244" [root@iZj6c7bp1a289r04izstw4Z ~]# systemctl restart logstash [root@iZj6c7bp1a289r04izstw4Z ~]# systemctl enable logstash [root@iZj6c7bp1a289r04izstw4Z ~]# netstat -lntp |grep 5044 tcp 0 0 0.0.0.0:10514 0.0.0.0:* LISTEN 5945/java [root@iZj6c7bp1a289r04izstw4Z ~]# netstat -lntp |grep 9600 tcp 0 0 172.31.116.244:9600 0.0.0.0:* LISTEN 6132/java [root@iZj6cdt6ht4gae9swlcd87Z ~]# curl 'localhost:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open cdn-log-2019.03.02 VGN_ljUjRIyerCZu8HHilA 5 1 584 0 1.4mb 1.4mb yellow open cdn-log-2019.03.03 BokIxrVKQpOsDnlgilS8XQ 5 1 1052 0 1.4mb 1.4mb green open .kibana_1 TiwdgoXhSsK6mcUZEJVmMw 1 0 2 0 8.6kb 8.6kb #新日志在目录后，index这里也更新了，而且docs.count数据和日志行数是对的。 [root@iZj6cdt6ht4gae9swlcd87Z cdn]# curl 'localhost:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open cdn-log-2019.03.02 VGN_ljUjRIyerCZu8HHilA 5 1 1720 0 2mb 2mb green open .kibana_1 TiwdgoXhSsK6mcUZEJVmMw 1 0 2 0 8.6kb 8.6kb yellow open cdn-log-2019.03.03 BokIxrVKQpOsDnlgilS8XQ 5 1 8534 0 6.4mb 6.4mb
访问grafana：
备注：注意时间要选择正确，要不没数据显示。