上一篇准备知识:https://www.jianshu.com/p/dd1a3a0df63f
一、ELK简介
开源实时日志分析ELK平台能够完美的解决我们上述的问题，ELK由ElasticSearch、Logstash和Kiabana三个开源工具组成：
ElasticSearch是一个基于Lucene的开源分布式搜索服务器。它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。它提供了一个分布式多用户能力的全文搜索引擎，基于RESTful web接口。Elasticsearch是用Java开发的，并作为Apache许可条款下的开放源码发布，是最流行的企业搜索引擎。设计用于云计算中，能够达到实时搜索，稳定，可靠，快速，安装使用方便。
在ElasticSearch中，所有节点的数据是均等的。
Logstash是一个完全开源的工具，他可以对你的日志进行收集、过滤、分析，并将其存储供以后使用（如，搜索），您可以使用它。说到搜索，logstash带有一个web界面，搜索和展示所有日志。
logstash收集日志基本流程: input-->codec-->filter-->codec-->output
input:从哪里收集日志。
filter:发出去前进行过滤
output:输出至Elasticsearch或Redis消息队列
codec:输出至前台，方便边实践边测试
数据量不大日志按照月来进行收集
Kibana 是一个基于浏览器页面的Elasticsearch前端展示工具，也是一个开源和免费的工具，它Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助您汇总、分析和搜索重要数据日志。
二、部署环境：
CentOS release 6.7，elasticsearch6.2.2，logstash62.2,kibana6.2.2;虚拟机内存要大于2G。关闭防火墙和SELinux。
1.java环境(es需要安装java才能使用)
[root@localhost ~]# yum install java
[root@localhost ~]# java -version
openjdk version "1.8.0_161"
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)
2、安装Elasticsearch
2.1. 导入elasticsearch PGP key
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
2.2.推荐去安装新版本的
地址:https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html
2.3.添加elasticsearch的yum仓库
[root@localhost ~]# vim /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
2.4.安装ElasticSearch
[root@localhost ~] # yum install -y elasticsearch
2.5.yum安装使用普通用户,需要配置limits
[root@localhost ~]# vim /etc/security/limits.conf
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
2.6. 创建数据目录并设置权限
[root@localhost ~]# mkdir -p /data/es-data
[root@localhost ~]# chown -R elasticsearch:elasticsearch /data/es-data
2.7. yum安装 默认配置文件在 /etc/elasticsearch
[root@localhost ~]# vim /etc/elasticsearch/elasticsearch.yml
[root@localhost ~]# grep '^[a-z]' /etc/elasticsearch/elasticsearch.yml
cluster.name: menghuan-elk-cluster #集群名称
node.name: linux-node #节点名称
path.data: /data/es-data #数据存放路径
path.logs: /var/log/elasticsearch #日志存放路径
bootstrap.memory_lock: false #6.x下测试 开启会有问题 或许是内存不充足导致还行调研
network.host: 192.168.56.11 #允许访问的ip
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.56.10","192.168.56.11"] #单播方式 组播在6.x下面貌似不好使
[root@localhost ~]# chkconfig --add elasticsearch
[root@localhost ~]# chkconfig elasticsearch on
[root@localhost ~]# chkconfig --list |grep elasticsearch
elasticsearch 0:off 1:off 2:on 3:on 4:on 5:on 6:off
[root@localhost ~]# service elasticsearch start
Starting elasticsearch: Exception in thread "main" 2018-03-10 09:39:39,721 main ERROR No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: ParsingException[Failed to parse object: expecting token of type [START_OBJECT] but found [VALUE_STRING]];
报错 解决: 参数的冒号前后没有加空格,加了之后就好
然后重新启动后显示:Starting elasticsearch: [ OK ] 这个时候还是有一些问题 通过查看/var/log/elasticsearch/menghuan-elk-culster.log日志文件
发现有以下2个错误
[1]: max number of threads [1024] for user [elasticsearch] is too low, increase to at least [4096]
[2]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
解决如下:
[1]: [root@localhost init.d]# vim /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.
* soft nproc 1024
* soft nproc 4096
root soft nproc unlimited
[2]:修改/etc/elasticsearch/elasticsearch.yml 增加配置：bootstrap.system_call_filter: false
修复完以上2个问题后重启 就可以启动成功了
[root@localhost ~]# ps -ef|grep elasticsearch
495 25551 1 73 10:21 ? 00:00:23 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.Cp7sEmq3 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/elasticsearch/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid -d
2.8. 6.x新版本 请安装此插件
/usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack
到此ES安装完毕
3.安装kibana
3.1. 推荐安装新版本
地址: https://www.elastic.co/guide/en/kibana/current/rpm.html
3.2.添加kibana到yum仓库
[root@localhost ~]# vim /etc/yum.repos.d/kibana.repo
[kibana-6.x]
name=Kibana repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
3.4.安装kibana
[root@localhost ~]# yum install -y kibana
3.5 kibana自启动
[root@localhost ~]# chkconfig --add kibana
[root@localhost ~]# chkconfig kibana on
[root@localhost ~]# chkconfig --list |grep kibana
kibana 0:off 1:off 2:on 3:on 4:on 5:on 6:off
3.6. 修改kibana配置文件
[root@localhost ~]# vim /etc/kibana/kibana.yml
[root@localhost kibana]# grep '^[a-z]' /etc/kibana/kibana.yml
server.port: 5601 #访问端口
server.host: "192.168.56.10" #允许访问主机 (建议内网)
elasticsearch.url: "http://192.168.56.10:9200" #es地址
kibana.index: ".kibana"
3.7. 运行Kibana
[root@localhost ~]# service kibana start
kibana started
[root@localhost ~]# ps -ef|grep kibana
kibana 2528 1 43 07:59 pts/0 00:00:02 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml
root 2537 1802 0 07:59 pts/0 00:00:00 grep kibana
3.8.访问kibana_url：http://192.168.56.10:5601/
3.9.创建kibana索引（创建之前得先装logstash，把数据给ES才行）
4.安装logstash
4.1.推荐安装新版本
地址:https://www.elastic.co/guide/en/logstash/current/installing-logstash.html#package-repositories
4.2.添加logstash的yum仓库
[root@localhost ~] # vim /etc/yum.repos.d/logstash.repo
[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
4.3.安装logstash
[root@localhost ~]# yum install -y logstash
4.4.配置Logstash以及自启动
logstash6生成init脚本后台启动
默认情况使用rpm包安装完logstash之后没有启动脚本，这一点我觉得算是开发不够彻底。官网给了一个脚本，需要根据不同的系统版本生成对应的启动脚本，而且官网没有给明使用方法，对于新用户来说算是个坑，不过在终端可以查看到脚本的使用帮助。
我的系统是CentOS6，这里说一下6的使用方法，7类似，具体方法如下：
4.4.1.查看脚本使用帮助
[root@localhost ~]# cd /usr/share/logstash/bin/
[root@localhost ~]# ./system-install --help
Usage: system-install [OPTIONSFILE] [STARTUPTYPE] [VERSION]
NOTE: These arguments are ordered, and co-dependent
OPTIONSFILE: Full path to a startup.options file
OPTIONSFILE is required if STARTUPTYPE is specified, but otherwise looks first
in /usr/share/logstash/config/startup.options and then /etc/logstash/startup.options
Last match wins
STARTUPTYPE: e.g. sysv, upstart, systemd, etc.
OPTIONSFILE is required to specify a STARTUPTYPE.
VERSION: The specified version of STARTUPTYPE to use. The default is usually preferred here, so it can safely be omitted.
Both OPTIONSFILE & STARTUPTYPE are required to specify a VERSION.
a、要跟startup.options文件的绝对路径，如果是rpm安装的在/etc/logstash/startup.options，如果是二进制包解压安装的则在解压目录下的config目录下面。
b、必须要跟启动类型，比如CentOS6是sysv，CentOS7是systemd。
4.4.2.执行脚本生成启动文件
[root@localhost ~]# /usr/share/logstash/bin/system-install /etc/logstash/startup.options sysv
Successfully created system startup script for Logstash
4.4.3.执行完就可以使用init脚本启动logstash了
[root@localhost ~]# service logstash start
logstash started
[root@localhost ~]# ps -ef|grep logstash
logstash 3086 1 99 09:36 pts/0 00:00:09 /usr/bin/java -Xms256m -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.5.jar:/usr/share/logstash/logstash-core/lib/jars/guava-22.0.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.0.8.jar:/usr/share/logstash/logstash-core/lib/jars/javac-shaded-9-dev-r4023-3.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings /etc/logstash
4.4.4 关闭logstash
rpm 安装的logstash有 initctl 的守护进程
使用initctl stop logstash来关闭 logstash
4.5.测试
[root@localhost ~]# /usr/share/logstash/bin/logstash -e 'input { stdin {} } output { stdout{codec => rubydebug} }'
虽然可以使用 但是会出现一些warngin如下图:
解决如下:
mkdir -p /usr/share/logstash/config/
ln -s /etc/logstash/* /usr/share/logstash/config
chown -R logstash:logstash /usr/share/logstash/config/
再次测试不提示warning
