ELK+Zookeeper+Kafka+Filebeat
一、实验目的：
1. 强大的搜索功能，elasticsearch可以以分布式搜索的方式快速检索，而且支持DSL的语法来进行搜索，简单的说，就是通过类似配置的语言，快速筛选数据。
2. 完美的展示功能，可以展示非常详细的图表信息，而且可以定制展示内容，将数据可视化发挥的淋漓尽致。
3. 分布式功能，能够解决大型集群运维工作很多问题，包括监控、预警、日志收集解析等。
二、实验逻辑思路：
1，使用Filebeat这个工具安装到客户端。
2，接收到的日志文件推送给Kafka上存储。
3，Zookeeper来协调管理Kafka日志队列最终推送给Logstash集群处理。
3，Logstash集群处理过的日志文集推送给Easticsearch集群来处理日志文件。
4，Easticsearch集群处理过后把信息推送给Kibana。
5，Kibana通过图片、图形、网页形式呈现给用户。
三、安装部署
1.为了方便实验的顺利进行，Linux防火墙和selinux我建议还是关闭比较好的选择，不必要为此折腾时间。
[root@server1 ~]# systemctl stop firewalld [root@server1 ~]#systemctl disable firewalld [root@server1 ~]#sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
2，实验要求的软件版本。
1，Easticsearch-6.3.2 2，Logstash-6.3.2 3，Kibana-6.3.2 4，JDK 1.8.0_181 5，zookeeper-3.4.12 6，filebeat-6.3.2-linux-x86_64 7，Kafka_2.10-0.10.0.1
3，在第一台主机上的配置server1。
[root@server1 ~]# rpm -qa |grep jdk [root@server1 ~]# rpm -e --nodeps java-1.7.0-openjdk-1.7.0.171-2.6.13.2.el7.x86_64 [root@server1 ~]# mkdir -pv /usr/local/ [root@server1 ~]# tar xf jdk-8u181-linux-x64.tar.gz -C /usr/local/ [root@server1 ~]# vim /etc/profile JAVA_HOME=/usr/local/jdk1.8.0_181 export PATH=$PATH:$JAVA_HOME/bin [root@server1 ~]# source /etc/profile [root@server1 ~]# java -version java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) [root@server1 ~]# tar -xf elasticsearch-6.4.2.tar.gz -C /usr/local/ [root@server1 ~]# cd /usr/local [root@server1 local]# mv elasticsearch-6.4.2 elasticsearch [root@server1 local]# groupadd elasticsearch [root@server1 local]# useradd -g elasticsearch elasticsearch -m [root@server1 local]# chown -R elasticsearch.elasticsearch /usr/local/ [root@server1 local]# ll /usr/local/elasticsearch/ [root@server1 local]# mkdir -p /data/elasticsearch [root@server1 local]# chown -R elasticsearch.elasticsearch /data/elasticsearch [root@server1 local]# cd /usr/local/elasticsearch/config/ [root@server1 config]# cp elasticsearch.yml elasticsearch.yml.bak
4，server1 上修改elasticsearch.yml配置文件。
[root@server1 ~]# vim /usr/local/elasticsearch/config/elasticsearch.yml cluster.name: ELK-Cluster node.name: server1 node.master: true node.data: true path.data: /data/elasticsearch path.logs: /usr/local/elasticsearch/logs network.host: 0.0.0.0 http.port: 9200 discovery.zen.minimum_master_nodes: 1 discovery.zen.ping_timeout: 3s http.cors.enabled: true http.cors.allow-origin: "*" discovery.zen.ping.unicast.hosts: ["10.93.58.66:9300","10.93.58.41:9300"]
5，在server1修改系统配置参数。
[root@server1 config]# echo "vm.max_map_count=262144" >> /etc/sysctl.conf [root@server1 config]# echo "fs.file-max=655360" >> /etc/sysctl.conf [root@server1 config]# sysctl -p vm.max_map_count = 262144 fs.file-max = 655360 [root@server1 config]# vim /etc/security/limits.conf 最后加上 * soft nproc 65536 * hard nproc 65536 * soft nofile 65536 * hard nofile 65536 [root@server1 config]# vim /etc/security/limits.d/20-nproc.conf * soft nproc 20480
6，切换用户启动elasticsearch。
earch [root@server1 ~]# cd /usr/local/elasticsearch/ [root@server1 elasticsearch]# ./bin/elasticsearch -d [root@server1 config]# netstat -lantup |grep java
7，server1安装elasticsearch-head。
[root@server1 ~]# yum install epel-release.noarch -y [root@server1 ~]# yum install -y nodejs npm [root@server1 ~]# yum install -y git [root@server1 ~]# cd /usr/local/ [root@server1 ~]# git clone git://github.com/mobz/elasticsearch-head.git [root@server1 ~]# npm config set registry=http://registry.npm.taobao.org/ [root@server1 ~]#cd elasticsearch-head/ [root@server1 elasticsearch-head]# npm install [root@server1 elasticsearch-head]# cd _site [root@server1 _site]# vim app.js "http://10.93.58.66:9200"; [root@server1 _site]# su - elasticsearch [elasticsearch@server1 ~]$ cd /usr/local/elasticsearch-head/ [elasticsearch@server1 elasticsearch-head]$ npm run start > [email protected] start /usr/local/elasticsearch-head > grunt server Running "connect:server" (connect) task Waiting forever... Started connect web server on http://localhost:9100
8，server2安装elasticsearch.yml配置。
cluster.name: ELK-Cluster node.name: server2 node.master: true node.data: true path.data: /data/elasticsearch path.logs: /usr/local/elasticsearch/logs network.host: 0.0.0.0 http.port: 9200 discovery.zen.minimum_master_nodes: 1 discovery.zen.ping_timeout: 3s http.cors.enabled: true http.cors.allow-origin: "*" discovery.zen.ping.unicast.hosts: ["10.93.58.66:9300","10.93.58.41:9300"]
9，server3安装elasticsearch.yml配置
cluster.name: ELK-Cluster node.name: server3 node.master: false node.data: true path.data: /data/elasticsearch path.logs: /usr/local/elasticsearch/logs network.host: 0.0.0.0 http.port: 9200 discovery.zen.minimum_master_nodes: 1 discovery.zen.ping_timeout: 3s http.cors.enabled: true http.cors.allow-origin: "*" discovery.zen.ping.unicast.hosts: ["10.93.58.66:9300","10.93.58.41:9300"]
10，kafkzk安装kafkzk-3.4.13配置。
关闭防火墙和selinux [root@kafkzk~]# systemctl stop firewalld [root@kafkzk~]#systemctl disable firewalld [root@kafkzk~]#sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
11，解决一些依赖的问题。
[root@kafkzk ~]#yum install cppunit -y [root@kafkzk ~]#yum install python-setuptools -y
12，新建目录上传Java安装包安装Java并配置环境变量。
[root@kafkzk ~]#rz [root@kafkzk ~]# mkdir -pv /usr/local/ [root@kafkzk ~]# tar xf jdk-8u181-linux-x64.tar.gz -C /usr/local/ [root@kafkzk ~]# vim /etc/profile 添加如下内容： JAVA_HOME=/usr/local/jdk1.8.0_181 export PATH=$PATH:$JAVA_HOME/bin [root@server1 ~]# source /etc/profile [root@server1 ~]# java -version java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
13，安装kafkzk加入环境变量并启动服务。
[root@kafkzk ~]#tar xf zookeeper-3.4.13.tar.gz -C /usr/local/ [root@kafkzk ~]#mv /usr/local/zookeeper-3.4.13 /usr/local/zookeeper [root@kafkzk ~]#cd /usr/local/zookeeper/conf [root@kafkzk ~]# cp zoo_sample.cfg zoo_sample.cfg.bak [root@kafkzk ~]#mv zoo_sample.cfg zoo.cfg [root@kafkzk ~]# grep -v "^*" /usr/local/zookeeper/conf/zoo_sample.cfg.bak |grep -v "^#" >/usr/local/zookeeper/conf/zoo.cfg [root@kafkzk ~]#vim zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data/zookeeper clientPort=2181 server.1=10.93.58.66:2888:3888 server.2=10.93.58.41:2888:3888 server.3=10.93.58.209:2888:3888
14，修改环境变量，添加如下内容。
[root@kafkzk ~]#vim /etc/profile export ZOOKEEPER_HOME=/usr/local/zookeeper export PATH=$PATH:$ZOOKEEPER_HOME/bin [root@kafkzk ~]#source /etc/profile [root@kafkzk ~]# zkServer.sh start ZooKeeper JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [root@kafkzk ~]# jps 1521 Jps /usr/local/logstash
15，未完成待写。