ELK+filebeat+kafka

ELK 流程图：

filebeat --> kafka --> logstash --> elasticsearch --> kibana

---

1.部署kafka

安装JAVA1.8(略过) 

下载 kafka_2.11-0.10.0.0.tgz

下载地址：http://kafka.apache.org/downloads.html

# yum -y install glibc.i686

# tar zxvf kafka_2.11-0.10.0.0.tgz

# mv kafka_2.11-0.10.0.0 kafka

配置zookeeper

# cd kafka

# [root@yqvm-elk kafka]# cat config/zookeeper.properties |grep -v ^# |grep -v ^$

dataDir=/tmp/zookeeper

clientPort=2181

maxClientCnxns=100

tickTime=2000

initLimit=10

启动zookeeper

# nohup ./bin/zookeeper-server-start.sh config/zookeeper.properties &

配置kafka

# cat config/server.properties |grep -v ^# |grep -v ^$

broker.id=0

listeners=PLAINTEXT://192.168.10.12:9092         #本机IP

num.network.threads=3

num.io.threads=8

socket.send.buffer.bytes=102400

socket.receive.buffer.bytes=102400

socket.request.max.bytes=104857600

log.dirs=/tmp/kafka-logs

num.partitions=1

num.recovery.threads.per.data.dir=1

log.retention.hours=168

log.segment.bytes=1073741824

log.retention.check.interval.ms=300000

zookeeper.connect=192.168.10.12:2181          #本机IP

zookeeper.connection.timeout.ms=6000

启动kafka

# nohup ./bin/kafka-server-start.sh config/server.properties &

---

2.部署filebeat

下载 filebeat-5.4.0 

下载地址：https://www.elastic.co/downloads

# rpm -ivh filebeat-5.4.0-x86_64.rpm

配置filebeat

# cat /etc/filebeat/filebeat.yml

filebeat.prospectors:

- input_type: log

paths:

- /home/test/tomcat01/logs/catalina.out

document_type: tomcat01

tail_files: true

- input_type: log

paths:

- /home/test/tomcat02/logs/catalina.out

document_type: tomcat02

tail_files: true

output.kafka:

enabled: true

hosts: ["192.168.10.12:9092"]

topic: test-tomcat

启动filebeat

# service filebeat start

测试filebeat日志是否输出到kafka

实时查看kafka消费

# ./bin/kafka-console-consumer.sh --zookeeper 192.168.10.12:2181 --topic test-tomcat

再新开一个会话，写点日志

# echo "haha" >> /home/test/tomcat01/logs/catalina.out

返回查看kafka消费

# ./bin/kafka-console-consumer.sh --zookeeper 192.168.10.12:2181 --topic test-tomcat

{"@timestamp":"2017-07-19T18:11:05.881Z","beat":{"hostname":"localhost.localdomain","name":"localhost.localdomain","version":"5.4.0"},"input_type":"log","message":"haha","offset":55,"source":"/home/test/tomcat01/logs/catalina.out","type":"tomcat01"}

如果出现上面的内容。说明filebeat 输出日志到 kafka 成功。

未完待续。