flume + kafka 日志采集
版权声明：本文为博主原创文章，如需转载，请注明转载地址： https://blog.csdn.net/liurui_wuhan/article/details/82985265
将系统产生日志信息通过flume采集，推送至kafka进行消费处理
架构图
| 服务 | ip | port | 备注 |
| flume collectors | 10.200.132.181 | 6333 | flume collectors |
| flume agent | 10.200.132.168 | flume采集器(目前使用一个agent) | |
| kafka | 10.200.132.181 | 9092 2181 | kafka和zookeeper |
一台机器部署一个flume agent，如果需要采集多个服务的日志，在配置文件里面可以配置多个collect，
关于flume内部架构的详细介绍，可以参考博客：https://blog.csdn.net/lsjseu/article/details/51670933
关于kafka的安装，前面我已经安装过了，可以参考博客：https://blog.csdn.net/liurui_wuhan/article/details/82984615
本文主要安装flume和如何实现日志采集
一、安装部署
1、下载安装flume
下载地址：http://flume.apache.org/download.html
目前最新是1.8版本 http://www.apache.org/dyn/closer.lua/flume/1.8.0/apache-flume-1.8.0-bin.tar.gz
[root@log-system opt]# tar -zxvf apache-flume-1.8.0-bin.tar.gz
2、10.200.132.181机器上配置
新建 flume-collecters.properties ，用于将收集到flume agent日志 推送到kafka
[root@log-system apache-flume-1.8.0-bin]# cd conf
[root@log-system conf]# vim flume-collecters.properties
#flume collecters
agent.sources = s1Flume
agent.channels = c1
agent.sinks =sinkKafka
# For each one of the sources, the type is defined
agent.sources.s1Flume.channels = c1
agent.sources.s1Flume.type = avro
#flume ip
agent.sources.s1Flume.bind = 10.200.132.181
# flume 端口
agent.sources.s1Flume.port = 6333
# The channel can be defined as follows.
agent.sources.s1Flume.channels = c1
# Each sink's type must be defined
agent.sinks.sinkKafka.type = org.apache.flume.sink.kafka.KafkaSink
# kafka消息队列名称
agent.sinks.sinkKafka.topic = topic-pear
# kafka ip:port
agent.sinks.sinkKafka.brokerList = 10.200.132.181:9092
agent.sinks.sinkKafka.requiredAcks = 1
agent.sinks.sinkKafka.batchSize = 20
agent.sinks.sinkKafka.channel = c1
#Specify the channel the sink should use
#agent.sinks.loggerSink.channel = memoryChannel
# Each channel's type is defined.
agent.channels.c1.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent.channels.c1.capacity = 100启动flume
[root@log-system apache-flume-1.8.0-bin]# bin/flume-ng agent -c conf -f conf/flume-collecters.properties -n agent -Dflume.root.logger=INFO,console,LOGFILE &
查看6333端口是否已启动
3、10.200.132.168上flume配置
[root@cm-elk-02 conf]# vim flume-test-collect.properties
agent.sources = fileSource
agent.channels = memoryChannel
agent.sinks = collecter1
agent.sinkgroups = gCollecters
agent.sinkgroups.gCollecters.sinks = collecter1
#sink调度模式 load_balance failover
agent.sinkgroups.gCollecters.processor.type = failover
#负载均衡模式 轮询 random round_robin
agent.sinkgroups.gCollecters.processor.selector=round_robin
#失效降级
agent.sinkgroups.gCollecters.processor.backoff=true
#降级时间30秒
agent.sinkgroups.gCollecters.processor.maxTimeOut=30000
agent.sources.fileSource.type = exec
# 监控的日志文件
agent.sources.fileSource.command = tail -F /opt/test/logs/test.log
#agent.sources.fileSource.charset=utf-8
agent.sources.fileSource.channels = memoryChannel
agent.sources.fileSource.restartThrottle = 10000
agent.sources.fileSource.restart = true
agent.sources.fileSource.logStdErr = true
# Each sink's type must be defined
agent.sinks.collecter1.channel = memoryChannel
agent.sinks.collecter1.type = avro
# flume 服务ip
agent.sinks.collecter1.hostname = 10.200.132.181
# flume 端口
agent.sinks.collecter1.port = 6333
agent.sinks.collecter1.batch-size = 10
# Each channel's type is defined.
agent.channels.memoryChannel.type = memory
# Other config values specific to each type of channel(sink or source)
#The max number of events stored in the channel
agent.channels.memoryChannel.capacity = 100
#The max number of events stored in the channel per transaction
agent.channels.memoryChannel.transactionCapacity = 100
#Timeout in seconds for adding or removing an event
agent.channels.memoryChannel.keep-alive=30
创建日志目录（如果没有就创建）
[root@cm-elk-02 conf]# mkdir -p /opt/test/logs/启动服务
[root@cm-elk-02 apache-flume-1.8.0-bin]# bin/flume-ng agent -c conf -f conf/flume-test-collect.properties -n agent -Dflume.root.logger=INFO,console,LOGFILE
部署配置基本完成
二、验证
登录10.200.132.181服务器，执行kafka消费消息
[root@log-system ~]# /opt/kafka_2.12-2.0.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic-pear --from-beginning登录10.200.132.168服务器，往日志文件写日志
[root@cm-elk-02 logs]# echo "hello world" >>test.log写入完成之后，大概等几秒钟，就可以看到kafka消费者消费的队列信息了。
自己也可以写一个springboot程序产生日志，修改flume agent 监控的日志目录文件，就可以实时的将日志通过flume推送至kafka