kafka与flume 的应用（实战）

版本号：

RedHat6.5   JDK1.8    flume-1.6.0   kafka_2.11-0.8.2.1

1.flume安装

RedHat6.5安装单机flume1.6： http://blog.leanote.com/post/[email protected]/26781d33b435

2.kafka安装

RedHat6.5安装kafka集群 ：  http://blog.leanote.com/post/[email protected]/0230848f841a

3.Flume和Kafka整合

在conf目录新建flume-kafka.conf文件：

1. touch /usr/local/flume/apache-flume-1.6.0-bin/conf/flume-kafka.conf
2. sudo gedit /usr/local/flume/apache-flume-1.6.0-bin/conf/flume-kafka.conf

输入以下内容：

1. # 指定Agent的组件名称  
2. agent1.sources = source1  
3. agent1.sinks = sink1  
4. agent1.channels = channel1  
5.  
6. # 指定Flume source(要监听的路径)  
7. agent1.sources.source1.type = spooldir  
8. agent1.sources.source1.spoolDir = /usr/local/flume/logtest
9.  
10. # 指定Flume sink  
11. #agent1.sinks.sink1.type = logger  
12. agent1.sinks.sink1.type = org.apache.flume.sink.kafka.KafkaSink  
13. agent1.sinks.sink1.topic = test  
14. agent1.sinks.sink1.brokerList = 192.168.168.200:9092  
15. agent1.sinks.sink1.requiredAcks = 1  
16. agent1.sinks.sink1.batchSize = 100   
17.  
18. # 指定Flume channel  
19. agent1.channels.channel1.type = memory  
20. agent1.channels.channel1.capacity = 1000  
21. agent1.channels.channel1.transactionCapacity = 100  
22.  
23. # 绑定source和sink到channel上  
24. agent1.sources.source1.channels = channel1  
25. agent1.sinks.sink1.channel = channel1  

agent1.sinks.sink1.topic = test   代表flume监听路径下发生变化时，会把消息发送到localhost机器上的test主题。

启动flume-kafka.conf：

1. cd /usr/local/flume/apache-flume-1.6.0-bin
2. bin/flume-ng agent --conf conf --conf-file conf/flume-kafka.conf --name agent1 -Dflume.root.logger=INFO,console

运行成功日志如下：

1. 2017-07-07 22:22:02,270 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:120)] Monitored counter group for type: SINK, name: sink1: Successfully registered new MBean.
2. 2017-07-07 22:22:02,270 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: SINK, name: sink1 started

启动kafka的消费者，监听topic主题：

1. kafka-console-consumer.sh --zookeeper localhost:2181 --topic test

testKafka.log ：

在/usr/local/flume目录下面新建一个testKafka.log日志文件，写入Flume connect Kafka success! 作为测试内容：

1. touch /usr/local/flume/testKafka.log
2. sudo gedit /usr/local/flume/testKafka.log

 

然后拷贝testKafka.log到flume监听路径/usr/local/flume/logtest下：

1. cp /usr/local/flume/testKafka.log /usr/local/flume/logtest

接着就可以在前一个终端看到刚刚采集的内容了,如下：

---------------------------------kafka------------------------------

1. [root@master kafka_2.11-0.9.0.0]# kafka-console-consumer.sh --zookeeper localhost:2181 --topic test  
2. [2017-07-07 22:36:38,687] INFO [Group Metadata Manager on Broker 200]: Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
3. Flume connect Kafka success!

 ---------------------------------flume------------------------------

1. 2017-07-07 22:41:32,602 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.rollCurrentFile(ReliableSpoolingFileEventReader.java:348)] Preparing to move file /usr/local/flume/logtest/testKafka.log to /usr/local/flume/logtest/testKafka.log.COMPLETED
2. 2017-07-07 22:41:35,669 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - kafka.utils.Logging$class.info(Logging.scala:68)] Fetching metadata from broker id:0,host:localhost,port:9092 with correlation id 0 for 1 topic(s) Set(test)
3. 2017-07-07 22:41:35,728 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - kafka.utils.Logging$class.info(Logging.scala:68)] Connected to localhost:9092 for producing
4. 2017-07-07 22:41:35,757 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - kafka.utils.Logging$class.info(Logging.scala:68)] Disconnecting from localhost:9092
5. 2017-07-07 22:41:35,791 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - kafka.utils.Logging$class.info(Logging.scala:68)] Connected to slave2:9092 for producing