Kafka集成Flume 客户端测试实现

前言:测试环境

LINUX:centos6.5

FLUME:1.6.0-cdh5.7.0

KAFKA:2.12-0.11.0


一、定义Kafka所使用的Topic

启动Kafka:

$ kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties

创建Topic:

$ kafka-topics.sh --create --zookeeper hadoopmaster:2181 --replication-factor 1 --partitions 1 --topic test_topic

查看Topic是否创建成功:

$ kafka-topics.sh --list --zookeeper hadoopmaster:2181


二、定义Flume Agent的类型

1、配置Agent1,类型为exec-memory-avro(监听某个文档是否有文档生成)

#定义agent name,source,sinks,channels的对应名称

exec-memory-avro.sources=exec-source

exec-memory-avro.sinks=avro-sink

exec-memory-avro.channels=memory-channel

#定义source为exec,监听目录及方式

exec-memory-avro.sources.exec-source.type=exec

exec-memory-avro.sources.exec-source.command=tail -F /home/hadoop/data/test.log

exec-memory-avro.sources.exec-source.shell=/bin/sh -c

#定义sinks为avro,对应的主机名及端口

exec-memory-avro.sinks.avro-sink.type = avro

exec-memory-avro.sinks.avro-sink.hostname = hadoopmaster

exec-memory-avro.sinks.avro-sink.port = 22222

#定义channel为memory方式,统一source和sink的channel

exec-memory-avro.channels.memory-channel.type = memory

exec-memory-avro.sources.exec-source.channels = memory-channel

exec-memory-avro.sinks.avro-sink.channel = memory-channel

2、配置Agent2,类型为avro-memory-kafka(监听某个文档是否有文档生成)

#同上,接收agent1 avro sink的消息

avro-memory-kafka.sources=avro-source

avro-memory-kafka.sinks=kafka-sink

avro-memory-kafka.channels=memory-channel

avro-memory-kafka.sources.avro-source.type=avro

avro-memory-kafka.sources.avro-source.bind=hadoopmaster

avro-memory-kafka.sources.avro-source.port=22222

#Kafka sinks配置,详情请参考flume官方文档“Kafka Sink”

avro-memory-kafka.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink

avro-memory-kafka.sinks.kafka-sink.brokerList = hadoopmaster:9092

avro-memory-kafka.sinks.kafka-sink.topic = test_topic

avro-memory-kafka.sinks.kafka-sink.batchSize = 5  ##当消息量多少条才处理

avro-memory-kafka.sinks.kafka-sink.requiredAcks = 1

avro-memory-kafka.channels.memory-channel.type = memory

avro-memory-kafka.sources.avro-source.channels = memory-channel

avro-memory-kafka.sinks.kafka-sink.channel = memory-channel


三、Flume与Kafka启动

1、先启动agent2(avro-memory-kafka)

$ flume-ng agent --name avro-memory-kafka --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/avro-memory-kafka -Dflume.root.logger = INFO,console

2、启动agent1(exec-memory-avro--flume)

$ flume-ng agent --name exec-memory-avro --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/exec-memory-avro.conf -Dflume.root.logger = INFO,console


3、启动Kafka Consumer

 $ kafka-console-consumer.sh --zookeeper hadoopmaster:2181 --topic test_topic


四、在监听目录上进行测试

$ echo test1 >> test.log

$ echo test2 >> test.log

$ echo test3 >> test.log

$ echo test4 >> test.log

$ echo test5 >> test.log

$ echo test6 >> test.log

若启动的Kafka Consumer控制台上,有对应消息输出,则测试成功。

你可能感兴趣的:(Kafka集成Flume 客户端测试实现)