Flume的汇总节点的配置

1 配置分析

因为汇总节点的sources是其他flume节点的输出.Avro Source
channel一般选择 Memory Channel. 有几个sink就需要有几个channel
sink需要分2个,一个给hbase落地,用AsyncHBaseSink,一个给kafka进行实时分析.用Kafka Sink

2 AsyncHBaseSink的配置详情

image.png

对channel table(表名) type cloumnFamily(列簇)必须填
对serialzer类需要自己下载源码二次开发
payloadColumn表示列簇下的列名

3 配置详情

image.png

source需要给2个channel用，所以，agent1.sources.r1.channels = hbaseC,kafkaC

image.png

注意把下面的c1切换成hbaseC

image.png

image.png

image.png

最终版本

agent1.sources = r1
agent1.channels = kafkaC hbaseC
agent1.sinks = kafkaSink hbaseSink
# **************************flume+Hbase***************************
agent1.sources.r1.type = avro
agent1.sources.r1.channels = hbaseC kafkaC
agent1.sources.r1.bind = bigdata-pro01.kfk.com
agent1.sources.r1.port = 5555
agent1.sources.r1.threads = 5

agent1.channels.hbaseC.type = memory
agent1.channels.hbaseC.capacity = 100000
agent1.channels.hbaseC.transactionCapacity = 100000
agent1.channels.hbaseC.keep-alive = 20

agent1.sinks.hbaseSink.type = asynchbase
agent1.sinks.hbaseSink.table = weblog
agent1.sinks.hbaseSink.columnFamily = info
agent1.sinks.hbaseSink.serializer = org.apache.flume.sink.hbase.KfkAsyncHbaseEventSerializer
agent1.sinks.hbaseSink.channel = hbaseC
agent1.sinks.hbaseSink.serializer.payloadColumn = datatime,userid,searchname,retorder,cliorder,cliurl

#******************************flume+kafka**********************************

agent1.channels.kafkaC.type = memory
agent1.channels.kafkaC.capacity = 100000
agent1.channels.kafkaC.transactionCapacity = 100000
agent1.channels.kafkaC.keep-alive = 20

agent1.sinks.kafkaSink.channel = kafkaC
agent1.sinks.kafkaSink.type = org.apache.flume.sink.kafka.KafkaSink
agent1.sinks.kafkaSink.kafka.topic = weblog
agent1.sinks.kafkaSink.kafka.bootstrap.servers = bigdata-pro01.kfk.com:9092,bigdata-pro02.kfk.com:9092,bigdata-pro03.kfk.com:9092
agent1.sinks.kafkaSink.zookeeperConnect = bigdata-pro01.kfk.com:2181,bigdata-pro02.kfk.com:2181,bigdata-pro03.kfk.com:2181
agent1.sinks.kafkaSink.requiredAcks = 1
agent1.sinks.kafkaSink.batchSize = 1
agent1.sinks.kafkaSink.serializer.class = kafka.serializer.StringEncoder