flume消费kafka到hbase

!!! hbase需要配置环境变量
1.压缩包版本
flume版本1.7.0
kafka版本0.10.2.1
2.flume配置文件
进入conf文件
创建 flume-kafka-hdfs.conf文件，配置文件代码如下：

# source alias
agent.sources = source_from_kafka
# channels alias
agent.channels = mem_channel
# sink alias
agent.sinks = hbase_sink
# define kafka source
agent.sources.source_from_kafka.type = org.apache.flume.source.kafka.KafkaSource
agent.sources.source_from_kafka.channels = mem_channel
agent.sources.source_from_kafka.batchSize = 5000
# set kafka broker address  
agent.sources.source_from_kafka.kafka.bootstrap.servers = master:9092,slave1:9092,slave2:9092
# set kafka topic
agent.sources.source_from_kafka.kafka.topics = test
# set kafka groupid
agent.sources.source_from_kafka.kafka.consumer.group.id = test-consumer-group

#输出源到hbase
agent.sinks.hbase_sink.type = hbase
#hbase表
agent.sinks.hbase_sink.table = weizhi
#列族
agent.sinks.hbase_sink.columnFamily = dili
agent.sinks.hbase_sink.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
agent.sinks.hbase_sink.channel = mem_channel
         
agent.channels.mem_channel.type = memory
agent.channels.mem_channel.capacity = 100000
agent.channels.mem_channel.transactionCapacity = 10000
                                                                                                                

3.启动flume命令

./flume-ng agent --conf ../conf/ --name agent --conf-file ../conf/flume-kafka-hdfs.conf -Dflume.root.logger=INFO,console

4.启动kafka

./kafka-server-start.sh  -daemon ../config/server.properties

5.开启发送者并发送消息

./kafka-console-producer.sh --broker-list 192.168.88.5:9092 --topic test

6.启动hbase shell chakan

scan 'weizhi'