flume采坑

问题1

安装配置kafka ---> flume ---> hdfs报错"responseBody={error_code=15,coordinator={node_id=-1,host=,port=-1}}"

flume配置文件如下

agent.sources = kafkaSource
agent.channels = memoryChannel
agent.sinks = hdfsSink
# The channel can be defined as follows.
agent.sources.kafkaSource.channels = memoryChannel
agent.sources.kafkaSource.type=org.apache.flume.source.kafka.KafkaSource
agent.sources.kafkaSource.kafka.bootstrap.servers = slave01:9092,slave02:9092,slave03:9092
agent.sources.kafkaSource.topic=maxwell

#agent.sources.kafkaSource.groupId=flume
agent.sources.kafkaSource.kafka.consumer.timeout.ms=100

agent.channels.memoryChannel.type=memory
agent.channels.memoryChannel.capacity=1000
agent.channels.memoryChannel.transactionCapacity=100


# the sink of hdfs
agent.sinks.hdfsSink.type=hdfs
agent.sinks.hdfsSink.channel = memoryChannel
agent.sinks.hdfsSink.hdfs.path=hdfs://master01:8020/flume
agent.sinks.hdfsSink.hdfs.writeFormat=Text
agent.sinks.hdfsSink.hdfs.fileType=DataStream

讲道理这个配置完成以后，我只要输入如下命令立刻妥妥地跑起来，但是一直报错"responseBody={error_code=15,coordinator={node_id=-1,host=,port=-1}}"

flume-ng agent --conf conf --conf-file example.conf  --name kafka   -Dflume.root.logger=INFO,console

解决

参考如下文章：https://stackoverflow.com/questions/42362911/kafka-high-level-consumer-error-code-15/42416232#42416232，我把kafka中的信息删除完后解决问题。

kafka系列之在zookeeper中存储结构参看下图：

kafka系列之在zookeeper中存储结构