Flume HA的配置
在此处我们为了防止单点故障，把flume配置成HA的高可用
1、首先准备3台机器，分别是hdp-01、hdp-02、hdp-03，其中hdp-01为source机器，而hdp-02、hdp-03为Flume高可用集群。
2、我们先设置hdp-02、hdp-03两台机器。cd apps/flume/myconf/avro-hdfs.conf （此处source端为avro模式，sink端为hdfs模式）
hdp-02机器中的配置信息
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = hdp-02
a1.sources.r1.port = 4141
# Describe the sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = /flume/events/
a1.sinks.k1.hdfs.filePrefix = events-
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
hdp-03机器中的配置信息
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = hdp-03
a1.sources.r1.port = 4141
# Describe the sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = /flume/events/
a1.sinks.k1.hdfs.filePrefix = events-
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
3、最后设置hdp-01机器中的配置信息
hdp-01机器中的配置信息
# Name the components on this agent
a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1
#set group，因为是HA，所以我们给HA集群中的机器配置一个sink组
a1.sinkgroups = g1
# Describe/configure the source
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /root/flumeSpool
a1.sources.r1.fileHeader = true
# Describe the sink
# set sink1，其实，sink1就是k1
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = hdp-03
a1.sinks.k1.port = 4141
# set sink2，其实，sink2就是k2
#a1.sinks.k2.channel = c1
a1.sinks.k2.type = avro
a1.sinks.k2.hostname = hdp-02
a1.sinks.k2.port = 4141
#set sink group
a1.sinkgroups.g1.sinks = k1 k2
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c1
#set failover,故障转移的配置信息，惩罚因子
a1.sinkgroups.g1.processor.type = failover
a1.sinkgroups.g1.processor.priority.k1 = 10
a1.sinkgroups.g1.processor.priority.k2 = 1
a1.sinkgroups.g1.processor.maxpenalty = 10000