Flume企业级应用 - 扇入扇出
水善利万物而不争，处众人之所恶，故几于道
文章目录
-
- 1. 扇入、扇出
- 2. 示例
-
- 1. 扇入（聚合）
-
- a1配置：a1_netcat_mem_avro.conf
- a2配置：a2_taildir_mem_avro.conf
- a3配置：a3_avro_mem_logger.conf
- 2. 扇出（复制）
-
- a1配置：a1_taildir_mem_avro.conf
- a2配置：a2_avro_mem_hdfs.conf
- a3配置：a3_avro_mem_file_roll.conf
1. 扇入、扇出
扇入、扇出就是利用多个flume实例进行数据的采集。扇入就是从多个数据源采集数据然后都发送到一个flume里面，像扇子一样进行不同数据源间数据的整合。扇出就是用一个flume将数据同时发往多个flume，将数据落到不同的地方。
这个扇入扇出我也是面试的时候被整不会了，面试官问我了解flume扇入扇出不？我一脸懵逼，听都没听过。然后查了下原来就是多个flume的使用，也叫聚合和复制…
2. 示例
1. 扇入（聚合）
以上图中的组件进行配置，演示扇入。
a1配置：a1_netcat_mem_avro.conf
a1.sources = r1
a1.channels = c1
a1.sinks = k1
a1.sources.r1.type = netcat
a1.sources.r1.bind = hadoop101
a1.sources.r1.port = 6666
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = hadoop103
a1.sinks.k1.port = 4545
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a2配置：a2_taildir_mem_avro.conf
a2.sources = r1
a2.channels = c1
a2.sinks = k1
a2.sources.r1.type = TAILDIR
a2.sources.r1.positionFile = /opt/module/flume-1.9.0/positionFile/taildir_position.json
a2.sources.r1.filegroups = f1
a2.sources.r1.filegroups.f1 = /opt/module/flume-1.9.0/app.*
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100
a2.sinks.k1.type = avro
a2.sinks.k1.hostname = hadoop103
a2.sinks.k1.port = 4545
a2.sources.r1.channels = c1
a2.sinks.k1.channel = c1
a3配置：a3_avro_mem_logger.conf
a3.sources = r1
a3.channels = c1
a3.sinks = k1
a3.sources.r1.type = avro
a3.sources.r1.bind = hadoop103
a3.sources.r1.port = 4545
a3.channels.c1.type = memory
a3.channels.c1.capacity = 1000
a3.channels.c1.transactionCapacity = 100
a3.sinks.k1.type = logger
a3.sources.r1.channels = c1
a3.sinks.k1.channel = c1
分别启动a3、a2、a1
bin/flume-ng agent -n a3 -c conf -f job/Into/a3_avro_mem_logger.conf -Dflume.root.logger=INFO,console
bin/flume-ng agent -n a2 -c conf -f job/Into/a2_taildir_mem_avro.conf -Dflume.root.logger=INFO,console
bin/flume-ng agent -n a1 -c conf -f job/Into/a1_netcat_mem_avro.conf -Dflume.root.logger=INFO,console
然后通过netcat向6666端口发数据观察到a3有日志输出，采集到了数据；向taildir监控的目录下app.log中追加内容，同样也观察到了a3有日志输出，也采集到了数据。
2. 扇出（复制）
以上图中的组件进行配置，演示扇出。
a1配置：a1_taildir_mem_avro.conf
a1.sources = r1
a1.channels = c1 c2
a1.sinks = k1 k2
a1.sources.r1.type = TAILDIR
a1.sources.r1.positionFile = /opt/module/flume-1.9.0/positionFile/taildir_position.json
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /opt/module/flume-1.9.0/app.*
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
a1.channels.c2.type = memory
a1.channels.c2.capacity = 1000
a1.channels.c2.transactionCapacity = 100
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = hadoop101
a1.sinks.k1.port = 4545
a1.sinks.k2.type = avro
a1.sinks.k2.hostname = hadoop101
a1.sinks.k2.port = 4546
a1.sources.r1.channels = c1 c2
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2
a2配置：a2_avro_mem_hdfs.conf
a2.sources = r1
a2.channels = c1
a2.sinks = k1
a2.sources.r1.type = avro
a2.sources.r1.bind = hadoop101
a2.sources.r1.port = 4545
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100
a2.sinks.k1.type = hdfs
a2.sinks.k1.hdfs.path = hdfs://hadoop101:8020/tmp/flume/applog/%Y-%m-%d/%H%M/%S
a2.sinks.k1.hdfs.filePrefix = applog-
# 是否按照时间滚动文件夹
a2.sinks.k1.hdfs.round = true
# 多少时间建立一个新的文件夹
a2.sinks.k1.hdfs.roundValue = 10
# 时间的单位
a2.sinks.k1.hdfs.roundUnit = second
a2.sinks.k1.hdfs.rollInterval = 60
a2.sinks.k1.hdfs.rollSize = 134217700
a2.sinks.k1.hdfs.rollCount = 0
# 是否使用本地时间戳 默认false使用header里面的时间戳
a2.sinks.k1.hdfs.useLocalTimeStamp = true
# 积攒多少个Event才flush到HDFS一次
a2.sinks.k1.hdfs.batchSize = 100
# 设置文件类型，可支持压缩
a2.sinks.k1.hdfs.fileType = CompressedStream
a2.sinks.k1.hdfs.codeC = gzip
a2.sources.r1.channels = c1
a2.sinks.k1.channel = c1
a3配置：a3_avro_mem_file_roll.conf
a3.sources = r1
a3.channels = c1
a3.sinks = k1
a3.sources.r1.type = avro
a3.sources.r1.bind = hadoop101
a3.sources.r1.port = 4546
a3.channels.c1.type = memory
a3.channels.c1.capacity = 1000
a3.channels.c1.transactionCapacity = 100
a3.sinks.k1.type = file_roll
a3.sinks.k1.sink.directory = /home/atguigu/flume_local_data
a3.sinks.k1.sink.rollInterval = 3600
a3.sources.r1.channels = c1
a3.sinks.k1.channel = c1
分别启动a3、a2、a1
bin/flume-ng agent -n a3 -c conf -f job/a3_avro_mem_file_roll.conf -Dflume.root.logger=INFO,console
bin/flume-ng agent -n a2 -c conf -f job/a2_avro_mem_hdfs.conf -Dflume.root.logger=INFO,console
bin/flume-ng agent -n a1 -c conf -f job/a1_taildir_mem_avro.conf -Dflume.root.logger=INFO,console
向文件中追加内容：
[atguigu@hadoop101 flume-1.9.0]$ echo $(date) >> app.log1
[atguigu@hadoop101 flume-1.9.0]$ echo $(date) >> app.log1
观察到控制台打印出的新的日志，说明已经采集到了，再去hdfs和本地路径中查看，出现了相应的文件。
