Flume系列:案例-Flume 聚合拓扑(常见的日志收集结构)

目录

Apache Hadoop生态-目录汇总-持续更新

 1:案例需求-实现聚合拓扑结构

3:实现步骤:

2.1:实现flume1.conf - sink端口4141

2.2:实现flume2.conf- sink端口4141

2.3:实现flume3.conf - 监听端口4141

3:启动传输链路


Apache Hadoop生态-目录汇总-持续更新

系统环境:centos7

Java环境:Java8

案例只演示通道流程,其中Source,channel,Sink的种类按需调整

Flume系列:案例-Flume 聚合拓扑(常见的日志收集结构)_第1张图片

 1:案例需求-实现聚合拓扑结构

Flume-1 监控文件/tmp/group3/log1.log

Flume-2 监控文件/tmp/group3/log2.log

Flume-1 与 Flume-2 将数据发送给Flume-3,Flume-3 将最终数据打印到控制台。

Flume系列:案例-Flume 聚合拓扑(常见的日志收集结构)_第2张图片

3:实现步骤:

2.1:实现flume1.conf - sink端口4141

vim flume1.conf

# 1:定义组件
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# 2:定义source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /tmp/group3/log1.log
a1.sources.r1.shell = /bin/bash -c

# 3:定义channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# 4:定义sink
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = worker215
a1.sinks.k1.port = 4111

# 5:定义关联关系
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

2.2:实现flume2.conf- sink端口4141

vim flume2.conf

# 1:定义组件
a2.sources = r1
a2.sinks = k1
a2.channels = c1

# 2:定义source
a2.sources.r1.type = exec
a2.sources.r1.command = tail -F /tmp/group3/log2.log
a2.sources.r1.shell = /bin/bash -c

# 3:定义channel
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100

# 4:定义sink
a2.sinks.k1.type = avro
a2.sinks.k1.hostname = worker215
a2.sinks.k1.port = 4111

# 5:定义关联关系
a2.sources.r1.channels = c1
a2.sinks.k1.channel = c1

2.3:实现flume3.conf - 监听端口4141

vim flume3.conf

# 1:定义组件
a3.sources = r1
a3.sinks = k1
a3.channels = c1

# 2:定义source
a3.sources.r1.type = avro
a3.sources.r1.bind = worker215
a3.sources.r1.port = 4111

# 3:定义channel
a3.channels.c1.type = memory
a3.channels.c1.capacity = 1000
a3.channels.c1.transactionCapacity = 100

# 4:定义sink
a3.sinks.k1.type = logger

# 5:定义关联关系
a3.sources.r1.channels = c1
a3.sinks.k1.channel = c1

3:启动传输链路

1)启动flume3,监听端口4141-收集其他服务传输的数据
flume-ng agent --name flume3 --conf-file flume3.conf  -Dflume.root.logger=INFO,console

2)启动flume1,sink端口4141
flume-ng agent --name flume1 --conf-file flume1.conf

3)启动flume2,sink端口4141
flume-ng agent --name flume2 --conf-file flume2.conf

 Flume系列

        Apache Hadoop生态部署-Flume采集节点安装

        Flume系列:Flume组件架构

        Flume系列:Flume Source使用

        Flume系列:Flume Channel使用

        Flume系列:Flume Sink使用

        Flume系列:Flume 自定义Interceptor拦截器

        Flume系列:Flume通道拓扑结构

        Flume系列:Flume常用采集通道案例

        Flume系列:案例-Flume复制(Replicating)和多路复用(Multiplexing)

        Flume系列:案例-Flume负载均衡和故障转移

        Flume系列:案例-Flume 聚合拓扑(常见的日志收集结构)

        Flume系列:Flume数据监控Ganglia

你可能感兴趣的:(Hadoop生态,#,Flume,flume,大数据,hadoop)