flume-ng安装,应用,几个简单测试例子!

官网文档:http://flume.apache.org/FlumeUserGuide.html


Flume是Cloudera提供的一个高可用的,高可靠的,分布式的海量日志采集、聚合和传输的系统,Flume支持在日志系统中定制各类数据发送方,用于收集数据;同时,Flume提供对数据进行简单处理,并写到各种数据接受方(可定制)的能力。
当前Flume有两个版本Flume 0.9X版本的统称Flume-og,Flume1.X版本的统称Flume-ng。由于Flume-ng经过重大重构,与Flume-og有很大不同,使用时请注意区分。
日志收集
Flume最早是Cloudera提供的日志收集系统,目前是Apache下的一个孵化项目,Flume支持在日志系统中定制各类数据发送方,用于收集数据。

数据处理

Flume提供对数据进行简单处理,并写到各种数据接受方(可定制)的能力 Flume提供了从console(控制台)、RPC(Thrift-RPC)、text(文件)、tail(UNIX tail)、syslog(syslog日志系统,支持TCP和UDP等2种模式),exec(命令执行)等数据源上收集数据的能力。


省去长篇介绍........



下载安装:
wget http://mirror.bit.edu.cn/apache/flume/1.7.0/apache-flume-1.7.0-bin.tar.gz
tar -zxvf apache-flume-1.7.0-bin.tar.gz

测试例子:
cd conf/
touch example.conf
vim  example.conf

# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1


# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444


# Describe the sink
a1.sinks.k1.type = logger


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1


启动:
$ bin/flume-ng agent --conf conf --conf-file conf/example.conf --name a1 -Dflume.root.logger=INFO,console


 
yum install telnet


telnet localhost 44444


helloworld 测试(a->b传送数据)


安装avro (cmark)
wget https://cmake.org/files/v3.8/cmake-3.8.1.tar.gz

1: wget http://mirrors.hust.edu.cn/apache/avro/avro-1.7.7/c/avro-c-1.7.7.tar.gz
2:tar -zxvf avro-src-1.7.7.tar.gz
3:cd avro-c-1.7.7
4:mkdir build
5:cd build
6:cmake .. -DCMAKE_INSTALL_PREFIX=$PREFIX -DCMAKE_BUILD_TYPE=RelWithDebinfo
7:make
8:make install


关闭防火墙
service iptables stop

-------------测试 HTTP Source--------------------------
conf:httppost.conf


# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1


# Describe/configure the source
a1.sources.r1.type = org.apache.flume.source.http.HTTPSource
a1.sources.r1.port = 5140
a1.sources.r1.channels = c1


# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


启动flume agent al
bin/flume-ng agent --conf conf --conf-file conf/httppost.conf --name a1 -Dflume.root.logger=INFO,console


发送一个请求
curl -XPOST  -d '[{"headers" : {"timestamp" : "434324343","host" : "random_host.example.com"},"body" : "random_body"},{"headers" : {"namenode" : "namenode.example.com","datanode" : "random_datanode.example.com"},"body" : "really_random_body" }]' http://localhost:5140


结果:控制台会打印出来   OK-----


-------------测试 avro Source--------------------------
conf:avrotest.conf


# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1


# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
a1.sources.r1.channels = c1


# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


启动flume agent al
bin/flume-ng agent --conf conf --conf-file conf/avrotest.conf --name a1 -Dflume.root.logger=INFO,console




echo "hello world" > test.log


发送文件到flume
bin/flume-ng avro-client -c conf -H 192.168.254.129 -p 44444 -F /home/rui/log/test.log


-------------测试 Spooling Directory Source--------------------------
conf:directorytest.conf


# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1


# Describe/configure the source
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /home/rui/log/flumespool
a1.sources.r1.fileHeader = true
a1.sources.r1.channels = c1


# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


启动flume agent al
bin/flume-ng agent --conf conf --conf-file conf/directorytest.conf --name a1 -Dflume.root.logger=INFO,console


测试:
cd /home/rui/log/flumespool
echo "hello world" > test.log

-------------测试 avro sink-----------a1服务--send-->a2服---------------
接收端:touch receive.conf
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1


# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44445
a1.sources.r1.channels = c1


# Describe the sink
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


启动flume agent al
bin/flume-ng agent --conf conf --conf-file conf/receive.conf --name a1 -Dflume.root.logger=INFO,console


发送端:send.conf

# Name the components on this agent
a2.sources = r1
a2.sinks = k1
a2.channels = c1


# Describe/configure the source
#或者目录监听方式
a2.sources.r1.type = syslogtcp
a2.sources.r1.bind = localhost
a2.sources.r1.port = 44446
a2.sources.r1.channels = c1


# Describe the sink !!!!!!!!!!!!!!!!!!!!The point is here
a2.sinks.k1.type = avro  
a2.sinks.k1.channel = c1
a2.sinks.k1.hostname = 127.0.0.1
a2.sinks.k1.port = 44445


# Use a channel which buffers events in memory
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100


启动flume agent al
bin/flume-ng agent --conf conf --conf-file conf/send.conf --name a2 -Dflume.root.logger=INFO,console


tcp方式测试:
echo "<37>hello via avro sink" | nc localhost 44446


如果没有nc 就 yum install nc


44445端口控制台打印现来:<37>hello via avro sink, 就成功了!

你可能感兴趣的:(大数据)