flume ng 1.3 安装配置方法

业务系统需要收集监控系统日志，想到了hadoop的flume。经过试验，虽说功能不算足够强大，但基本上能够满足功能需求。Flume 是一个分布式、可靠和高可用的服务日志收集工具，能够和hadoop,hive等配置完成日志收集，存储，分析处理等工作，更详细的介绍可以参见apache网站。下面介绍下简单的安装配置方法

1,网上下载flume-ng安装包，分别部署在收集和接收日志文件的服务器上，服务器上需安装jdk 1.6以上，
http://flume.apache.org/download.html
tar -zxvf apache-flume-1.3.0-bin.tar.gz
2, 日志文件接收端端新建conf/flume-conf.properties server端的具体配置如下

从avro source端接收数据，然后写入到HDFS文件系统中

[flume@ conf]$ cat  flume-conf.properties
agent.sources = avrosrc
agent.channels = memoryChanne3
agent.sinks = hdfsSink

# For each one of the sources, the type is defined
agent.sources.avrosrc.type = avro
agent.sources.avrosrc.bind = 172.16.251.1
agent.sources.avrosrc.port = 44444

# The channel can be defined as follows.
agent.sources.avrosrc.channels = memoryChanne3

# Each channel's type is defined.
agent.channels.memoryChanne3.type = memory
agent.channels.memoryChanne3.keep-alive = 10
agent.channels.memoryChanne3.capacity = 100000
agent.channels.memoryChanne3.transactionCapacity =100000

# Each sink's type must be defined
agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.channel = memoryChanne3
agent.sinks.hdfsSink.hdfs.path = /logdata/%{hostname}_linux/%Y%m%d_date
agent.sinks.hdfsSink.hdfs.filePrefix = %{datacenter}_
agent.sinks.hdfsSink.hdfs.rollInterval = 0
agent.sinks.hdfsSink.hdfs.rollSize = 4000000
agent.sinks.hdfsSink.hdfs.rollCount = 0
agent.sinks.hdfsSink.hdfs.writeFormat = Text
agent.sinks.hdfsSink.hdfs.fileType = DataStream
agent.sinks.hdfsSink.hdfs.batchSize = 10

[flume@ conf]$ cat  flume-conf.properties
agent.sources = avrosrc
agent.channels = memoryChanne3
agent.sinks = hdfsSink

# For each one of the sources, the type is defined
agent.sources.avrosrc.type = avro
agent.sources.avrosrc.bind = 172.16.251.1
agent.sources.avrosrc.port = 44444

# The channel can be defined as follows.
agent.sources.avrosrc.channels = memoryChanne3

# Each channel's type is defined.
agent.channels.memoryChanne3.type = memory
agent.channels.memoryChanne3.keep-alive = 10
agent.channels.memoryChanne3.capacity = 100000
agent.channels.memoryChanne3.transactionCapacity =100000

# Each sink's type must be defined
agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.channel = memoryChanne3
agent.sinks.hdfsSink.hdfs.path = /logdata/%{hostname}_linux/%Y%m%d_date
agent.sinks.hdfsSink.hdfs.filePrefix = %{datacenter}_
agent.sinks.hdfsSink.hdfs.rollInterval = 0
agent.sinks.hdfsSink.hdfs.rollSize = 4000000
agent.sinks.hdfsSink.hdfs.rollCount = 0
agent.sinks.hdfsSink.hdfs.writeFormat = Text
agent.sinks.hdfsSink.hdfs.fileType = DataStream
agent.sinks.hdfsSink.hdfs.batchSize = 10

如果flume和hadoop不是同一用户，需要注意相关权限问题

3，日志收集端的conf/flume-conf.properties server文件配置，这里收集二个日志文件到收集端

agent.sources = tailsource-1 tailsource-2
agent.channels = memoryChannel-1 memoryChannel-2
agent.sinks = remotesink remotesink-2

agent.sources.tailsource-1.type = exec
agent.sources.tailsource-1.command = tail -F /tmp/linux2.log
agent.sources.tailsource-1.channels = memoryChannel-1

agent.sources.tailsource-2.type = exec
agent.sources.tailsource-2.command = tail -F /tmp/linux2_2.log
agent.sources.tailsource-2.channels = memoryChannel-2

agent.sources.tailsource-1.interceptors = host_int timestamp_int inter1
agent.sources.tailsource-1.interceptors.host_int.type = host
agent.sources.tailsource-1.interceptors.host_int.hostHeader = hostname

agent.sources.tailsource-1.interceptors.timestamp_int.type = org.apache.flume.interceptor.TimestampInterceptor$Builder

#agent.sources.tailsource-1.interceptors = inter1
agent.sources.tailsource-1.interceptors.inter1.type = static
agent.sources.tailsource-1.interceptors.inter1.key = datacenter
agent.sources.tailsource-1.interceptors.inter1.value = BEIJING

agent.sources.tailsource-2.interceptors = host_int timestamp_int inter1
agent.sources.tailsource-2.interceptors.host_int.type = host
agent.sources.tailsource-2.interceptors.host_int.hostHeader = hostname

agent.sources.tailsource-2.interceptors.timestamp_int.type = org.apache.flume.interceptor.TimestampInterceptor$Builder

#agent.sources.tailsource-1.interceptors = inter1
agent.sources.tailsource-2.interceptors.inter1.type = static
agent.sources.tailsource-2.interceptors.inter1.key = datacenter
agent.sources.tailsource-2.interceptors.inter1.value = linux2_2

agent.channels.memoryChannel-1.type = memory
agent.channels.memoryChannel-1.keep-alive = 10
agent.channels.memoryChannel-1.capacity = 100000
agent.channels.memoryChannel-1.transactionCapacity =100000

agent.channels.memoryChannel-2.type = memory
agent.channels.memoryChannel-2.keep-alive = 10
agent.channels.memoryChannel-2.capacity = 100000
agent.channels.memoryChannel-2.transactionCapacity =100000

agent.sinks.remotesink.type = avro
agent.sinks.remotesink.hostname = 172.16.251.1
agent.sinks.remotesink.port = 44444
agent.sinks.remotesink.channel = memoryChannel-1

agent.sinks.remotesink-2.type = avro
agent.sinks.remotesink-2.hostname = 172.16.251.1
agent.sinks.remotesink-2.port = 44444
agent.sinks.remotesink-2.channel = memoryChannel-2

agent.sources = tailsource-1 tailsource-2
agent.channels = memoryChannel-1 memoryChannel-2
agent.sinks = remotesink remotesink-2

agent.sources.tailsource-1.type = exec
agent.sources.tailsource-1.command = tail -F /tmp/linux2.log
agent.sources.tailsource-1.channels = memoryChannel-1

agent.sources.tailsource-2.type = exec
agent.sources.tailsource-2.command = tail -F /tmp/linux2_2.log
agent.sources.tailsource-2.channels = memoryChannel-2

agent.sources.tailsource-1.interceptors = host_int timestamp_int inter1
agent.sources.tailsource-1.interceptors.host_int.type = host
agent.sources.tailsource-1.interceptors.host_int.hostHeader = hostname

agent.sources.tailsource-1.interceptors.timestamp_int.type = org.apache.flume.interceptor.TimestampInterceptor$Builder

#agent.sources.tailsource-1.interceptors = inter1
agent.sources.tailsource-1.interceptors.inter1.type = static
agent.sources.tailsource-1.interceptors.inter1.key = datacenter
agent.sources.tailsource-1.interceptors.inter1.value = BEIJING

agent.sources.tailsource-2.interceptors = host_int timestamp_int inter1
agent.sources.tailsource-2.interceptors.host_int.type = host
agent.sources.tailsource-2.interceptors.host_int.hostHeader = hostname

agent.sources.tailsource-2.interceptors.timestamp_int.type = org.apache.flume.interceptor.TimestampInterceptor$Builder

#agent.sources.tailsource-1.interceptors = inter1
agent.sources.tailsource-2.interceptors.inter1.type = static
agent.sources.tailsource-2.interceptors.inter1.key = datacenter
agent.sources.tailsource-2.interceptors.inter1.value = linux2_2

agent.channels.memoryChannel-1.type = memory
agent.channels.memoryChannel-1.keep-alive = 10
agent.channels.memoryChannel-1.capacity = 100000
agent.channels.memoryChannel-1.transactionCapacity =100000

agent.channels.memoryChannel-2.type = memory
agent.channels.memoryChannel-2.keep-alive = 10
agent.channels.memoryChannel-2.capacity = 100000
agent.channels.memoryChannel-2.transactionCapacity =100000

agent.sinks.remotesink.type = avro
agent.sinks.remotesink.hostname = 172.16.251.1
agent.sinks.remotesink.port = 44444
agent.sinks.remotesink.channel = memoryChannel-1

agent.sinks.remotesink-2.type = avro
agent.sinks.remotesink-2.hostname = 172.16.251.1
agent.sinks.remotesink-2.port = 44444
agent.sinks.remotesink-2.channel = memoryChannel-2

4，后台运行
nohup bin/flume-ng agent -n agent -c conf -f conf/flume-conf.properties >1.log &

查看日志vi flume.log
端口连接情况 netstat -an|grep 44444
[flume@dtydb6 flume-1.4]$ netstat -an|grep 44444
tcp        0      0 ::ffff:172.16.251.1:44444   :::*                        LISTEN  

5,测试方法

可以使用如下类似的脚本，定期向日志文件写入来进行测试

for i in {1..1000000}; do echo "LINUX2  PRESS ************* Flume log rotation $i" >> /tmp/linux3.log; sleep 0.0001; done

参考资料：
http://flume.apache.org/FlumeUserGuide.html