flume-ng 1.4 安装配置

记录一下flume-ng 的安装配置



三个基本概念


source：数据源

channel：数据传输通道

sink：数据目的地，如存储在hdfs

配置：


flume.conf



#define

agent1.sources = source1

agent1.channels = channel1

agent1.sinks = sink1

 

#

agent1.sources.source1.type = spooldir

agent1.sources.source1.spoolDir = /tmp/test/

agent1.sources.source1.deletePolicy = immediate

 

#Describe the sink

agent1.sinks.sink1.type = logger

 

# Use a channel which buffers events in memory

agent1.channels.channel1.type = memory

agent1.channels.channel1.capacity = 1000

agent1.channels.channel1.transactionCapacity = 100

 

#Bind the source and sink to the channel

agent1.sources.source1.channels = channel1

agent1.sinks.sink1.channel = channel1

##sink配置


sink配置为logger类型，存储在日志文件中，测试无问题后再改为存储在hdfs中。


##源配置


将源的类型设置为spooldir用于监视目录下的文件


指定spooDir为tmp下的test文件夹


删除策略设置为及时删除，默认是不删除。


 


启动

flume-ng agent --conf /usr/local/flume/conf/ --conf-file flume.conf --name agent1 -Dflume.root.logger=INFO,console


 测试

cd /tmp/test/

echo 'good day' > 123

 结果


2014-06-22 10:03:39,540 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.deleteCurrentFile(ReliableSpoolingFileEventReader.java:366)] Preparing to delete file /tmp/test/123

2014-06-22 10:03:40,934 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{file=/tmp/test/123} body: 67 6F 6F 64 20 64 61 79                         good day }