hadoop学习过程中，用到安装flume组件，过程记录如下：


  1. 安装flume服务，yum -y install flume-ng*， service flume-ng-agent start

  2. 查看flume安装位置 whereis flume-ng

  3. 在flume安装位置的conf文件夹下，新建配置文件 testflume.conf  


tail1.sources = src1

tail1.channels = ch1

tail1.sinks = sink1


tail1.sources.src1.type = exec

tail1.sources.src1.command = tail -F /var/log/httpd/access_log

tail1.sources.src1.channels = ch1


tail1.channels.ch1.type = memory

tail1.channels.ch1.capacity = 500


tail1.sinks.sink1.type = avro

tail1.sinks.sink1.hostname = agent

tail1.sinks.sink1.port = 6000

tail1.sinks.sink1.batch-size = 1

tail1.sinks.sink1.channel = ch1


##

agent1.sources = r1 #seqGenSrc

agent1.channels = c1 #memoryChannel

agent1.sinks = k1 #loggerSink


agent1.sources.r1.type = avro

agent1.sources.r1.bind = agent

agent1.sources.r1.port = 6000

agent1.sources.r1.channels = c1


agent1.sinks.k1.type = hdfs

agent1.sinks.k1.hdfs.path = flume/testflume

agent1.sinks.k1.hdfs.filePrefix = access_log

agent1.sinks.k1.channel = c1


agent1.channels.c1.type = memory

agent1.channels.c1.capacity = 500


  4. 依次执行  flume-ng agent -c /etc/flume-ng/conf -f /etc/flume-ng/conf/testflume.conf -n agent1 -Dflume.root.logger=INFO,console

           flume-ng agent -c /etc/flume-ng/conf -f /etc/flume-ng/conf/testflume.conf -n tail1 -Dflume.root.logger=INFO,console

  5. 查看 hadoop fs -ls /user/guowen/flume/testflume，看到有抓取的文件生成