flume学习02-flume安装配置启动

上一篇博客简单介绍了flume的一些基本概念和架构，更详细的内容可以去官网用户手册学习，这篇博客主要介绍如何安装配置flume，以及作个简单示例如何使用

下载flume

• wget http://apache.dataguru.cn/flume/1.5.2/apache-flume-1.5.2-bin.tar.gz
• tar -zxvf apache-flume-1.5.2-bin.tar.gz
• cp conf/flume-env.sh.template conf/flume-env.sh
• cp conf/flume-conf.properties.template simple-agent.properties

启动和停止

• 修改配置文件simple-agent.properties（这里需要根据你自己的业务去修改这个文件，这里举个简单的例子）:

# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#  http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.


# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per agent, 
# in this case called 'agent'

agent.sources = so1
agent.channels = c1
agent.sinks = s1

# For each one of the sources, the type is defined
agent.sources.so1.type = netcat
agent.sources.so1.bind = localhost
agent.sources.so1.port = 44444

# The channel can be defined as follows.
# agent.sources.seqGenSrc.channels = memoryChannel

# Each sink's type must be defined
agent.sinks.s1.type = logger

#Specify the channel the sink should use
# agent.sinks.loggerSink.channel = memoryChannel

# Each channel's type is defined.
agent.channels.c1.type = memory
agent.channels.c1.capacity = 1000
agent.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
agent.sources.so1.channels = c1
agent.sinks.s1.channel = c1


# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
# agent.channels.memoryChannel.capacity = 100

这就是最简单的一个配置，收集到的日志直接在flume的logger中打印出来

• 启动
  bin/flume-ng agent –conf ./conf/ -f conf/simple-agent.properties -Dflume.root.logger=DEBUG,console -n agent
  注意点：
  –conf：指定flume-env.sh的目录
  -f：flume定义组件的配置文件
  -n：启动Agent的名称，该名称在组件配置文件中定义
  -Dflume.root.logger：flume自身运行状态的日志，根据需要配置

• 停止
  直接ctrl+c就好了

• 脚本启动暂停
  非后台运行的flume看上去比较怪，所以我们还是写2个脚本来实现后台启动和停止
  start.sh和stop.sh
  使用
  ./start.sh simple-agent.properties agent
  参数1：指定启动的配置文件
  参数2：指定agent的名字

  ./stop.sh 44444
  参数1：配置文件中指定的启动端口

#!/bin/bash
nohup bin/flume-ng agent --conf ./conf/ -f conf/$1 -Dflume.root.logger=DEBUG,console -n $2 &

#!/bin/bash
kill -9 $(lsof -i:12306 | awk '{print $2}' | tail -n 1)

测试

由于配置中的source是netcat，那么我们直接telnet

xxx@xxx:~$ telnet localhost 44444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
dasdasd
OK

我们再看flume console的输出

2015-02-12 10:39:14,886 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 64 61 73 64 61 73 64 0D                         dasdasd. }

可以看到正常执行了

安装配置启动就简单得介绍到这里，毕竟我学这个是用来收集日志的，所以接下去几篇都会介绍用flume如何去接受log4j的输出日志，有可能涉及到日志的分析处理。