Flume-ng集群安装文档

Flume-ng集群安装文档

（apache-flume-1.6.0-bin）

一、环境搭建

1. 下载并解压安装包

tar  xvf  apache-flume-1.6.0-bin.tar.gz

2. 配置环境变量

root用户：#vi /etc/profile 或

cdh5用户：$vi~/.bashrc  

以上两个任选一种，增加以下环境变量：

export FLUME_HOME=/app/data/flume/apache-flume-1.6.0-bin

export FLUME_CONF_DIR=$FLUME_HOME/conf

export PATH=$PATH:$FLUME_HOME/bin

生效修改：#source /etc/profile  或 $source~/.bashrc

3. 修改配置文件

flume-env.sh

cp  flume-env.sh.template  flume-env.sh   （修改文件名）

配置

export JAVA_HOME=/usr/jdk1.7.0_80

export JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote"

flume.conf

cp flume-conf.properties.template  flume.conf      （修改文件名）

配置

# The configuration file needs to define the sources,

# the channels and the sinks.

# Sources, channels and sinks are defined per agent,

# in this case called 'agent'

 

 

a1.sources = r1

a1.sinks = k1

a1.channels = c1

 

# For each one of the sources, the type is defined

a1.sources.r1.type = exec

a1.sources.r1.bind = localhost

a1.sources.r1.port = 5858

a1.sources.r1.shell = /bin/bash -c

a1.sources.r1.command = tail -F /tomcat_app/nohup.out

a1.sources.r1.restartThrottle = 300000

a1.sources.r1.restart = true

a1.sources.r1.logStdErr = false

a1.soources.r1.batchSize = 5

 

# The channel can be defined as follows.

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

a1.channels.c1.keep-alive = 30

a1.channels.c1.byteCapacityBufferPercentage = 80%

a1.channels.c1.byteCapacity = 1024000000

 

# Each sink's type must be defined

a1.sinks.k1.type = logger

 

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

启动

启动命令：

nohup bin/flume-ng agent --conf conf --conf-file conf/flume.conf --name a1 -Dflume.root.logger=INFO,console &        （nohup命令表示不打印启动日志）

如要查看启动日志，则使用启动命令：

bin/flume-ng agent --conf conf --conf-file conf/flume.conf --name a1 -Dflume.root.logger=INFO,console &       

验证是否启动成功

bin/flume-ng version

显示如下即启动成功

Flume 1.6.0

Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git

Revision: 2561a23240a71ba20bf288c7c2cda88f443c2080

Compiled by hshreedharan on Mon May 11 11:15:44 PDT 2015

From source with checksum b29e416802ce9ece3269d34233baf43f

其他机器同样执行以上步骤安装启动flume即可