Hadoop笔记之十一——Flume安装及简单实例

===================安装====================================
1、解压安装
2、配置flume-env.sh（JAVA_HOME）
3、完事儿

===================简单使用================================
Flume常用参数
--conf,-c          use configs in directory
--name,-n          the name of this agent (required)
--conf-file,-f     specify a config file (required if -z missing)
例子
    监听某个端口，实时收集端口的数据（telnet）

$ bin/flume-ng  agent -c conf/ -f conf/flume-telnet.properties -n a1 -Dflume.root.logger=info,console

============实例===============================
实时获取hive的日志，放到我们的hdfs上面

1、准备好源数据，我们的hive日志文件目录

2、flume要使用到hadoop的jar包，将四个jar包考到flume的lib目录下
$ cp hdfs/hadoop-hdfs-2.5.0-cdh5.3.6.jar /opt/modules/flume-1.5.0-cdh5.3.6-bin/lib/
$ cp common/hadoop-common-2.5.0-cdh5.3.6.jar /opt/modules/flume-1.5.0-cdh5.3.6-bin/lib/
$ cp tools/lib/commons-configuration-1.6.jar /opt/modules/flume-1.5.0-cdh5.3.6-bin/lib/
$ cp tools/lib/hadoop-auth-2.5.0-cdh5.3.6.jar /opt/modules/flume-1.5.0-cdh5.3.6-bin/lib/

3、配置了一个flume-hivelog.properties文件
a2.sources = r2
a2.sinks = k2
a2.channels = c2

# Describe/configure the source
a2.sources.r2.type = exec
a2.sources.r2.command = tail -F /opt/modules/hive-0.13.1-cdh5.3.6/logs/hive.log
a2.sources.r2.shell = /bin/sh -c

# Describe the sink
a2.sinks.k2.type = hdfs
a2.sinks.k2.hdfs.path = hdfs://192.168.17.128:8020/flume/%Y%m%d/%H
a2.sinks.k2.hdfs.filePrefix = hivelog
a2.sinks.k2.hdfs.fileType = DataStream
a2.sinks.k2.hdfs.writeFormat = Text
a2.sinks.k2.hdfs.batchSize = 100 

# 按照每天每个小时生成一个文件夹来保存对于的数据
a2.sinks.k2.hdfs.round = true
a2.sinks.k2.hdfs.roundValue = 1
a2.sinks.k2.hdfs.roundUnit = hour

# 设置文件回滚条件
a2.sinks.k2.hdfs.rollInterval = 120
a2.sinks.k2.hdfs.rollSize = 128000000
a2.sinks.k2.hdfs.rollCount = 0

# 必须要设置为1，否则上面几条配置不生效 
a2.sinks.k2.hdfs.minBlockReplicas = 1

# 允许使用本地的时间
a2.sinks.k2.hdfs.useLocalTimeStamp = true 

# Use a channel which buffers events in memory
a2.channels.c2.type = memory
a2.channels.c2.capacity = 1000
a2.channels.c2.transactionCapacity = 100

# Bind the source and sink to the channel
a2.sources.r2.channels = c2
a2.sinks.k2.channel = c2

4、运行测试
$ bin/flume-ng  agent -c conf/ -f conf/flume-hivelog.properties -n a2