CentOS 7.0安装flume

环境

CentOS 7.0

hadoop 2.7.3 CentOS 7.0+hadoop 2.7搭建集群

flume 1.8.0

下载并安装

下载

下载地址：http://flume.apache.org/download.html

百度网盘：https://pan.baidu.com/s/1dFCSh3N 密码：wnko

也可以通过wget直接下载:

wget http://mirrors.hust.edu.cn/apache/flume/1.8.0/apache-flume-1.8.0-bin.tar.gz

安装

我下载到的目录是/data/software，安装到/opt/flume目录下

mkdir /opt/flume
tar -zxvf apache-flume-1.8.0-bin.tar.gz -C /opt/flume/
cd /opt/flume

将apache-flume-1.8.0-bin改名为apache-flume-1.8.0

mv apache-flume-1.8.0-bin/ apache-flume-1.8.0

配置

添加flume环境变量

vi /etc/profile

添加上

export FLUME_HOME=/opt/flume/apache-flume-1.8.0
export FLUME_CONF_DIR=$FLUME_HOME/conf
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HIVE_HOME/bin:$ZK_HOME/bin:$HBASE_HOME/bin:$FLUME_HOME/bin

修改flume-env.sh

cd /opt/flume/apache-flume-1.8.0/conf
# 拷贝出模板
cp flume-env.sh.template flume-env.sh
# 修改配置文件
vi flume-env.sh

添加上

export JAVA_HOME=/opt/java/jdk1.8.0_60
export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"

修改flume-conf

cp flume-conf.properties.template flume-conf.properties
vi flume-conf.properties

添加agent、source、channel、sink的相关信息

a1.sources=r1
a1.channels=c1
a1.sinks=k1
#Describe/configure the source
a1.sources.r1.type=netcat
a1.sources.r1.bind=192.168.122.128
a1.sources.r1.port=44444
a1.sources.r1.channels=c1
#Describe the sink
a1.sinks.k1.type=logger
a1.sinks.k1.channel=c1
#Describe the channel
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transcationCapacity=100

启动

注：位于flume的安装目录启动

cd /opt/flume/apache-flume-1.8.0
# 启动命令
./bin/flume-ng agent --conf ./conf --conf-file ./conf/flume-conf.properties --name a1 -Dflume.root.logger=INFO,console

输出：

...
...
2017-12-05 15:34:26,837 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:62)] Configuration provider starting
2017-12-05 15:34:26,850 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:134)] Reloading configuration file:./conf/flume-conf.properties
2017-12-05 15:34:26,861 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: k1 Agent: a1
2017-12-05 15:34:26,862 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k1
2017-12-05 15:34:26,862 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k1
2017-12-05 15:34:26,891 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [a1]
2017-12-05 15:34:26,891 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:147)] Creating channels
2017-12-05 15:34:26,904 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:42)] Creating instance of channel c1 type memory
2017-12-05 15:34:26,923 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:201)] Created channel c1
2017-12-05 15:34:26,924 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)] Creating instance of source r1, type netcat
2017-12-05 15:34:26,951 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: k1, type: logger
2017-12-05 15:34:26,957 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:116)] Channel c1 connected to [r1, k1]
2017-12-05 15:34:26,974 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:137)] Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:r1,state:IDLE} }} sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@7abb4f13 counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }
2017-12-05 15:34:26,996 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:144)] Starting Channel c1
2017-12-05 15:34:26,998 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:159)] Waiting for channel: c1 to start. Sleeping for 500 ms
2017-12-05 15:34:27,002 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
2017-12-05 15:34:27,004 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: CHANNEL, name: c1 started
2017-12-05 15:34:27,500 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:171)] Starting Sink k1
2017-12-05 15:34:27,502 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:182)] Starting Source r1
2017-12-05 15:34:27,503 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:155)] Source starting
2017-12-05 15:34:27,508 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:166)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/192.168.122.128:44444]

flume启动成功

说明：启动命令格式

./bin/flume-ng agent --conf <conf路径> --conf-file <flume-conf.properties路径> --name <agent名称> -Dflume.root.logger=INFO,console

注意：-Dflume.root.logger=INFO,console 只有一个短横线

测试

在另外两台机器上192.168.122.129 、192.168.122.130 上telnet开启flume的这台机器

# 192.168.122.129 输入hello world
telnet 192.168.122.128 44444

hello world

# 192.168.122.130 输入hello flume
telnet 192.168.122.128 44444

hello flume

输出：

2017-12-05 15:48:01,684 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64 0D             hello world. }

2017-12-05 15:51:44,665 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 68 65 6C 6C 6F 20 66 6C 75 6D 65 0D             hello flume. }

说明：使用telnet命令，需先安装telnet

yum install -y telnet

补充

由于前期排版规划没排好，只好将类似日志输出到HDFS的配置补到下篇

---

参考：Hadoop入门进阶课程12–Flume介绍、安装与应用案例

flume1.8.0安装配置