hadoop是一个分布式系统,跟hadoop配合的一般也是分布式系统,分布式系统带来的就是分布式日志,分布式日志带来1. 日志数量多 2. 日志数据量大, 所以无论是采集分布式的日志还是存储海量的日志到hadoop,都需要一个日志收集系统,这就是flume。不过其实关系也不是太大,日志方面没有太大需求的人其实可以跳过flume的学习
或者是这样的
$ yum install -y flume-ng flume-ng-agent flume-ng-doc
$ flume-ng help
$ vim /etc/flume-ng/conf/example.conf
# example.conf: A single-node Flume configuration
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
$ flume-ng agent --conf conf --conf-file /etc/flume-ng/conf/example.conf --name a1 -Dflume.root.logger=INFO,console
$ curl -X GET http://localhost:44444?w=helloworld
OK
OK
OK
OK
OK
14/07/04 11:39:12 INFO sink.LoggerSink: Event: { headers:{} body: 47 45 54 20 2F 3F 77 3D 68 65 6C 6C 6F 77 6F 72 GET /?w=hellowor }
14/07/04 11:39:12 INFO sink.LoggerSink: Event: { headers:{} body: 55 73 65 72 2D 41 67 65 6E 74 3A 20 63 75 72 6C User-Agent: curl }
14/07/04 11:39:12 INFO sink.LoggerSink: Event: { headers:{} body: 48 6F 73 74 3A 20 6C 6F 63 61 6C 68 6F 73 74 3A Host: localhost: }
14/07/04 11:39:12 INFO sink.LoggerSink: Event: { headers:{} body: 41 63 63 65 70 74 3A 20 2A 2F 2A 0D Accept: */*. }
14/07/04 11:39:12 INFO sink.LoggerSink: Event: { headers:{} body: 0D
以上是flume的最简单的例子,接下来介绍一个稍微复杂一点,但是真正实用的例子
# source, channel, sink definition
agent.channels = mem-channel
agent.sources = log4j-avro-source
agent.sinks = hdfs-sink
# Channel
# Define a memory channel called mem-channel on agent
agent.channels.mem-channel.type = memory
# Source
# Define an Avro source called log4j-avro-channel on agent and tell it
# to bind to host1:12345. Connect it to channel mem-channel.
agent.sources.log4j-avro-source.type = avro
agent.sources.log4j-avro-source.bind = 192.168.1.126
agent.sources.log4j-avro-source.port = 12345
agent.sources.log4j-avro-source.channels = mem-channel
# Sink
# Define a logger sink that simply logs all events it receives
# and connect it to the other end of the same channel.
agent.sinks.hdfs-sink.type = hdfs
agent.sinks.hdfs-sink.hdfs.path = hdfs://mycluster/flume/events/
agent.sinks.hdfs-sink.channel = mem-channel
service flume-ng-agent restart
sudo -u flume hdfs dfs -mkdir -p /flume/events/
4.0.0
org.crazycake
play-flume
0.0.1-SNAPSHOT
jar
play-flume
http://maven.apache.org
UTF-8
log4j
log4j
1.2.17
org.apache.flume.flume-ng-clients
flume-ng-log4jappender
1.4.0
junit
junit
4.11
${project.artifactId}-${timestamp}
org.apache.maven.plugins
maven-compiler-plugin
2.5.1
true
1.6
org.apache.maven.plugins
maven-jar-plugin
2.4
false
true
# Define the root logger with appender file
log4j.rootLogger = DEBUG, stdout, flume
# Redirect log messages to console
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
# Define the flume appender
log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Hostname = host1
log4j.appender.flume.Port = 12345
log4j.appender.flume.UnsafeMode = false
log4j.appender.flume.layout=org.apache.log4j.PatternLayout
package org.crazycake.play_flume;
import org.apache.log4j.Logger;
public class FlumeLog {
public static void main(String[] args)
{
Logger logger = Logger.getLogger(App.class);
logger.info("hello world");
logger.info("My name is alex");
logger.info("How are you?");
}
}