[置顶] Flume-ng生产环境实践（一）Flume-ng生产环境编译
1.首先在系统上安装jdk1.6+和mvn3.X：
|
$ java -version
java version "1.6.0_31"
Java(TM) SE Runtime Environment (build 1.6.0_31-b04-415-11M3635) |
2.下载source code：
| $ svn co https://svn.apache.org/viewvc/flume/trunk/ flume |
$ cd flume
$ mvn clean install -DskipTests如果不跳过测试的话，会出现编译验证失败，由于测试过程中，在flume-ng-core项目下生成event.txt测试文件。需要删除才可以编译通过。详情了解地址：
https://issues.apache.org/jira/browse/FLUME-1372
$ cp flume-ng-dist/target/flume-ng-dist-1.3.0-SNAPSHOT-dist.tar.gz . $ tar -zxvf apache-flume-1.3.0-SNAPSHOT-dist.tar.gz $ cd flume-1.3.0-SNAPSHOT
$ cp conf/flume-conf.properties.template conf/flume.conf
$ cp conf/flume-env.sh.template conf/flume-env.sh通过配置即可以开始启动flume-ng这个分布式日志收集系统了。
编译过程可能发现内存溢出问题。
windows下修改mvn.bat
增加这一行即可：
set MAVEN_OPTS= -Xms512m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=512m
linux下修改/etc/profile
编译过程发现
flume-ng-core下scripts/saveVersion.sh无权限执行的时候，给这个脚本赋权就好了
chmod 755 scripts/saveVersion.sh
编译过程可能发生
Failed to execute goal org.apache.rat:apache-rat-plugin:0.7:check (test.
rat) on project flume-ng-core: Too many unapproved licenses: 1 -> [Help 1]
删除flume-ng-core的\src\test\resources\event.txt即可4.线上测试：
a.单机测试
flume-ng启动agent的命令为：
b.flume-ng 测试过程中event丢失部分body数据
$ bin/flume-ng agent -n $agent_name -c conf -f conf/flume-conf.properties.template
添加如下配置文件
# Name the components on this agentagent1.sources = source1agent1.sinks = sink1agent1.channels = channel1
# Describe/configure source1agent1.sources.source1.type = netcatagent1.sources.source1.bind = localhostagent1.sources.source1.port = 44444
# Describe sink1agent1.sinks.sink1.type = logger
# Use a channel which buffers events in memoryagent1.channels.channel1.type = memoryagent1.channels.channel1.capacity = 1000agent1.channels.channel1.transactionCapactiy = 100
# Bind the source and sink to the channelagent1.sources.source1.channels = channel1agent1.sinks.sink1.channel = channel1以上配置信息描述的是：该配置文件中 配置了一个代理agent1在代理agent中配置了一个source（源）一个sink（接收器）和一个channel（通道），分别为：source1，sink1，channel1source1的类型定义为netcat，对应该类型参数为bind和port 分别为localhost和44444sink1的类型定义为logger，直接输出到日志文件中channel的类型定义为内存方式，设置其参数capactiy和transactionCapacity分别为1000和100指定source1和sink1的channel为channel1
现在我们可以启动看看我们的flume-ng是否成功。在flume-ng的home目录下
$ bin/flume-ng agent --conf-file example.conf --name agent1 -Dflume.root.logger=INFO,console --conf = conf
启动之后日志信息如下：
2012-07-30 15:42:40,780 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 68 61 68 61 0D haha. } 说明我们的flume-ng运行良好。可以继续下一步操作…………
经过测试发现，当source端单event的body数据大于16字节后，输出到目标只剩下16字节。进过多源代码的分析，发现，源代码中进行了截取。
在LoggerSink.java中：
public static String dumpEvent(Event event, int maxBytes) {StringBuilder buffer = new StringBuilder();if (event == null || event.getBody() == null) {buffer.append("null");} else if (event.getBody().length == 0) {// do nothing... in this case, HexDump.dump() will throw an exception} else {byte[] body = event.getBody();byte[] data = Arrays.copyOf(body, Math.min(body.length, maxBytes));ByteArrayOutputStream out = new ByteArrayOutputStream();try {HexDump.dump(data, 0, out, 0);String hexDump = new String(out.toByteArray());// remove offset since it's not relevant for such a small datasetif(hexDump.startsWith(HEXDUMP_OFFSET)) {hexDump = hexDump.substring(HEXDUMP_OFFSET.length());}buffer.append(hexDump);} catch (Exception e) {if(LOGGER.isInfoEnabled()) {LOGGER.info("Exception while dumping event", e);}buffer.append("...Exception while dumping: ").append(e.getMessage());}String result = buffer.toString();if(result.endsWith(EOL) && buffer.length() > EOL.length()) {buffer.delete(buffer.length() - EOL.length(), buffer.length()).toString();}}return "{ headers:" + event.getHeaders() + " body:" + buffer + " }";}不难看出，在event处理过程中，发生了数据截取操作。
c.flume-ng avro2log demo配置详解(多机器)
flume-ng启动agent的命令为：$ bin/flume-ng agent -n $agent_name -c conf -f conf/flume-conf.properties.template添加如下配置文件
# Describe/configure source1agent1.sources.source1.type = avroagent1.sources.source1.bind = 192.168.1.101agent1.sources.source1.port = 44444
# Describe sink1agent1.sinks.sink1.type = logger
# Use a channel which buffers events in memoryagent1.channels.channel1.type = memoryagent1.channels.channel1.capacity = 1000agent1.channels.channel1.transactionCapactiy = 100
# Bind the source and sink to the channelagent1.sources.source1.channels = channel1agent1.sinks.sink1.channel = channel1现在我们可以启动看看我们的flume-ng是否成功。在flume-ng的home目录下$ bin/flume-ng agent --conf-file avro2log.conf --name agent1 -Dflume.root.logger=INFO,console --conf = conf启动之后日志信息如下：