用Maven编译Apache flume-ng 1.5.0源码及问题解决

在今年的5月22号，Flume-ng 1.5.0版本正式发布，关于Flume-ng 1.5.0版本的新特性可以参见本博客的《Apache Flume-ng 1.5.0正式发布》进行了解。关于Apache flume-ng 1.4.0版本的编译可以参见本博客《基于Hadoop-2.2.0编译flume-ng 1.4.0及错误解决》。本文将讲述如何用Maven编译Apache flume-ng 1.5.0源码。

一、到官方网站下载相应版本的flume-ng源码

1 # wget http://mirror.bit.edu.cn/apache/flume/1.5.0/apache-flume-1.5.0-src.tar.gz
2 # tar -zxf apache-flume-1.5.0-src.tar.gz
3 # cd apache-flume-1.5.0-src

二、用maven在相应的Hadoop上编译Flume-ng 1.5.0源码

Flume-ng 1.5.0默认的Hadoop版本是1.0.1，Hbase是0.92.1。我们可以在编译的时候加上一些参数，指定Hadoop 2.2.0版本进行编译：

1 -Phadoop-2

三、Hadoop、Hbase对应的test不能下载

　　如果在编译的时候出现了类似如下的情况，请将相应pom.xml文件中的Hadoop-test和Hbase-test依赖包去掉：

01
10
11         org.apache.hbase
12         hbase
13         ${hbase.version}
14         tests
15         test
16
17
18
19          org.apache.hadoop
20          hadoop-test
21          ${hadoop.version}
22

四、elasticsearch-0.90.1.jar read错误

　　如果你在编译的过程中出现一下异常：

1 [ERROR] Failed to execute goal org.apache.maven.plugins:
2 maven-compiler-plugin:2.3.2:compile (default-compile) on
3 project flume-ng-elasticsearch-sink: Compilation failure
4 [ERROR] error: error reading
5 /org/elasticsearch/elasticsearch/0.90.1/elasticsearch-0.90.1.jar;
6 error in opening zip file
7 [ERROR] -> [Help 1]

　　只需要到你maven本地仓库将elasticsearch-0.90.1.jar文件删除，然后重新下载即可：

1 # rm elasticsearch-0.90.1.jar
2 wget http://central.maven.org/maven2/org/elasticsearch/elasticsearch/0.90.1/elasticsearch-0.90.1.jar

五、ua-parser-1.3.0下载失败

编译过程中，下载ua-parser-1.3.0.pom可能失败，出现如下错误：

01 [ERROR] Failed to execute goal on project flume-ng-morphline-solr-sink:
02  Could not resolve dependencies for project
03 org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.5.0:
04 Failed to collect dependencies for [org.apache.flume:flume-ng-core:jar:1.5.0
05 (compile), org.slf4j:slf4j-api:jar:1.6.1 (compile),
06 org.kitesdk:kite-morphlines-all:pom:0.12.0 (compile?),
07  org.slf4j:jcl-over-slf4j:jar:1.6.1 (provided),
08 org.apache.solr:solr-test-framework:jar:4.3.0 (test),
09 org.kitesdk:kite-morphlines-solr-core:jar:tests:0.12.0 (test),
10 junit:junit:jar:4.10 (test)]: Failed to read artifact descriptor
11 for ua_parser:ua-parser:jar:1.3.0: Could not transfer artifact
12 ua_parser:ua-parser:pom:1.3.0 from/to maven-twttr (http://maven.twttr.com):
13  Connection to http://maven.twttr.com refused: Connection timed out -> [Help 1]

只需要加一个新的repository即可：

假如上面的repository仍然不能用，可以google “ua-parser-1.3.0.pom”获取其他源。

六、编译Flume-ng 1.5.0源码

1 # mvn install -Phadoop-2 -DskipTests -Dtar

编译需要一段时间，这个需要看你编译时的网速。如果一切顺利的话，你将会看到以下输出，说明你的Flume-ng 1.5.0编译成功：

01 [INFO] ------------------------------------------------------------------------
02 [INFO] Reactor Summary:
03 [INFO]
04 [INFO] Apache Flume ...................................... SUCCESS [1.360s]
05 [INFO] Flume NG SDK ...................................... SUCCESS [2.903s]
06 [INFO] Flume NG Configuration ............................ SUCCESS [0.483s]
07 [INFO] Flume NG Core ..................................... SUCCESS [2.531s]
08 [INFO] Flume NG Sinks .................................... SUCCESS [0.110s]
09 [INFO] Flume NG HDFS Sink ................................ SUCCESS [1.668s]
10 [INFO] Flume NG IRC Sink ................................. SUCCESS [0.484s]
11 [INFO] Flume NG Channels ................................. SUCCESS [0.074s]
12 [INFO] Flume NG JDBC channel ............................. SUCCESS [0.579s]
13 [INFO] Flume NG file-based channel ....................... SUCCESS [0.712s]
14 [INFO] Flume NG Spillable Memory channel ................. SUCCESS [0.552s]
15 [INFO] Flume NG Node ..................................... SUCCESS [0.619s]
16 [INFO] Flume NG Embedded Agent ........................... SUCCESS [1.001s]
17 [INFO] Flume NG HBase Sink ............................... SUCCESS [2.373s]
18 [INFO] Flume NG ElasticSearch Sink ....................... SUCCESS [0.899s]
19 [INFO] Flume NG Morphline Solr Sink ...................... SUCCESS [2.483s]
20 [INFO] Flume NG Kite Dataset Sink ........................ SUCCESS [35.925s]
21 [INFO] Flume Sources ..................................... SUCCESS [0.054s]
22 [INFO] Flume Scribe Source ............................... SUCCESS [1.231s]
23 [INFO] Flume JMS Source .................................. SUCCESS [2.133s]
24 [INFO] Flume Twitter Source .............................. SUCCESS [1.204s]
25 [INFO] Flume legacy Sources .............................. SUCCESS [0.043s]
26 [INFO] Flume legacy Avro source .......................... SUCCESS [1.234s]
27 [INFO] Flume legacy Thrift Source ........................ SUCCESS [1.320s]
28 [INFO] Flume NG Clients .................................. SUCCESS [0.038s]
29 [INFO] Flume NG Log4j Appender ........................... SUCCESS [6.175s]
30 [INFO] Flume NG Tools .................................... SUCCESS [1.559s]
31 [INFO] Flume NG distribution ............................. SUCCESS [1:51.993s]
32 [INFO] Flume NG Integration Tests ........................ SUCCESS [1.517s]
33 [INFO] ------------------------------------------------------------------------
34 [INFO] BUILD SUCCESS
35 [INFO] ------------------------------------------------------------------------
36 [INFO] Total time: 3:03.921s
37 [INFO] Finished at: Mon Jun 16 23:37:51 CST 2014
38 [INFO] Final Memory: 135M/745M
39 [INFO] ------------------------------------------------------------------------

最后将会在flume-ng-dist/target目录下产生以下的文件：

1 total 29208
2 drwxr-xr-x 3 wyp wyp     4096 Jun 16 23:36 apache-flume-1.5.0-bin
3 -rw-r--r-- 1 wyp wyp 27307850 Jun 16 23:36 apache-flume-1.5.0-bin.tar.gz
4 drwxr-xr-x 3 wyp wyp     4096 Jun 16 23:36 apache-flume-1.5.0-src
5 -rw-r--r-- 1 wyp wyp  2543174 Jun 16 23:37 apache-flume-1.5.0-src.tar.gz
6 drwxr-xr-x 2 wyp wyp     4096 Jun 16 23:36 archive-tmp
7 drwxr-xr-x 3 wyp wyp     4096 Jun 16 23:36 maven-shared-archive-resources
8 -rw-r--r-- 1 wyp wyp      990 Jun 16 23:37 rat.txt

apache-flume-1.5.0-bin这个文件夹里面的东西可以直接拿来用。