Ambari源码编译

Ambari为Apache基金会的一个顶级项目。作为一个为Hadoop系统提供一站式维护的工具，ambari支持通过web页面/RestfulApi的方式进行对集群的一键部署，最重要的是可以自己编写部署脚本部署自己的分布式系统。这个是富有价值的，现在有的公司使用的是自研的分布式框架，想要自动部署大多也是自研一套分布式框架。现在Ambari的出现，缓解了这个问题，用户只需要使用python按照规则编写一套部署在自己系统上的安装/启动脚本，就可以将自己的系统使用Ambari进行一键式部署。

Ambari不仅提供集群组件的部署功能，还提供了集群的监控功能，不过现在的监控只支持Hadoop生态中的服务，如果想使用Ambari进行监控，可能需要二次开发，这里我没有使用Ambari进行过对自研计算框架或其他服务的监控。

这次，我也要系统的学习一下Ambari底层的实现，以及监控功能如何使用。所以，现在先从编译开始，一步一步进行总结。

首先，我的环境是Mac OS，不过其他Linux环境也可以。

1.安装maven 下面是Mac环境的 Linux/Windows可以百度，具体安装大致相似。

    (1) maven 3.0.4下载链接(注意版本一定要3.0.4 或 3.0.5) http://archive.apache.org/dist/maven/maven-3/3.0.4/binaries/apache-maven-3.0.4-bin.tar.gz

    (2) 解包并copy到对应目录

tar zxvf apache-maven-3.0.4-bin.tar.gz
cp apache-maven-3.0.4 /opt/local/share/java/maven3.0.4

    (3) vi ~/.bash_profile 文件中依次配置 

M3_HOME=/opt/local/share/java/maven3.0.4
PATH=$M3_HOME/bin:$PATH
export M3_HOME
export PATH

    (4) source ~/.bash_profile 并通过mvn --version 查看版本是否正确

      maven的版本一定要是3.0.4或3.0.5，对于其他的三系版本，可能在Mac系统下会有bug，在编译的时候显示缺少对应的类。详情可以参见maven的jira: https://issues.apache.org/jira/browse/MNG-5787

      具体的错误可能如下:

[INFO] 
[INFO] >>> maven-eclipse-plugin:2.10:eclipse (default-cli) > generate-resources @ ambari-admin >>>
[INFO] 
[INFO] --- build-helper-maven-plugin:1.8:regex-property (parse-package-version) @ ambari-admin ---
[INFO] 
[INFO] --- build-helper-maven-plugin:1.8:regex-property (parse-package-release) @ ambari-admin ---
[INFO] 
[INFO] --- build-helper-maven-plugin:1.8:parse-version (parse-version) @ ambari-admin ---
[INFO] 
[INFO] --- build-helper-maven-plugin:1.8:regex-property (regex-property) @ ambari-admin ---
[INFO] 
[INFO] --- frontend-maven-plugin:0.0.16:install-node-and-npm (install node and npm) @ ambari-admin ---
Downloading: http://maven.oschina.net/content/groups/public/org/codehaus/plexus/plexus-classworlds/2.4/plexus-classworlds-2.4.jar
Downloaded: http://maven.oschina.net/content/groups/public/org/codehaus/plexus/plexus-classworlds/2.4/plexus-classworlds-2.4.jar (46 KB at 165.3 KB/sec)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Ambari Main ........................................ SUCCESS [  1.323 s]
[INFO] Apache Ambari Project POM .......................... SUCCESS [  0.033 s]
[INFO] Ambari Web ......................................... SUCCESS [  0.029 s]
[INFO] Ambari Views ....................................... SUCCESS [  0.270 s]
[INFO] Ambari Admin View .................................. FAILURE [  3.559 s]
[INFO] ambari-metrics ..................................... SKIPPED
[INFO] Ambari Metrics Common .............................. SKIPPED
[INFO] Ambari Metrics Hadoop Sink ......................... SKIPPED
[INFO] Ambari Metrics Flume Sink .......................... SKIPPED
[INFO] Ambari Metrics Kafka Sink .......................... SKIPPED
[INFO] Ambari Metrics Storm Sink .......................... SKIPPED
[INFO] Ambari Metrics Collector ........................... SKIPPED
[INFO] Ambari Metrics Monitor ............................. SKIPPED
[INFO] Ambari Metrics Assembly ............................ SKIPPED
[INFO] Ambari Server ...................................... SKIPPED
[INFO] Ambari Agent ....................................... SKIPPED
[INFO] Ambari Client ...................................... SKIPPED
[INFO] Ambari Python Client ............................... SKIPPED
[INFO] Ambari Groovy Client ............................... SKIPPED
[INFO] Ambari Shell ....................................... SKIPPED
[INFO] Ambari Python Shell ................................ SKIPPED
[INFO] Ambari Groovy Shell ................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:42 min
[INFO] Finished at: 2016-01-26T22:15:04+08:00
[INFO] Final Memory: 23M/165M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.16:install-node-and-npm (install node and npm) on project ambari-admin: Execution install node and npm of goal com.github.eirslett:frontend-maven-plugin:0.0.16:install-node-and-npm failed: A required class was missing while executing com.github.eirslett:frontend-maven-plugin:0.0.16:install-node-and-npm: org/slf4j/helpers/MarkerIgnoringBase
[ERROR] -----------------------------------------------------
[ERROR] realm =    plugin>com.github.eirslett:frontend-maven-plugin:0.0.16
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] = file:/opt/local/mvnlib/m3/com/github/eirslett/frontend-maven-plugin/0.0.16/frontend-maven-plugin-0.0.16.jar
[ERROR] urls[1] = file:/opt/local/mvnlib/m3/com/github/eirslett/frontend-plugin-core/0.0.16/frontend-plugin-core-0.0.16.jar
[ERROR] urls[2] = file:/opt/local/mvnlib/m3/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar
[ERROR] urls[3] = file:/opt/local/mvnlib/m3/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar
[ERROR] urls[4] = file:/opt/local/mvnlib/m3/org/apache/commons/commons-compress/1.5/commons-compress-1.5.jar
[ERROR] urls[5] = file:/opt/local/mvnlib/m3/org/tukaani/xz/1.2/xz-1.2.jar
[ERROR] urls[6] = file:/opt/local/mvnlib/m3/commons-io/commons-io/1.3.2/commons-io-1.3.2.jar
[ERROR] urls[7] = file:/opt/local/mvnlib/m3/org/apache/httpcomponents/httpclient/4.3.1/httpclient-4.3.1.jar
[ERROR] urls[8] = file:/opt/local/mvnlib/m3/org/apache/httpcomponents/httpcore/4.3/httpcore-4.3.jar
[ERROR] urls[9] = file:/opt/local/mvnlib/m3/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar
[ERROR] urls[10] = file:/opt/local/mvnlib/m3/commons-codec/commons-codec/1.6/commons-codec-1.6.jar
[ERROR] urls[11] = file:/opt/local/mvnlib/m3/org/codehaus/plexus/plexus-utils/3.0.10/plexus-utils-3.0.10.jar
[ERROR] urls[12] = file:/opt/local/mvnlib/m3/javax/enterprise/cdi-api/1.0/cdi-api-1.0.jar
[ERROR] urls[13] = file:/opt/local/mvnlib/m3/javax/annotation/jsr250-api/1.0/jsr250-api-1.0.jar
[ERROR] urls[14] = file:/opt/local/mvnlib/m3/com/google/guava/guava/10.0.1/guava-10.0.1.jar
[ERROR] urls[15] = file:/opt/local/mvnlib/m3/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar
[ERROR] urls[16] = file:/opt/local/mvnlib/m3/org/sonatype/sisu/sisu-guice/3.1.0/sisu-guice-3.1.0-no_aop.jar
[ERROR] urls[17] = file:/opt/local/mvnlib/m3/aopalliance/aopalliance/1.0/aopalliance-1.0.jar
[ERROR] urls[18] = file:/opt/local/mvnlib/m3/org/eclipse/sisu/org.eclipse.sisu.inject/0.0.0.M2a/org.eclipse.sisu.inject-0.0.0.M2a.jar
[ERROR] urls[19] = file:/opt/local/mvnlib/m3/asm/asm/3.3.1/asm-3.3.1.jar
[ERROR] urls[20] = file:/opt/local/mvnlib/m3/org/codehaus/plexus/plexus-component-annotations/1.5.5/plexus-component-annotations-1.5.5.jar
[ERROR] urls[21] = file:/opt/local/mvnlib/m3/org/apache/maven/plugin-tools/maven-plugin-annotations/3.2/maven-plugin-annotations-3.2.jar
[ERROR] urls[22] = file:/opt/local/mvnlib/m3/com/googlecode/slf4j-maven-plugin-log/slf4j-maven-plugin-log/1.0.0/slf4j-maven-plugin-log-1.0.0.jar
[ERROR] Number of foreign imports: 1
[ERROR] import: Entry[import  from realm ClassRealm[maven.api, parent: null]]
[ERROR] 
[ERROR] -----------------------------------------------------: org.slf4j.helpers.MarkerIgnoringBase
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginContainerException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :ambari-admin

2.安装 python 具体过程请自行搜索。

3.Ambari官网下载源码 Ambari 2.2.0下载链接:http://mirrors.hust.edu.cn/apache/ambari/ambari-2.2.0/apache-ambari-2.2.0-src.tar.gz 使用华科的镜像比较快

4.解包源码后 使用mvn 进行编译，由于我是用eclipse进行源码阅读，所以直接使用下面命令生成ecilpse项目。

mvn eclipse:eclipse

由于Ambari运行环境不支持Mac如果需要编译成binary的需要参照官方文档：https://cwiki.apache.org/confluence/display/AMBARI/Build+and+install+Ambari+2.2.0+from+Source选择生成的是rpm 还是 jdeb格式

5.遇到问题如下：

    (1)下载phantomjs-1.9.7-macosx.zip失败 配置的链接被墙。

[INFO] > [email protected] install /Users/prime7/dc/ambari-2.2.0/ambari-admin/src/main/resources/ui/admin-web/node_modules/phantomjs
[INFO] > node install.js
[INFO]
[INFO] Downloading https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.7-macosx.zip
[INFO] Saving to /var/folders/5c/s73d64p57gq4572b4hxjvn040000gn/T/phantomjs/phantomjs-1.9.7-macosx.zip
[INFO] Receiving...
[INFO]
[INFO] Error making request.
[INFO] Error: socket hang up
[INFO]     at createHangUpError (http.js:1472:15)
[INFO]     at CleartextStream.socketCloseListener (http.js:1522:23)
[INFO]     at CleartextStream.EventEmitter.emit (events.js:117:20)
[INFO]     at tls.js:696:10
[INFO]     at process._tickCallback (node.js:415:13)
[INFO]
[INFO] Please report this full log at https://github.com/Medium/phantomjs
[INFO] npm ERR! [email protected] install: `node install.js`
[INFO] npm ERR! Exit status 1
[INFO] npm ERR!
[INFO] npm ERR! Failed at the [email protected] install script.
[INFO] npm ERR! This is most likely a problem with the phantomjs package,
[INFO] npm ERR! not with npm itself.
[INFO] npm ERR! Tell the author that this fails on your system:
[INFO] npm ERR!     node install.js
[INFO] npm ERR! You can get their info via:
[INFO] npm ERR!     npm owner ls phantomjs
[INFO] npm ERR! There is likely additional logging output above.
[INFO]
[INFO] npm ERR! System Darwin 15.0.0
[INFO] npm ERR! command "/Users/prime7/dc/ambari-2.2.0/ambari-admin/src/main/resources/ui/admin-web/node/node" "/Users/prime7/dc/ambari-2.2.0/ambari-admin/src/main/resources/ui/admin-web/node/npm/bin/npm-cli.js" "install" "--unsafe-perm" "--color=false"
[INFO] npm ERR! cwd /Users/prime7/dc/ambari-2.2.0/ambari-admin/src/main/resources/ui/admin-web
[INFO] npm ERR! node -v v0.10.26
[INFO] npm ERR! npm -v 1.4.3
[INFO] npm ERR! code ELIFECYCLE
[INFO] npm ERR!
[INFO] npm ERR! Additional logging details can be found in:
[INFO] npm ERR!     /Users/prime7/dc/ambari-2.2.0/ambari-admin/src/main/resources/ui/admin-web/npm-debug.log
[INFO] npm ERR! not ok code 0
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ....................................... SUCCESS [0.740s]
[INFO] Apache Ambari Project POM ......................... SUCCESS [0.491s]
[INFO] Ambari Web ........................................ SUCCESS [0.064s]
[INFO] Ambari Views ...................................... SUCCESS [0.208s]
[INFO] Ambari Admin View ................................. FAILURE [44.926s]
[INFO] ambari-metrics .................................... SKIPPED
[INFO] Ambari Metrics Common ............................. SKIPPED
[INFO] Ambari Metrics Hadoop Sink ........................ SKIPPED
[INFO] Ambari Metrics Flume Sink ......................... SKIPPED
[INFO] Ambari Metrics Kafka Sink ......................... SKIPPED
[INFO] Ambari Metrics Storm Sink ......................... SKIPPED
[INFO] Ambari Metrics Collector .......................... SKIPPED
[INFO] Ambari Metrics Monitor ............................ SKIPPED
[INFO] Ambari Metrics Assembly ........................... SKIPPED
[INFO] Ambari Server ..................................... SKIPPED
[INFO] Ambari Agent ...................................... SKIPPED
[INFO] Ambari Client ..................................... SKIPPED
[INFO] Ambari Python Client .............................. SKIPPED
[INFO] Ambari Groovy Client .............................. SKIPPED
[INFO] Ambari Shell ...................................... SKIPPED
[INFO] Ambari Python Shell ............................... SKIPPED
[INFO] Ambari Groovy Shell ............................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 47.645s
[INFO] Finished at: Sat Jan 23 14:42:30 CST 2016
[INFO] Final Memory: 23M/229M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.16:npm (npm install) on project ambari-admin: Failed to run task: 'npm install --unsafe-perm --color=false' failed. (error code 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :ambari-admin

解决过程：查看/Users/prime7/dc/ambari-2.2.0/ambari-admin/src/main/resources/ui/admin-web/npm-debug.log  发现下载的对应链接无法访问，上网查了下是被墙了，可以自行百度phantomjs-1.9.7-macosx.zip进行下载。下载后参照上面错误第一行的uri将对应下载后的zip拷贝到对应路径下。这个是国内镜像下载地址 http://npm.taobao.org/dist/phantomjs/

    (2).编译 Ambari Metrics Collector 缺少ambari-metrics-common jar包。参见 https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development

官方做法是

cd ambari-metrics
mvn clean package -Dbuild-rpm -DskipTests

For Ubuntu:
cd ambari-metrics
mvn clean package -Dbuild-deb -DskipTests

实际在Mac上可以不用生成对应rpm deb 

cd ambari-metrics
 
  
mvn  package -DskipTests
 
  
mvn  install -DskipTests
 
  
 
  
 
  
 
  
对应错误：
 
  
 
  
[INFO] >>> maven-eclipse-plugin:2.10:eclipse (default-cli) @ ambari-metrics-timelineservice >>>
Downloading: http://repo.hortonworks.com/content/groups/public/org/apache/ambari/ambari-metrics-common/2.2.0.0.0/ambari-metrics-common-2.2.0.0.0.jar
Downloading: http://repo.maven.apache.org/maven2/org/apache/ambari/ambari-metrics-common/2.2.0.0.0/ambari-metrics-common-2.2.0.0.0.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ....................................... SUCCESS [0.714s]
[INFO] Apache Ambari Project POM ......................... SUCCESS [0.041s]
[INFO] Ambari Web ........................................ SUCCESS [0.047s]
[INFO] Ambari Views ...................................... SUCCESS [0.137s]
[INFO] Ambari Admin View ................................. SUCCESS [9.250s]
[INFO] ambari-metrics .................................... SUCCESS [0.454s]
[INFO] Ambari Metrics Common ............................. SUCCESS [0.189s]
[INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [0.496s]
[INFO] Ambari Metrics Flume Sink ......................... SUCCESS [0.290s]
[INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [0.093s]
[INFO] Ambari Metrics Storm Sink ......................... SUCCESS [0.376s]
[INFO] Ambari Metrics Collector .......................... FAILURE [6.079s]
[INFO] Ambari Metrics Monitor ............................ SKIPPED
[INFO] Ambari Metrics Assembly ........................... SKIPPED
[INFO] Ambari Server ..................................... SKIPPED
[INFO] Ambari Agent ...................................... SKIPPED
[INFO] Ambari Client ..................................... SKIPPED
[INFO] Ambari Python Client .............................. SKIPPED
[INFO] Ambari Groovy Client .............................. SKIPPED
[INFO] Ambari Shell ...................................... SKIPPED
[INFO] Ambari Python Shell ............................... SKIPPED
[INFO] Ambari Groovy Shell ............................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 19.204s
[INFO] Finished at: Sat Jan 23 15:16:46 CST 2016
[INFO] Final Memory: 35M/241M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project ambari-metrics-timelineservice: Could not resolve dependencies for project org.apache.ambari:ambari-metrics-timelineservice:jar:2.2.0.0.0: Could not find artifact org.apache.ambari:ambari-metrics-common:jar:2.2.0.0.0 in apache-hadoop (http://repo.hortonworks.com/content/groups/public/) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :ambari-metrics-timelineservice
 
  
 
  
 
  
 
  
(3) phantomjs-1.9.8-macosx.zip 下载失败 具体原因和(1)一致。