基于arm架构编译安装Ambari的一种方法
引言
目前市场上主流的大数据平台以CDH和HDP两种为主，而两种大数据平台都是基于Apache原生态版本改进的，HDP版本的更贴近于原生态版本，开源率100%，基本上支持所有的开源大数据组件。CDH版本大数据平台经过二次开发与原生态版本有较大差异，且在2018年已经闭源，但市场占有率较大。以上两种大数据平台都仅支持x86架构，早在几年期RedHat联手cloudera公司发表声明将推出64位ARM版，据悉目前Centos7.5以上版本系统已经支持arm架构，cloudera公司产品未见推出arm机构大数据平台，且Cloudera和Hortonworks合并后的CDH和HDP版本都无法下载，猜测开始进行收费。目前Cloudera官方最新大数据平台CDP是CDH与HDP两个产品合并后的新产品，之前有咨询官方客服，收到回复最新版HDP支持arm架构，但无法下载。
以下为各个版本支持架构
| 产品 | 支持架构 | 是否收费 |
|---|---|---|
| CDH | x86 | 6.3版本后需从官方下载 |
| HDP | x86、Power 8、Power 9 | 官方最新3.1.5，githup最新2.7.5 |
| HD（华为） | x86、arm（已知） | 是 |
| TDH（星环） | x86、arm（已知） | 是 |
| CRH（红象云腾） | x86、arm（已知） | 是 |
编译环境准备
下面开始基于arm架构编译Ambari
参考地址http://www.360doc.com/content/19/1027/03/40492717_869278661.shtml
- 安装基础环境
gcc , c++ ,python-dev, git , bower, brunch,gulp
- 安装python-dev
apt-get install python-dev
- 安装Phantomjs
apt-get isntall phantomjs
修改配置 /usr/bin/phantomjs
#!/bin/sh
LD_LIBRARY_PATH="/usr/lib/phantomjs:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH
export QT_QPA_PLATFORM=offscreen
export QT_QPA_FONTDIR=/usr/share/fonts
exec "/usr/lib/phantomjs/phantomjs" "$@"
检查版本
phantomjs -v
2.1.1
- 编译安装leveldb
wget http://pkgs.fedoraproject.org/repo/pkgs/snappy/snappy-1.0.5.tar.gz/4c0af044e654f5983f4acbf00d1ac236/snappy-1.0.5.tar.gz
tar -xf snappy-1.0.5.tar.gz ; cd snappy-1.0.5
./configure --disable-shared --with-pic --host aarch64-unknown-linux --build arrch64-unknown-linux
make -j4
- 编译leveldb
git clone https://github.com/google/leveldb.git
sudo apt-get install cmake
cd leveldb
cmake CMakeLists.txt
make
生成：leveldb.a文件
- 安装leveldbjni
https://opsx.alibaba.com/mirror/search?q=leveldbjni&lang=zh-CN
阿里源下载leveldbjni-javadoc-1.8-19.fc29.noarch.rpm
通过alien 转换为deb包
Dpkg 解压安装
- 编译安装frontend-maven-plugin
git clone https://github.com/eirslett/frontend-maven-plugin.git
cd frontend-maven-plugin
git checkout -b 0.0.16 frontend-plugins-0.0.16
mvn clean -DskipTests install -U
编译ambari源码
- 下载ambari源码
wget http://archive.apache.org/dist/ambari/ambari-2.6.1/apache-ambari-2.6.1-src.tar.gz
tar xfvz apache-ambari-2.6.1-src.tar.gz
cd apache-ambari-2.6.1-s
- 按照网页对文件进行修改
http://www.360doc.com/content/19/1027/03/40492717_869278661.shtml
- 修改完毕后进行编译(编译部分为ambari-metrics)
mvn versions:set -DnewVersion=2.6.1.0.0
pushd ambari-metrics
mvn clean install -U
popd
执行成功后如图：（只要Ambari Metrics编过即可）
当前目录为(编译部分为ambari整体)：
/server/apache-ambari-2.6.1-src
执行：
mvn -B clean install package rpm:rpm -DskipTests -Dpython.ver="python >= 2.6" -Preplaceurl -Drat.ignoreErrors=true -U
如果报错：
[INFO] Compiling 114 source files to /server/apache-ambari-2.6.1-src/ambari-metrics/ambari-metrics-timelineservice/target/classes
编译器 (1.8.0_91) 中出现异常错误。如果在 Bug Database (http://bugs.java.com) 中没有找到该错误, 请通过 Java Bug 报告页 (http://bugreport.java.com) 建立该 Java 编译器 Bug。请在报告中附上您的程序和以下诊断信息。谢谢。
java.lang.NullPointerException
at com.sun.tools.javac.comp.Lower.visitSelect(Lower.java:3837)
at com.sun.tools.javac.tree.JCTree$JCFieldAccess.accept(JCTree.java:1897)
at com.sun.tools.javac.tree.TreeTranslator.translate(TreeTranslator.java:58)
at com.sun.tools.javac.comp.Lower.translate(Lower.java:2371)
at com.sun.tools.javac.comp.Lower.translate(Lower.java:2390)
at com.sun.tools.javac.comp.Lower.visitAssign(Lower.java:3202)
at com.sun.tools.javac.tree.JCTree$JCAssign.accept(JCTree.java:1686)
解决方法：
第二次执行时,该位置编译成功又出问题，报错和上面一样，需要手动删除 /server/apache-ambari-2.6.1-src/ambari-metrics/ambari-metrics-timelineservice/target
使用http服务，搭建本地源
apt-get isntall npm
npm install http-server -g
apt install nodejs-legacy
/usr/local/bin/http-server
安装ambari-server和agent
- 启动agent报错
解决
vim /var/lib/ambari-agent/bin/ambari-agent
export PYTHON=/usr/bin/python
将buildNumber修改为VERSION
报错
解决
cp -r /usr/lib/ambari-server/lib/resource_management/ /usr/lib/ambari-agent/lib/
cp -r /usr/lib/ambari-server/lib/resource_management/ /usr/lib/python2.6/site-packages/
cp -r /usr/lib/ambari-server/lib/ambari_jinja2/ /usr/lib/python2.6/site-packages/
- server端修改
vim /etc/init.d/ambari-server
- 修改base_alert.py | 添加如下代码
vim /usr/lib/python2.6/site-packages/ambari_agent/alerts/base_alert.py
import sys
reload(sys)
sys.setdefaultencoding('utf8')
mysql相关配置
- 修改：vim /etc/mysql/mysql.conf.d/mysqld.cnf
将bind-address 127.0.0.1修改为0.0.0.0
#bind-address = 127.0.0.1
bind-address = 0.0.0.0
重启mysql
- 将mysql驱动包复制到/usr/share/java/下；
修改配置文件：vim /etc/ambari-server/conf/ambari.properties
server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar
- 启动前需要手动在mysql数据库中创建ambari库、用户、表
mysql> create database ambari DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)
mysql> create user 'ambari'@'%' identified by 'bigdata';
Query OK, 0 rows affected (0.00 sec)
mysql> grant select,insert,update,delete,create on ambari.* to ambari;
Query OK, 0 rows affected (0.00 sec)
mysql> grant all privileges on *.* to ambari@'localhost' identified by 'bigdata';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> grant all privileges on *.* to ambari@'node1' identified by 'bigdata';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
执行：
mysql -uambari -p ambari </var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
密码输入：bigdata
- 修改配置文件：
vim /etc/ambari-agent/conf/ambari-agent.ini
[server]
hostname=node1
url_port=8440
secured_url_port=8441
connect_retry_delay=10
max_reconnect_retry_delay=30
将[server]下的[hostname]值为ambari-server所在的主机名。
重启ambari-agent。
- Ambari-server setup
[Unauthorized System] root:# /etc/init.d/ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
WARNING: Could not run /usr/sbin/sestatus: OK
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):ambari
Adjusting ambari-server permissions and ownership...
Checking firewall status...
/bin/bash: ufw: command not found
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
=============================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/lib/jvm/java-8-openjdk-arm64
Validating JDK on Ambari Server...done.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? n
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
=============================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
=============================================================
Enter choice (1): 3
Hostname (localhost):
Port (3306):
Database name (ambari):
Username (ambari):
Enter Database Password (bigdata):
Configuring ambari database...
Proceed with configuring remote database connection properties [y/n] (y)?
Extracting system views...
ambari-admin-2.6.1.0.0.jar
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
- 创建python链接
ln -s /usr/bin/python /usr/bin/ambari-python-wrap
- 启动ambari-server
/etc/init.d/ambari-server start
- 启动ambari-agent
/etc/init.d/ambari-agent start
- 访问界面
经过重重困难终于安装成功，但是仍然有一些amd安装包需要进行重新打包安装，另外此平台还有很多需测试和解决问题，依靠个人力量仍然无法完成，希望各位兄弟支持共同完成。
最后转载两个华为大佬编译博客，强烈建议参考
https://bbs.huaweicloud.com/forum/thread-38329-1-1.html?from=groupmessage
https://bbs.huaweicloud.com/forum/thread-28829-1-1.html