国产麒麟系统部署Ambari + HDP
前言
Ambari部署在国产麒麟系统上，之前尝试过一次部署，卡在yum模块，后续因为其他事情没有继续进行，最近在网上看到有大神处理了yum模块的内容，重新部署一次，特意记录下来；
参考：
https://blog.csdn.net/Jason1511/article/details/131481518
https://blog.csdn.net/Jason1511/article/details/131580626
版本
| 系统版本 | Kylin Linux Advanced Server release V10 (Tercel) |
|---|---|
| 系统架构 | x86_64 |
| ambari | ambari-2.7.5.0-centos7.tar.gz |
| HDP | HDP-3.1.5.0-centos7-rpm.tar.gz |
| HDP-GPL-3.1.5.0-centos7-gpl.tar.gz | |
| HDP-UTILS-1.1.0.22-centos7.tar.gz |
一、虚拟机部署麒麟系统
麒麟系统虚拟机安装教程
系统镜像名称：（去银河麒麟官网下载，速度更快）
Kylin-Server-10-SP1-Release-Build20-20210518-x86_64.iso
二、安装包准备
官网不能下载了，需要收费，提供一个百度云链接
链接：[https://pan.baidu.com/s/1sw8prtB29mR1JsZ-DOU-Pw](https://pan.baidu.com/s/1sw8prtB29mR1JsZ-DOU-Pw)
提取码：4p4n
三、环境准备
| IP | 主机名 |
|---|---|
| 192.168.119.129 | master |
| 192.168.119.130 | node1 |
3.1、环境设置（所有节点执行）
参考：环境设置
3.2、JDK卸载与安装（所有节点执行）
麒麟系统自带有JDK1.8和JDK11两个版本，这两个版本都需要卸载并重新安装
rpm -qa|grep java
rpm -e --nodeps java-1.8.0-openjdk-1.8.0.242.b08-1.h5.ky10.x86_64
rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.242.b08-1.h5.ky10.x86_64
rpm -e --nodeps java-11-openjdk-headless-11.0.6.10-4.ky10.ky10.x86_64
rpm -e --nodeps java-11-openjdk-11.0.6.10-4.ky10.ky10.x86_64
部署
rpm -ivh jdk-8u152-linux-x64.rpm
vim /etc/profile
JAVA_HOME=/usr/java/jdk1.8.0_152
CLASSPATH=$JAVA_HOME/lib/
PATH=$PATH:$JAVA_HOME/bin
export PATH JAVA_HOME CLASSPATH
3.3、YUM源设置(离线安装需要配置)
3.4、时间服务器同步
# master节点执行
yum -y install chrony
vim /etc/chrony.conf
server 192.168.110.130 iburst
allow 192.168.119.0/24
local stratum 10
systemctl restart chronyd.service
systemctl enable chronyd.service
# 查看时间同步状态
chronyc -a makestep
chronyc sourcestats
chronyc sources -v
# 其他节点执行
yum -y install chrony
vim /etc/chrony.conf
server 192.168.110.130 iburst
systemctl restart chronyd.service
systemctl enable chronyd.service
3.5、MySQL部署（主节点执行）
MySQL部署
四、安装基础服务（主节点执行）
# 安装HTTP服务
yum -y install httpd
systemctl restart httpd
systemctl enable httpd
# 将ambari + HDP安装包上传至服务器
tar -zxvf ambari-2.7.5.0-centos7.tar.gz -C /var/www/html/
tar -zxvf HDP-3.1.5.0-centos7-rpm.tar.gz -C /var/www/html/
tar -zxvf HDP-GPL-3.1.5.0-centos7-gpl.tar.gz -C /var/www/html/
tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/
mkdir /var/www/html/libtrpc
mv libtirpc-0.2.4-0.16.el7.x86_64.rpm /var/www/html/libtrpc
mv libtirpc-devel-0.2.4-0.16.el7.x86_64.rpm /var/www/html/libtrpc
cd /var/www/html/libtrpc
createrepo .
# 制作本地YUM源
vim /etc/yum.repos.d/ambari.repo
[Ambari-2.7.5.0]
name=Ambari-2.7.5.0
baseurl=http://master/ambari/centos7/2.7.5.0-72/
gpgcheck=0
enabled=1
priority=1
vim /etc/yum.repos.d/HDP.repo
[HDP-3.1.5.0]
name=HDP Version - HDP-3.1.5.0
baseurl=http://master/HDP/centos7/3.1.5.0-152/
gpgcheck=0
enabled=1
priority=1
[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://master/HDP-UTILS/centos7/1.1.0.22/
gpgcheck=0
enabled=1
priority=1
[HDP-GPL-3.1.5.0]
name=HDP-GPL Version - HDP-GPL-3.1.5.0
baseurl=http://master/HDP-GPL/centos7/3.1.5.0-152
gpgcheck=0
enabled=1
priority=1
vim /etc/yum.repos.d/libtrpc.repo
[libtirpc_repo]
name=libtirpc-0.2.4-0.16
baseurl=http://master/libtrpc/
gpgcheck=0
enabled=1
priority=1
scp /etc/yum.repos.d/ambari.repo node1:/etc/yum.repos.d/
scp /etc/yum.repos.d/HDP.repo node1:/etc/yum.repos.d/
scp /etc/yum.repos.d/libtrpc.repo node1:/etc/yum.repos.d/
#分别在每台机器上执行：
cd /etc/yum.repos.d/
yum clean all
yum makecache
五、Ambari-Server部署
5.1、安装ambari-server
yum -y install ambari-server
报错信息如下：
解决：**yum remove libpq**
cp /software/mysql-connector-java-5.1.40-bin.jar /usr/share/java/
cp /software/mysql-connector-java-5.1.40-bin.jar /var/lib/ambari-server/resources/
vim /etc/ambari-server/conf/ambari.properties
server.jdbc.driver.path=/software/mysql-connector-java-5.1.40-bin.jar
5.2、解决KylinV10系统上的问题
关键信息来了：yum 安装完server后，需要修改ambari-server的系统检查设置
一共需要修改三处信息:
vim /usr/lib/ambari-server/lib/ambari_commons/os_check.py
第一处：在80行左右相似格式的地方插入如下代码(注意格式对齐)
_IS_KYLIN_LINUX = os.path.exists('/etc/kylin-release')
第二处：在90行左右相似格式的地方插入如下代码
def _is_kylin_linux():
return _IS_KYLIN_LINUX
第三处：在210行左右相似格式的地方插入如下代码
elif _is_kylin_linux():
distribution =("centos","7","core")
5.3、配置ambari-server
ambari-server setup --jdbc-db=mysql --jdbc-driver=/software/mysql-connector-java-5.1.40-bin.jar
ambari-server setup
[root@master software]# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)?
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/java/jdk1.8.0_152
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)?
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost):
Port (3306):
Database name (ambari):
Username (ambari): root
Enter Database Password (bigdata):
Re-enter password:
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.7.5.0.72.jar
....
Ambari repo file doesn't contain latest json url, skipping repoinfos modification
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
5.4、配置MySQL表
mysql -uroot -proot
create database ambari;
use ambari;
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
5.5、安装ambari-agent（所有节点执行）
yum -y install ambari-agent
vim /etc/ambari-agent/conf/ambari-agent.ini
systemctl start ambari-agent
5.6、启动ambari-server
ambari-server start
启动成功会提示webUI地址：http://hostname:8080
账户：admin 密码：admin
注意：ambari-server在启动时会报错：Cannot register host with not supported os type, hostname=node1, serverOsType=centos7, agentOsType=kylin linux advanced server release v10
解决：执行下面命令后重启
cp /etc/system-release /etc/system-release-bak
vim /etc/system-release
CentOS Linux release 7.5.1804 (Core)
六、可视化界面配置
http://192.168.119.130:8080
6.1、异常一：Authorized users only
在这一步会报错，日志如下：
Authorized users only. All activities may be monitored and reported.
Load key "/var/run/ambari-server/bootstrap/2/sshKey": invalid format
root@master: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
原因分析：ssh 私钥是通过web界面的文本框传过去的，感觉是web端的bug，没有将私钥写入服务器
解决：将sshKeyFile这个变量写死
修改源码：
cp /root/.ssh/id_rsa /var/run/ambari-server/bootstrap/1/sshKey
vim /usr/lib/ambari-server/lib/ambari_server/bootstrap.py (910行)
6.2、异常二：/usr/sbin/hst: 没有那个文件或目录
Failed to execute command: /usr/sbin/hst agent-status; Exit code: 127; stdout: ; stderr: /bin/sh: /usr/sbin/hst: 没有那个文件或目录
解决：
yum install smartsense-hst
yum install hdp-select
6.3、异常三：ImportError: No module named yum
解决：将centos7里面python2.7的site-packages包替换过来
mv /usr/lib/python2.7/site-packages/ /usr/lib/python2.7/site-packages-bak
scp -r /usr/lib/python2.7/site-packages node1:/usr/lib/python2.7/
6.4、异常四：ImportError: No module named pycurl
解决：
python -m ensurepip --upgrade
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
yum install -y libidn
yum install -y libidn-devel
yum install -y nss-pem
yum install -y libcurl
yum install -y libcurl-devel
pip install pycurl
6.5、异常五：ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other)
解决：
yum install -y openssl-devel
pip uninstall pycurl
export PYCURL_SSL_LIBRARY=openssl
pip install pycurl
6.6、异常六： ImportError: No module named sqlitecachec
解决：
rpm -ivh http://mirror.centos.org/centos/7/os/x86_64/Packages/yum-metadata-parser-1.1.4-10.el7.x86_64.rpm --force
6.7、异常七：nothing provides redhat-lsb needed
解决：
mkdir -p /software/rpm
cd /software/rpm
wget http://mirrors.sohu.com/centos/7/os/x86_64/Packages/redhat-lsb-4.1-27.el7.centos.1.x86_64.rpm
wget http://mirrors.sohu.com/centos/7/os/x86_64/Packages/redhat-lsb-core-4.1-27.el7.centos.1.x86_64.rpm
wget http://mirrors.sohu.com/centos/7/os/x86_64/Packages/redhat-lsb-cxx-4.1-27.el7.centos.1.x86_64.rpm
wget http://mirrors.sohu.com/centos/7/os/x86_64/Packages/redhat-lsb-desktop-4.1-27.el7.centos.1.x86_64.rpm
wget http://mirrors.sohu.com/centos/7/os/x86_64/Packages/redhat-lsb-languages-4.1-27.el7.centos.1.x86_64.rpm
wget http://mirrors.sohu.com/centos/7/os/x86_64/Packages/redhat-lsb-printing-4.1-27.el7.centos.1.x86_64.rpm
wget http://mirrors.sohu.com/centos/7/os/x86_64/Packages/redhat-lsb-submod-multimedia-4.1-27.el7.centos.1.x86_64.rpm
wget http://mirrors.sohu.com/centos/7/os/x86_64/Packages/redhat-lsb-submod-security-4.1-27.el7.centos.1.x86_64.rpm
wget http://mirrors.sohu.com/centos/7/os/x86_64/Packages/redhat-lsb-supplemental-4.1-27.el7.centos.1.x86_64.rpm
wget http://mirrors.sohu.com/centos/7/os/x86_64/Packages/redhat-lsb-trialuse-4.1-27.el7.centos.1.x86_64.rpm
rpm -ivh *rpm --nodeps --force
