麒麟系统安装HDP【已解决】

麒麟系统安装HDP

1、软件版本介绍

操作系统：Kylin-Server-10-SP2-x86_64
HDP版本： 3.1.5.0-152

2、文件替换

本安装文件中的kylin.zip

本计划逐个检查然后手动添加，由于比较紧急，直接替换了。
如果时间充裕喜欢研究，可以对比下和原本的有啥区别。

比较好的一个对比软件：Beyond_Compare-4-Win-Release.exe

os_check.py
os_utils(agent).py
os_utils(server).py
yum_manager.py
做好备份
修改oc_check.py伪装麒麟系统成centos8
/var/lib/ambari-agent/tmp/ambari_commons/os_check.py
/usr/lib/ambari-server/lib/ambari_commons/os_check.py
/usr/lib/ambari-agent/lib/ambari_commons/os_check.py

/usr/lib/ambari-server/lib/ambari_commons/repo_manager/yum_manager.py

以os_check.py为例，基本就是把kylin伪装成centos
----ps:这里开始伪装成了7，后面的其他操作也报错了，改成了8
----麒麟v10与centos8类似

188行

80行

91行

3 报错解决

3.1 解决KeyError: ‘HDP-3.1’（所有机器）

修该完成后，给其他机器分发即可
scp -r /etc/yum.repos.d/hdp.repo root@vm1213:/etc/yum.repos.d/
scp -r /etc/yum.repos.d/hdp.repo root@vm1214:/etc/yum.repos.d/

vim /var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/params.py

代码中识别的小写名称（所有机器）
之前安装使用的大写名称，这里就直接复制一个小写名称
cp /etc/yum.repos.d/HDP.repo /etc/yum.repos.d/hdp.repo

内容还需要检查调整。直接修改小写即可

根据代码可以看到会对中括号进行操作，这里修改的时候 [] 里面填写了对应的代码中的值
[HDP-3.1]
[HDP-UTILS-1.1.0.22]
[HDP-3.1-GPL]

vim hdp.repo

[HDP-3.1]
name=HDP-3.1
baseurl=http://vm1212/HDP/centos7/3.1.5.0-152
gpgcheck=1
gpgkey=http://vm1212/HDP/centos7/3.1.5.0-152/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://vm1212/HDP-UTILS/centos7/1.1.0.22
gpgcheck=1
gpgkey=http://vm1212/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-3.1-GPL]
name=HDP-GPL Version - HDP-GPL-3.1.5.0
baseurl=http://vm1212/HDP-GPL/centos7/3.1.5.0-152
gpgcheck=1
gpgkey=http://vm1212/HDP-GPL/centos7/3.1.5.0-152/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

3.2 安装smartsense-hst（所有机器）

yum -y install smartsense-hst

3.3 解决Non-ASCII character ‘\xe5’ in file（所有机器）

/var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/params.py

报错编码的问题
在文件首行加#coding=utf-8没起作用，定位到报错的行数，是中文注释，直接把中文注释删了（具体是哪个文件，根据报错提示修改）

3.4 解决ImportError: No module named yum（所有机器）

ImportError: No module named yum
ImportError: dnf.cli

>>> import yum
Traceback (most recent call last):
  File "", line 1, in <module>
ImportError: No module named yum

卸载

此问题主要是由于版本的问题造成了，做了一些尝试无果。直接卸载重新安装
本操作卸载了python2 python3 yum 谨慎操作

卸载python

# 卸载
rpm -qa|grep python|xargs rpm -ev --allmatches --nodeps 
# 删除所有残余文件
whereis python |xargs rm -frv 
# 验证删除，返回无结果表示卸载干净了
whereis python

卸载yum

# 卸载
rpm -qa|grep yum|xargs rpm -ev --allmatches --nodeps 
# 删除所有残余文件
whereis yum |xargs rm -frv 
# 验证删除，返回无结果表示卸载干净了
whereis yum

python2

如果卸载过python和yum需要下载重装
新建一个文件夹路径存放下载到的依赖包

wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-help-2.7.18-1.se.01.ky10.noarch.rpm 
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-2.7.18-1.se.01.ky10.x86_64.rpm 
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-setuptools-44.1.1-1.ky10.noarch.rpm 
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-tools-2.7.18-1.se.01.ky10.x86_64.rpm 
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-debug-2.7.18-1.se.01.ky10.x86_64.rpm 
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-rpm-4.15.1-17.ky10.x86_64.rpm 
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-tkinter-2.7.18-1.se.01.ky10.x86_64.rpm 
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-devel-2.7.18-1.se.01.ky10.x86_64.rpm 
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-libxml2-2.9.10-11.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-rpm-4.15.1-17.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-ipaddress-1.0.23-1.ky10.noarch.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/yum-4.2.23-3.p02.ky10.noarch.rpm  
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-libdnf-0.48.0-1.p01.ky10.x86_64.rpm  

yum

wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/yum-4.2.23-3.p02.ky10.noarch.rpm 
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/yum-langpacks-0.4.5-10.oe1.noarch.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/yum-langpacks-help-0.4.5-10.oe1.noarch.rpm 
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/yum-metadata-parser-1.1.4-24.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/yum-metadata-parser-help-1.1.4-24.ky10.x86_64.rpm

python3和dnf

wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/dnf-4.2.23-3.p02.ky10.noarch.rpm 
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/dnf-automatic-4.2.23-3.p02.ky10.noarch.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/dnf-help-4.2.23-3.p02.ky10.noarch.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/dnf-plugins-core-4.0.17-1.ky10.noarch.rpm 
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/dnf-plugins-core-help-4.0.17-1.ky10.noarch.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python3-3.7.9-6.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python3-gpgme-1.14.0-1.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python3-rpm-4.15.1-17.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python3-libdnf-0.48.0-1.p01.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python3-libcomps-0.1.10-2.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python3-hawkey-0.48.0-1.p01.ky10.x86_64.rpm 
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python3-dnf-4.2.23-3.p02.ky10.noarch.rpm 

安装

rpm -Uvh --replacepkgs * --nodeps --force

更新三个内容

yum install python2
yum install python3
yum install dnf

可能还会报错site-packages

这里采用了其他方式。打包了centos7系统上安装的python2的site-packages文件
替换现有python2包
路径：/usr/lib/python2.7/site-packages所有文件
替换全部服务器/usr/lib/python2.7/site-packages（做好服务器备份）

本安装文件中的site-packages.tar.gz

其他以下依赖包根据报错自行下载

openssl、nmap、libssh2、util-linux-devel

wget https://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/openssl-1.1.1f-4.p01.ky10.x86_64.rpm 
wget https://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/openssl-libs-1.1.1f-4.p01.ky10.x86_64.rpm 
wget https://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/nmap-7.80-1.ky10.x86_64.rpm 

wget 
https://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/libssh2-1.9.0-6.ky10.x86_64.rpm 
wget https://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/libssh2-help-1.9.0-6.ky10.noarch.rpm
wget https://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/libssh2-devel-1.9.0-6.ky10.x86_64.rpm

wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/util-linux-devel-2.35.2-10.p01.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/util-linux-devel-2.35.2-10.p02.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/util-linux-devel-2.35.2-8.p01.ky10.x86_64.rpm

3.5 解决redhat-lsb报错（所有机器）

redhat-lsb依赖 错误，麒麟仓库该依赖不支持下载centos8强制安装解决
安装

wget https://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/kylin-lsb-5.0-1.p01.ky10.x86_64.rpm 

保证依赖可用
验证：lsb_release

强制安装一下el8.x86_64.rpm结尾的依赖
http://mirrors.aliyun.com/centos/8.5.2111/AppStream/x86_64/os/Packages/?spm=a2c6h.25603864.0.0.50f21a7abxdrEu/

本文件包含redhat-lsb的麒麟版本但属于aarch64结构，并没有使用

安装成功后检测是否可用

3.6 解决No package hadoop_${stack_version}

resource_management.core.exceptions.Fail: No package found for -yarn(expected name: hadoop_3_1-yarn)

直接写死stack_version的版本

yum_manager.py使用的是本文档中的脚本

vim /usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py

3.7 解决hive无法连接数据库

输入hive命令如无法连接数据库，报下图错误

检查用户名密码配置

Mysql的hive库权限要给对应的机器
主要执行GRANT

CREATE USER 'hive'@'vm1213' IDENTIFIED BY 'bigdata'; 
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'vm1213'; 
CREATE USER 'hive'@'vm1214' IDENTIFIED BY 'bigdata'; 
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'vm1214';

3.8 报错ambari-server setup

主要是伪装成centos

如果还报错把ambari.properties替换成本文中的ambari.properties
vim /etc/ambari-server/conf/ambari.properties

4、依赖报错离线下载

根据需求修改上下文：

启动安装时如依赖找不到，依据自身情况下载
麒麟仓库：https://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/
麒麟官网
https://update.cs2c.com.cn/NS/V10/V10SP1/os/adv/lic/base/aarch64/Packages/
https://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/aarch64/Packages/
麒麟
https://archive.kylinos.cn/yum/v10/kylin-openstack-q/aarch64/aarch64/