Hue-release-4.3.0集成Ambari2.7.3
文章目录
- 环境搭建
- hue-release-4.3.0.zip
- hue-4.3.0-stacks
- hue-4.3.0-common-services
- 重启ambari服务-添加Hue服务
- 问题修复
- 问题1：hue_env_python = '
- 问题2：Failed to execute command '/usr/bin/yum -y install saslwrapper-devel', exited with code '1', message: 'Error: Nothing to do
- 问题3：Execution of 'http://10.180.210.241/hue/hue-4.3.0.tgz | xargs wget -O hue.tgz' returned 123. /bin/bash: http://10.180.210.241/hue/hue-4.3.0.tgz: No such file or directory
- 问题4：Execution of 'cd /usr/local/hue && make apps' returned 1. /bin/bash: line 0: cd: /usr/local/hue: No such file or directory
- 问题5：/bin/bash: mvn: command not found
- 问题6：/usr/local/hue目录错误
- 问题7：Execution of '/var/lib/ambari-agent/cache/common-services/HUE/4.3.0/package/files/configs.sh set manager241.bigdata HD45 hive-site 'hive.security.authorization.sqlstd.confwhitelist.append' 'hive.server2.logging.operation.verbose'' returned 1. [ERROR] "hive-site" not found in server response.
- 问题8：Execution of 'cd /user/local/hue && source ./build/env/bin/activate' returned 1. /bin/bash: line 0: cd: /user/local/hue: No such file or directory
- 问题9：Configuration parameter 'java_home' was not found in configurations dictionary!
- 问题10：Invalid HTTP_HOST header: '10.180.210.242:8888'. You may need to add u'10.180.210.242' to ALLOWED_HOSTS.
- 问题11：Table 'hue.auth_user' doesn't exist
- 问题12：访问不存在的接口出现Debug错误信息，不显示404页面错误
- Hue集成Ambari完毕
- Hue集成大数据组件
- Hue-Hive
- 问题1：hue集成hive时获取hive_server_hosts错误
- Hue-Spark
- 问题1：hue集成spark_sql时报错获取sql_server_host,sql_server_port错误
- Hue-Oozie
- 问题1：配置文件pseudo-distributed.ini oozie_url=http://localhost:11000/oozie 连接显示localhost，获取不到oozie服务节点
- Hue-Zookeeper
- 问题1：配置文件pseudo-distributed.ini [zookeeper] [[clusters]] [[[default]]] host_ports=
环境搭建
| 组件 | 描述 |
|---|---|
| Ambari | 2.7.1 |
| HDP | 3.0.1 |
| Hue | 4.3.0 |
- Ambari2.7.1集群搭建流程参考：https://blog.csdn.net/Happy_Sunshine_Boy/article/details/86595945
- 下载hue-4.3.0源码包：hue-release-4.3.0.zip
- https://github.com/cloudera/hue/releases/tag/release-4.3.0
- 下载依赖：
python-simplejson-3.8.2-12.1.x86_64.rpm
saslwrapper-0.16-5.el7.x86_64.rpm
saslwrapper-devel-0.16-5.el7.x86_64.rpm
- 安装完毕Ambari集群，下载ambari-hue-service：http://github.com/snakecy/ambari-hue-service
git clone https://github.com/snakecy/ambari-hue-service.git
hue-release-4.3.0.zip
# hue编译环境
yum install -y ant asciidoc cyrus-sasl-devel cyrus-sasl-gssapi cyrus-sasl-plain gcc gcc-c++ krb5-devel libffi-devel libxml2-devel libxslt-devel make mysql mysql-devel openldap-devel python-devel sqlite-devel gmp-devel libtidy maven
yum install -y gcc openssl-devel
[root@manager241 tgm]# unzip hue-release-4.3.0.zip
[root@manager241 tgm]# mv hue-release-4.3.0 /opt/
[root@manager241 tgm]# cd /opt/
[root@manager241 opt]# mv hue-release-4.3.0 hue-4.3.0
[root@manager241 opt]# cd hue-4.3.0
[root@manager241 hue-4.3.0]# make apps
[root@manager241 opt]# touch hue-4.3.0/build/env/bin/replace_py_pwd.py
[root@manager241 opt]# cd hue-4.3.0/build/env/bin/
[root@manager241 bin]# vim replace_py_pwd.py
import os
import re
files = os.listdir('./')
def replaceStr(file):
with open(file, 'r') as f:
str = f.read()
str1 = re.sub("/opt/hue-4.3.0/build/env/bin/python2.7", "/usr/local/hue-4.3.0/build/env/bin/python2.7", str)
with open(file, 'w') as f:
f.write(str1)
for f in files:
replaceStr(f)
[root@manager241 bin]# chmod +x replace_py_pwd.py
[root@manager241 opt]# tar -zcvf hue-4.3.0.tgz hue-4.3.0/
[root@manager241 opt]# mkdir -p /var/www/html/hue
[root@manager241 opt]# cp hue-4.3.0.tgz /var/www/html/hue/
[root@manager241 opt]# cd /root/tgm/
[root@manager241 tgm]# cp python-simplejson-3.8.2-12.1.x86_64.rpm /var/www/html/hue/
[root@manager241 tgm]# cp saslwrapper-* /var/www/html/hue/
[root@manager241 tgm]# cd /var/www/html/hue/
[root@manager241 hue]# yum install createrepo -y
[root@manager241 hue]# createrepo /var/www/html/hue/
hue-4.3.0-stacks
[root@manager241 hue]# cd /var/lib/ambari-server/resources/stacks/HDP/3.0/services/
[root@manager241 services]# mkdir HUE
[root@manager241 services]# cd HUE/
[root@manager241 HUE]# touch metainfo.xml
[root@manager241 HUE]# chmod +x metainfo.xml
# metainfo.xml 内容如下:
<metainfo>
<schemaVersion>2.0schemaVersion>
<services>
<service>
<name>HUEname>
<extends>common-services/HUE/4.3.0extends>
<version>4.3.0version>
<osSpecifics>
<osSpecific>
<osFamily>anyosFamily>
<packages>
<package><name>wgetname>package>
<package><name>tarname>package>
<package><name>asciidocname>package>
<package><name>krb5-develname>package>
<package><name>libxml2-develname>package>
<package><name>libxslt-develname>package>
<package><name>openldap-develname>package>
<package><name>python-develname>package>
<package><name>python-setuptoolsname>package>
<package><name>python-psycopg2name>package>
<package><name>sqlite-develname>package>
<package><name>rsyncname>package>
<package><name>saslwrapper-develname>package>
<package><name>pycryptoname>package>
<package><name>gmp-develname>package>
<package><name>libyaml-develname>package>
<package><name>cyrus-sasl-plainname>package>
<package><name>cyrus-sasl-develname>package>
<package><name>cyrus-sasl-gssapiname>package>
<package><name>libffi-develname>package>
<package><name>MySQL-clientname>package>
packages>
osSpecific>
osSpecifics>
service>
services>
metainfo>
hue-4.3.0-common-services
[root@manager241 HUE]# cd /var/lib/ambari-server/resources/common-services/
[root@manager241 common-services]# mkdir HUE
[root@manager241 tgm]# cp -r ambari-hue-service/ /var/lib/ambari-server/resources/common-services/HUE/
[root@manager241 common-services]# cd HUE/
[root@manager241 HUE]# mv ambari-hue-service 4.3.0
[root@manager241 HUE]# chmod -R +x 4.3.0/
重启ambari服务-添加Hue服务
[root@manager241 ~]# ambari-server restart
[root@manager241 ~]# ambari-agent restart
Ambari页面-服务-添加服务-选中hue服务-点击下一步-配置数据库-进行部署
在部署的过程中出现各种问题，问题解决，参考后面“问题修复”
问题修复
问题1：hue_env_python = ’
- 解决：
vim /var/lib/ambari-server/resources/common-services/HUE/4.3.0/package/scripts/params.py
- 修改为：
- 或者download_url修改为：
import commands
download_url = commands.getoutput('cat /etc/yum.repo.d/hue.repo | grep "baseurl" | awk -F= \'{ print $2"hue.tgz"}\'')
问题2：Failed to execute command ‘/usr/bin/yum -y install saslwrapper-devel’, exited with code ‘1’, message: 'Error: Nothing to do
- 解决：
[root@manager241 ~]# cd /etc/yum.repos.d/
[root@manager241 yum.repos.d]# cp centos.repo hue.repo
[root@manager241 yum.repos.d]# vim hue.repo
[hue]
name=hue
baseurl=http://10.180.210.241/hue/
gpgcheck=0
enabled=1
[root@manager241 yum.repos.d]# scp hue.repo master241.bigdata:$PWD
[root@manager241 yum.repos.d]# scp hue.repo worker241.bigdata:$PWD
问题3：Execution of ‘http://10.180.210.241/hue/hue-4.3.0.tgz | xargs wget -O hue.tgz’ returned 123. /bin/bash: http://10.180.210.241/hue/hue-4.3.0.tgz: No such file or directory
- 解决：
vim /var/lib/ambari-server/resources/common-services/HUE/4.3.0/package/scripts/common.py
- 修改为：
问题4：Execution of ‘cd /usr/local/hue && make apps’ returned 1. /bin/bash: line 0: cd: /usr/local/hue: No such file or directory
- 解决：
vim /var/lib/ambari-server/resources/common-services/HUE/4.3.0/package/scripts/params.py
- 修改为：
问题5：/bin/bash: mvn: command not found
- 解决：问题原因，在hue节点执行make apps时需要maven环境，所以在hue的安装节点需要搭建hue编译的环境
# 搭建maven环境
参考：https://blog.csdn.net/Happy_Sunshine_Boy/article/details/91375219
# 安装环境
yum install -y ant asciidoc cyrus-sasl-devel cyrus-sasl-gssapi cyrus-sasl-plain gcc gcc-c++ krb5-devel libffi-devel libxml2-devel libxslt-devel make mysql mysql-devel openldap-devel python-devel sqlite-devel gmp-devel libtidy maven
问题6：/usr/local/hue目录错误
- 解决：
vim /var/lib/ambari-server/resources/common-services/HUE/4.3.0/package/scripts/common.py
- 修改为：
问题7：Execution of ‘/var/lib/ambari-agent/cache/common-services/HUE/4.3.0/package/files/configs.sh set manager241.bigdata HD45 hive-site ‘hive.security.authorization.sqlstd.confwhitelist.append’ ‘hive.server2.logging.operation.verbose’’ returned 1. [ERROR] “hive-site” not found in server response.
- 解决：遇到这种问题，安装相对应的组件（Hive、Oozie、Hbase、Spark2）
问题8：Execution of ‘cd /user/local/hue && source ./build/env/bin/activate’ returned 1. /bin/bash: line 0: cd: /user/local/hue: No such file or directory
- 解决：
vim /var/lib/ambari-server/resources/common-services/HUE/4.3.0/package/scripts/hue_server.py
- 修改为：
问题9：Configuration parameter ‘java_home’ was not found in configurations dictionary!
- 解决：
vim /var/lib/ambari-server/resources/common-services/HUE/4.3.0/package/scripts/params.py
- 修改为：
问题10：Invalid HTTP_HOST header: ‘10.180.210.242:8888’. You may need to add u’10.180.210.242’ to ALLOWED_HOSTS.
- 解决：
vim /usr/local/hue-4.3.0/desktop/core/src/desktop/settings.py
- 修改为：
问题11：Table ‘hue.auth_user’ doesn’t exist
- 解决：
登录ambari页面-点击服务选中hue-点击操作-点击Metastoresync，同步数据库
问题12：访问不存在的接口出现Debug错误信息，不显示404页面错误
vim /var/lib/ambari-server/resources/common-services/HUE/4.3.0/configuration/hue-desktop-site.xml
- Ambari页面临时解决：
- 服务-HUE-配置参数-ADVANCED-高级设置hue-desktop-site
Django Debug Mode设置为false - 服务-HUE-配置参数-ADVANCED-高级设置pseudo-distributed.ini
allowed_hosts={desktop_allowed_hosts}
修改为
allowed_hosts=*
- 另外如果后期永久修改需要在配置文件中修改解决：
- 修改为：
Hue集成Ambari完毕
Hue集成大数据组件
Hue-Hive
问题1：hue集成hive时获取hive_server_hosts错误
问题描述：
在hue操作hive的时候，获取不到hive的hiveserver2连接（hive_server_host,hive_server_port）
，手动修改配置文件/usr/local/hue/desktop/conf/pseudo-distributed.ini中有关hive的信息，重启hue时会重置配置文件，使手动配置的hiveserver2信息丢失。
解决：
vim /var/lib/ambari-agent/cache/common-services/HUE/4.3.0/package/scripts/params.py
修改为：
hive_server_hosts = default("/clusterHostInfo/hive_server_hosts", [])
hive_server_host = config['clusterHostInfo']['hive_server_hosts'][0]
Hue-Spark
问题1：hue集成spark_sql时报错获取sql_server_host,sql_server_port错误
解决：
vim /var/lib/ambari-server/resources/common-services/HUE/4.3.0/package/scripts/params.py
404 spark_thriftserver_hosts = default("/clusterHostInfo/spark_thriftserver_hosts", [])
405 spark_thriftserver_host = "localhost"
406 spark_hiveserver2_thrift_port = "10002"
407 spark_history_server_url = ''
408 if len(spark_thriftserver_hosts) > 0:
409 spark_thriftserver_host = spark_thriftserver_hosts[0]
410 spark_hiveserver2_thrift_port = str(config['configurations']['spark-hive-site-override']['hive.server2.thrift.port']).strip()
411 spark_jobhistoryserver_hosts = default("/clusterHostInfo/spark_jobhistoryserver_hosts", [])
412 if len(spark_jobhistoryserver_hosts) > 0:
413 spark_history_server_host = spark_jobhistoryserver_hosts[0]
414 else:
415 spark_history_server_host = "localhost"
416 spark_history_ui_port = config['configurations']['spark-defaults']['spark.history.ui.port']
417 spark_history_server_url = format("http://{spark_history_server_host}:{spark_history_ui_port}")
修改为：
403 # configurations of Spark
404 #spark_thriftserver_hosts = default("/clusterHostInfo/spark_thriftserver_hosts", [])
405 #spark_thriftserver_host = "localhost"
406 #spark_hiveserver2_thrift_port = "10002"
407 #spark_history_server_url = ''
408 #if len(spark_thriftserver_hosts) > 0:
409 # spark_thriftserver_host = spark_thriftserver_hosts[0]
410 # spark_hiveserver2_thrift_port = str(config['configurations']['spark-hive-site-override']['hive.server2.thrift.port']).strip()
411 # spark_jobhistoryserver_hosts = default("/clusterHostInfo/spark_jobhistoryserver_hosts", [])
412 # if len(spark_jobhistoryserver_hosts) > 0:
413 # spark_history_server_host = spark_jobhistoryserver_hosts[0]
414 # else:
415 # spark_history_server_host = "localhost"
416 # spark_history_ui_port = config['configurations']['spark-defaults']['spark.history.ui.port']
417 # spark_history_server_url = format("http://{spark_history_server_host}:{spark_history_ui_port}")
418 spark_thriftserver_host = default("/clusterHostInfo/spark2_thriftserver_hosts", [])[0]
419 spark_hiveserver2_thrift_port = str(config['configurations']['spark2-hive-site-override']['hive.server2.thrift.port']).strip()
420 spark_jobhistoryserver_host = default("/clusterHostInfo/spark2_jobhistoryserver_hosts", [])[0]
421 spark_history_ui_port = config['configurations']['spark2-defaults']['spark.history.ui.port']
422 spark_history_server_url = format("http://{spark_jobhistoryserver_host}:{spark_history_ui_port}")
Hue-Oozie
问题1：配置文件pseudo-distributed.ini oozie_url=http://localhost:11000/oozie 连接显示localhost，获取不到oozie服务节点
解决：
vim /var/lib/ambari-server/resources/common-services/HUE/4.3.0/package/scripts/params.py
vim /var/lib/ambari-agent/cache/common-services/HUE/4.3.0/package/scripts/params.py
oozie_servers_hosts = default("/clusterHostInfo/oozie_server", [])
修改为：
oozie_servers_hosts = default("/clusterHostInfo/oozie_server_hosts", [])
Hue-Zookeeper
问题1：配置文件pseudo-distributed.ini [zookeeper] [[clusters]] [[[default]]] host_ports=
vim /var/lib/ambari-server/resources/common-services/HUE/4.3.0/package/scripts/params.py
vim /var/lib/ambari-agent/cache/common-services/HUE/4.3.0/package/scripts/params.py
解决：
zookeeper_hosts = default("/clusterHostInfo/zookeeper_hosts", [])
修改为：
zookeeper_hosts = default("/clusterHostInfo/zookeeper_server_hosts", [])
代码参考：https://github.com/tgm2018/Hue4.3.0