HDP安装以及遇到的一些坑

1 关闭防火墙 systemctl stop firewalld 

systemctl disable firewalld

2 ssh 免密 

每个节点上 ssh-keygen 

然后每个节点 ssh-copy-id user@ip

expect -c "

        set timeout -1;

        spawn ssh-keygen

        expect {

                \"*Enter file*\" {send \"\r\"; exp_continue}

                \"*Enter passphrase*\" {send \"\r\"; exp_continue}

                \"*Enter same passphrase*\" {send \"\r\"; exp_continue}

                \"*The key fingerprint*\" {send \"\r\"; exp_continue}

        }"

 

3 安装ambari-server

ambari 默认使用pstgresql 可配置成mysql

安装mysql

后期可配置mysql 密码 以及允许root远程登录

alter user 'root'@'localhost' identified by 'your_password';

开启root 远程访问 GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456'

flush privileges;  

4 配置本地源 因为 如果在线安装的话，非常大，近10G

//开启http服务，以便其他机器可以通过网络下载具体目录为 /var/www/html

yum install httpd

systemctl start(enable) httpd

离线下载 （我一般选用最新版的）

在下面这个网址里

https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-installation/content/ambari_repositories.html

ambari  https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-installation/content/ambari_repositories.html

需要注意的是需要下载 tar.gz 包

 

HDP https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-installation/content/hdp_25_repositories.html

注意 需要下载 HDP 和HDP-utils 对应的linux版本

 

解压 HDP HDP-UTILS ambari 

cp * /var/www/html 

配置yum 源 

cat ambari.repo 

#VERSION_NUMBER=2.4.1.0-22

[Updates-ambari-2.7.3.0]

name=ambari-2.7.3.0

baseurl=http://ip/ambari/centos7/2.7.3.0-139

gpgcheck=1

gpgkey=http://ip/ambari/centos7/2.7.3.0-139/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

enabled=1

priority=1

/////////////////////////////////////////////////////////////////////////////////////////////////////

cat hdp.repo

[HDP-3.1.1.0]

name=HDP Version - HDP-3.1.0.0

baseurl=http://192.234324324/HDP/centos7/3.1.0.0-78

gpgcheck=1

gpgkey=http://192.164352345/HDP/centos7/3.1.0.0-78/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

enabled=1

priority=1

 

[HDP-UTILS-1.1.0.22]

name=HDP-UTILS Version - HDP-UTILS-1.1.0.22

baseurl=http://192.56343/HDP-UTILS/centos7/1.1.0.22

gpgcheck=1

gpgkey=http://56363/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

enabled=1

priority=1

//////////////////////////////////////////////////////////////////////////////////////////////////////

mv *repo /etc/yum.repo.d/

yum clean all

yum makecache 

yum install -y ambari-server

 

#进入数据库创建ambari库和导入数据
mysql>create database ambari DEFAULT CHARSET utf8;
mysql>use ambari
mysql>source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

 

b.配置ambari

[root@ambari-server hdp]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'permissive'
WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
OK to continue [y/n] (y)? y
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? y
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 1
To download the Oracle JDK and the Java Cryptography Extension (JCE) Policy Files you must accept the license terms found at http://www.or
acle.com/technetwork/java/javase/terms/license/index.html and not accepting will cancel the Ambari Server setup and you must install the JDK and JCE files manually.Do you accept the Oracle Binary Code License Agreement [y/n] (y)? y
Downloading JDK from http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u77-linux-x64.tar.gz to /var/lib/ambari-server/resources/jdk-8u77
-linux-x64.tar.gzjdk-8u77-linux-x64.tar.gz... 100% (173.0 MB of 173.0 MB)
Successfully downloaded JDK distribution to /var/lib/ambari-server/resources/jdk-8u77-linux-x64.tar.gz
Installing JDK to /usr/jdk64/
Successfully installed JDK to /usr/jdk64/
Downloading JCE Policy archive from http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-8.zip to /var/lib/ambari-server/resources/jc
e_policy-8.zip
Successfully downloaded JCE Policy archive to /var/lib/ambari-server/resources/jce_policy-8.zip
Installing JCE policy...
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (3): 3
Hostname (ambari-server): ambari-server
Port (3306): 
Database name (ambari): 
Username (root): root
Enter Database Password (mysqladmin): 
Configuring ambari database...
Copying JDBC drivers to server resources...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/r
esources/Ambari-DDL-MySQL-CREATE.sqlProceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
............
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
#启动服务
ambari-server start

 

ip：8080 admin admin

launch install wizard

use local repo 

配置 对应的url 即上面repo设置的地址

如 http://ip/ambari 没有端口 

ssh 密钥 id_isa 主机的/root/.ssh/id_isa

https://blog.51cto.com/kaliarch/2121505

接下来慢慢等待吧

因为它会报错！！！ 

有很多错，怪不得HDP用的人没有CDH多，虽然CDH也不是多么容易安装 ，基本都是每安装一次就会有一个新坑

 

//每台机器上都要有ambari-agent 

ambari-server只在一台机器上

 

1 Starting MySQL… ERROR! The server quit without updating PID file (/var/lib/mysql/server.vasco.com.pid)

解决办法   rm -rf /etc/my.cnf

2 ERROR 2018-12-27 16:59:24,797 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:618)

ERROR 2018-12-27 16:59:24,797 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions.

o resolve this issue, add the following property in ambari-agent.ini (/etc/ambari-agent/conf/ambari-agent.ini) file under [security]and restart ambari-agent:

解决办法：/etc/ambari-agent/conf/ambari- agent.ini

中添加

[security]

force_https_protocol=PROTOCOL_TLSv1_2

3 注册成功但是一直卡在Please wait while the hosts are being checked for potential problems这个问题上，解决方法是把系统的语言设置为英文。被运维给实力坑了一把 

文件位置：/etc/locale.conf

# localectl set-locale LANG=zh_CN.utf8

# localectl set-locale LANG=en_US.UTF-8

 

 

pyton 看最后一条就是问题所在

 

5 Ambari 错误 install-activity-analyzer.sh：command not found 

解决办法

重装smartsense

yum -y install smartsense-hst

 

//主要看日志  多个节点，如果一个安装好了，而其他的有错误做对比 最后一条错误日志

 

安装hive时需要提前设置

ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

 

hdp ambari version 
3.1.0.0-78  2.7.3.0-139

最后采用了 低版本的高版本的有意想不到的坑 ，所以以后一定不要装最新的软件，只装最稳定的！！！

ambari  HDP

2.6.1.5-3  2.6.4.0-91

完美安装成功 

遇到一些小问题：

id_isa 为私钥

hosts 文件的127.0.0.1 不能删除

kafka metrics  smartsense 等无法安装时  rpm -qa | grep ambari-metrics 删除相应的包，再重新安装

 

遇到软连接 symlink exist not a symlink ore broken link 

因为之前有过安装然后重新安装没有删除干净导致

主要是删除 /usr/hdp 目录 还有一些指定的目录配置项

yum remove hive\*
yum remove oozie\*
yum remove pig\*
yum remove zookeeper\*
yum remove tez\*
yum remove hbase\*
yum remove ranger\*
yum remove knox\*
yum remove storm\*
yum remove accumulo\*
yum remove falcon\*
yum remove ambari-metrics-hadoop-sink 
yum remove smartsense-hst
yum remove slider_2_4_2_0_258
yum remove ambari-metrics-monitor
yum remove spark2_2_5_3_0_37-yarn-shuffle
yum remove spark_2_5_3_0_37-yarn-shuffle
yum remove ambari-infra-solr-client
 

\*会自动匹配全部

 

ambari-server stop

ambari-agent stop

yum erase ambari-server

yum erase ambari-agent

 

1. rm -rf /etc/yum.repos.d/ambari.repo /etc/yum.repos.d/HDP*

2. yum clean all

 

rm -rf /var/log/ambari-agent
rm -rf /var/log/ambari-metrics-grafana
rm -rf /var/log/ambari-metrics-monitor
rm -rf /var/log/ambari-server/
rm -rf /var/log/falcon
rm -rf /var/log/flume
rm -rf /var/log/hadoop
rm -rf /var/log/hadoop-mapreduce
rm -rf /var/log/hadoop-yarn
rm -rf /var/log/hive
rm -rf /var/log/hive-hcatalog
rm -rf /var/log/hive2
rm -rf /var/log/hst
rm -rf /var/log/knox
rm -rf /var/log/oozie
rm -rf /var/log/solr
rm -rf /var/log/zookeeper
 

rm -rf /hadoop/*
rm -rf /hdfs/hadoop
rm -rf /hdfs/lost+found
rm -rf /hdfs/var
rm -rf /local/opt/hadoop
rm -rf /tmp/hadoop
rm -rf /usr/bin/hadoop
rm -rf /usr/hdp
rm -rf /var/hadoop

 

rm -rf /etc/ambari-agent
rm -rf /etc/ambari-metrics-grafana
rm -rf /etc/ambari-server
rm -rf /etc/ams-hbase
rm -rf /etc/falcon
rm -rf /etc/flume
rm -rf /etc/hadoop
rm -rf /etc/hadoop-httpfs
rm -rf /etc/hbase
rm -rf /etc/hive 
rm -rf /etc/hive-hcatalog
rm -rf /etc/hive-webhcat
rm -rf /etc/hive2
rm -rf /etc/hst
rm -rf /etc/knox 
rm -rf /etc/livy
rm -rf /etc/mahout 
rm -rf /etc/oozie
rm -rf /etc/phoenix
rm -rf /etc/pig 
rm -rf /etc/ranger-admin
rm -rf /etc/ranger-usersync
rm -rf /etc/spark2
rm -rf /etc/tez
rm -rf /etc/tez_hive2
rm -rf /etc/zookeeper
 

rm -rf /var/run/ambari-agent
rm -rf /var/run/ambari-metrics-grafana
rm -rf /var/run/ambari-server
rm -rf /var/run/falcon
rm -rf /var/run/flume
rm -rf /var/run/hadoop 
rm -rf /var/run/hadoop-mapreduce
rm -rf /var/run/hadoop-yarn
rm -rf /var/run/hbase
rm -rf /var/run/hive
rm -rf /var/run/hive-hcatalog
rm -rf /var/run/hive2
rm -rf /var/run/hst
rm -rf /var/run/knox
rm -rf /var/run/oozie 
rm -rf /var/run/webhcat
rm -rf /var/run/zookeeper

 

rm -rf /usr/lib/ambari-agent
rm -rf /usr/lib/ambari-infra-solr-client
rm -rf /usr/lib/ambari-metrics-hadoop-sink
rm -rf /usr/lib/ambari-metrics-kafka-sink
rm -rf /usr/lib/ambari-server-backups
rm -rf /usr/lib/ams-hbase
rm -rf /usr/lib/mysql
rm -rf /var/lib/ambari-agent
rm -rf /var/lib/ambari-metrics-grafana
rm -rf /var/lib/ambari-server
rm -rf /var/lib/flume
rm -rf /var/lib/hadoop-hdfs
rm -rf /var/lib/hadoop-mapreduce
rm -rf /var/lib/hadoop-yarn 
rm -rf /var/lib/hive2
rm -rf /var/lib/knox
rm -rf /var/lib/smartsense
rm -rf /var/lib/storm

删除配置的ambari的数据库 

reboot

https://blog.csdn.net/zsj777/article/details/80577712

查找是否还有hdp的安装参与

 yum list installed | grep HDP | awk '{print $1}' | sed '1,3d' | awk -F '_|-' '{print $1}'

yum -y remove xxx