Ambari2.7.4+HDP3.1.4在centos7.6部署

一.下载安装包

使用在线安装特别慢，所有的安装包加起来有9个G左右，所以需要先下载相关安装包到服务器，通过配置本地源的方式来实现的离线安装。

在私有集群中部署时，客户机房的网络情况不可预测，因此也需要使用离线部署的方式。

根据官方文档

https://docs.cloudera.com/HDPDocuments/Ambari-2.7.4.0/bk_ambari-installation/content/hdp_314_repositories.html

通过ambari安装需要下载下面的四个主要包：

1.ambari 2.7.4  （1.9G）    http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2. 7.4.0

2.HDP 3.1.4   (8.9G)  http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/HDP-3.1.4.0-centos7-rpm.tar.gz

3.HDP-UTILS-1.1.0.22  (87M)  http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz

4.HDP-GPL-3.1.4(158k) http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.1.4.0/HDP-GPL-3.1.4.0-centos7-gpl.tar.gz

二.修改本机名和/etc/hosts文件

2.1 修改主机名(所有机器)

#对master sudo hostnamectl set-hostname master #对slave1 sudo hostnamectl set-hostname slave1 #对slave2 sudo hostnamectl set-hostname slave2

2.2 修改/etc/hosts文件。(所有机器)

这里主要是为了可以实现通过名称来查找相应的服务器。

[root@master ~]# vi /etc/hosts 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6 172.16.130.202 master 172.16.130.203 slave1 172.16.130.204 slave2

2.3 修改/etc/sysconfig/network（所有机器）

[root@master ~]# vi /etc/sysconfig/network NETWORKING=yes

接下来就可以通过测试一下是否可以通过ping主机名来找到对应的服务器。

[root@master ~]# ping slave1 PING slave1 (172.16.130.203) 56(84) bytes of data. 64 bytes from slave1 (172.16.130.203): icmp_seq=1 ttl=64 time=0.300 ms 64 bytes from slave1 (172.16.130.203): icmp_seq=2 ttl=64 time=0.191 ms 64 bytes from slave1 (172.16.130.203): icmp_seq=3 ttl=64 time=0.198 ms 64 bytes from slave1 (172.16.130.203): icmp_seq=4 ttl=64 time=0.202 ms ^C --- slave1 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3000ms rtt min/avg/max/mdev = 0.191/0.222/0.300/0.048 ms

2.3 修改文件打开限制（所有机器）

[root@master ~]# vi /etc/security/limits.conf # End of file * soft nofile 65536 * hard nofile 65536 * soft nproc 131072 * hard nproc 131072

2.3 关闭防火墙（所有机器）

[root@master~]#systemctl disable firewalld [root@master~]#systemctl stop firewalld

然后

[root@master ~]# cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: #     enforcing - SELinux security policy is enforced. #     permissive - SELinux prints warnings instead of enforcing. #     disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of these two values: #     targeted - Targeted processes are protected, #     mls - Multi Level Security protection. SELINUXTYPE=targeted

三.同步时钟

3.1 安装ntp服务(所有机器)

[root@master ~]#yum -y install ntp

3.2 设置master为主服务器，开启nptd服务(主服务器)

[root@master ~]#vi /etc/ntp.conf

如图 restrict 改为集群所在网关及子网掩码。server和fudge按图中配置。

[root@master ~]# systemctl start ntpd.service       #启动服务 [root@master ~]# systemctl enable ntpd.service      #开机自启动

3.3 子节点设置同步(子节点)

主服务器开启ntp服务器以后，子节点就不需要开启了，因为当server与client之间的时间误差过大时（可能是1000秒），处于对修改时间可能对系统和应用带来不可预知的问题，NTP将停止时间同步！所以如果发现NTP启动之后时间并不进行同步时，应该考虑到可能是时间差过大引起的，此时需要先手动进行时间同步！所以直接使用定时手动同步的方式就可以了。

[root@slave1 data]# crontab -e 0-59/10 * * * * /usr/sbin/ntpdate master crontab: installing new crontab [root@slave1 data]# crontab -l 0-59/10 * * * * /usr/sbin/ntpdate master

四. SSH无密码登录(主节点)

配置master节点无密码登录到其他节点，在master节点上操作

[root@master ~]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Created directory '/root/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:NMVPXezqq+MPf95MtP5bzaghnvXSljcC4BH/QFCvm+I root@master The key's randomart image is: +---[RSA 2048]----+ |         o+. . o.| |         o..o . .| |        o +o . . | |       . + oo   .| |        S o.o  ..| |         . .o..+o| |         ..o=o.o=| |        ...+oB*+=| |         Eo.+**BO| +----[SHA256]-----+ [root@master ~]# ssh-copy-id slave1 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'slave1 (192.168.12.102)' can't be established. ECDSA key fingerprint is SHA256:GG4+viH+NDpVisSLMobnjEYOs171XRVUDoG85EI+mI8. ECDSA key fingerprint is MD5:f3:bf:63:a6:89:61:c2:08:3c:7d:2a:ed:13:be:d3:ae. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@slave1's password: Permission denied, please try again. root@slave1's password: Permission denied, please try again. root@slave1's password: Number of key(s) added: 1 Now try logging into the machine, with:   "ssh 'slave1'" and check to make sure that only the key(s) you wanted were added. [root@master ~]# ssh-copy-id slave2 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host 'slave2 (192.168.12.103)' can't be established. ECDSA key fingerprint is SHA256:Yv86oeVgTcxv8FCm5FLpiEUBTAnXgH2GOkgGboyoR+8. ECDSA key fingerprint is MD5:8c:ae:b6:c4:f1:77:1e:ab:a5:16:38:a2:d2:da:4c:af. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@slave2's password: Number of key(s) added: 1 Now try logging into the machine, with:   "ssh 'slave2'" and check to make sure that only the key(s) you wanted were added.

测试是否实现了无密码登录

[root@master hdp274]# ssh slave1 date ;ssh slave2 date;ssh master date; Thu Sep 19 19:44:37 CST 2019 Thu Sep 19 19:44:37 CST 2019 Thu Sep 19 19:44:39 CST 2019

将创建的秘钥拷贝出来，因为后面ambari安装的时候需要上传这个秘钥。创建秘钥是在隐藏文件夹/root/.ssh/下面的，所以需要先把秘钥拷贝到可见区域，然后拷贝到电脑上。下面名为id_rsa的文件就是秘钥文件。

[root@master ~]# cd /root/.ssh/ [root@master .ssh]# ls authorized_keys  id_rsa  id_rsa.pub  known_hosts [root@master .ssh]# cp id_rsa /root/ [root@master .ssh]# ls /root/ anaconda-ks.cfg   id_rsa  jdk-8u161-linux-x64.rpm

五.修改yum源，实现离线安装

5.1 安装httpd服务(主服务器)

[root@master ~]# yum -y install httpd [root@master ~]# service httpd restart Redirecting to /bin/systemctl restart httpd.service [root@master ~]# chkconfig httpd on

5.2 将上面下载的四个包放到/var/www/html目录下（主服务器）

[root@master hdp274]# ls ambari-2.7.4.0-centos7.tar.gz   HDP-GPL-3.1.4.0-centos7-gpl.tar.gz HDP-3.1.4.0-centos7-rpm.tar.gz  HDP-UTILS-1.1.0.22-centos7.tar.gz [root@master hdp274]# cp * /var/www/html/ambari/ [root@master hdp274]# cd /var/www/html/ambari/ [root@master ambari]# ls ambari-2.7.4.0-centos7.tar.gz   HDP-GPL-3.1.4.0-centos7-gpl.tar.gz HDP-3.1.4.0-centos7-rpm.tar.gz  HDP-UTILS-1.1.0.22-centos7.tar.gz

5.3 解压压缩包

[root@master ambari]# tar -zxvf ambari-2.7.4.0-centos7.tar.gz [root@master ambari]# tar -zxvf HDP-3.1.4.0-centos7-rpm.tar.gz [root@master ambari]# tar -zxvf HDP-GPL-3.1.4.0-centos7-gpl.tar.gz [root@master ambari]# mkdir HDP-UTILS-1.1.0.22 [root@master ambari]# tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz -C HDP-UTILS-1.1.0.22 [root@master ambari]# rm *.gz -rf

5.4 通过浏览器查看是否能够访问这个源了。

5.5 制作本地源

（1）安装本地源制作相关工具（主服务器）

[root@master ambari]# yum install yum-utils createrepo yum-plugin-priorities -y [root@master ambari]#  createrepo  ./

（2）修改文件里面的源地址（主服务器）

vi ambari/centos7/2.7.4.0-118/ambari.repo

[root@master ambari]# vi HDP-UTILS-1.1.0.22/HDP-UTILS/centos7/1.1.0.22/hdp-utils.repo

将修改好的文件拷贝到所有机器的/etc/yum.repos.d/路径下

[root@master ambari]# cp ambari/centos7/2.7.4.0-118/ambari.repo  /etc/yum.repos.d/ [root@master ambari]# scp /etc/yum.repos.d/ambari.repo slave1:/etc/yum.repos.d/ [root@master ambari]# scp /etc/yum.repos.d/ambari.repo slave2:/etc/yum.repos.d/ [root@master ambari]# cp HDP-UTILS-1.1.0.22/HDP-UTILS/centos7/1.1.0.22/hdp-utils.repo  /etc/yum.repos.d/ [root@master ambari]# scp /etc/yum.repos.d/hdp-utils.repo slave1:/etc/yum.repos.d/ [root@master ambari]# scp /etc/yum.repos.d/hdp-utils.repo slave2:/etc/yum.repos.d/

清除yum缓存

[root@master ambari]# yum clean all [root@master ambari]# yum makecache [root@master ambari]# yum repolist

六.安装ambari-server

6.1 安装和配置ambari-server

[root@master ~]# yum -y install ambari-server [root@master ~]# ambari-server setup Using python  /usr/bin/python Setup ambari-server Checking SELinux... SELinux status is 'enabled' SELinux mode is 'permissive' WARNING: SELinux is set to 'permissive' mode and temporarily disabled. OK to continue [y/n] (y)? y Customize user account for ambari-server daemon [y/n] (n)? y Enter user account for ambari-server daemon (root):root Adjusting ambari-server permissions and ownership... Checking firewall status... Checking JDK... [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Custom JDK ============================================================================== Enter choice (1): 1 JDK already exists, using /var/lib/ambari-server/resources/jdk-8u112-linux-x64.tar.gz Installing JDK to /usr/jdk64/ Successfully installed JDK to /usr/jdk64/ JCE Policy archive already exists, using /var/lib/ambari-server/resources/jce_policy-8.zip Installing JCE policy... Check JDK version for Ambari Server... JDK version found: 8 Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server. Checking GPL software agreement... GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y Completing setup... Configuring database... Enter advanced database configuration [y/n] (n)? y Configuring database... ============================================================================== Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL / MariaDB [4] - PostgreSQL [5] - Microsoft SQL Server (Tech Preview) [6] - SQL Anywhere [7] - BDB ============================================================================== Enter choice (1): 1 Database admin user (postgres): admin Database name (ambari): admi^H^H Postgres schema (ambari): ^C Aborting ... Keyboard Interrupt. [root@master ~]# ambari-server setup Using python  /usr/bin/python Setup ambari-server Checking SELinux... SELinux status is 'enabled' SELinux mode is 'permissive' WARNING: SELinux is set to 'permissive' mode and temporarily disabled. OK to continue [y/n] (y)? y Customize user account for ambari-server daemon [y/n] (n)? y Enter user account for ambari-server daemon (root):root Adjusting ambari-server permissions and ownership... Checking firewall status... Checking JDK... Do you want to change Oracle JDK [y/n] (n)? y [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Custom JDK ============================================================================== Enter choice (1): 1 JDK already exists, using /var/lib/ambari-server/resources/jdk-8u112-linux-x64.tar.gz Installing JDK to /usr/jdk64/ Successfully installed JDK to /usr/jdk64/ JCE Policy archive already exists, using /var/lib/ambari-server/resources/jce_policy-8.zip Installing JCE policy... Check JDK version for Ambari Server... JDK version found: 8 Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server. Checking GPL software agreement... Completing setup... Configuring database... Enter advanced database configuration [y/n] (n)? n Configuring database... Default properties detected. Using built-in database. Configuring ambari database... Checking PostgreSQL... Running initdb: This may take up to a minute. About to start PostgreSQL Configuring local database... Configuring PostgreSQL... Backup for pg_hba found, reconfiguration not required Creating schema and user... done. Creating tables... done. Extracting system views... ambari-admin-2.7.4.0.118.jar .... Ambari repo file contains latest json url http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks repoinfos with it... Adjusting ambari-server permissions and ownership... Ambari Server 'setup' completed successfully.

6.2 启动ambari

[root@master ~]# ambari-server start

成功启动后在浏览器输入Ambari地址：

http://master:8080

七.设置Use Local Repository

配置成本地源

http://master/ambari/HDP/centos7/3.1.4.0-315

http://master/ambari/HDP-GPL/centos7/3.1.4.0-315

http://master/ambari/HDP-UTILS-1.1.0.22/HDP-UTILS/centos7/1.1.0.22

注意:需要修改本机hosts文件

172.16.130.202 master 172.16.130.203 slave1 172.16.130.204 slave2

八.指定要安装集群的主机列表和之前的id_rsa文件

九.安装hive前需要下载jdbc驱动

根据下面的问题单上的描述，得知ambari安装hive时的说明有误。https://issues.apache.org/jira/browse/AMBARI-23355

[root@fidcmpt01 hdp]# ls mysql-connector-java-8.0.17.jar [root@fidcmpt01 hdp]# pwd /opt/hdp [root@fidcmpt01 hdp]# ambari-server setup --jdbc-db=mysql --jdbc-driver=/opt/hdp/mysql-connector-java-8.0.17.jar

附录一:卸载集群

1）停止Ambari中的所有服务

2）在所有集群节点上运行python脚本

[root@master var]# python /usr/lib/python2.6/site-packages/ambari_agent/HostCleanup.py –silent –skip=users

3）停止ambari-server和所有机器上的ambari-agent,并卸载

[root@master var]# ambari-server stop [root@master var]# ambari-agent stop [root@master opt]# yum erase ambari-server [root@master opt]# yum erase ambari-agent [root@master ~]# yum erase postgresql [root@master opt]# rm -rf /etc/ambari-server [root@master opt]# rm -rf /usr/lib/python2.6/site-packages/ambari_server [root@master opt]# rm -rf /etc/ambari-agent/ [root@master opt]# rm -rf /usr/lib/python2.6/site-packages/ambari_agent [root@master opt]# rm -rf /usr/lib/python2.6/site-packages/resource_monitoring

3）删除数据库。 删除MySQL和Postgres的实例，以便Ambari安装和配置新的数据库。

yum remove mysql mysql-server yum erase postgresql rm -rf /var/lib/pgsql rm -rf /var/lib/mysql

4）删除数据库。 删除MySQL和Postgres的实例，以便Ambari安装和配置新的数据库。

附录二.出错后重装ambari-server

如果出现错误，请注意查看日志，根据具体的错误内容进行处理，默认ambari-server的日志在/var/log/ambari-server/ambari-server.log里面。如果在处理日志的过程中或者后面安装的过程中出现一些莫名的错误，可以重置的安装。如果上面进行的默认数据库的配置，可以使用下面的代码重置ambari-server

[root@master ~]# ambari-server stop [root@master ~]# ambari-server reset [root@master ~]# ambari-server setup

如果在安装的过程中出现了错误，又想重新安装，可以在ambari-server开启的情况下，执行下面的语句来移除已安装的包，然后再通过不同的情况选择上面两种方式的一种对ambari-server进行重置

python /usr/lib/python2.6/site-packages/ambari_agent/HostCleanup.py --silent

附录三. YARN和MAPREDUCE2的服务删除

由于YARN和MAPREDUCE2这两个服务在ambari中是相互依赖的，无法在界面上删除，可以使用接口方式。

curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://localhost:8080/api/v1/clusters/hadoop2/services/YARN