HDP安装部署流程及遇到的问题(鲲鹏ARM64)

1.检查操作系统版本。

CentOS7.6
cat /etc/redhat-release

openEuler20.03
cat /etc/openEuler-release

说明:
对于TaiShan服务器支持的操作系统为CentOS 7.6/openEuler20.03。
2.检查操作系统默认语言。

echo $LANG

若操作系统默认语言为en_US.UTF-8,则进入3。
若操作系统默认语言非en_US.UTF-8,则需要切换成英文。
将默认语言由中文切换为英文。
echo ‘export LANG=en_US.UTF-8’>> ~/.bashrc

将当前会话关闭后重新登录。
3.设置操作系统时区。

操作系统时区统一设置为(Asia/Shanghai)。

cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

说明:

如果安装OS时已经将时区设置为Asia/Shanghai,则3可忽略。

设置域名解析
集群所有主机都必须配置域名解析,才能实现Server对Agent的顺利访问。建议由系统管理员统一配置,配置方法如下:
1.在每台主机上设置相应的主机名。

编辑网络配置文件。
vi /etc/sysconfig/network

设置文件内容为:

NETWORKING=yes
HOSTNAME= {主机名}

设置主机名(假设主机名为server1,agent1,agent2,agent3)。
hostnamectl --static set-hostname {主机名}

2.配置“/etc/hosts”。

为集群中每一台主机添加一行包含主机IP地址与完整主机名的信息。
vi /etc/hosts

例如,步骤中设置的主机名为server1,agent1,agent2,agent3:

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6IP1 server1IP2 agent1IP3 agent2IP4 agent3

可以在一台主机上配置,然后同步“/etc/hosts”到集群中所有节点上。
scp /etc/hosts root@{目标ip}:/etc/hosts

重启所有节点网络服务。
说明:

CentOS中执行:
systemctl restart network

openeuler中执行:
systemctl restart NetworkManager

3.验证。
返回主机IP地址。
hostname -i

返回主机名称。
hostname -f

关闭防火墙
说明:
Ambari部署Hadoop集群时需要保证Ambari Server能够连接到Ambari Agent。
进行安装任务,集群所有主机都必须关闭防火墙。
systemctl stop firewalld
systemctl disable firewalld
关闭SElinux和配置limits参数
1.更改SELinux为宽容模式。

setenforce 0

2.编辑“/etc/selinux/config”文件,永久禁止SELinux自动启动。

vi /etc/selinux/config

设置SELINUX=disabled,如下所示:

This file controls the state of SELinux on the system.
SELINUX= can take one of these three values:
enforcing - SELinux security policy is enforced.
permissive - SELinux prints warnings instead of enforcing.
disabled - No SELinux policy is loaded.
SELINUX=disabled
SELINUXTYPE= can take one of three values:
targeted - Targeted processes are protected,
minimum - Modification of targeted policy. Only selected processes are protected.
mls - Multi Level Security protection.
SELINUXTYPE=targeted

3.编辑“/etc/security/limits.conf”文件。

vi /etc/security/limits.conf

添加如下最后四行的内容:

#* soft core 0
#* hard rss 10000
#@student hard nproc 20
#@faculty soft nproc 20
#@faculty hard nproc 50
#ftp hard nproc 0
#@student - maxlogins 4

#End of file

  • hard nproc 65535
  • soft nproc 65535
  • hard nofile 65535
  • soft nofile 65535
    禁用交换分区
    在群内所有主机禁用交换分区。
    sysctl vm.swappiness=0echo vm.swappiness=0 >> /etc/sysctl.conf
    设置umask
    为集群所有主机设置umask,设置用户创建目录默认权限。不同的系统采用不同的操作。
    CentOS 7.6(鲲鹏/x86)
    CentOS 7.6操作系统。
    umask 0022echo umask 0022 >> /etc/profilesource /etc/profile
    设置节点互信
    Ambari Server与Ambari Agent服务器之间需要配置ssh无密访问,Ambari Server通过SSH公钥授权远程登录其他主机。
    配置方法如下。
    1.产生ssh密钥。

在集群每台主机产生SSH认证密钥对。
ssh-keygen -t rsa

弹出对话框直接回车即可。
[root@server1 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:FscQ5D3nOcVjXY7T8wdVo+R51tvnmbVXyhi5kNkqV8Y root@server1
The key’s randomart image is:
±–[RSA 2048]----+
| .+. . .=|
| . + o.o*+|
| o = .+O+=|
| o O =o+=|
| S + E .B|
| . = * oO|
| . o o o+o|
| o .|
| |
±—[SHA256]-----+

配置ssh免密登录。

将Ambari Server生成的公钥文件id_rsa.pub拷贝至集群所有主机(包含Ambari Server本身)。

在Ambari Server主机执行:

cd ~/.ssh/
ssh-copy-id -i ~/.ssh/id_rsa.pub root@{target_host}

说明:

{target_host}包括hadoop集群的所有主机,可以是主机名或IP地址。
该命令完成后会在{target_ost}的~/.ssh/目录下生成文件authorized_keys。

2.所有主机设置文件目录权限。

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

3.Ambari Sever主机分别免密码登录检查。

ssh {tarqet.host}

配置OS镜像源

集群中每台服务器均需配置本地镜像源。
1.进入存放OS镜像文件的目录,挂载镜像源。

mount YOUR_OS.iso /media/ -o loop
(这里直接上传一个iso文件,并在iso目录执行此行命令)
说明:

YOUR_OS.iso用实际的iso包名代替。例如:

CentOS7.6(鲲鹏)
CentOS-7-aarch64-Everything-1810.iso
CentOS7.6(x86)
CentOS-7-x86_64-DVD-1810.iso

2.配置本地源。

备份repo。
cd /etc/yum.repos.d
mkdir -p /etc/yum.repos.d/bak
mv *.repo bak
创建并编辑local.repo文件。
vi local.repo
在local.repo文件中添加如下内容:
[local]
name=local repo
baseurl=file:///media
enabled=1
gpgcheck=0
3.生效本地源。
yum clean all
yum makecache
安装依赖软件

CentOS7.6(鲲鹏/x86):
安装"Infrastructure Server"和"Development Tools"。
yum groupinstall “Infrastructure Server” --setopt=group_package_types=mandatory,default,optional -y
yum groupinstall “Development Tools” --setopt=group_package_types=mandatory,default,optional -y

安装JDK
说明:
集群内所有节点均需安装JDK,如果需要开启kerberos,还需jce安装包。
下载安装包(根据具体环境选择下载)
x86环境OpenJDK下载地址:
https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u222-b10/OpenJDK8U-jdk_x64_linux_hotspot_8u222b10.tar.gz

鲲鹏环境OpenJDK下载地址:
https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u222-b10/OpenJDK8U-jdk_aarch64_linux_hotspot_8u222b10.tar.gz

鲲鹏环境毕昇JDK下载地址:
https://www.huaweicloud.com/kunpeng/software/bishengjdk.html

说明:
在AdoptOpenJDK官网下载最新版的对应平台的OpenJDK,推荐使用OpenJDK 8u272及以上版本或毕昇JDK。
x86服务器以OpenJDK8U-jdk_x64_linux_hotspot_8u222b10.tar为例,TaiShan服务器以jdk-8u222-linux-aarch64.tar.gz为例。
安装JDK
wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u222-b10/OpenJDK8U-jdk_aarch64_linux_hotspot_8u222b10.tar.gz

1.下载OpenJDK。

2.解压及修改权限。

tar -zxvf OpenJDK8U-jdk_aarch64_linux_hotspot_8u222b10.tar.gz
chown -R root:root jdk8u222-b10/
chmod -R 755 jdk8u222-b10/

3.移动到“/usr/local”目录下。

mv jdk8u222-b10/ /usr/local/

4.打开环境变量的配置文件。

vi /etc/profile

5.添加下面语句至文件末尾处。
export JAVA_HOME=/usr/local/jdk8u222-b10
export PATH= J A V A H O M E / b i n : JAVA_HOME/bin: JAVAHOME/bin:PATH
6.使环境变量生效。
source /etc/profile

警告:
JAVA_HOME当前配置为系统默认配置的Java版本,请根据实际情况调整,并确认配置目录可以被所有用户访问到。
搭建Yum源服务器
下载rpm包
说明:
以防遗漏,请全部下载。
鲲鹏(CentOS 7.6)rpm下载地址:
https://mirrors.huaweicloud.com/kunpeng/yum/el/7/bigdata/HDP-UTILS-1.1.0.22/repos/HDP-UTILS-1.1.0.22-centos7.tar.gz

https://mirrors.huaweicloud.com/kunpeng/yum/el/7/bigdata/HDP-GPL/3.x/updates/3.1.0.0/HDP-GPL-3.1.0.0-centos7-gpl.tar.gz

https://mirrors.huaweicloud.com/kunpeng/yum/el/7/bigdata/ambari/2.x/updates/2.7.3.0/ambari-2.7.3.0-centos7.tar.gz

https://mirrors.huaweicloud.com/kunpeng/yum/el/7/bigdata/HDP/3.x/updates/3.1.0.0/HDP-3.1.0.0-centos7-rpm.tar.gz

安装httpd组件
1.安装httpd和createrepo。
yum install httpd createrepo

2.开启httpd服务。
service httpd start
3.验证。
systemctl status httpd.service
若状态为running,则安装并启动成功:

● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
Active: active (running) since Mon 2019-10-14 16:22:09 CST; 18h ago
Docs: man:httpd(8)
man:apachectl(8)
Main PID: 22995 (httpd)
Status: “Total requests: 34; Current requests/sec: 0; Current traffic: 0 B/sec”
CGroup: /system.slice/httpd.service
├─22995 /usr/sbin/httpd -DFOREGROUND
├─22996 /usr/sbin/httpd -DFOREGROUND
├─22997 /usr/sbin/httpd -DFOREGROUND
├─22998 /usr/sbin/httpd -DFOREGROUND
├─22999 /usr/sbin/httpd -DFOREGROUND
├─23000 /usr/sbin/httpd -DFOREGROUND
├─23015 /usr/sbin/httpd -DFOREGROUND
├─25332 /usr/sbin/httpd -DFOREGROUND
└─46379 /usr/sbin/httpd -DFOREGROUND

Oct 14 16:22:09 DataNode-1 systemd[1]: Starting The Apache HTTP Server…
Oct 14 16:22:09 DataNode-1 httpd[22995]: AH00558: httpd: Could not reliably determine the server’s fully qualified domain name, using xxx.xxx.xxx.xxx. Set the ‘ServerName’ d…this message
Oct 14 16:22:09 DataNode-1 systemd[1]: Started The Apache HTTP Server.
Hint: Some lines were ellipsized, use -l to show in full.
管理rpm包
说明:
以鲲鹏(CentOS 7.6)场景为例进行说明,仅供参考,其它场景请酌情调整。
Ambari rpm包:
分别创建如下目录,管理ambari的rpm包。

mkdir -p /var/www/html/entry/ambari/2.7.3.0.0/centos7/aarch64

将之前下载的rpm包进行分类,鲲鹏计算平台的包放到aarch64目录(解压)
aarch64
│ ├── ambari-agent-2.7.3.0-0.aarch64.rpm
│ ├── ambari-infra-manager-2.7.3.0-0.aarch64.rpm
│ ├── ambari-infra-solr-2.7.3.0-0.aarch64.rpm
│ ├── ambari-infra-solr-client-2.7.3.0-0.aarch64.rpm
│ ├── ambari-logsearch-logfeeder-2.7.3.0-0.aarch64.rpm
│ ├── ambari-logsearch-portal-2.7.3.0-0.aarch64.rpm
│ ├── ambari-metrics-collector-2.7.3.0-0.aarch64.rpm
│ ├── ambari-metrics-grafana-2.7.3.0-0.aarch64.rpm
│ ├── ambari-metrics-hadoop-sink-2.7.3.0-0.aarch64.rpm
│ ├── ambari-metrics-monitor-2.7.3.0-0.aarch64.rpm
│ ├── ambari-server-2.7.3.0-0.aarch64.rpm
│ └── smartsense-hst-1.5.1.2.7.3.0-1.aarch64.rpm

HDP rpm包:
分别创建如下目录,管理hdp的rpm包。
mkdir -p /var/www/html/entry/hdp/3.1/centos7/aarch64
其中Arm的包放到aarch64目录
aarch64
│ ├── atlas-metadata_3_1_0_0_78-1.1.0.3.1.0.0-78.aarch64.rpm
│ ├── bigtop-jsvc-1.0.15-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-client-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-conf-pseudo-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-hdfs-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-hdfs-datanode-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-hdfs-fuse-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-hdfs-journalnode-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-hdfs-namenode-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-hdfs-secondarynamenode-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-hdfs-zkfc-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-httpfs-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-httpfs-server-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-libhdfs-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-mapreduce-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-mapreduce-historyserver-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-source-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-yarn-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-yarn-nodemanager-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-yarn-proxyserver-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-yarn-registrydns-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-yarn-resourcemanager-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-yarn-timelinereader-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hadoop_3_1_0_0_78-yarn-timelineserver-3.1.1.3.1.0.0-78.aarch64.rpm
│ ├── hbase_3_1_0_0_78-2.0.2.3.1.0.0-78.aarch64.rpm
│ ├── hive_3_1_0_0_78-3.1.0.3.1.0.0-78.aarch64.rpm
│ ├── hive_3_1_0_0_78-jdbc-3.1.0.3.1.0.0-78.aarch64.rpm
│ ├── livy2_3_1_0_0_78-0.5.0.3.1.0.0-78.aarch64.rpm
│ ├── phoenix_3_1_0_0_78-5.0.0.3.1.0.0-78.aarch64.rpm
│ ├── ranger_3_1_0_0_78-hbase-plugin-1.2.0.3.1.0.0-78.aarch64.rpm
│ ├── ranger_3_1_0_0_78-hdfs-plugin-1.2.0.3.1.0.0-78.aarch64.rpm
│ ├── ranger_3_1_0_0_78-hive-plugin-1.2.0.3.1.0.0-78.aarch64.rpm
│ ├── ranger_3_1_0_0_78-yarn-plugin-1.2.0.3.1.0.0-78.aarch64.rpm
│ ├── spark2_3_1_0_0_78-2.3.2.3.1.0.0-78.aarch64.rpm
│ ├── spark2_3_1_0_0_78-yarn-shuffle-2.3.2.3.1.0.0-78.aarch64.rpm
│ └── tez_3_1_0_0_78-0.9.1.3.1.0.0-78.aarch64.rpm

创建Yum源仓库
1.使用createrepo创建Yum源仓库。
cd /var/www/html/entry/ambari/2.7.3.0.0/centos7 && createrepo .cd /var/www/html/entry/hdp/3.1/centos7 && createrepo .

须知:

createrepo后需要加路径,上述命令最后的“.”,表示当前路径,不可缺少,否则会报错。
创建完成后将在上述两个目录分别生成repodata文件夹。
[root@Management-1 centos7]# ll
total 12
drwxr-xr-x 2 root root 4096 Oct 15 16:34 aarch64
drwxr-xr-x 2 root root 6 Oct 15 16:34 noarch
drwxr-xr-x 2 root root 4096 Oct 15 19:01 repodata
drwxr-xr-x 2 root root 4096 Oct 15 16:37 x86_64

2.配置repo文件。

说明:
该步骤需在集群中每台服务器上执行。

创建并编辑ambari.repo文件。
cd /etc/yum.repos.d
vi ambari.repo

在ambari.repo文件中添加如下内容:

[ambari]
name=ambari repo
baseurl=http://{Yum源服务器ip}/entry/ambari/2.7.3.0.0/centos7
enabled=1
gpgcheck=0

创建并编辑hdp.repo文件。
vi hdp.repo

在hdp.repo文件中添加如下内容:

[hdp]
name=hdp repo
baseurl=http://{Yum源服务器ip}/entry/hdp/3.1/centos7
enabled=1
gpgcheck=0

说明:
baseurl可以在浏览器中验证是否能够正常访问。(把大括号删除)
3.生效源地址。

yum clean all
yum makecache

须知:

文档中有将rpm包分别管理,如Taishan的包放到aarch64目录,noarch目录下放公共的包,在实际部署时,将RPM包统一放在“/var/www/html/entry/ambari/2.7.3.0.0/centos7/aarch64”目录。(但是在我的部署过程中,放到这个目录下依旧扫不到包,可直接放到/var/www/html/entry/hdp/3.1/centos7/aarch64下)
在“/etc/yum.repod.d”中创建ambari.repo时,需要将url路径写到上一层目录,即:“baseurl=http://ip地址/entry/ambari/2.7.3.0.0/centos7/”。
注意baseurl字符均为小写,否则生效Yum源时会报错。

安装数据库
安装数据库服务。
说明:
在集群中server服务器上执行即可。
1.安装mariadb及mysql相关服务。
yum install mariadb mariadb-server.aarch64 mysql-connector-java.noarch -y
2.
下载mysql-connector-java.jar,并放置到“/usr/share/java”目录下(jar包版本可酌情调整):
mkdir -p /usr/share/java
wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.49/mysql-connector-java-5.1.49.jar -O /usr/share/java/mysql-connector-java.jar
启动并设置mariadb为开机启动。
systemctl start mariadb
systemctl enable mariadb

启动状态查询。
systemctl status mariadb.service

状态为running则启动成功:

[root@server1 etc]# systemctl status mariadb.service
● mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-10-16 17:21:14 CST; 4 days ago
Main PID: 6191 (mysqld_safe)
CGroup: /system.slice/mariadb.service
├─6191 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
└─6353 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/ma…

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
1.登录并修改默认密码。
mysql -uroot -p
默认没有密码,按回车即可登录。
[root@server1 yum.repos.d]# mysql -uroot -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 5.5.60-MariaDB MariaDB Server

Copyright © 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

MariaDB [(none)]>
2.设置密码。
set password = password(‘密码’);flush privileges;
MariaDB [(none)]> set password = password(‘root’);
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.00 sec)
须知:
flush privileges必须执行,否则设置不生效。
3.验证密码。
退出后重新登录,查看密码是否修改成功。
MariaDB [(none)]> exit;
Bye
[root@server1 yum.repos.d]# mysql -uroot -p
Enter password:root
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 4
Server version: 5.5.60-MariaDB MariaDB Server

Copyright © 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

MariaDB [(none)]>
2.创建Ambari数据库并配置权限。(这里要创建一个本机名的USER如:
MariaDB [ambari]> CREATE USER ‘ambari’@‘本机名’ IDENTIFIED BY ‘ambari123’;
Query OK, 0 rows affected (0.01 sec)

MariaDB [ambari]> GRANT ALL PRIVILEGES on . To ‘ambari’@‘本机名’ IDENTIFIED BY ‘ambari123’;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> create database ambari;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> use ambari;
Database changed
MariaDB [ambari]> CREATE USER ‘ambari’@’%’ IDENTIFIED BY ‘ambari123’;
Query OK, 0 rows affected (0.00 sec)

MariaDB [ambari]> GRANT ALL PRIVILEGES on . To ‘ambari’@’%’ IDENTIFIED BY ‘ambari123’;
Query OK, 0 rows affected (0.00 sec)

MariaDB [ambari]> CREATE USER ‘ambari’@‘server1’ IDENTIFIED BY ‘ambari123’;
Query OK, 0 rows affected (0.01 sec)

MariaDB [ambari]> GRANT ALL PRIVILEGES on . To ‘ambari’@‘server1’ IDENTIFIED BY ‘ambari123’;
Query OK, 0 rows affected (0.00 sec)

MariaDB [ambari]> CREATE USER ‘ambari’@‘agent1’ IDENTIFIED BY ‘ambari123’;
Query OK, 0 rows affected (0.00 sec)

MariaDB [ambari]> GRANT ALL PRIVILEGES on . To ‘ambari’@‘agent1’ IDENTIFIED BY ‘ambari123’;
Query OK, 0 rows affected (0.00 sec)

MariaDB [ambari]> CREATE USER ‘ambari’@‘agent2’ IDENTIFIED BY ‘ambari123’;
Query OK, 0 rows affected (0.00 sec)

MariaDB [ambari]> GRANT ALL PRIVILEGES on . To ‘ambari’@‘agent2’ IDENTIFIED BY ‘ambari123’;
Query OK, 0 rows affected (0.00 sec)

MariaDB [ambari]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
说明:
根据实际情况进行修改,需要对集群中每一台服务器创建用户。
部署Ambari-server
下载安装Ambari-server
root用户执行:
yum install ambari-server.aarch64 -y
初始化Ambari数据库连接配置
1.在集群中所有服务器上切换回root用户添加环境变量。
vim /etc/profile
2.
添加内容如下:
export buildNumber=2.7.3.0.0
3.生效环境变量。
source /etc/profile

4.初始化Ambari数据库连接配置。

ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
Using python /usr/bin/python
Setup ambari-server
Copying /usr/share/java/mysql-connector-java.jar to /var/lib/ambari-server/resources/mysql-connector-java.jar
If you are updating existing jdbc driver jar for mysql with mysql-connector-java.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts.
JDBC driver was successfully initialized.
Ambari Server ‘setup’ completed successfully.

Ambari-server安装配置
1.安装Ambari-server。(根据以下提示一个一个选,)
ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux…
SELinux status is ‘enabled’
SELinux mode is ‘permissive’
WARNING: SELinux is set to ‘permissive’ mode and temporarily disabled.
OK to continue [y/n] (y)? y
Customize user account for ambari-server daemon [y/n] (n)? nEnter user account for ambari-server daemon (root):bigdata
Adjusting ambari-server permissions and ownership…
Checking firewall status…
Checking JDK…
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK

Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/local/jdk8u222-b10
Validating JDK on Ambari Server…done.
Check JDK version for Ambari Server…
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.

选择Custom JDK,路径以上面步骤中实际的路径为准。

Checking GPL software agreement…
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y
Completing setup…
Configuring database…
Enter advanced database configuration [y/n] (n)? y
Configuring database…

Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB

Enter choice (1):3
Hostname (localhost): 本机名
Port (3306):
Database name (ambari):
Username (ambari):
Enter Database Password (bigdata):ambari123
Re-enter password:
Configuring ambari database…
参数Enter Database Password(bigdata)输入安装数据库中设置的密码。
根据实际情况输入后回车。
Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y
Configuring remote database connection properties…
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views…
ambari-admin-2.7.3.0.0.jar

Ambari repo file doesn’t contain latest json url, skipping repoinfos modification
Adjusting ambari-server permissions and ownership…
Ambari Server ‘setup’ completed successfully.

初始化Ambari数据库
1.登录mysql。
mysql -uroot -p
MariaDB [(none)]> use ambari;
Database changed
MariaDB [ambari]> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
2.验证脚本是否初始化成功,出现以下table列表:
MariaDB [ambari]> show tables;
±------------------------------+
| Tables_in_ambari |
±------------------------------+
| ClusterHostMapping |
| QRTZ_BLOB_TRIGGERS |
| QRTZ_CALENDARS |
| QRTZ_CRON_TRIGGERS |
| QRTZ_FIRED_TRIGGERS |
| QRTZ_JOB_DETAILS |
| QRTZ_LOCKS |
| QRTZ_PAUSED_TRIGGER_GRPS |
| QRTZ_SCHEDULER_STATE |
| QRTZ_SIMPLE_TRIGGERS |
| QRTZ_SIMPROP_TRIGGERS |
| QRTZ_TRIGGERS |
| adminpermission |

MariaDB [(none)]> exit;
Bye
启动Ambari服务
ambari-server start
[root@server1 yum.repos.d]# ambari-server start
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources…
Ambari database consistency check started…
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start…
Server started listening on 8080

DB configs consistency check: no errors and warnings were found.
Ambari Server ‘start’ completed successfully.
查看Ambari server状态。
ambari-server status
[root@Namenode1 centos7]# ambari-server status
Using python /usr/bin/python
Ambari-server status
Ambari Server running
Found Ambari Server PID: 18130 at: /var/run/ambari-server/ambari-server.pid
(自选)停止Ambari server服务。
ambari-server stop
登录Ambari界面
启动ambari-server后,通过客户端浏览器登录Ambari的界面。
浏览器网址:http://:8080。
默认登录用户名/密码为:admin/admin。
启动Ambari部署向导
1.单击“Launch Install Wizard”搭建集群。
2.输入集群的名字,不能使用空格和其他特殊字符。
例如:“cluster1”。
3.单击“下一步”,进入选择版本。
选择版本
1.将除了redhat7之外的os选项通过右边的remove删除。
2.在redhat7对应的Base URL中填入hdp.repo文件中的地址,这里为http://{Yum源服务器ip}/entry/hdp/3.1/centos7。(这里事先把所有HDP包解压到这个目录下,验证:通过url可以看到所有包)
3.单击“下一步”,进入安装选项。

安装选项
1.在目标主机中填入集群中所有服务器的主机名。
2.填入server主机的ssh密钥。
通过命令获取。
cat ~/.ssh/id_rsa
3.单击“REGISTER AND CONFIRM”,进入确认主机。
确认主机
1.安装ambari-agent,并检测主机,保证检查结果没问题。
2.单击“下一步”,进入部署基础服务。
部署基础服务
说明:
没有特别说明直接单击下一步即可。
1.选择服务,单击“下一步”。
2.先安装HDFS、ZooKeeper、Ambari Metrics。
(由于之后安装比较自主,所以自行百度)以上可参考链接:
https://support.huaweicloud.com/dpmg-hdp-kunpengbds/kunpengambarihdp_04_0018.html
(我的安装部署均参考的此链接,其中有许多注意事项已经我自己遇到的问题,均在文中有所提到)

你可能感兴趣的:(大数据,centos,arm,ambari,hadoop)