Apache Ambari 2.7.4 + HDP3.1.4 安装

版本信息

名称	版本
OS	CentOS7
JDK	1.8
Ambari	2.7.4
HDP	3.1.4
HDP-UTILS	1.1.0.22

机器配置

主机名称	ip	备注
hadoop33	192.168.0.33	master，仓库
hadoop81	192.168.0.81	slave1
hadoop90	192.168.0.90	slave2

一、在三台服务器分别修改hosts

vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.33 hadoop33
192.168.0.81 hadoop81
192.168.0.90 hadoop90

二、在三台服务器分别修改hostname（需要重启才能生效）

# 修改hadoop33
hostnamectl set-hostname hadoop33
# 修改hadoop81
hostnamectl set-hostname hadoop81
# 修改hadoop90
hostnamectl set-hostname hadoop90

三、配置免密

# 持续回车即可
ssh-keygen -t rsa
# 交换密钥
ssh-copy-id hadoop33 # 注意：本机也需要执行
ssh-copy-id hadoop81
ssh-copy-id hadoop90

四、安装JDK

tar -zxvf openjdk-8u40-b25-linux-x64-10_feb_2015.tar.gz -C /usr/local/tools
vim /etc/profile
export JAVA_HOME=/usr/local/tools/java-se-8u40-ri/
export PATH=$PATH:$JAVA_HOME/bin

source /etc/profile

五、HDP下载地址

http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.4.0/ambari-2.7.4.0-centos7.tar.gz
http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/HDP-3.1.4.0-centos7-rpm.tar.gz
http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz

六、安装依赖

yum -y remove mariadb-libs
yum -y install psmisc ntp ntpdate httpd createrepo wget git

七、关闭防火墙

# 关闭防火墙
systemctl stop firewalld
# 关闭开机自启
systemctl disable firewalld

八、关闭SELinux

setenforce 0
vim /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted 

九、同步时间

crontab -e
*/1 * * * * /usr/sbin/ntpdate ntp.sjtu.edu.cn

十、解压安装包到/var/www/html目录下，准备制作本地源

mkdir /var/www/html/hdp-utils
tar -zxvf /root/ambari-2.7.4.0-centos7.tar.gz -C /var/www/html/
tar -zxvf /root/HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/hdp-utils/
tar -zxvf /root/HDP-3.1.4.0-centos7-rpm.tar.gz -C /var/www/html/

十一、开启httpd服务

# 开启httpd服务
systemctl start httpd
# 开启开机自启
systemctl enable httpd

十二、制作本地源

createrepo /var/www/html/

十三、检查站点是否做好

curl http://hadoop33/ambari/ | grep centos
curl http://hadoop33/HDP/ | grep centos
curl http://hadoop33/hdp-utils/ | grep HDP-UTILS

十四、安装mysql

# 卸载mariadb
yum -y remove mariadb-libs

# 安装mysql
wget http://repo.mysql.com/mysql57-community-release-el7.rpm
rpm -ivh mysql57-community-release-el7.rpm
yum install -y mysql-server

# 启动mysql
systemctl start mysqld
# 设置开机自启
systemctl enable mysqld
# 获取密码
cat /var/log/mysqld.log | grep password

# 为了可以设置简单密码
set global validate_password_policy=0;
set global validate_password_length=4;
# 立即修改密码，执行其他操作报错：
SET PASSWORD FOR 'root'@'localhost' = PASSWORD('abc123');
# 我们创建密码为abc123
 
mysql -uroot -pabc123
CREATE USER 'root'@'%' IDENTIFIED BY 'abc123';
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%';
CREATE USER 'root'@'localhost' IDENTIFIED BY 'abc123';
GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost';
CREATE USER 'root'@'hadoop33' IDENTIFIED BY 'abc123';
GRANT ALL PRIVILEGES ON *.* TO 'root'@'hadoop33';  //本地主机名
FLUSH PRIVILEGES;

## 创建HDP需要用到的几个数据库
create database ambari default character set = 'utf8';
create database hive default character set = 'utf8';
create database oozie default character set = 'utf8';

十五、下载mysql驱动包

mkdir -p /usr/share/java/
wget -O /usr/share/java/mysql-connector-java.jar https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.27/mysql-connector-java-5.1.27.jar

十六、配置本地源repo

vim /etc/yum.repos.d/ambari.repo
[ambari-2.7.4.0]
name=ambari-2.7.4.0
baseurl=http://hadoop33/ambari/centos7/2.7.4.0-118
gpgcheck=0
gpgkey=http://hadoop33/ambari/centos7/2.7.4.0-118/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

vim /var/www/html/hdp-utils/hdp-util.repo
[HDP-UTILS-1.1.0.22]
name=Hortonworks Data Platform Version - HDP-UTILS-1.1.0.22
baseurl=http://hadoop33/hdp-utils/HDP-UTILS/centos7/1.1.0.22
gpgcheck=0
enabled=1
priority=1

vim /var/www/html/HDP/centos7/hdp.repo
#VERSION_NUMBER=3.1.4.0-315
[HDP-3.1.4.0]
name=HDP Version - HDP-3.1.4.0
baseurl=http://hadoop33/HDP/centos7/3.1.4.0-315
gpgcheck=0
gpgkey=http://hadoop33/HDP/centos7/3.1.4.0-315/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://hadoop33/hdp-utils/HDP-UTILS/centos7/1.1.0.22
gpgcheck=0
gpgkey=http://hadoop33/hdp-utils/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

十七、在hadoop33节点安装ambari-server

yum -y install ambari-server
cp /usr/share/java/mysql-connector-java.jar /var/lib/ambari-server/resources/mysql-jdbc-driver.jar
vim /etc/ambari-server/conf/ambari.properties
server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar

十八、使用ambari的sql脚本生成表

mysql -uroot -pabc123
use ambari
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
quit

十九、ambari-server初始化

ambari-server setup

Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'permissive'
WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
OK to continue [y/n] (y)? y
# 第一步输入y，开启守护进程
Customize user account for ambari-server daemon [y/n] (n)? y
# 第二步输入root，设置ambari-server的用户是root
Enter user account for ambari-server daemon (root):root
# 第三步选择jdk，如果没有安装jdk可以输入1默认安装一个，但是我们此时选择2使用我们已经安装过的jdk
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? y
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
# 当上一步输入的是2的时候，让你选择你之前安装过jdk的JAVA_HOME的路径，我们输入/usr/local/tools/java-se-8u40-ri
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/local/tools/java-se-8u40-ri
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
# 第5步输入y进入高级数据库配置，我们这里选择3使用mysql
Checking GPL software agreement...
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost): hadoop33
Port (3306): 3306
Database name (ambari): ambari
Username (ambari): ambari
Enter Database Password (bigdata): 
Re-enter password: 
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
# 配置远程数据库连接属性
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.7.4.0.118.jar
....
Ambari repo file doesn't contain latest json url, skipping repoinfos modification
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

二十、启动ambari-server

ambari-server start

Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.......................
Server started listening on 8080

DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.

# 这时候可以通过 http://hadoop33:8080 链接访问 ambari server, 默认账户为(admin/admin)

二十一、分别在三台服务器安装ambari-agent

# 分发本地源ambari.repo到其他两台服务器
scp /etc/yum.repos.d/ambari.repo [email protected]:/etc/yum.repos.d/
scp /etc/yum.repos.d/ambari.repo [email protected]:/etc/yum.repos.d/
# 分别在三台服务器安装ambari-agent
yum -y install ambari-agent

二十二、修改ambari-agent的配置文件

vim /etc/ambari-agent/conf/ambari-agent.ini
[server]
hostname=hadoop33 # 将localhost改为ambari-server的主机名，我这里的ambari-server主机名是hadoop33
url_port=8440
secured_url_port=8441
connect_retry_delay=10
max_reconnect_retry_delay=30

二十三、分别在三台服务器启动ambari-agent

ambari-agent start

Verifying Python version compatibility...
Using python  /usr/bin/python
Checking for previously running Ambari Agent...
Checking ambari-common dir...
Starting ambari-agent
Verifying ambari-agent process status...
Ambari Agent successfully started
Agent PID at: /run/ambari-agent/ambari-agent.pid
Agent out at: /var/log/ambari-agent/ambari-agent.out
Agent log at: /var/log/ambari-agent/ambari-agent.log