Ambari 与hdp 集群搭建教程

                Ambari 与hdp 集群搭建教程

集群环境os为centos6.8 server

 

1,各个节点进行:网络(静态IP)配制(

192.168.10.30

master

8g

192.168.10.31

slave1

8g

192.168.10.32

Slave2

8g

192.168.10.33

Slave3

8g

192.168.10.34

Slave4

8g

192.168.10.35

Slave5

8g

Vi /etc/sysconfig/network-scripts/ifcfg-eth0

:

DEVICE=eth0

HWADDR=00:50:56:A7:45:10

TYPE=Ethernet

UUID=010e7a3c-f81c-4914-a69a-01da66b9cfb3

ONBOOT=yes

NM_CONTROLLED=yes

BOOTPROTO=static

IPADDR=192.168.10.30

NETMASK=255.255.240.0

GATEWAY=192.168.2.1

DNS1=202.98.96.68

DNS2=202.98.96.69

 

 

Vi /etc/sysconfig/network

:

NETWORKING=yes

HOSTNAME=master

GATEWAY=192.168.2.1

 

 

重启网络服务:

/etc/init.d/network restart

 

2,Hosts配置

在各个节点:vi /etc/hosts

  添加集群主机:  

 

 

192.168.10.30  master

192.168.10.31  slave1

192.168.10.32  slave2

192.168.10.33  slave3

192.168.10.34  slave4

192.168.10.35  slave5

                  ………………………….

 

 

3,关闭防火墙和SELinux(各个节点)。

chkconfig iptables off (重启后生效)

  vi   /etc/selinux/config

     修改SELINUX=disabled (重启后永久生效

 

Ssh 无密码登录

master 节点下:

ssh-keygen -t rsa

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

chmod 600 ~/.ssh/authorized_keys

scp ~/.ssh/authorized_keys root@slave1:~/.ssh/

scp ~/.ssh/authorized_keys root@slave2:~/.ssh/

 

……………………..

 

 

4,Ntp 即时服务

 

  master 为ntp校准服务器

   Master  的 vi /etc/ntp.conf

:

 

restrict 127.0.0.1

restrict -6 ::1

 

# Hosts on local network are less restricted.

#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

 

 

restrict 192.168.10.0 mask 255.255.240.0 nomodify notrap  //允许的网络字段连接ip限制

 

 

server 127.127.1.0 perfer   //以本机时间为准

fudge 127.127.1.0 stratum 10

 

#server 0.centos.pool.ntp.org iburst  //全部注释掉

#server 1.centos.pool.ntp.org iburst

#server 2.centos.pool.ntp.org iburst

#server 3.centos.pool.ntp.org iburst

 

 

各个客户端(slave1,2,3,4,5)配置如下

Vi /etc/ntp.conf

 

  Server master   //指定ntp 服务器,为master

#server 0.centos.pool.ntp.org iburst  //全部注释掉

#server 1.centos.pool.ntp.org iburst

#server 2.centos.pool.ntp.org iburst

#server 3.centos.pool.ntp.org iburst

 

 

 

启动全部节点ntp服务;

 

service ntpd start

检查是否成功,用ntpstat命令查看同步状态,出现以下状态代表启动成功

 

请求服务器前,请先使用ntpdate手动同步一下时间:ntpdate -umaster (主节点ntp服务器)

这里可能出现同步失败的情况,请不要着急,一般是本地的ntp服务器还没有正常启动,一般需要等待5-10分钟才可以正常同步。启动服务:

service ntpd start

 

 

5,在各个节点jdk配置,:

rpm –ivh jdk-8u101-linux-x64.rpm

 此时jdk安装目录为/usr/java/….

配置jdk环境:

Vi /etc/profile

 

 

export JAVA_HOME=/usr/java/jdk1.8.0_101

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

 

export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin

 

 

source /etc/profile

 

 

 6,安装http服务器(主节点)

yum install httpd

service httpd start

#修改网站默认的跟目录

vi /etc/httpd/conf/httpd.conf

#

# DocumentRoot: Thedirectory out of which you will serve your

# documents. Bydefault, all requests are taken from this directory, but

# symbolic links andaliases may be used to point to other locations.

#

DocumentRoot"/var/www/html"

#

# This should bechanged to whatever you set DocumentRoot to.

#

var/www/html">

 

 

7,安装mysql(主节点)

rpm –ivh MySQL-server-5.5.36-1.el6.x86_64.rpm

rpm –ivh MySQL-client-5.5.36-1.el6.x86_64.rpm

service mysql start

service mysql start

4.mysqladmin -u root password 'xxxx'

5.mysql -uroot -pxxxx   (进入mysql命令行)

进入mysql命令行之后,开始执行下面sql语

#创建用户并分配权限

CREATE USER 'ambari'@'master'IDENTIFIED BY 'ambari';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'master';

GRANT ALL PRIVILEGES ON *.* TO root@'master';

FLUSH PRIVILEGES;

 

 

 

8,创建用户组及用户(在主节点)

 

 Groupadd ambari

添加ambari安装、运行用户到组.

useradd -G ambari ambari

 

 

9, 配置安装源

:鉴于多数集群生产环境不能上网,所以安装时采用本地源的方式.具体为先下载附录中的ambariHDPHDP-UTILStar包然后再配置到本地源中。

#ambariHDP(HDP-UTILS)

a)Ambari2.2.2

操作系统

格式

URL地址

RedHat 6

CentOS 6

Oracle Linux 6

Base URL

http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.2.0

Repo File

http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.2.0/ambari.repo

Tarball md5 | asc

http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.2.0/

ambari-2.2.2.0-centos6.tar.gz

 

b)HDP 2.4

操作系统

版本号

源名称

格式

URL地址

RedHat 6

CentOS 6

Oracle Linux 6

HDP-2.4.2.0

HDP

Base URL

http://public-repo-1.hortonworks.com/HDP/centos6/2.x/

updates/2.4.2.0

Repo File

http://public-repo-1.hortonworks.com/HDP/centos6/2.x/

updates/2.4.2.0/hdp.repo

Tarball md5 | asc

http://public-repo-1.hortonworks.com/HDP/centos6/2.x/

updates/2.4.2.0/HDP-2.4.2.0-centos6-rpm.tar.gz

HDP-UTILS

Base URL

http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/

repos/centos6

Repo File

http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/

repos/centos6/HDP-UTILS-1.1.0.20-centos6.tar.gz

 

1)  下载如下组件并拷贝到主节点的httpd网站根目录,/var/www/html内后解压

Ambari 2.2.2:

http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.2.0/ambari-2.2.2.0-centos6.tar.gz

HDP 2.4.2:

http://s3.amazonaws.com/public-repo-1.hortonworks.com/HDP/centos6/2.x/2.4-latest/HDP-2.4-latest-centos6-rpm.tar.gz

HDP-UTILS-1.1.0.20

http://s3.amazonaws.com/public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6/HDP-UTILS-1.1.0.20-centos6.tar.gz

2) 配置ambariHDPHDP-UTILS的源

:事实证明HDPHDP-UTILS源不需要配置,ambari安装时可以指定本地源,并在各个节点里创建repo文件.ambari的源需要配置,因为安装采用yum install ambari方式.

vi ambari.repo(http目录下,即解压目录:/var/www/html/)

#VERSION_NUMBER=2.2.2.0-460

[Updates-ambari-2.2.2.0]

name=ambari-2.2.2.0 -Updates

baseurl=http://192.168.10.30/AMBARI-2.2.2.0/centos6/2.2.2.0-460/

gpgcheck=1

gpgkey=http://192.168.10.30/AMBARI-2.2.2.0/centos6/2.2.2.0-460/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

enabled=1

priority=1

 

vi hdp.repo(http目录下,即解压目录:/var/www/html/)

#VERSION_NUMBER=2.4.2.0-258

[HDP-2.4.2.0]

name=HDP Version -HDP-2.4.2.0

baseurl=http://192.168.10.30/HDP/centos6/2.x/updates/2.4.2.0

gpgcheck=1

gpgkey=http://192.168.10.30/HDP/centos6/2.x/updates/2.4.2.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

enabled=1

priority=1

[HDP-UTILS-1.1.0.20]

name=HDP Utils Version- HDP-UTILS-1.1.0.20

baseurl=http://192.168.10.30/HDP-UTILS-1.1.0.20/repos/centos6

gpgcheck=1

gpgkey=http://192.168.10.30/HDP/centos6/2.x/updates/2.4.2.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

enabled=1

priority=1

 

ambari.repo ,hdp.repo复制到/etc/ yum.repos.d/目录下

接着执行以下指令:

yum clean all

yum list update

yum makecache

yum repolist

 

10. 执行安装

Yum方式安装ambari

 安装ambari2.2.2

1)安装ambari

yum install ambari-server

2)配置amabri

ambari-server setup
Customize user account for ambari-server daemon [y/n] (n)?
输入:y
Enter user account for ambari-server daemon (root):

:这里亦可选择用户amabri,届时会让提供该用户的密码等信息.
输入:root

检查防火墙是否关闭
Adjusting ambari-server permissions and ownership...
Checking firewall...
WARNING: iptables is running. Confirm the necessary Ambari ports areaccessible. Refer to the Ambari documentation for more details on ports.

OK to continue [y/n] (y)?
输入y

检查JDK
Checking JDK...

[1] - Oracle JDK 1.7 + Java Cryptography Extension (JCE)Policy Files 7
[2] - Oracle JDK 1.6 + Java Cryptography Extension (JCE) Policy Files 6
[3] - Custom JDK
==============================================================================
Enter choice (1):
输入3
输入:/自己的jdk安装目录

 Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL
[4] - PostgreSQL
==============================================================================
Enter choice (1):
输入3

Hostname (localhost):
Port (3306):
Database Name (ambari):
Username (ambari):
输入:(什么也不输入,直接回车

Enter Database Password (cluster):
Re-enter password:
输入ambari

WARNING: Before starting Ambari Server, youmust run the following DDL against the database to create the schema:/var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
:这里需要以mysqlamabri用户在ambari数据库里source上面的sql文件,作用时初始化ambari数据库。

Proceed with configuring remote database connectionproperties [y/n] (y)?
输入y

简单图示如下:

 

 Ambari 与hdp 集群搭建教程_第1张图片


 

 Ambari 与hdp 集群搭建教程_第2张图片



Ambari 与hdp 集群搭建教程_第3张图片Ambari 与hdp 集群搭建教程_第4张图片Ambari 与hdp 集群搭建教程_第5张图片



注意:启动ambari server之前一定要运行上述提到的Ambari-DDL-MySQL-CREATE.sql脚本,否则ambari无法创建数据库,无法启动。

 

登录mysql,执行如下命令:

"source/var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql"

11,启动ambari服务

ambari-server start

Using python  /usr/bin/python2.6

Starting ambari-server

Ambari Server running with 'root' privileges.

Organizing resource files at/var/lib/ambari-server/resources...

Server PID at: /var/run/ambari-server/ambari-server.pid

Server out at: /var/log/ambari-server/ambari-server.out

Server log at: /var/log/ambari-server/ambari-server.log

Waiting for server start....................

Ambari Server 'start' completed successfully.

:启动ambari访问前,请确保mysql驱动已经放置在/usr/share/java内且名字是mysql-connector-java.jar不然ambari server启动时会报错mysql驱动找不到的错误.具体报错见:

 

 

12,访问master192.168.10.30:8080进入安装界面,用户密码:admin/admin

 

1,选 launch install wizard

   Ambari 与hdp 集群搭建教程_第6张图片

2,配置本地源,如下图示;指定本地源,修改base url,master,如下图中红线所示;否则会联网下载,安装失败。

 Ambari 与hdp 集群搭建教程_第7张图片

 

3,配置集群主机,将主节点master下的.ssh文件下id_rsa作为ssh private key,如下图示:

 

 Ambari 与hdp 集群搭建教程_第8张图片

 

 

 

 

4,主机检查

 Ambari 与hdp 集群搭建教程_第9张图片

 

主机检查时出现TransparentHuge Pages Issues (1)

解决方法:执行下面的命令可临时解决这个警告:

echo never >/sys/kernel/mm/redhat_transparent_hugepage/defrag

永久解决:

vi  /etc/rc.local

在文件的最后添加下面的行,保存即可。

echo never >/sys/kernel/mm/redhat_transparent_hugepage/defrag

echo never >/sys/kernel/mm/transparent_hugepage/enabled

 5,选择服务;

 

 Ambari 与hdp 集群搭建教程_第10张图片

 

6,选择服务安装的server

Ambari 与hdp 集群搭建教程_第11张图片Ambari 与hdp 集群搭建教程_第12张图片

 Ambari 与hdp 集群搭建教程_第13张图片

在选server host 时,自己将几乎所有server 都放在了master节点上,包括hbase server ,hive server2 /metastore  ,namenode,secondary,hdfs等,除了accumulo,其他可以按配置放。

 

 

 

 

 

 

 

7,注意配置hive config,如下图示,一定要选mapreduce

 Ambari 与hdp 集群搭建教程_第14张图片

Ambari 与hdp 集群搭建教程_第15张图片



测试之前在mysql执行以下命令:

CREATE USER 'hive'@'master' IDENTIFIED BY 'hive';

GRANT ALL PRIVILEGES ON *.* TO 'hive'@'master' IDENTIFIED BY 'hive' WITH GRANT OPTION;

Flush privileges

 

 

8,开始安装

 

 Ambari 与hdp 集群搭建教程_第16张图片

 Ambari 与hdp 集群搭建教程_第17张图片

 安装较慢,估计40分钟左右

  出错时,按提示解决:

9.安装成功后如下示:

 Ambari 与hdp 集群搭建教程_第18张图片

 

10,Sqoop测试成功。

 Ambari 与hdp 集群搭建教程_第19张图片

 

安装参考链接:http://blog.csdn.net/shenliang1985/article/details/51478151

集群扩容新增硬盘:链接:http://www.centoscn.com/CentOS/config/2014/0403/2711.html


你可能感兴趣的:(大数据,平台,集群)