centos7安装hadoop3.03和jdk1.8的高可用集群模式

centos7安装hadoop3.03和jdk1.8的高可用集群模式

规划三台服务器
配置如下：
master 192.168.145.200
slave1 192.168.145.201
slave2 192.168.145.202

添加用户hadoop3 并加入hadoop用户组中
安装目录如下：
/home/hadoop3
添加/home/hadoop3/app目录
安装如下软件:
/home/hadoop3/app/jdk
/home/hadoop3/app/hadoop
/home/hadoop3/data

配置SSH免密码通信
先在三台服务器中/home/hadoop3/目录下新建目录.ssh
Master主机上操作如下：
su hadoop3
cd /home/hadoop3
mkdir .ssh
ssh-keygen –t rsa
cd .ssh
ls
cat id_rsa.pub >>authorized_keys
cd ..
chmod 700 .ssh
chmod 600 .ssh/*
ssh master //第一次执行需要输入yes
ssh master //第二次执行就可以直接访问了

slave1和slave2主机操作如下：
su hadoop3
cd /home/hadoop3
mkdir .ssh
ssh-keygen –t rsa
cd .ssh
ls
cd ..
chmod 700 .ssh
chmod 600 .ssh/*

将另外2个节点中的共钥id_rsa.pub拷贝到master节点中的authorized_keys文件中
如：把slave1和slave2主机节点上的/home/hadoop3/.ssh/id_rsa.pub改名为id_rsa201.pub和id_rsa202.pub拷贝到主机master中/home/hadoop3/.ssh/中

[hadoop3@master .ssh]$
cat id_rsa201.pub >> authorized_keys
cat id_rsa202.pub >> authorized_keys
合并后authorized_keys内容为：
Ssh-rsa ……………………hadoop3@master
Ssh-rsa ……………………hadoop3@slave1
Ssh-rsa…………………….hadoop3@slave2

然后将master中的authorized_keys文件分发到其他2个节点上。

scp -r authorized_keys hadoop3@slave1:~/.ssh/
scp -r authorized_keys hadoop3@slave2:~/.ssh/
测试免密命令如下：
ssh master
ssh slave1
ssh slave2

三台主机/etc/hosts 配置内容为：
127.0.0.1 core localhost.localdomain localhost
192.168.145.200 master
192.168.145.201 slave1
192.168.145.202 slave2

脚本工具的使用
在master节点，hadoop3用户下创建/home/hadoop3/tools目录。

[hadoop3@master ~]$ mkdir /home/hadoop3/tools
cd /home/hadoop3/tools

    将本地脚本文件上传至/home/hadoop3/tools目录下，这些脚本大家可以自己写， 如果不熟练也可以直接使用。

[hadoop3@master tools]$ rz deploy.conf
[hadoop3@master tools]$ rz deploy.sh
[hadoop3@master tools]$ rz runRemoteCmd.sh
[hadoop3@master tools]$ ls
deploy.conf  deploy.sh  runRemoteCmd.sh

查看一下deploy.conf配置文件内容。

[hadoop3@master tools]$ cat deploy.conf
master,all,namenode,zookeeper,resourcemanager,
slave1,all,slave,namenode,zookeeper,resourcemanager,
slave2,all,slave,datanode,zookeeper,

查看一下deploy.sh远程复制文件脚本内容。

[hadoop3@master tools]$ cat deploy.sh
#!/bin/bash
#set -x
if [ $# -lt 3 ]
then 
  echo "Usage: ./deply.sh srcFile(or Dir) descFile(or Dir) MachineTag"
  echo "Usage: ./deply.sh srcFile(or Dir) descFile(or Dir) MachineTag confFile"
  exit 
fi
src=$1
dest=$2
tag=$3
if [ 'a'$4'a' == 'aa' ]
then
  confFile=/home/hadoop3/tools/deploy.conf
else 
  confFile=$4
fi
if [ -f $confFile ]
then
  if [ -f $src ]
  then
    for server in `cat $confFile|grep -v '^#'|grep ','$tag','|awk -F',' '{print $1}'` 
    do
       scp $src $server":"${dest}
    done 
  elif [ -d $src ]
  then
    for server in `cat $confFile|grep -v '^#'|grep ','$tag','|awk -F',' '{print $1}'` 
    do
       scp -r $src $server":"${dest}
    done 
  else
      echo "Error: No source file exist"
  fi
else
  echo "Error: Please assign config file or run deploy.sh command with deploy.conf in same directory"
fi

查看一下runRemoteCmd.sh远程执行命令脚本内容。
[hadoop3@master tools]$ cat runRemoteCmd.sh

#!/bin/bash
#set -x
if [ $# -lt 2 ]
then 
  echo "Usage: ./runRemoteCmd.sh Command MachineTag"
  echo "Usage: ./runRemoteCmd.sh Command MachineTag confFile"
  exit 
fi
cmd=$1
tag=$2
if [ 'a'$3'a' == 'aa' ]
then

  confFile=/home/hadoop3/tools/deploy.conf
else 
  confFile=$3
fi
if [ -f $confFile ]
then
    for server in `cat $confFile|grep -v '^#'|grep ','$tag','|awk -F',' '{print $1}'` 
    do
       echo "*******************$server***************************"
       ssh $server "source /etc/profile; $cmd"
    done 
else
  echo "Error: Please assign config file or run deploy.sh command with deploy.conf in same directory"
fi

   三个文件，方便我们搭建hadoop3分布式集群。具体如何使用看后面如何操作。
   如果我们想直接使用脚本，还需要给脚本添加执行权限。

[hadoop3@master tools]$ chmod u+x deploy.sh
[hadoop3@master tools]$ chmod u+x runRemoteCmd.sh

   同时我们需要将/home/hadoop3/tools目录配置到PATH路径中。

[hadoop3@master tools]$vi ~/.bashrc
PATH=/home/hadoop3/tools:$PATH
export PATH

如果执行runRemoteCmd.sh文件报错：解析器错误，执行下面命令:

Vi runRemoteCmd.sh
:set ff=unix
:wq

centos7中hadoop各节点时间同步方法

采用NTP(Network Time Protocol)方式来实现, 选择一台机器, 作为集群的时间同步服务器, 然后分别配置服务端和集群其他机器。我这里以master机器时间为准，其他机器同这台机器时间做同步。

(一）NTP服务端

安装ntp服务（在master服务器上)

sudo su -

yum install ntp -y

配置/etc/ntp.conf，这边采用本地机器作为时间的原点

注释server列表：

#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

添加如下内容：　　

server 127.127.1.0 prefer
fudge 127.127.1.0 stratum 8
logfile /var/log/ntp.log

启动ntpd服务

systemctl start ntpd

查看ntp服务状态

systemctl status ntpd

加入开机启动

systemctl enable ntpd

（二) NTP客户端 (在slave1和slave2服务器上)

安装ntp

yum install ntpdate

配置crontab任务主动同步

ssh slave1
sudo su -
yum install ntpdate
vi /etc/crontab

# crontab -e
*/10 * * * * /usr/sbin/ntpdate 192.168.145.200;hwclock -w

ssh slave2
sudo su -
yum install ntpdate
vi /etc/crontab

# crontab -e
*/10 * * * * /usr/sbin/ntpdate 192.168.145.200;hwclock -w

然后重新启动hadoop集群服务器reboot.

Zookeeper安装

安装到/home/hadoop3/app/zookeeper目录
zookeeper-3.4.12.tar.gz
tar –zxvf zookeeper-3.4.12.tar.gz
mv zookeeper-2.4.12 zookeeper
配置vi /home/hadoop3/app/zookeeper/conf/zoo.cfg
Cp zoo_sample.cfg zoo.cfg
Vi zoo.cfg
dataDir=/home/hadoop3/data/zookeeper/zkdata
dataLogDir=/home/hadoop3/data/zookeeper/zkdatalog

master主机配置方法

server.1=0.0.0.0:2888:3888
server.2=192.168.145.201:2888:3888
server.3=192.168.145.202:2888:3888

slave1主机配置方法

server.1=192.168.145.200:2888:3888
server.2=0.0.0.0:2888:3888
server.3=192.168.145.202:2888:3888

slave2主机配置方法

server.1=192.168.145.200:2888:3888
server.2=192.168.145.201:2888:3888
server.3=0.0.0.0:2888:3888

1 2 3代表服务编号；2888代表Zookeeper节点通信端口；3888代表zook选举端口

远程拷贝zookeeper
通过远程脚本deploy.sh将Zookeeper安装目录拷贝到其他节点

[hadoop3@master app]$ deploy.sh zookeeper /home/hadoop3/app/ slave

所有节点创建数据目录和日志目录

[hadoop3@master app]$ runRemoteCmd.sh "mkdir -p /home/hadoop3/data/zookeeper/zkdata" all  
[hadoop3@master app]$ runRemoteCmd.sh "mkdir -p /home/hadoop3/data/zookeeper/zkdatalog" all 

创建myid文件

Vi /home/hadoop3/data/zookeeper/zkdata/myid
在各个节点上，在 dataDir 所指定的目录下创一个名为 myid 的文件， 文件内容为各个server 点后面的数字。
[hadoop3@master zkdata]$ vi myid

[hadoop3@slave1 zkdata]$ vi myid

[hadoop3@slave2 zkdata]$ vi myid

测试运行

runRemoteCmd.sh “/home/hadoop3/app/zookeeper/bin/zkServer.sh start” zookeeper
runRemoteCmd.sh “jps” all
runRemoteCmd.sh “/home/hadoop3/app/zookeeper/bin/zkServer.sh status” all
如果一个节点为leader，另两个节点为follower，则说明Zookeeper安装成功。

三台主机vi ~/.bashrc文件内容为：

# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi
# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=
# User specific aliases and functions
#下面为新加的内容
# add to path for /home/hadoop3/app/jdk   and   /home/hadoop3/tools
JAVA_HOME=/home/hadoop3/app/jdk
HADOOP_HOME=/home/hadoop3/app/hadoop
ZOOKEEPER_HOME=/home/hadoop3/app/zookeeper
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
PATH=$JAVA_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$ZOOKEEPER_HOME/bin:/home/hadoop3/tools:$PATH
export JAVA_HOME CLASSPATH  HADOOP_HOME ZOOKEEPER_HOME PATH

编辑配置文件workers 从节点列表

vi /home/hadoop3/app/hadoop/etc/hadoop/workers
master
slave1
slave2

配置hadoop-env.sh

export JAVA_HOME=/home/hadoop3/app/jdk
export HADOOP_HOME=/home/hadoop3/app/hadoop

配置mapred-env.sh、yarn-env.sh

export JAVA_HOME=/home/hadoop3/app/jdk
export HADOOP_HOME=/home/hadoop3/app/hadoop

编辑配置文件core-site.xml

vi /home/hadoop3/app/hadoop/etc/hadoop/core-site.xml

<configuration>
    <property>
      <name>fs.defaultFSname>
      <value>hdfs://myclustervalue>
    property>
    <property>
      <name>hadoop.tmp.dirname>
      <value>/home/hadoop3/data/tmpvalue>
    property>
    <property>
       <name>ha.zookeeper.quorumname>
       <value>master:2181,slave1:2181,slave2:2181value>
    property>
configuration>

编辑配置文件hdfs-site.xml

Vi /home/hadoop3/app/hadoop/etc/hadoop/hdfs-site.xml

<configuration>
    <property>
      <name>dfs.nameservicesname>
      <value>myclustervalue>
    property>
    <property>
      <name>dfs.permissions.enabledname>
      <value>falsevalue>
    property>
    <property>
      <name>dfs.ha.namenodes.myclustername>
      <value>nn1,nn2,nn3value>
    property>
    <property>
      <name>dfs.namenode.rpc-address.mycluster.nn1name>
      <value>master:9820value>
    property>
    <property>
      <name>dfs.namenode.rpc-address.mycluster.nn2name>
      <value>slave1:9820value>
    property>
    <property>
      <name>dfs.namenode.rpc-address.mycluster.nn3name>
      <value>slave2:9820value>
    property>
    <property>
      <name>dfs.namenode.http-address.mycluster.nn1name>
      <value>master:9870value>
    property>
    <property>
      <name>dfs.namenode.http-address.mycluster.nn2name>
      <value>slave1:9870value>
    property>
    <property>
      <name>dfs.namenode.http-address.mycluster.nn3name>
      <value>slave2:9870value>
    property> 
      <name>dfs.namenode.shared.edits.dirname>
      <value>qjournal://master:8485;slave1:8485;slave2:8485/myclustervalue>
    property>
    <property>
      <name>dfs.journalnode.edits.dirname>
      <value>/home/hadoop3/data/journaldata/jnvalue>
    property>
    <property>
      <name>dfs.client.failover.proxy.provider.myclustername>
      <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvidervalue>
    property>
    <property>
      <name>dfs.ha.fencing.methodsname>
      <value>
           sshfence
           shell(/bin/true)
      value>
    property>
    <property>
      <name>dfs.ha.fencing.ssh.private-key-filesname>
      <value>/home/hadoop3/.ssh/id_rsavalue>
    property>
    <property>
      <name>dfs.ha.fencing.ssh.connect-timeoutname>
      <value>10000value>
    property>
    <property>
      <name>dfs.namenode.handler.countname>
      <value>100value>
    property>
    <property>
       <name>dfs.ha.automatic-failover.enabledname>
       <value>truevalue>
     property>
configuration>

编辑配置文件mapred-site.xml

Vi /home/hadoop3/app/hadoop/etc/hadoop/mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.namename>
        <value>yarnvalue>
    property>
    <property>
        <name>mapreduce.application.classpathname>
        <value>
            /home/hadoop3/app/hadoop/etc/hadoop,
            /home/hadoop3/app/hadoop/share/hadoop/common/*,
            /home/hadoop3/app/hadoop/share/hadoop/common/lib/*,
            /home/hadoop3/app/hadoop/share/hadoop/hdfs/*,
            /home/hadoop3/app/hadoop/share/hadoop/hdfs/lib/*,
            /home/hadoop3/app/hadoop/share/hadoop/mapreduce/*,
            /home/hadoop3/app/hadoop/share/hadoop/mapreduce/lib/*,
            /home/hadoop3/app/hadoop/share/hadoop/yarn/*,
            /home/hadoop3/app/hadoop/share/hadoop/yarn/lib/*
        value>
    property>
configuration>

编辑配置文件yarn-site.xml

Vi /home/hadoop3/app/hadoop/etc/hadoop/yarn-site.xml

<configuration>


    <property>
        <name>yarn.resourcemanager.connect.retry-interval.msname>
        <value>2000value>
    property>
    <property>
        <name>yarn.resourcemanager.ha.enabledname>
        <value>truevalue>
    property>
    <property>
        <name>yarn.resourcemanager.ha.automatic-failover.enabledname>
        <value>truevalue>
    property>
    <property>
        <name>yarn.resourcemanager.ha.automatic-failover.embeddedname>
        <value>truevalue>
    property>
    <property>
        <name>yarn.resourcemanager.cluster-idname>
        <value>yarn-rm-clustervalue>
    property>
    <property>
        <name>yarn.resourcemanager.ha.rm-idsname>
        <value>rm1,rm2value>
    property>
    <property>
        <name>yarn.resourcemanager.hostname.rm1name>
        <value>mastervalue>
    property>
    <property>
        <name>yarn.resourcemanager.hostname.rm2name>
        <value>slave1value>
    property>
    <property>
        <name>yarn.resourcemanager.recovery.enabledname>
        <value>truevalue>
    property>
    <property>
       <description>The class to use as the persistent store.description>
       <name>yarn.resourcemanager.store.classname>
       <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStorevalue>
     property>
    <property>
        <name>yarn.resourcemanager.zk.state-store.addressname>
        <value>master:2181,slave1:2181,slave2:2181value>
    property>
    <property>
        <name>yarn.resourcemanager.zk-addressname>
        <value>master:2181,slave1:2181,slave2:2181value>
    property>
    <property>
        <name>yarn.resourcemanager.address.rm1name>
        <value>master:8032value>
    property>
    <property>
        <name>yarn.resourcemanager.scheduler.address.rm1name>
        <value>master:8034value>
    property>
    <property>
        <name>yarn.resourcemanager.webapp.address.rm1name>
        <value>master:8088value>
    property>
    <property>
        <name>yarn.resourcemanager.address.rm2name>
        <value>slave1:8032value>
    property>
    <property>
        <name>yarn.resourcemanager.scheduler.address.rm2name>
        <value>slave1:8034value>
    property>
    <property>
        <name>yarn.resourcemanager.webapp.address.rm2name>
        <value>slave1:8088value>
    property>
    <property>
        <name>yarn.nodemanager.aux-servicesname>
        <value>mapreduce_shufflevalue>
    property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce_shuffle.classname>
        <value>org.apache.hadoop.mapred.ShuffleHandlervalue>
    property>
configuration>

格式化hdfs

先启动zookeeper

runRemoteCmd.sh “/home/hadoop3/app/zookeeper/bin/zkServer.sh start” all

接着启动journalnode

runRemoteCmd.sh “/home/hadoop3/app/hadoop/sbin/hadoop-daemon.sh start journalnode” all
runRemoteCmd.sh “/home/hadoop3/app/hadoop/sbin/hadoop-daemon.sh stop journalnode” all

在master节点上执行格式化

bin/hdfs namenode -format / /namenode 格式化
bin/hdfs zkfc -formatZK //格式化高可用
sbin/start-dfs.sh

备用节点slave1 slave2通过master节点元数据信息，分别在slave1、slave2节点上执行。

bin/hdfs namenode -bootstrapStandby
bin/hdfs namenode -bootstrapStandby

测试运行HDFS

查看HDFS Web界面，这里配置的master slave1 slave2节点都为namenode
http://192.168.145.200:9870
http://192.168.145.201:9870
http://192.168.145.202:9870
关闭active状态的namenode，检查是否会自动切换其他节点

测试hdfs文件上传

vi djt.txt
hadoop
hadoop
hadoop
dajiangtai
dajiangtai
dajiangtai
hsg
qq.com

bin/hdfs dfs -mkdir /dajiangtai
bin/hdfs dfs -put djt.txt /dajiangtai
bin/hdfs dfs -cat /dajiangtai/djt.txt

脚本分发修改的yarn配置

deploy.sh mapred-site.xml /home/hadoop3/app/hadoop/etc/hadoop/ slave
deploy.sh yarn-site.xml /home/hadoop3/app/hadoop/etc/hadoop/ slave

启动yarn

在master节点启动resourcemanager

bin/yarn –daemon start resourcemanager
在slave1节点启动resourcemanager
bin/yarn –daemon start resourcemanager
在3个节点分别启动nodemanager
runRemoeteCmd.sh “yarn –daemon start nodemanager” all

通过Web查看YARN

http://192.168.145.200:8088

检查ResourceManager状态

Bin/yarn rmadmin –getServiceState rm1
Bin/yarn rmadmin –getServiceState rm2
Bin/yarn rmadmin –getServiceState rm3

关闭active 状态的resourcemanager，检查另外一个节点是否能称为active状态。

测试运行WordCount

Cd /home/hadoop3/app/hadoop
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.3.jar wordcount /dajiangtai/djt.txt /dajiangtai/output

查看yarn的Web界面

查看运行结果

bin/hdfs dfs -cat /dajiangtai/output/*
hdfs dfs -cat /dajiangtai/output/*
dajiangtai 3
hadoop 3
hsg 1
qq.com 1
[hadoop3@master ~]$

   如果以上操作没有问题，说明Hadoop3.0分布式高可用集群成功搭建完毕。

参考1url: http://www.dajiangtai.com/community/18389.do
参考2url: https://blog.csdn.net/hliq5399/article/details/78193113