hadoop3.1.3集群搭建(ha+yarn)

当前环境：

centos6.5,jdk8

准备工作：

1.服务器之间免密登录

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys

2.服务器之间时间同步

3.安装zookeeper集群

搭建步骤：

1.下载apache hadoop3.1.3并上传至服务器解压

https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-3.1.3/hadoop-3.1.3.tar.gz

2.环境变量设置：

export BASE_DIR=/data/br
export JAVA_HOME=$BASE_DIR/base/jdk1.8.0_181
export HADOOP_HOME=$BASE_DIR/base/hadoop
export ZOOKEEPER_HOME=/data/br/base/zookeeper

export HDFS_JOURNALNODE_USER=root
export HDFS_ZKFC_USER=root
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_NODEMANAGER_USER=root
export YARN_RESOURCEMANAGER_USER=root

export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin:$KAFKA_HOME/bin

3.hadoop-env.sh设置

export JAVA_HOME=/data/br/base/jdk1.8.0_181

4.core-site.xml配置

    fs.defaultFS
    hdfs://mycluster
  
  
    hadoop.tmp.dir
    /data/br/cache/hadoop/ha
  
  
  
     ha.zookeeper.quorum
     bonree01:2181,bonree02:2181,bonree03:2181
   

5.hdfs-site.xml配置

  
    dfs.replication
    2
  
  
    dfs.nameservices
    mycluster
  
  
    dfs.ha.namenodes.mycluster
    nn1,nn2
  
  
    dfs.namenode.rpc-address.mycluster.nn1
    bonree01:8020
  
  
    dfs.namenode.rpc-address.mycluster.nn2
    bonree02:8020
  
  
    dfs.namenode.shared.edits.dir
    qjournal://bonree01:8485;bonree02:8485;bonree03:8485/mycluster
  
  
    dfs.client.failover.proxy.provider.mycluster
    org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
  
  
    dfs.ha.fencing.methods
    sshfence
  
  
    dfs.ha.fencing.ssh.private-key-files
    /root/.ssh/id_dsa
  
  
    dfs.ha.automatic-failover.enabled
    true
  

6.mapred-site.xml设置

  
    mapreduce.framework.name
    yarn
  
  
        mapreduce.application.classpath
        $HADOOP_HOME/share/hadoop/mapreduce/*:$HADOOP_HOME/share/hadoop/mapreduce/lib/*
    

7.yarn-site.xml配置

    yarn.nodemanager.aux-services
    mapreduce_shuffle


    yarn.nodemanager.env-whitelist
    JAVA_HOME,HADOOP_HOME



  yarn.resourcemanager.ha.enabled
  true



  yarn.resourcemanager.cluster-id
  rmhacluster1



  yarn.resourcemanager.ha.rm-ids
  rm1,rm2



  yarn.resourcemanager.hostname.rm1
  bonree02


  yarn.resourcemanager.hostname.rm2
  bonree03



  yarn.resourcemanager.webapp.address.rm1
  bonree02:8088


  yarn.resourcemanager.webapp.address.rm2
  bonree03:8088



  yarn.resourcemanager.zk-address
  bonree01:2181,bonree02:2181,bonree03:2181

8.works指定datanode

bonree01
bonree02
bonree03

以上配置拷贝到所有集群上，配置完成，若需要添加其他配置参数，参考官网：https://hadoop.apache.org/docs/r3.1.3/hadoop-project-dist/hadoop-common/ClusterSetup.html

 

二、格式化相关操作（前提zookeeper集群已启动）

1.在指定的所有journalnode机器上执行命令启动journalnode

   hdfs --daemon start journalnode

2.在某台机器上执行namenode格式化（zk进程所在节点）

   hdfs namenode -format

3.在所在某一台namenode机器执行，启动namenode

   hdfs --daemon start namenode

4.在其余namenode机器上执行，同步active namenode信息，作为secondarynamenode

   hdfs namenode -bootstrapStandby

5.在active namenode所在节点执行，初始化zookeeper上NameNode的状态

  hdfs zkfc -formatZK

6.start-dfs.sh 启动ha，在指定的resource manager所在机器上执行start-yarn.sh启动resourcemanager、nodemanager

再次检查每台机器进程是否都正常运行，搭建完成。

hdfs web url：http://bonree01:9870, 

resourcemanager web url：http://bonree02:8088

说明：搭建说明及配置简陋，仅保证能正常运行。仅供参考。