Hadoop-HA高可用

一、集群规划

二、HDFS高可用

官方地址

1. 在opt目录下创建一个ha文件夹，将/opt/module/下的 hadoop-3.1.3拷贝到/opt/ha目录下（记得删除data 和 log目录）
   
   

2. 配置core-site.xml
   

3. hdfs-site.xml

  
  
    dfs.namenode.name.dir
    file://${hadoop.tmp.dir}/name
  

  
  
    dfs.datanode.data.dir
    file://${hadoop.tmp.dir}/data
  

  
  
    dfs.journalnode.edits.dir
    ${hadoop.tmp.dir}/jn
  

  
  
    dfs.nameservices
    mycluster
  

  
  
    dfs.ha.namenodes.mycluster
    nn1,nn2,nn3
  

  
  
    dfs.namenode.rpc-address.mycluster.nn1
    hadoop102:8020
  
  
    dfs.namenode.rpc-address.mycluster.nn2
    hadoop103:8020
  
  
    dfs.namenode.rpc-address.mycluster.nn3
    hadoop104:8020
  

  
  
    dfs.namenode.http-address.mycluster.nn1
    hadoop102:9870
  
  
    dfs.namenode.http-address.mycluster.nn2
    hadoop103:9870
  
  
    dfs.namenode.http-address.mycluster.nn3
    hadoop104:9870
  

  
  
    dfs.namenode.shared.edits.dir
qjournal://hadoop102:8485;hadoop103:8485;hadoop104:8485/mycluster
  

  
  
    dfs.client.failover.proxy.provider.mycluster
    org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
  

  
  
    dfs.ha.fencing.methods
    sshfence
  

  
  
    dfs.ha.fencing.ssh.private-key-files
    /home/atguigu/.ssh/id_rsa
  

4. 分发配置好的hadoop环境到其他节点
   
   
5. 启动HDFS-HA集群
   将HADOOP_HOME环境变量更改到HA目录(三台机器)
   
   
   去三台机器上source环境变量
   
6. 在各个JournalNode节点上，输入命令启动journalnode服务
   
7. 在[nn1]上，对其进行格式化，并启动；在[nn2]和[nn3]上，同步nn1的元数据信息
   
   
   
   
8. 启动[nn2]和[nn3]
   
   
9. 查看web页面显示
   
   
   
10. 在所有节点上，启动datanode
    
    
    
11. 将[nn1]切换为Active
    

三、 HDFS-HA自动故障转移工作机制

自动故障转移为HDFS部署增加了两个新组件：ZooKeeper和ZKFailoverController（ZKFC）进程，如图所示。ZooKeeper是维护少量协调数据，通知客户端这些数据的改变和监视客户端故障的高可用服务。

1. 架构规划
   

2. 修改hdfs-site.xml， 启用nn故障自动转移
   

3. 在core-site.xml文件中增加，zkfc要连接的zkServer地址
   

4. 修改后分发配置文件
   

5. 启动
   关闭所有HDFS服务
   启动Zookeeper集群
   
   
   

6. 启动Zookeeper以后，然后再初始化HA在Zookeeper中状态
   

7. 启动HDFS
   
   
   
   验证：将Active NameNode进程kill，查看网页端三台Namenode的状态变化

四、配置YARN-HA集群

1. 集群规划
   
2. 修改yarn-site.xml

    
        yarn.nodemanager.aux-services
        mapreduce_shuffle
    

    
    
        yarn.resourcemanager.ha.enabled
        true
    
 
    
    
        yarn.resourcemanager.cluster-id
        cluster-yarn1
    

    
    
        yarn.resourcemanager.ha.rm-ids
        rm1,rm2,rm3
    

    
    
        yarn.resourcemanager.hostname.rm1
        hadoop102
    

    
    
        yarn.resourcemanager.webapp.address.rm1
        hadoop102:8088
    

    
    
        yarn.resourcemanager.address.rm1
        hadoop102:8032
    

    
    
        yarn.resourcemanager.scheduler.address.rm1  
        hadoop102:8030
    

      
    
    yarn.resourcemanager.resource-tracker.address.rm1
        hadoop102:8031
    


    
    
        yarn.resourcemanager.hostname.rm2
        hadoop103
    
    
        yarn.resourcemanager.webapp.address.rm2
        hadoop103:8088
    
    
        yarn.resourcemanager.address.rm2
        hadoop103:8032
    
    
        yarn.resourcemanager.scheduler.address.rm2
        hadoop103:8030
    

    
yarn.resourcemanager.resource-tracker.address.rm2
        hadoop103:8031
    


    
    
        yarn.resourcemanager.hostname.rm3
        hadoop104
    
    
    
        yarn.resourcemanager.webapp.address.rm3
        hadoop104:8088
    
    
    
        yarn.resourcemanager.address.rm3
        hadoop104:8032
    
    
    
        yarn.resourcemanager.scheduler.address.rm3  
        hadoop104:8030
    

      
    
    yarn.resourcemanager.resource-tracker.address.rm3
        hadoop104:8031
    

     
    
        yarn.resourcemanager.zk-address
        hadoop102:2181,hadoop103:2181,hadoop104:2181
    

     
    
        yarn.resourcemanager.recovery.enabled
        true
    
 
     
    
        yarn.resourcemanager.store.class     org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore


    
    
        yarn.nodemanager.env-whitelist
        JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME
    

3. 启动zk yarn
   zk.sh start
   start-yarn.sh